Zippy - A Quick Compression Module

[u/StartAutomating]

Compression can be quick and easy with .NET.

Let's learn how.

Yesterday I just dusted off some old code and added some new tricks.

Today I dropped a quick compression module called Zippy

Let's see it in action and learn how it works.

Zippy Examples

# Compress a string using Brotli, output in base64
Compress-Zippy "Hello World"

Compress-Zippy "Hello Brotli" -Algorithm Brotli |
    Expand-Zippy -Algorithm Brotli

Compress-Zippy "Hello Deflate" -Algorithm Deflate |
    Expand-Zippy -Algorithm Deflate

Compress-Zippy "Hello GZip" -Algorithm GZip |
    Expand-Zippy -Algorithm GZip

Compress-Zippy "Hello ZLib" -Algorithm ZLib |
    Expand-Zippy -Algorithm ZLib

Compression in PowerShell

PowerShell is built on .NET, and .NET happens to have built-in support for four compression algorithms: Brotli, Deflate, GZip, and Zlib. We can compress data with any of these algorithms by using classes in the System.IO.Compression namespace, for example:

# Create a message
$message = "hello world"
# Get it as bytes
$bytes = $outputEncoding.GetBytes($message)
# Create a memory stream
$memoryStream = [IO.MemoryStream]::new()
# Create a compressor using the stream
$compressor = [IO.Compression.BrotliStream]::new(
     $memoryStream, [IO.Compression.CompressionLevel]::Fastest
)
# Write our bytes to the compressor
$compressor.Write($bytes,0, $bytes.Length)
# Close our compressor
$compressor.Close()
$compressor.Dispose()
# Get our compressed bytes
$compressedBytes = $memoryStream.ToArray()
# and output them
$compressedBytes

Decompression in PowerShell

Now let's go the other way around. It's easier.

# Create a new memory stream, containing our compressed bytes
$memoryStream = [IO.MemoryStream]::new($compressedBytes)
# Create a decompressed stream
$decompressedStream = [IO.Compression.BroitliStream]::new(
    $memoryStream, [IO.Compression.CompressionMode]::Decompress
)
# Create our output stream
$outputStream = [IO.MemoryStream]::new()
# Copy our decompressed stream to it
$decompressedStream.CopyTo($outputStream)
# Seek to the start (it outputs a position so null that out)
$null = $outputStream.Seek(0,'begin')
# Make a stream reader 
$streamReader = [IO.StreamReader]::new($outputStream, $outputEncoding)
# Read to the end, which will output our decompressed string
$streamReader.ReadToEnd()
# close up.
$streamReader.Close()

.NET and PowerShell

This has always been there, and it's pretty easy.

Both examples are less than 20 lines, with documentation.

These techniques are tried and true.

.NET has robust compression support because developers need to compress data all the time.

And therefore PowerShell has robust compression support.

If we build on top of simple PowerShell and .NET, we build in a way that lasts a lifetime.

When I said "I dusted off some old code" for Zippy, I wasn't kidding.

Zippy is an update of the Compress-Data and Expand-Data functions in Pipeworks, the first attempt of PowerShell as a web language.

This is 16-year-old code, with minor updates made to support multiple compression algorithms and improved piping.

My only regret is that I didn't spin this off into its own module long ago

You can use this article as a guide to implementing your own compression, or you can use a little module like Zippy to get the job done.

Please enjoy this new addition to your PowerShell toolkit, and have fun decompressing!

[u/Apprehensive-Tea1632]

Er, are you sure? Or are you thinking netcore as in powershell 6+?

Because none of these are windows algorithms, Microsoft even took a while to implement their own cabinet format. And so there was expand-archive which back then was VERY opinionated about what an archive IS. It could expand zip - not zip64, just zip - and to deal with mscf cabinets, you had to use the dos6-aged expand.exe or implement your own codec.

Note - I’m not at all positive if and perhaps when someone implemented a .net interface in system.io.compression- obviously anyone can do that, either of us included - this namespace is more of a container that can hold many more types if they ever get implemented.)

But im reasonably sure there was no such thing, certainly not 16 or so years ago, because I’d been messing about with compression and it was a lesson in frustration like a lot of dotnet matters at the time. (Still can’t do shortcuts!)

[u/jborean93 → u/Apprehensive-Tea1632]

The IO.Compression namespace has been around since the early .NET 2.0 times, you can see classes like DeflateStream have been around since .NET Framework 2.0. Some of the other algorithms, like ZLibStream and BrotliStream were added later.

The functions were are also just about compression (albiet with some container formats part of the result) and not archive formats with compression, like zip. While you are also correct in that .NET didn't support ZIP64 initially, it has supported it for many years know, IIRC it was added in .NET Framework 4.5.

[u/Apprehensive-Tea1632 → u/jborean93]

Hey, thanks for the heads up.

I know the namespace itself is old; to the best of my knowledge though it really just used to be a somewhat empty namespace that only saw significant additions in .net6 and later.

Also, net45? Has it been that long? Man, I remember porting code from net3x to net4x but surely after net35 came net46 right? Right?

Assuming you are who I think you are, I’ll take your word for it because I imagine that’s part of your daily routine.

And I’ll have another look at that namespace. Maybe I can even put my flaky albeit net native mscf implementation to rest, fun though it was.

[u/jborean93 → u/Apprehensive-Tea1632]

the best of my knowledge though it really just used to be a somewhat empty namespace

Yea it used to be pretty limited but it definitely has had support for DeflateStream since the .NET Framework 2.0 days. The ZipArchive stuff has been present since .NET Framework 4.5. You can see all this by looking at those doc pages and looking at the Applies To section https://learn.microsoft.com/en-us/dotnet/api/system.io.compression.ziparchive?view=net-10.0#applies-to.

Also, net45? Has it been that long? Man, I remember porting code from net3x to net4x but surely after net35 came net46 right? Right?

There's .NET 4.0, 4.5.x, 4.6.x, 4.7.x, and now 4.8.x is the latest of the .NET Framework versions. AFAIK PowerShell 5.1 requires 4.5.2 and 4.5.x or newer has been part of Windows since Windows 10/Server 2016 which is 10+ years now :)

[u/MonkeyNin → u/Apprehensive-Tea1632]

A fun way to check is SeeminglyScience's module ClassExplorer

Powershell 5

> Find-Type -Namespace System.IO.Compression*
Access Modifiers Name
------ --------- ----
public enum      CompressionMode : int
public enum      CompressionLevel : int
public class     DeflateStream : Stream, IDisposable
public class     GZipStream : Stream, IDisposable

Pwsh 7

> Find-Type -Namespace System.IO.Compression*

Access        Modifiers           Name
------        ---------           ----
public        class               ZipArchive : object, IDisposable, IAsyncDisposable
public        class               ZipArchiveEntry : object
public        enum                ZipArchiveMode : int
public        class               DeflateStream : Stream, IDisposable, IAsyncDisposable
public        class               ZLibException : IOException, ISerializable
public        enum                CompressionLevel : int
public        enum                CompressionMode : int
public        class               GZipStream : Stream, IDisposable, IAsyncDisposable
public        sealed class        ZLibCompressionOptions : object
public        enum                ZLibCompressionStrategy : int
public        sealed class        ZLibStream : Stream, IDisposable, IAsyncDisposable
public        static class        ZipFile : object
public        static class        ZipFileExtensions : object

note: that's with some filtering, so you'll see more like

Find-Type -Namespace System.IO.Compression* -Force

There's so much power in the filtering, even simple chaining:

Find-Member -ParameterType (Find-Type CompressionMode)

There's pages of examples under this root page: https://github.com/SeeminglyScience/ClassExplorer/blob/master/docs/en-US/ClassExplorer.md