Extract any file

File compression reduces redundancy so the same information takes fewer bits. The upper bound on how far you can go is governed by information theory: for lossless compression, the limit is the entropy of the source (see Shannon’s source coding theorem and his original 1948 paper “A Mathematical Theory of Communication”). For lossy compression, the trade-off between rate and quality is captured by rate–distortion theory.

Two pillars: modeling and coding

Most compressors have two stages. First, a model predicts or exposes structure in the data. Second, a coder turns those predictions into near-optimal bit patterns. A classic modeling family is Lempel–Ziv: LZ77 (1977) and LZ78 (1978) detect repeated substrings and emit references instead of raw bytes. On the coding side, Huffman coding (see the original paper 1952) assigns shorter codes to more likely symbols. Arithmetic coding and range coding are finer-grained alternatives that squeeze closer to the entropy limit, while modern Asymmetric Numeral Systems (ANS) achieves similar compression with fast table-driven implementations.

What common formats actually do

DEFLATE (used by gzip, zlib, and ZIP) combines LZ77 with Huffman coding. Its specs are public: DEFLATE RFC 1951, zlib wrapper RFC 1950, and gzip file format RFC 1952. Gzip is framed for streaming and explicitly does not attempt to provide random access. PNG images standardize DEFLATE as their only compression method (with a max 32 KiB window), per the PNG spec “Compression method 0… deflate/inflate… at most 32768 bytes” and W3C/ISO PNG 2nd Edition.

Zstandard (zstd): a newer general-purpose compressor designed for high ratios with very fast decompression. The format is documented in RFC 8878 (also HTML mirror) and the reference spec on GitHub. Like gzip, the basic frame doesn’t aim for random access. One of zstd’s superpowers is dictionaries: small samples from your corpus that dramatically improve compression on many tiny or similar files (see python-zstandard dictionary docs and Nigel Tao’s worked example). Implementations accept both “unstructured” and “structured” dictionaries (discussion).

Brotli: optimized for web content (e.g., WOFF2 fonts, HTTP). It mixes a static dictionary with a DEFLATE-like LZ+entropy core. The spec is RFC 7932, which also notes a sliding window of 2^WBITS−16 with WBITS in [10, 24] (1 KiB−16 B up to 16 MiB−16 B) and that it does not attempt random access. Brotli often beats gzip on web text while decoding quickly.

ZIP container: ZIP is a file archive that can store entries with various compression methods (deflate, store, zstd, etc.). The de facto standard is PKWARE’s APPNOTE (see APPNOTE portal, a hosted copy, and LC overviews ZIP File Format (PKWARE) / ZIP 6.3.3).

Speed vs. ratio: where formats land

LZ4 targets raw speed with modest ratios. See its project page (“extremely fast compression”) and frame format. It’s ideal for in-memory caches, telemetry, or hot paths where decompression must be near RAM speed.

XZ / LZMA push for density (great ratios) with relatively slow compression. XZ is a container; the heavy lifting is typically LZMA/LZMA2 (LZ77-like modeling + range coding). See .xz file format, the LZMA spec (Pavlov), and Linux kernel notes on XZ Embedded. XZ usually out-compresses gzip and often competes with high-ratio modern codecs, but with slower encode times.

bzip2 applies the Burrows–Wheeler Transform (BWT), move-to-front, RLE, and Huffman coding. It’s typically smaller than gzip but slower; see the official manual and man pages (Linux).

Windows, blocks, and random access

“Window size” matters. DEFLATE references can only look back 32 KiB (RFC 1951 and PNG’s 32 KiB cap noted here). Brotli’s window ranges from about 1 KiB to 16 MiB (RFC 7932). Zstd tunes window and search depth by level (RFC 8878). Basic gzip/zstd/brotli streams are designed for sequential decoding; the base formats don’t promise random access, though containers (e.g., tar indexes, chunked framing, or format-specific indexes) can layer it on.

Lossless vs. lossy

The formats above are lossless: you can reconstruct exact bytes. Media codecs are often lossy: they discard imperceptible detail to hit lower bitrates. In images, classic JPEG (DCT, quantization, entropy coding) is standardized in ITU-T T.81 / ISO/IEC 10918-1. In audio, MP3 (MPEG-1 Layer III) and AAC (MPEG-2/4) rely on perceptual models and MDCT transforms (see ISO/IEC 11172-3, ISO/IEC 13818-7, and an MDCT overview here). Lossy and lossless can coexist (e.g., PNG for UI assets; Web codecs for images/video/audio).

Practical tips

• Pick for the job. Web text and fonts: brotli. General files and backups: zstd (great decompression speed and levels to trade time for ratio). Ultra-fast pipes and telemetry: lz4. Maximum density for long-term archives where encode time is OK: xz/LZMA.
• Small files? Train and ship dictionaries with zstd (docs) / (example). They can shrink dozens of tiny, similar objects dramatically.
• Interoperability. When exchanging multiple files, prefer a container (ZIP, tar) plus a compressor. ZIP’s APPNOTE defines method IDs and features; see PKWARE APPNOTE and LC overviews here.
• Measure on your data. Ratios and speeds vary by corpus. Many repos publish benchmarks (e.g., LZ4’s README cites Silesia corpus here), but always validate locally.

Key references (deep dives)

Theory: Shannon 1948 · Rate–distortion · Coding: Huffman 1952 · Arithmetic coding · Range coding · ANS. Formats: DEFLATE · zlib · gzip · Zstandard · Brotli · LZ4 frame · XZ format. BWT stack: Burrows–Wheeler (1994) · bzip2 manual. Media: JPEG T.81 · MP3 ISO/IEC 11172-3 · AAC ISO/IEC 13818-7 · MDCT.

Bottom line: choose a compressor that matches your data and constraints, measure on real inputs, and don’t forget the gains from dictionaries and smart framing. With the right pairing, you can get smaller files, faster transfers, and snappier apps — without sacrificing correctness or portability.