31% Smaller and Up to 83% Faster Than LZ4

lzvolt reads LZ4's own blocks — not a similar format, the actual bytes liblz4 wrote, per the LZ4 block specification. On 20 of 27 measured shapes it reads them faster than liblz4 does. So the cheapest adoption is: keep liblz4 as your packer, change your decoder, stop. LZ4_decompress_safe becomes lzvolt_decompress_lz4, same arguments. Stored data never moves, the write path is untouched, and there is nothing to roll back because nothing was converted. For many systems that is the right place to stay — decoding is what you do a thousand times per write. Re-encoding into the native format is a second, optional step. That is where the 31% and the factor of two live. And it is reversible: decode with the new one, pack with the old one. The round trip is bounded by liblz4's encoder — 2.51 GiB/s against 11.86 — so a terabyte of logical data rolls back in about seven core-minutes. Two rules if you convert. Readers first: an LZ4 decoder correctly rejects a native stream, so one unrestarted replica spoils the day. And while both formats coexist, keep a flag per value — there is deliberately no magic number, so a blob can't be sniffed.

Capacity and throughput multiply rather than add: 44.7% more live data in the same RAM and 36% faster per core. On 64 KiB records that product is +101.5% on the M2 Max, +106.6% on Neoverse V2, +96.3% on Xeon and +79.1% on EPYC. In purchasing terms, a cache holding a given working set at a given read rate needs about half the boxes. It is not confined to the best-compressing shape, either. On varied 64 KiB data the capacity is 1.4% worse than LZ4, and the combination is still +62% to +89%, carried by decode alone. And my sizes include a length header, while LZ4's bare blocks carry no length at all. A standalone LZ4 value means the frame format, which costs 23 bytes, flat. The one record cell I lose — 512 bytes, by 1.6% — I win by 9.3% against a self-contained LZ4 value.

No new algorithm. LZ77 with the same two primitives everyone uses, literal runs and back-references, plus three decisions about spending bits that the usual implementations don't make. Two token shapes. LZ4's token is one byte: four bits of literal-run length, four of match length. On varied 4 KiB records, 49% of match lengths fall below the four-bit ceiling and 47% above — astride it, so the deciding branch keeps changing direction and the predictor is useless. The format therefore carries two layouts, and a stream may switch between them exactly once. The wide layout trades two literal bits for a fifth match bit, moving the escape from 19 bytes to 35, plus one bit for a repeat flag: same offset as last time. The length lives in the stream. A one-to-four byte header, shaped like QUIC's variable-length integers, carries the decoded length, and the shortest encoding is mandatory. The decoder allocates once, exactly, and treats truncation as an error instead of a plausible short answer. The packer refuses. A quick probe skips non-repetitive data without running a match search, and a threshold of one eighth decides the rest. Below it the value is stored raw and compress returns false, because reading a packed value costs four to five times reading a stored one. liblz4 has no such step.

                      saved >= 8 && saved * 8 >= original
                    

A decoder is a loop over its input. On 64 KiB records mine walks 5524 bytes where liblz4 walks 7992, for the same 65536 out. Give a loop 31% less input and you have done 31% less of nearly everything. The size win and the speed win are one result measured twice. Three more mechanisms. A third of blocks never load an offset — 493 of 1567 carry the repeat flag. A single 32-bit load carries token, offset and match length when the literal run is empty: 56% of blocks on varied 4 KiB, 88% on 64 KiB records. And above a threshold memcpy beats a hand-written loop, because glibc's is tuned by people who do nothing else. The kernel is latency-bound, not issue-bound: the copy loop runs at IPC 5.37 against an issue width of six. Deleting an instruction buys nothing; deleting a dependent load from the critical chain buys three cycles. One honest caveat: I have not ablated these mechanisms. The individual numbers are measured; their ranking is inference, not a measured decomposition.

Three machines, three runs each, medians of three, interleaved against liblz4 in one process — same bytes, same machine, same minute. Decoding the native format: eighteen cells, eighteen wins, +14% to +83%. Reading LZ4's own blocks wins 20 of 27; the seven losses cluster on small incompressible values on x86, where I give up 12–15%. On size, four shapes of six are smaller than LZ4 and two are larger: varied 64 KiB by 1.4%, 512-byte records by 1.6%. liblz4 is the stock distribution package, while my side is hand-written assembly dispatched per part line — honest for deployment, not a like-for-like comparison of effort. CPU governor, turbo and core pinning were not controlled, so treat single-digit margins with suspicion. The codec ships five separately assembled decoder bodies, because identical changes measured opposite signs on different parts: aligning the source before the copy is +49% on Neoverse V2 and −7.1% on the M2 Max, both aarch64. A benchmark number with no machine name attached is noise. I measured against liblz4 and lz4_flex — not Oodle's Selkie, lzturbo, Snappy, LZO or zstd at negative levels. So, precisely: on the format the industry standardised on, across four processors, LZ4 is no longer the fastest thing that reads it. That is all the numbers support.

About twenty deliberate changes measured zero or negative. Three carried the project, and all three deleted work rather than making existing work cheaper. The measuring harness lied four times, the last through 4 KiB aliasing: same binary, 32% apart, because malloc placed the buffers differently in every process. It now uses one arena at fixed offsets, takes the median of five ratios and prints a resolution floor of 6% per cell. Three bugs survived green test suites, among them a decoder that executed 512 bytes of its own lookup table as machine code, and a provenance violation only Miri found. Every test started from a stream the packer wrote; an attacker has no such restriction. Now assembly changes are accepted only after deliberate corruption turns the suite red, and four cargo-fuzz targets ran 115,090,390 executions under AddressSanitizer with no crashes — not a proof of absence. The broader lesson: every late problem was a claim about the code that nobody had executed. Run the sentence before you write it down.