# Stream VByte: Faster Byte-Oriented Integer Compression Source: [Stream VByte: Faster Byte-Oriented Integer Compression]() Collection index: [llms.txt]() Data updated: 2025-10-28T13:56:34.383Z Authors: Daniel Lemire, Nathan Kurz, Christoph Rupp Published: 2017-09-25T14:45:20.000Z ## Abstract Arrays of integers are often compressed in search engines. Though there are many ways to compress integers, we are interested in the popular byte-oriented integer compression techniques (e.g., VByte or Google's Varint-GB). They are appealing due to their simplicity and engineering convenience. Amazon's varint-G8IU is one of the fastest byte-oriented compression technique published so far. It makes judicious use of the powerful single-instruction-multiple-data (SIMD) instructions available in commodity processors. To surpass varint-G8IU, we present Stream VByte, a novel byte-oriented compression technique that separates the control stream from the encoded data. Like varint-G8IU, Stream VByte is well suited for SIMD instructions. We show that Stream VByte decoding can be up to twice as fast as varint-G8IU decoding over real data sets. In this sense, Stream VByte establishes new speed records for byte-oriented integer compression, at times exceeding the speed of the memcpy function. On a 3.4GHz Haswell processor, it decodes more than 4 billion differentially-coded integers per second from RAM to L1 cache. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Sparse Retrieval Efficiency on MSMARCO v1 (test)](): Sparse Retrieval Efficiency; MSMARCO v1 (test); Latency @ 90% Acc (µs): 402. ## Original sources and implementation [Original paper]() [PDF]()