How Time-Series Database Compression Keeps Historians Fast

A single machine can produce millions of samples per day. Historians stay manageable because they compress aggressively - and knowing how helps you decide what to store, at what resolution, and for how long.

Two Kinds of Compression

Lossless compression encodes the raw samples so they can be reproduced exactly. Techniques include delta encoding (storing the difference between consecutive values) followed by entropy coding, plus run-length encoding for repeated values. Industrial data compresses well because most values change slowly.

Lossy compression discards information the consumer will not miss. The most common approach in historians is swinging-door compression: samples are kept only when the value leaves a tolerance band around the line defined by the previous kept points. The result is a piecewise-linear approximation with a guaranteed maximum error.

Tuning the Compression Deviation

The compression deviation is the key parameter. Set it too small and you store noise; too large and you distort peaks that matter for analysis. A good practice is to set the deviation relative to the sensor's accuracy and to keep a separate, uncompressed stream for critical safety or quality signals.

Practical Guidance

• Store raw data at the edge for short retention (hours to days) and compressed aggregates for the long term.
• Choose deadband and sample rates per signal class, not globally - temperatures and vibration spectra have very different information content.
• Use database-native compression plus OS-level deduplication carefully; double compression rarely helps and complicates backups.