PostgreSQL compression, TOAST, and LZ4 storage

The TOAST boundary

PostgreSQL does not store every large value directly inside the main heap tuple. When a value becomes too large, TOAST can compress it, move it out of line, or do both, depending on the column storage strategy and the data.

That means compression experiments need to look past the visible table definition. A text column, JSON document, or bytea value may live partly in a per-table TOAST table, and the size change may appear there rather than where the test first looks.

The practical question is not "did I enable compression?" It is "which values crossed the threshold, how compressible were they, and where did PostgreSQL store the result?"

pglz and LZ4

PostgreSQL has historically used pglz compression for TOASTed values. Newer versions can use LZ4 when PostgreSQL is built with support for it and the column or default configuration selects it.

LZ4 is usually attractive because it is fast. It may not always produce the smallest possible output, but storage systems often care about CPU cost as much as compression ratio. A slower method that saves a little more space can be the wrong trade-off for hot data.

The right method depends on the value shape. Repetitive JSON or long text may compress well. Already-compressed images, encrypted blobs, and many binary formats may not.

Why size may not change

A trivial test can appear to show that compression does nothing. Usually the test is measuring the wrong thing or using values that do not cross the relevant storage boundary.

Small values may remain inline. Values can also be too random to compress. Existing rows may not be rewritten until an update, rewrite, or table maintenance operation touches them. Indexes and heap overhead may dominate the total relation size in a tiny dataset.

That is why a useful test needs enough rows, realistic values, and size measurements for the heap, TOAST table, and indexes separately.

Designing an experiment

A good experiment inserts representative payloads into two comparable tables or columns, one using the baseline compression and one using the candidate method. It records row counts, average value sizes, total relation size, TOAST relation size, and query timings.

The data should include the shapes expected in production: short values, medium values, very large values, repetitive text, and less-compressible content. A thousand copies of the same string will prove that compression works, but not whether it helps the real application.

After changing compression settings, make sure the rows are actually rewritten. Otherwise the test may compare metadata rather than stored values.

Interpreting results

The best outcome is not always the smallest table. Compression changes storage, CPU, cache behavior, vacuum work, and I/O. For cold archival values, size may dominate. For frequently read values, decompression cost and latency matter.

Look for clear wins. If LZ4 saves meaningful space with no measurable read penalty, it is attractive. If it barely changes size, leave the simpler default alone. If compression saves space but moves CPU into a hot request path, the application may need a different data model instead.

TOAST is an implementation detail until performance or storage pressure makes it visible. Once visible, it deserves the same measurement discipline as any other database change.

Checking compression in practice

PostgreSQL compression is mostly invisible until row size, TOAST storage, or backup volume becomes a problem. The first step is to measure the column and table, not to assume compression is helping.

SELECT
  pg_size_pretty(pg_total_relation_size('documents')) AS total_size,
  pg_size_pretty(pg_relation_size('documents')) AS heap_size;

SELECT id, pg_column_size(body) AS body_bytes
FROM documents
ORDER BY body_bytes DESC
LIMIT 20;

On PostgreSQL versions that support column compression choices, the setting can be made explicit. That still does not mean every value will compress well. Already-compressed JSON blobs, images, or encrypted payloads may resist compression.

ALTER TABLE documents
ALTER COLUMN body SET COMPRESSION lz4;

VACUUM FULL documents;

The rewrite matters: changing the setting affects future storage, while existing rows may need to be rewritten before the space profile changes. Compression is a storage policy plus a data lifecycle, not a single switch.

Cost of changing storage

Changing compression policy can be expensive because storage layout changes when rows are rewritten. On a large table, that can mean locks, vacuum pressure, WAL volume, replica lag, and backup churn. The operational plan matters as much as the setting.

A safer rollout starts with one table or partition, records before-and-after sizes, and watches query latency for rows that are frequently read. Faster compression is not automatically better if the workload is dominated by small values that never leave the main heap or by data that was already compressed before it reached PostgreSQL.

Compression decisions also interact with application behavior. If a column is read on every request, decompression cost can matter more than disk savings. If it is mostly archival, space reduction may be worth a slower occasional read.