JSON Compress, lossless JSON compression whose output is still JSON

Send a thousand records over an API and you send a thousand copies of every key name. A status with four possible values is spelled out a thousand times, and so is a currency that is always "GBP". Minifying takes out the whitespace, which is the small part of the problem. The repetition stays.

JSON Compress is our open-source library for the repetition. It writes each key name once, turns arrays of like-shaped objects into columns and rows, and stores repeated values once. On record-shaped data the result is typically 45 to 75% smaller than minified JSON, and it comes back byte for byte. The output is still JSON, so it goes wherever the original went: an HTTP body, a WebSocket frame, a Postgres jsonb column, a queue message.

What it does to a document

There are four transformations, and each can be turned off:

A key catalogue. Every distinct key name is written once and replaced by a short token, with the busiest key getting the shortest one.
Columnar tables. An array of like-shaped objects becomes a column list and a matrix of rows. This is the biggest win, and it is why record sets compress far better than nested configuration.
Constant and dictionary columns. A column whose value never changes is stored once. A column drawn from a small set of strings becomes a short list plus small integers.
A string catalogue. Anything else that repeats is written once, but only when the reference costs fewer bytes than the copies would. That is measured, not guessed.

Four records are too few to show a saving, but they show the shape:

[
  { "id": 1, "name": "Ada", "role": "admin", "team": "core" },
  { "id": 2, "name": "Bob", "role": "admin", "team": "core" },
  { "id": 3, "name": "Cy",  "role": "user",  "team": "core" },
  { "id": 4, "name": "Dee", "role": "user",  "team": "core" }
]

becomes

{
  "jc": 1,
  "k": ["id", "name", "role", "team"],
  "s": ["admin", "user"],
  "d": {
    "~t": [0, 1, 2, 3],
    "~e": [0, 0, 1, 2],
    "~cv": ["core"],
    "~r": [[1, "Ada", 0], [2, "Bob", 0], [3, "Cy", 1], [4, "Dee", 1]]
  }
}

At four thousand rows the key names, the word "core" and the two roles are still written exactly once each.

The numbers, including the bad one

These come from javascript/examples/benchmark.js in the repository, which round-trips every case before it reports a size:

An API record set of 1,000 users: 206.2 KB minified, 54.8 KB compressed, 73% smaller. With gzip on both, still 20% smaller than gzip alone.
A time series of 2,000 readings: 73% smaller, and 19% over gzip.
1,500 structured log lines: 76% smaller, and 13% over gzip.
A nested API response of orders with lines: 59% smaller, and 6% over gzip.
GeoJSON with 400 features: 46% smaller, and 5% over gzip.
A small config file with little repetition: 29% smaller, but 22% larger than gzip alone.

The last row is there on purpose. Most JSON meets gzip on its way out of a server, and gzip already removes much of this. On a small document with little repetition the envelope costs more than it saves. measure() and the stats command tell you which case you are in before you change anything.

Lossless, including the awkward cases

Getting 70% on tidy data is easy. The documents nobody thought about are where a library like this succeeds or fails. Key order survives, even inside packed tables. A missing key stays different from a null one. Keys that look like numbers keep their position. A key named __proto__ comes back as data. Circular references throw instead of hanging the process.

Every test run also builds 2,000 seeded random documents, with ragged rows and hostile key names, and compares each round trip byte for byte with what JSON itself produces. All sixteen combinations of options run against the same documents.

Six languages, the same bytes

The format is written down in a specification, and there are six implementations of it. Each was written from the specification rather than translated from the JavaScript, and given the same document they all write the same bytes. So you can compress in a Python job and restore in a Go worker, or compress in the browser and restore in .NET.

npm install @tech-style/json-compress
dotnet add package TechStyle.JsonCompress
pip install json-compress
go get github.com/eharain/JSON-Compress/go
cargo add json-compress

JavaScript, .NET, Python, Go and Rust are released at 1.0.3. PHP passes the same tests and installs from the repository until its Packagist listing is ready. Java is planned. CI checks every port against a shared corpus of 144 documents in both directions: each must restore all 144 envelopes and must write the same 144 itself. None of the implementations has a dependency.

In a browser there is no build step at all. The bundle is served by jsDelivr straight from the repository, about 21 KB minified and 7 KB over the wire. Our online JSON minifier loads it from that same URL and does all its work in the page, so nothing you paste is uploaded.

Where it earns its place

It does most for JSON that travels: paginated API responses, WebSocket and SSE traffic with no per-message compression, localStorage with its small quota, JSON columns, structured logs, and queue messages that hit a broker's size limit. For a small, varied config file, leave it alone.

JSON Compress is MIT licensed. It is a Tech Style product: tech-style.co/product-json-compress. The source and the specification are at github.com/eharain/JSON-Compress.

Search This Blog

Tech Style