Notes/RollupDB

RollupDB Launches, an Aggregate Database That Keeps Years of Statistics and Lets the Raw Events Go

RollupDB, the studio’s eleventh project, is an aggregate database in a single file. You declare up front which statistics you’ll want, and it keeps those for years while the raw readings go. Old buckets fold into coarser ones as they age: seconds into minutes, minutes into hours, hours into days. Version 0.1 came out on 17 October as one Go binary for Linux, macOS and Windows, with a Go library and a small loopback HTTP service on the same engine. The same day, 0.1.1 followed: it refuses a clock that jumps far ahead unless you lift the limit, and gives HTTP snapshots more time. The code is open source under Apache 2.0.

Count, sum, min, max and mean are exact. Variance is stored so it merges correctly across buckets, and percentiles come from a KLL sketch, labelled approximate, with the error model in each answer. In one recorded test, five years of readings from two services, 2.8 million of them, took about 207 MB as raw JSON lines and 15.4 MB in RollupDB. Half the readings gave a 15.3 MB file. What moves the size is history kept and the number of series: eight services instead of two made it 16.3 MB.

The test write-up checks the numbers against a reference computed outside the database. Over five years the exact statistics matched it exactly, and the 99th percentile stayed well inside its stated error. Five kill tests, each run three times, cut the process off mid-write; every reopen kept what was acknowledged and nothing else. That’s one run on one 4-CPU Linux runner, and it covers process death, not power loss.

The limits are plain. A breakdown you didn’t declare can’t be rebuilt later, there’s no SQL or clustering, rollups run when you call advance, and one process owns a file at a time. Where Precomputing keeps answers ready inside a SQLite file, RollupDB is built for the long tail: years of statistics in a file that stays small.