jvq, short for just very quickly, is the studio’s nineteenth project: a performance budget gate for CI. Most teams that run benchmarks in CI compare today’s numbers with last week’s, taken on another machine, and shared runners can drift 5 to 20% between runs. The alarms stop meaning anything. jvq measures the base commit and the change in the same job, taking turns, so both sides see the same machine. Version 0.1.0 came out on 18 October for Go benchmarks, as one binary with no dependencies, with binaries for Linux, macOS and Windows and a GitHub Action that checks the download’s checksum. It’s open source under Apache 2.0.
A run checks out both commits into git worktrees, runs the benchmarks in alternating rounds after a warm-up, and estimates the slowdown with a rank-based confidence interval. Only when the whole interval sits over the budget, 5% by default, does the pull request fail. When the interval straddles the line, jvq adds rounds, up to 30 a side, and if it still can’t tell it says so and passes. Bytes and allocations per op get budgets of their own, down to “no new allocation”.
The test write-up runs the real binary hundreds of times on a benchmark whose cost is set by hand. On a noisy two-core cloud machine it raised no alarm in 200 runs with no change and caught 42 of 50 runs at 10% more work and every run at 20%. On a GitHub-hosted runner, where the noise was 0.4%, it caught all 50 runs at 10%. One extra allocation per op was caught in 100 runs of 100 on both.
Next is any language: timing a whole command, a small JSON format, and comparing two saved reports. Adapters for Rust, C++, Python and JavaScript benchmark tools follow in 0.3.