jvq, the studio’s CI gate that fails a pull request when it makes benchmarks slower than their budget, read only Go’s benchmark output in its first release. Version 0.2.0 opens it to any language. A time adapter times the whole command, so a Python script or a compiled binary can be gated with no change to its code, and a json adapter reads a line per result that any benchmark can print. A new jvq compare judges two saved outputs with the same statistics. The release has binaries for Linux, macOS and Windows.
One change goes deeper than the adapters. When a run prints several values for one benchmark, jvq now takes their median as that run’s single value, since values from one run share that run’s machine and aren’t independent.
The release also adds recorded tests in harder settings. With each run 2% slower than the last and nothing changed, running all of base before all of head faked a regression in 37 of 50 runs on a two-core VM and 50 of 50 on a GitHub runner; jvq’s turn-taking order faked none. Beside a process holding half the CPUs, and on Python scripts through both new adapters, it raised no false alarm on either machine.
Adapters for Rust’s criterion, Google Benchmark, pytest-benchmark and vitest, and a gate on binary size, are next in 0.3. The first release is covered in https://softdevstudio.com/jvq-launches-a-ci-gate-that-fails-pull-requests-which-make-benchmarks-slower/.