Notes/jvq

jvq Adds a Second Gate That Quarantines Flaky Tests Without Hiding Real Failures

jvq 0.7.0 adds a second gate to the same binary. jvq flaky reads JUnit XML and go test -json from each CI run, keeps a small history, and calls a test flaky only when it passed and failed on the same commit. Two flaky commits in its last 100 runs put a test in quarantine: its failures still show up and still get annotated, but they stop blocking the merge. Thirty passing runs in a row take it back out. A test that fails three attempts in a row counts as broken and blocks again, and a test that always fails never goes in.

In a simulation of 300 commits with five flaky tests among 200 stable ones, red first runs fell from 53.7% to 14.6%, and no stable test was ever quarantined. With two retries, a quarantined test that really broke was blocked on the run that broke it, 50 times in 50.

An independent review found five bugs, fixed in 0.7.1. The worst let a compile error pass: build failures shared an entry with run-time package failures, so once those flaked, a real build break could sit in quarantine. Build failures now have an entry of their own that can never be quarantined. The review also closed a way for crafted test names to inject GitHub Actions commands, and on pull_request_target runs jvq now reads the checked-out commit instead of the base branch’s. The case study replays 26 of jvq’s own CI pushes across Linux, macOS and Windows: nothing was quarantined, and a Windows test bug was blocked four pushes in a row, as it should be. The setup guide covers GitHub Actions, pytest and Go retries. Binaries are out for Linux, macOS and Windows. The launch note covers the performance budget gate.