CodeInstances Adds Shrink and Test, Narrowing a Failure Down and Keeping It as a Regression Test

CodeInstances reached its third release on its first day. Version 0.3 adds two commands from the same step of its plan. codeinstances shrink takes a failing run and a passing one and finds which of the failing answers cause the failure. codeinstances test replays a folder of recorded runs as regression tests. Binaries are out for Linux, macOS and Windows.

Shrink is delta debugging over recorded answers. It replays the program offline with the passing run’s answers swapped in for some of the failing ones, keeps narrowing, and stops when no single answer left can go without the failure going away. What’s left is written out as a small failing instance. Test then holds two rules: a run that passed must replay exactly, and a recorded failure must now exit 0 against the answers that broke it.

In the recorded demo, 22 answers differed between a good run and one where the service broke a single item. Shrink found that one answer in 10 runs and 339 ms, and test failed the program with the bug and passed a fixed build. 77 Go tests pass, the race detector is clean, and an independent review before release made the run limit a hard one and stopped Ctrl-C from being counted as a result. Grouping repeated failures into one bug is next on the plan.