Crucible: the machine farm and its distributed tasks

← Foundry architecture · Yuchao Cao · October 2026
How a change gets from a coding agent to the protected master: tested in parallel on a farm of machines, merged one at a time.

Bug database one bug = one step PM dispatches Agent A own git worktree Agent B own git worktree Agent C own git worktree dispatch Check-in queue candidates waiting submit branch, move to next bug Farm: 4 machines, each pulls its own work testbox006 1 rebase onto current master 2 static gates, then 28,000+ tests 3 every test inside its time budget testbox007 1 rebase onto current master 2 static gates, then 28,000+ tests 3 every test inside its time budget testbox009 1 rebase onto current master 2 static gates, then 28,000+ tests 3 every test inside its time budget testbox010 1 rebase onto current master 2 static gates, then 28,000+ tests 3 every test inside its time budget atomic claim: one candidate per machine Integrator single, serialized: one merge at a time Protected master no direct pushes fast-forward merge pass fail, conflict or too slow: master untouched, bug reopens with the failing tests merged: bug resolved, the next dependent step is unblocked
Testing is distributed; merging is not. Each farm machine runs a small service that claims one candidate at a time from the shared queue, so there is no central dispatcher. A machine that reboots simply resumes, and adding a machine adds throughput. Only the integrator writes to master, so two passing changes can never merge on top of each other untested.
Why a farm: the same 8 check-ins, one machine against four 1 machine testbox006 testbox007 testbox009 testbox010 integrator farm done here: about 1/4 of the time each block is one full-suite run (median about 13.5 minutes) green ticks: merges, still one at a time
Each full-suite run takes about 13.5 minutes at the median. Four machines finish four of them in the time one machine finishes one. Merges are seconds long, so serializing them costs almost nothing.
Measured (Crucible records, 2026)
Full-suite runs in one week743, on 4 machines
Run timemedian 809 s, mean 946 s
Machine time that weekabout 195 hours, about 8 days of serial testing
Check-ins since July 20269,400+: 5,600 merged, 2,800 stopped before master