What mixing taught me about evals
Reading copy. The published version of this note lives on the lab at coconutlabs.org/research/mixing-and-evals. That copy is canonical.
Mixing a track and evaluating an AI system are the same job wearing different clothes: judgment under unreliable perception. I stopped treating that as a cute analogy the month I gave my synthesizer a regression suite. The instrument I make music with now has more automated checks than most services I have shipped, and building that harness taught me the mapping in both directions. Ears teach evals. Evals teach ears.
Ears teach evals
Nobody mixes in a vacuum. You keep reference tracks, finished songs in the lane you are aiming for, and you flip to them constantly to recalibrate what good sounds like. That is a golden set: known-quality exemplars consulted on every change, guarding your standards against fatigue drift. The mixer's core move, the instant A/B, is pairwise judgment, the most reliable shape there is: comparisons hold where absolute scores wander.
And every mixer learns their ears are a biased judge. Louder always wins unfairly, so you volume-match before comparing. That is verbosity bias with a gain knob. Late-night ears flatter everything, so you bounce it and listen tomorrow. That is offline evaluation on a fresh judge. A track that works in the studio must survive the car and the phone speaker, so you check translation. That is production monitoring. Music solved judgment-under-bias with structure, not talent, and eval builders are reinventing the same structures for text.
Evals teach ears
Here is the direction I did not expect. Building syn1, my Audio Unit synth, I wanted the discipline my ears cannot provide. Ears cannot hear a two percent regression. A harness can. So the DSP core carries a test suite the way a data platform carries one, and its checks are worth listing, because each one replaces a judgment ears would have fumbled.
Read that card as an eval engineer and every row is familiar. The CPU budget is a latency SLO. The aliasing floor is a threshold gate: below audibility, by a margin, always. The pink-noise slope is a measurement against a known ideal, which is exactly what a golden set is. Block-size invariance is the determinism law from streaming, applied to audio buffers: the same input must produce the same bytes no matter how the work is chunked. And the null test, render twice and subtract to silence, is the cleanest regression check I know in any field.
The judge still gets the last word
The harness does not decide if a patch sounds good. It decides whether the machine did what the math promised, so that when I audition a sound, the only variable left is taste. That division is the whole lesson. Measurement handles everything measurable, precisely so judgment can be spent where only judgment works. In eval terms: automated gates for the floor, human judges for the ceiling, and never confuse which layer you are standing in.
The takeaway
If you build eval harnesses, study mixing: references, fast pairwise switches, bias rituals, morning ears, translation checks. If you mix, steal the harness: budgets, thresholds, null tests, determinism. Both crafts converge on one sentence. Measure everything the machine can measure, so your ears are free to judge the only thing that matters.