Methods 8 min
Testing without an oracle
When you cannot say what the right answer is, you can still say what must be true about it. Nine relations that catch bugs no example test would.
Most testing assumes you know the answer. You call the function with a known input, compare against a known output, and the test passes or fails. This works whenever an oracle exists — a reference implementation, a worked example from a paper, a value you can derive by hand.
For a lot of this lab’s code, no oracle exists. What is the correct output of sampling a metamer polytope’s boundary with seed 21 and fourteen directions? Nobody knows. There is no table to check it against, and computing it by hand means reimplementing the thing under test.
The technique that works here is metamorphic testing: instead of asserting what the output is, assert a relationship between outputs that must hold regardless of what they are.
The idea in one example
Take color space conversion. I may not know the Oklab coordinates of an arbitrary XYZ triple off the top of my head. But I know this: if I convert XYZ to Oklab and back, I must land where I started.
xyzToOklab then oklabToXyz ≈ identity
That is a test I can run on ten thousand random inputs without knowing a single correct answer. And it is a real test — it catches transposed matrices, swapped coefficients, a missing cube root, a sign error, and any asymmetry between the forward and inverse paths.
The relation does not need an oracle because it is a property of the pair of functions, not of any particular value.
The nine
The relations this lab runs, and what each is actually good at catching:
Round-trip identity. Every invertible conversion composed with its inverse is the identity, within tolerance. Catches transcription errors in matrices and inverse matrices independently.
Colour difference is symmetric. ΔE(a, b) = ΔE(b, a). Sounds trivial; CIEDE2000 has enough asymmetric-looking terms in its definition that an implementation error here is easy and invisible in casual use.
Colour difference is zero exactly when the colors are identical. Catches an epsilon that got hardcoded somewhere, and division-by-near-zero handling in the hue terms.
Illuminant scaling leaves relative colorimetry unchanged. Double the intensity of a light and relative colorimetry — the normalized version — must not move. Catches a missing normalization, which is the single most common colorimetry bug and produces results that look plausible right up until someone compares two illuminants.
Zero-severity CVD simulation is the identity. A dichromacy model at severity zero must return its input untouched. Catches off-by-one indexing into the severity table, and interpolation that does not pin its endpoints.
Full-severity simulation is idempotent. Simulating full deuteranopia twice equals simulating it once — the second pass has nothing left to remove. Catches a projection that is not actually a projection.
Gamut mapping is idempotent on in-gamut colors. A color already inside the gamut must come back unchanged. Catches an unconditional transform applied where a conditional one was meant.
Permutation invariance. Reorder the inputs to a batch operation and the outputs must be reordered identically, with the same values. Catches state leaking between iterations of a loop — a surprisingly common bug in kernels that reuse a scratch buffer.
Spectral scaling linearity. Scale a reflectance by a constant and the resulting XYZ scales by the same constant. Catches a nonlinearity applied at the wrong stage of the pipeline.
Each of these runs against both the TypeScript reference engine and the native C++ engine, on generated inputs.
The bug in the test harness
Worth telling because it is the kind of mistake that makes a test suite lie.
The runner originally worked like this: pick a relation, then generate random inputs for it, then check. Reasonable-looking, and wrong in differential mode.
When you use the same seed to compare two engines, you expect both to see the same inputs. But relation selection consumed random draws from the same stream that generated the inputs. Change which relations ran, or their order, and the inputs shifted underneath. Two engines that agreed perfectly could be tested on different data and reported as disagreeing.
The fix is to draw every input up front, before any relation is chosen, so the input stream cannot depend on control flow. Obvious in hindsight. It took a confusing failure on a correct implementation to find it.
The same class of bug reappeared later in the fuzzer, where each tool’s inputs depended on how many random draws the previous tools had consumed, so changing the iteration count silently changed which tools got covered. Shared mutable random state is a recurring trap in test infrastructure, and it always presents as a mysterious nondeterminism rather than as itself.
Where metamorphic testing stops
Relations constrain; they do not pin down. An implementation that returns zero for everything satisfies symmetry, satisfies zero-on-identical, and satisfies several others. Relations must be paired with example-based tests — here, the 401 golden vectors — that anchor the absolute values.
The division of labour is clean. Golden vectors say this input gives this output. Relations say whatever the outputs are, they hang together correctly. Vectors catch drift in values; relations catch structural errors on inputs no one recorded. Neither substitutes for the other, and a suite with only one of them has a predictable blind spot.
The reason to reach for relations is simple: they are the only kind of test available when the answer is genuinely unknown, which in numerical work is most of the time.