benchmarks

Correctness Suite

Metamer confusion pairs, nine metamorphic relations, and 401 conformance vectors for color code.

figure generated by the instrument itself. hover to let the light in.

The problem

Two color implementations can both look conformant and still disagree numerically. ICC conformance testing validates profile structure, not the computations, and nothing publishes certified numeric vectors for the colorimetric transforms themselves. On the benchmark side, adversarial-lighting datasets perturb the lamp rather than the surface, so there is no benchmark of physically real surface pairs that match under one light and split under another.

The method

generate_metamer_benchmark samples opposite boundary points of the metamer set for anchor colors, using the Logvinenko and Godau-Funt convex-set framework, then ranks pairs by ΔE_OK divergence under the split illuminant. Every item is a physically valid reflectance pair, identical under the match illuminant (D65 by default) and divergent under the split illuminant (A by default). crosscheck_color_implementations builds on established metamorphic and differential testing: nine algebraic relations any correct implementation must satisfy (round trips, gamut-map idempotence, adaptation white preservation, metamer tristimulus preservation, ΔE metric axioms, CVD severity-zero identity, SPD scaling linearity), with failures shrunk to minimal witnesses; the color-specific relation suite is the new part, the methods are not. generate_conformance_vectors emits deterministic vectors from published formulae (IEC 61966-2-1 sRGB, Ottosson 2020 Oklab, CIE/ISO 11664-6 CIEDE2000 in the Sharma-Wu-Dalal formulation, Machado 2009 CVD) with tolerances derived from double-precision round-trip error, or scores a candidate implementation against them.

What it promises

Benchmark items are physically realizable reflectance pairs: matched under the match illuminant with the residual ΔE_OK reported per item, divergent under the split illuminant, rated trivial, moderate, or hard. No pixel perturbation, no out-of-gamut fiction.
Nine metamorphic relations with failures shrunk to minimal witnesses. Differential mode diffs another library's outputs on the same generated inputs and localizes disagreements. The suite has already caught a real bug in our own engine: selecting a relation subset shifted the seeded RNG stream so the same seed generated different test inputs, fixed by drawing all inputs up front.
401 golden vectors generated from the TS reference engine pin the C++ core in core/tests/golden; regenerating them requires a PR labeled golden-update with justification, never a silent refresh.
Stated limit, in the tool's own description: the conformance suite is a reference suite. This engine is the oracle, and its authority is published formulae, stated tolerances, and metamorphic self-consistency, not independent ground truth.

Calling it

{
  "tool": "generate_metamer_benchmark",
  "arguments": {
    "anchor_count": 5,
    "match_illuminant": "D65",
    "split_illuminant": "A",
    "samples_per_anchor": 12,
    "items_per_anchor": 2,
    "seed": 42
  }
}

Clone the repo, build the server (cd mcp && npm install && npm run build), then: claude mcp add maryslab -- node mcp/dist/server.js

Known limits

This engine is the conformance oracle, so a disagreement beyond tolerance means one of the two implementations has a defect to explain, in either direction. It is not independent ground truth.
Benchmark divergence comes from sampling metamer-set boundary points, so the headline maxSplitDeltaEOk is the largest divergence found, not a proven maximum, and unrealizable anchors fail loudly with NO_SOLUTION.
The benchmark's split illuminants are limited to D65, D50, A, and E, and the conformance suite covers eight transforms, not the full engine surface.