Reproducible is not pre-registered
"Reproducible evaluation artefacts", "reproducible test evidence", "stability and reproducibility of results": the words appear in EU sandbox programmes, insurance examination tools and NIST documentation templates. They describe a real property. They do not describe the property most of those documents assume they are getting.
Two questions that get merged into one word
- Can I get this number again? That is reproducibility. It is answered by pinned code, pinned data, pinned seeds and an environment someone else can run. It protects against drift, silent dependency changes and honest mistakes.
- Was this the bar all along? That is pre-registration. It is answered by a record of the acceptance criteria whose existence at a given time can be checked by someone who does not trust the author. It protects against the threshold being chosen, or quietly adjusted, after the results were known.
ISO/IEC 17000 describes conformity assessment as a demonstration that specified requirements are fulfilled, in a sequence: selection, determination, review, decision, attestation. The sequence assumes the specified requirements came first. Nothing in the sequence proves that they did. Reproducibility strengthens the determination step; pre-registration is evidence about the input to that step.
Where the confusion shows up
- EU AI Act readiness guidance that tells providers to "produce reproducible evaluation artefacts for conformity dossiers" and stops there.
- Insurance examination questionnaires that ask how a model was validated and when it was last tested, but not what it had to achieve or when that was decided.
- Documentation templates that record the stability of evaluation results and treat stability as the reader's main uncertainty about a performance claim.
- Model cards that publish a threshold and a pass, with no way for a reader to establish which came first.
Why the second property is cheap
Reproducibility is expensive: it needs environments, data custody and compute. Pre-registration needs one small file and one independent clock. The steps:
- Write the criteria down as one file before validation runs: the metric, the comparison rule, the threshold, the evaluation dataset (name and a hash of the exact files) and, if randomness is involved, the seed. Keep it short; it is a record, not a report.
- Hash the file.
sha256sum criteria.yaml. The 64-character digest identifies that exact set of bytes; change one character and the digest changes. - Get a time that you did not assign yourself. A file date, a git commit date or a signed memo is provider-attested: the party being examined chose the clock. Independent options that cost nothing: an RFC 3161 timestamp token from a public time-stamping authority over the digest, or an entry in a public transparency log. Either lets a third party check the time without trusting you.
- Bind the record to what was tested. The criteria file should name the model version identifier and the dataset version it applies to, so the chronology cannot be reattached to a different run later.
- Keep the token with the validation report. When an examiner asks, you hand over three things: the criteria file, its digest, and the independent time evidence. They can recompute the digest and verify the token themselves.
- If the criteria changed after results were seen, say so. A second record with its own time evidence, and one sentence on why. A revised bar with a visible history is stronger evidence than a bar that was never written down.
The result is not authority. It does not say the criteria were adequate or the model good; a model can fail a locked bar. It says which criteria object existed at the relevant time, and it says so in a way a third party can check. Failing a locked bar, on the record, is better evidence than passing a bar nobody can date.
Where PRML fits, if you want a standard form. The steps above need no particular tool. PRML is an open, vendor-neutral record format for exactly this object (metric, comparator, threshold, dataset identity and hash, seed) with a canonical byte encoding, a registered media type (application/vnd.prml+yaml) and free verifiers. It is evidence, not authority: the governance process still decides; the record independently proves which criteria object existed at the relevant time. What is PRML?
Sources and related reading
- ISO/IEC 17000:2020, Conformity assessment — Vocabulary and general principles (definitions of conformity assessment, determination, attestation)
- NIST AI 300-1, field 6.5: evaluation criteria
- NAIC Exhibit C, field 11: how the model was validated
- Threshold disclosure: what published model evaluations actually state
If my evaluation is fully reproducible, do I still need to pre-register the criteria?
Yes, if anyone other than you needs to trust the pass. Reproducibility proves the number can be regenerated; it says nothing about whether the threshold it was compared against existed before the number. Both properties are independent, and the second costs one file and one timestamp.
Is a git commit an independent timestamp?
No. The commit date is set by the committer's machine and can be rewritten. A push to a public host adds a server-side receipt time that is harder to alter, but it is still a single party's record. An RFC 3161 timestamp token from a time-stamping authority, or a public transparency-log entry, is evidence a third party can verify without trusting you or your host.
Does pre-registering criteria mean the criteria are good?
No. It means they existed at a verifiable time. Whether they were the right criteria is a governance judgement that the record leaves to the governance process. Evidence, not authority.