Which frameworks require pre-specified criteria — and what do they accept as proof?

If an evaluation is supposed to be judged against criteria fixed in advance, someone eventually has to show that they were. This page reports what five regulatory documents actually say, measured the same way: the primary text of each was read in full and searched for the mechanisms it defines — for the dataset, for the model, and for the acceptance criteria themselves.

What each framework requires

• EU GMP Annex 22 (EudraLex Volume 4, consultation draft, July 2025 — not in force; the text in force is Annex 11). §4.2: acceptance criteria "should be documented and approved before the start of acceptance testing"; §7.2 requires an approved test plan containing "the pre-defined metrics and acceptance criteria" before the test is initiated. For the test data it is far more specific: §6 requires access control, audit trail logging accesses and changes, no copies outside the repository, staff separation and a four-eyes principle. §10.2 requires measures to "detect any unauthorised change" to the model. For the criteria: no such mechanism.
• ITU, Designing Regulatory Experimentation (June 2026). Methodological Note 3: "Every experimentation space must define regulatory hypotheses from the design stage (ex ante) and contrast them with observed results at the close (ex post), guaranteeing traceability between evidence and the normative decision." The clearest ex ante requirement in the set. The document contains no technical mechanism at all: no hash, no timestamp, no audit trail, no integrity.
• ETSI TS 104 008 (CABCA, January 2026). The one that goes furthest. Thresholds live in an Operationalization Specification that is "provided to Stakeholders for a formal Initial Review and Approvement"; artefacts are "version-controlled" and reports "Versioned and timestamped"; §8.1.4.2 states that "the acceptance threshold shall be justified by intended use and deployment context". What is missing is not a mechanism but an independent one: the words hash and cryptographic do not appear, and the version and timestamp are produced inside the system of the party being audited or of its auditor.
• NIST AI 300-1 ipd (July 2026, initial public draft). The dataset profile offers field 1.5.2 "Cryptographic Signature — cryptographically signed manifest containing one or more cryptographic hashes for the dataset instances", and the model profile offers the same at 1.6.2 alongside a version identifier. Both optional. The evaluation field 6.5(*) asks only for "Evaluation Metrics and Results" and "timing and frequency of evaluations"; its subfields cover intended use, robustness and evaluation reliability. The word threshold does not appear anywhere in the document.
• NAIC AI Risk Evaluation Supplement v5.0 (exposure draft, comments closed 29 September 2026). Exhibit C field 11 asks the insurer to "discuss testing model outputs … and how the model was validated prior to being deployed". Free narrative. The document contains no hash, no audit trail, no version control requirement.

The same lens, five documents

Each primary text was downloaded and searched with stem matching. Counts are of the whole document.

What the pattern is

Three of these frameworks know how to protect the integrity of an object and do it. NIST offers a cryptographic signature field for a dataset and for a model. Annex 22 puts access control, a copy prohibition and a four-eyes principle around test data, and requires unauthorised-change detection for the model. ETSI version-controls and timestamps its audit artefacts.

None of the five defines an equivalent for the acceptance criteria. Where the criteria are addressed at all, the requirement is procedural — documented, approved, justified, discussed — and the evidence that they existed at the stated time is a record held by the party whose work is being assessed. A reader who was not present still has to take that party's word for the ordering.

The clearest single illustration is NIST AI 300-1: a document that offers cryptographic signatures for datasets and models, in seventeen thousand words, never uses the word threshold.

A commitment to record changes is not the same as a checkable record

A fair objection to everything above: plenty of organisations do keep a record, and some now promise to publish one. That is true, and the promise is worth stating precisely, because it shows where the remaining gap actually is.

The clearest current example comes from the party being regulated rather than the regulator. OpenAI's Frontier Governance Framework (28 May 2026), written to map its practices onto the EU AI Act's Code of Practice for General-Purpose AI and California's Transparency in Frontier AI Act, commits in §7.1 that material updates will be presented to the board's Safety and Security Committee "with changes and justifications for material updates documented in a changelog and published within 30 days of the update", and in §7.2 to a framework assessment at least every twelve months. Of the documents surveyed here and their industry counterparts, it is the first to commit to recording the change itself and the reason for it.

That commitment closes the question what changed. It does not close the question who can check. The changelog is published by the organisation whose safeguards it describes; the document contains no hash, no timestamp requirement and no version identifier of its own. A reader who wants to know whether an entry is complete, whether the thirty days were kept, or what the previous text said, has one source for all three answers: the same organisation. The commitment is only as checkable as the party making it.

This matters because the fix is small and does not touch the substance of the promise. Publishing the digest of a changelog entry, and anchoring that digest to a time the publisher does not control, turns "we recorded it" into something a third party can verify without extending any trust. The commitment stays exactly as it is; only its verifiability changes.

The same reasoning applies to every row in the table above. The gap these documents share is not an absence of records. It is that the record which would settle the ordering is held, dated and attested by the party being assessed — which is close to the thing the requirement exists to rule out.

The same gap, named at the top of the field — and funded on one side of it

Two documents published since this page was first written describe the same gap from opposite ends of the stack, and neither names a mechanism for it.

The UN Independent International Scientific Panel on AI — the first global scientific body on AI, established under General Assembly resolution 79/325 — gives as the first of its assessment challenges, in the preliminary report it presented in July 2026, that "there is information asymmetry in safety validation between companies and society… Safety evaluation methodologies are currently designed largely by the companies being evaluated", and that developer-set risk thresholds "remain defined by the developers, without standardized evaluation or external verification". Across 27,795 words that report uses evaluation 55 times and benchmark 25 times, and hash, cryptographic, timestamp, provenance, tamper, attestation and pre-registration not once. It is an assessment rather than a recommendation: it diagnoses the gap and leaves the mechanism slot empty.

The second is sharper, because it shows the gap is not an inability to think in these terms. A Transformative AI Strategy for Europe, published on 14 September 2026 by a senior expert council convened by Monika Schnitzer, with contributors including Margrethe Vestager, Yoshua Bengio, Daron Acemoglu and Philippe Aghion, uses verification 72 times. Forty-four of those are about hardware. It proposes at least €250 million over five years for on-chip trusted execution environments, "tamper-evident enclosures for AI chips", retrofittable secure modules and "privacy-preserving methods for verifying a chip’s workload", and public procurement for "attestable AI compute" and "location attestation for imported AI chips". On the evaluation side the same document asks for an ecosystem of third-party evaluators and auditors — an institutional answer — and specifies no integrity mechanism for the criteria those evaluators would apply. Its single use of pre-registration in 85,911 words is about research-funding experiments, not model evaluation.

Put together: across the documents measured here, the integrity of the chip carries a proposed budget line and a procurement instrument, while the integrity of the criterion appears in none of them as a requirement. The difference is one of attention rather than difficulty. A criteria file is small, and fixing it in time needs a hash and a timestamp from a party the producer does not control — older and cheaper technology than anything proposed for the silicon.

Where a regulator has worked through this directly, it reached the same conclusion

Everything above measures what these documents do not say. One European decision says it outright, in a different domain.

On 16 July 2026 the Commission adopted a specification decision under the Digital Markets Act requiring Alphabet to share anonymised Google Search data with competing search services (case DMA.100209, published in the Official Journal as C/2026/4443). Almost all of it is about anonymisation and security rather than model evaluation. But in deciding how compliance with those measures should be checked, it works through the same problem this page is about, and states the answer plainly.

On one side, the decision rejects letting the assessed party attest to itself: "reliance on self-attestation by third-party [search engines] would likewise fail to provide an objective or verifiable assessment of compliance with the anonymisation, integrity, and confidentiality contractual measures. Self-attestation lacks independent oversight and would rely solely on the representations of the third-party [search engine], which may have incentives to interpret the contractual measures in a manner favourable to its own operational needs." On the other side, it rejects letting the gatekeeper run the check: a verification mechanism "controlled or substantially influenced by Alphabet would create a structural conflict of interest and could undermine the fairness and neutrality of the verification process."

What it requires instead is the shape this page keeps asking for. The Commission defines the assurance objectives itself rather than accepting the assessed party’s. It requires independent assurance under ISAE 3000, noting that such engagements "do not only attest to the existence of policies or controls at a point in time, but also provide assurance on the design, implementation and operating effectiveness of controls over a defined period". It requires access logs to be protected against "unauthorised access, alteration, or tampering" and retained for the annual audit. And the acceptance criteria for residual re-identification risk were written by an independent expert the Commission commissioned, published in the decision text as an explicit list, with two further experts commissioned separately to run adversarial testing against them.

None of that reasoning is ours, and none of it is controversial. What has not happened is the same reasoning applied to evaluation criteria. In the five frameworks in the table above, the metric, the comparison rule and the dataset are still described by the party whose result they judge, and no mechanism is specified for establishing when they were fixed.

What this page is not claiming

• This is a five-document measurement, not a survey of regulation. It says nothing about frameworks not read here.
• The NIST fields are optional, and the Annex 22 draft binds no one — it is a consultation text whose scope excludes generative models and LLMs from critical GMP applications altogether.
• ETSI is not missing a mechanism; it is missing an independent one. The distinction matters and collapsing it would misrepresent the document.
• None of these bodies has endorsed, adopted or mentioned any particular tool, including ours. What is reported here is what the texts say.
• DMA.100209 is about data access, anonymisation and security compliance, not about AI evaluation criteria. It is quoted for how it reasons about self-attestation and conflict of interest, not as a requirement that applies to evaluations. Its own limit is visible too: the expert report and the adversarial testing are cited by redacted document identifiers, so a reader outside the case cannot check the testing itself.
• Neither of the two documents in the section above is a binding instrument, and neither recommends any mechanism. The UN Panel report is non-prescriptive by its mandate; the European strategy is an independent expert report, not an EU institution document, and its €250 million is a proposal, not an allocation. Both are quoted for what they say about the state of practice.
• What the UN Panel asks for is larger than what a criteria record provides. It calls for standardized, rigorous, independent third-party assessment on the model of the pharmaceutical and aeronautical industries. A record of when the criteria were fixed is one small component of that, and not a substitute for it.
• Nobody here is accused of hiding anything. The Frontier Governance Framework commits to publishing changes and their justifications; that is more than any of the five regulatory documents asks for. The point of that section is the narrower one: a published record still relies on the publisher.

How a provider or an examiner can close the gap now

1. Write the criteria down as one file before validation runs: the metric, the comparison rule, the threshold, the evaluation dataset (name and a hash of the exact files) and, if randomness is involved, the seed. Keep it short; it is a record, not a report.
2. Hash the file. sha256sum criteria.yaml. The 64-character digest identifies that exact set of bytes; change one character and the digest changes.
3. Get a time that you did not assign yourself. A file date, a git commit date or a signed memo is provider-attested: the party being examined chose the clock. Independent options that cost nothing: an RFC 3161 timestamp token from a public time-stamping authority over the digest, or an entry in a public transparency log. Either lets a third party check the time without trusting you.
4. Bind the record to what was tested. The criteria file should name the model version identifier and the dataset version it applies to, so the chronology cannot be reattached to a different run later.
5. Keep the token with the validation report. When an examiner asks, you hand over three things: the criteria file, its digest, and the independent time evidence. They can recompute the digest and verify the token themselves.
6. If the criteria changed after results were seen, say so. A second record with its own time evidence, and one sentence on why. A revised bar with a visible history is stronger evidence than a bar that was never written down.

Where PRML fits, if you want a standard form. The steps above need no particular tool. PRML is an open, vendor-neutral record format for exactly this object (metric, comparator, threshold, dataset identity and hash, seed) with a canonical byte encoding, a registered media type (application/vnd.prml+yaml) and free verifiers. It is evidence, not authority: the governance process still decides; the record independently proves which criteria object existed at the relevant time. What is PRML?

Sources

• European Commission — Case DMA.100209, SP – Alphabet – Article 6(11), specification decision of 16 July 2026, recitals on the verification mechanism; OJ C/2026/4443.
• UN Independent International Scientific Panel on AI — Preliminary Report, July 2026, §2.1.
• A Transformative AI Strategy for Europe — full report, senior expert council convened by Monika Schnitzer, 14 September 2026.
• EU GMP Annex 22 — European Commission, Annex 22: Artificial Intelligence, EudraLex Volume 4 consultation draft, published 7 July 2025, consultation closed 7 October 2025: consultation page.
• ITU — Designing Regulatory Experimentation: A Practical Guide for the Public Sector, Telecommunication Development Bureau, ISBN 978-92-61-42681-1: itu.int.
• ETSI — TS 104 008 V1.1.1 (2026-01), Continuous Auditing Based Conformity Assessment for AI-enabled systems: etsi.org.
• NIST — AI 300-1 ipd, Guidance and Templates for Public-Facing AI Documentation, July 2026, doi 10.6028/NIST.AI.300-1.ipd: nvlpubs.nist.gov.
• NAIC — AI Risk Evaluation Supplement v5.0 exposure draft, Big Data and Artificial Intelligence (H) Working Group.

The five-document table was measured 13 September 2026, and the two documents in the section above on 14 September 2026, each against its primary text. Counts use stem matching; a boundary match misses plurals such as hashes.