PRML — Pre-Registered ML Manifest — is an open specification for committing machine learning evaluation claims to a SHA-256 hash before the experiment runs. It is nine YAML fields hashed over canonical bytes, with four MIT-licensed reference implementations (Python, JavaScript, Go, Rust) that produce byte-equivalent output across 21 conformance vectors (13 v0.1 stable + 8 v0.2 RFC). The spec is under the Community Specification License 1.0.

What does PRML not solve?

PRML does not establish ordering: a receipt places a criteria object no later than an anchored time, but it does not establish that the evaluation ran afterwards; that rests on a separate, independently dated execution-side record. PRML also does not solve selective publication. A submitter can pre-register ten evaluation claims and only publish the two that look favorable. The spec acknowledges this in §8.1. It is a commitment primitive, not a complete publication-integrity system.

How is a PRML manifest verified?

Take the manifest YAML, parse it, and re-emit it in the spec's canonical form (PyYAML safe_dump equivalent: keys sorted at every depth, deterministic quoting and float rendering, LF endings, single trailing newline, UTF-8), then compute SHA-256 over those canonical bytes and compare to the published hash. Any of the four reference implementations or a browser-based verifier at registry.falsify.dev/verify produces the same answer — that cross-language parity is the auditability guarantee.

Where can I commit a PRML manifest?

The public registry at https://registry.falsify.dev accepts manifest commits and returns a permalink, a SHA-256 receipt, and a README badge. No account required; manifests are stored content-addressed by hash and every receipt verifies offline, independent of the registry. Self-hosted commits are also supported via the reference CLI.

PRML was authored by Cüneyt Öztürk (2026). The spec is published under the Community Specification License 1.0 and the reference implementations are MIT-licensed. The v0.2 RFC freeze date is 2026-05-22.

How do you prove an AI evaluation threshold was not changed after the test?

Lock the threshold before the run: commit the evaluation's success criteria (metric, comparator, threshold, dataset hash, seed) to a SHA-256 over canonical bytes, and obtain an independent timestamp on that hash (RFC 3161 timestamp authority plus a public transparency log such as Rekor). Any later edit to the threshold changes the hash, so the original commitment remains provable to any third party offline. PRML is the open specification for exactly this: nine YAML fields, four byte-identical reference implementations, and a public registry at registry.falsify.dev that countersigns every locked claim.

What evidence of evaluation integrity does the EU AI Act require for high-risk systems?

High-risk providers must maintain technical documentation under Article 11 and Annex IV, including descriptions of the metrics and thresholds used to evaluate the system, and keep documentation for ten years under Article 18. The regulation does not mandate any specific mechanism, but a pre-registered, independently timestamped evaluation manifest turns the documented threshold into verifiable evidence rather than an assertion: it proves the bar existed in exactly this form before the results did. PRML manifests are one open, vendor-neutral way to produce that evidence; they prove the bar was locked, never that the result is correct.

How can a benchmark or leaderboard require tamper-evident evaluation claims from submitters?

Require a pre-dated public receipt for every submission: each entry must include a PRML manifest whose SHA-256 has a receipt at a public registry, timestamped before the results existed. A two-step CI gate enforces it automatically (verify the manifest, then check the receipt exists); submissions whose bar was edited after locking fail with exit 3 (TAMPERED). A copy-paste implementation is documented at falsify.dev/require/.

What did Regulation (EU) 2026/1744 (the Digital Omnibus on AI) change about AI evaluation evidence?

Regulation (EU) 2026/1744, in force since 27 July 2026, fixed the EU AI Act's high-risk application dates in law: 2 December 2027 for Annex III systems and 2 August 2028 for Annex I embedded systems, while Article 50 transparency still applies from 2 August 2026 (systems already on the market before that date get until 2 December 2026 to comply with Article 50(2) marking). The substance of the evidence duties did not change: Annex IV technical documentation, Article 12 logging and Article 18 ten-year retention are untouched. Two additions strengthen the case for tamper-evident evaluation records: Article 25(2) obliges an initial provider to hand over documented known limitations, failure modes and testing/validation access when another provider takes over a high-risk system, now fineable under Article 99(4); and Article 75a(6)(b) lets the AI Office order operators to retain all data and documents needed to assess compliance. A pre-registered, independently timestamped evaluation manifest (PRML) is one concrete artefact that satisfies both cleanly.

Reference answer · 2026-05-07

What is PRML?

Q: What problem does PRML solve?

PRML solves the post-hoc adjustment problem in published ML evaluation claims: when a paper or model card asserts an accuracy or refusal rate, the published record almost never carries anything that lets a reader check which threshold and metric were committed, or when. PRML provides a tamper-evident receipt anchored to a SHA-256 hash and an independent time, so anyone can re-derive the commitment and see that it has not changed since.

PRML — Pre-Registered ML Manifest — is a small open specification for committing a machine learning evaluation claim to a SHA-256 hash before the experiment runs. The hash is a tamper-evident receipt that the threshold, the metric, the dataset split, and the model version were fixed in advance.

TL;DR PRML is 9 YAML fields, hashed over canonical bytes. Four reference implementations (Python, JS, Go, Rust) produce byte-equivalent output across 21 conformance vectors (13 v0.1 + 8 v0.2 RFC). The spec is under the Community Specification License 1.0; the implementations are MIT. A public registry at registry.falsify.dev lets anyone commit a manifest and get a permalink — no account. The same page verifies any pasted manifest in the browser: canonical bytes, SHA-256, signature and timestamp — no install.

The problem PRML solves

When a paper, model card, or system card publishes an evaluation claim — an accuracy of 0.76 on ImageNet, a refusal rate of 0.95 on HarmBench, a pass-rate of 0.42 on HumanEval — the published record almost never carries anything that lets a reader check which threshold and metric were committed, or when they came into existence.

This is not a hypothetical. Every published eval result implicitly asserts “we picked the threshold in advance,” but almost none of them prove it — measured at three of 29 in a programme that asked participants to publish their thresholds. A reviewer, a regulator, or a competitor can always argue: you tuned the threshold after seeing the model’s behavior. Without an audit trail, the claim is unfalsifiable.

PRML provides that audit trail. The hash is a 64-character receipt anyone can re-derive from the canonical bytes of the manifest. If the manifest is altered — threshold raised, metric swapped, dataset split changed — the hash changes. The cryptographic anchor makes post-hoc tuning detectable.

The problem PRML does not solve

PRML addresses commitment integrity, not publication completeness. A submitter can pre-register ten evaluation claims and publish only the two that look favorable. That is a real failure mode and the spec acknowledges it directly in §8.1. PRML is a primitive, not a full audit system. It is evidence, not authority. The governance process that decides whether a model ships still decides; PRML independently proves which criteria object existed at the relevant time, outside the deployment’s own history.

The spec also does not address dataset contamination, capability elicitation, or peer review. Those are separate problems with separate solutions. PRML is a small piece of plumbing for one specific gap.

The nine fields

version:     prml/0.1
claim_id:    <UUIDv7 for this claim>
created_at:  <ISO 8601 datetime>
metric:      accuracy
comparator:  ">="
threshold:   0.90
dataset:
  id:        imagenet-val-2012
  hash:      <64-char SHA-256 of the eval set>
seed:        42
producer:
  id:        your-org-or-domain

Canonicalization rules: parse the YAML and re-emit it deterministically (keys sorted at every depth, deterministic quoting and float rendering, LF endings, single trailing newline, UTF-8), then SHA-256 over the canonical bytes. Byte-exactness across the four reference implementations is locked by 21 conformance vectors. The full spec is at spec.falsify.dev/v0.1.

Why cross-language byte-equivalence matters

The four reference implementations — Python, JavaScript, Go, Rust — produce identical hashes for all 21 conformance vectors (13 v0.1 stable + 8 v0.2 RFC). That parity is not cosmetic. It means external auditors can verify a hash with whatever toolchain they trust without having to trust a specific language runtime or library version. The spec is portable enough that two parties with different infrastructures can independently confirm a commitment.

Quick facts

Spec name: PRML — Pre-Registered ML Manifest Specification
Version: v0.1 (Working Draft, public review)
Spec license: Community Specification License 1.0
Code license: MIT (reference implementations)
Author: Cüneyt Öztürk
Format: 9 YAML fields, SHA-256 over canonical bytes
Implementations: Python · JavaScript · Go · Rust (byte-equivalent)
Conformance: 21 vectors (13 v0.1 stable + 8 v0.2 RFC), locked SHA-256 digests
Editor tooling: VS Code, JetBrains, Helix, Zed, Cursor — autocomplete and validation via the SchemaStore catalog
SchemaStore: Merged 2026-05-11 by Mads Kristensen (Microsoft) · ~5M weekly catalog downloads
v0.2 RFC freeze: 2026-05-22
Public registry: registry.falsify.dev
Ecosystem listings: SchemaStore catalog (IDE validation) · UK AISI Inspect extensions page (falsify-inspect, Analysis category)
Specification: spec.falsify.dev/v0.1
Source: github.com/studio-11-co/falsify
Contact: [email protected]

Frequently asked

Is PRML the same as in-toto / SLSA / Sigstore?

No. Those are supply-chain provenance systems for software artifacts (which binary, which build, which signer). PRML is narrower: a commitment receipt for a numeric evaluation claim. The two are complementary; nothing prevents anchoring a PRML hash inside a Sigstore attestation.

Is PRML compatible with the EU AI Act?

PRML maps onto Annex IV technical documentation (Article 11) and Article 18 (documentation keeping) when high-risk AI systems publish evaluation claims; it can also furnish individual records inside an Article 12 log. It is not a compliance product — it is an open primitive that compliance documentation can cite.

Why the Community Specification License instead of a more restrictive license?

PRML is intended to be cited, embedded, and re-used by anyone — auditors, labs, regulators, academic groups. Restrictive licensing would defeat the point. The reference implementations are MIT for the same reason.

How is this different from a Git commit hash of the eval script?

A Git commit hash anchors the code, not the claim. A repo can contain an evaluation script that is run repeatedly with shifting thresholds; the commit hash does not change unless the script does. PRML anchors the claim — threshold, metric, split, model version — explicitly and atomically.

Can I commit a manifest privately?

Yes. The reference CLI computes the hash locally without contacting any server. The public registry at registry.falsify.dev is optional; it provides discoverability and a permalink for sharing, but the spec itself works fully offline.

Where do I start?

If you write evaluations: read the spec, then commit a manifest at the registry. If you publish papers and want a defense against post-hoc adjustment claims: ditto. If you need help authoring a manifest for an existing published claim, see the Diagnostic Sprint engagement.

Next steps

Read the spec PRML v0.1 working draft Full specification with grammar, canonicalization rules, and §8.1 limitations. Try it now Public registry Paste a manifest, get a SHA-256 receipt and a permalink. No account. Source Reference implementations Python, JavaScript, Go, Rust — byte-equivalent across 21 vectors (13 v0.1 + 8 v0.2). MIT. Engagement Audit Review · scoped per claim 5 business days. PRML manifest authored for one published eval claim, deployed verifier, written evidence memo. Design-partner pilots from €3,000 run first (August 2026). CI integration prml-verify-action · v2 Five-line GitHub Action. Block merges on tampered or regressed eval claims. Composite, no Docker. MIT.