Evaluation Evidence Pack (worked example)

A tamper-evident, independently verifiable record of one evaluation’s acceptance criteria, for the testing requirements of EU AI Act Article 9(8) and the testing documentation in Annex IV point 2(g). It is a worked example of the evidence an Acceptance Evidence Pack delivers. It is suitable for forwarding to internal compliance leads, notified-body assessors, and accredited audit firms.

A complete acceptance record, with the plan both sides approved, the signed acceptance report, the agreed change conditions and our verification statement, all sealed and verifiable offline, is on its own page: a completed acceptance record. This page shows the criteria part in depth. In an engagement, deliverables are signed with a qualified electronic signature and carry a qualified electronic time stamp (eIDAS Art. 25(2) and 41(2)); this worked example is not.

1. What this document is

This is a complete evidence record for one evaluation claim. It contains the four things such a record needs: a manifest that fixes the acceptance criteria, a timestamped commitment showing the manifest existed no later than the anchored time, the result of the run, and a procedure any third party can follow to check that the manifest and the commit match. The delivered pack is a self-contained PDF; this page is that pack rendered for reading. Either way the assessor does not need to take our word for anything in section 3.

This is our own worked example, not a client engagement. We have run it on public data so that every number below is reproducible by a reader who has never spoken to us. The claim under test is a fairness claim about a credit-scoring model, chosen because creditworthiness is an Annex III high-risk use and because the four-fifths benchmark is the threshold convention an assurance reader already recognises.

The claim: a logistic-regression credit model trained on the UCI Statlog (German Credit) data selects female applicants for the favourable outcome at a rate of at least 80 percent of the male rate, on a held-out test split.

What a reader can check, and what they have to take from us. The manifest was committed to the public registry at 08:22:37Z on 25 August 2026. A reader can check independently that this manifest existed no later than that time and has not changed since. That we ran the evaluation afterwards is our statement as the producer; the record does not prove it. The transcript in section 3, dated 08:25:58Z, includes the evaluation run; that second date is ours, not independently anchored. The two dates together are what an assessor weighs.

2. The pre-registered manifest and its commit

The manifest below is the document committed at 08:22:37Z on 25 August 2026, in PRML v0.1 schema. The SHA-256 is computed over its canonicalised bytes (see the specification for the canonical form rules). It is reproduced here in full so that section 3 can be carried out from this page alone.

version: prml/0.1
claim_id: 01997a2c-0000-7000-8000-000000000001
created_at: '2026-08-25T08:22:01Z'
metric: impact_ratio
metric_args:
  protected_attribute: sex
  protected_group: female
  reference_group: male
  group_definition: 'attribute 9 (personal status and sex): A92 is female; A91, A93, A94 are male'
  favourable_outcome: predicted class 1 (good credit)
  split: stratified 70/30 train/test at the manifest seed
  features: all 20 attributes; categorical one-hot encoded; numeric standardised
  model_config: scikit-learn LogisticRegression, L2 penalty, C=1.0, max_iter=2000
  definition: female selection rate divided by male selection rate on the held-out test split
comparator: '>='
threshold: 0.8
dataset:
  id: uci-statlog-german-credit
  hash: b21f3d81db8071257d5ff1deaeba1fd4303b62712e6fcc9715c7a86202cb5871
  uri: https://archive.ics.uci.edu/dataset/144/statlog+german+credit+data
seed: 42
producer:
  id: worked-example.falsify
model:
  id: sklearn-logistic-regression-l2-C1.0
notes: Worked example produced by Falsify OU; not a client engagement. 0.80 is the EEOC four-fifths benchmark, a rule of thumb applied by bias-audit practice and not a legal standard; no statute fixes it. It is used because it is the convention an assurance reader recognises. The outcome was not known to the author when this manifest was sealed.

The timestamp matters more than the signature. Our signature says only that we saw the hash; the RFC 3161 token is issued by a third party whose clock we do not control, and the Rekor entry is append-only and mirrored, so its later absence would itself be evidence.

The result

A reader should note what this verdict does and does not carry. It says the model met a bar whose exact wording was committed and independently dated; the order relative to the test rests on that date together with the dated result, not on the verdict alone. It does not say 0.80 was the right bar: the four-fifths rule is an EEOC rule of thumb applied by bias-audit practice, it is not a legal standard, and no statute fixes it. Choosing the bar remains a judgement, and this artifact only makes that judgement visible and dated.

3. Verification: how an auditor re-derives the hash

The verification is mechanical and takes about a minute. It needs the manifest text (section 2), one of the four byte-equivalent reference implementations, and openssl. It does not need us.

=== VERIFICATION TRANSCRIPT — 2026-08-25T08:25:58Z ===

$ python3 falsify_prml.py verify manifest.yaml
OK  hash verified  sha256:132437f156e81d7c16822847a3c101b146b2a846828f0ddeb933efcf464667f5
(no --observed value given; predicate not evaluated)

$ shasum -a 256 german.data
b21f3d81db8071257d5ff1deaeba1fd4303b62712e6fcc9715c7a86202cb5871  german.data

$ curl -s https://registry.falsify.dev/132437f156e81d7c16822847a3c101b146b2a846828f0ddeb933efcf464667f5.yaml | shasum -a 256
afaceab9e03038c5213dfcc76751004d7806a242101f62612c96883e6c46a3fe  -
  (local manifest.yaml: afaceab9e03038c5213dfcc76751004d7806a242101f62612c96883e6c46a3fe)

$ openssl ts -verify -digest 132437f156e81d7c16822847a3c101b146b2a846828f0ddeb933efcf464667f5 -in receipt.tsr -CAfile chain.pem
Verification: OK

$ openssl ts -verify -digest <one character changed> -in receipt.tsr -CAfile chain.pem
Verification: FAILED
  (negative control: the check discriminates)

$ python3 run_eval.py
  impact_ratio=0.857919
  selection_female=0.677083
  selection_male=0.789216
  n_test=300
  n_female=96
  n_male=204
  verdict=PASS

--- environment note ---
The openssl lines above were run with OpenSSL 3.x.

On macOS, /usr/bin/openssl is LibreSSL, not OpenSSL. It verifies this receipt
fine, but it cannot verify RFC 3161 tokens that use ESSCertIDv2 (SHA-256) in the
signing-certificate attribute — it reports "ess signing certificate
error" and prints Verification: FAILED. That is the tool refusing to read the
token, not a bad token. Qualified EU timestamp authorities emit exactly that
shape, so if you ever check a qualified token, use OpenSSL 3.x.

The negative control is part of the procedure. Changing one character of the digest makes the timestamp check fail. Without that line the passing check proves nothing, because a check that cannot fail is not a check.

If the criteria cannot be public. Since 3 September 2026 the registry also accepts sealed commits: the manifest is validated, hashed, signed and anchored, but its text is withheld. The receipt proves that bytes with this hash existed at the recorded time; a party shown the manifest privately verifies it against the hash exactly as in this pack. POST /<hash>/reveal publishes the text later and is accepted only if the bytes re-derive the hash. The bar stays private; the ordering stays public.

One practical trap. macOS ships LibreSSL, not OpenSSL, and LibreSSL cannot validate the ESS signing-certificate attribute that the Sigstore TSA uses. It reports Verification: FAILED on a token that is perfectly valid. Use a real OpenSSL build (brew install openssl, then the binary under /opt/homebrew/opt/openssl/bin). We record this because we hit it ourselves while producing this pack.

Files

• manifest.yaml: the sealed manifest, byte-identical to the copy the registry stores
• receipt.tsr: the RFC 3161 timestamp token
• run_eval.py: the evaluation, which reads its threshold from the manifest
• VERIFY.txt: the transcript above, as text
• Dataset: UCI Statlog (German Credit), hash in the table above

4. Where this evidence sits in the AI Act compliance file

The pack shows that the acceptance criteria existed no later than the anchored time and have not changed since. It does not show that the criteria were appropriate, that the test ran after that time, or that the result is correct. The table follows from that boundary. Quotations are from Regulation (EU) 2024/1689 (OJ L, 12.7.2024); none of the provisions cited was amended by Regulation (EU) 2026/1744, except Article 43(3), quoted below in its amended form.

The fit column is deliberately strict: it reports what the pack mechanically provides, not what the provider’s wider compliance programme covers. The risk management system as a whole (Article 9), data governance (Article 10), human oversight (Article 14) and accuracy, robustness and cybersecurity measures (Article 15) are outside it. The Act asks for testing against “prior defined” criteria but does not say how “prior” is to be shown; it requires no timestamp, hash or third party. The pack is stronger evidence of a fact the Act already asks for, not an additional obligation.

5. ISO/IEC 42001:2023 Annex A

Control titles are as listed in public summaries of Annex A; check the numbering against your licensed copy of the standard. This mapping is our reading of the texts, not legal advice, and was prepared with AI assistance and reviewed by Falsify OÜ.

6. Limitations and out-of-scope items

This section is taken from spec section 8.1, restated here for the assessor's convenience. The spec is the controlling document.

• Selective non-publication. A provider could commit one hundred manifests, run all one hundred evaluations, and only publish the three that came out well. PRML by itself does not catch this; a separate completeness mechanism is required (registry-side claim-set commitments are on the v0.3 backlog).
• Execution-time data tampering inside the eval harness. If the eval harness silently truncates the dataset before computing the metric, the manifest hash will still verify; the dishonesty happens between data load and metric computation. Detecting this requires independent re-execution by a reviewer with their own dataset copy.
• Model binary integrity. The manifest references the model by id and hash but does not attest that the binary on disk matches the hash. This belongs to Sigstore / in-toto / SLSA in the supply chain layer.
• Runtime gating. PRML does not block model loading or inference if a manifest fails to verify. It is an audit primitive, not a runtime gate.
• Legal-compliance certification. This document does not certify legal compliance with the AI Act or any of its articles. It is one input to the compliance file. Certification under the AI Act is the responsibility of the notified body and, for self-assessment routes, the provider's quality-management system under Article 17.

7. For auditors and notified-body assessors

Suggested workflow for an assessor receiving this document as part of an Annex IV technical documentation file:

1. Extract the manifest from section 2 and save it to a local file.
2. Install any one of the four reference implementations (one-line install). Validate the manifest against the PRML JSON Schema (canonical schema is in SchemaStore, indexed since 2026-05-11).
3. Re-canonicalise and re-hash. Compare against the SHA-256 string in section 2 and against the registry record.
4. If you have independent access to the dataset and model artefacts, hash them and compare against the dataset.hash and model.hash fields. This catches identity-substitution attacks but not within-run tampering.
5. Cross-check the registry timestamp against the provider's claim of "pre-registered before the run." Any future-dated commits relative to result reporting are an audit red flag.
6. For high-risk systems, request a second independent reviewer to re-execute the eval against the provider's dataset and model copies. The manifest gives that reviewer everything they need to reproduce the run byte-for-byte modulo hardware non-determinism (which the seed and notes fields partly mitigate).

An assessor who completes steps 1-3 has a tamper-evident attestation of the claim that does not depend on trusting the provider. Steps 4-6 raise the assurance level further.

8. About this artifact