Article 12 Evidence Pack (worked example)

A tamper-evident, cryptographically self-verifying technical artifact for EU AI Act Article 12 record-keeping and Annex IV section 2(d) documentation. This is a sample of the deliverable a Sprint Audit Review produces. It is suitable for forwarding to internal compliance leads, notified-body assessors, and accredited audit firms.

1. What this document is

This is a complete Article 12 record for one evaluation claim. It contains the four things such a record needs: a pre-registered manifest that fixes the claim, a cryptographic commit of that manifest made before the run, the result of the run, and a procedure any third party can follow to check that the manifest and the commit match. The delivered pack is a self-contained PDF; this page is that pack rendered for reading. Either way the assessor does not need to take our word for anything in section 3.

This is our own worked example, not a client engagement. We have run it on public data so that every number below is reproducible by a reader who has never spoken to us. The claim under test is a fairness claim about a credit-scoring model, chosen because creditworthiness is an Annex III high-risk use and because the four-fifths benchmark is the threshold convention an assurance reader already recognises.

The claim: a logistic-regression credit model trained on the UCI Statlog (German Credit) data selects female applicants for the favourable outcome at a rate of at least 80 percent of the male rate, on a held-out test split.

The outcome was not known when the manifest was sealed. The manifest was committed to the public registry at 08:22:37Z on 25 August 2026; the evaluation was run afterwards. That ordering is the whole point of the artifact, and it is the part a reader can check independently rather than trust.

2. The pre-registered manifest and its commit

The manifest below is the document that was fixed before the evaluation ran, in PRML v0.1 schema. The SHA-256 is computed over its canonicalised bytes (see the specification for the canonical form rules). It is reproduced here in full so that section 3 can be carried out from this page alone.

version: prml/0.1
claim_id: 01997a2c-0000-7000-8000-000000000001
created_at: '2026-08-25T08:22:01Z'
metric: impact_ratio
metric_args:
  protected_attribute: sex
  protected_group: female
  reference_group: male
  group_definition: 'attribute 9 (personal status and sex): A92 is female; A91, A93, A94 are male'
  favourable_outcome: predicted class 1 (good credit)
  split: stratified 70/30 train/test at the manifest seed
  features: all 20 attributes; categorical one-hot encoded; numeric standardised
  model_config: scikit-learn LogisticRegression, L2 penalty, C=1.0, max_iter=2000
  definition: female selection rate divided by male selection rate on the held-out test split
comparator: '>='
threshold: 0.8
dataset:
  id: uci-statlog-german-credit
  hash: b21f3d81db8071257d5ff1deaeba1fd4303b62712e6fcc9715c7a86202cb5871
  uri: https://archive.ics.uci.edu/dataset/144/statlog+german+credit+data
seed: 42
producer:
  id: worked-example.falsify
model:
  id: sklearn-logistic-regression-l2-C1.0
notes: Worked example produced by Falsify OU; not a client engagement. 0.80 is the EEOC four-fifths benchmark, a rule of thumb applied by bias-audit practice and not a legal standard; no statute fixes it. It is used because it is the convention an assurance reader recognises. The outcome was not known to the author when this manifest was sealed.

The timestamp matters more than the signature. Our signature says only that we saw the hash; the RFC 3161 token is issued by a third party whose clock we do not control, and the Rekor entry is append-only and mirrored, so its later absence would itself be evidence.

The result

A reader should note what this verdict does and does not carry. It says the model met a bar that was fixed before the test. It does not say 0.80 was the right bar: the four-fifths rule is an EEOC rule of thumb applied by bias-audit practice, it is not a legal standard, and no statute fixes it. Choosing the bar remains a judgement, and this artifact only makes that judgement visible and dated.

3. Verification: how an auditor re-derives the hash

The verification is mechanical and takes about a minute. It needs the manifest text (section 2), one of the four byte-equivalent reference implementations, and openssl. It does not need us.

=== VERIFICATION TRANSCRIPT — 2026-08-25T08:25:58Z ===

$ python3 falsify_prml.py verify manifest.yaml
OK  hash verified  sha256:132437f156e81d7c16822847a3c101b146b2a846828f0ddeb933efcf464667f5
(no --observed value given; predicate not evaluated)

$ shasum -a 256 german.data
b21f3d81db8071257d5ff1deaeba1fd4303b62712e6fcc9715c7a86202cb5871  german.data

$ curl -s https://registry.falsify.dev/132437f156e81d7c16822847a3c101b146b2a846828f0ddeb933efcf464667f5.yaml | shasum -a 256
afaceab9e03038c5213dfcc76751004d7806a242101f62612c96883e6c46a3fe  -
  (local manifest.yaml: afaceab9e03038c5213dfcc76751004d7806a242101f62612c96883e6c46a3fe)

$ openssl ts -verify -digest 132437f156e81d7c16822847a3c101b146b2a846828f0ddeb933efcf464667f5 -in receipt.tsr -CAfile chain.pem
Verification: OK

$ openssl ts -verify -digest <one character changed> -in receipt.tsr -CAfile chain.pem
Verification: FAILED
  (negative control: the check discriminates)

$ python3 run_eval.py
  impact_ratio=0.857919
  selection_female=0.677083
  selection_male=0.789216
  n_test=300
  n_female=96
  n_male=204
  verdict=PASS

The negative control is part of the procedure. Changing one character of the digest makes the timestamp check fail. Without that line the passing check proves nothing, because a check that cannot fail is not a check.

One practical trap. macOS ships LibreSSL, not OpenSSL, and LibreSSL cannot validate the ESS signing-certificate attribute that the Sigstore TSA uses. It reports Verification: FAILED on a token that is perfectly valid. Use a real OpenSSL build (brew install openssl, then the binary under /opt/homebrew/opt/openssl/bin). We record this because we hit it ourselves while producing this pack.

Files

• manifest.yaml — the sealed manifest, byte-identical to the copy the registry stores
• receipt.tsr — the RFC 3161 timestamp token
• run_eval.py — the evaluation, which reads its threshold from the manifest
• VERIFY.txt — the transcript above, as text
• Dataset: UCI Statlog (German Credit), hash in the table above

4. Mapping to EU AI Act Article 12 and Annex IV section 2

The table below states what this evidence pack covers and what it does not, against the operative text of Article 12 of Regulation (EU) 2024/1689 and Annex IV section 2.

The "FULL / PARTIAL / NONE" column is deliberately strict: it reports what the evidence pack mechanically provides, not what the provider's broader compliance program covers. Most rows that read PARTIAL or NONE here will read FULL elsewhere in the provider's technical documentation file under Annex IV.

5. Mapping to ISO/IEC 42001 controls

For providers under, or pursuing, an ISO/IEC 42001:2023 (AI Management System) management system, the mapping below shows where this evidence pack contributes. References are to the published 42001 control text.

This mapping uses control numbering from the 42001:2023 published text. For a provider running parallel certifications under 42001 and the AI Act, the column "Annex IV §2(d) coverage" in section 4 above and the column "A.9.3 coverage" here will overlap heavily; both describe the same evaluation event from different normative perspectives.

6. Limitations and out-of-scope items

This section is taken from spec section 8.1, restated here for the assessor's convenience. The spec is the controlling document.

• Selective non-publication. A provider could commit one hundred manifests, run all one hundred evaluations, and only publish the three that came out well. PRML by itself does not catch this; a separate completeness mechanism is required (registry-side claim-set commitments are on the v0.3 backlog).
• Execution-time data tampering inside the eval harness. If the eval harness silently truncates the dataset before computing the metric, the manifest hash will still verify; the dishonesty happens between data load and metric computation. Detecting this requires independent re-execution by a reviewer with their own dataset copy.
• Model binary integrity. The manifest references the model by id and hash but does not attest that the binary on disk matches the hash. This belongs to Sigstore / in-toto / SLSA in the supply chain layer.
• Runtime gating. PRML does not block model loading or inference if a manifest fails to verify. It is an audit primitive, not a runtime gate.
• Legal-compliance certification. This document does not certify legal compliance with Article 12, 15, or 18. It is one input to the compliance file. Certification under the AI Act is the responsibility of the notified body and, for self-assessment routes, the provider's quality-management system under Article 17.

7. For auditors and notified-body assessors

Suggested workflow for an assessor receiving this document as part of an Annex IV technical documentation file:

1. Extract the manifest from section 2 and save it to a local file.
2. Install any one of the four reference implementations (one-line install). Validate the manifest against the PRML JSON Schema (canonical schema is in SchemaStore, indexed since 2026-05-11).
3. Re-canonicalise and re-hash. Compare against the SHA-256 string in section 2 and against the registry record.
4. If you have independent access to the dataset and model artefacts, hash them and compare against the dataset.hash and model.hash fields. This catches identity-substitution attacks but not within-run tampering.
5. Cross-check the registry timestamp against the provider's claim of "pre-registered before the run." Any future-dated commits relative to result reporting are an audit red flag.
6. For high-risk systems, request a second independent reviewer to re-execute the eval against the provider's dataset and model copies. The manifest gives that reviewer everything they need to reproduce the run byte-for-byte modulo hardware non-determinism (which the seed and notes fields partly mitigate).

An assessor who completes steps 1-3 has a tamper-evident attestation of the claim that does not depend on trusting the provider. Steps 4-6 raise the assurance level further.

8. About this artifact