Answer · NAIC · 2026-09-09

NAIC Exhibit C, field 11: how do you show how the model was validated before deployment?

Version 5.0 of the NAIC AI Risk Evaluation Supplement asks insurers, for each high-risk AI model, to discuss testing of model outputs and how the model was validated prior to being deployed. A narrative answers the question. It does not, on its own, let the examiner check it.

Short answer State the acceptance criteria the model had to meet (metric, comparison rule, threshold, evaluation dataset), say when those criteria were fixed, and point to a record that establishes they were fixed before the validation results were available. If you only describe the procedure and the results, the examiner cannot tell a bar set before testing from a bar chosen after seeing the numbers. The Supplement's own materiality logic already works this way: the threshold used is disclosed, not left implicit.

What the Supplement asks

AI model risk criteria are set by the insurance company. Note that materiality should be a consideration in determining the level of risk.
Exhibit C, purpose statement, v5.0
Discuss testing model outputs (e.g. model drift, accuracy, unfair trade practices, unfair discrimination, performance degradation, etc.) and how the model was validated prior to being deployed as well as how its performance is monitored on an ongoing basis.
Exhibit C, field 11
Last date of model testing
Exhibit C, field 12

Fields 1 (name and version), 4 (implementation date) and 7 (risk classification) frame the answer. Exhibit B, item 3a, separately asks whether validation and testing are performed by someone independent from development. Nothing in v5.0 asks for the acceptance criteria themselves or for when they were fixed. That is the gap a careful answer fills voluntarily, because the examiner's follow-up question is predictable: what did the model have to achieve, and who decided that, and when?

Why a narrative is not enough

Two models can report identical validation results while one was held to criteria fixed before testing and the other to criteria selected after the results were known. From the response alone the examiner cannot distinguish the two. The risk criteria are the company's to set; the Supplement says so. What makes them credible is not who set them but whether they can be shown to have existed before the outcome did.

How to make it verifiable, without any tool

  1. Write the criteria down as one file before validation runs: the metric, the comparison rule, the threshold, the evaluation dataset (name and a hash of the exact files) and, if randomness is involved, the seed. Keep it short; it is a record, not a report.
  2. Hash the file. sha256sum criteria.yaml. The 64-character digest identifies that exact set of bytes; change one character and the digest changes.
  3. Get a time that you did not assign yourself. A file date, a git commit date or a signed memo is provider-attested: the party being examined chose the clock. Independent options that cost nothing: an RFC 3161 timestamp token from a public time-stamping authority over the digest, or an entry in a public transparency log. Either lets a third party check the time without trusting you.
  4. Bind the record to what was tested. The criteria file should name the model version identifier and the dataset version it applies to, so the chronology cannot be reattached to a different run later.
  5. Keep the token with the validation report. When an examiner asks, you hand over three things: the criteria file, its digest, and the independent time evidence. They can recompute the digest and verify the token themselves.
  6. If the criteria changed after results were seen, say so. A second record with its own time evidence, and one sentence on why. A revised bar with a visible history is stronger evidence than a bar that was never written down.

A template answer for field 11

Pre-deployment validation. The model (name, version 2.3.1) was required to meet
the following acceptance criteria, fixed on 2026-06-12 by the Model Risk Committee
before validation results were available: [metric] [>=/<=] [threshold] on
[evaluation dataset, version, SHA-256 …]; [second criterion …].
Record: criteria file SHA-256 […], RFC 3161 timestamp token dated 2026-06-12
(retained with the validation report). Validation was performed by [independent
function] on 2026-06-19; results: [metric] = […] (criterion met / not met).
Criteria have not been revised since. Ongoing monitoring: […].

Where PRML fits, if you want a standard form. The steps above need no particular tool. PRML is an open, vendor-neutral record format for exactly this object (metric, comparator, threshold, dataset identity and hash, seed) with a canonical byte encoding, a registered media type (application/vnd.prml+yaml) and free verifiers. It is evidence, not authority: the governance process still decides; the record independently proves which criteria object existed at the relevant time. What is PRML?

Sources

Does the NAIC AI Risk Evaluation Supplement require pre-registered acceptance criteria?

No. Version 5.0 asks how the model was validated prior to deployment (Exhibit C, field 11) and for the last date of testing (field 12), and states that risk criteria are set by the insurance company. It does not ask which acceptance criteria were used or when they were fixed. Falsify has proposed an optional field for that; as of September 2026 it is a proposal, not a requirement.

Is a signed and dated memo enough to prove when the criteria were fixed?

It proves that someone signed a document carrying a date. It does not prove the document existed on that date, because the signer controls the clock. Independent time evidence, such as an RFC 3161 timestamp token or a transparency-log entry over the file's hash, lets an examiner verify the time without trusting the company.

Does pre-committing the criteria prove the model is safe or compliant?

No. It proves which criteria existed before the results, nothing about whether the criteria were adequate or the results good. A model can fail a locked bar. That is still useful evidence; a passed bar with no record of when it was set is weaker.