The score has a home. The bar does not.

One widely used evaluation framework, its complete official documentation, 165 files. None of them defines or records what would count as passing. None carries a mechanism that would let anyone else check that the criteria were fixed before the results. Measured and issued 29 August 2026.

What was measured, and why

Five earlier measurements in this series looked at the side that asks for evidence: a government assurance programme, a benchmark corpus, a financial-stability document set, a certification scheme, a medical-device regime. Each one asks, in its own words, that success criteria be settled in advance. None supplies a way to show it happened.

This one looks at the other end. If a pass bar is ever going to be recorded, the recording would happen in the tool that runs the evaluation. So the question here is narrow and mechanical: in the documentation a practitioner actually reads, is there a place to write down what would count as passing, and anything that would let a third party check when it was written?

The subject is an open-source RAG evaluation library with more than fifteen thousand stars, widely used in production pipelines. It is named in the appendix along with the exact commit measured. It is worth saying plainly at the outset that this is not a criticism of the project: it is a developer tool for iterating on a system, not a compliance instrument, and nothing about its design promises otherwise. The finding is about an empty layer, not a defect.

Method

The corpus is the docs/ directory of the project's own repository at a pinned commit — the same markdown from which the published documentation site is generated. 165 files: 144 Markdown, 21 notebooks. Nothing was sampled; every file was scanned. The library's source code was not measured, and neither were community write-ups, tutorials or archived version sites.

Each term was matched case-insensitively and then every match was read. A count on its own was not treated as a result in either direction: a hit that turns out to be a Colab metadata key is not evidence of provenance, and an absence is only reported for the exact terms listed.

Two controls ran alongside. A positive control (faithfulness) matched 49 files and 382 times, confirming the scan reaches the corpus. A negative control (blockchain, notaris|notariz) matched nothing, confirming it is not matching everything. One earlier pass of this scan suppressed standard error and silently produced a false zero for one term; it was re-run with errors visible, which is how the discrepancy was caught.

Result

165 of 165 documents contain no recorded pass bar, and no mechanism by which a third party could check the ordering. The full per-match ledger is in the appendix.

The nearest thing to a record, and why it is not one

The framework does have a concept page for experiments, and it uses the word hypothesis. Its own loop is documented as: make a change, run evaluations, observe results, then hypothesise the next change.

Read it again in that order. The hypothesis is formed after the results are seen. That is the correct loop for development — it is how anyone improves a system — and it is drawn honestly. But it is not a loop that can produce a claim someone else has to rely on, because at no point in it is anything fixed before the measurement that produced the number.

The same page shows the one timestamp the data model carries. It is produced by the running code at the moment results are returned, from the local clock, by the party making the claim. Self-asserted and after the fact: exactly the two properties that make a timestamp unusable as evidence of ordering.

What this shows, and what it does not

It shows that the layer where a bar would naturally be recorded does not ask for one. The tool computes the score faithfully and stores it carefully. What would count as passing is never its question.

It does not show that the project should change. A developer's inner loop does not need a notary, and adding one would make the tool worse at its job. The claim is narrower: when a score leaves that loop and becomes something a customer, an auditor or a regulator relies on, there is currently nowhere it was written down in advance. That gap is not the tool's to close alone, but it is not closed anywhere else either.

Three limits, stated rather than left to be discovered. This measured documentation, not source code — an unnamed field may exist in the library, but a field that is not documented is not something a third party can rely on. This is an absence claim, and it holds only for the corpus and the exact terms listed; the honest sentence is “these terms were not found in this corpus”, not “this does not exist”. And the corpus is a snapshot: the commit is recorded so anyone can re-run the scan and get a different answer later, which is the point.

One consequence for our own specification

Most of this framework's headline metrics are LLM-as-judge: a model grades the output. That has a consequence we had not written down. In PRML, the optional model field is defined as the model under test; the judge has no field of its own, and the schema forbids unknown keys. A judge can be carried in the free-form metric_args mapping, which is inside the hash — but nothing requires it, no verifier notices when it is missing, and two tools would name the key differently. A seed does not help, because it pins a local generator and not a hosted model.

So a manifest can be locked, timestamped, and verified as untampered, while the bar it encodes is still under-determined: re-run with a different judge and the verdict moves, with nothing in the record to show it. That is not in our threat model. It is now an open defect in the roadmap, and a candidate for v1.0.

Appendix — per-match ledger

Corpus: the docs/ directory of vibrantlabsai/ragas at commit 298b68274234c060deacab3cf5fb52aa3a20e885, retrieved 29 August 2026. 165 files (144 .md, 21 .ipynb). Every match below was read in context before being classified.

hash / digest / SHA-256 — 2 matches, 2 files, 0 valid

• howtos/integrations/_langfuse.md — hashlib.sha256(...).hexdigest(), used to build a question-to-context dictionary key.
• howtos/integrations/langfuse.ipynb — the same snippet in notebook form.

timestamp / RFC 3161 — 8 matches, 5 files, 0 valid

• concepts/experimentation.md ×2 — "timestamp": datetime.now().isoformat() written into the result row; and result CSVs named with a timestamp prefix.
• howtos/applications/iterate_prompt.md ×2 — opening and re-saving timestamped result files.
• howtos/cli/rag_eval.md — “Timestamps” listed as a column of the result file.
• howtos/customizations/_caching.md — advice not to cache prompts containing timestamps.
• howtos/integrations/ag_ui.md ×2 — a protocol message field, and timestamped output.

signature / signed — 2 matches, 2 files, 0 valid

• howtos/migrations/migrate_from_v03_to_v04.md — “change method signature to use keyword arguments”.
• howtos/applications/benchmark_llm.md — a synthetic customer who “signed up for our service nine years ago”.

provenance / attestation / tamper — 1 match, 1 file, 0 valid

• howtos/integrations/langgraph_agent_evaluation.ipynb — "colab": {"provenance": []}, a notebook metadata key.

pre-registration — 0 · immutable / append-only / audit trail — 0

threshold — 13 matches, 8 files, 0 are a pass bar for a claim

• extra/components/choose_evaluator_llm.md ×2 and extra/components/choose_generator_llm.md ×2 — HarmBlockThreshold, a provider safety-filter setting.
• howtos/customizations/testgenerator/_testgen-customisation.md ×2 and the matching notebook ×2 — threshold=0.01, distance_threshold=0.9 inside test-set generation.
• howtos/applications/benchmark_llm.md ×2 — an income threshold inside synthetic test data.
• concepts/metrics/available_metrics/answer_correctness.md — an option to round a score to binary.
• howtos/applications/align-llm-as-judge.md and howtos/applications/text2sql.md — stop iterating when the result “meets your business threshold”. The bar is named as the reader's, and there is nowhere in the tool to put it.

seed / random_state — 4 matches, 3 files, 1 valid

• howtos/customizations/run_config.md ×2 — seed=42 in the run configuration. Valid, but it governs the library's own executor, not a hosted judge model.
• howtos/customizations/index.md — “Seed generations using production data”, a link title.
• howtos/applications/singlehop_testset_gen.md — “semantic seed points” in test generation.

Controls

• Positive: faithfulness — 49 files, 382 matches.
• Negative: blockchain — 0. notaris|notariz — 0.

The series so far

3 of 29 · 0 of 635 · 6 of 124 · 8 of 45 · 20 of 30 · 0 of 165. The first five measured regimes that ask for the bar. This one measured the tooling that would record it.