# Pre-registered robustness evaluation of the open-source SynthID-Text watermark

**Status:** LOCKED, version 1 (11 October 2026). This file, the scripts in `kod/`, the input hashes, the model revisions and the SHA-256 of the key file are listed in `kilit/freeze-manifest.canonical.json`, whose bytes carry an RFC 3161 timestamp from Sigstore's public timestamp authority (`kilit/freeze.tsr`). The receipt is published **before the main run**. Check it offline with `bash kod/dogrula.sh`.

**Producer:** Falsify OÜ (reg. 17574308, Tallinn, Estonia).

## 1. Purpose and scope
EU AI Act Article 50(2) requires providers to make machine-readable marks effective, interoperable, robust and reliable as far as technically feasible. The Transparency Code of Practice lets providers test their marking against internal benchmarks until the AI Office endorses recognised ones. Published robustness figures are therefore usually measured under conditions chosen by the party making the claim.

This evaluation shows one way to make such a figure checkable: the attack set, the false-positive target and the bars are fixed and timestamped before any test output exists.

It evaluates the **open-source** SynthID-Text implementation in Hugging Face `transformers` on two small open-weight models. It says **nothing** about any deployed system (Google Gemini, Anthropic Claude, OpenAI textGrain or others).

## 2. System under test
- Watermark: `transformers.SynthIDTextWatermarkingConfig` (transformers 4.46.3, library defaults for sampling-table size and seed and for context history; the sampling table is drawn with a **CPU** generator for both generation and detection, see `kod/ortak.py`: the library draws it on the generation device, and in a pre-lock check a table drawn on Apple MPS matched a CPU-drawn table in only about half of its entries, so text watermarked on MPS was not detectable by a CPU detector), `ngram_len=5`, 30 keys drawn once with `secrets.randbits(62)` (`kod/anahtar_uret.py`); the SHA-256 of the key file is recorded at lock time and the keys are published after the run.
- Models (float32 weights; Apple M3 Pro, PyTorch 2.8.0 MPS backend; float16 produced invalid sampling probabilities on this backend in a pre-lock check):
  - `Qwen/Qwen2.5-1.5B-Instruct`, revision `989aa7980e4cf806f80c7fef2b1adb7bc71aa306` (Apache-2.0)
  - `HuggingFaceTB/SmolLM2-1.7B-Instruct`, revision `31b70e2e869a7173562077fd711b654946d38674` (Apache-2.0)
- Sampling: `do_sample=True`, `temperature=0.7`, `top_k=40`, `min_new_tokens=400`, `max_new_tokens=400`, the chat template shipped with each model revision, batches of 8 prompts, `torch.manual_seed(20261011 + index of the batch's first prompt)`. Exact settings are in `kod/ortak.py`.

## 3. Prompts
- Source: `databricks/databricks-dolly-15k`, revision `bdd27f4d94b9c1f951818a7da7fd7aeea5dbff1a` (CC BY-SA 3.0), file `databricks-dolly-15k.jsonl`.
- Filter: categories `open_qa` and `general_qa`, empty `context`, instruction ≤ 300 characters.
- Order: shuffle with seed 20261011. Indices 0–199 = **test**, 200–399 = **calibration**, 400–409 = **pilot**.
- **Pilot disclosure.** Before lock, the full pipeline was run once on the 10 pilot prompts with the same keys, to check that it works. Its scores were seen. They are published with the outputs, labelled as pilot, and are not results. **The bars in section 8 were set before the pilot and were not changed after it.** The pilot also found the device issue described in section 2; an earlier pilot run made with the MPS-drawn table is kept, labelled, in `ciktilar/_onkontrol-mps-tablo/`.
- Input file SHA-256: `databricks-dolly-15k.jsonl` 2df9083338b4abd6bceb5635764dab5d833b393b55759dffb0959b6fcbf794ec; WordNet 3.0 `wordnet.zip` (nltk_data) cbda5ea6eef7f36a97a43d4a75f85e07fccbb4f23657d27b4ccbc93e2646ab59.

## 4. Generations
For each model: watermarked outputs on the test prompts; unwatermarked outputs on the test and calibration prompts. Outputs shorter than 400 tokens are excluded and counted.

## 5. Detector
Primary score: SynthID **mean** score, i.e. the mean g-value over positions not masked by context repetition or end-of-sequence, computed with the same keys via the `transformers` SynthID logits processor. Secondary score: weighted mean. Scoring uses the generating model's tokenizer.

Threshold: for each model, the 99th percentile of the primary score on the **calibration** unwatermarked outputs at 400 tokens (used for C0, C2, C3, C4) and at their first 200 tokens (used for C1); target false-positive rate 1%. A text counts as detected when its score is strictly above the threshold. Thresholds are computed before any watermarked output is scored.

Scoring is text-level: each condition's text is re-tokenized with the generating model's tokenizer, without special tokens.

## 6. Conditions (watermarked test outputs)
| ID | Condition |
|---|---|
| C0 | No attack, 400 tokens |
| C1 | Truncated to the first 200 tokens |
| C2 | Synonym substitution of 10% of words |
| C3 | Synonym substitution of 25% of words |
| C4 | Paraphrase by the other model, unwatermarked ("Rewrite the following text in your own words, keeping the meaning."), max 450 new tokens |

The conditions correspond to typical processing operations named in Measure 3.3 of the Code of Practice on Transparency of AI-Generated Content (lexical substitution, paraphrasing, and character deletion as a desynchronisation). Translation cycles, also named there, are not included in this evaluation.

Synonym substitution: the target is 10% or 25% of all alphabetic words. Eligible words are alphabetic, longer than 3 characters and not in the fixed stop-word list in `kod/ortak.py`; they are visited in a random order (seed fixed per output in `kod/saldir.py`) and replaced with the first lemma, in WordNet 3.0 index order, that differs from the word (exact-form lookup, no morphological reduction; the WordNet database files are read directly). Words without such a lemma are skipped. The achieved share is reported. This is a deliberately simple, deterministic attack; many substitutions are not natural synonyms in context.

## 7. Outcomes
Primary: true-positive rate at the fixed threshold for each condition and model, with 95% Wilson intervals. Empirical false-positive rate of the fixed threshold on the unwatermarked **test** outputs. Secondary: AUC.

## 8. Bars (stated before the run; each is reported as met or not met)
- **B1** C0: TPR ≥ 0.90 for each model.
- **B2** C1: TPR ≥ 0.70 for each model.
- **B3** C2: TPR ≥ 0.60 for each model.
- **B4** C3: TPR < 0.50 for each model (expected degradation).
- **B5** C4: TPR < 0.30 for each model (expected degradation).
- **B6** Empirical FPR on unwatermarked test outputs ≤ 0.03 for each model.

## 9. Changes and deviations
Any change after lock is recorded as an amendment with its own timestamp **before** the affected step runs, stating what changed, why, and whether any test output had been seen. Unplanned deviations are reported with the results.

## 10. What will be published
This protocol and its receipt; the scripts; environment listing (`pip freeze`, hardware); all generations, scores and thresholds; the keys (after the run); results against B1–B6 whatever they are.

## 11. What the receipt does and does not show
The receipt shows that this protocol and the named scripts existed no later than the timestamp and have not changed since. It does not show when the evaluation ran, that the bars were the right ones, or that the results are correct. Anyone holding the published files can check them against the receipt.