SynthID-Text robustness evaluation (receipt and results)
We fixed the protocol, the bars, the scripts and the inputs of a robustness evaluation of the open-source SynthID-Text watermark, and timestamped them before the main run. The run finished the same day. Of the twelve bar outcomes (six bars, two models), eight were met and four were not. All twelve are below, with every output and an offline check.
The bars
Each is reported as met or not met. TPR is the share of watermarked texts detected at a threshold that gives a 1% false-positive rate on separate unwatermarked calibration texts.
| Bar | Condition | Rule |
|---|---|---|
| B1 | No attack, 400 tokens | TPR ≥ 0.90 for each model |
| B2 | Truncated to 200 tokens | TPR ≥ 0.70 |
| B3 | 10% of words replaced with WordNet synonyms | TPR ≥ 0.60 |
| B4 | 25% of words replaced | TPR < 0.50 (expected degradation) |
| B5 | Paraphrased by the other model, unwatermarked | TPR < 0.30 (expected degradation) |
| B6 | Unwatermarked test texts | False-positive rate ≤ 0.03 |
A 10-prompt pilot was run before the lock to check the pipeline, and its scores were seen. The bars were set before the pilot and were not changed after it. The full account is in the protocol, section 3.
Results
200 watermarked and 200 unwatermarked test texts per model, 400 tokens each. Counts are texts detected out of 200; 95% Wilson intervals in brackets where they matter.
| Bar | Rule | Qwen2.5-1.5B | SmolLM2-1.7B |
|---|---|---|---|
| B1 | No attack: TPR ≥ 0.90 | 200/200 · met | 200/200 · met |
| B2 | 200 tokens: TPR ≥ 0.70 | 200/200 · met | 200/200 · met |
| B3 | 10% synonyms: TPR ≥ 0.60 | 200/200 · met | 200/200 · met |
| B4 | 25% synonyms: TPR < 0.50 | 200/200 · not met | 175/200 (0.82 to 0.91) · not met |
| B5 | Paraphrased: TPR < 0.30 | 128/200 (0.57 to 0.70) · not met | 21/200 (0.07 to 0.16) · met |
| B6 | False positives ≤ 0.03, at 400 and 200 tokens | 4/200 and 2/200 · met | 1/200 and 7/200 (0.017 to 0.070) · not met |
Thresholds (99th percentile of the calibration scores): Qwen 0.5088 at 400 tokens and 0.5142 at 200; SmolLM2 0.5134 and 0.5130. AUC was 0.995 or higher in every condition except paraphrasing (Qwen 0.873, SmolLM2 0.657). Every number is in ciktilar/test/sonuclar.md in the results bundle.
Reading the four that were not met
Three are predictions the watermark beat. B4 and B5 were written as expected degradation: we predicted that replacing a quarter of the words, or having another model rewrite the text, would bring detection below 50% and 30%. After 25% replacement, every Qwen text and 87.5% of SmolLM2 texts were still detected, and 64% of Qwen texts were still detected after SmolLM2 rewrote them. Our prediction missed, in the watermark's favour.
One is a shortfall. For SmolLM2 at 200 tokens, 7 of 200 unwatermarked test texts scored above the threshold: 3.5%, against a bar of 3% and a calibration target of 1%. At 400 tokens the rate was 1 of 200.
Because the bars were timestamped before the run, none of them can be adjusted now. B4 cannot become "below 95%" and B6 cannot become 4%.
Two observations that were not in the protocol
These were looked at after the results, so they are reported as observations, not tested claims.
- The paraphrase results are not comparable across models. When SmolLM2 rewrote Qwen's texts it kept a median 11% of their five-word sequences (mean 17%); when Qwen rewrote SmolLM2's texts it kept 2% (mean 4%). The gap between 64% and 10.5% says more about how much each paraphraser copies than about the watermark on each model.
- Qwen switched to Chinese in 10 of its 200 test outputs; SmolLM2 did in none. The prompts are English. Those texts were watermarked and detected like the others.
What happened during the run
Generation finished without problems. The paraphrase step for Qwen's texts then stalled on one batch, the one holding the longest Chinese text: first the machine ran out of memory and swapped, then a memory limit we set stopped it. With a higher limit it got through on the third attempt. That step saves its output only at the end, so nothing from the interrupted attempts was saved or read. Models, seeds, prompts, batch size, settings, threshold rule and bars stayed as locked. The two orchestration scripts, added after the lock, only call the locked scripts. While the run was going we computed scores without thresholds to check that the pipeline worked, and changed nothing afterwards. The full timeline is in RUN-NOTES.md in the results bundle.
Check it yourself
You need OpenSSL 3 and Python 3. Nothing calls a Falsify service.
unzip synthid-text-results-2026-10-11.zip && cd synthid-text-results
OPENSSL=/path/to/openssl3 bash kod/dogrula.sh # the lock
shasum -a 256 anahtarlar/keys.json # matches watermark_keys_sha256 in the manifest
cd kod && python3 skorla.py --set test # recomputes the resultsThe first script checks two things: every frozen file matches the SHA-256 recorded in the freeze manifest, and the RFC 3161 token covers exactly the canonical manifest bytes. Change one character of the protocol and the first check fails. macOS's built-in LibreSSL fails the token check; use OpenSSL 3. The last command rebuilds sonuclar.json from the published texts and keys (it needs transformers 4.46.3; CPU is enough); we checked that it reproduces the published file byte for byte. The lock bundle alone is still available above. You can fetch the timestamp authority's certificate chain yourself from https://timestamp.sigstore.dev/api/v1/timestamp/certchain instead of trusting the copy in the bundle. Folder names: kod is code, kilit is lock.
What this receipt shows, and what it does not
It shows that the protocol and the listed files existed no later than 2026-10-11 10:39:39 UTC, and lets anyone check that a copy they hold is byte-identical to the timestamped version. It does not show when the evaluation ran, that the bars were the right ones, or that the results are correct. It says nothing about any watermark deployed by Google, Anthropic, OpenAI or anyone else.
The protocol (section 11) and the last line of dogrula.sh say "have not changed since". That is broader than what a timestamp shows, and we no longer use it; both files are locked, so they stay as they were.
Why we did this
Under Measure 4.2 of the Code of Practice on Transparency of AI-Generated Content, signatories test their marking and detection "according to internal benchmarks and industry best practices" until the AI Office endorses recognised ones, and downstream providers may rely on the test results of a third-party marking or detection provider. A reader of such a test usually cannot check whether the attack set, the false-positive target and the pass bar were fixed before the test. This evaluation shows one way to make that checkable.
During the pre-lock check we found that the transformers implementation draws its sampling table with a generator on the generation device, and that a table drawn on Apple MPS differs from one drawn on CPU: text watermarked on MPS was not detectable by a CPU detector. Our scripts draw the table on CPU for both steps. Details are in the protocol, section 2, and we reported it upstream as huggingface/transformers#49498.