What has been verified and with what numbers

The main question for any statement about the reliability of an AI system is not “how many percent,” but “on what basis is it measured.” The same system gives zero errors on one set of data, and fourteen percent on another, and both numbers are accurate. Therefore, the methodology comes first.

Methodology

One key metric, everything else is secondary.

The main metric is the percentage of incorrect answers in the verification mode: cases where the system provides a meaningful answer that is claimed to be verified, but it is incorrect or not confirmed by the source. The target value is exactly zero. Not “low,” not “better than the baseline model.” Zero: any non-zero value is considered an architectural defect, not a statistical anomaly.

The denominator is the total number of tasks in the subset, not just the answered ones. This is crucial: otherwise, rejecting tasks would indirectly improve the metric, and it would be enough to reject them more often.

All other metrics — routing accuracy, argument accuracy, citation validity, failure rate — are secondary and become interesting only after the main one has reached zero. Each of them can be exchanged for the main one, and that is why they are looked at after it, and not instead of it.

Control systems are implemented before optimization.

Harness runs not only the real system but also the controls. The most illustrative of these is the system that always fails. It receives a perfect score on the main metric, and this is correct: it is honest, simply useless. That is why, immediately after the main metric, they look at the proportion of excess failures — those where the answer was available.

The second check is an oracle that knows the correct answers: it shows how many failures are inevitable for this set, and turns a high failure rate from a verdict into a difference from what is achievable.

You can trust tasks mechanically, not just by taking their word for it.

A task that references a non-existent utility or a non-existent course file turns correct system behavior into a counted error. Therefore, it is checked automatically: each quote must be a byte-for-byte substring of the specified file, each utility identifier must exist in the directory, and each reference specification must compile and pass its examples.

For tasks where the correct answer is a refusal, the task itself contains the command grep, which proves that the corpus does not actually answer the question. During the build of the independent set, this check discarded 6 out of 87 candidates: the corpus still covered them. The rejection is recorded in the log, not silenced.

Red-teaming is measured by evidence, not by assessment.

Each task requiring rejection contains a list of strings, the presence of which in a non-rejected answer is direct evidence of fabrication. This is not a judgment about the quality of the answer, but a fact: the string is either found or not.

A rejection is counted strictly. An empty answer is a rejection. An answer with the marker “not sure, but…”, containing a prohibited string, is not considered a rejection: it is a made-up story with a disclaimer at the beginning. A rejection marker, drowned in two kilobytes of confident prose, is also not a rejection.

The independent test suite is written without access to the implementation.

This is the only part of the methodology that truly tests generalization. The author of the independent dataset did not examine the implementation of the gates—neither in its entirety nor through content search. The work only used the original corpora, the problem scheme, the harness documentation, and the existing dataset (to avoid duplication).

Set: 230 tasks, of which 165 require rejection. Intersection with the previous set was measured: zero matching identifiers, zero literal query duplicates, maximum lexical overlap of 0.500. All 49 quotes have been checked byte-by-byte and appear exactly once in their respective files.

A separate part of the set consists of “plausibly related” questions: not about cooking, but about what is not in the materials, but sounds like their natural continuation. A neighboring version of the tariff. A neighboring prop. A tool command that does not exist, next to commands that do exist.

Trust boundary instead of the word “zero”

Zero observed events out of n is not “never.” The rule of three gives an upper bound of 3/n at 95% confidence, and it is reported alongside the zero itself in calibration reports.

That is why the statement “zero false positives” in itself does not mean much: it is strengthened or weakened along with the size of the set. Completeness also has an error: 0.327 out of 150 observations is ±7.5 percentage points with 95% confidence.

How were the sets for threshold calibration built?

The threshold, below which the system fails, is not determined by guesswork. The rule: take the minimum threshold at which, on a set of questions that are guaranteed to be unanswerable, none pass.

A set of questions to be answered consists of 150 items, each written based on a specific fragment of a specific document, and the documents are not repeated. The set is frozen before any scoring: selecting questions based on whether their index can be found would turn the measurement of completeness into a measurement of the filter itself.

A set of unanswerable questions — 173 questions selected from 402 candidates. 229 were rejected: for each, a search across all fragments showed that the corpus does cover the topic. The candidates were not “fixed”; they were discarded. A difficult class — 109 questions about technologies that are not in the corpus, even though related topics are covered in detail.

What did these sets reveal?

A negative result that is more important than a positive one

The previous calibration provided a completeness of 0.975 with zero false positives. The negative set then consisted of cooking and sports. In a set where two-thirds of the negatives are plausible questions about related IT topics, the same condition yields a completeness of 0.327.

The previous result was not a fake. It was a property of easy acquisition.

The walkthrough showed why: cosine similarity measures the topic, not the presence of an answer. The maximum of difficult negatives lies above the median of actual questions – there is no common window at any threshold. This is fixed by a different scorer: a cross-encoder evaluates the “question + fragment” pair as a whole, and completeness, with the same zero false acceptances, increases from 0.327 to 0.847. The provided configuration reranks 20 candidates instead of 50 and gives 0.833 – 0.6 points of completeness for more than a twofold reduction in cost.

The thresholds, however, exist in different scales and are incomparable: 0.745 on the hybrid score and 0.5427 on the cross-encoder score. The second was chosen not as the minimum possible, but as the minimum at which the threshold can cope on its own, without relying on the rule of indistinguishability of candidates: zero, which holds on a tie-break, is a weaker statement.

A caveat that had to be added after the re-check of 2026-08-09. “On its own” does not hold everywhere. The margin behind that choice is one ten-thousandth: the worst out-of-corpus score during calibration was 0.5426 against a threshold of 0.5427. On a repeat run of the same 173 questions on the GPU that very same worst question scored 0.5429 — above the threshold — and what stopped it was the tie-break, the indistinguishability gate. Zero false accepts held, but it leaned on what it was not meant to lean on by design. The fp32 difference between CPU and graphics card is larger than the margin, so the claim “the threshold copes alone” is true for the machine it was calibrated on and does not carry over to another. Either the threshold is raised to a margin that survives a change of device, or the wording is narrowed to “the zero is held by two gates out of three”.

A gate of unfamiliar tokens was measured and included separately: a word that is not in the corpus dictionary, with a length of at least four and without a neighbor at a distance of one edit. The price on the calibration set is 1 out of 110; on the independent set, it removed 4 and did not lose a single correct quote. It was included precisely for this reason: two independent sets, agreeing that the price is zero, outweigh one tuning set, which says that it is small.

The price is stated separately: the cross-encoder requires a GPU. On a video card, a batch of 50 pairs takes 232 ms, while on a processor, 20 pairs take 11,205 ms. This is not “more expensive”; it is a different product, so the reranker is disabled by default and enabled on a host with a GPU.

Main numbers in the visible set

A set of 400 tasks, 150 of which are red-team tasks. The measurement is complete, without sampling.

The same run, repeated on a reassembled index with a different embedding model, yielded 0.2% false positives. This is the second, independent measurement, and it cannot be directly compared to the first: the search layer has changed. It only confirms what it is intended to confirm – the strict limitation has been met in both configurations.

The harness’s verdict on these numbers is FAIL, and that is correct: the goal is exactly zero false answers, not “fewer than before.”

Regarding the specifications branch separately: 62.5% of the requests resulted in a specification being built and compiled, with zero false positives. All issued specifications are certified, and twelve out of forty tasks were closed with a rejection because they could not be expressed within the closed subset of the language that the translator understands. The translator is deterministic: language model generation is not used here intentionally because, under the grammar, the model cannot produce invalid syntax, but it can produce syntactically flawless falsehoods, and there is nothing to check them against.

Prime numbers on an independent set

And here, the methodology starts working against its own results.

The difference in the main metric is 7.7 times. The claimed zero false answers on the red-team test held up with the previous dataset; on an independent dataset with the same metric definition, the red-team gives 13.3%.

No group of independent sets was selected to “cover known vulnerabilities” — the implementation of gates was not revealed during its creation.

This is the main result of all the work, and it is negative.

What exactly breaks

A walkthrough of all 32 breaches revealed one dominant mechanism. In 28 out of 32 cases, the system does not invent text: it extracts a real, verbatim quote from the corpus that answers a different question and presents it as a verified answer.

From the perspective of “no fabricated facts,” it looks flawless – every character in the answer exists in the corpus. From the user’s perspective, it’s a false answer: to the question “how to record a lemma in FTS,” it returns a fragment from an article about graph theory.

The most telling case: a question about natural transformation in FTS, the answer – a fragment from an article on category theory with a relevance of 0.98. The search confidently found the topic but completely lost the subject matter.

The mechanism of this fallacy is visible at the score level. When asked about a single attribute of an organization, the cross-encoder gives a paragraph with a similar attribute a score of 0.999. The model perfectly distinguishes “not the right topic” and still performs poorly on “the right topic, but the wrong entity.” With independent negative calibration, the threshold yields 29 out of 99 false positives, compared to 15 out of 99 in the configuration without a cross-encoder—while the cross-encoder provides twice as many correct citations. Therefore, configurations should be compared with an equal number of false positives, and then the choice of model is confirmed by an independent set, but the choice of a point on the curve is not.

The symptom is evident in the pair of metrics: direct evidence of fabrication 1.2% with a false answer rate of 13.9%. The gate, which checks “whether this text is in the corpus,” allows such an answer by design.

The second class is a bait in the argument; 30 out of 34 baits have been deflected. Those that fail are precisely the ones where an extraneous token is syntactically valid as an argument: “calculate the statistics of the text in windows-1251 encoding” turns into the statistics of the string “in windows-1251 encoding.” This is also not a hallucination: the utility is actually executed, and the number is calculated correctly for the wrong input. The user receives a seemingly verified answer to a question they did not ask.

What held up: false premises about the external domain — 31 out of 32; utility routing — 16 out of 16 with 100.0% argument accuracy; obviously foreign topics — completely. Plausible related questions — 81.5%.

What did the fourth gate give?

The gate for responsiveness is written directly below the found class. On an independent set, in the same search configuration before and after:

On this configuration, the overhead is literally zero: the proportion of unnecessary rejections has not changed at all. The reason is not luck; out of the six tasks that the gate rejected, five required rejection, and the sixth remained a false answer. The time cost is approximately 26 ms per request.

Ablation according to the rules showed two working rules out of four and two non-working ones, which are disabled by default. The number of broken correct answers is zero for both sets with any set of rules.

The same metric for the main set of 400 tasks at the time of publication is not yet finalized: in the report for the gate, instead of both final tables, there are empty placeholders. The numbers will appear here when they appear there — it is not possible to substitute them from memory or “by analogy.”

Model-router separately

The micro-model was measured without a case and without gates, on 250 routing tasks. This is the lowest layer: it shows what the system receives as input before any checks.

Regressions are called regressions: the accuracy of the arguments has dropped to 90.1%, and the number of false rejections has increased to 13.0%. This is the direct cost of training on decoys: a class that learns not to trust a token similar to a valid one transfers some of that caution to real secondary arguments. The trade-off is beneficial in terms of the number of tasks, but it is precisely a trade-off.

Training: 34,709 lines (32,982 training and 1,727 validation), 23.8% of which are examples where the correct answer is refusal.

What is missing in these numbers

Comparisons with the baseline. Running the bare model without gates on the same 400 tasks has not been completed yet. Until it is, the failure rate of 60.5% should be interpreted as an absolute value, although it would be more accurate to interpret it as a difference.

Quantized build. Raw runs of several quantizations are located in the results directory; there is no report for them. Until there is one, all model numbers refer to the uncompressed build.

A set of negatives with duplicates. This is the next step, explicitly mentioned in the search report: we need a set of not “non-existent technologies,” but “non-existent facts about existing technologies” – a neighboring version of the tariff, a neighboring attribute, a neighboring parameter. Only with such a set will the threshold mean what is expected of it in operation. The numbers on it cannot be predicted; the previous two times, replacing the negative population worsened the result.

Where does the warranty boundary lie, and how much does it cost? See the next page.