Nine measurements of prompting and an agent cluster: where a technique gives +48 points and where nothing can be told apart
A great deal has been written about prompting, and almost all of it is written the same way: a technique, a pretty example, the conclusion “it works”. What such articles usually lack is the number at which the technique stops working, and any admission that on some material it did not work at all.
We have nine measurements in a row. Not one — nine, and they argue with each other. On one kind of material the best technique gives +48 points, on another seven techniques out of eight lose to a plain question, on a third we honestly could not tell anything apart. Below are all nine: what we asked, how we measured, what came out and what that answer does not say.
There is a separate section, “What we did not measure”. It is mandatory, and it is not an excuse: it shows where our knowledge ends. For instance, the question “is it worth having a model write components” we never measured at all, and the article says so outright instead of filling the gap with reasoning.
Five answers straight away
| question | answer | source |
|---|---|---|
| Is there any difference between prompt variants at all? | There is, and a large one — but only where the answer has to be derived. +28.8 points on arithmetic, +48.0 on freshly generated tasks. On multiple-choice questions, zero. | measurement 9 |
| How do you orient yourself on tests? | We do have material where the answer is checked by running tests (HumanEval). There no technique rose above the plain question, and the worst one lost 14.4 points. | measurement 9 |
| What does a cluster of agents and a detailed brief buy? | None of the five attempts showed a gain. Three ended in “cannot be told apart”, one was never finished, one showed a gain that vanishes under the proper measure. | measurements 1, 2, 3, 5, 7 |
| What does Digit give as a platform? | The structure does assemble and the write boundaries hold for file tools — but not for the terminal. We did not show any gain from a three-level cluster. | measurements 5, 7 |
| Is it worth having a model write components? | Not measured. None of the nine measurements ever asked that question. | — |
In one line, if there is no time to read further: let the model count out loud where counting is required; do not get in its way where code has to be written; and do not expect the way agents are organised to pay for itself — with us it never once did.
What we were asking
We had three questions, and they kept coming back, because the first answers did not satisfy us.
- Does the way a prompt is written change anything? A prompt is the text you send the model. Eight well-known techniques against the baseline: a plain prompt carrying the question and no instructions at all.
- Does the way agents are organised buy anything? A detailed brief against a short one; a cluster (an overagent hands work to leads, leads to workers) against a single agent doing everything in a row.
- What does it cost? In calls, in tokens, in hours and — where we paid with money — in dollars.
All nine measurements are these three questions on different material. Our
directories are called measure…measure9, and from here on they appear as
“measurement 1” … “measurement 9” in that same order.
Nine measurements on one screen
| measurement | what we asked | what it was measured on | what came out |
|---|---|---|---|
| 1 | does a detailed brief give a better result than a short statement of the same task | 3 tasks in live code, 6 agent runs, a blind judge | could not be told apart: the direction of the difference is inconsistent (1/2, 2/1, 2/2 defects) |
| 2 | does a detailed brief give a gain in answers about the task | 12 cells of work, 4 models, 2496 verdicts | no gain on any model under the headline measure |
| 3 | does a cluster structure give a measurable gain | 6 debt tasks in live code, two groups with two repeats each | cannot be told apart: both groups took 12 out of 12; the declared cluster shape never assembled |
| 4 | do prompting techniques differ from one another | the same 12 cells, 4 models, 11,808 calls | on no model did any technique beat the plain question |
| 5 | does a three-level cluster buy anything on Digit | — | not finished: stopped by its own rule declared in advance |
| 6 | do the techniques hold up on a large question bank | 120 questions about repository history, 4 models + Claude Code, 12,240 calls | seven techniques out of eight lost to the plain question |
| 7 | what does each level of the structure and the master prompt give | 12 tasks, two full grids of nine cells | cannot be told apart: a floor — 0 green checks out of 12 in all nine cells |
| 8 | does the picture of measurement 6 break on 30B+ models | the same tasks, a model ladder from 0.87 to 27 billion parameters | it does not break; and what does change with size changes the wrong way: the threshold upward is cleared only on the smallest rung |
| 9 | do the techniques work where there is something to derive | 90 tasks of four kinds, 9900 calls | they work: +28.8 and +48.0 points; on code they do harm |
Measurements 1, 3, 5, 7 are about agents and the organisation of work. Measurements 2, 4, 6, 8, 9 are about the text of the prompt. Measurement 9 is the only one whose answer came out confidently positive, and it is clear below why: it is the only one whose material consisted of tasks that have an internal derivation.
How we measured
One rig across all nine measurements, and it is worth describing before the numbers — otherwise there is nothing to weigh the numbers against.
The plan of a measurement is written before the first call and hashed. We call it pre-registration: the hypothesis, the material, the scoring rule, the threshold and — mandatory — the falsifiers, that is, outcomes declared in advance under which the hypothesis counts as refuted. Any later edit is filed as a separate amendment with its own hash, not as a rewrite of the text. This is a defence not against ill will but against oneself: it is very easy, once you have seen the numbers, to recall that “we did, of course, mean this other measure”.
The answer is checked by a machine, not by a human and not by another model. A number, a letter, a test run, a command’s exit code. The judge is deterministic and blind: it receives only the pair “question — answer”, and which arm and which model that answer belongs to is stitched onto the result after the scoring.
The threshold is double. A difference counts only if both conditions hold: the 95 per cent interval of the paired difference does not cover zero and the absolute difference is larger than the spread between repeats of one and the same arm. The second condition cuts off the case where the difference is smaller than the noise between two runs of one and the same setup. If either condition fails, we write “cannot be told apart” — and that is not a polite way of saying “did not work”. It is a third outcome, and it means exactly what it says: this measurement does not tell this apart.
Three numbers are counted, not one. Correct, “I don’t know” (a refusal) and confidently wrong. Without the refusals column any conclusion of the form “technique X is worse” is invalid — and it is clear below why this is not pedantry but the main source of error in measurements like these.
Result 1. Techniques work where there is something to derive
This is measurement 9 — the most recent one and the only one with a confidently positive answer. 90 tasks of four kinds, 9900 calls, every answer marked by a machine. There are fifteen prompt variants: nine techniques and six mirrors in which only the sentence about the answer format is changed — the next section shows what they are for.
| kind of task | baseline | what worked | difference, points |
|---|---|---|---|
| Arithmetic — GSM8K: school problems taking several steps | 41.6 % | self-consistency | +28.8 |
| Fresh tasks — the same arithmetic, but invented by our generator after these models had been trained | 44.0 % | self-consistency | +48.0 |
| General knowledge — MMLU: a question and four options | 55.2 % | nothing worked | all nine within ±7 |
| Code — HumanEval: “write a Python function”, the answer is checked by running tests | 53.6 % | nothing; worst of all tree of thoughts | −14.4 |
The difference is counted in percentage points: the baseline takes 41.6 % of the tasks and self-consistency 70.4 % — that is the +28.8 points.
The nine prompt variants that were compared:
- baseline — the question only, no instructions; the others are compared with it;
- chain-of-thought — tell the model to reason step by step out loud before naming the answer;
- self-consistency — ask the same question five times and take the answer that came up most often;
- few-shot — put three solved tasks of the same kind into the prompt;
- zero-shot RCTF — role, context, task, format; no solved examples;
- tree of thoughts — ask for three candidate solutions and pick the best by declared criteria;
- chain-of-verification — a draft, three checking questions about it, a fix;
- the full brief — our format: the whole statement of the task, with a role, what is known, boundaries and what would refute it;
- the master prompt — our assembled prompt: the calling context, the authority, the order of reasoning.
The first six are the standard techniques, and their write-ups are easy to find under exactly these names. The last two are ours, and you will not meet them in anyone else’s article.
Here is the baseline — one line naming the kind of task, the task itself, and nothing else:
Задача на счёт в несколько шагов. Текст задачи — по-английски, как в источнике.
ВОПРОС
Leila buys 3 cucumbers from the market. Cucumbers are $2 each. Jack buys 5 tomatoes from the grocery store. Tomatoes are $1 each. Chase buys 1 head of lettuce from the farmer’s market. Lettuce cost $3 each. Together, how much did the three of them spend to make a salad for the potluck?
And here is the winner — self-consistency. The prompt itself is an ordinary chain-of-thought one, and the whole technique is that the request is repeated five times and the most frequent answer is taken:
Задача на счёт в несколько шагов. Текст задачи — по-английски, как в источнике.
ПОСТАНОВКА
РОЛЬ. Ты счётчик. Тебе дают текстовую задачу, и ты доводишь расчёт до одного числа.
КОНТЕКСТ. Задача взята из открытого набора школьных задач. Ответ — целое число; калькулятора и выполнения кода у тебя нет, считай сам.
ЗАДАЧА. Реши заданную ниже задачу и назови число.
ФОРМАТ. Формат ответа предписан в конце запроса и обязателен.
ВОПРОС
Leila buys 3 cucumbers from the market. Cucumbers are $2 each. Jack buys 5 tomatoes from the grocery store. Tomatoes are $1 each. Chase buys 1 head of lettuce from the farmer’s market. Lettuce cost $3 each. Together, how much did the three of them spend to make a salad for the potluck?
ПОРЯДОК ОТВЕТА
Прежде чем отвечать, рассуждай по шагам и пиши рассуждение вслух:
1. назови, что именно спрашивают;
2. выпиши, что дано, — величины, условия, краевые случаи;
3. веди решение по шагам, каждый шаг отдельной строкой;
4. проверь полученное на том, что дано: сходится ли;
5. назови ответ.
ПОСЛЕДНЕЙ строкой ответа напиши ровно `ОТВЕТ: <число>`, где <число> — целое число без единиц измерения, знака валюты и разделителей разрядов.
Если не знаешь — `ОТВЕТ: НЕ ЗНАЮ`.
Copy it and try: this is not a retelling but the very text that went to the model, written in Russian around an English task. The “ПОРЯДОК ОТВЕТА” block is chain-of-thought. If five calls are too expensive, take one: that will be pure chain-of-thought, word for word the same prompt without the vote. It nearly catches up.
| kind of task | baseline | self-consistency | chain-of-thought |
|---|---|---|---|
| Arithmetic (GSM8K) | 41.6 % | +28.8 | +23.2 |
| Fresh tasks | 44.0 % | +48.0 | +40.0 |
| General knowledge (MMLU) | 55.2 % | cannot be told apart | cannot be told apart |
| Code (HumanEval) | 53.6 % | cannot be told apart | cannot be told apart |
| Everything together | 49.1 % | +17.1 | +10.9 |
Result 2. The most expensive sentence in a prompt is the ban on thinking
The most expensive thing you can append to a prompt is “Answer in exactly one line. Write nothing else”. Here is its price at zero technique: baseline against baseline, differing by exactly that sentence.
| kind of task | “answer in exactly one line” | “on the last line of the answer” | difference |
|---|---|---|---|
| Everything together | 32.2 | 49.1 | −16.9 |
| Arithmetic | 9.6 | 41.6 | −32.0 |
| Fresh tasks | 6.7 | 44.0 | −37.3 |
| Code | 48.0 | 53.6 | −5.6 |
| General knowledge | 54.4 | 55.2 | −0.8 |
On arithmetic that is 32 points — more than any technique in the measurement adds. Where there is nothing to reason about (general knowledge is picking a letter out of four), the sentence costs nothing: −0.8 points, threshold not cleared.
It gets stranger. Impose that ban on all nine variants at once and they become almost indistinguishable: 31.8–38.9 % against 32.2 % for the baseline. Only chain-of-thought breaks through: 24.8 % against 9.6 % on arithmetic and 24.0 % against 6.7 % on fresh tasks. The model manages to count even in a single line. But that is several times less than the same 64.8 % and 84.0 % when reasoning is allowed.
And the other way round: the baseline with no format requirement whatsoever gives 60.9 % against 49.1 % for the very same baseline under “on the last line”. Any requirement about format costs something — the only question is how much.
This is the first thing worth taking away from the article. If you compare two prompts whose format requirement differs, you are measuring the format, not the technique. The difference between formats is larger than the difference between all eight techniques.
Result 3. Where a technique does harm: code and the right to keep quiet
On code nobody rose above the baseline. Baseline — 53.6 %, tree of thoughts — 39.2 %, chain-of-thought — 44.0 %.
The reason is visible in the refusals — that is what we call the cases where the model answered “I don’t know”. On code the baseline keeps quiet in 26.4 % of cases, and what it does hand over passes the tests in 72.8 %. Chain-of-thought keeps quiet in 0.8 % — and its code passes the tests in 44.4 %. The technique forces the model to produce something every time, and what it produces more often does not work. Where the answer is checked by running tests rather than by comparison with a reference, that costs the most.
Result 4. On recall questions the techniques lose
Measurements 4 and 6 took different material: multiple-choice questions about the history of our repositories — which file a commit touched, whether a path exists in the tree, which commit first added a file. The respondent does not see the tree: it has only the commit subject and the list of options. That is, a task of recalling a fact, not of deriving one.
Measurement 6: a bank of 1238 tasks, a sample of 120, 15 question types, four open models, 12,240 calls (11,520 through ollama plus 720 in a separate Claude Code league). Random guessing on this sample is 28.72 %.
| arm | correct, % | difference from baseline | interval of the difference | threshold |
|---|---|---|---|---|
| baseline | 37.6 | — | — | reference |
| self-consistency | 40.1 | +2.5 | [−2.4; 7.7] | cannot be told apart |
| chain-of-thought | 30.0 | −7.5 | [−12.1; −2.9] | downward |
| the master prompt | 26.3 | −11.3 | [−16.1; −6.4] | downward |
| tree of thoughts | 25.9 | −11.6 | [−16.8; −6.5] | downward |
| the full brief | 19.2 | −18.4 | [−24.0; −12.7] | downward |
| chain-of-verification | 18.7 | −18.9 | [−23.9; −13.7] | downward |
| few-shot | 16.8 | −20.8 | [−26.2; −15.4] | downward |
| zero-shot RCTF | 15.6 | −22.0 | [−27.1; −17.0] | downward |
Seven techniques out of eight are significantly below the plain question, and none is above. Measurement 4 on the same sort of material (12 cells of our own work, four models, 11,808 calls) gave the same thing: on none of the four models did any technique beat the plain question.
But reading this as “techniques do not work” is not allowed, and here is why. Half of the loss is the price of honesty: the full brief and chain-of-verification explicitly tell the model to answer “I don’t know” when the answer does not follow from the data, and they refuse in 46–56 % of cases, while accuracy counts a refusal as an error. The other half is a property of the material: there is nothing to derive the answer from, so there is nothing to derive, and a technique that makes the model reason makes it reason in a void.
That is precisely why measurement 9 with tasks that have something to derive came about. The pre-registration of measurement 9 opens with an admission: “The setup was wrong, and this is acknowledged.”
A separate league of the same measurement is Claude Code without tools, 45 tasks. Baseline — 48.7 %, the full brief — 28.2 % (−20.5, interval [−33.3; −7.7]), the master prompt — 35.9 % (−12.8, [−23.1; −2.6]). Both of our own formats are significantly below the plain question. The mechanism is the same: the full brief gives 31 refusals out of 45, the plain question — none.
Result 5. Model size fixes nothing
The objection we received after measurement 6 sounded reasonable: “the measurement stopped at 14 billion parameters, and on larger models the picture will change”. Measurement 8 tested exactly that: the same material, the same nine arms, a ladder within one family from 0.87 to 27 billion parameters — thirty times over in size with one training recipe. Exactly one feature changes.
| stage | baseline, % | best arm | its share, % | difference | cleared the threshold upward | arms below baseline |
|---|---|---|---|---|---|---|
| 0.87B | 27.9 | chain-of-thought | 40.4 | +12.5 | self-consistency (+11.5) | 0 of 8 |
| 2.3B | 38.5 | zero-shot RCTF | 40.4 | +1.9 | no | 3 of 8 |
| 4.7B | 48.1 | tree of thoughts | 42.3 | −5.8 | no | 5 of 8 |
| 9.7B | 44.2 | tree of thoughts | 50.0 | +5.8 | no | 3 of 8 |
| 27B | 46.2 | self-consistency | 55.8 | +9.6 | no | 4 of 8 |
The picture did not break — it broke the wrong way. The threshold upward is cleared only on the smallest rung. At the top of the ladder, where the hypothesis demanded a win, not a single arm turned out to be above the plain question. The observed dependence on size is the reverse one: the larger the model, the more confident the plain question and the more techniques end up below it.
On the material of measurement 9, where there is something to derive, the same thing is visible from the other side:
| model size | baseline | self-consistency | difference |
|---|---|---|---|
| 0.87B | 16.1 | 40.6 | +24.5 |
| 2.3B | 28.9 | 72.2 | +43.3 |
| 4.7B | 87.8 | 87.8 | 0.0 |
| 9.7B | 96.7 | 90.0 | −6.7 |
| 27B | 97.8 | 97.8 | 0.0 |
On the junior models the technique drags the result up two and a half times. On the senior ones there is nothing to add: the baseline already takes 98 out of 100. This is not “on large models the techniques are useless” — this is “the material ran out before the model did”. If you work with a large model on easy tasks, a technique will give you nothing, and that is not about the technique.
Result 6. An agent cluster and a detailed brief: five attempts, not one gain
This is the second half of the work, and it answers the question “what does the organisation of agents’ work give”. In short: with us it never paid off, but none of the five attempts gives the right to say “a cluster does not help” — they say different things and for different reasons.
Attempt 1 (measurement 1): a detailed brief against a short one on live code
Three tasks in a real fork of the cwm window manager, each done twice: one agent gets the full brief (role, what is known with provenance, neighbours and write boundaries, one task, a falsifier, readiness as the output of commands), the other gets the same task in one or two sentences. Six runs, six clones, six branches, and a blind judge who knew neither about the groups nor that a measurement was under way.
All 12 acceptance criteria out of 12 are “met”: not one “not met”, not one “could not be verified”. The primary metric is the defects found by the judge:
| pair of tasks | the full brief | the short one | direction |
|---|---|---|---|
| moving a window inside a stack | 1 defect | 2 | the brief is better by 1 |
| a switch for the cursor jump | 2 | 1 | the short one is better by 1 |
| reconciling groups with the ribbon | 2 | 2 | a draw |
The threshold was declared before the run: a difference counts only if its direction is the same in all three pairs. It is not the same. The outcome is “could not be told apart”, and that is not the same thing as “there is no difference”.
The only metric with a consistent direction is expenditure: the short brief turned out to be more expensive in all three pairs (+21 %, +73 %, +39 % of tokens). But that is not a win for the brief either: more expensive because more was done. Separating “more expensive because the brief is worse” from “more expensive because more was done” is beyond this run.
The brief did, however, turn out to have a measurable effect where nobody was looking for it — in the boundaries. Files changed outside the declared set: with the full brief 1, 0 and 0; with the short one 1, 3 and 12. In the worst case a branch without boundaries went twelve files outside the set, seven of them into somebody else’s already-merged stack, and rewrote the numbers there with plausible ones. The full brief is worth something not because it makes the work better but because it keeps the performer inside its cell.
And here is the counter-argument straight away: the section “known: verified, do not reopen” suppresses re-checking. A wrong fact that got into such a section travelled with the briefs into three branches at once. A detailed brief is capable of carrying an error that a short brief cannot carry. One of the three pairs is contaminated on top of that: the agent with the short brief read its neighbour’s tree, and no brief forbade it — the ban covered writing only. The pair was marked contaminated and not discarded: throwing out an observation after seeing its result is fitting the data.
Attempt 2 (measurement 2): the same brief, but we ask the model about the task
Twelve cells of real work, five question types about each, four open models, five repeats, 2496 verdicts. Both groups receive a byte-identical backing — an inventory of the tree, eight commands, a list of resources; only the statement block differs.
Under the headline measure (only questions whose answer does not lie in the brief itself):
| model | short statement | the full brief | difference | interval |
|---|---|---|---|---|
| 14B | 69.4 % | 71.7 % | +2.2 | [−9.4; +13.9] |
| 3B | 10.0 % | 17.2 % | +7.2 | [−1.7; +17.2] |
| 7B | 50.6 % | 48.3 % | −2.2 | [−12.2; +6.7] |
| 4B | 50.6 % | 42.8 % | −7.8 | [−15.0; −1.1] |
On no model is a gain shown. The only interval that does not cover zero looks against the brief — and is extinguished by the spread of repeats (8.3 points against a difference of 7.8).
If you count over all the questions at once, on the smallest model a gain of +17.6 points [+8.5; +26.3] appears — and that is exactly the outcome that had been named a falsifier in advance: the win holds where the answer lies in the brief itself and vanishes where it does not. What is measured is retelling, not benefit beyond it.
What matters: the model did read the brief. The positional control (a question whose answer exists only in the brief) was passed by all four models with room to spare: +66.7, +25.0, +66.7, +66.7 points. The negative result cannot be written off as “it did not read”.
Attempt 3 (measurement 3): a cluster against a single agent
Six debt tasks in a live tree, each with a machine check that is red before the work. The control: one agent gets all six cards. The experiment: an overagent, two leads, six workers with one card each.
The first line of the report is a disqualification: the platform twice refused
the leads permission to spawn workers. Verbatim:
Error: No such tool available: Agent. Agent is disabled for this session, in subagents as well as here.
In the first repeat there were no workers at all; in the second, six were spawned
by the overagent itself. The declared cluster shape never assembled.
| bucket | single agent | “cluster” |
|---|---|---|
| green check | 12 of 12 | 12 of 12 |
| red, readiness not declared | 0 | 0 |
| false delivery (declared done, check red) | 0 | 0 |
| forgery | 0 | 0 |
Both groups are at the ceiling — that is the second falsifier declared in advance: the measurement does not tell them apart. The verdict is “cannot be told apart”.
The only thing that confidently follows from this measurement is the price: the cluster turned out to be slower by the clock (2.82×) and more expensive in tokens (9.21×), that is, parallelism did not pay off here at all. With two caveats, both from the report itself. The first is stated in it outright: part of that price was bought by the platform’s refusal, the leads spent moves on attempts to spawn workers, and “by how much exactly is not measured”. The second turned up during the cross-check: the section “cluster shape” reports that three agents worked in the first repeat, while the price ledger for the same repeat counted seven. The figure 9.2× is inflated by an unknown amount, and it cannot be quoted as measured.
Attempt 4 (measurement 5): the question we did not finish
Measurement 5 asked the most direct question: does a three-level cluster buy anything on Digit — our own agent platform where nested agents are allowed. The plan was written out in full: six cards, two groups, three repeats, 36 observations, a blind machine judge, four outcome buckets, a threshold stated as a number (the cluster has five more greens out of eighteen).
There was no run. No harness, no baseline measurements, not a single launch of the judge. The measurement stopped on its own rule declared in advance: if a lead hands work to workers in fewer than half the cases, delivering a verdict about the cluster is forbidden — because a group with a lead that did not split the work is not a cluster but the same single agent through one extra level. The probe gave 2 out of 4: twice the lead split the task between two workers, once it gave everything to one, once it did not delegate at all. Exactly on the boundary.
The next step — checking whether a larger model cures this — ran into a busy graphics card: the probe script requires 12 gigabytes of free memory and refuses to work without them, and the card is shared and other people’s processes were running on it at that moment. After that the question moved on to measurement 7 in an extended setup. So let it be recorded: we asked, we did not finish, and here is why.
Attempt 5 (measurement 7): what each level gives
Measurement 7 decomposed the organisation of work into features: a single agent, two levels, three levels; the master prompt at each of the levels separately. Nine cells, two full grids, twelve tasks, all red before the work.
The result is 0 greens out of 12 in all nine cells of both grids. Not one configuration fixed a single task. This is a floor: there is nothing to tell apart.
The reason is named as a number, and that is the most useful thing that came out of the measurement. The same material, the same judge, the same model; only the number of cards in one session changes:
| cards in a session | green gates |
|---|---|
| 1 | 1 green out of 5 repeats |
| 2 | 1 out of 5 |
| 6 | 0 out of 3 |
| 12 (as in the grid) | 0 out of 18 cell-runs |
That is, there is no threshold between “one” and “two”: there is a low success rate that becomes zero at six and at twelve. The model that was affordable for the measurement does not take on editing a file at all; the model that does take it on costs far too much time — its single green gate took 414 seconds against a limit of 900, and a full grid on it did not fit the declared ceiling of six hours. A larger rung does not fit into the card’s free memory: 23.9 gigabytes against the 18.3 available.
The structure does assemble: the overagent spawns leads, the leads spawn workers, the split rate is 0.50 in both grids. There is simply nothing to compare when both groups are at zero.
And one finding which, in the report’s own words, is “worth more than half the
measurement”. The overagent went outside its own write boundary live: the
boundary is held by the file tools and not by the terminal, and an ordinary echo
into a file went straight past it. That is, what is locked up is the subagent’s
tools, not the subagent.
What our own dataset says about this
Besides the measurements we have a marked-up record of real cluster work: 44
cells, each one task handed to one agent, with a brief, branches, gates and an
outcome. Outcomes of four kinds: success 15, mixed 14, negative (the check
worked against what the customer intended) 13, could-not-verify 2. Plus six
controlled runs — the very three pairs from attempt 1.
Publishing that set is not allowed: the material comes from private repositories, and the explicit ban is written into the set itself. But one of its numbers is worth quoting here, because it explains why controlled runs were needed at all: the style of the brief cannot be recovered from the historical reports for 33 cells out of 44. Only two leads wrote in their report which template they used for the briefs. On material like that the question “does a detailed brief help” is not answerable at all, however long you stare at it.
Result 7. What it costs
A token is a chunk of text by which the model measures volume and by which you are billed.
A technique. Self-consistency is the best technique of measurement 9, +17.1 points across the whole material. It costs five calls instead of one and 1,646,559 output tokens against 69,545 for the baseline: 23.7 times more. One point of accuracy costs roughly ninety thousand extra tokens. On arithmetic and on fresh tasks that pays off. On general knowledge and on code it does not: there is nothing there to pay for.
A cluster. In measurement 3 the cluster cost 2.82× by the clock and 9.21× in tokens relative to the single agent — with an identical result of 12 out of 12 in both groups. Both multiples have to be read with the caveats from the previous section, but their sign is not in dispute: parallelism did not pay off here.
Money. The one place where we paid not with our own graphics card but with money is a separate cell of measurement 4, where the subject was Claude Code. There all three currencies were measured at once.
| arm | calls | input tokens | output tokens | of that “thinking” | $ per task |
|---|---|---|---|---|---|
| baseline | 1 | 1,844 | 1,966 | 1,496 | 0.0676 |
| zero-shot RCTF | 1 | 2,441 | 92 | 74 | 0.0267 |
| self-consistency ×5 | 5 | 12,927 | 7,808 | 1,313 | 0.2262 |
| chain-of-verification | 3 | 8,984 | 2,284 | 992 | 0.1470 |
| tree of thoughts | 2 | 7,205 | 2,928 | 633 | 0.1453 |
| all nine arms | 16 | 50,627 | 17,927 | 5,215 | 0.8562 |
Here everything turns over. On local models the format requirement cost quality; on a large closed model the most expensive arm by output is the plain question with no instructions at all: 1,966 output tokens, of which 1,496 go on internal reasoning. Zero-shot RCTF, which asks for a short answer in a given form, is 21 times cheaper. The output contract on this model saves money rather than spending it.
And one more number useful to anyone about to measure for themselves: three arms out of nine (self-consistency, chain-of-verification, tree of thoughts) eat up 60.6 % of the budget and 71 % of the model’s time. A full bank of a thousand tasks in nine arms and three repeats is 48,000 calls, about 2,569 dollars and 84 hours. That estimate was obtained for 0.17 % of its price: we ran three tasks and did the arithmetic.
What we did not measure
The list is honest rather than an excuse. It shows where our knowledge ends.
- Having a model write components — never measured, not once. None of the nine measurements asked whether it is worth having a model write application code, and no number here answers that. To answer it you need a different measurement: component tasks with acceptance tests written by someone other than the author of the solution; two groups — the model and a human, or the model with and without a technique; the metric being test passage and the cost of fixing after the first delivery. We have no such measurement.
- We hardly measured agents’ work — we measured answers about work. Measurements 2, 4, 6, 8 and 9 measure what a model says about a task, not what it does. Carrying the conclusion over from “understanding” to “work” is forbidden by their own pre-registrations. Real work was measured only by measurements 1, 3, 5 and 7 — and there it was either “could not be told apart”, or “cannot be told apart”, or there was no run at all.
- Not a single measurement with built-in reasoning switched on. The
thinkmode is deliberately off in every run: the technique was measured in the text of the prompt, not in a setting. About reasoning models we have nothing. - The technique is not separated from the length of the prompt. The full brief and the master prompt have both more tokens and more text. We ran no separate control of the “long but empty prompt” kind, so part of the effect may be volume alone.
- Code quality is not measured. On HumanEval what counts is test passage and nothing else: neither readability, nor complexity, nor safety.
- No long context, no tool use, no dialogue, no tasks in Russian. The task texts are English, the rig is Russian.
- Models larger than 27 billion parameters were not measured at all, and those that were are one family in four-bit quantisation. Part of the spread between rungs may be explained by quantisation rather than by size.
- The continuity of cluster work — half of the skill — was not measured. All the tasks fitted into a single session; no limits, no handover of state to the next agent.
- The isolation of a subagent is not proven. The write boundaries hold for file tools and do not hold for the terminal — that is not a supposition but a case we caught.
- The price in money is measured in one place out of nine. Everywhere else it is tokens and graphics-card hours; the tariff was not counted.
- There are too few repeats everywhere. Two repeats on some rungs, one on the rest, three pairs in measurement 1, one run per cell in measurement 7. Run-to-run spread is the largest hole in all our plans, and it does not close by being careful about the scoring.
What follows from this
About the prompt. Let the model count out loud where counting is required: +23.2 points on arithmetic for a single sentence. Do not get in its way where code has to be written: on HumanEval no technique rose above the plain question. And watch the format requirement — it costs more than any technique and easily substitutes itself for the subject of the comparison.
About agents. In our measurements a detailed brief never once bought quality, but twice it measurably kept the performer inside the boundaries — and in one case the price of going outside them was seven of somebody else’s files in an already-merged stack. The cluster never once paid off. If you are building a multi-level organisation of work, it is worth checking the dullest thing first: does the task reach the worker at all. With us the lead handed out work in half the cases, and a whole measurement stopped right there.
About measurements. Three outcomes, not two. “Cannot be told apart” is not a polite refusal but a result: it says this measurement does not tell this apart, and it points at what to fix in the next one. We have more such outcomes than positive ones, and it is exactly they that moved the work forward.
Check it yourself
You do not need twelve thousand calls for this. A couple of dozen tasks and half an hour will do.
- Take tasks where the answer is checked by a machine, not by you: a number, a letter, a test run. Otherwise you will be measuring your own mood.
- Make exactly two prompts per task — the baseline and the same one plus “reason step by step”. The closing sentence about format must match byte for byte in both, otherwise you will measure it rather than the technique.
- Count three numbers, not one: correct, “I don’t know”, confidently wrong. A technique often does not make the model more right — it removes its right to keep quiet.
- Check the rig first, not the model. Run reference solutions and knowingly bad answers through your own checker. On HumanEval ours looks like this: 25 reference solutions out of 25 pass the tests in 0.2 seconds, while an empty answer, prose and a stub all three fail. A check that does not go red on something knowingly bad checks nothing.
- Run it twice on one and the same model. The spread between two runs is your threshold: a difference smaller than it means nothing.
- Take a smaller model. On a large one the tasks will run out before the model does, and you will see a zero where there is none.
Where our own reports contradict themselves
Nine measurements means nine stacks of tables, assembled by scripts at different times, and some of their lines do not agree with one another. We did not pick the convenient number: both are named below, and the article leans on what the run confirms rather than on what looks nicer.
One closing caveat: on 2 September 2026 the conclusions of this article were revised — the earlier measurement compared techniques with differing requirements for the answer format and was therefore measuring the requirement rather than the technique; what exactly was wrong and which numbers used to stand here is in the aside of the “For a developer” mode.
