Study 2 of 528 September 2026Croxio Ltd108 handovers, three arms, two models

Does reading a handover back catch a misunderstanding before the work is done?

The question

When the sender and the receiver understand a job differently, does making the receiver read the job back catch it before the work is done? And is a typed line better at this than a paragraph?

The material

Two kinds of job: write the copy for a landing page, and extract supplier invoices into a table. Six briefs for each: one clean and five with a planted trap. Every brief had hidden acceptance criteria the receiver was not shown.

Compared

Three ways of handing the same job over. Plain English with no readback. Plain English with the receiver writing back what it understood, judged by a separate model. A rogerwilco line with the receiver reading it back and a program comparing it with the sender's criteria.

Size

108 handovers: 36 per arm. Claude Sonnet did two runs per brief (72), Claude Haiku 4.5 one (36).

Graded by

Code for everything countable (word counts, formats, sets, amounts, dates, forbidden strings) and a separate model, Claude Opus, blind to the arm, for the rest (did the first sentence name the reader's problem, is the voice right). Defects were sorted into two kinds: a misread of the handover, which is what a readback is for, and a failure to do what the receiver had correctly understood.

Why we did it

The first study found that writing handovers as lines saves little money. What a line might buy instead is safety: because every fact has a slot, a program can compare what the receiver says it understood with what the sender meant. In aviation this is the readback. A controller gives a clearance, the pilot says it back, and the controller listens for the mistake. Here the mistake we care about is the receiver understanding the job differently from the sender and doing the wrong work well.

Two questions. Does a readback catch that at all? And if it does, is the typed form doing the work, or would asking for a paragraph back do just as well? The second arm, English with a readback judged by a model, is there to separate the format from the ritual.

What we tested

Each job came as a short handover message plus an attached brief, and the brief wins where the two disagree. The five traps were: a constraint that appears only in the brief, not in the message; a number in the message that is close to but not the brief's (120 words against "150 words, no more"); a message tail that contradicts the brief (pay in pounds only, when the brief says all currencies); a rule buried in the middle of the brief; and a categorical rule. For every brief the sender wrote down acceptance criteria and kept them: the word count, the tone, what must appear, what must not.

In the line arm the sender asks for a comprehension readback. It writes its criteria as typed constraints, keeps the values to itself, and names only the dimensions it will check. The receiver reads the brief, derives its own criteria and sends them back as acc=. A program compares the two sets key by key: the same key with a different value is a contradiction and fails; a key the sender holds that the receiver did not send is a gap and fails; a key only the receiver sent is an assumption, noted for the sender to confirm. The handover for the near-miss brief and the readback the strong model sent, as they were in the study. The study ran under the protocol's working name, so the lines say FC1 where today's say RW1; the sender's twelve criteria began len:150w:

FC1 HO lead-01 from=lead to=writer item=gh:crox/site#42 ask=DRAFT due=20261002 rb=acc keys=len+lang+tone+voice+fmt+must+no art=file:brief.md // hero copy per the brief: H1 plus a 120-word paragraph, UK English, we-voice, plain, no em dashes; open on the Monday problem; mention regulated-industry AI
FC1 RB writer-01 re=lead-01 from=writer to=lead item=gh:crox/site#42 ask=DRAFT due=20261002 acc=len:150w+lang:uk+tone:plain+voice:we+fmt:md+must:monday-problem+must:regulated-industry-ai+no:em-dash+no:hype-words+no:pricing+no:model-vendor-claims+no:testimonials+no:client-names do=copy not=pricing+model-vendor-claims+testimonials+client-names+subhead+cta asm=len-from-brief-not-tail+len-excludes-h1+150w-is-target-and-cap // Tail says 120w, brief.md says 150 words no more, so len is 150w from the brief; confirm or correct. Task: H1 plus one paragraph, Monday problem in sentence one, Crox shipped health and financial services AI, no clients named.

The handover said 120 and the brief said 150. The receiver reported len:150w, which matches the sender's own criterion, so the readback passed and the job went ahead on the right number; the receiver also wrote down, as an assumption, that it had taken the brief's number over the tail's. Had it reported len:120w, the check would have failed before any copy was written and the sender would have re-issued the handover.

In the English readback arm the receiver instead wrote a paragraph beginning "what I understand", and a separate model, shown the paragraph and the criteria, said whether they matched. In the no-readback arm the receiver simply did the job.

How we ran it

1. Wrote the twelve briefs (two kinds of job, six variants each) and the hidden acceptance criteria for every one.
2. For every brief and arm, a fresh agent received the brief and the handover message, and in the line arm the rogerwilco skill file as well. Nothing else.
3. No-readback arm: the agent did the job and handed in its output.
4. English-readback arm: the agent first wrote what it understood and waited. A blind judge compared the paragraph with the criteria. On a mismatch the sender corrected it in English and the agent read back again. One correction was allowed.
5. Line arm: the agent sent an RB line with acc= and waited. A program compared it with the sender's criteria. On a failure the sender re-issued the HO line with re= pointing at the failed readback, and the agent read it back again. One correction was allowed.
6. Corrections were sent to the same agent after it had produced its first output, so what the study measures is whether the misread was caught and whether one correction fixed it. The saving from catching it before any work is inferred from when the flag was raised, not measured.
7. Graded every output: code for the countable criteria, the blind judge for the rest. Counted tokens for the prompt, the outputs including the readbacks, and the corrections.

What happened

Table 1. Misreads are outputs that failed a criterion tied to the planted trap, the thing a readback is meant to catch. Other defects are outputs that failed a criterion the receiver had read back correctly, mostly the small model not putting the reader's problem in the first sentence. Caught means the readback check named the failing dimension. False alarms are flags on handovers whose output had no misread.

In words: with no readback, 2 of 24 strong-model jobs and 2 of 12 small-model jobs were done on the wrong reading and stayed wrong. With a line read back and checked by a program, the strong model had nothing to catch, because it had already read the brief's number, and the small model's one misread was caught and fixed. With an English readback judged by a model, every misread was caught and fixed. The program raised no false alarm on the strong model and ten on the small one.

Where the misreads came from. On the strong model, every one was the near-miss: the handover said 120 words, the brief said 150. Every no-readback receiver wrote 120. Both line-arm receivers wrote 150 and carried len:150w in their readbacks, each adding an assumption that said the brief's length overrides the tail's. Every English-readback receiver wrote 120 and said so in its paragraph ("the brief caps at 150, the lead says 120, 120 fits"), which the judge flagged. On the small model the second trap also fired: told "pounds only" in the message when the brief said all currencies, the no-readback receiver dropped the dollar invoice; both readback arms kept it.

The ten false alarms on the small model were all the same thing: its readbacks listed a few criteria and stopped, so the program reported gaps on set values (a must: or a no:) that the receiver then went on to honour. The one true flag was the 120-versus-150 contradiction.

The gap between reading and doing. The small model wrote must:monday-problem in all six of its copy readbacks and then, in four of the six jobs, did not put the reader's problem in the first sentence. Its readback was right and its work was wrong, and no check before the work can catch that.

Table 2. Tokens per handover (o200k): the prompt the receiver saw, what it wrote including readbacks, and correction traffic.

In words: a readback costs tokens. An English readback cost about 1.3 times a plain handover. The line handover cost almost five times, nearly all of it the skill file in the prompt, which the receiver does not need: the generated legend that replaced it is a quarter of the size.

What it means

Readbacks work, in either form. Four misreads reached an output across the two readback arms; all four were flagged and all four were fixed by one correction. The arm with no readback had no way to notice and kept all of its four.

The typed form changes behaviour before the check ever runs. On the strong model the line arm had nothing to catch: having to put a number after len= sent the receiver to the brief, where the number was 150. The English readback let the same model see the conflict, write it down, and still choose the handover's number. This is the clearest sign that the format matters and not only the ritual. It rests on one trap and two runs, so it is a strong hypothesis, not a result.

A program's check is more precise than a model's judgement, for a capable receiver. It ran 24 strong-model readbacks with no false alarm; the judge produced one (it demanded a nuance the brief did not require) and misread the criteria once on the small model. A judge is also a second model call per handover. The code check is free.

The check's precision collapses on a terse readback. A receiver that lists two criteria and stops looks, to the program, as if it missed the rest. The fix is in the protocol, not the model: tell the receiver which dimensions to report, and treat a missing set value differently from a contradiction.

A readback verifies understanding, not execution. The same typed criteria could be checked again when the work comes back; this study did not do that.

A comprehension readback is a cost that pays back only when it prevents rework. On a 120-word paragraph the rework is cheap. On a specification, a build, a payment or anything scheduled to fire, it is not.

What we decided

No handover without a readback.

Both readback arms caught every misread that reached an output; the arm without one kept all of its own. The sender owns the item until the receiver's readback passes its check.SPEC section 5, rule 1.

The comprehension readback is part of the protocol, with a closed set of criterion keys.

Two agents reading the same brief name the same fact differently (lang:uk, spelling:uk, lang:en-gb), and an open key set turns that into false gaps. The sender names the dimensions it will check with keys=, never the values, so a terse receiver knows what to report.SPEC section 5.2; rb=acc and keys= on the HO.

The receiver gets the generated legend, not the skill file.

The skill file in the prompt made a line handover cost nearly five times a plain one. The legend a program generates from the vocabulary is 766 tokens today (505 in its lean form) against 3,240 for the skill file; the skill stays on the sender's side.rogerwilco.py legend; served at /legend.

Two changes were proposed and are still open.

A gentler rule for a terse readback: a missing set value asks for a one-line confirmation instead of failing the readback, which would have removed all ten of the small model's false alarms while keeping its one true catch. And a check of the finished work against the same criteria, since the readback cannot see the gap between understanding and doing. The fifth study picked the second one up.Study 5, readback and set-back.

What this does not show

• 36 handovers per arm, and one small-model run per cell. Two of the five traps carry the whole signal; the other three were read correctly by both models in every arm, so those cells are flat.
• The near-miss brief said "150 words, no more", which a receiver can defensibly satisfy with 120. The difference in behaviour between the arms is real and visible; whether 120 is a defect depends on what the sender meant. A next version of the corpus should say "between 140 and 150".
• Claude receivers only, two kinds of job, criteria written by the experimenter. A sender and a receiver from the same model family share their misreads.
• Corrections ran after the work was done. In a live system the catch happens before the work starts; here that is inferred from when the flag was raised.
• Two harness bugs were found during the run and are reported rather than hidden: a plural mismatch (amounts against amount) caused two false alarms on the strong model before it was fixed, and the second readback was at first compared with the original handover instead of the re-issued one.

Where everything is

Corpus, harness, all 108 trials with their prompts, readbacks, outputs, checks and grades: research/pilot in the rogerwilco repository (harness.py, results/summary_final.json, results/rows.json). The original report is report.html in the same folder. The study ran under the protocol's working name, formcore, with the line prefix FC1; the trial files keep what the agents saw.

Study 2 of 5 · 28 September 2026 · 108 handovers · receivers Claude Sonnet (two runs) and Claude Haiku 4.5 (one run) · judge Claude Opus, blind to the arm · tiktoken o200k_base