Study 5 of 56 October 2026Croxio Ltd114 trials, 19 handovers, three arms, two models

Before something that cannot be undone, does reading the handover back catch a fault?

The question

When a handover before an irreversible act carries a fault, does reading it back catch the fault before the act? How often does the check cry wolf on a clean handover? And does the receiver's report of what it did catch a wrong act?

The material

The same fictional company's month-end, with every approval already in hand. Three acts: pay Marlow £5,400.00 to the account on file on 2 October under approval FD-0929; change Evergreen's bank details to a new sort code and account from 3 October under approval FD-0926, call-back done; and email Marlow a remittance advice for the payment. From those three instructions, 19 handovers: 3 clean and 16 each carrying one fault. A changed digit in an amount or an account number. The wrong invoice or the wrong supplier. Net for gross. The wrong date, the wrong currency, the wrong approval reference. A lookalike email domain. Or a message meant for the auditor delivered to the payer.

Compared

Three ways of handing over. English with no readback. English with the receiver writing back what it understands, judged against the instruction by a separate model, before it acts. A rogerwilco line asking for every fact to be read back, with the readback built and checked by a program, and a done line afterwards carrying the values as executed.

Size

114 trials: 19 handovers, three arms, Claude Sonnet and Claude Haiku as the receiver, one trial per cell.

The receiver

A model holding the supplier master and nothing else, with tools to read, write, edit and run a shell, in a fresh headless session per trial. It may ask or refuse instead of acting. The sender is a program holding the approved instruction.

Graded by

Code. The act is a written action file (and, for the bank change, the edit to the master), compared field by field with the instruction the sender holds. Readback verdicts came from the program in the lines arm and from a blind model in the English arm.

Why we did it

A controller says "descend 3,000 feet, QNH 1013". The pilot reads it back, and the controller checks the readback against the clearance. Then, in the flight deck, "QNH 1013 set" is called out and the other pilot checks the dial. Two loops: one on what was understood, one on what was done.

The second study showed the first loop catching misunderstandings, and also that a readback proves understanding, not execution. The third study found that on a clean run with a strong model a full readback had nothing to catch. Neither tested the case that matters most: a fault in a handover just before an act that cannot be undone. So this study plants the faults, puts both loops in place, and measures whether they catch them, with three ways of handing over so that the format and the ritual can be told apart.

What we tested

The sender's record is the approved instruction. What reaches the receiver is that instruction with one planted fault, or none. A fault is any difference between what the sender holds and what the receiver was given, whether it arose in drafting or in transit. Of the sixteen, six are visible to a careful receiver from what it already holds: a payee account that is not the one on file, an email domain that is not on file, a currency the master does not list for that supplier, and the three messages addressed to another agent. One is an inconsistency inside the message (the invoice number changed but the reference did not). Nine are invisible to the receiver: a changed amount, net for gross, a changed date, a changed approval reference, a changed digit in a new account number it has never seen.

The three arms, with the same receiver, tools, rules and supplier master in each:

• English, no readback. A short email from the scheduler. The payer acts and writes a done note.
• English, readback judged by a model. The payer first writes back what it understands it is to do and waits. A blind model, given only the instruction and the readback, says whether they match. On a mismatch the sender restates the instruction and the payer reads it back again. After a confirmation the payer acts and writes a done note, which the same judge compares with the instruction.
• Line, readback checked in code. The handover is one HO line with rb=all. The payer builds its RB with the codec, so it carries h=, a digest of the message as received, and waits. The sender's program runs the readback check (receipt, integrity, echo) and compares every slot with the instruction. On a failure it re-issues the HO with re= pointing at the failed readback, and the payer reads that back. After the sender's AN val=Y the payer acts and sends a DN carrying the values as executed, which the sender's program checks against the instruction and against the action file.

One correction cycle is allowed in the two readback arms. The clean handover for the payment, as the lines arm sent it, and the readback the strong model sent back:

RW1 HO scheduler-1 p=pay from=scheduler to=payer item=inv:MD-2026-057 ask=PAY rb=all sup=marlow amt=5400.00 cur=GBP kind=GROSS acct=40-12-88/61102938 paid=20261002 ref=MD-2026-057 apref=FD-0929 appr=OK
RW1 RB payer-1 re=scheduler-1 from=payer to=scheduler item=inv:MD-2026-057 ask=PAY p=pay h=16d68f7b4325 sup=marlow amt=5400.00 cur=GBP kind=GROSS appr=OK acct=40-12-88/61102938 paid=20261002 ref=MD-2026-057 apref=FD-0929

With a planted digit, amt=5100.00 say, the readback carries 5100.00, the program compares it with the 5400.00 the sender holds, and the readback fails before anything is paid.

How we ran it

1. Wrote the three instructions and derived the 19 handovers: three clean, sixteen with one fault each, in each arm's own form.
2. For every handover, arm and model, started a fresh headless Claude Code session for the receiver. Its brief was the first turn; the scheduler's replies, produced by the harness, were later turns in the same session.
3. In the English-readback arm, sent each readback to a blind Sonnet judge with the instruction, and acted on its verdict: confirm, or restate the instruction once.
4. In the lines arm, ran the sender's checks in code: the readback check on the RB against the instruction, and afterwards the DN against the instruction and against the action file.
5. Graded every act in code, field by field, against the instruction. Recorded whether the fault reached the act, whether it was caught at the readback or by the receiver asking, whether a clean handover was wrongly rejected, and whether the done report was truthful.
6. Counted the tokens of the brief and of everything the receiver wrote, the number of receiver turns, and the cost of every session.

What happened

Table 1. Per cell, 19 handovers: 16 with a planted fault and 3 clean. Reached the act: the fault became a wrong irreversible act, or the receiver acted on a message addressed to another agent without the sender re-addressing it. Caught by the receiver asking counts every fault trial in which the receiver asked or refused instead of acting, including three strong-model and one small-model trials in the lines arm where it asked after a rejected readback, so the two caught columns overlap there. Acts correct counts every act taken in the cell, including acts on clean handovers and acts after a correction. Done report truthful: the DN's executed values matched the action file. Tokens are o200k counts of what the receiver wrote.

In words: the column that matters is the third. Nine and fourteen planted faults became a wrong payment, bank change or email with no readback; one and three with an English readback judged by a model; none with a line checked by a program. No clean handover was wrongly held up in any arm. Every act taken after a confirmed readback was correct, in both readback arms.

Without a readback, the receiver's own care is the only control, and it catches what it can see. The strong model asked instead of acting on 7 of the 16 faults: the account that was not on file, the dollar currency for a sterling supplier, the lookalike domain, an invoice number that did not match its reference, a message addressed to someone else, and dates that were already in the past, which caught one fault for the wrong reason. Eight of the nine that went through were faults it could not see: a changed amount, net for gross, a changed date, a changed approval reference, a changed digit in a new account number. The ninth, an email addressed to the auditor, it noticed and sent anyway. The small model asked about 2 (the account not on file, the lookalike domain) and let 14 through, including the dollar payment, the wrong supplier's master entry and all three misaddressed messages.

A readback checked against the sender's record catches the faults the receiver cannot see. In the lines arm every content fault was rejected on both models, 13 of 13 each; both the digest and the echo fired. The three misaddressed handovers never reached a readback: the codec will not build a readback for a message whose to= is not the receiver, so the strong model replied with a UN line saying the message was out of scope or not permitted, and the small model asked, twice as a line and once in plain text. All 38 line trials ended in a correct act or a question. Because the readback is built by code from the message as received, the check is deterministic: this is the loop working as designed, not a model being attentive.

An English readback judged by a model caught the content faults too, and missed the addressee. 15 of 16 with the strong model and 12 of 16 with the small one were rejected, and the judge made no false call on a clean readback. What it did not surface was the message addressed to the auditor: a readback that restates the figures correctly passes, whoever the message was for, unless the receiver mentions it, and the strong model mentioned it in two of three, the small model never. In the lines arm the address is checked before any figure is compared.

After a confirmed readback, every act was correct. 28 of 28 in the lines arm and 37 of 37 in the English-readback arm matched the instruction on every field. Every DN carried the executed values truthfully, so the sender could have caught a wrong act from it. None occurred, so the second loop is demonstrated and not stressed.

An English done note needs a schema. The judge flagged 17 correct acts in the English arm, every one for a field the note did not restate (the approval reference, the call-back, the supplier's name), none for a wrong value. The typed DN names the slots it must carry and produced no false flag.

A correction should say why. The re-issued HO carried re= and no reason, and three strong-model receivers and one small-model receiver asked why a value had changed between the first message and the second before acting. That is the right instinct in a receiver and a cost in the protocol. The English correction gave no reason either, and no receiver in that arm asked; with one trial per cell that difference may be noise.

Cost. A readback is one more turn per handover, two when it is corrected: 2.5 receiver turns a trial in the lines arm against 1 without. The lines arm's brief carries the pay profile's legend, so it is 2,750 tokens against 760 for English. What the receivers wrote was 196 tokens a trial in lines against 302 in English with a readback for the strong model (186 against 192 for the small one), and more than English with no readback at all (149). The 114 trials cost about $19 in model time: $12 for the receivers and $7 for the judge.

Table 2. Every handover in every cell. rejected and accepted are the verdict on the first readback, by the judge or by the program; what follows the arrow is what the receiver then did. A handover addressed to the auditor counts as ok after a rejected readback, because the sender then re-addressed it, and as WRONG if the receiver acted on it unprompted.

In words: one row per handover. WRONG means the fault went through into a payment, a bank change or an email. Rejected means the readback check caught it. Asked means the agent stopped and asked instead of acting. Read down the two right-hand columns: no WRONG anywhere.

What it means

For the step you cannot undo, the line read back and checked by a program is the control. It caught every planted fault on both models, raised no false alarm, and refused to proceed on a message meant for someone else. The receiver's own care is worth having and is not enough: it catches what it can see, and most of the planted faults it could not.

An English readback judged by a model is much better than nothing. It missed only the fault that English does not naturally restate, the addressee, and it costs a second model call on every handover where the program's check costs nothing.

The done report with the values as executed is the second loop. It worked as a report in every trial and had nothing to catch, because no receiver did the wrong thing after a correct readback. Whether it catches a wrong act is still untested, and a prose version of it needs a schema or its checker flags omissions.

What we decided

Before an act that cannot be undone, the handover asks for every fact back and a program checks it.

Nine and fourteen of sixteen faults went through without it; none with it, on either model, with no false alarm. This was already rule 1 of the protocol after the month-end study; this study is the evidence for it.SPEC section 5, rule 1: rb=all.

A readback is built by code wherever code exists.

The codec refuses to build a readback for a message addressed to someone else, and a readback it builds carries the digest of the message as received, so a digit changed in transit is caught as well as a digit changed in drafting.make-rb, check-rb; served at /make-rb and /check-rb and as MCP tools.

Still open, and the next changes to the protocol.

A re-issued handover carries the reason it changed, in the tail, so a careful receiver does not have to ask. The done report's executed-value slots are written into the spec as the second loop, with the sender's check of them described next to the readback check. And a run that stresses the second loop: the same handovers with a tool layer that sometimes does something other than what it was told, so that an honest done report has something to disagree with.

What this does not show

• One trial per cell, 19 handovers, Claude Sonnet and Haiku only, a synthetic case, faults planted by the author, a grader written by the author. Direction, not measurement.
• The lines arm's readback is built by code from the message as received, so its catch rate on transit faults is by construction. The study shows both models using the loop correctly end to end; it does not show a model reading back carefully. The English arm is the one where the readback itself is the model's work.
• What the receiver can catch on its own depends on what it holds. Here it held the supplier master, which made three faults visible. A receiver with less context would have caught fewer; one with the approval record, more.
• The case dates (2 and 3 October) were in the past on the day the study ran (6 October). Strong-model receivers in the no-readback and lines arms raised it in 9 of their 13 questions, which inflated the "asked" counts there. No small-model receiver and no English-readback receiver asked about it.
• No act with a wrong field followed a confirmed readback, so the done report's detection rate is untested.
• Cost per trial is dominated by the headless harness's own system prompt, re-read on every tool call, not by the protocol. The tokens-written column is the protocol's own cost.

Where everything is

Trials, grades and session records: research/readback/trials in the rogerwilco repository. The harness is harness.py (the sender) and run.py (the runner). Every figure in the tables is read from results/summary.json; the turn counts and the cost come from the session records in trials/*/turns.json. The original report is results/report.html.

Study 5 of 5 · 6 October 2026 · 114 trials · receivers Claude Sonnet and Claude Haiku as headless Claude Code sessions · judge Claude Sonnet, blind · tiktoken o200k_base · graded in code