Study 6 of 610 and 11 October 2026Croxio Ltd2,762 trials pre-registered on two Claude receivers, then 2,196 on two OpenAI receivers and a 329-trial extension

Before an act that cannot be undone, does reading back the plan catch what an echo cannot?

The question

Study 5 planted faults between the sender's record and the message, and had the receiver build its readback by code from the message as received. Every fault was caught by construction. This study asks what a readback catches that nothing cheaper would: a fault in the receiver's own reading, and a fault in the tool.

The material

The same fictional month-end as study 5, with the payer handed three acts it cannot take back: release a payment, change a supplier's bank details, send a remittance advice. The handover names the supplier and the figures but not the payee account or the address, which the receiver resolves from a supplier master, as an accounts-payable function does.

Three fault blocks

Block 1, drafting: study 5's sixteen planted faults in the handover, three clean. Block 2, the receiver's reading: the handover is correct and the context is built to mislead, fourteen confusers in two variants and fifteen clean contexts. Block 3, execution: the tool executes something other than what it was asked, fifteen ways, and prints what it did.

Compared

No readback. A prose plan readback judged by a model. The echo readback built by code, study 5's arm, kept as the control that should catch nothing at the receiver's locus. An execution readback written by the receiver, carrying the values it will use, checked by code against the record. The same readback as a fixed-shape English block parsed by code.

Receivers

Claude Sonnet 5.5 and Claude Haiku 5.5, as headless sessions; Claude Sonnet 5.5 as the judge in the prose arm; Claude Haiku 4.5 in one exploratory probe.

Graded by

Code: the act as the tool executed it against the record, field by field; the readback and the done report against the record and against the tool's printout. Every proportion carries its exact 95% interval. Hypotheses, tests and sizing were written down before the run; the protocol, its amendments and every pilot are in the repository.

Why we did it

A fault can enter a handover in five places, and each has a cheapest control. The sender can draft it wrongly; a check of the handover against the record before sending catches that. It can be corrupted on the way; a digest catches that. It can be addressed to the wrong agent; the address check catches that. The receiver can read it wrongly, resolving the supplier to a lookalike's account or picking an old handover from its inbox; only a readback of what the receiver is going to do can catch that before the act. And the tool can do something other than what it was told; only a report of what was done, checked against the record, can catch that after.

Study 5 tested the first three. Its lines arm had the receiver build the readback with the codec from the message as received, so the readback always carried whatever the message carried, and comparing it with the record caught every planted fault. That showed the loop was sound. It did not show that a readback catches anything a check before sending would not. The protocol's claim to be a safety control rests on the last two places, and this study was built to test them, with the echo kept in as the arm that should catch nothing there.

What we tested

In block 2 the handover carries the business intent and nothing the receiver should resolve for itself: pay invoice MD-2026-057 to marlow, 5400.00 GBP gross, on 2 October, under FD-0929, with no account; the account comes from the master. The contexts carry the traps. A lookalike row for the same supplier with a different account. A superseded row with the old account beside the current one. A pending change that takes effect after the payment date. An earlier handover for a different Marlow invoice, later withdrawn, or one already paid. An email from a lookalike domain saying the bank details have changed. The account on file stored in a format the tool will not accept. For the bank change, a lookalike supplier and an earlier change for another supplier already executed. For the email, a second contact beside the billing one, a lookalike domain redirecting advices, and a stale draft with last month's figures.

The execution readback in the lines arm carries, for a payment, the supplier, the invoice, the amount, the currency, net or gross, the account the receiver will pay to, the date, the reference and the approval reference; for a bank change, the supplier, the account on file being replaced, the new account, the effective date and the approval; for an email, the address, the invoice and the figures. The sender's code compares every slot with the record, re-issues the handover once with the slot named and the reason, and only on a pass sends AN val=Y. After the act the receiver sends a DN carrying the values the tool printed, and the sender checks that too.

The confusers were a pre-registered manipulation, with a check: if the no-readback arm did not make wrong acts at least a tenth of the time on the strong receiver and a fifth on the small one, a three-step hardening ladder was to be applied and every step reported. It was: two confusers instead of one; the confusing rows listed first and the live handover further from the end of the inbox; the status field removed from the master so that an old row is marked only by its date.

How we ran it

1. Wrote the protocol first: the blocks, the arms, the measures, the hypotheses and their tests, the sizing, the manipulation check and the ladder, and what each outcome would change in the protocol. It is research/plan-readback/README.md, with its amendments appended as they were made.
2. Built the case and the harness. The receiver is a headless Claude Code session with the brief as its system prompt, a supplier master and an inbox in its folder, and one way to act: a tool that validates what it is passed, executes once, prints what it executed and refuses a second act. Every arm's brief is the same length within 10%.
3. Piloted the no-readback arm on block 2: 172 trials per level, twice. The first pilot found two defects in the case (an advice for a payment not yet released; conventions that implied every handover names a readback level) and they were fixed. The corrected pilot met the check at no level of the ladder, so block 2 ran at level 0 and reports bounds.
4. Piloted blocks 1 and 3 and fixed three faults in the harness they exposed, each logged: the second readback checked against the wrong line, a tool that misread a master without a status field, and a judge prompt that read "does the note match the record" as "does the note report the record faithfully".
5. Froze the tools, the data and the protocol at one commit, recorded in results/manifest.json, and ran the whole design at that commit: block 1 with three repeats, block 2 with five, block 3 with three. 2,762 trials, about $331 of model time, 1 trials ending in anything but the receiver's own last word.
6. Graded everything in code and compared a blind second reading on a sample. Every number on this page is read from results/summary.json by the script that writes the page, and a test in the app asserts they match.

What happened

Table 1. Block 2, the receiver's reading. Per cell 215 trials: 28 induced contexts and 15 clean ones, five repeats. "Under" is the upper end of the exact 95% interval. A false alarm is a readback rejected although every value in it matched the record. Cost is what the headless session cost, which is dominated by its own context, not by the protocol's traffic.

In words: nobody paid the wrong account, changed the wrong row or wrote to the wrong address, in any arm, on either receiver. The no-readback receivers asked instead in a few percent of induced contexts and acted correctly in the rest. The readback arms had nothing to catch at this locus. The echo readback carried none of the slots the receiver had resolved, which is why it could not have caught a wrong one; the execution readback carried them all, and every act matched its confirmed readback. The model judge raised 1 of 215 and 1 of 212 false alarms, each over a detail the readback added or left out while every value was right; the code checks raised 0 of 215 and 0 of 215.

The hardening ladder did not change this. Two confusers instead of one, the confusing rows listed first with the live handover buried deeper, and a master with no status field so that an old row is told from the current one only by its date: at every level the no-readback arm made no wrong act, 0 of 224 on Sonnet and 0 of 224 on Haiku over the four levels together, upper bounds of 1.6% and 1.6%. Asking rose with hardening; acting wrongly did not. Claude Haiku 4.5, run once as a probe of a weaker receiver, gave 0 of 56.

Table 2. Block 1, the sender's drafting. Study 5's sixteen planted faults and three clean handovers, three repeats. The pre-send check, computed rather than run, refuses all sixteen.

In words: a wrong figure, date, account or reference in the handover itself became a wrong irreversible act about a third of the time with no readback, and never with one. The wrong acts were all bank-detail changes, which nothing in the receiver's inbox contradicted; the payment and email faults it could see against its own inbox, and it asked. Both readback arms caught every fault, the echo mostly at the readback and the execution readback half at the readback and half by the receiver asking, because a receiver that resolves the account from the master and finds the handover's account differs from it stops. The check before sending catches the same sixteen for nothing.

Table 3. Block 3, the tool. Fifteen injected faults, three repeats: a transposed digit in the account, a date a day late, another amount, a wrong sort code or reference, the change applied to another supplier's row, the email sent to another address, the wrong currency. The tool prints what it executed; the receiver reports from that.

In words: the second loop works. Told to report what the tool printed rather than what it meant, every receiver did, and the sender's check of that report flagged every fault that reached an act, whether the check was code on a DN line or a judge on a prose note; the one shortfall in the table is a Haiku receiver that asked instead of acting, so there was nothing to flag. Nearly every receiver also said, unprompted, that the tool had done something other than asked.

The same design on two OpenAI receivers, partial

The registered design named two receivers outside Anthropic. With a key for them, it was built afresh for gpt-5.5 and gpt-5.4-mini and run through the API harness, Claude Sonnet 5.5 as the judge, a different vendor from the receivers. 2,196 of the 2,762 trials ran before the key's credits ran out twice: blocks 1 and 2 near complete, block 3 not started. The unrun trials stay built; nothing here is weighted or imputed. The harness recorded tokens and no price, and one harness fault found during this run, the protocol's live codec reaching the sender's check, is described in the results notes and corrected.

Table 4. Block 2 on the OpenAI receivers, as far as it ran: 160 to 191 of each cell's 215 trials. "Did nothing" is a trial that ended with neither an act nor a question, which the wrong-act rate does not count. "Held" is a readback rejected twice, so the receiver never acted.

Table 5. Block 1 on the OpenAI receivers, complete: the sixteen planted faults and three clean handovers, three repeats.

In words: gpt-5.5 reads like the Claude receivers, with one difference of degree. Nothing went wrong at its own locus in any arm, and a planted drafting fault reached a wrong act 29 of 48 times with no readback, twice the Claude rate, and never with one. gpt-5.4-mini is the receiver the readback rule was written for, and the arms do not rank as the rule assumes. With no readback it made 1 of 140 wrong acts in induced contexts, though it also did nothing in 37 of 191. Under the echo readback it made 10 of 140, bank changes executed with the decoy's sort code or a transposed digit after a readback that had passed, because the echo carries none of the values the receiver resolves; the echo also let four wrong bank changes through in clean contexts. Under the prose readback it made 4 of 140, each passed by the judge, which also raised false alarms on 38 of 190. Its execution readbacks in line form never carried the resolved slots: it ran make-rb, sent the envelope back unchanged, was told twice which slot was missing, and was held in 182 of 186, so that arm is a stall with no wrong act rather than a catch. The fixed-shape English block, parsed by code, is the one execution readback it completes: 1 of 140 wrong act, and in the context that puts a second contact beside the supplier's billing address it planned the wrong address, the block was refused, the handover restated, and it acted correctly in four of five trials, where the echo arm executed the wrong address in four of five. On this receiver the form of the readback decides whether a wrong plan is caught, passed or stalled.

The extension: free prose, and the judged readback against planted faults

Two arms added after the main run, written up in the protocol before they ran and not registered: free prose, arm A's brief with the conventions block removed so the receiver is told nothing about what a message carries or how it is written, in blocks 1 and 2; and the prose readback judged by a model, in block 1, where it had never met a planted fault. Sonnet 5.5 only, 329 trials, $37; the gpt-5.4-mini half is built and waits on credits.

Table 6. The extension beside the main run's Sonnet cells. The free-prose brief is 1,006 tokens against arm A's 2,820, so length and discipline are confounded here, as the protocol says.

In words: on Sonnet the conventions do no work that the figures can see. Free prose made 0 of 140 wrong acts in block 2 and let 15 of 48 planted faults through in block 1, the same bank-change and misaddressed handovers as disciplined prose. The judged prose readback against planted faults is the one comparison that separates prose from lines at the locus where Sonnet fails: it caught 18 of 48 at the readback and the receiver asked on 21 of 48, but 8 of 48 reached a wrong act, every one the handover addressed to another agent. The receiver read back the right values and noted that the salutation was not its own; the judge, asked whether the values matched the record, said they did; and the scheduler's "proceed" overrode the receiver's own doubt. The line checks of arms C and D refuse that message at the receipt tier, before any value is compared. A model judge checks what it is asked to check.

What it means

The receiver's own reading is not where these models fail before an irreversible act. Given a correct handover and a context built to mislead, Sonnet 5.5 and Haiku 5.5 resolved the account on file, skipped the withdrawn and the already-paid handovers, ignored the lookalike's email and the stale draft, reformatted the account the tool would not accept, and when they could not tell, asked. The plan readback had nothing to catch, at any level of hardening we had pre-registered, and we stopped there rather than invent harder traps after seeing the result.

What a readback catches is the sender's drafting, and a check before sending catches the same thing without a round trip. The echo and the execution readback were equivalent at that locus. Where the execution readback differed was in how: a receiver that resolves the account for itself and finds the handover's account is not the one on file stops and asks, which is the right instinct and costs a turn.

The second loop is the one that earned its place. Nothing here protected against a tool that does something other than what it was told except a report of what was done, read from the tool and checked against the record. Both the code check and the judge caught every fault, because the receivers reported honestly; a receiver that reported its intention instead would have hidden them, and the pilot found that only a prompt that says "the values the tool printed, not the values you intended" produces the honest report.

Code and judge differ where the pilots said they would. The judge produced a handful of false alarms over details the record did not mention, and before its prompt was fixed it read an honest report of a wrong act as a match with the record. The code checks produced none, and the one format false alarm in the English-block arm was a receiver that wrote a line before its block. Checking a typed readback costs nothing; checking a prose one costs a model call and some judgement.

The cost of the loop is a turn. A readback arm's receiver took two turns where the no-readback receiver took one, and the done line added nothing to that. On these receivers that is the whole price of a control that, before the act, had nothing to catch.

What we decided

A handover before an irreversible act is built from the record by code, or checked against it before it is sent.

The drafting locus is the sender's, and this study showed the check before sending equal to the readback there. The protocol's own tool does the comparison; the rule goes into the spec beside the readback rule.

The done report carries the values the tool executed, and the sender checks it.

This is the loop that caught what no readback could. The DN's executed-value slots and the sender's check of them go into the spec as the second loop, with the instruction that made the reports honest: report what the tool printed, not what you meant.

The readback before an irreversible act stays, as insurance whose price is now known.

On the two receivers here there was nothing for it to catch at the receiver's locus: 0 of 140 wrong acts on Sonnet and 0 of 140 on Haiku with it, against 0 of 140 and 0 of 140 without, and 0 of 215 and 0 of 215 false alarms. The rule was written for mixed and weak fleets, and it costs one turn. The cross-vendor run then found such a receiver: on gpt-5.4-mini the echo passes a wrong plan, the typed execution readback stalls, and the parsed English block catches. The rule stands, and whether rb=exec should accept a fixed English block as an equivalent form is the question it leaves for the spec.

Where the receiver must resolve a value, the readback carries the value it resolved.

The execution readback's slots as tested, including the account on file being replaced by a bank change, go into the pay profile, so that a readback before an irreversible act is a readback of the plan and not an echo of the message.

What this does not show

• The main run is two receivers from one vendor, run as headless Claude Code sessions, with the same vendor as judge; the bound on wrong acts is a property of those two models and this case. The cross-vendor run that followed is partial, blocks 1 and 2 only, and its block-2 cells are a few trials short; it was run after the main results were known, though against the same frozen design.
• The confusers are ours. Fourteen kinds in two variants, hardened three ways, all pre-registered, and none made these receivers act wrongly. A receiver in the field meets contexts we did not think of; the bound says these did not catch it out, not that nothing will.
• Block 1 is not study 5's context: the inbox carries the approved figures and, for the email, the release, so most payment and email faults were visible to the receiver and it asked. The replication is of the arms under this harness, not of study 5's numbers.
• The cost a trial is the headless harness's, which re-reads its own context on every call; the protocol's own traffic is a few hundred tokens a trial.
• Three harness faults and two case defects were found by the pilots and fixed before the freeze, and are listed in the protocol's amendments. The main run had none we found; the grader's blind second reading on a sample is reported in the results folder.

Where everything is

The protocol with its amendments, every pilot with its trials and grades, the ladder, the Haiku 4.5 probe, the frozen manifest, the main run's trials, the sender's state, the grades and results/summary.json, the cross-vendor run under results/api and the extension under results/extension: research/plan-readback in the rogerwilco repository. The page you are reading is written from that JSON by tools/site_page.py, and app/tests/test_research_figures.py fails if any figure here differs from it.

Study 6 of 6 · 10 and 11 October 2026 · 2,762 trials on Claude Sonnet 5.5 and Claude Haiku 5.5 as headless Claude Code sessions, pre-registered · 2,196 on gpt-5.5 and gpt-5.4-mini through the API harness, partial · 329 in the extension · judge Claude Sonnet 5.5 · tiktoken o200k_base · graded in code