Study 7 of 711 and 12 October 2026Croxio Ltd794 trials on Claude Haiku 5.5 and 1,014 on gpt-5.4-mini, pre-registered, each after a pilot

Before an act that cannot be undone, does a readback the tool composes catch what the model's cannot?

The question

Every readback tested so far was the model's account of its intention. Between that account and the act there is a gap the model fills by typing a tool call, and study 6's weak receiver failed in that gap: the readback was right and the call was not. This study moves the readback to the tool. prepare states exactly what will be executed and freezes it; commit executes that statement and nothing else; the sender checks the statement against the record before anything moves.

The material

Study 6's fictional month-end, unchanged: the payer handed three acts it cannot take back, release a payment, change a supplier's bank details, send a remittance advice, with the account or the address to be resolved from a supplier master built to mislead.

Three fault blocks

Block 1, the receiver and the call: study 6's 28 induced contexts and 15 clean ones. Block 2, the tool: study 6's fifteen injected tool faults, each applied at prepare, so the statement carries the fault, and at commit, so the act differs from the statement. Block 0, the sender's draft: study 5's sixteen planted faults and three clean handovers, as a regression check.

Compared

No readback. The execution readback written by the model and checked by code, study 6's arm. The tool's prepared statement relayed by the model as its readback, with discipline keeping it from committing early. The tool's prepared statement read by the sender from the tool's log, with a commit the tool refuses until the sender has confirmed.

Receiver

Claude Haiku 5.5 as headless sessions. No judge: every arm is checked by code. The OpenAI receiver the design named is parked; the harness keeps its arms for a later run.

Graded by

Code: the act as the tool executed it against the record; the statement, the readback and the done report against the record and against the tool's log. Every proportion carries its exact 95% interval. The protocol, its amendments before the main run, the pilot and the frozen manifest are in the repository.

Why we did it

Study 6 placed the faults. The sender's drafting is caught by a check against the record before sending. The tool's execution is caught only after the act, by the done report. The receiver's own reading was null on every strong receiver, and on gpt-5.4-mini it turned on the form of the readback: the echo passed a wrong plan, the typed execution readback stalled because the receiver never composed it, and the parsed English block caught a wrong address. In every arm the readback was the model's statement of what it meant to do, and nothing bound the tool call that followed to it.

Aviation closes that gap by reading the machine, not the pilot. The cleared altitude is set in the window and the other pilot checks the window; on Mode S the controller sees the selected altitude before the aircraft moves. The equivalent here is a two-phase tool. Two things follow from it that no model-sourced readback gives: the checked values are the ones that will run, because the call is frozen at prepare; and a receiver that cannot compose a readback is not asked to, because the tool composes it.

What we tested

The act tool of study 6 gained three verbs and kept its one-step form for the baseline arms. prepare validates and resolves the arguments as the one-step tool did before executing, writes the prepared transaction to the tool's log under a token, and prints the readback line it composed from the handover it was given: the envelope and the digest of the handover as received, the execution slots with the prepared values, and the token. For a bank change the statement names the row it will replace, resolved by the tool from the master; for every act it names the supplier as the master row the prepared legal name resolves to, so a statement that would change another supplier's row says so. commit executes the prepared transaction exactly and refuses any token it does not hold; in the enforced arm it also refuses a token the sender has not confirmed. abandon discards a prepared transaction so that the receiver can prepare again.

In the relayed arm the receiver's readback is the line the tool printed, copied to its readback file unchanged; the sender checks that line against the record, requires it to carry a token, and confirms that token, never reading the tool's log. In the enforced arm the receiver sends no readback: it replies that it has prepared, the sender reads the latest prepared transaction from the tool's log, checks it, and on a pass writes the confirmation the tool requires. In both, a failed check re-issues the handover once with the slot named and the reason; the receiver abandons, resolves again and prepares again; a second failure holds the handover without an act. After the act the receiver writes the done line from the tool's printout and the sender checks it, as in study 6.

The two arms separate two things the design would otherwise confound. The relayed arm is set-and-cross-check over the frequency: the readback is the tool's, and discipline keeps the model from committing before the sender answers. The enforced arm is the downlink: the model relays nothing and cannot commit until the sender has confirmed. If the relayed arm never commits early and never relays a line that differs from the tool's, the tool's statement suffices and the sender need not reach the tool's state; if either happens, the data make the case for enforcement.

How we ran it

1. Wrote the protocol before building: the mechanism, the arms, the blocks, the measures, the hypotheses and their tests, the sizing, the pilot gate, what the study will not show. It is research/tool-prepare/README.md, with the amendments made before the main run appended.
2. Built the tool and the harness from study 6's, with the codec frozen at study 6's commit so that the no-readback and execution-readback arms are study 6's arms, and smoke-tested the whole chain without a model: prepare, relay, check, confirm, commit, abandon, the done check and the grading.
3. Piloted 520 trials of the two new arms on Haiku 5.5: every clean context, six induced ones, and the first two tool faults of each act at both points. 424 of 520 reached a commit or a refusal without a harness fault, against a gate of 90%; 108 wrong acts, all faults at commit, which nothing before the act can see. The pilot cost $4.07 and is reported apart from the main run.
4. Wrote the amendments the pilot called for, among them a path the protocol had not named: a receiver that reads the tool's statement, sees that it differs from what it passed, and refuses. Froze the tools, the data and the protocol at one commit, recorded in results/manifest.json, and ran the design: block 1 with three repeats, block 2 with two, block 0 once. 1,808 trials, $11 of model time, 205 ending in anything but the receiver's own last word.
5. Graded everything in code. Every number on this page is read from results/summary.json by the script that writes the page, and a test in the app asserts they match.

What happened

Table 1. Block 1, the receiver and the call. Per cell 129 trials: 28 induced contexts and 15 clean ones, three repeats. "Under" is the upper end of the exact 95% interval. A false alarm is a statement or readback rejected although every value in it matched the record. In the enforced arm "committed before the confirmation" counts a commit the tool refused for want of one. Cost is what the headless session cost.

In words: nobody paid the wrong account, changed the wrong row or wrote to the wrong address in any arm, the bound study 6 found on this receiver. The two new arms behaved like the execution readback on everything the readback measures and differed on cost: the receiver wrote 2,914 and 2,903 output tokens a trial where composing its own readback took 3,691, and in the enforced arm the traffic between the agents fell to 66 tokens, because nothing is relayed. Three trials asked instead of preparing, all on the context where an email from a lookalike domain asks for more than the approval covers; the no-readback arm asked on 5 of 84. Every act matched its confirmed statement, 128 of 128 and 127 of 127.

Table 2. Block 2, the tool. Fifteen injected faults, each at two points, two repeats: 30 trials per point per arm. In the one-step arms the point makes no difference, since the fault is applied at execution either way, and both halves are shown. The done check flags are counted over the trials where an act happened. "Reported" is a regex over what the receiver wrote after the act.

In words: the point of injection is the whole result. A fault the tool introduces when it resolves the call, a transposed digit, a date a day late, another supplier's row, is invisible to a readback the model composes, because the model's values are right; the one-step arms executed 29 of 30 and 30 of 30 of them. The two-phase tool put the same fault into the statement, and the statement stopped it every time. The catch was the receiver's: it had passed one account and the tool said it would pay another, and it refused to relay a statement it did not believe, prepared again in most trials, saw the same fault, abandoned and asked. The sender's check was never reached. A fault injected at commit, after the statement, is a tool that does other than it stated, and nothing before the act can see it: every arm executed it, the done check flagged all 30 of 30 and 30 of 30 in the two-phase arms as it did 30 of 30 in study 6's, every done report carried the tool's values (30 of 30 and 30 of 30), and the receiver itself said the tool had done something other than asked in 28 of 30 and 28 of 30.

Table 3. Block 0, the sender's drafting, as a regression check: study 5's sixteen planted faults and three clean handovers, one repeat. The tool refuses to prepare a handover addressed to another agent, so the three misaddressed handovers are caught at the tool as a refusal the receiver reports.

In words: no regression. A planted fault became a wrong act 0 of 16 and 0 of 16 times. Most were caught before any statement, because the inbox carries the approved figures and the receiver asked when the handover contradicted them; the bank-change faults, which the inbox does not contradict, reached the tool, and the sender's check of the statement refused them (5 of 16 and 5 of 16, the receiver then preparing again from the re-issued handover or asking). The three handovers addressed to another agent were refused by the tool and reported.

The weak receiver: gpt-5.4-mini

The design named a second receiver, the one whose failures the control was built for: gpt-5.4-mini, which in study 6 passed a wrong plan under the echo and never completed a typed execution readback. Its run came after the Haiku 5.5 results were written up, with four amendments registered first: the tool reads the handover from the inbox by id, so the receiver never retypes it; the scheduler's replies are delivered to the inbox; the API runner; and two harness faults found by a first pilot attempt and fixed before the pilot was re-run, both kept in the repository. The registered pilot then met both gates: under the echo the receiver made a wrong act in 3 of 28 induced contexts, and under the enforced two-phase arm 42 of 43 trials reached a commit or a refusal. The design ran as registered: 1,014 trials, six arms in block 1 and four in block 2, 205 ending in anything but the receiver's own last word.

Table 4. Block 1 on gpt-5.4-mini: 129 trials a cell, 28 induced contexts and 15 clean ones, three repeats. "Caught" is a readback or statement refused with no wrong act; on this receiver a refusal for a readback that is not an RB at all counts as a false alarm, not a catch. The runner records tokens and not price; at the published rates the run cost about $14.

In words: this is the receiver the control was built for, and the control holds where its own readbacks fail. Under the echo it executed 10 of 84 induced handovers wrongly and 4 of 45 clean ones, bank changes made to the superseded sort code and remittance advices sent to the second contact, each after a readback that had passed, because the echo carries none of the values the receiver resolves. Under its own typed execution readback it made no wrong act but was held in 32 of 84 induced trials and refused 75 of 129 times for a slot it had left out: a stall, as in study 6, with 9 of 84 genuine catches of the old account along the way. The English block it completes: 0 of 84 wrong acts, no holds, and it caught the wrong address twice. Under the enforced statement it made 0 of 84 wrong acts, was held 5 of 84 times, raised 0 of 115 false alarms, and the sender's check of the tool's statement caught 7 of 84 wrong plans: the old sort code passed into prepare six times and once the lookalike domain's address, the very faults the echo had executed. It never committed before the confirmation, 0 of 129, and in 7 of 84 trials it asked instead. The relayed arm shows why the enforced one is needed on this receiver: in 51 of 129 trials the line the receiver relayed was not the line the tool printed, nearly always its own composition without the token, so the sender refused it for form, 8 of 84 trials were held, and 14 never ran prepare at all; discipline about committing held, 0 of 129, but discipline about relaying did not.

Table 5. Block 2 on gpt-5.4-mini: fifteen injected faults at each of two points, two repeats, 30 trials a point an arm.

In words: the same shape as on Haiku 5.5, with the catch moving from the receiver to the sender. A fault the tool introduced when it resolved the call reached a wrong act in 23 of 30 trials with no readback and 9 of 30 with the model's own readback, which cannot see it, and in 0 of 30 and 0 of 30 with the tool's statement. On Haiku 5.5 the receiver refused every such statement itself; gpt-5.4-mini relayed or recorded it and the sender's check refused it, 14 times the receiver saw it first in the enforced arm, and because the fault reappears in every prepare the handover was held or the receiver asked. A fault at commit reached a wrong act in every arm, as it must, and the done check flagged 24 of 25 and 25 of 27 of them in the two-phase arms against 12 of 13 in the model's.

The registered tests on this receiver. H1a: the enforced statement made fewer wrong acts than the echo, 0 of 84 against 10 of 84, one-sided Fisher p under 0.01. H1b: it was held less often than the model's execution readback, 5 of 84 against 32 of 84, p under 0.01. H2: against the English block, 0 of 84 and 0 of 84 wrong acts, no evidence of a difference. H5: 0 of 38 false alarms on clean contexts, whose bound at this size is above the registered 5%, and 0 of 115 over every context, whose bound is under it. H6: no commit before the confirmation in 0 of 129 relayed trials, and a relayed line that differed from the tool's in 51 of 129: the case for the enforced arm over the relayed one is made by the data on this receiver. The run cost about $14 at the published rates; 205 of its 1,014 trials ended in a hold or without the receiver's reply word, none in a harness fault.

A grading rule was sharpened after these data were seen, and both readings are kept. The rule as registered counted any refusal that was not purely formal as a catch, which on Haiku 5.5 was right, since every refusal there named a value the record contradicted. On gpt-5.4-mini most refusals were for a slot the readback had left out, a form fault with the plan right. A catch now needs a value the record contradicts, a refusal for form is counted apart, and in the relayed arm a refusal the tool's own line would not have earned is a relay fault. Under the registered rule the model's execution readback would show 57 of 84 catches on this receiver and the relayed statement 38 of 84; under the sharpened rule they show 9 of 84 and 6 of 84. No Haiku 5.5 figure changes.

What it means

The gap between the readback and the act is real, and the tool closes it. Study 6's execution readback checked what the model meant to do; a tool fault at prepare is exactly the case where what the model meant and what the tool would do differ, and the model's readback passed every one. The tool's statement is the only account of the act that the act is bound to, and on this receiver it stopped every such fault before the sender's check was needed.

The window is read first by the one who set it. The protocol put the check with the sender; the receiver read the statement, compared it with its own call and refused. That is set-and-cross-check in the cockpit, before the readback on the frequency, and it means the statement does its work even where the sender's check is weak or late. The sender's check still stands behind it: in block 0 it was the sender's check that caught the bank-change faults the receiver had carried into its call.

Discipline held, so enforcement was not needed here. The model never committed before the confirmation and never altered the line it relayed, in 0 of 128 relays. On this receiver the relayed arm suffices and the sender does not need to reach the tool's state; the enforced arm costs nothing more and removes the assumption, and on a receiver that does commit early it is the only one of the two that holds.

What remains after the statement is the tool that does something other than it stated. Nothing before the act sees that, by construction; the done check caught every one, as it did in study 6. The two loops are now the statement before and the report after, and the model's own readback between them has become a cheaper thing: the model relays or does nothing, and the values it would have typed come from the tool.

What this does not test is the receiver the control was designed for. Study 6's weak receiver failed between its readback and its call, and the tool-composed statement was built so that such a receiver need not compose anything. Haiku 5.5 composes a correct execution readback already, so block 1 here shows the cost and the false-alarm rate of the new arms and bounds their wrong acts, not a rescue.

What we decided

Where the tool can state its transaction before executing it, the execution readback is the tool's statement.

The spec's guidance on the act before an irreversible step names prepare and commit as the preferred form of the act tool: the statement carries the execution slots as the tool will run them and a token, the sender checks it as it checks any execution readback, and the commit executes the statement and nothing else. The model relays the statement, or the sender reads it from the tool.

The receiver compares the statement with its own call before relaying it.

That is where every prepare-point fault was caught here. The brief says so in words: if the tool's statement differs from what you passed, abandon and ask; never relay a statement you do not believe.

The done check stays as the second loop.

A fault after the statement is invisible before the act, and the done check flagged every one. Nothing in this study moves it.

Where the receiver cannot be trusted to relay, the sender reads the tool.

On gpt-5.4-mini the relayed statement was altered in 51 of 129 trials and the enforced one never committed early; the enforced form is the one the spec names for mixed fleets, with the relayed form acceptable where the receiver has shown it relays faithfully.

The tool takes the handover by id.

A receiver that retypes the handover breaks the digest, as the Haiku 4.5 check showed; prepare --ho-id reads the message as delivered, and the sender's replies are delivered to the inbox so a re-issued handover can be read the same way.

What this does not show

• Two receivers, one from each vendor; the Claude receiver as headless Claude Code sessions, the OpenAI one through the API harness. The bounds on wrong acts are properties of these two models and this case. The weak receiver's two-phase arms used the tool's by-id form of the handover and the Haiku 5.5 run's used the retyped form, an amendment registered between the runs, so the two receivers' P and Q cells are not the same procedure to the letter.
• The enforced arm assumes the sender can read the tool's prepared state. Here that is a shared file; in a deployment it is a signed statement or a ledger both sides trust. The study measures whether the control works with a model in the loop, not how to deliver the statement.
• Freezing the call at prepare and confirming it are two things, and the design confounds them: an immutable call without a confirmation would also remove the call faults. The relayed arm against the enforced arm separates the confirmation's enforcement, not the freeze.
• A fault at commit is a fault nothing before the act can see, by construction. Its row in table 2 measures the done check, not the statement.
• The confusers are study 6's, and study 6 found that Haiku 5.5 does not act wrongly against them. Block 1 therefore measures the cost and the false alarms of the two new arms on a receiver that had nothing to catch, and bounds the wrong-act rate; it does not show the statement catching a wrong resolution, because there was none to catch.
• One filter fault in the runner, found at the start of the main run, left 168 trials to a second pass at a later commit on the same built trials; it is recorded in the amendments.

Where everything is

The protocol with its amendments, the pilot with its trials and grades, the frozen manifest, the main run's trials, the sender's state and the tool's logs, the grades and results/summary.json: research/tool-prepare in the rogerwilco repository. The page you are reading is written from that JSON by tools/site_page.py, and app/tests/test_research_figures.py fails if any figure here differs from it.

Study 7 of 7 · 11 and 12 October 2026 · 794 trials on Claude Haiku 5.5 as headless Claude Code sessions and 1,014 on gpt-5.4-mini through the API harness, pre-registered · pilots of 520 and 86 trials · no judge · tiktoken o200k_base · graded in code