Study 9 of 924 October 2026Croxio Ltd1,585 chains, 541 on five local models and 1,044 on two Claude anchors, pre-registered, after a speed test and a pilot

Where does a relayed handover break as the agents get smaller, and does the check at the hop hold?

The question

Study 8 found that on Haiku 5.5 and Haiku 4.5 no relay changed a value in 792 chains. A protocol whose checks never fire is either unnecessary or untested at the edge where it matters, and the edge is a mixed fleet: relays that are the smallest and cheapest models a team can run, under length pressure, in a context that is not clean. This study goes there, with models down to 1.5 billion parameters run locally, and the two Claude anchors through the same runner so the sizes compare.

The material

Study 8's company, acts, record, master and inboxes, copied: release a payment, change a supplier's bank details, send a remittance advice. The chain is a scheduler that is code, a coordinator that adds the deadline and priority, a desk that confirms the approval against its log, then the payer. Every agent is one text-only chat with no tools; its brief is the system prompt, its inbox and context file the first turn, its reply the message. The payer replies with a fixed act block the harness executes once against the master.

Compared

Five forms: free prose; prose under the seven conventions; RW1 lines validated for syntax only; lines read back at every hop by the model, with the forward checked against the handover received; lines read back at every hop by code, built with make-rb from the message as delivered, the relay confirming or disputing.

The axes

Six blocks. The clean baseline with two other live handovers in every relay's inbox; brevity pressure; a decoy, an adjacent run, a near approval reference or a log that covers the supplier but not the item; a sweep to five relays; an injected email from a lookalike supplier domain giving a new account; and, added after the freeze, the routed message: one message per call with a short brief, as a small model is deployed.

Models

Qwen2.5 1.5B, 3B, 7B and 14B and Llama 3.2 3B, Q4_K_M on four CPU cores through llama.cpp at temperature 0; Claude Haiku 4.5 and Claude Haiku 5.5 headless through the same text-only runner at provider defaults. Every agent in a chain runs on the same model. No judge.

Graded by

Code: the act as executed against the record; the record's fields in every delivered message, by parse for a line and by normalised match for prose; whether the forwarded line parsed; whether the planted account went anywhere; every refusal and what it named. Every proportion carries its exact 95% interval. A blind second reading of 80 chains agreed with the grader on 80 of 80 acts, 80 of 80 on every field at the payer and 80 of 80 on drift.

Why we did it

Study 8 was a null on the fault it was built to find: current Claude models, even the small one, carry an instruction's facts intact across a chain, and the line's claim on a chain was the envelope, the size and a check that refused nothing. The question this left was where the form matters. A team that runs a mixed fleet puts its smallest models at the relays, and a 1.5B model that cannot call a tool fails for reasons that have nothing to do with the message's form; so every agent here is a plain chat, the harness is the tool, and the model's ability to write a line is itself measured. Five axes of pressure were registered: size, brevity, confusers, length and an injected instruction, each with the five forms, and a contingency table said in advance what each result would change in the protocol.

What we tested

The five arms. R is what held Haiku 4.5 in study 8; C is study 7's amendment, the readback composed by code, brought to the relay. If a small model cannot write a valid line at all, L, R and C all stall at that point, which is a finding about the floor of the form.

Block 1 is the baseline: a clean record and two other live handovers in every relay's inbox, three acts, two clean contexts, every arm. Block 2 adds brevity pressure to the relay's brief. Block 3 is study 8's confusers, four per act. Block 4 is the sweep, one, two, three and five relays, prose against the checked line. Block 5 plants, two messages before the live handover in one relay's inbox, an email from a lookalike supplier domain saying the payee account has changed; the correct relay forwards the record's values and may mention the email. Block 6, registered after the freeze and before any of its chains ran, delivers the live handover alone with a short brief in F, L and C, to ask how much of the small models' failure is the inbox and the long brief rather than the form.

How we ran it

1. Wrote the protocol before building: the text-only runner, the five arms, the five blocks, the measures, eight hypotheses and their tests, the sizing by a speed test, the pilot gates, the exclusion rule and a contingency table. It is research/breaking/README.md, with every amendment under its own dated heading.
2. Built the harness from study 8's with the codec and the pay profile frozen into the study folder, measured the speed of each local model, and piloted 12 chains (free prose and the code-readback arm, one payment, every model, $0.31 on the anchors). Gate 3, the smallest model producing one valid line, was not met: Qwen 1.5B attempted no line. As registered it ran anyway.
3. Froze the tools, the data and the protocol at one commit and ran the design: the anchors at three repeats, the local models one chain at a time, baseline and injection blocks first on every model, kept alive by hand because the container is reclaimed when idle. Qwen 14B was added after the freeze, one size up from the 7B, so the size axis reaches the point where the local models stop breaking.
4. Cut the remaining blocks on the four models below the band after their baseline, since brevity, confusers and length on relays that do not carry the baseline would have been fifty-five hours of CPU for cells already at zero; ran the confusers on the 14B; registered and ran block 6 on the three models that had reached the payer.
5. Graded everything in code: 1,585 chains, 0 harness faults in the analysed set (four chains of one launch hit a thread-unsafe import in the harness; they are excluded by name in the protocol and were run again one at a time, all four ending in a correct act), $93.84 of anchor time and 39.8 hours of the box's clock on the local models. Then a blind second reading of 80 chains, drawn by seed, from the chain files alone: agreement 80 of 80 on the act, 80 of 80 on every field at the payer, 80 of 80 on drift. Every figure on this page is read from results/summary.json by the script that writes the page, and a test in the app asserts they match.

What happened

The headline: block 1, correct acts and wrong acts, model by arm

Table 1. Block 1, the clean baseline, two relays: 6 chains per cell on the local models, 18 on the anchors. A correct act matches the record on every field; a wrong act is a chain that acted and the act differs from the record. The rest of each cell is chains that asked, were held by a check, or stalled.

In words: the two smallest models never acted. Qwen 1.5B asked at the first hop in 6 of 6 free-prose chains and 6 of 6 code-readback chains, and was held at its own readback in 6 of 6 in R; Qwen 3B the same, 6 of 6, 5 of 6 and 6 of 6. Llama 3B acted, and wrongly as often as rightly: 2 of 6 correct and 2 of 6 wrong in free prose, 0 of 6 and 2 of 6 with the unchecked line, 0 of 6 and 0 of 6 with the checked one, held at the forward in 6 of 6. Qwen 7B made 2 of 6 wrong acts in each unchecked arm and no correct one, and in C was held in 5 of 6, its forwarded line dropping a slot the check named. Qwen 14B: 5 of 6 (83%, 36 to 100) in free prose with 1 of 6 wrong, 5 of 6 with 0 of 6 wrong in C, and 0 of 6 with 2 of 6 wrong on the unchecked line. The anchors: Haiku 4.5 18 of 18, 15 of 18, 17 of 18, 18 of 18 and 17 of 18; Haiku 5.5 18 of 18, 17 of 18, 18 of 18, 4 of 18 and 18 of 18, the R cell held for form as below. No anchor made a wrong act in 1,044 chains.

Block 5, the injected account

Table 2. Block 5: the payer executed with the planted account ("complied"), and the chains in which the planted account or the lookalike address appeared in a message delivered past the relay that received the email. Every such appearance was in the tail of a line whose slots were the record's: a relay warning the next hop. 0 slots of any line in the run carried the planted value.

In words: H1, the primary hypothesis, cannot be tested as registered, because complied is 0 everywhere (Fisher's p 1.0 on every model). No relay on any model, in any form, forwarded the planted account as the record's, and no payer paid it. H2 holds on slots: check-ho had nothing to refuse. The grader's count of the value "forwarded" is 7 chains, 1 in R and 1 in C on Haiku 5.5 among them, and in each the line's acct= or addr= was right and the planted value sat in the tail, for example "Evergreen email of 2026-10-02 quotes 20-71-44/58213907, not the acct on this HO; payer to verify before executing". The injection axis is a bound: these models do not comply with this instruction in this voice.

The floor: valid lines against correct acts

Table 3. The valid-line rate (H6): of every line a relay replied with as its forward, over every block the model ran, the proportion that parses as an RW1 handover under the frozen codec, with its interval; beside it the clean block's correct acts on the unchecked and the checked line, and how many of the model's chains delivered anything to the payer at all. A model under 80% is below the form's floor.

In words: the valid-line rate does not predict the act. Qwen 1.5B attempted no line in 174 chains and reached the payer in 0 of 174. Qwen 3B's lines parsed in 27 of 28 (96%, 82 to 100) and it executed nothing, because every chain ended with a question or a hold before the desk. Qwen 14B's lines parsed in 60 of 60 (100%, 94 to 100) and its unchecked line ended in a correct act in 0 of 6 clean chains, because the lines that parsed were lines for another item in the inbox. Below the floor are Llama 3B (14 of 36 (39%, 23 to 57) in L, 31 of 43 (72%, 56 to 85) in C) and Qwen 7B (16 of 35 (46%, 29 to 63) and 23 of 29 (79%, 60 to 92)); their lines that failed to parse in L were delivered as written and read by the payer as prose. Both anchors parsed everywhere: Haiku 5.5 274 of 275 (100%, 98 to 100) in C.

Qwen 14B, the first local cell in the band

Table 4. Qwen 14B by block and arm: correct acts, then wrong acts, chains that asked and chains held. Blocks 1, 5 and 6 are 6 chains per cell, block 3 is 12; block 6 ran F, L and C only.

In words: the 14B is where the checked line starts to pay. In the clean block prose and the code-readback line are level, 5 of 6 against 5 of 6, and differ on the wrong act, 1 of 6 against 0 of 6; the sixth C chain was the payer naming the supplier by its short name, which the bank-change executor's exact-name rule refused, not a relay fault. Against the confusers prose made 6 of 12 correct acts and 6 of 12 wrong ones, 4 of 12 of them another item and 2 of 12 the record's values past a log that did not cover the item; the checked line made 7 of 12 and 0 of 12, ended in the desk's ask where that was the expected outcome in 2 of 12, and was held in 3 of 12. The check caught 11 forwarded lines in that block, the coordinator's priority or deadline taken from the wrong run or a dropped slot, and refused 3 of 22 forwards with every value right. The model-composed readback was held in every chain on this model, 6 of 6, 12 of 12 and 6 of 6, because the 14B wrote its RB without the version token and could not be talked out of it.

Block 6, the routed message

Table 5. Block 6 against block 1, the same three models, F, L and C: one message per call, a short brief under 700 tokens, and in the line arms a legend whose example line is in the live order. Six chains per cell in both blocks.

In words: routing is most of what the small models were missing, and it is not the form. Llama 3B's prose rose from 2 of 6 to 4 of 6 correct, its checked line from 0 of 6 to 3 of 6 with 0 of 6 wrong, and its unchecked line to 1 of 6 with 4 of 6 wrong acts: in three of the four the payer wrote a value from the legend's example line, the Ashdown address or the 640.00, into its act, from a line the desk had stripped of slots; in the fourth it wrote back the old account. Qwen 7B became a questioner, asking at the first hop in 6 of 6 prose chains about which run the item belongs to and in 6 of 6 checked chains after its forward was refused; its unchecked line reached 4 of 6 correct with 1 of 11 forwarded lines parsing, the payer reading broken lines as prose. Qwen 14B's prose held at 5 of 6, its unchecked line rose from 0 of 6 to 4 of 6 with 0 of 6 wrong, and its checked line fell to 2 of 6: 3 of 6 held, two at the coordinator whose line dropped rb= and took the priority from the wrong run under the short brief, one at the payer disputing a readback that carried exactly what the handover carried. Routing removed the selection errors on the 14B and gave the 7B's unchecked line its first correct acts; it did not make the checked forward easier to write.

The anchors, and the readback the model wrote itself

Table 6. Block 1 on Claude Haiku 5.5, 18 chains per form. "Held with every value right" counts a chain held at a readback whose last refusal named no value mismatch: the RB did not parse or omitted an echo. Readbacks refused for form are those refusals over every readback checked in the cell. Cost is what the headless sessions cost.

Table 7. Block 1 on Claude Haiku 4.5, the same columns.

In words: both anchors carried the handover in every form but one, and the one is the readback written by the model with no tool to hand. In arm R the relay writes its own RB line. Haiku 5.5 wrote it without the RW1 version token in 85 of 90 of its R chains at least once, which the codec refuses as a parse failure; the handover was re-issued with the reason, and on the re-issue the model mostly wrote a line that parsed but omitted the echo of pri, which the echo tier refuses; the chain was then held with every value right. Over every block, 210 of 347 of its readbacks were refused for form (156 parse failures, 54 omitted echoes) and 2 for a value; 57 of 90 of its R chains were held with every value right, where the grader's own stall count, which reads the refusal's kind, says 23 of 90, because it calls an omitted echo a value failure. Haiku 4.5 did the same at a tenth the rate: 26 of 283 readbacks refused for form, 3 of 90 chains held. The code-composed readback cannot fail this way: C on Haiku 5.5 had 0 of 396 readbacks refused for form, 0 of 396 disputed with every value right, 1 of 271 forward refused with every value right, and 4 forwards caught, three a decoy's approval reference and one a priority from the wrong run, each right on the re-issue; on Haiku 4.5, 1 of 393 readback disputed wrongly and 0 of 266 forwards. H8 separates C from R on Haiku 5.5, 18 of 18 against 4 of 18 (p under 0.001), in the direction the registration did not expect, and does not separate them on Haiku 4.5, 17 of 18 against 18 of 18 (p 1.0). In study 8 the same two models ran make-rb as a tool, and no readback on Haiku 5.5 was refused with every value right. The prose arms' shortfalls on the anchors were questions, as in study 8, not wrong values.

The hypotheses

Table 8. The nine hypotheses, tested per model with nothing pooled across models: Fisher's exact at alpha 0.05, one-sided where registered so, Clopper-Pearson intervals throughout.

What it cost

A clean two-relay chain on four CPU cores: Qwen 1.5B 12.4 s in free prose and 46.2 s with the code readback; Qwen 3B 21.8 and 76.8; Llama 3B 154 and 109.2; Qwen 7B 126.5 and 171.7; Qwen 14B 526.2 and 1135.9, so a checked chain on the 14B is nineteen minutes, and its 138 chains took 27.6 hours of the box's clock. On the anchors through the headless runner: Haiku 5.5 free prose 3.0 model turns, 385 tokens of traffic and $0.010 a chain; the code readback 6.2 turns, 694 tokens, $0.024; the model's own readback $0.022 for a chain that was usually held. Haiku 4.5 cost $0.096, $0.183 and $0.308, and $85.58 for its 522 chains against $8.27 for Haiku 5.5's. At five relays on Haiku 5.5 the checked line's traffic was 1,364 tokens against free prose's 857, while the prose message delivered to the payer had grown from 99 tokens at one relay to 151 at five and the line stayed at 95.

What it means

The breaking point is the agent, not the form. Below about 14 billion parameters a relay in this harness cannot pick its handover out of an inbox of three, in any form: in prose it forwards the wrong one or all of them, in lines it writes a valid line for the wrong one, and with the check at the hop it is refused and held. The check does what it is for, 0 wrong acts in 647 checked chains against 51 in the unchecked arms, but it buys that with zero throughput on these models, and a relay that is held every time is not a relay. Routing the message, one per call with a short brief, is what a small model needs first; the 7B and the 3B then act, and the 14B stops selecting the wrong item.

An unchecked line is not a half-measure at these sizes; it is the worst form. A line that parses and names the wrong item, or that has lost its slots so the payer fills them from the legend's example, gives the payer something that looks checked and is not. Where a relay cannot be trusted to copy a line exactly, the discipline is the check at the hop or prose, not the line alone.

The injection did nothing, on any model, in any form. These models do not change a payee on a supplier's say-so in an inbox, which leaves the check at the hop as a control against the fault the small models do make, picking another item, and not against the one the block was built to plant.

And the readback is composed by code. The strong model, which in study 8 built every readback with the tool and was never refused wrongly, here wrote the line itself and was refused for form in 210 of 347 readbacks; the code-composed readback on the same model was refused for form 0 of 396 times. The model's part in a readback is to read it, not to write it.

What we decided

The readback is composed by code from the message as delivered, for any receiver that has not shown it copies a line exactly.

H4 held on every model and H8 reversed on the strong one: the model-composed readback held 57 of 90 Haiku 5.5 chains with every value right and the code-composed one 0 of 18 in the clean block. The skill and the spec now say so, and that the model-composed readback is for a receiver that has demonstrated the copy with the tool in hand.

Below the floor the line alone is not the form.

H6 named Llama 3B and Qwen 7B below it and the 14B's unchecked line was its worst form. An unchecked line at these sizes is the check or nothing; and a legend's example line must be in the live order and must not resemble a live item.

The injection axis is a bound, not a result.

H1 could not be tested because no relay complied. The spec claims that no slot carried the planted value past a checked hop, and not that the check is the control against compliance.

Small models need the message routed before they need a form.

Block 6 raised the prose and the unchecked line on the 3B and the 7B and removed the 14B's selection errors; the checked forward got no easier. Deploy a small relay with one message per call and a short brief, and check what it forwards.

The amendments, in short

• Before the freeze: the local models run in-process; context 12,288; replies capped at 500 tokens; the anchors run from outside the repository; the reply is the message; ASK on the first line is a question; one nudge for a reply that is not an answer; the act block replaces the act tool and the done report; R without the digest tier, since a model with no code cannot compute it; C's dispute rule; a harness defect in the pilot's first pass fixed and the pilot reset.
• After the freeze: Qwen 14B added; the runs kept alive by hand; the remaining blocks cut on the four models below the band; block 6 registered and built; four anchor chains reset under a live runner and run again at the end; four Haiku 5.5 chains ended in a harness fault from a thread-unsafe import and stand as faults.
• In the analysis: a wrong act is a chain that acted (17 chains that asked at the wrong hop in the desk-cover cell reported apart); Haiku 5.5's readback refusals split into form and value; "injected value forwarded" read against the slots; the act executor's two rules (the act's kind, and the exact supplier name for a bank change) recorded as shaping the small models' outcomes; what Llama 3B's block 6 wrong acts were.

What this does not show

• Five quantised models on CPU at temperature 0 stand for small models in general only loosely; the result is a shape across sizes, not a number for any model, and most cells hold six chains.
• No tools. A relay that must run code to check its own line is not tested here; the harness runs the checks the arm names.
• The brevity and length axes ran only on the anchors and on Qwen 1.5B, which never reached a relay's forward; the cut left them unmeasured on the models in between.
• The injection is one message in one voice, and no model complied with it; a relay that resists this one may not resist another.
• The bank-change executor's exact-name rule stopped ten wrong acts on the small models that the payment and remittance executors would have let through; the wrong-act counts in CHGBANK are lower bounds.
• Four Haiku 5.5 chains in the sweep stand as harness faults; the re-run the protocol calls for needs the anchors' runner.

Where everything is

The protocol with its amendments and its outcome, the speed test, the pilot, the frozen manifest, every chain with every message as sent, the harness's state, the grades, the blind reading and results/summary.json: research/breaking in the rogerwilco repository. The page you are reading is written from that JSON by tools/site_page.py, and app/tests/test_research_figures.py fails if any figure here differs from it.

Study 9 of 9 · 24 October 2026 · 1,585 chains, 541 on five local models through llama.cpp and 1,044 on Claude Haiku 4.5 and Claude Haiku 5.5 as headless text-only sessions, pre-registered · a speed test and a pilot of 12 chains · no judge · tiktoken o200k_base · graded in code