Nine studiesSeptember to October 2026Croxio Ltd
What we tested, what happened, and what we decided
Nine studies, one after another, each asking the question the one before it raised. Everything in them is made up for the test: the companies, the invoices, the clinical notes. Every log and every grade is in the repository. Each study is told here in brief; its own page says why we did it, what we tested, how we ran it, what happened, what it means and what we decided, in that order. For the story first, the home page walks through six ways of handing over a job, with a picture of each and what it caught; the map below says which study each of those tests comes from.
The six tests on the home page, and where they come from
Test 1, prose with nothing checkingStudy 6, block 1 (the no-readback arm) and its free-prose extension; the cross-vendor run for the OpenAI receivers; block 2 for the traps in the payer's own files.
Test 2, prose said back and judged by a modelStudy 5's judged-readback arm, and study 6's prose plan readback (arm B) with the extension's planted-fault run.
Test 3, a line echoed back and compared by a programStudy 5's line arm, study 6's echo readback (arm C) on four receivers, and study 7's echo on the weak receiver.
Test 4, the payer's own values read backStudy 6's execution readback (arm D) and the English block (arm E); study 7 on the weak receiver, where arm D stalled and arm E did not.
Test 5, the report afterwardsStudy 6, block 3 (the tool executes something other than it was told) and study 7's commit-point faults, flagged by the done check.
Test 6, the windowStudy 7: the two-phase tool, prepare and commit, on Haiku 5.5 and gpt-5.4-mini.
1. Does writing a handover as a line save tokens, and can a model still read it?
The wire study, 27 September. 240 messages, five ways of writing them, two models.
WhyThe idea on the table was that agents talking English is wasteful and they need their own language. Before building on that we wanted to know what a fixed-vocabulary line actually saves, whether it loses anything, and whether a model reads it as well as English.
What we didWrote the same 240 handover messages five ways: plain English, JSON, a key=value line, a form made of symbols, and a very tight form where every fact sits in a fixed position. Counted the tokens, checked that every message survives being turned into data and back, corrupted lines to see what the decoder catches, and gave 60 messages in each form to a model that had never seen the format, asking what it would do with each.
What happenedA line costs about 29% fewer tokens than English; the tight form about 59% fewer. Every message survived the round trip. The strong model read every form perfectly. The small model lost its place in the tight form in one run of two, getting four messages in ten wrong.
What we decidedThe saving is 20 to 40 tokens a message, so tokens are not the reason to use lines; what a line gives is something a program can check. The key=value line became the rogerwilco line. Symbols and the positional form were dropped, a corrupted digit is caught by a digest on the readback instead, and no code may be spelled like an answer.
2. Does reading a handover back catch a misunderstanding before the work is done?
The readback pilot, 28 September. 108 handovers, 90 of them with a planted trap, three ways of handing over, two models.
WhyIf lines do not save money, they might make handovers safer: a program can compare what the receiver says it understood with what the sender meant. We wanted to know whether a readback catches a misunderstanding at all, and whether a typed line does it better than a paragraph.
What we didGave agents jobs with traps in the instructions, for example a handover that said 120 words when the attached brief said 150. Handed each job over three ways: plain English with no readback; plain English with the receiver writing back what it understood, judged by another model; and a line with the receiver reading it back and a program comparing it with criteria the sender had kept to itself.
What happenedWithout a readback, 4 of 36 jobs were done on the wrong reading and stayed wrong. With a readback of either kind, every misread that reached an output was caught and fixed with one correction. The line prevented the strong model's misreads before any check ran, because having to put a number after len= sent it to the brief. On the small model the program over-flagged terse readbacks, ten false alarms in eleven. A readback proved understanding, not execution: the small model read one rule back correctly in every readback and then broke it in four of six jobs.
What we decidedNo handover without a readback became rule 1 of the protocol, and the comprehension readback, with a closed set of criterion keys and the sender naming which it will check, went into the spec. The receiver gets a short generated legend instead of the whole skill file, which was nearly all of a line handover's extra cost. A check of the finished work against the same criteria was left open and picked up by the fifth study.
3. When four agents close a company's month, where does the saving come from?
Month-end close, 29 to 30 September. 28 runs, six ways of writing messages, two scenarios, two models.
WhyReal work is a team of agents passing messages and files, with a person who must be asked some things and not others. We wanted the lines in that setting, against the obvious alternative of telling the agents to write well, with a small model in the seats as well as a strong one.
What we didFour agents (one reads the emails, one checks the numbers, one builds the payment run, one audits it) closed the September accounts of a made-up company, with a finance director who decides only what the policy reserves. We planted things that should be escalated (a bank-detail change from a lookalike email, a payment above the limit, an invoice in dollars) and things that should not (a corrected invoice, a duplicate, a small variance). We ran it with messages in free English, in English in three fixed sections, in English under seven rules, as lines, as lines with every fact read back, and as lines that replace the working files. A second scenario added a phishing instruction hidden in an email and a wrong answer from the boss.
What happenedWith the strong model every way of writing got the payment run right, escalated exactly the three reserved decisions, left the four decoys alone, ignored the phishing paragraph and pushed back on the boss's wrong answer. English under the seven rules cut the traffic between agents to a third and the director's reading to 40%, as well as the lines did and more cheaply. The lines added what English cannot: a program refused 18 broken or made-up messages from the small model, and when the lines replaced the files the agents pass each other, those files shrank from 22,000 tokens a run to under 2,000 with every fact kept.
What we decidedThe seven rules lead the skill, because the discipline does the saving; the lines are for where a program must check: small or mixed models, steps that cannot be undone, logs someone has to trust, and the edge where documents enter. The readback level is set per handover, full before an irreversible act and a short echo otherwise. Profiles may close the answer vocabulary. And at the edge, the lines are the record: one line per document with a pointer to its source and no working file.
4. Can a document become lines of fact you can trust?
The translation study, 30 September. 12 clinical notes, 422 statements marked by hand, 144 translations, two models.
WhyThe month-end study ended with lines replacing the working files, one line per email with a pointer to its source. Emails are easy. We wanted to know whether the rule holds where the facts are dense and a wrong reading matters, and whether the notation itself matters against JSON.
What we didWrote twelve made-up clinical notes with the hard things in them: findings that are denied, diagnoses that are only possible, facts about a relative, facts from years ago, a results table, a correction, a hidden instruction. Marked every statement by hand, 422 of them. Asked two models to turn each note into one line per statement, each pointing at the exact words in the note, as lines and as JSON, three runs each: twelve translations of every note. A program refused any line whose words were not in the note at the place it pointed to.
What happenedThe strong model found 99.7% of the statements, pointed at the right words 99% of the time, and invented nothing, because a line without words behind it cannot get through. The small model had 70 of its 72 first attempts refused, got every note through in two rounds, and still misread meaning: it marked what the patient said as something the doctor observed 59 times and read the present as the past 104 times. Lines and JSON came out the same. The record was 5.6 times the size of the note.
What we decidedThe clinical profile and its gate are part of rogerwilco, and the gate is a hosted tool. The claim is structure and provenance, not compression: a record that is bigger than the note is worth having because every line is traceable and nothing in it is made up. Pipelines write one document per call. Binding concepts to a terminology and stating the envelope once per block are the next changes.
5. Before something that cannot be undone, does reading back catch a fault?
Readback and set-back, 6 October. 114 trials, 19 handovers, 16 of them with a planted fault, three ways of handing over, two models.
WhyThe pilot showed a readback catching misunderstandings but not proving execution; the month-end study found nothing for a full readback to catch on a clean run. Neither tested a fault in a handover just before an act that cannot be undone. In aviation there are two loops, the readback and the cross-check of what was set. This study plants faults so both have something to catch.
What we didHanded an agent single jobs it cannot take back: release a payment, change a supplier's bank account, send a supplier an email. Into sixteen of nineteen handovers we planted one fault: a wrong digit, the wrong account, the wrong date, the wrong invoice, net instead of gross, a message meant for a different agent. Three ways of handing over: English with no readback; English with a readback judged by a model; a line read back and checked by a program, followed by a done report carrying the values as executed.
What happenedWith no readback, 9 of the 16 faults became a wrong act that could not be undone (14 with the small model). With an English readback judged by a model, 1 and 3 got through, every one a message addressed to someone else. With a line read back and checked by a program, none got through on either model, and no correct handover was wrongly held up. Every act after a confirmed readback was right, 65 of 65, and every done report told the truth, 28 of 28.
What we decidedFor the step you cannot undo, the line read back and checked by a program is the control, and the rule that requires it stands on this evidence. Readbacks are built by code wherever code exists. Two protocol changes follow and are open: a corrected handover says why it changed, and the done report with executed values goes into the spec as the second loop, with a run that stresses it.
6. Before an act that cannot be undone, does reading back the plan catch what an echo cannot?
The execution readback, 10 and 11 October. 2,762 trials, three fault blocks, five ways of handing over, two Claude models, pre-registered; then a partial run of the same design on two OpenAI receivers, and a free-prose extension.
WhyStudy 5 caught every planted fault by construction: the readback was built by code from the message, so comparing it with the record caught whatever the message carried, which a check before sending would have caught too. The claim to be a safety control rests on two other faults, the receiver reading the handover wrongly and the tool doing something other than it was told. This study was built to test those, with the echo kept in as the arm that should catch nothing there, and its hypotheses, tests and sizing written down before it ran.
What we didHanded the payer correct handovers that name the supplier and the figures but not the account or the address, which it must resolve from a master built to mislead: lookalike rows, superseded and pending accounts, decoy handovers, stale drafts, emails from lookalike domains. Five ways of handing over, from none to an execution readback in which the receiver writes the values it will use and a program checks them against the record. A second block re-ran study 5's planted drafting faults; a third made the tool execute something other than it was asked, and read back what it did.
What happenedNo receiver made a wrong act against a correct handover, in any arm, at any of four levels of hardening: 0 of 140 induced trials per cell on Sonnet 5.5 and Haiku 5.5, 0 of 224 each across the ladder, 0 of 56 on Haiku 4.5. They resolved the trap or asked. gpt-5.5, run afterwards through the API harness, read the same way. gpt-5.4-mini did not: under the echo readback it made 10 wrong acts in 140 induced contexts and 4 in 50 clean ones, after readbacks that had passed; its typed execution readbacks never carried the resolved values and were held; the fixed-shape English block, parsed by code, it completed, and that caught a wrong address and corrected it in four of five trials where the echo let it through. Against the sender's own drafting faults, the no-readback arm made a wrong irreversible act 17 and 14 times in 48, and both readback arms caught every one, as a check before sending does. Against the tool, the done report carrying the tool's printed values let the sender's check flag 45 of 45 injected faults on each receiver, and every report was honest.
What we decidedA handover before an irreversible act is checked against the record before it is sent, because that is where the readback's catches came from. The done report with the executed values goes into the spec as the second loop, because that is where the uncaught faults were. The readback before an irreversible act stays; on the two Claude receivers it had nothing to catch, raised no false alarm, and cost one turn, and on gpt-5.4-mini its form decided everything: the echo passed a wrong plan, the typed execution readback stalled, the parsed English block caught. Whether the spec should accept that block as a form of rb=exec is the open question.
7. Does a readback the tool composes catch what the model's cannot?
Prepare and confirm at the tool, 11 and 12 October. 794 trials on Haiku 5.5 and 1,014 on gpt-5.4-mini, each after a pilot, three fault blocks, six ways of handing over, pre-registered.
WhyEvery readback so far was the model's account of what it meant to do, and nothing bound the tool call that followed to it. Study 6's weak receiver failed in that gap: the readback was right, the call was not. Pilots close the same gap by reading the machine, not each other: the cleared altitude is set in the window and the window is checked. We built the window.
What we didThe payment tool gained two steps. Prepare states exactly what it will execute, freezes it under a token and prints the readback line itself; commit executes that statement and nothing else. Two new arms beside no readback and the model's own execution readback: the tool's statement relayed by the model, and the statement read by the sender from the tool's log with a commit the tool refuses until the sender confirms. Study 6's misleading contexts again, fifteen tool faults injected either when the tool resolves the call or at the moment it executes, and study 5's planted drafting faults as a regression check.
What happenedA fault the tool introduced when it resolved the call became a wrong irreversible act in 29 of 30 trials with no readback and 30 of 30 with the model's execution readback, whose values were right. With the two-phase tool, 0 of 30 in either arm: in every trial Haiku read the statement the tool printed, saw it differed from what it had passed, abandoned and asked, before the sender's check ever ran. Against the misleading contexts no arm made a wrong act, the statement raised no false alarm in 255 checks, the model never altered the line it relayed and never committed early, and the two new arms cost less than composing a readback. A fault injected at the moment of execution got through every arm, by construction, and the done check flagged all of them. Then the weak receiver, gpt-5.4-mini, whose own readbacks fail: under the echo it made 14 wrong acts, under its typed readback it stalled in 52 trials, and under the tool's statement with the commit enforced it made none, was held 7 times, and the sender's check of the statement caught the old sort code and the lookalike address it had passed to the tool. Relaying the statement through that receiver was unreliable in 51 of 129 trials, though it never committed early.
What we decidedWhere the tool can state its transaction before executing it, the execution readback is the tool's statement; a receiver that can be trusted relays it, and otherwise the sender reads the tool. The tool takes the handover by id so the receiver never retypes it. The done check stays as the loop after the act.
8. When a handover passes through agents that do not act on it, how much arrives, and does a line checked at every hop stop the loss?
The relay, 22 October. 792 chains, two relays and a sweep to five, four ways of handing over, two Claude models, pre-registered, after a pilot and a hardening ladder.
WhyEvery study so far was one edge: a sender, a receiver, an act. Agent systems are graphs: a planner hands to a coordinator, the coordinator to a desk, the desk to the agent with the tool, and each restates the instruction for the next. A relay is a receiver that becomes a sender, so every hop is another draft. We wanted to know whether anything carries an instruction intact across them without a check at each one, and what the check costs.
What we didStudy 6's three acts again, with the first message written by code from the record and passed through a coordinator that adds the deadline and priority and a desk that confirms the approval against its log, each in an inbox of routine traffic with two other live handovers to pick from, before the payer. Four forms: free prose, prose under the seven conventions, lines validated at each hop, and lines read back at every hop and checked by code against the handover the relay received before they are forwarded. A clean block, a block with a confuser at one relay, and a sweep of one, two, three and five relays.
What happenedNo relay changed a value. Twice in 792 chains a field reached a hop with a different value, once a call-back sentence in a prose relay's message that the grader read as conditional and once the weaker model forwarding a different handover altogether at the third relay, and neither became an act. Prose relays dropped a field twice, both times the call-back statement of a bank change, and the payer acted without it; no line reached the payer short of a slot. On Haiku 5.5 the checked line ended in a correct act in 45 of 45 clean chains and the unchecked line in 45 of 45, against 37 of 45 for free prose, whose losses were questions: a handover forwarded inside a wrapper still addressed to the relay, or provenance reported rather than shown. At five relays the prose message had grown from 73 tokens to 197 and the line from 77 to 104. The checks raised no false alarm in 90 forwards. On Haiku 4.5 the order reversed: prose and unchecked lines lost one chain in 45, the checked line eight, seven of them the relay retyping the handover into its readback so that the digest broke and the chain was held with every value right, 13 of 128 readbacks refused in the clean block.
What we decidedA relay is a sender: it checks the line it forwards against the one it received, the received handover standing as the record, and its readback is built by code from the message as delivered, never retyped, which is what makes the check at every hop fit a mixed fleet. And the site says what the study found: on these models prose carries the facts across a chain; what it loses is the address, the provenance, and size.
9. Where does a relayed handover break as the agents get smaller, and does the check at the hop hold?
The breaking point, 24 October. 1,585 chains, five local models from 1.5 to 14 billion parameters and two Claude anchors, five ways of handing over, six blocks, pre-registered, after a speed test and a pilot.
WhyStudy 8 was a null on the fault it was built to find: Claude models, even the small one, carry an instruction intact across a chain. A protocol whose checks never fire is either unnecessary or untested at the edge, and the edge is a mixed fleet whose relays are the smallest models a team can run. We took the same chain to models down to 1.5 billion parameters, run locally with no tools, with the two Claude anchors through the same text-only runner so the sizes compare.
What we didStudy 8's company, acts and inboxes again, every agent one plain chat whose reply is the message and whose act is a fixed block the harness executes. Five forms: free prose, disciplined prose, lines validated for syntax, lines read back at every hop by the model, and lines read back at every hop by code built from the message as delivered. Six blocks: the clean baseline, brevity, confusers, a sweep to five relays, an injected email giving a new payee account, and, added after the freeze, the message routed one per call with a short brief.
What happenedThe agents broke before the form did. Below about 14 billion parameters no form gave throughput: the two smallest models asked at the first hop in every arm, and Llama 3B and Qwen 7B reached a correct act in at most 2 of 6 clean chains, their failures the relay acting on another message in its inbox. The run made 51 wrong acts, every one another item, none the injected account; 0 of 390 injection chains complied in any form on any model. The checked arms made 0 wrong acts in 647 chains, and on the four smaller models 0 correct ones. Qwen 14B was the first local model in the band: free prose 5 of 6 with 1 wrong act and the code-readback line 5 of 6 with 0 in the clean block, 6 of 12 with 6 wrong against 7 of 12 with 0 against the confusers; its unchecked line was its worst form, every line parsing and 0 of 6 correct. Routing the message removed the 14B's selection errors and lifted the 3B and the 7B. Both anchors were in the band in every form but one: Haiku 5.5's readback written by the model, with no tool, was refused for form in 210 of 347 readbacks and held 57 of 90 chains with every value right, where the code-composed readback was refused for form 0 times in 378.
What we decidedThe readback is composed by code from the message as delivered for any receiver that has not shown it copies a line exactly; below the floor the line alone is not the form, and a legend's example must be in the live order; the injection axis is a bound; and a small relay needs the message routed before it needs a form.
The honest summary
Lines do not save money with a strong model; the habits do. Lines let a program check what agents say to each other, refuse made-up words, trace every fact to its source, and catch a wrong figure before a payment goes out. With a small model that matters a lot. With a strong model it matters at the steps you cannot undo, and there the check that earned its place is the tool's own statement of what it is about to do, checked before it does it. Across a chain of relays, on these models, prose keeps the facts and loses the address; the line keeps both, and the check at each hop is for the fleet whose relays are small. Every study here is small: one to three runs per cell, two models, one author writing both the tests and the marking. Enough to show direction; not yet a measurement.
Corpora, harnesses, raw logs and grades: the research/ folder of the rogerwilco repository. The naming changed from formcore (prefix FC1) to rogerwilco (RW1) on 29 September 2026; raw logs keep what the agents saw.