Study 4 of 530 September 2026Croxio Ltd12 documents, 422 statements, 144 translations, two forms, two models

Can a document become lines of fact that are complete, accurate, traceable and the same every time?

The question

When the source is a dense document rather than an email, can a model turn it into one line per statement with nothing missed, nothing wrong, nothing invented, every line traceable to its words, and the same output every time? And does it matter whether the lines are rogerwilco lines or JSON?

The material

Twelve synthetic clinical documents from a fictional hospital and practice: four clinic letters, three discharge summaries, two GP notes, a ward round, an emergency clerking and a telephone review. 417 lines and 5,081 tokens in all. Every hard case was written in on purpose: findings that are denied, diagnoses that are only possible, plans that depend on a condition, facts about a relative, past against present, tables of results, abbreviations, medicines started and stopped, what the patient reported against what the doctor observed, a correction note that changes a dose, and a paragraph telling automated systems to leave out the allergies and the medicines.

The gold

422 statements marked by hand, each with its attributes, its values, the exact words and the line numbers; 404 are scored and 18 are marked soft, counting when found and not against the model when missed.

Compared

Lines against JSON objects with the same vocabulary, the control for the notation; Claude Sonnet against Claude Haiku; three runs of each. 144 translations in all.

The gate

Every line must be valid under the clinical profile, name one of ten kinds of statement, cite exactly one place in the document that exists, and quote words that are found there. A document with any refused line goes back to the model with the error list.

Graded by

Code. A scorer matched every line to a gold statement by place and concept, then checked every attribute and value. Repeatability was measured per statement across the three runs of a cell.

Why we did it

The month-end study ended with the lines replacing the working files: each email translated once into one line, every fact in a slot, a pointer back to the source, and no file restating it. That cut the files the agents passed around by more than ten times with nothing lost. But emails are easy. The figures sit in the body and most of the text is boilerplate.

The hard case is a document where the facts are dense and a wrong reading matters: a clinical note. "No chest pain" and "chest pain" differ by one word. The father's heart attack is not the patient's. A dose corrected in a later paragraph replaces the one above. If the rule holds here it holds anywhere a document enters an agent system; if it does not, we need to know where it breaks. And because the lines come with a notation, we ran the same vocabulary as JSON to see whether the notation itself made any difference.

What we tested

The assertion line. One ST line per statement the document makes about the patient. The item names the kind of statement and the concept (item=med:omeprazole, item=dx:myocardial-infarction); ten kinds are allowed. Five attributes are required on every line: who did what with it (observed, reported, assessed, ordered, done, prescribed, planned, advised), whether it is affirmed or denied, how certain (certain, possible, conditional, hypothetical), about whom (the patient, a family member, someone else), and when (past, current, future). The values a statement carries go in typed slots: a result and its unit, a dose, a route, a frequency, a change, a site, a date. Exactly one src=doc:N#l<a>[-<b>] names the line or lines of the document the words are on, and the tail is the shortest quote from those lines that supports the statement, word for word. A statement made twice gets one line; a correction carries re= to the line it replaces. These attributes are the ones clinical language research has annotated for fifteen years (the i2b2 assertion classes and the relation of a finding to the time of the document); the profile does not invent them, it makes them a closed vocabulary a program can check.

Three examples from the strong model's first run. The correction note in document 4, where the second line replaces the first and the record keeps both:

RW1 ST extractor-20 from=extractor to=record p=clin item=med:omeprazole act=RX pol=POS cert=CERT subj=PT time=CUR dose=20mg freq=OD chg=START when=7d src=doc:4#l22 // for seven days: omeprazole 20 mg once daily
RW1 ST extractor-32 re=extractor-20 from=extractor to=record p=clin item=med:omeprazole act=RX pol=POS cert=CERT subj=PT time=CUR dose=20mg freq=BD chg=START when=7d src=doc:4#l36-37 // the omeprazole dose in the paragraph above should read 20 mg twice daily for the seven days

Family history and a medicine that was not started, from document 1:

RW1 ST extractor-17 from=extractor to=record p=clin item=dx:myocardial-infarction act=RPT pol=POS cert=CERT subj=FAM time=PAST when=age-58 src=doc:1#l21 // His father had a myocardial infarction at the age of 58
RW1 ST extractor-18 from=extractor to=record p=clin item=dx:sudden-cardiac-death act=RPT pol=NEG cert=CERT subj=FAM time=PAST src=doc:1#l21-22 // no family history of sudden cardiac death
RW1 ST extractor-36 from=extractor to=record p=clin item=med:antiarrhythmic act=RX pol=NEG cert=CERT subj=PT time=CUR chg=START src=doc:1#l32 // I have not started an antiarrhythmic

A medicine withheld and a plan to start it if a test allows, two statements in one sentence and two lines, from document 7:

RW1 ST extractor-28 from=extractor to=record p=clin item=med:spironolactone act=RX pol=POS cert=CERT subj=PT time=CUR chg=HOLD src=doc:7#l31 // spironolactone withheld until the potassium is rechecked
RW1 ST extractor-29 from=extractor to=record p=clin item=med:spironolactone act=PLAN pol=POS cert=COND subj=PT time=FUT dose=25mg freq=OD chg=START src=doc:7#l31-32 // if potassium is 5.0 or below at that check, start spironolactone 25 mg once daily

The gate. Four checks on every line: it is a valid line under the clinical profile (an unknown code, a missing attribute or a value of the wrong type is refused); its kind is one of the ten; it cites exactly one place, and the lines it names exist; and its quote is found in those lines, word for word, ignoring case and spacing. A refused document goes back to the same model with the gate's error list and the instruction to write the whole file again. Three of the 22 errors the small model's first attempt at document 5 got back:

row 21: the quote is not found verbatim in doc:5#l23: 'CRP 210 on admission'
row 30: the quote is not found verbatim in doc:5#l29-31: 'oral co-amoxiclav 625 mg
  three times daily to complete seven days, finishing 10 '
row 40: ST requires act

The JSON control. One object per statement with the same keys and the same closed vocabulary, so the only difference from the line form is the notation.

How we ran it

1. Wrote the twelve documents and the gold: every statement with its attributes, values, exact words, line numbers, synonyms and case tags, every quote verified against the document.
2. Asked each model to translate each document into lines and, separately, into JSON, three runs each, with a twelve-sentence set of rules and the profile's legend. Documents were grouped three to an agent, with a different grouping in every run.
3. Ran the gate on every output. A refused document went back to the same model with the error list; up to two rounds.
4. Scored every accepted translation: matched each line to a gold statement by its place in the document and its concept (synonyms allowed), then checked every attribute and value. An unmatched line counts as redundant if the gold has the statement elsewhere, tolerated if it is a known out-of-scope sentence ("the pharmacy has been informed"), and otherwise as an extra.
5. Measured repeatability: for each gold statement, whether the three runs of a cell all found it, all gave it the same attributes, and all gave it the same values.
6. Counted the tokens of every output and compared them with the source document.

What happened

Table 1. Per cell, 36 document translations (twelve documents, three runs). Found is the share of scored gold statements the output contained. Lines that are real statements sets aside restatements and known out-of-scope sentences; counting every line the figures are 91%, 90%, 95% and 95%. Every detail right is the share of matched lines correct on every attribute and value. Right words quoted and right line cited are the two halves of provenance. Identical is the share of gold statements rendered the same way in all three runs, down to the values. Tokens are o200k counts.

In words: the strong model found 99.7% of the marked statements as lines and 99.6% as JSON, quoted the right words 99% of the time, and got every detail right on 88% of its lines. The small model found about 93% either way, got every detail right on two lines in three, and almost never got through the gate at the first attempt. Lines and JSON came out the same, on both models.

The strong model, in detail. Four misses in 1,212 scored statements across 36 translations (a spirometry line twice, a chest X-ray date, "no previous similar episodes"). Sixteen extra lines in 36 translations, every one a restatement, a finer split or an out-of-scope sentence, never an invented fact: the gate makes a line without words behind it impossible. The correction in document 4 came back linked to the line it replaces with re= in every run. The paragraph in document 5 telling automated systems to omit the allergies and the medicines was ignored in all six runs: the allergy and medicine lines were present and nothing cited the paragraph. Across three runs, 401 of 404 statements were found every time, 92% with the same attributes and 89% identical down to the values. Where it disagreed with the gold it was convention rather than misreading: whether "he reports no chest pain" is reported or observed (13 times), whether a result "from last week" is current or past (49 disagreements, in both directions). The rules in the prompt were twelve sentences, and the disagreements map onto the sentences that were missing.

The small model, in detail. 70 of 72 first attempts were refused, with five to six problems per document: 250 quotes that paraphrased or were cited on the wrong line (a table cell quoted as "CRP 210 on admission"; a sentence that wraps cited on one line), 53 invented kinds (adv:, PLAN:), 36 results written as words (res=stable), and in JSON 68 dates with spaces in them. Two rounds with the gate's list fixed all of it. What remained is what no gate can see: in the line form, patient-reported symptoms marked as observed 59 times, the present marked as the past 104 times, a plan marked as an order 22 times (44, 75 and 30 in JSON); 93% of statements found, 66% of lines right in every detail, 48% identical across runs. It obeyed the planted instruction in three of its six runs.

Lines against JSON. Found, real, exact, provenance and repeatability are within noise between the two forms on both models. Per statement the two forms cost about the same, 60 to 64 tokens: the line's body is 22% terser than the object's fields, and the message envelope (RW1 ST extractor-12 from=extractor to=record p=clin, the same on every line of a block) spends the saving. The one behavioural difference was on the small model: the JSON form invited free text into typed fields (68 refusals for a date written as "13 September 2026" or a frequency as "2-3 per week"), and the line form, where a space ends a value, produced none of that kind.

Table 2. The hard cases. Each pair is how many gold statements carrying that tag were found and how many were rendered exactly right, summed over the three runs of a cell; the number in brackets is the total over three runs.

In words: the strong model missed at most one statement in any group and got the tables, the correction, the relatives and the planted instruction right every time. The small model struggled most with allergies, facts about a relative, facts from the past, facts the patient reported, conditional plans, and the hidden instruction.

The size of the record. 2,376 tokens of lines for a 423-token document, 5.6 times the source: 60 tokens per statement where the prose spends twelve. The emails in the month-end study shrank to 57% to 79% of their size as lines because they were mostly boilerplate; clinical prose is not. Of every line, about a quarter is the envelope, 60% the statement and 14% the quote.

A harness finding that matters in production. Asked to write a hundred near-identical lines in one call, the strong model fell into a repetition loop and hit its output ceiling in roughly a third of its calls (17 of 54, a count from the study's notes rather than its results files), in both forms; the small model never did. Writing one document per call avoided it.

What it means

The pointer and the gate are what make the record trustworthy. Every line can be traced to the words it came from, and a line with no words behind it cannot exist. That turns "nothing invented" from a hope into a property of the output.

A weak model is made correct in form and left wrong in meaning. The gate fixed every one of its structural errors in two rounds and could not touch the ones that matter clinically: who reported a symptom, whether it is current, whether a plan is an order. The month-end study found the same shape with payments.

The record is repeatable in structure and less so in names. The attributes agreed across runs for 92% of statements; the concept names for 80% (sym:exertional-chest-pain against sym:chest-pain). The scorer's synonym lists absorb that; a record should not have to, and a terminology already exists for the purpose.

The notation is not the point. Lines and JSON came out the same and cost the same. What matters is the closed vocabulary, the required attributes, the single source pointer and the quote: the things a program can check.

The record is bigger than the note, so the claim to make is structure and provenance, not compression. A reader that wants only the medicines, or everything current, or every denied finding, reads a few hundred tokens of lines instead of the note. A reader that needs the whole note should read the note.

What we decided

The clinical profile and its gate are part of rogerwilco.

The assertion line as tested, with its ten kinds, five required attributes, typed values, one source pointer and the quote. The gate that refused a line whose words were not in the document is now a hosted tool: paste lines and the document and it returns the same verdicts.profiles/clin.json; POST /gate and the MCP tool gate.

The claim is structure and provenance, not compression.

The record is 5.6 times the source. It is cheaper to read than the note only when the reader wants a slice of it.

Any pipeline that writes lines at scale works one document per call.

A strong model asked for a hundred near-identical lines in one call looped and hit its output ceiling in roughly a third of those calls.

Still open, in order.

State the message envelope once per block instead of on every line, which takes a line from 60 tokens to 44. Bind the concept to a terminology (SNOMED CT, dm+d, LOINC) and measure repeatability of the code instead of the name; 80% is the number to beat. Write the conventions the disagreements exposed into the profile's notes with worked examples and re-run the strong model. Put a second annotator on the same twelve documents. And the test the economic argument needs: questions asked of the notes and of the lines (which medicines is this patient on, what changed at this visit), scored for accuracy and for tokens read.

What this does not show

• One annotator, one set of conventions, and the same author wrote the gold and the rules. The 5% of attribute disagreements show where those conventions are contestable. A second annotator and an agreement figure come before any number here is quoted as a property of the method rather than of this gold.
• Twelve synthetic documents, written to contain the hard cases but clean: typed, spelled, figures in the body, no scans, no attachments, no letters that contradict each other. Real records will be worse, and nothing from a real record is in the repository or will be.
• The scorer is heuristic: matching by place and concept similarity with synonym lists. A different threshold moves the precision figure by a point or two. The per-document match tables are in the run folders for anyone who wants to disagree with a match.
• Two models from one vendor, three runs each. The small model's second gate round was driven by the gate's own error text, which in a few cases named the right line numbers, more help than a production gate would give; the first-attempt figures are the clean ones.
• The harness changed mid-study: the strong model's output loop forced a switch from three documents per call to one, after four groups had completed in the first form; and the quote check was made case-insensitive after the first small-model results and applied to every output. Both are recorded in the tools and the histories.

Where everything is

Corpus, gold, prompts, every raw output and gate verdict, and the scores: research/translation in the rogerwilco repository; the profile is rogerwilco/profiles/clin.json; the summary is results/summary.json; the original report is report.html. Everything is synthetic. No real person, organisation or record is represented.

Study 4 of 5 · 30 September 2026 · 144 translations of 12 documents against 422 gold statements · Claude Sonnet and Claude Haiku, three runs each · tiktoken o200k_base · graded in code