What we investigated, how we measured it, and what we concluded — including the results that sent us backwards.
12 notes
EvaluationSpecterShipped
A benchmark that could not fail on row selection
Every question in our answer-quality corpus had its answer near the top of its source table. The tables ran to tens of thousands of rows. The benchmark was scoring retrieval only where retrieval cannot lose.
The sampler that built the corpus walked each table’s rows in ascending order and stopped at the first matches it found. Combined with a row-fetch limit, a wide table’s deep rows were never even read. The bias was structural: ascending-and-break can only ever produce top rows.
Those top rows are exactly the ones the renderer emits for free when no row hint exists — so the corpus asked nothing that required selecting a row at all.
The symptom that exposed it: two genuinely correct retrieval fixes shipped and moved the measured number by nothing.
The fix is a deep-row question slice and a sampler that cannot only produce top rows. An instrument that cannot register a failure will report progress that is not there.
EvaluationSpecterReverted
A confidence signal that inherited a calibration it never earned
We shipped a new evidence-confidence discriminator on the strength of a calibration measured on a different signal — one production cannot compute. We re-measured it, and took it back out.
The calibrated signal required the ground-truth answer, which production does not have. The shipped code substituted a proxy that had never been calibrated — and the pull request, the commit message and the docstring all quoted the original figure.
Re-measured against the full question corpus, the shipped proxy retained roughly a third of the separation the calibrated signal had. It called the majority of turns “not strong”, and most of those turns did contain the answer. A threshold sweep found nothing better that was not overfitted to the data that scored it.
We reverted to the previous value and kept the candidate and its harness in the tree, so a retry is designed and measured before it ships rather than after.
A justification can outlive the substitution that invalidates it. The number stays in the text while the thing it measured is quietly swapped out.
EvaluationShared engineShipped
Validate a benchmark before you spend on it
A benchmark plan endpoint that refuses runs it can prove will not work, arms that name the model that actually ran, and a results store so a run’s numbers outlive the process that produced them.
A plan step validates a benchmark before it is paid for. It caught a plan that had green-lit four evaluations that could not run at all.
Paired harnesses are checked against themselves: two identical arms must report no delta. When they did report one, that was a harness defect, not a finding.
Every arm now names the model that ran it, which sounds obvious and was not true before — one investigation traced reported “measurements” that were nothing of the kind back to three separate causes.
Results are written to a store rather than to files on a box, and any model in the catalogue can be benchmarked, not only the one a job already runs. Scores across models are compared on one screen.
EvaluationSpecterShipped
Judged evaluations, bound to preregistered identity
Golden structure evaluations and a judged reporter evaluation, with every paid phase bound to the exact prompt and code that produced it.
A golden evaluation harness for generated analysis structure, with the provenance of each generation exposed rather than inferred.
A second-tier reporter evaluation runs with clean-context seams and an explicit judge protocol, so the judge cannot be primed by the material it is grading.
Every paid evaluation phase is bound to preregistered prompt and code identity. A result you cannot trace to exactly what produced it is not a result you can act on.
Calibration of the judge itself is registered alongside the run — including the cases where a systemic defect must abort the campaign rather than be averaged into it.
ModelsShared engineShipped
One dispatcher, declared capabilities, and honest refusals
Every model call is named as a task and resolved through a binding. A model’s capabilities are declared up front, and the engine refuses what a model cannot do rather than discovering it mid-run.
A task registry names every LLM functionality in the system, so a call site asks for a task rather than for a model. Bindings map tasks to endpoints, and are consulted on every call.
Providers sit behind one seam — Anthropic, Gemini, Bedrock and OpenRouter among them — and every adapter renders images rather than silently dropping them.
Per-binding parameters are typed and unknown keys are refused: timeouts, output caps, temperature, and a concurrency lane of its own.
Offered models are probed for retirement before a customer finds one that has gone. An early version of the probe assumed capabilities instead of measuring them; that was itself a finding, and the probe now narrows only on evidence.
RetrievalSpecterIn progress
An explicit context budget, and abstraction when it runs out
A report that silently drops evidence is worse than one that says what it could not fit. So the budget became explicit, and what does not fit is abstracted rather than discarded.
A shared budget packer replaced ad-hoc truncation across the assembly path, with file-affinity bucketing and quantized salience tiering deciding what gets full text.
When completeness cannot be met inside the budget, an abstraction stage produces a schema-forced summary of the members that did not fit, with per-reference locators so a citation still points at something real.
Abstracts are cached against validated membership, with corruption treated as a cache miss rather than as data.
Re-planning runs as a round-indexed state machine with monotone per-level states, so coverage waves converge instead of oscillating, and per-run budget telemetry records what was fitted and what was not.
It ships dark. The packer and the abstraction stage are behind flags that default to off, and they stay there until they are measured against the path they would replace — the whole point of the telemetry is to make that comparison possible before the switch is thrown.
EvaluationSpecterShipped
Golden cases for the valuation engine
A financial model that is confidently wrong is worse than no model. Tier-1 golden cases and a Tier-2 extraction evaluation pin the engine to cases whose answers we already know.
A Tier-1 golden-case harness covers the core engine and its adapter against hand-checked cases.
A Tier-2 extraction evaluation runs against midterm gold sets, separating an extraction failure from a modelling one.
The language contract is gated on an evaluation of its own: a completion that is truncated is rejected rather than rendered, and generated prose is checked in every supported language rather than only in English.
Diagnostics explain a valuation’s sign rather than asserting a figure, and staleness fingerprints mark a projection whose inputs have moved underneath it.
SystemsShared engineIn progress
Paying for OCR once
A file whose content hash is already known rejoins the pipeline after the vision stage, instead of being read again.
Data rooms are full of the same document under different names, and every copy used to be read from scratch.
A content-hash cache lets a hit skip the vision stage entirely and rejoin at evidence construction, where the work is genuinely per-project.
The cache is keyed on content, not filename, so a renamed or re-exported copy is recognized as the same document.
It runs behind a flag that defaults to off. A cache that returns the wrong document is worse than paying twice, so it stays gated until the identity check has been exercised at production scale.
RetrievalShared engineShipped
Removing the row-level embedding channel
We embedded every table row, and it was the largest structure in the system. A paired before/after study said retrieval did not need it. We removed it.
Method: paired before/after on the same golden set, across three question tiers — deterministic exact-value questions, model-generated semantic questions, and a document control tier that a table-channel change must not move. If the control tier moves, the measurement is broken, not the system.
Post-rerank is the number that counts, because it is the order a user actually sees. Stage-1 candidates are reported but are not the verdict.
Queries that could not complete inside the time bound were dropped from both sides rather than scored as misses, and every exclusion was reported. Time bounds the sample; it must never bound the score.
The decisive tier was run four times, because a fresh question set moves single digits run to run and one draw is not a result.
Outcome: retrieval-neutral post-rerank on the customer-scale library across all three tiers, with the control tier flat — which is what makes the measurement trustworthy. Exact-value lookups now resolve through SQL over the retained cells. Index footprint and query latency both fell.
RetrievalShared engineShipped
What retrieval actually does, measured rather than assumed
Six channels, rank-fused at equal weight, re-scored by a cross-encoder, then rendered into a budget. Writing it down against the source is how we found the parts that were described wrongly.
Keyword and dense channels run in parallel across evidence, tables and rows. Reciprocal rank fusion combines them at equal weight — raw scores are discarded, only ranks survive, which keeps one channel’s scale from dominating the others.
A cross-encoder then re-scores the fused candidates. For a table it scores the metadata signal together with the matched rows, taking the best — so which rows were matched decides what the reranker, and ultimately the model, ever sees.
Japanese keyword search had been described in our own docs as dead. Run against live data, it turned out to be partially functional and term-biased. Sharpening it with a per-library stoplist of the highest document-frequency tokens measurably improved correctness.
Each claim in the reference is marked as measured at runtime or reasoned from code, and the two are never mixed. Where the code and the prose disagree, the code wins and the prose gets corrected.
Document AINebulaShipped
Layout-aware cropping, and routing by region type
A page is not one thing. Segmenting it, cropping each region, and reading each crop with an instruction chosen for that region beats one prompt over the whole page.
Layout detection segments the page; each region is cropped and read with a per-crop instruction resolved from a central store, so an operator can change how a region type is read without a deploy.
Handwritten regions are routed to the model that reads them best, rather than being handed to whichever model is reading the rest of the page.
Pre-OCR orientation correction turns pages the right way up before anything reads them, with a backup model when the primary one fails — a scanned page fed in sideways is a preventable class of error, not an accuracy problem.
Throughput came from a distributed rate governor with per-model limits, batch-level sub-PDFs, and a pipelined render/OCR stage — density without tripping provider limits.
ModelsNebulaShipped
A Japanese fine-tune, and a switch to measure it against
Frontier models read Japanese business documents well. They do not read Japanese investor-relations layouts as well as something trained on them. So we trained one — and kept the base engine one command away.
A LoRA fine-tune of an open-weight vision-language model on Japanese investor-relations and financial material, served on dedicated GPU capacity so throughput does not sit behind a third party’s queue.
A one-command toggle switches the engine between the base model and the fine-tune. Without it, a comparison is an argument; with it, both run over exactly the same inputs.
Per-page timeouts are configurable, so one dense page cannot hold a batch hostage — a bound on the worst case, not an average.
The fine-tune sits alongside frontier models rather than replacing them: routing picks per document, and a page type nobody has fine-tuned for still gets the best general model available.