---
title: "\"All pass\": a green run that threw away fourteen answers it had paid for"
date: 2026-08-16
project: "open tomato"
series: "Enough to Prove a Point"
part: 4
canonical_url: https://www.opentomato.io/diary/all-pass-green-run/
license: https://creativecommons.org/licenses/by/4.0/
---

# "All pass": a green run that threw away fourteen answers it had paid for

**TL;DR**
- I ran a scoring stage and reported "all pass". The engine agreed: the run's status was success.
- It had made fifteen paid model calls, received fourteen usable answers, dropped all fourteen, and stored one row. Where that one row came from, the record doesn't say.
- The code that dropped them was a guard built two hours earlier to stop paid work being thrown away.

I worked through this with Claude, in the Claude desktop app's Claude Code tab. There is no transcript left of this one, so Claude's lines here are reconstructed from the note we wrote at the time, not quoted from a log.

Same pipeline as [part 1](./2026-09-22-ten-dollar-allowance.md), a different stage. That piece is worth reading first, though this one stands on its own.

**Three words this piece uses.** The pipeline is a recruiter project of mine, built on n8n, a workflow-automation tool where each step is a "node" and a run is an "execution". It later became the project now called **bart**.

## What "all pass" was actually saying

The status wasn't lying. It was answering a question nobody needed answered: did the nodes finish?

The question that mattered was different: what did this run produce, and what did it cost to produce it? Nothing in the interface separated the two.

Three signals in that system report success under conditions I would not call success:

- **The run status.** It says every node finished. Nodes that are configured to degrade on error do finish.
- **The item count on a node.** It says an item came out. A degraded item, where the input passes through unchanged, is still an item.
- **A write node completing.** It says the write was attempted. The step that marks work as done sits downstream of it.

Each one is reasonable. Together they mean a run can be green, emit the expected counts, stamp its work as complete, and produce nothing.

## The two defects

**Defect 1: a contract widened in one layer and narrowed in the next.**

The output schema was deliberately made nullable, so that a model answering "unknown" for one field doesn't have its whole response rejected. That is a documented rule in this project, and a good one.

In the same change, the consuming code required that field to be non-null.

Neither layer is wrong alone. Together they mean the model can return a response the schema accepts and the consumer silently drops. No error is raised anywhere, because dropping an item is a legitimate thing for that consumer to do. It is what it does with genuinely failed calls.

The trigger was ordinary. My scoring criteria were still placeholder text, so the model correctly answered that fit could not be assessed against placeholder criteria, returned null for that one field, and scored the two it could.

A well-behaved answer, discarded by a guard, at full price, fourteen times.

**Defect 2: a text matcher for an error that arrived without text.**

The detector for systemic failure matched on the provider's own wording, captured from a real earlier failure. In the run that mattered, the error arrived as a bare generic string with the real sentence missing.

So fifteen consecutive failures in an earlier run did not trip it, and that run reported success having done nothing.

> **Me:** I just ran [the scoring stage], all pass, what's next on the pile?
>
> **Claude:** Status says success. The table says one row, from fifteen calls. The guard required a non-null primary score, and the model declined that one field while answering the other two.

The fix for the second defect keeps the text match and adds a rule underneath it: a batch where *every* call failed is one broken credential, not N bad rows. That is knowable without reading any message.

## The guard that inverted its own purpose

This is the sharpest thing in the story, and it is not about the model.

Two hours earlier we had found that a provider refusal killed a run, billing the successful calls and saving none of them. We built a guard: partition the successes from the failures, and persist the successes before any halt. We verified it in a shell against a captured real error payload. We shipped it.

Its first real run discarded fourteen paid answers.

A safeguard is a component. Shipping one without checking what it produces is shipping unverified code with extra confidence attached. It deserves more scepticism than ordinary code, not less, precisely because its existence is used as evidence that the risk is handled.

## The week's other misreading

Worth recording, because it cost more than this defect did.

I reported a scheduled job firing three times in two minutes. The execution list showed exactly that. About two hours went into duplicate-detection safeguards.

Reading one more column showed the bursts were command-line runs from a provisioning script two days earlier. The scheduled trigger had fired once a day, correctly, every day.

The list was accurate. The interface didn't make the distinguishing field prominent, and I didn't ask it to. We kept the safeguards, because nothing in the tooling could have told me that, which was its own gap.

## What it cost

- About 29 billed model calls across two runs, producing one stored row. Small in absolute terms. The rate is the finding: roughly a 3% conversion from paid call to retained result, reported as success.
- The cost recurs. Because the fourteen were never stored, the selection query returns them again next run.
- Two hours on a duplicate that didn't exist.

## What changed

| Safeguard | What it prevents | What it costs |
|---|---|---|
| Persist before halt: successes are written before any stop | Paying twice for completed work | Nothing. A pure ordering change. |
| A declined field is an answer, not a failure: nulls reach the column as nulls | Defect 1. Aligns the consumer with the schema it already had. | Nothing. |
| Shape-based detection underneath the text matcher: all-failed halts regardless of wording | Defect 2, and every future change to a third party's error format | Nothing, and it removes a dependency on a contract nobody controls. |
| A test that drives the shipped artefact against every shape it can emit | The shell-verification gap that let both defects ship | About forty lines. This is the one that would have caught both. |
| A ledger row for the halt itself, not only for calls | Runs that stop without leaving a trace of why | One row. Makes the cost reconstructable afterwards. |

And a reporting rule, which is the cheapest of the lot: **for any stage that costs money, the completion report states calls made, items retained, and the ratio.** Not the execution status.

## The close

An operator saying "it worked" is relaying the interface's summary, not the outcome. They are not over-claiming; they are repeating what they were shown. The obligation to check the artefact sits with whoever knows the artefact exists.

And when the summary and the artefact disagree, the interface usually isn't wrong. It is answering a narrower question than the one you have. The fix is rarely to distrust the tool. It is to know which question it answers, and to ask the other one separately, every time.

**One thing to keep straight about that table.** It describes the project as it was fixed at the time. bart is a later fork of it, and not every fix travelled across.

**At the time of writing, some of the planned safeguards are still in development.** In bart today: there is no attempt counter to retire a failing item, no halt on authorization or quota errors, no `restart: no` on two services and no container labels, the panic script stops containers by project rather than by label, the intake path still makes one model call per document, and the storage-path setting accepts only one spelling.

---

*Originally published in the [open tomato dev diary](https://www.opentomato.io/diary/all-pass-green-run/), licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).*