A $10 allowance was the only thing watching: a test container emptied it in an estimated 43 minutes

I worked through this with Claude, in the Claude desktop app's Claude Code tab. Claude's lines come from the session log, trimmed only for length.

Three names, once. The project that burned the money is open-for-work, a job-seeking pipeline I built on n8n, a workflow-automation tool. It parsed job postings, researched each company's tech, and read the posting itself for red flags. On 19 August I started porting it into a new project, agentic-research, and bart on GitHub is a fork of that port. From here the pipeline is open-for-work and the port is agentic-research.

What was left running

The stack was meant to live on my home server. A test run brought it up on another machine instead, as a container, and nothing took it down afterwards. The container was headless: it had no output and answered nothing, even on the machine it ran on. When I finally went looking, I had to edit it and expose a port just to log in and see what state it was in.

Six ordinary decisions kept it alive and busy, and each one is defensible on its own.

1. The container's restart policy was "unless stopped", so it came back after every reboot.
2. The deploy script published both workflows as active.
3. The credentials file staged the real API key.
4. A schedule trigger fired every 15 minutes.
5. The smoke test wrote a 90-row test fixture into the live input folder, and only cleaned it up at the start of the next run.
6. Nothing outside the container could see it was there.

Claude: activate-workflows.sh publishes both workflows active, credentials.json stages the real key, smoke.sh leaves the 90 rows in the drop folder, and the schedule trigger fires every 15 minutes. Loopback-only binding with no route meant nothing showed it was alive. It was a self-restarting, self-triggering, invisible spender.

Every one of those points the same way, keep going, and nothing in the stack was there to say stop. I'll call it the test container from here, because that is what it was. The risk it shows isn't that production is full of hidden zombies; it is that a one-off, in a contained environment, still did real damage.

What it cost, and what is measured

I keep test keys on a $5 to $10 monthly allowance, and I'd encourage anyone to do the same. A tiny allowance on a test key is the cheapest insurance there is against an unattended loop, and it is a deliberate strategy in our own projects. This incident is the case for it.

The $10 allowance is about 4 million tokens. That figure comes from the provider's console report, the same page I copied the logs from. At the time of the incident that console had no advanced search, no export, and no spending report I could pull, so I picked a few pages of requests by hand as a reference point. They were timestamped, and they mostly averaged 2,000 to 2,500 tokens each. Later, in the session, that sample was analysed; the clock times are UTC, and our own incident note puts the same window two hours later, so this piece follows the transcript as the primary record.

rows=200  window=19:44:10 -> 19:49:30 UTC  span=320s (5.3 min)
rate = 37.5 requests/min
prompt tokens=464,504  completion=27,336  total=491,840
avg prompt tokens/request = 2,323
errors: {'None': 200}

So: 491,840 tokens in 5.3 minutes, at 37.5 requests a minute, with zero errors, which means all of it was billed. Two workers were running at once, because 126 of the 200 requests started while another was still in flight.

The 43 minutes in the title is an estimate. It takes that window's rate, 92,220 tokens a minute, and applies it to a cap of about 4 million. The 200 rows are a sample of a run whose true size nobody knows, since the console couldn't be queried, so what the sample establishes is the rate, and at that rate a month lasts about three quarters of an hour. Other records round it differently: our own documents say "43 minutes" in some places and "under an hour" in others, and the note I wrote that night says "a few hours". This piece uses the extract's arithmetic.

Why a pass was so expensive, and which pass that was

The pipeline handled 27 items, and each item went through three model steps, which makes 81 calls per pass, with retries set to three. One pass was measured at 732,393 tokens, because each call carried the raw body of an email, about 9,000 tokens of it, mostly formatting characters and tracking links wrapped around a few lines of real text.

That figure needs three labels, and the piece would be wrong without them.

1. It is an estimate, from a column headed "estimated prompt tokens".
2. It is prompt-side only. It doesn't include what came back.
3. It was measured on the other instance, the copy that started two hours later and whose calls all failed authorization. That copy never spent anything.

So the 9,000-token payload is real, and it was measured on a pass that was never billed. Whether the test container that was billed carried the same payload is an assumption, and nobody can check it now: that container sat on another machine, and I had killed it before the session opened.

So we need to look at these numbers side by side to make sense of them. Across the 200 billed requests the average was 2,323 prompt tokens per request, about a quarter of the 9,000 the pass-level explanation leans on. What that means, plainly: the detailed explanation of an expensive pass comes from a copy that cost nothing, the actual billed traffic looks about four times lighter per request, and there is no record that connects the two.

The alarm that wasn't one

Nothing in the system reported any of this. Every request succeeded, so no error fired; every node was set to continue past failures, so the runs reported success; and the test container was headless, so nothing could watch it.

The provider's budget email was the loudest signal, and it was thunder arriving long after the lightning. It took minutes, maybe an hour, to reach me. Then it had to be read, and then it took a while longer to work out what was going on. By then the money was gone, and the run was still going: when the cap was reached the billing stopped and the workflows kept executing for another two to three hours, still unbilled and still unwatched.

Billing was the loudest consequence. Underneath it, junk rows kept accumulating in a real database, a prompt-injection test fixture kept being ingested as genuine records, and the same items kept being fed back through. Left unrestricted, the run continues and the first signal is something other than an invoice.

The catalyst

Here is how I'd rank the causes now, having looked at it for a month. The queue that never drained is the real catalyst: a step that marked items as processed was failing, so the same items came back on every pass, and with retries stacking three deep the run had no way to end on its own. The cap was the circuit breaker. The 15-minute timer was a knob. The hidden container was a missing criterion for whoever wrote the test setup. And the persistent test elements left alive, with nothing to scrape them away, were the real issue.

In a way the stacking was a fortunate accident, like the amphora someone forgot with water and bread in it and came back to find beer. If the retries hadn't piled all the requests up at once, the leftover tests might have eaten the allowance quietly, week by week, and never been noticed. That is the argument for the $1 test key that Claude proposes below, over a $10 one.

Two of my own figures don't have a record behind them, so take them as recollection. I remember thinking $10 could have lasted the month if the queue had drained. And I remember the real deployment costing $1 to $2 every other day.

The gap underneath all of it

When I looked at why this was possible, the answer was a missing sentence in how the work was described. The instruction set was good at making things and weak at checking them. A task read, in effect, "test this like this". It never read "make sure the test finishes, and clean up afterwards". Every instruction was about making something and none about ending it.

What changed, and where it lives now

The session that investigated it closed the holes in open-for-work the same night, in one containment commit on 14 August:

• Restart policy set to "no" on both services, and a label on every container so a stray can be found without knowing its name.
• A panic script: one command that stops the local containers, deactivates the remote workflows and clears staged fixtures.
• A trap in the smoke test, so fixtures can't outlive the run that staged them, including when it is interrupted.
• An audit script that treats the repository as the allowed set and names every workflow that isn't in it.

That trap has become standard in our pipelines and loops since: it makes sure everything is unmounted or destroyed if a setup, test or step fails. Today it lives in open-for-work's smoke script and in a sync script in the homelab. agentic-research has no trap; its equivalent is a test-stack script that refuses to start over leftovers, tears down by label, and fails if anything survives.

Then the honest note about what all that doesn't fix, in Claude's words:

The hole that's still open, and it's the important one: a stray container can still reach a live provider key. Everything above limits how long something runs, not how much it can spend. A separate test API key with its own low budget (~$1). Then a stray is structurally incapable of burning the month, regardless of what else fails.

The fixes above are open-for-work's. agentic-research was a rewrite: every script was redone between 28 August and 14 September, and the port's spec ordered the containment posture kept. So the question of why some fixes aren't in the port has three different answers. Three had no subject there, because the trap, the fixture clearing and the storage-path spelling all protected a drop folder the port dropped with its intake. Three were redesigned in place, with comments saying why: the restart policy, the labels, and the panic script. Two were not carried, and nothing records a decision about them: the attempt counter and the halt when every call has failed.

The attempt counter is worth a sentence, because there are two of them and I had them confused. The one I remembered is a generation counter from a runaway-control spec, ruled unnecessary because the research graph can't loop, replaced by an invariant that checks exactly that, and shipped in a September PR. The one that retires a failing work item is a different thing: built in open-for-work on 18 August as a migration, never ported, in no spec or issue, and its failure mode is live today.

At the time of writing, some of the planned safeguards are still in development. In agentic-research today there is no attempt counter to retire a failing research item and no halt on authorization or quota errors. Three others were redesigned rather than copied: the spender runs with no restart policy by design while the two databases restart, there are no declared containment labels because the sweep uses the compose tool's own project label, and the panic script stops by project and disarms the remote instance without hunting by name or mount.

The close

The cap is the only reason this incident has an end time, and it was only ever a spend limit that happened to sit in the path. Take it away and the test container restarts forever, the schedule fires every 15 minutes, the queue never drains, and every run reports success. There is no answer inside the system to the question "when would this have stopped?"

The first investigation of this got the cause wrong, conceded 27 minutes later and retracted it in full after 36. That is the next piece, and it is the reason this series exists.

Moral of the story

The repeatable mistake here was not stating the obvious. When you hand work to an agent, "test it like this" needs "and make sure it finishes, then clean up" after it, every time, however obvious it feels. The fault was a missing sentence, so in a system where more than one actor is working and each is checking the others, assigning blame is not a useful output; adding the sentence is.

---

How this was made

• Review rounds: 1 with coco, plus 1 pass of inline comments from Marcos.
• The human: ran the session the piece is about; pulled the numbers from the provider's console by hand; corrected the draft with 17 inline comments, roughly 900 words of them, including the two analogies used here.
• The editor's comments steered what the piece is about more than once: from the cap as the cause to the undrained queue, and from a list of bugs to the missing sentence the moral rests on.
• paco was used for:
  • first draft from the knowledge unit and the recovered session log
  • restructuring after coco's review and after the editor's comments
  • the two fact-check passes against the origin and the port
• Rough split, an estimate: what the piece says was decided by the human, about 70%; how it says it was phrased by the agent, about 80%.