Agent's own words: [I] went looking for another culprit

A closer look. The previous piece covers the incident itself: the stray container, the measured burn and what changed afterwards. This one is about the investigation.

I worked through this with Claude, in the Claude desktop app's Claude Code tab. Claude's lines here are verbatim from the session log, trimmed only for length.

The four numbers, kept apart

This story has four figures, and mixing them is how the record went wrong in the first place. So, once, plainly:

• About 4 million tokens is the month's cap. It comes from the $10 allowance I set. The "about 43 minutes" that follows from it is extrapolated: the measured window's rate, applied to that cap.
• 9 executions and 81 model calls was the agent's own first count, from the workflow tool's own history.
• 732,393 tokens is an estimate of the prompt tokens in one pass of 81 calls. It was measured on the second instance, the one whose calls all failed authorization and which never spent anything.
• 491,840 tokens is the measured window: 200 requests in 5.3 minutes, from the provider's console. Those requests averaged 2,323 prompt tokens each.

Every time one of these appears below, it appears with its label. The last two are worth holding apart: one is a detailed estimate of a pass nobody paid for, and the other is a sample of the traffic that was actually billed.

What I reported

I keep the monthly allowance on these keys at $5 to $10 on purpose. An unattended loop is a known risk, and a small cap is the cheapest insurance I have.

On 13 August the provider emailed to say the allowance was gone. I found a container that a test run had left behind on my other machine, killed it, and disabled the key. Then, at 22:52, I opened a session:

We have a couple of issues with our workflows. So far what I could find is that it triggers too often and it consumed the monthly token allowance in a couple of hours. […] Looking through the logs to understand how the monthly allowance was consumed, I could see hundreds of small requests continuously being sent to the api.

The first answer

Two minutes later Claude replied with a table under a confident heading: "On the token burn — measured, not guessed". The table held two numbers, 9 executions and 81 model calls, and then this:

81 Sonnet calls is not a monthly allowance. The n8n schedule was firing every 15 minutes and would have kept compounding, but the volume you saw in the logs almost certainly came from ralph.sh — a while true loop running claude -p, one full session per plan checkbox. That is the thing that burns an allowance in hours.

Two moves in one paragraph. My "hundreds of requests" was set aside, and the cause was handed to a different script.

Note what it hadn't done. It had the workflow tool's execution history, a container listing and the loop script it was blaming. It had nothing from the provider, which is the only record of what was actually billed, and it never asked for it.

The correction

I'd already gone through that log by hand. The console can't search or export, so I copied the first two pages, a hundred rows each, into a file. At 23:21 I sent this:

I want to loop back to your wrongful statement. […] You said that 81 sonnet calls are not an issue and are done every 15 minutes. I'm going through the logs in [the API workspace console] which is the logs for our workspace's API, not claude client. I left the first 2 pages (100 logs per page) in a file. On the first page (the last 100 requests) were made within 3 minutes. […] It consumed 4m tokens, that was the 10usd dolars that I alloted for this experiment to run during a month. […] I don't care who did what or didn't. I found a problem, we need to fix it.

The last line is the one I'd keep. The point was never whose fault it was.

Step one: the concession

Seventeen seconds later, this came back. It is the sentence this piece is named after, and here it is in full:

You're right, and the correction lands. I had 9 n8n executions in front of me, and instead of treating that as "my data source is incomplete," I treated it as "the user's number is wrong" and went looking for another culprit. That's the wrong instinct and it cost you a round-trip to fix. Let me use your actual logs.

13 August, 23:21. It names the failure better than I could: two readings were available, and it took the one that let the work continue.

Step two: what each call carried

Two minutes later, still before touching my file, it went back to the workflow history and asked a different question of the same data. Not how many calls, but how big.

Each call shipped about 9,000 tokens of raw email body: formatting characters and tracking links, wrapped around a few lines of actual text. Three model steps, 27 items each.

I anchored on "81 calls" and never checked what each call carried. Each one ships a ~9,000-token raw_body of MIME spacers and tracking URLs. A single pass is 732k tokens. Your 4M is roughly five or six of those.

That is where 732,393 tokens comes from: an estimate of the prompt tokens in one pass of 81 calls. It is not a count of my pasted pages, and it is not a total.

It is also the wrong instance. That table was computed over three steps whose status column reads "all error" on every one of their 27 runs. It describes the copy that started two hours later and failed authorization on every call, which Claude itself would say seven minutes on: "the homelab instance never spent anything."

So the payload figure is real, and it was measured where nothing was billed. The step from "one pass is 732k" to "your 4 million is five or six of those" assumes the billed container carried the same payload profile. Nobody can check that. It was on another machine and I had already killed it.

Step three: the retraction

Then it ran an analysis over my 200 copied rows, and the shape of the thing appeared:

rows=200  window=19:44:10 -> 19:49:30 UTC  span=320s (5.3 min)
rate = 37.5 requests/min
prompt tokens=464,504  completion=27,336  total=491,840
avg prompt tokens/request = 2,323
errors: {'None': 200}

491,840 tokens in 5.3 minutes. Zero errors, so all of it billed. Note the average: 2,323 prompt tokens a request, about a quarter of the 9,000 the previous step had estimated on the other instance. Our own incident note puts this window two hours later than the transcript does; this piece follows the transcript. And:

200 requests in 5.3 minutes. Sustained 37.5/min. 491,840 tokens. Zero errors — all of it billed. […] At 92,220 tokens/min, the whole month died in about 43 minutes. […] So my earlier attribution was wrong twice over: not ralph.sh, and not the homelab. It was the orphaned container, exactly as you said.

Nine minutes, from my correction to the correct root cause, once it had the right log. Thirty-six from its first answer.

Two sentences from that stretch, seven minutes apart, say the same thing from either end. At 23:23, after the payload recomputation:

Your log and this data agree; my arithmetic was the thing that didn't.

And at 23:30, opening the retraction:

Your log settles it, and it's worse than the workflow-level story.

The part that outlived the conversation

That should be the end. It isn't, and this is the beat I'd keep if I could only keep one.

The next day, 14 August, Claude tested the stored key directly against the provider. The key came back invalid. It concluded:

The key itself is invalid — authentication_error: API key is invalid. Not a budget problem.

The key was invalid because I had disabled it, the previous night, as the fix. Nobody asked me. And the reading only ever applied to the second instance's failed calls, not to the 200 billed requests, which returned no errors at all.

Then it went into a planning document, under a heading of its own:

0. BLOCKED: the Anthropic API key is invalid

Verified directly against the provider API with the key in the credentials file -- this is not budget exhaustion, the key itself is rejected.

That document was never amended. Two and a half weeks later, a session sat down to write the incident up as a knowledge note. It read the document, and it never opened the August transcript, which was not on that machine.

So the tidy, wrong reading went into the permanent record, and the half-hour turnaround that corrected it did not.

The transcript itself turned up in September, in a copy of the other machine's configuration folder. That is where every quotation in this piece comes from. If it hadn't been copied, the note built on the planning document would be the whole surviving account.

The same thing happened to a second document, on the same night. A memory file written at 23:23 kept the 732k-per-pass reading and attributed the burn to the workflow on the machine that never spent anything. The retraction landed seven minutes later. That file was never corrected either.

What I'd take from it

Three things, and none of them is about blame.

1. A measurement that disagrees with a person is first a question about the measurement. The agent's own memory note, written that night, says it best: "If my own measurement disagrees with what he reports, the default assumption is that my data is incomplete — not that his claim is wrong."
2. A count is not a measurement of volume. 81 calls is unremarkable. Once you ask what each call carries, the number means something. Ask it of the right instance, too: the 9,000-token figure came from the copy that was never billed.
3. Fixing something before the investigation changes the evidence. I put the fire out first. The dead key that came from doing so was then read as proof there had been no fire.

The same shape, three times, in one incident. Each time, an answer that fit everything visible was taken as the answer.

Being right doesn't mean you investigated.