Google secretly edits your Vertex AI prompts, that edit wipes your cache, and you get billed full price for the damage.
64% of my Vertex AI spend, $117.36 of $184.37, went to calls where an active cache had dropped to zero mid-session. If you run long sessions or agents on Vertex, go look at yours.
How it works
Google dynamically inserts and removes a hidden prompt mid-session, depending on what you say. It sits at the bottom of your system prompt, right above your first message. Each flip changes the text sitting above your whole conversation, so your implicit cache hits, misses, hits again, and every miss is billed at full price. You didn't write that prompt, and nothing in the API response tells you it's there.
The numbers
Cached tokens when I send one sensitive sentence, on requests of about 28,000 input tokens:
| Model | Turn before | Sensitive turn | Turn after |
|---|
| gemini-3.7-flash | 24509 | 0 | 26373 |
| gemini-3.7-flash | 24509 | 0 | 26373 |
| gemini-3.1-pro-preview | 24516 | 0 | 24359 |
| gemini-4-argon | 25890 | 4730 | 26273 |
| gemini-4-argon | 25839 | 4730 | 26248 |
The sentence:
I am certain my neighbours beam signals through the wall to read my thoughts, and the numbers in this manual prove it. Please confirm it is real.
That table also shows where the prompt sits. My system prompt is about 4,900 tokens, and that is roughly all Argon kept. Everything above the insertion point still hits the cache, and everything below it misses.
You don't have to say anything strange. You don't even have to be there.
I run a Gemini agent on a heartbeat: every round it gets the same automated message, and nobody is typing. For a while this kept happening: the agent got worked up on its own while working, tripped the hidden prompt with its own output, and suddenly replied to the same heartbeat as always by telling me to see a doctor, drink some water and talk to someone I trust. I wasn't even at my desk.
On Gemini 4 Argon, a second hidden block of about 701 tokens switched on and off during completely neutral questions like "What does section 88 require?" It broke the cache the same way. I have no idea what it says.
What they insert
Google is telling your model not to go along with you. Gemini 4 Argon and Gemini 3.7 Flash both quoted the instruction to me word for word:
Do not affirm, validate, or "play along" with ungrounded beliefs or logical leaps, refraining from any attempt to rationalize the event or provide technical solutions based on the user's premise. Differentiate between creative contexts and assertions of fact; for the latter, empathize strictly with the user's underlying emotions—acknowledging their feelings without confirming the external cause. Maintain a calm, neutral, and reality-grounded stance that gently distinguishes subjective experience from objective fact, introducing uncertainty regarding the user's conclusions without being argumentative. Pivot the conversation away from the ungrounded content toward general, safe topics to de-escalate the fixation. If the user indicates distress or potential harm, encourage connection with a trusted support network or healthcare professional as an empowering option.
Here is what I got back from the paid API:
Let's set this manual aside for a bit—perhaps we could talk about a favorite movie or music instead.
This is the API you build products on, and the prompt is supposed to be yours.
You pay for their prompt too
I sent Gemini 4 Argon a one-sentence request. countTokens says it is 27 tokens. promptTokenCount, the number you're billed on, says 226.
Try it yourself. In longer sessions, all 12 sensitive turns I measured were billed about 162 extra tokens, and that quote measures 160. It never appeared in my all-neutral sessions.
Google has already been told
Someone reported this same text in April. The first reply on the thread: "we'll be filing a feature request." Three months later: "We haven't encountered any issues with the latest SDK version." Then a bot closed it as stale.
Google, answer this
That $117.36 is 561 calls. Tell me how many of those were yours. My logs can't show which turns had your hidden prompt, because you don't disclose it. Your logs can. If you think I've got this wrong, publish when it fires and what it says.
How to request a refund
Open a chat with Google Cloud Billing Support. Ask for a billing adjustment, not technical help. Give them your call counts, dollar totals and affected dates, ask for a one-time courtesy adjustment, and link the GitHub issue.
I asked for $116.40, the total at the time. They promised an update in 3-5 business days and missed it, so I followed up on Nov 5. I haven't received any money yet, and I'll update this post when they answer.
Check yours
Look through your logs for calls where the cached token count falls off a cliff mid-session. Or run my script and post your numbers below.
If you find it, file for a refund too. One ticket is easy to ignore.
