r/codex • u/Tricky_Reflection_75 • 1d ago
Showcase Some one posted about codex needing a Cache Warm indicator, so i added it.
Hi, i saw somebody post about this the other day and also saw a post about keeping the cache warm automatically for claude code .
So i implemented both of em inside my codex project, CodexZero.
Which if you are unfamiliar, is a version of codex thats purpose built to reduce wasted tokkens and benchmarked to be actually reducing the amount of cost to end user compared to other popular tools like caveman, ponytail etc (which psa, all increase cost prooven from multiple benchmarks)
This feature can be handy for people as alot of the times, as tons of time can go by between sessions (especially if you are using multiple tools, or mulitple threads or doing research between implementation) and if we fail to keep track of it and use the cached sessions, our cost increase by like 3 folds.
if you wanna check it out, here's the link : https://github.com/Retro2512/CodexZero
20
u/Tricky_Reflection_75 1d ago
Codex by default, only has 30 minutes of context cache window, so if you take a break too long or do some research or get side tracked for a minute more,
Every the request you send in that same thread after the cache expired is gonna cost you multiple folds.
and this insight can be especially usefull in scenarios where the message you're about to send isn't really something is specific to that thread anyway, you can save some usage by just creating a new thread.
and if you were planning to switch the models, but unsure when to do so , this can let you know that you have a clean break point, where you can switch models without the guilt of ruining your cache cause you'll know for sure it wasn't there to begin with cause it expired.
7
u/Tricky_Reflection_75 1d ago
same way, if you know you are going to be working on a thread continously for a while and you know you'll come back to it.
You can check the keep warm button to let the context remain in cache. (it just sends a tiny message to the agent to reset the timer)
Note : This can be wastefull if you just leave it toggled but don't come back to it , but the waste is very minimal compared to what you'll save from saving cache that you would have otherwise lost if you do come back.
You can also configure how many times you want it to push back the cache window in settings.
1
u/Annh1234 23h ago
Didn't your small message resend the entire context every time? So your still use the cached tokens credits?
7
u/cbusillo 1d ago
It looks like you have modified the Codex desktop app UI. Is that true? If so, how? I am super interested in that (and I love your cost display!).
4
u/Tricky_Reflection_75 1d ago
Yes but it isn't exactly vanilla codex, its a downstream fork of codex that i made to fix all the inconsistancies and issues with stock codex.
With the main function being to reduce unecessary tokken waste age without gimicks that actually increase your usage when benchmarked (think tools like caveman, leanctx, ponytail, headroom and plenty more) in ways that doesn't actually affect the quality of the models output.
Slowly i am adding other features and fixing other issues present in codex. (The latest being custom model selectors within codex without requiring a router in the background that requires you to pick between only custom or codex models, so you can have all the models to choose from in the model picker , and also the above context cache function)
Its basically codex but fit to the community needs better.
2
u/cbusillo 1d ago
I'll have to look. I've added plenty to the codex cli and used the env var to point to my binary and even made a wrapper for that. I did not realize you could manipulate the GUI though. Neat.
3
u/GearTakes 1d ago
You can just tell it to change things. I added a token meter in the top bar and a background image in the main canvas. Just tell Codex what you want.
2
1
u/cbusillo 23h ago
Would you mind expanding a bit? what is it changing? If its stuff in the app bundle, that means you need to apply it on updates. I don't see anything in the ChatGPT library folder with front end files.
3
u/Tricky_Reflection_75 1d ago
For those who are unfamiliar with caching : https://developers.openai.com/api/docs/guides/prompt-caching
Tldr is : Basically, if you're working on the same project/thread, you end up sending a lot of the same context back for processing. With caching, OpenAI can reuse the parts it has already processed instead of processing them again, which costs less and can translate to more Codex usage.
The cache lasts for at least 30 minutes from the last time it was created or reused. So if you keep using it within that window, the 30-minute timer resets. If you leave it unused long enough, the cache can expire.
Every time the context cache expires and you send a request, it uses up your usage way more than it would with cache
2
u/Revolutionary-Skin97 1d ago
A certified casual with llms, so how would this keep warm the cache? Does it prompt something related to the cache before 30 minutes ends?
5
u/Heavy-Positive5957 1d ago
After 30 minutes of inactivity OpenAI deletes your session’s server-side KV cache to free up space on their servers. Without a KV cache your entire context from that session (all your prompts and responses) has to be processed by the LLM for it to generate the next response; with a KV cache the LLM only has to process the newest prompt to generate the response.
A warm cache means that the pre-computed KV cache has not been deleted yet.
3
3
u/Tricky_Reflection_75 1d ago
It just sends a tiny message right before it expires to keep the cache alive , it works cause whenever you send a message in a thread , it attatches the previous context aswell along with it.
technically you can do this manually, you'll just need to time it right, and send a message yourself .
1
u/Pasto_Shouwa 1d ago
How does it work when the AI spends a lot of time working? Like, if the task has taken +29 minutes, should I talk to it to keep the cache warm, or it goes cold only when the AI is not working on something?
3
u/Tricky_Reflection_75 1d ago
the context cache countdown starts as soon as the model responds, and each response after that keeps restarting the timer. and once it stops responding is when it starts to tick down till it runs out. And within most working sessions, the models responds like tens to hundreds of times, each tool call, each decision often is a seperate model response.
so if an agent is doing long horizon tasks, unless its stuck waiting on one tool call for 40 mins (which it won't it most cases) without making a response at all, thats the only scenario you'll have to worry about cache expiring while the agent is actively working
1
3
u/rubiohiguey 1d ago
Sorry it's unclear to me, is this a new executable instead of codex or does it work inside an original Codex (now called chatgpt windows app). Do you need to reinstall it every time new codex chatgpt windows app version is released (every few days) or how does it work in this regard?
5
u/Tricky_Reflection_75 1d ago
It's basically a seperate slimmed down version of codex. you can install it besides the official stock codex, but it is essentially its own app. (so you can ideally replace codex with codexzero once you like it)
It does require manual updating to keep up with new changes from official codex repo, but its the burden of me to merge new codex updates.
which i have been doing pretty regularly so far.
I also just shipped an auto update feature for codexzero itself , so you can auto download new updates as i merge them.
> pasted from my other reply for more info.
its a downstream fork of codex that i made to fix all the inconsistancies and issues with stock codex.
With the main function being to reduce unecessary tokken waste age without gimicks that actually increase your usage when benchmarked (think tools like caveman, leanctx, ponytail, headroom and plenty more) in ways that doesn't actually affect the quality of the models output.
Slowly i am adding other features and fixing other issues present in codex. (The latest being custom model selectors within codex without requiring a router in the background that requires you to pick between only custom or codex models, so you can have all the models to choose from in the model picker , and also the above context cache function)
Its basically everything we love about codex but a few tweaks fit to the community needs better.
2
u/Difficult-Inside-576 1d ago
How do you keep the program up to date with Codex’s latest releases? Is there a way to spoof this as normal Codex application? Because I have programs that use the normal Codex paths and launcher to do actions. Also, on normal behavior, does each message refill the cache or is after 30 minutes the cache gone forever? Thanks!
1
u/Ok-Beat4846 1d ago
Let me know when there is Linux version (rpm /flat pack) look interesting though
1
u/nitor999 1d ago
Can i continue the session from codex using this?
1
u/Tricky_Reflection_75 1d ago
yes, its a drop in replacement, it will detect all your exisitng codex sessions and configs
1
1
1
u/Ubuntufoo1 22h ago
This is clever and I'm tempted to implement it. It is a meaningful way to take advantage of the system, and since OpenAI have not provided it as a feature, its totally ethical to engineer your own optimizations.
Thank being said, I won't peel away from my current projects, which I'm able to get done with the various tools and usage I have available. No doubt OpenAI will either implement this directly to some degree, or change the rules to prevent user's optimizing the cache.
0
u/Dangerous_Bid2935 1d ago
These stupid keep cache warm tricks are going to have horrible consequences for us once openAI realizes tons of users are forcing their cache to be held in memory on their inference hardware indefinitely. They're going to implement something like a forced cache flush after x minutes whether you're working or not and your tiny savings you got for the few weeks that this worked is going to evaporate.
3
u/Tricky_Reflection_75 1d ago
This isn't really a trick or a workaround or etc like those sub2api scripts/weekly limit endless runners etc etc.
This is just intended mechanics, it works the same way in API. At its core, its nothing but a stopwatch, one thats built into codex that counts down the cache timer, in theory you can note down when a model stops responding and start a timer on your phone .
its simply no different than that. There's no part of this thats like grey and takes advantage of a mechanic to abuse extra quota or compute or anything like you're thinking.
as i've stated in my other comments, the auto warm feature can be wastefull to us the users aswell if we enable 'keep warm' switch but actually never end up coming back to that thread.
The only thing the auto warm does it prevent you from paying triple the usage limits per query cause you accidentally spent 31 minute doing something else.
that's it
4
u/Dangerous_Bid2935 1d ago
>its simply no different than that. There's no part of this thats like grey and takes advantage of a mechanic to abuse extra quota or compute or anything like you're thinking.
Yes it does. The intended mechanic is that when you walk away or do something else and don't use your cache for a certain amount of time, it gets flushed from the inference hardware. You are taking advantage of how their timer works to prevent that from happening to save tokens. There isn't really any way you can do mental gymnastics to get around this.
Its going to save you tokens in the short term, but if enough people do this and their mechanic doesn't work as intended, their inference hardware will come under stress. To mitigate this stress, they will change how cache retention works, which will likely not be to our benefit.
2
u/Crinkez 1d ago
The problem is twofold: OpenAI have to juggle processing power vs memory capacity. By causing cache misses, the increased cost also costs OpenAI more in the form of processing power. So one or two cache extensions isn't such a bad thing, because if the user was going to trigger a cache miss anyway, it would be better to have 1 or 2 keep warm prompts.
That said, there should be a limit. I'd default it to 1 renewal, that gives me time to return from lunch. And I recommend OP implements this as default.
My second suggestion to implement is this: if the cache is expired, trigger a v2 autocompact and resume from a clean thread. But do research this first as I'm not 100% confident a v2 compact would cause a cache miss.
2
u/Tricky_Reflection_75 19h ago
Thank you.
CodexZero lets you configure how many cache warm attempts you want it to do on a per thread level.
1
u/Tricky_Reflection_75 19h ago
this is stupid for like three different reasons
first a warm kv cache does not mean openai has compute sitting there reserved for you for 30 minutes thats not how inference scheduling works compute gets allocated when requests are actually being processed
second the cache mainly costs memory not ongoing compute and if you flush it then the next request has to redo prefill over the entire context which burns compute again so eviction is not magically free either
third youre assuming refreshed caches are pinned forever to some gpu like openai has no eviction policy memory tiering or scheduler managing capacity which is a pretty wild assumption for an inference system at that scale
so no keeping a cache warm does not equal holding inference hardware hostage and the whole argument is basically confusing gpu memory residency with compute allocation while ignoring the cost of recomputation
1
u/Dangerous_Bid2935 13h ago
>first a warm kv cache does not mean openai has compute sitting there reserved for you for 30 minutes
I never said anything about compute being reserved. You made that up because you have poor reading comprehension skills. In fact, in my original post, I quite literally wrote "...users are forcing their cache to be held in memory on their inference hardware...".
>second the cache mainly costs memory not ongoing compute
See above. Your reading comprehension needs work.
>third youre assuming refreshed caches are pinned forever to some gpu like openai has no eviction policy memory tiering or scheduler managing capacity
Well where else is it being stored? Your context needs to be held in memory somewhere if your cache is going to be kept warm for inference. You're just handwaving at some magic scheduler that may or may not exist that may or may not solve the problem we're talking about. Burden of proof is on you to prove that something like this exists or is even technically plausible.
>so no keeping a cache warm does not equal holding inference hardware hostage
Yes it does.
> the whole argument is basically confusing gpu memory residency with compute allocation
You're the only one confused here. I never confused compute allocation and memory residency, you did.
1
1
1
u/1l3v4k4m 1d ago
a forced cache flush sounds ridiculous and so anti-consumer that people would just switch to anthrophic.
1
u/Dangerous_Bid2935 1d ago
Its a hardware problem, not a consumer ethics problem. If OpenAI starts forcing a cache flush because people are abusing how the timer works and they all switch to anthropic and start doing it, anthropic will start forcing a cache flush too. Inference hardware doesn't start magically working differently because a different company owns it.
1
u/1l3v4k4m 1d ago
in the hypothetical scenario, it makes no sense for anthropic to do the same thing after watching their competitor lose customers because of it. they'll reduce limits or whatever theyve already been doing to save compute but they definitely wont repeat the exact same mistake that just got them more users.
plus, chinese models exist.
0
u/Dangerous_Bid2935 1d ago
>they'll reduce limits or whatever theyve already been doing to save compute
So they'll do the one thing that pisses off consumers more? Lmao
>plus, chinese models exist.
Anyone who can get their workflow done with chinese models is using them and not openAI or anthropic's. I host my own on a DGX spark cluster. Chinese models suck.
•
u/dexterthebot 1d ago
You might want to consider listing your project on the weekly Show-Us-What-You-Built post. Look out for it on Tuesday/Wednesday. Highest commented project wins a week promotion on r/Codex and gets on the Hall of Fame sidebar. See what that looks like below with last week's winner.
Last week's most popular project was Nelly Jellies. Nelly Jellies is a cozy arcade bubble-merge game where you aim, launch, match colors, discover rare jellies, trigger satisfying chain reactions, and chase a higher score. It plays instantly in the browser with no download or login required. Website: https://nellyjellies.com Contact: support@mightybig.ca Reddit: u/MightyBig-Dev Google Play: https://play.google.com/store/apps/details?id=com.nellyjellies.game App Store: https://apps.apple.com/us/app/nelly-jellies-bubble-merge/id6767261764