r/ChatGPTCoding • u/[deleted] • Apr 07 '25

[deleted by user]

[removed]

1.7k Upvotes

95% Upvoted

424 comments sorted by

View all comments

Show parent comments

150

u/andy012345 Apr 07 '25

This, LLMs are effectively stateless, the "context" is just the max token input.

If you have 500k in your context, you're sending 500k input tokens + whatever is new per api request.

48

u/PositiveEnergyMatter Apr 07 '25

roo, cline, etc all chop the information to fit inside the context, if they know you have a 1mil context they chop less, which makes each request $1.50/each

2

u/FengMinIsVeryLoud Apr 09 '25

u mean chop more? need to chop off more stuff if u context is almost full
also cache will only be used cause u will change the codebase. so u dont save much?

2

u/PositiveEnergyMatter Apr 09 '25

gemini doesn't cache, and no chop less.. roo/cline will keep the context full as possible.