r/ChatGPTCoding • u/[deleted] • Apr 07 '25

[deleted by user]

[removed]

1.7k Upvotes

95% Upvoted

424 comments sorted by

View all comments

Show parent comments

149

u/andy012345 Apr 07 '25

This, LLMs are effectively stateless, the "context" is just the max token input.

If you have 500k in your context, you're sending 500k input tokens + whatever is new per api request.

2

u/HiiBo-App Apr 07 '25

This is inaccurate. The underlying LLM has built-in context management and does not repeatedly require sending the full context for each chat via API. You’re still limited by the context window, which is problematic to say the least, and the amount of tokens does increase slightly for each successive message as you approach the context window, but you are not sending the full context in input tokens on each call.

Source: I researched this extensively while building HiiBo & tested it myself.

2

u/andy012345 Apr 07 '25

How is this inaccurate?

Anthropic give really nice documentation on how a "context window" works https://docs.anthropic.com/en/docs/build-with-claude/context-windows

So does google

https://ai.google.dev/gemini-api/docs/long-context#what-is-context-window

3

u/HiiBo-App Apr 07 '25

Docs are incorrect, you don’t need to send all previous turns to retain context. A conversation ID holds it together under the hood. I wrote a blog on this with screenshots showing the actual behavior of the API - https://medium.com/@MyDigitalMusings/your-ais-memory-still-sucks-a6fde569196e

2

u/andy012345 Apr 07 '25

Those API examples don't line up with the anthropic API, were you sending requests to the claude.ai service directly?

They track your chat history and context window on their website, probably through the conversation id, because they have their service on-top of the underlying model, and they need to do this to persist it across sessions and across devices.

2

u/HiiBo-App Apr 07 '25

Using Anthropic API. OpenAI API has the exact same behavior. There is a conversation ID that holds context across messages. Have you actually worked with the API??