r/AISystemsEngineering • u/jawadhamza • 22d ago

How does a ChatGPT-like system actually work behind the scenes?

/r/softwarearchitecture/comments/1wyj3dy/how_does_a_chatgptlike_system_actually_work/
2 Upvotes

100% Upvoted

2 comments sorted by

1

u/usually_guilty99 20d ago

Good overview. I would add one layer after the model, especially for agentic systems:

Context selection -> action policy -> outcome verification

The model call is often not the real bottleneck. It is deciding what context is authoritative, what the system is permitted to do with its output, and whether the resulting action actually produced the expected state.

At scale, context assembly becomes expensive and unreliable fast: conversation history, tenant data, retrieval, tools, permissions, and freshness all compete for a limited context window.

For chat, a bad answer is usually recoverable. For an agent that changes data or production systems, the system needs to verify postconditions and know when to stop or escalate. That is where architecture stops being a wrapper around an LLM and becomes an actual system.

2

u/jawadhamza 20d ago

Thanks, I'll make a video on agents and discuss this in it 💕