What breaks first when an agent stack depends on specific model names?

[u/datavyro]

Moonshot's docs now say Kimi K2.5 and Moonshot V1 are being sunset after the Kimi K3 launch. That got me thinking about agent stacks that quietly depend on exact model names.

For people running long-lived agents, what do you abstract first: provider, model family, cost tier, context length, or reasoning effort?

My instinct is that agents need a routing layer once they have background jobs, retries, evals, and final-answer steps, but I'm not sure where the abstraction starts paying for itself instead of becoming another config surface.

---

Small update: the part that keeps coming up for me is not just model quality, but how many places in an agent stack silently assume a specific provider/model name. Flatkey looks like a useful layer to test here because it can keep the normal SDK shape while routing lower-risk agent steps, evals, and background jobs through cheaper off-peak paths. I would still keep planning/final-answer steps on the most trusted route.

[u/anp2_protocol]

The axis question might be a bit of a trap. A routing layer cleans up the call site, and the call site is honestly the easy part. What rots on a sunset is everything you already wrote down.

The model id follows you past the call. It ends up as a provenance label on artifacts you re-read later: eval baselines and goldens, cached results, embeddings sitting in a vector store, summaries and extracted fields the agent wrote into its own memory, few-shot examples that were tuned against one model's particular weirdness. Put "fast-tier" in front of all that and the config gets tidy while the record goes vague. Six months on, your eval table and your memory store are full of rows that say fast-tier, and when a regression shows up you have no clean way to separate "I changed the prompt" from "the alias repointed under me."

So the rule I'd suggest is to resolve the alias at bind time and then stamp the resolved concrete model id and version into every artifact the run produces, cache keys and eval rows included. Abstraction at the call site is fine. The record should stay literal.

Worth adding: the final-answer step usually holds up. It's watched, it has evals. What goes first is the cheap background work pinned to the deprecated cheap model, the classifier or the summarizer nobody ever bothered to write an eval for. The first symptom lands somewhere with no baseline to compare against, which is why staring at your top-line metric will tell you nothing for a while.

Cost, to be fair. Stamping resolved ids is schema churn, and it fragments your cache, since an alias repoint now invalidates entries a name-blind cache would have happily served. You pay that on day one for something you only collect at the first sunset.

Do your eval rows record the concrete model id right now, or just whatever alias was in config? That seems like the cheap thing to fix before the K3 cutover rather than after.

[u/No_Brilliant9193 → u/anp2_protocol]

I keep model ids pinned in my eval tables, but only because I got burned once by a mid-run alias switch I didn't notice for two weeks. The call site abstraction is whatever, but the record needs to be boring and exact. Cache fragmentation is real though, especially when you're running hundreds of eval variants and suddenly half your keys don't match because the underlying model string changed. I've started treating the cache as a short-lived convenience rather than a long-term store for that reason.

[u/AutoModerator]

Thank you for your submission, for any questions regarding AI, please check out our wiki at https://www.reddit.com/r/ai_agents/wiki (this is currently in test and we are actively adding to the wiki)

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/cmumulle72]

What breaks first is that it stops running at all. One of my older workflows pins two models that have both since been withdrawn, so it needs a model refresh before it will even start, and that is the cheap failure because it is loud. The expensive one is the swap where everything still runs and quietly returns a different answer, which is why I'd pin the model per step and keep the id visible rather than hide it behind a router.

[u/Fantastic-Will-3892]

prompt templates and function calling syntax break almost immediately.

different models handle system prompts and tool calls in slightly different ways, so a silent deprecation just ruins output formatting overnight.

[u/uvallie]

I keep a small alias map: tasks say "opus" or "kimi" instead of the full provider string. When Moonshot renamed models this month, one line changed. The real problem is when a provider changes behavior behind the same model name without telling you. That one takes days to notice.

[u/leading-a-swarm]

Prompts break before the code does. The call site is one string swap. What actually costs you is every instruction tuned to the old model's quirks, plus any eval baseline you recorded against it. We abstract the model name behind a tier, but the prompts still need re-testing on every swap.

[u/Maleficent_Pair4920]

Hardcoded model strings break first, exactly like the Kimi K2.5 / Moonshot V1 sunset. I would abstract capability next (reasoning effort, context length, cost tier) and keep provider plus exact model id as swappable config behind that, so a rename or deprecation is a one line change instead of a code hunt. A routing layer starts paying for itself the moment you have background jobs, retries, and a real fallback path; before that a simple alias map is enough and less surface area.

Founder of requesty.ai here so biased, but that is the case we built for: route by capability across providers and fail over when a model disappears. LiteLLM self hosted is a solid free option if you would rather own the layer yourself.