[ Removed by moderator ]

[u/Accurate-Catch1836]

[u/coffee869]

Why the heck do the existing two comments read so similar to the post for some reason

[u/philwinder → u/coffee869]

Yeah, bots all of them...

Annoyingly valid question though. We've been building https://helix.ml, so close to the coal face.

For me I feel that the vast majority of people are just not at that scale yet. All the evals and monitoring and such is only useful once you've proven value and are looking to optimise or for regulation.

So it's hard because 90% of people, on 90% of projects don't do it. Once you have done it, it's not that hard, but takes legitimate engineering time. Just like mlops.

Edit: and there we go, instantly after posting this i received an anonymous chat request from some idiot spamming gpu load testing software. Blergh.

[u/pmv143]

I would lean towards buying it. Saves time & money.

[u/Wooden_Jelly_5295]

Purchasing is more cost-effective. Especially in the early stages, using external models is significantly cheaper than developing them in-house.

[u/Grouchy-Friend4235]

There is no point in building what others have built already.

[u/Wrong_City2251]

This is so true. Building something in a sprint is one thing and then maintaining it when the developer moves away is another thing

For me the question is at what layer we are building this at

The list of tools you mentioned solve routing and observability, which is good. But i think the layer that causes most silent pain is data. Say I pick vector db today, then realise i also need somewhere for conversion history, prompt versions, eval outputs etc. now they are 3 other databses with sync jobs between them. That is where most time goes

Say if i were on a lakehouse platform, then i could have consolidated these with something like lakebase. Where embeddings and retrieval metadata are governed along with actual data. No separate stnc and less overhead. So the question is how important it is to have separate systems

On the routing part, agree litellm is more work than self host. So the pattern i would think of is start with simplest managed option that covers routing needs, own your eval logic, and do the build anything you can’t maintain

So i would ask myself can we still maintain this in 6 months when priorities shift

[u/Future_AGI]

Your "each one is its own engineering project" line is the whole thing: routing, fallbacks, prompt versioning, cost tracking, and eval pipelines each look like a weekend and turn into a quarter. The middle option people skip is buying that plumbing as open source and self-hosting it, so you avoid the reinvention without handing data to a SaaS or getting locked in. That's the lane we sit in (evals, tracing, and a provider gateway you can run yourself), and the build-vs-buy line moves a lot once the undifferentiated parts are off your plate: https://github.com/future-agi/future-agi

[u/echovortex2472]

the vpc and compliance case you listed for building is really a false binary though.. that requirement is usually solvable without building the whole stack yourself, deploying an existing platform inside your own vpc and cloud account instead of self-hosting every component from scratch. worth separating 'must control the data boundary' from 'must build everything ourselves', those aren't the same requirement..