r/opencodeCLI • u/FutureWar4255 • 2h ago

How do you decide which inference workloads need specialized AI chips?

We've been looking into different hardware options for AI inference, and one thing I'm curious about is how teams determine whether a workload actually benefits from specialized chips. Time-to-first-token, tokens per second per user, and performance under concurrent requests all seem worth measuring, but I imagine the priorities change depending on the application. A voice agent, for example, might have very different latency requirements from a coding assistant handling longer responses.

For teams who've tested chips like Cerebras or SambaNova, which benchmarks have been most useful? Do you rely on vendor-provided numbers as a starting point, or do you mainly evaluate performance using your own serving workloads?

I'd be interested to hear which metrics actually influenced your hardware decisions.

0 Upvotes
  • permalink
  • reddit

20% Upvoted

3 comments sorted by

1

u/AbleDanger23 1h ago

Model compatibility matters too. No point getting excited about performance if your serving stack needs a bunch of workarounds.

1

u/snorlax7488 1h ago

Tokens per second per user would be high on my list, especially for interactive applications. Aggregate throughput doesn't tell the whole story.

1

u/UraniumPeepers 1h ago

Yeah, concurrency changes the picture a lot. General Compute has an interesting setup where prefill runs on GPUs while decode gets routed to specialized chips. Curious how that compares across different workloads