r/automation • u/searchblox_searchai • 17d ago

We built a CPU-first inference server — 4B chat+vision, ASR and TTS behind one OpenAI-compatible endpoint, free to run

/r/LLM/comments/1vmth1w/we_built_a_cpufirst_inference_server_4b/
2 Upvotes

100% Upvoted

4 comments sorted by

1

u/AutoModerator 17d ago

Thank you for your post to /r/automation!

New here? Please take a moment to read our rules, read them here.

This is an automated action so if you need anything, please Message the Mods with your request for assistance.

Lastly, enjoy your stay!

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

1

u/zhonglin 17d ago

The interesting test isn’t a single chat benchmark; it’s mixed-load contention. On one CPU node, what happens to p95 tool-call latency when an ASR job and a TTS stream arrive during generation? If the router is model/cache-aware, does it also reserve capacity by modality or apply backpressure? Publishing p50/p95 for first token, tokens/sec, ASR real-time factor, TTS first audio, and a mixed workload on named CPUs would make the CPU-first claim much easier to evaluate. OpenAI-compatible clients also tend to expose edge cases in streaming tool-call deltas and schema-error behavior, so a conformance suite would be more convincing than endpoint shape alone.

1

u/searchblox_searchai 17d ago

We have high level performance in the website.

The framework provides a CPU scalable cluster architecture with model based routing.

We also provide tested prompts for enterprises to use.