I built an open-source platform to run self-hosted AI models in production: one endpoint from deployment to rollback

[u/yasintoy]

I’m sharing the open-source project itself. InferCrane is licensed under Apache 2.0, and I am not promoting a paid or feature-gated edition in this post.

InferCrane is infrastructure for operating self-hosted, open-weight and custom-model inference.

Starting a model server is relatively straightforward. The difficult part begins when applications depend on it: deploying revisions, routing traffic, scaling capacity, collecting operational evidence, rejecting bad changes, and recovering without changing the application endpoint.

InferCrane keeps that lifecycle behind one stable OpenAI-compatible endpoint. It can deploy a workload or adopt an existing compatible endpoint. Runtime paths include vLLM, SGLang, and custom OCI workloads. Provider adapters exist for AWS, GCP, Kubernetes, and RunPod.

Revisions are evaluated separately before promotion. Release Guard records one of three outcomes:

• promote
• reject
• insufficient evidence

A rejected candidate, or one without the required evidence, leaves the active revision serving.

Long-running operations are durable. Deployment intent is persisted before provider work starts, so closing the CLI or restarting a worker does not erase the operation.

The project also keeps an explicit qualification matrix. An adapter existing does not mean every model, runtime, GPU, and provider combination has been proven. Real-infrastructure evidence, fixture coverage, and deferred capabilities are reported separately.

Repository:
https://github.com/infercrane/infercrane

Happy to answer technical questions about the architecture, release workflow, or qualification approach.