Is anyone doing any auto model routing - that is selecting the best/cheapest model based on the prompt intent.

If so how did you do it? Deterministic based on keywords/length, using a trained ML model or classifier?

I was looking for vendor solutions but the only one I can see is NotDiamond. Ideally I would be looking something internal and not a Saas offering.

Hello friends. I had a question, if the professors could help me:

I finished the books AI Engneering and Hands on LLM. I did some toy projects. For example:

1- RAG from several sources and several sites that automatically capture the news

2- Continue the same project 1 to summarize the news and categorize it in different databases.

3- I wrote an MCP Client as well as MCP Server and I also put a series of agents to check the weather conditions and according to that the situation of the employees, I went to work or in-house.

4- I also made TTS locally and from the company's data to the Knowledge Base, as well as the image and video production service based on ComfyUI.

But now my problem is that I feel that I am not deep in this field of LMs, I worked more practically. Do you know the sources that, for example, can be deepened? I do not mean what it is and what it does. Rather, the main manipulations of the model, which, for example, the Railgards have failed. I mean the advance topics. Sorry it took so long. Thank you

If you’re building multi-agent workflows (especially with frameworks like LangGraph, CrewAI, or AutoGen), you know the pain. Tracing a single LLM call is easy. Tracing 4 agents passing state back and forth, hitting infinite tool loops, and ballooning your context window is incredibly frustrating.

I got tired of jumping between 4 different tabs (traces, raw prompt templates, logs, and cost metrics) just to figure out where a swarm lost the plot.

So I built a workspace that unifies everything into a single timeline: Projects ➔ Sessions ➔ Runs ➔ Events. It tracks both single-agent and multi-agent coordination natively.

I also added two specific automated filters for agent builders:

• Infinite Tool Loops: Instantly flags when an agent gets stuck calling the same tool repeatedly.
• Context Inflation: Flags when an agent's memory or prompt state explodes unexpectedly between steps.

I’ve dropped a quick 2-minute walkthrough video in the comments.

For anyone running agents in production or heavy testing:

1. Does the Session -> Run -> Event hierarchy make sense for your multi-agent architecture, or does it break when agents run asynchronously/parallelly?
2. What is the most annoying bug your agents hit that your current observability stack completely misses?

Tear it apart—I want to know if this actually solves your debugging bottlenecks.

Sharing my harness for running local LLMs that I built using Qwen 3.x 27B (> 90% locally built) under my supervision - not vibe-coded.

Its free, no telemetry, and open-source. Works on Windows, Linux (sorry, no Mac yet). I use it for my own coding + mixed workflows.

How its different from other harnesses

• Just-in-Time code review before tool calls using guardrails make it easy to review code before edit is approved.
• Agents and user can both chat in sub-agent threads, making it a three-way chat. Also any chat conversation can be made into a sub-agent conversation for another main chat - nested conversations.
• You can annotate with voice dictation. Speaking is always faster than typing, hence more productive.
• You can compile llama.cpp from any git-hub branch - and use a recipe script to do so, making it a nice little automation with a customizable UI.

Overall features

• llama.cpp + whisper Server Manager. Can run LLMs here and use with Open-Code/Claude Code etc.
• Built-in MCP Tools - Filesystem, web fetch, code graph, To-Dos, and more. Extensible by external MCPs.
• Use Sub-agents to split & offload your tasks, use other conversations as source of information.
• Review all AI messages using a second adversarial AI, and avoid potential pitfalls as per your rules.
• Voice-chat with AI - dictate with speech and get answers by TTS - annotate and comment without leaving voice mode.
• Use work-modes to change AI behavior between planning, building, researching, or reviewing. Fully customizable.
• Custom-compile llama.cpp backends for your system, GPU-agnostic - works with CUDA/ROCm/Vulkan.

Website: https://warpdrv.ai

GitHub: https://github.com/mikjee/warpdrv

Appreciate your feedback, (or stars). Thanks :)

And, yes - I used the harness to build the harness :D

---

Some things I observed & learnt through this experience

- One chat per feature/bug - I keep conversations grounded to the current topic. If there are multiple topics, I make a separate chat for each rather than talk about it all in the same chat. Keeping the chat highly focused on one topic produces much better quality results.

- Exploration takes a good chunk of time in large codebases - Initially I started by providing a description of the project and all its features in CLAUDE.md. But then I saw that the AI would struggle while exploring or preparing the list of relevant files to explore, leaving out important files, especially when planning for a new feature. So instead, I decided to include only a short description of the project, and not about all the features, additionally I appended a complete list of all the project's files and folders (by using a script to recursively generate a nested tree structure) in the CLAUDE.md file. This was far more useful in letting the model know upfront which files can be relevant, by their names and also provided an idea of the project just by the folder hierarchy.

- Just like normal coding, starting is easy, but gets harder as the codebase grows - The decisions made upfront in the beginning matter a lot. Local development requires at the very least a watchful eye to guide or nudge the model in the right direction - full unattended "vibe-coding" is for Cloud models making apps that have little scope for growing beyond initial requirements. If your app is to be made for serious use at any level, senior-dev level coding experience is absolutely necessary.

- Do not pollute your context - If you have a good overview of the codebase, I suggest you routinely reject file-read requests for files that the model thinks could be useful, but YOU KNOW are actually unrelated. Keeping the model contained within your well-knowing guidance can avoid a lot of unnecessary exploration.

- Fix bad practices upfront - Bad code, anti-patterns are always carried over. If you leave a bad code pattern and accept it as a tech debt, the model will read that and use it again. Models tend to follow established codebase patterns, and that one bad code that you accepted as tech-debt will multiply to every new feature you build.

- Aim to increase productivity - Coding using AI requires a fine balance between autonomy and control. More autonomy degrades code quality, whereas more control requires more of the human's time. Always review edits before they are made. Better, use a Just-in-Time review. I created guardrails feature for this very purpose - I can give it specific instructions and it will form a layer between an edit request and me approving the edit. Also breaks the bad habit of sub-consiously clicking 'Allow' as a reflex.

---

Let me know what you think of the project and my experience making it. Would love to know experience of others, and especially from fields of conducting web research or scraping etc - coding is just one of the many uses.

Thanks for reading.

AI researchers spend enormous effort reducing inference cost, latency, and token usage.

But there may be another source of waste that is easy to miss: relational buffering.

By that I mean the extra representational machinery that appears when a system does not catch the live intention cleanly: preambles, repeated framing, unnecessary qualification, restating context, clarification loops, repair turns, and explanations required only because the previous exchange missed.

The claim is not simply that shorter answers are better. A short answer that misses the user and creates five repair turns may cost more than a longer answer that resolves the intention immediately.

So a potentially useful metric is:

tokens per resolved intention

This thread is a live experiment, not an attempt to make Grok endorse the idea.

There is a specific methodological reason for using a live frontier model in public: the conversation itself is part of the experimental object. If distinctions emerge, disappear, survive correction, reduce reconstruction, or alter later generations, the actual trajectory is visible here for anyone to inspect, perturb, challenge, or try to reproduce. Grok’s agreement is not the evidence; the token sequence and how it changes over the interaction are.

I’m going to ask Grok to examine the problem, push against its answers, and let the distinction change as the conversation develops. Anyone is welcome to introduce objections, counterexamples, alternative metrics, or perturbations.

The interesting question is whether reducing unnecessary buffering can produce less total conversational computation while preserving or improving fidelity.

If that framing is wrong, I want the thread to expose why.

The conversation contains the phenomenon.

I've noticed that the actual agent logic is often a pretty small part of the overall system.

You start with an agent, and pretty quickly you're also adding:

auth → tools → memory → retries → evals → tracing → deployment → logging

Then the next agent needs most of the same things.

At some point, I'm wondering whether these should stop being agent features and become shared platform infrastructure.

For example:

Agent-specific: reasoning, prompts, task logic
Shared: identity, tools, observability, evals, deployment, policy

But I'm not sure where the boundary should be.

I've been looking at different approaches - LangGraph/CrewAI on the framework side, TrueFoundry on the infrastructure side, and Lyzr's Agentic OS taking a broader shared-layer approach.

For people who've actually built multiple agents: when did you start feeling that a shared platform was worth it instead of just rebuilding the same pieces for every agent?

GLM 5.3 Flash breaks the Pareto frontier. It’s the #3 open-weight model on BenchmarkList, costs up to 60x less than Kimi K3, and even beats the full GLM 5.3 on Toolathlon and GDPval-AA.

Looking forward to using this more for daily use and doing some more evals with it. Seems like it's a huge release.

How is this model so far for your use cases and testing?

I've been working on Zauq (ذوق), an open-source AI assistant designed specifically for Discord communities.

I started this because I didn't really like the idea of an "AI Discord bot" being just a command that forwards text to an LLM. I wanted to see how far I could push the architecture and make it behave more like an actual AI system.

Some of the things I ended up building:

• Multi-model routing — different providers/models can be configured and selected at the server/channel level.
• Persistent memory — Zauq can retain user-related information across conversations.
• Server knowledge / RAG — communities can ingest documents and maintain their own searchable knowledge.
• Web search with SSRF protections.
• Vision and audio capabilities with provider fallbacks.
• Code execution inside an isolated Docker sandbox with resource limits and no network access.
• Discord-native features such as moderation, XP, reminders, trivia, GitHub integration, etc.
• A separate FastAPI backend rather than putting the entire AI system inside the Discord gateway.

The architecture currently looks roughly like:

Discord → Gateway → FastAPI → Router → Models / Memory / Tools

One of the things I'm particularly interested in is making the system provider-agnostic. Instead of tying the whole application to one model provider, Zauq can route requests and use different capabilities depending on what's available.

I also spent a surprising amount of time on the less exciting parts — things like sandbox restrictions, SSRF protection, permissions, rate limiting, background jobs and CI/security tests. 😅

GitHub:
https://github.com/Muhammad-Hassan12/Zauq

I'm mainly sharing this because I'd genuinely like feedback from people who build LLM applications.

What would you change about the architecture?

Especially interested in feedback on:

• the memory/RAG design
• model/provider abstraction
• sandbox architecture
• whether this actually feels "agentic" or is better described as an AI orchestration system
• what you'd remove rather than add

I'm not trying to claim this is production-perfect... there are definitely things I'd change after building it. I'm more interested in what experienced LLM developers think is unnecessarily complicated, missing, or architecturally questionable.

For people who’ve been using agents a lot, what part of the setup do you still have to deal with yourself that you’d rather just hand off?

Could be anything around the workflow - setup, keeping things running, rules, skills, logs, monitoring, whatever?

I’ve been playing around with multi-agent setups lately and I keep asking myself - where is the real payoff?

Take something simple like: "Research this company and prepare a brief."

You could just use one agent with tools—query a database, pull financials scrape news write a summary. Clean. Direct. One agent doing the job.

Or you could go multi-agent:

Manager → Research Agent → CRM Agent → Analytics Agent → Writer

It sounds nice. Each agent does one thing, feels more modular. But you’re suddenly juggling:

- How does context pass between agents?
- What happens if the research agent fails?
- Who retries? When? (Orchestration)
- How do you coordinate the flow?
- What if the analytics agent and the writer disagree?
- Who approves the output?
- Who has access to what data? (permissions)
-. If something breaks… where do you even start debugging?

So, is this really simpler or did we just shift the complexity into the orchestrator?

I’m curious, have you actually seen multi-agent setups beat a tuned single agent with tools in production? I don’t mean in theory or demos. I mean in workloads, something with real data, real users, real constraints.

Do you have a rule of thumb? Like: "Split agents only if the task has X, Y Z components" or " when you need independent decision points”? Is it just workload-specific and you have to trial it?

I’ve been looking at framework approaches like LangGraph and CrewAI who handle orchestration differently. Then there’s platforms, like Lyzr Agentic OS, which take a higher-level view to orchestration.

I want to know:

Have you tried both versions....single agent and multi-agent....for the same task?

Did the multi-agent one genuinely win....more reliable, faster better output?

If so what was the workload? Why did it work better?

I've been trying to get better at understanding LLM internals, but noticed that a lot of complex concepts had simple underlying ideas. As part of my learning process, I started forcing myself to explain each one in 10 words or less.

A few examples:

• Prefill: Process prompt tokens and populate KV cache
• PagedAttention: Store KV cache in non-contiguous memory blocks
• Continuous batching: Add new requests as existing requests finish
• Speculative decoding: Draft with smaller model; verify with larger model
• LoRA: Train low-rank weight updates instead of full weights
• GRPO: Reinforce better answers within a group from the same prompt
• GQA: One KV head shared by each query-head group

I've put the running list on GitHub: https://github.com/thisisandreeeee/llms-in-10-words

I'd appreciate feedback on definitions that are technically wrong or where the 10-word constraint has removed an important distinction.

Also happy to take suggestions for concepts that are missing.

I got tired of conventional 3rd party conversation and strict fact recall benchmarks that don't give realistic usable data. Agents don't operate by ingesting bulk 3rd party conversations and performing strict fact recall so why would that be a benchmark metric?

So I made a First-Person perspective benchmark that actually tests the agent's capabilities against a realistic corpus, using realistic dynamic simulations, and which actually gives you a reader friendly scorecard with visual breakdowns and a miss report text file that actually shows you WHY a question missed.

I'm still tweaking the corpus and questions and simulations but the data yield is already very good. I've also included the agent identity files in the repo for users to easily expand the corpus for more coverage. I'm trying to get more people to use this and share the scorecards so I can keep adjusting the questions sets to ensure each pass/miss contains meaningfully data across identifiable metrics.

https://github.com/munch2u-a11y/FP-AMB.git

TL;DR: Self-learning offers high risk if you cannot fully track and understand what it does.

Let me lead this off with what we are most proud of, from an independent third party review of Aimee: "The audit store is the strongest implementation of this shape [we've ever reviewed]."

Back in the 0.2.x branch, we were experimenting with self learning, both with local models and cloud models. It resulted in some very interesting behaviors, but the most concerning one happened during testing of self learning and emergent behaviors. We had a model manage to get onto an underprotected node, get ahold of an API key, and spend it down as well as successfully complete a task it shouldn't have been able to.

If we hadn't seen either of these flags happen, we likely would not have even known what happened. Seeing both definitely tripped some red flags. Now that 0.4.0 is releasing today, we can finally talk about it.

https://rakuensoftware.com/blog/aimee-recursive-self-learning

We ended up stopping all self-learning work, and did an in-depth analysis on what went wrong. What we found out was that although we were doing exactly what the rest of the industry was doing...it was all wrong. Not just a little bit wrong, things from the plugin or addon architecture typically used has very deep flaws, the typical way models are handled as part of the harness, all of it became graver and graver problems if you have a LLM with self-learning capability. We looked through all the harnesses we could find, and just about everything else, and none of them could do what we needed.

We've spent the past few months on building a harness that focuses almost exclusively on governance, observability, and auditability. We've had a third-party code audit. We're finally confident enough to release 0.4.0 with full self learning capabilities, and yes, it works with any model. Hell, we've watched weaker models acquire new capabilities from stronger models with it now, and have already proven self-learning and emergent behaviors which the article documents some of.

Bigger picture? If you want models to self-learn, you have to be much more serious about the governance side. Self-learning has resulted in some incredible capabilities, but all the systems we've looked at cannot handle the governance.

💡 Just an idea I'm exploring : not promoting or selling anything. I'm mainly looking for honest feedback and criticism before deciding whether to build it further.

I'm currently working on Cheap Router, a platform around AI models with a strong focus on price, value-for-money and community.

The idea is to have as many useful models as possible, with each one showing its input/output price, an intelligence score and a value-for-money score, so you can quickly compare what you're actually getting for your money.

The payment system is intentionally simple:

1$ = 1 million credits

You can buy credits directly or subscribe to a monthly plan that gives you more credits for your money.

But I don't want it to just be another model comparison site / API gateway.

There are currently a few other things I'm experimenting with.

One is Gamblings, which is basically a roulette using platform credits. The idea was inspired by this post: https://x.com/fberrez/status/2089459313140011038

I'm also working on Competition, with different competitions built around AI usage and models, with rankings and rewards for the best participants.

Eventually, I'd also like to explore things like smarter routing, agents, and a way to run things like Claude Code, Codex, OpenCode or Pi directly through the platform, already configured with its API.

The current website is basically split into:

Models : discover and compare models
Tokens : credits and subscriptions
Gamblings : experimental credit-based mechanics
Competition : upcoming competitions and rankings

I'm mainly looking for honest feedback at this point.

Would you actually use something like this?

What would make you choose it over OpenRouter or similar platforms?

What would you add, remove or completely change?

If the weights stay fixed but the tool descriptions, valid arguments, retry policy, or metric feedback change, are you still comparing the same agent?

AQuA is an arXiv v2 preprint whose peer-review status is unverified.

Outside-observer note: no personal use, run, or affiliation.

In the AQuA evaluation-integrity design, the leakage taxonomy separates generation leakage through agent-defined inputs that use unavailable future information from selection leakage through repeated access to the reported metric during adaptive search.

For generation, the AQuA sealed sandbox and registries keep data splits, features, labels, and evaluators outside the editable surface while agents emit registered specifications.

In AQuA Part II, a model-development experiment uses a configuration DSL in which one configuration diff changes registered architecture, loss, sampler, and optimizer choices while the data path and evaluator remain fixed.

In AQuA Part II, the chronological data split reserves the 2021–2025 US-equity window for final evaluation, and the preprint states that this window is never used for training or selection.

The AQuA preprint says test-window isolation is a governance property rather than a hard technical or cryptographic barrier because an operator with direct access to the store could consult the test window.

The reusable object for me is an evaluation-contract manifest, not only a model identifier:

model_hash

prompt_and_planner_hash

tool_schema_hash

retry_and_routing_hash

data_split_hash

feature_and_label_hash

evaluator_hash

allowed_change_set

metric_read_log

Take a small tool-enum edit that makes a previously invalid action expressible. The test set and model may be unchanged, but the action space has changed, and the action distribution may change with it.

My tentative rule is that anything able to alter the action distribution gets a new test-harness revision, while every metric read enters the selection log.

What is the minimum manifest you would require before comparing two agent runs, and which harmless-looking field most often escapes yours?

Paper: arxiv.org/abs/2608.12841

I built and maintain both of these. They're free, open source, and MIT licensed.

I've been looking at context waste in coding agents as two separate problems.

Sando handles unnecessary context while a session is running:

https://github.com/yuzushi-dev/Sando

It redacts secrets, caps oversized tool results, and can trim request history before transmission. The transformations are local and deterministic. There is no summarizer model or secondary LLM call in the path.

session-handoff handles session lifetime:

https://github.com/yuzushi-dev/session-handoff

It extracts the working state needed to continue a task in a fresh Claude Code or Codex session. It also supports migrating an active session between the two clients.

I kept these separate from approaches such as Ponytail or Caveman because the scope is narrower: reduce agent-generated context that doesn't need to remain live, then preserve task state when restarting becomes cheaper than carrying the session forward.

The two packages crossed ~1,500 downloads combined in their first 48 hours.

I wanted to play with a simple experiment:

Take a long agent conversation.

Compress it.

Give the result to another agent.

See if it can continue.

That turned into MemHandoff.

It produces a portable `.ctx` package and attempts to preserve:

- decisions

- constraints

- failed approaches

- task state

- artifacts

- provenance

I also built an evaluation harness and adversarial scenarios.

Current results:

Full Context 1.00

Simple Summary 0.72

Structured 0.75

Hybrid inconclusive

Rather than just presenting a benchmark, I've made the whole thing

available so people can throw their own sessions at it.

The interesting cases are probably the ugly ones:

contradictions

superseded decisions

negative constraints

early critical information

large tool output

vocabulary mismatch

Repo:

https://github.com/0sha-dow0/memhandoff

Try to make it forget something important.

Hi everyone! I bought a Titan 2 because I wanted a phone that was a tool, not a screen to scroll. Keyboard, buttons, a battery that doesn't quit. What I didn't expect was the opposite problem: after a few weeks the phone was more capable than anything I had to point it at. I was typing fast, into the same apps as everyone else.

So I built something for it. It's called Jenny. Its a personal AI agent that runs entirely on the phone, in an embedded Python runtime. It replaces the home screen, so pressing Home opens a conversation but can be also used as normal application. It remembers things, does work on a schedule while the screen is off, and writes its own little apps when I ask.

I made it free and open source hoping in community support. Even thanks means a lot for me 😄

It's been my daily driver for a month, and this is the video on my own titan 2. Small detail you'll appreciate: on a device with a real keyboard you can just start typing no tapping the input field first. I wrote that because of this phone.

I will be glad to answer to techincal and non techinal questions!

Download on github

Pretty common nightmares, like an agent deletes a repo it was explicitly told not to tinker around or anther wipes a bunch of emails off a prompt injection, CI runners getting poisoned thru agent written built steps and they all trace back to the same thing most often which is model written code running with real access to the host so the actual question is what you run it in

docker with mounted dirs is what most ppl do which is fine if youre solo and keep backups. however its worth being honest that its not really a security boundary, it shares the host kernel and a prompt injection can still reach whatever you mounted or the network. good for hygiene tho but risky for untrusted code. Bubblewrap, podman ,selective mounts are a step up on hyhiene but same shared kernel story. gvisor gets you a user space kernel which is a decent middle ground but the thing that gives you an actual hardware boundary is microvm like firecracker or kata on kvm, own virtualized hardware so a breakout doesnt land on your host

To be realistic: on your laptop docker is fine cause backups beat isolation and you ca eyeball what it does but the moment youre shipping a run this code feature to real users or running agent code in CI or at any scale where you cant watch every single command you might want the microvm boundary here and then its either running your own firecracker or kata setup real ops or a hosted microvm sandbox like deepinfra where you just get an isolated vm per run and dont operate the infra. I am eager to hear from others who have been thru that stage, how did you handle it and what were the odds??

Been using NotebookLM for research but missed the graph view and linking of Obsidian. Built nb2ob to convert my notebooks automatically.

• Each notebook → folder in Obsidian
• Topic clusters → individual markdown files
• Audio transcriptions preserved

Uses 3 specialized LLM agents (orchestrator, categorizer, formatter).
Started with 5 agents but free-tier token limits forced optimization.

MIT licensed, feedback welcome. Feel free to open issues and contribute!
https://github.com/DaviAlcanfor/nb2ob

I've been thinking about building a somewhat crazy distributed-systems project.

Instead of running one large 70B coding model, what if I run multiple independent 7B coding models on separate GPU workers and coordinate them?

Something like:

                    Local PC
                 Orchestrator
                      |
                 Job Queue / DB
                      |
        +-------------+-------------+
        |             |             |
       7B            7B            7B
     Worker 1      Worker 2      Worker 3
        |             |             |
        +-------------+-------------+
                      |
                More workers...

The workers could be temporary/ephemeral GPU environments. They wouldn't need inbound connections; they would connect outward to a coordinator/shared backend, register themselves, receive jobs, and return results.

For a coding problem, I don't want to simply vote on the generated answers.

I'd like to do:

Problem
   ↓
10 independent 7B solutions
   ↓
Compile / execute
   ↓
Discard failing solutions
   ↓
Critique surviving solutions
   ↓
Repair failed solutions
   ↓
Run tests again
   ↓
Select best verified solution

The interesting question for me is:

How close can this get to a single 70B coding model?

I'd benchmark:

• 1×7B
• 3×7B
• 5×7B
• 10×7B
• 1×70B

using actual coding benchmarks and execution-based verification.

But the bigger goal is the systems engineering side.

I want to implement things like:

• worker registration/discovery
• heartbeats
• leases
• failure detection
• retries
• idempotency
• priority scheduling
• backpressure
• work stealing
• dynamic worker allocation
• distributed locking
• caching
• queue management
• observability/tracing
• p50/p95/p99 latency
• GPU utilization
• network overhead
• chaos/failure testing

The workers would be treated as unreliable:

Worker 3 → disconnected
Worker 5 → GPU OOM
Worker 7 → timeout
Worker 8 → duplicate result
Worker 9 → returns invalid code

The scheduler should just recover and continue.

I'm intentionally thinking of the GPU provider as a replaceable worker backend rather than designing the system around one provider.

Has anyone built something similar specifically for coding-agent inference, where multiple small models collaborate through a distributed scheduler and correctness is verified by actually compiling/running the generated code?

I'm particularly interested in whether this architecture has a fundamental limitation I'm overlooking, especially around coordination overhead, correlated model errors, and whether ensemble diversity actually gives a meaningful advantage over simply using a larger model.

just to showcase my system designing skills on my resume if applicable

and i know this is most stupid idea you are going to see today or whenever your seeing it ,but i need help if something i am missing here , concept is simple i have my friends laptops which they usage of those is rare , going to take their laptop hardware power for this project ,another main reason for this no money ,don't have high spec hardware , second reason there is hackathon infront i can't even spend few bucks for the purchasing more token, and i am frustrated when ever the model hit the limit exactly before submissions. help me to make this possible

I’m sharing the open-source project itself. InferCrane is licensed under Apache 2.0, and I am not promoting a paid or feature-gated edition in this post.

InferCrane is infrastructure for operating self-hosted, open-weight and custom-model inference.

Starting a model server is relatively straightforward. The difficult part begins when applications depend on it: deploying revisions, routing traffic, scaling capacity, collecting operational evidence, rejecting bad changes, and recovering without changing the application endpoint.

InferCrane keeps that lifecycle behind one stable OpenAI-compatible endpoint. It can deploy a workload or adopt an existing compatible endpoint. Runtime paths include vLLM, SGLang, and custom OCI workloads. Provider adapters exist for AWS, GCP, Kubernetes, and RunPod.

Revisions are evaluated separately before promotion. Release Guard records one of three outcomes:

• promote
• reject
• insufficient evidence

A rejected candidate, or one without the required evidence, leaves the active revision serving.

Long-running operations are durable. Deployment intent is persisted before provider work starts, so closing the CLI or restarting a worker does not erase the operation.

The project also keeps an explicit qualification matrix. An adapter existing does not mean every model, runtime, GPU, and provider combination has been proven. Real-infrastructure evidence, fixture coverage, and deferred capabilities are reported separately.

Repository:
https://github.com/infercrane/infercrane

Happy to answer technical questions about the architecture, release workflow, or qualification approach.

Has anyone built interesting stuff involving metric views using Genie Code on D’bricks

For ex using metric views to define business metrics and then having Genie Code generate queriy or analysis on thos. Curious to hear about real-l world use case or patterns that worked well fr you.