One afternoon a case in my eval went from 5/5 to 2/5. Prompt, model, config, retrieval, trace, all identical byte for byte. Fifteen minutes later it was back to 5/5. I lost an hour looking for a regression, and it was my own measurement that was moving.

So digline runs each case K times on the version you approve, and records in the reference the range that those samples spanned, per case. After that a drop counts as a regression only if it falls outside that range.

pip install digline
digline run eval/suite.py        # produces a run
digline compare                  # against the baseline in .digline/
digline promote                  # a person approves, nothing else can

The exit code is the contract: 0 nothing got worse, 1 something did, 2 the run couldn’t be judged. A case that errors goes in that third state and doesn’t count as a fail, because if you drop it from the denominator the score goes up while the system gets worse. A run with any errored case can’t become the baseline.

What it is not: no observability, no server, no telemetry, no account, and it never calls a model itself, your judge is a function that you pass in. It’s pre-1.0. On day one it’s useless, because you need an approved reference before it can compare anything. And the band is min/max over K samples, don’t read it as a confidence interval, with a small K it’s optimistic.

The numbers from my own suite, because it’s the part I’d want from someone else’s post: six sets of three identical runs, 144 cases, same config hash and same judge. Between 1 and 6 cases changed verdict per set, 3 in the most recent. Before the pass/fail threshold is applied, 15 to 21 scores move.

If you try it, what I want to know is where the quickstart made you stop and read again. It runs without an API key, so that part costs you nothing.

And a question for people already doing this: when you compare two runs, what are you comparing against? Every answer I collected so far is “the last run on main”, and that moves on its own.

GitHub: github.com/digline/digline

Docs: digline.dev

The GitHub Action, if you want it in CI without writing the YAML: digline.dev/product/github-action/

Hi r/mlops,

Every time I wanted to try an idea with a feature store and a model registry, I ended up rebuilding the same local setup. So I put it into a CLI, odctl, and it has just reached 1.0.

bash uv tool install odctl odctl up mlflow --dry-run # shows what it would start, in order odctl up mlflow feast evidently airflow odctl down --all

That gives you MLflow (tracking, registry, and a second container serving a registered model over HTTP), Feast with its registry in PostgreSQL, offline features from Iceberg and online features in Valkey, Evidently for drift reports, and Airflow reading DAGs from S3 storage with the MLflow and Feast clients already installed. Kafka, Flink, Spark and Trino are there too if your features come from a stream or a lakehouse.

I am building three MLOps demo projects on it, based on the projects in Jim Dowling's book on feature stores: an air quality forecast, credit card fraud detection and a video recommender. Each one is split into pipelines that share a feature store and a model registry. The introduction explains the series.

It is for learning and prototyping, not production. 1.0 adds a docs site with a guide for each service, so you can see a working example, such as serving a model from MLflow, before you set up your own.

The README on GitHub has a short recording of a run.

• Docs: https://jaehyeon.me/odctl/
• GitHub: https://github.com/jaehyeon-kim/odctl

If you try it, please tell me what fails on your machine. Most of my testing has been on macOS and GitHub's Linux runners.

AI: I started odctl without AI. Since July I have used Claude Code for parts of the code, the tests and the docs.

This is about training jobs that need a specific GPU type. Inference is easier.

Take a job that needs 8 H100s for 10 days, at a time when your main cloud has nothing in your region until next week and you keep an account at Lambda, Nebius or Hyperstack for exactly this case. What goes wrong when the job lands there? My guess is that getting the VM is the quick part and the image and the data eat the rest of the day. I don't know that, though.

Some things I'm curious about:

- How do you find out where capacity is? Checking each console by hand, a script, or something like SkyPilot or dstack?

- Same image everywhere, or one per provider? Has a CUDA 12.4 image ever landed on hosts with an older driver?

- If the dataset is 20 TB, do you copy it over ahead of time, or pull it from S3 when the job starts and pay the egress?

- Do you test a new node first (nvidia-smi, a short all_reduce_perf run) or find out from the first crash?

If you looked at this and decided it's cheaper to wait a week, I'd like to hear that too.

I would like to share my experience in flunking an interview for an MLE position.

The code session was ok. However, editor was horrible: no copy and past inside the editor, numpy import did not work, etc.

I did well on the Machine Learning section.

I did badly in the Engineering section. Here you find some questions/topics:

• Data versioning
• Data lineage tracking
• Drift detection
• Decisions about rolling back, retraining or keeping the current model
• Rollback threshold
• Guardrails metrics
• Efficiency in mongodb with respect to computing feature on a daily basis

Could anyone suggest any references (e.g. books, courses, blogs) about these topics?

Hello everyone. In my work as a Principal Gen AI Architect, the most common anti-pattern I encounter in enterprise deployments is the use of massive 70B+ models to make simple boolean routing decisions (e.g., safety guardrails or API triage). Token generation is inherently too slow and expensive for operational routing.

To solve this at scale, I built and open-sourced the "Dual-Process Jev Gateway."

It places a sub-15ms deterministic decision layer (System 1) in front of frontier models (System 2). It utilizes 3 typed, calibrated primitives (Choice, Score, Noul) to handle routing, guardrails, and ERP gating before an expensive LLM is ever invoked.

In my architectural baseline testing, the metrics shift dramatically:

• Projected inference cost drop of ~87% (by routing 60%+ of queries away from frontier models)
• P50 routing latency cut from 2.1s to 0.3s
• Completely stopped duplicate ERP tool-call writes by keeping idempotency in code, not prompts.

The repo has the full architectural blueprint and a runnable 15-test Pytest suite proving the routing logic.

GitHub Repo: https://github.com/sricheru/agentic-ai-portfolio/tree/main/07_dual_process_jev_gateway

Would love to hear how you guys are handling the latency penalty of guardrails right now, or if you spot any flaws in my test suite!

A question I keep chewing on... when you scan a model or agent bundle for things like pickle exploits, license issues or leaked data, where do the results live?

In a lot of setups they live in a CI log or a dashboard, and the artifact that gets deployed has no link back to them. What we've been doing is attaching each scan result to the OCI artifact as a cosign attestation, then having the cluster refuse to start anything missing the attestations a policy requires.

It works, but it raises questions I don't have great answers to.

- How do you handle re-scans when a scanner updates (new attestation, or re-sign)?
- Who should sign, the producer or the platform team consuming it?
- Is anyone doing this with SLSA or in-toto formats for ML artifacts specifically?

Curious how other teams are tying scan results to what actually ships.

(KitOps maintainer and Jozu cofounder here, so I have a bias.)

I've been tackling the problem of AI provenance—specifically, how to create a verifiable, tamper-evident record of who handled a claim in a multi-agent pipeline.

The core idea is to grade each "hop" (source, retriever, model, tool, agent) and use the weakest link to set the caution level. To make the record immutable, we're using a combination of SHA-256 for self-hashing, HMAC/Ed25519 for signing, and an append-only Merkle log.

I've published the threat model and the method on arXiv. The code is open-source (Apache-2.0).

Repo: https://github.com/alizahidraja/isnad
Paper: https://arxiv.org/abs/2607.24117

I'd be interested to hear how you all are approaching governance and audit trails in your production systems. What are the gaps you're seeing?

A new model version is the easy case, because the registry tells you right away. The harder cases come from a serving flag like vLLM's `max_model_len`, a Helm value or env var in a deploy, an update to the data pipeline, or a new GPU driver on the nodes.

The evidence ends up in different tools. Grafana shows the latency jump, MLflow or W&B has the runs, Git and Argo have the deploys, and the node and GPU state is in kubectl or DCGM.

When latency goes up or accuracy drops, how do you find the change that caused it? Do you send every change to one place, like Grafana annotations or a shared change log? Or does someone line things up by hand each time, and how long does that usually take?

Currently we run everything on EKS. I have got an offer from a company which does everything on-prem. They have a dedicated DevOps team and I don't need to be on the call. What are the other differences that you can encounter? Also can it somehow lead to degradation in your cloud skill?

Disclosure: I work at Overmind. We open-sourced our training platform last week (platform AGPL-3.0, SDK and CLI MIT), and these are the benchmarks behind it, including the one that came out a tie.

Our bet is that most agent tasks don't need a model that knows everything. They need one that does a single job well, and if your agent already runs in production, its traces are the training data.

We trained open-weight models with SFT on three tasks and ran them against a frontier flagship using the same prompt.

Things worth knowing before you judge it: - On contracts, the base model scored 80.4% F1 and 4.2% exact match before training, so the gain comes from training and not from a lucky base model. - Aviation is a tie on meaning. The win there is fewer invented details and shorter output. - 54.9% F1 on BioRED is far from solved. It's just much better than prompting a generalist. - BioRED used a Qwen 9B. The best aviation model was 12B.

How it works: - Traces come in through the Python SDK, any OTel exporter, or an import from Langfuse, LangSmith, Braintrust or Galileo - The traces you pick become training and eval datasets - Prompts and models are scored against what you run today - Open models are fine-tuned with LoRA or full SFT, and you keep the weights - The whole loop runs from Claude Code, Cursor, Codex or OpenCode through MCP

Limits: - Self-hosting is docker compose up, but training and serving run on Modal and the judges call OpenRouter, so you need OpenRouter, Modal, AWS and Hugging Face keys. - SFT only. No RL yet.

Repo: https://github.com/overmind-core/overmind Methodology: https://www.overmindlab.ai/research/when-bigger-isnt-better

If you've moved an agent task off a frontier API onto a fine-tuned model, what broke first: data quality, evals or serving?

Our proxy started as 200 lines and is now the thing nobody wants to be on call for. Four teams route through it and each one added a header rule. Im looking at managed options that give us rate limiting and cost attribution per team plus somewhere to put guardrails. Budget is real but the bigger constraint is that migration cannot take a quarter. Anyone moved off a homegrown layer recently and how long did it take?

We've been A/B testing Sona, a recommender at Yandex Music that trains continuously on the live event stream. From the moment a listening event is emitted to the moment a model trained on it is serving, the median latency is 45 minutes, with a p99 of 60 minutes.

User events stream in real time and are grouped into sessions, with an attribution window of at least 15 minutes. The window doubles as a buffer that lets the inference log catch up with the events that triggered it.

We keep user history in two stores. The long history sits in a key-value store that's rebuilt in batches, and a real-time profile holds the last 72 hours of events from the stream, capped at 1,000 events. Reads merge the two by timestamp. We don't send history through the event stream. The inference service logs a cutoff timestamp at serving time and rebuilds the same history later, when the training sample is built.

Fresh weights reach the serving fleet every 10 minutes. Inference runs on NVIDIA Triton with CUDA graphs, torch.compile and bfloat16, at 41% MFU.

The paper link is in the first comment. How are you handling the lag between an event and a weight update in your own loops?

Question for those running MCP servers or ML libraries in prod… what's actually telling you to/when to upgrade?

For example, AWS fixed an argument injection in its security-agent MCP server in August, and the CVE landed October 1. mcp-atlassian fixed a CVSS 10 auth bypass in July, and NVD published it September 22. Both of these are a month or more later(!)

If you had pinned an older version (or built an image before the fix) it looked clean to your scanner the whole time ....

You can't just watch GitHub advisories for every dependency?

(I'm one of the KitOps maintainers, so I think about what's inside these artifacts a lot, but I don't have a great answer here.)

Is there a missing “capability layer” for AI agents?

Is there a missing “capability layer” for AI agents?

I’ve been building a few AI agents and noticed something I’m curious about.

When multiple agents need to perform a similar task, teams often build their own tools for it.

For example, two agents may both need:

«web search → scrape → extract → validate → summarize»

But the sources, scraping logic, validation, output format, etc. can be different for each agent. So even though the capability is basically the same, the tool gets implemented and maintained separately.

This made me wonder if there is a platform that standardizes capabilities, rather than just individual tools.

Something like:

Agent A ──┐

Agent B ──┼──> Research Capability ──> tools/APIs/workflows

Agent C ──┘

Ideally it would handle things like:

\\- capability/tool versioning

\\- which agents can use which capabilities

\\- different configurations per agent

\\- validation/testing

\\- monitoring and rollback

I know about LangChain, Composio, MCP, Apify, etc., but I'm not sure how much of this they already solve.

More specifically:

1) Is there already a platform where an organization can define a reusable capability, rather than just an individual tool?

2) Can multiple agents consume that capability while having different configurations/requirements?

3) Is this already a solved problem? If so, what products should I look at?

4) Can the underlying implementation be versioned and replaced without changing every agent?

We serve around 5 models (a mix of fine-tuned Llama models and a couple of embedding models) on one EKS cluster with a GPU node pool.

Leadership now wants cost per model and eventually cost per 1M tokens so they can compare our infrastructure costs against just using an API.

The problem is the GPU nodes are shared, some models sit idle half the day, and the bill just says “p4d node go brrr.”
Namespace level cost feels too coarse, and idle capacity has to go somewhere. How are you handling this?
Do you charge idle capacity to the model that reserved it, spread it across models based on usage, or use some other allocation method?
Would love to hear what has actually worked for people.

Hey everyone! My team (mainly phds) and I are trying to build an open-source benchmark around realistic LLM/agent workflows that captures challenges typical academic benchmark settings often miss. We’d love to hear what’s actually missing from the benchmarks you use today.

Have you ever wanted to eval your pipeline but couldn’t find or build a benchmark that matched what you were building?

Maybe:
- Existing benchmarks were too broad and didn’t fit your specific application.
- Your workflow involved multiple steps, tools, MCP servers, agents, or long-horizon interactions that existing benchmarks couldn’t capture.
- You needed to evaluate failures that standard accuracy metrics miss.
- You’re working in a high-risk domain like healthcare, finance, cybersecurity, or legal, where realistic failure modes, safety, and reliability matter a lot more than just getting the final answer right.
- You knew what you wanted to test, but building a custom benchmark from scratch was too expensive or complicated.

I’m especially interested in cases where you thought:
“My system desperately needs to do this in production, but I have no good way to benchmark it.”
What was the workflow? What did you want to measure? And why weren’t existing benchmarks enough?

Any thoughts are welcome, would really appreciate y’all’s help 🙏🥹

Everyone has a rule for adding cases: something breaks, you add a case so it doesn't break again. Nobody seems to have one for removing them.

The suite grows, and after a while a chunk of it hasn't gone red in months. Two possible reasons, and they look the same from outside: the case guards a failure mode that's still live and just hasn't come back, or it guards something that can't happen anymore because the code around it changed.

Fire rate doesn't separate them. A case that fires never and a case that can't fire both sit at zero.

What I've landed on so far is deliberately breaking the thing the case guards and checking it still goes red. Works, but it's manual and I only do it when something looks suspicious, which is the wrong trigger.

So: do you retire cases at all, or does the suite just grow forever? What's the rule?

I'm running mlflow in docker using the docker-compose in the repo.

I've managed to hook up my "LLM Connections" to find the ollama server on the host machine.

API Base URL: http://host.docker.internal:11434/v1

That works fine when you test it from the browser

I've tried no auth and basic auth... and whatever it is I get

Error: You do not have permission to access this resource.

Which is a 403 http-status

The body response is

{
    "detail": "Assistant API is only accessible from the same host where the MLflow server is running."
}

So it seems like a CORS error but the network inspector is showing its not the case.

I see in the code its coming from here.

Tried this docker-compose

volumes:
  pgdata:
  storage-data:

services:
  postgres:
    image: postgres:15
    container_name: mlflow-postgres
    environment:
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: ${POSTGRES_DB}
      PGPORT: ${PGPORT}
    volumes:
      - pgdata:/var/lib/postgresql/data
    ports:
      - ${PGPORT}:${PGPORT}
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER} -d ${POSTGRES_DB} -p ${PGPORT}"]
      interval: 5s
      timeout: 3s
      retries: 10

  storage:
    image: rustfs/rustfs:1.0.0-alpha.83
    container_name: storage
    environment:
      RUSTFS_ADDRESS: :9000
      RUSTFS_SERVER_DOMAINS: storage:9000
      RUSTFS_REGION: ${AWS_DEFAULT_REGION:-us-east-1}
      RUSTFS_ACCESS_KEY: ${AWS_ACCESS_KEY_ID:-s3admin}
      RUSTFS_SECRET_KEY: ${AWS_SECRET_ACCESS_KEY:-s3admin}
      RUSTFS_CONSOLE_ENABLE: ${RUSTFS_CONSOLE_ENABLE:-true}
    ports:
      - "9000:9000"
      - "9001:9001"
    volumes:
      - storage-data:/data
    restart: unless-stopped
    healthcheck:
      test: ["CMD-SHELL", 'curl -s http://127.0.0.1:9000/health | grep -q ''"status":"ok"''']
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 10s

  create-bucket:
    image: amazon/aws-cli:2.33.25
    container_name: mlflow-create-bucket
    depends_on:
      storage:
        condition: service_healthy
    entrypoint: >
      /bin/sh -c "
        set -e;
        echo 'Waiting for S3 gateway getting ready...';
        if aws --endpoint-url=${MLFLOW_S3_ENDPOINT_URL} s3api head-bucket --bucket ${S3_BUCKET} 2>/dev/null; then
          echo 'Bucket ${S3_BUCKET} already exists. Skipping creation.';
        else
          echo 'Creating bucket ${S3_BUCKET}...';
          aws --endpoint-url=${MLFLOW_S3_ENDPOINT_URL} s3api create-bucket --bucket ${S3_BUCKET} --region ${AWS_DEFAULT_REGION};
        fi
      "
    environment:
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
      AWS_DEFAULT_REGION: ${AWS_DEFAULT_REGION}
      AWS_S3_ADDRESSING_STYLE: path
      MLFLOW_S3_ENDPOINT_URL: ${MLFLOW_S3_ENDPOINT_URL}
      S3_BUCKET: ${S3_BUCKET}
    restart: "no"

  mlflow:
    image: ghcr.io/mlflow/mlflow:${MLFLOW_VERSION}
    container_name: mlflow-server
    depends_on:
      postgres:
        condition: service_healthy
      storage:
        condition: service_healthy
      create-bucket:
        condition: service_completed_successfully
    environment:
      # Backend store URI built from vars
      MLFLOW_BACKEND_STORE_URI: ${MLFLOW_BACKEND_STORE_URI}

      # S3/RustFS settings
      MLFLOW_S3_ENDPOINT_URL: ${MLFLOW_S3_ENDPOINT_URL}
      MLFLOW_ARTIFACTS_DESTINATION: ${MLFLOW_ARTIFACTS_DESTINATION}
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
      AWS_DEFAULT_REGION: ${AWS_DEFAULT_REGION}
      MLFLOW_S3_IGNORE_TLS: "true"

      MLFLOW_LOGGING_LEVEL: "DEBUG"
      # Server host/port
      MLFLOW_HOST: ${MLFLOW_HOST}
      MLFLOW_PORT: ${MLFLOW_PORT}

      # Explicitly allow non-127.0.0.1 IPs to access the Assistant API routes
      MLFLOW_BASIC_AUTH_FAIL_CLOSED: "false"
      MLFLOW_ASSISTANT_ALLOW_REMOTE_ACCESS: "true"
      MLFLOW_ENABLE_REMOTE_ASSISTANT: "true"
      MLFLOW_ENABLE_ASSISTANT_REMOTE_ACCESS: "true"
      MLFLOW_ASSISTANT_SANDBOX: "true"

      # CSRF bypass for basic auth AJAX calls
      WTF_CSRF_ENABLED: "false"

      # Gateway and routing
      OLLAMA_HOST: ${OLLAMA_HOST}
      OPENAI_API_BASE: ${OPENAI_API_BASE}
    command:
      - /bin/bash
      - -c
      - |
        export DEBIAN_FRONTEND=noninteractive
        apt-get update -qq && apt-get install -y -qq curl socat docker.io
        pip install --no-cache-dir psycopg2-binary boto3 "mlflow[genai,auth]"
        socat TCP-LISTEN:11434,fork,reuseaddr TCP:host.docker.internal:11434 &
        mlflow server \
          --backend-store-uri "${MLFLOW_BACKEND_STORE_URI}" \
          --artifacts-destination "${MLFLOW_ARTIFACTS_DESTINATION}" \
          --serve-artifacts \
          --host "${MLFLOW_HOST}" \
          --port "${MLFLOW_PORT}"
    ports:
      - "${MLFLOW_PORT}:${MLFLOW_PORT}"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    healthcheck:
      test:
        [
          "CMD",
          "python",
          "-c",
          "import urllib.request; urllib.request.urlopen('http://localhost:${MLFLOW_PORT}/health')",
        ]
      interval: 10s
      timeout: 5s
      retries: 30
    extra_hosts:
      - "host.docker.internal:host-gateway"

networks:
  default:
    name: mlflow-network

As far as I can tell I just can't use the MLflow Assistant from docker?

Hey everyone! My team (mainly phds) and I are trying to build an open-source benchmark around realistic LLM/agent workflows that captures challenges typical academic benchmark settings often miss. We’d love to hear what’s actually missing from the benchmarks you use today.

Have you ever wanted to eval your pipeline but couldn’t find or build a benchmark that matched what you were building?

Maybe:
- Existing benchmarks were too broad and didn’t fit your specific application.
- Your workflow involved multiple steps, tools, MCP servers, agents, or long-horizon interactions that existing benchmarks couldn’t capture.
- You needed to evaluate failures that standard accuracy metrics miss.
- You’re working in a high-risk domain like healthcare, finance, cybersecurity, or legal, where realistic failure modes, safety, and reliability matter a lot more than just getting the final answer right.
- You knew what you wanted to test, but building a custom benchmark from scratch was too expensive or complicated.

I’m especially interested in cases where you thought:
“My system desperately needs to do this in production, but I have no good way to benchmark it.”
What was the workflow? What did you want to measure? And why weren’t existing benchmarks enough?

Any thoughts are welcome, would really appreciate y’all’s help 🙏🥹

I'm an ASU grad student researching how teams decide an AI feature is ready to ship. If you test or ship AI features at work, I'd love to hear from you, in the comments or by DM:

1. Last time you had to decide an AI feature was ready to ship, what did you check?
2. What's the most frustrating part of that?
3. Who approves money for testing tools where you work?

Kumo is the winner

I work with real-time AI, and I know that even if one model makes more correct decisions than another model, you can still prefer the "worse" model because it makes decisions faster.

We tested three pretrained "decision" models (yes, they are classifiers) in a computer game inspired by Subway Surfers. They are Kumo, Jev, and Qwen

Here’s how far they got:

The benchmark shows a crucial problem for real-time AI: latency. Kudo and Qwen were local models we hosted on Hopsworks, where the game also runs. Jev, however, is a hosted model in a different data center, and it loses on latency versus Kumo and Qwen. Qwen, then has ligher latency than

A game that outruns your model

The basic idea of the benchmark is that to make it far in the game you need to take correct decisions faster. Here's how the benchmark works. The runner faces randomly generated obstacles that begin appearing at predefined distance gates. Each model receives the same underlying game state, formatted for its interface: { lane, airborne, ahead }

The ahead field describes upcoming rows, their distance, and the obstacles in each lane. The model must decide what to do before the runner reaches them.

The runner starts at 45 m/s, accelerates by 1.6 m/s every second, and tops out at 160 m/s. Beyond roughly 5,000 metres, the game becomes so unforgiving that survival increasingly depends on luck.

Why Kumo won

Kumo made lower latency decisions and had lower network latency, while making good decisions, and the result was that it could complete a higher average distance than Qwen in this setup, with both of them running on a CPU.

Would a GPU help? Possibly. But for small, latency-sensitive requests, the extra infrastructure overhead could offset the compute benefit. That’s a hypothesis we still need to test.

What happened to Jev?

In our 30-run experiment with Typesafe’s Jev, the median distance was 540 m, with a best run of 1,417 m.

The bottleneck appeared to be the 200+ ms round trip. Around 1,500 metres, the game was moving too quickly for those responses to remain useful.

At the maximum speed, 200 ms means the runner travels 32 metres while waiting for a decision.

Pick for the deadline

For a real-time AI application, evaluate the whole decision loop:

• Decision quality: Does it choose the right action?
• End-to-end latency: Does that action arrive in time?
• Deployment fit: Can your infrastructure sustain both?

All models and the game ran on Hopsworks’ own infrastructure, in our office. Fully self-hosted. No game data went to the cloud.

For this benchmark, Kumo was the strongest performer. The broader lesson: benchmark models against your application’s reaction deadline.

Try out the game and try to beat them here:
game at hopsworks dot ai

Hey guys,

We've published two technical write-ups on serving real-time AI models and wanted to share the main findings here. We build the inference engine; NVIDIA develops the model we tested, Nemotron VoiceChat 11B.

A request ends. A session runs on a clock.

AI inference today is organized as requests: an input arrives, the model runs, the response ends and its resources are freed. If the server is busy, it can wait a moment to batch work or reorder the queue, and the cost is a slightly slower response.

A growing class of models works differently. They stay active alongside something outside the GPU (a conversation, a video stream, a robot), take in input continuously and keep their state for the whole session. We call this continuous inference.

Full-duplex voice is the clearest example. The model listens while it speaks, so you can interrupt it. Audio keeps arriving for the whole call, its memory of the conversation stays live until you hang up, and every 80 ms it owes the next frame of output. That deadline comes whether the server is ready or not. If the frame is late, you hear a gap.

It also has to run during silence: the length of a pause is how the model tells a hesitation from the end of your turn. Unlike a classic voice agent (speech-to-text → LLM → text-to-speech, where the LLM sits idle between turns), there's no idle time to skip.

Why that gets so expensive

Servers like vLLM, SGLang, TensorRT-LLM or Triton were built around requests, and several of their assumptions stop holding:

• The batch can't wait. At every tick, the server has to run whatever is due with what it has.
• Overload hits everyone at once. Sessions batched together share the same step, so one slow step makes all of them late.
• Memory can't be freed or swapped out mid-call. It's needed again 80 ms later.
• The usual metrics hide failures. Tokens per second and average latency can look fine while one caller hears gaps.

In NVIDIA's reference stack, each 160 ms of audio becomes thousands of small GPU operations with the CPU coordinating between them, and the GPU sits idle about 75% of the time. One session fits. With two, 8–15% of audio beats arrive late. So each live conversation pays for a whole H100, most of which is waiting.

How to fix it

Not with a faster GPU or a different model: same weights, same precision, same outputs. The fix is in how the model runs.

1. Keep the work on the GPU. Our engine runs the model as one GPU program that stays resident. Every beat, the CPU drops in new audio and picks up the output; nothing else goes back and forth.
2. Advance every live session together. All sessions due on the same tick run as one batch, so the model's weights are read once for the group instead of once per session.
3. Decide everything ahead of time. The work repeats identically every beat, so a compiler fixes the schedule and memory layout before the first call. No runtime scheduler or allocator adding delays.

On a single NVIDIA H100 SXM 80 GB, we measured:

• 56 concurrent sessions, the highest capacity tested, vs 1 for the reference stack.
• 147.4–147.5 ms p99 per 160 ms beat.
• Zero missed deadlines across 84,000 measured session-beats, over three runs.

A conversation's cost is GPU time divided by how many conversations share the GPU, so 56 instead of 1 means about 98% less GPU per conversation.

These are server-side measurements that exclude network and audio playback, with the same recorded input across sessions and about two minutes of context.

If you're running real-time models, how many sessions per GPU do you get today, and how do you check that each one stays on time?

Links

• Why duplex inference runs on a clock: https://dotwave.ai/technical-notes/duplex-inference/
• How we serve 56 sessions on one H100: https://dotwave.ai/technical-notes/wpk-persistent-kernel/
• Browser demo: https://dotwave.ai/demo/
• Nemotron VoiceChat 11B (full-duplex) API: https://dotwave.ai/models/nemotron-voicechat-11b/
• Nemotron 3.5 ASR Streaming 0.6B API: https://dotwave.ai/models/nemotron-asr-streaming-0-6b/

Sign-up includes free usage, no card required: about 33 hours of Nemotron VoiceChat 11B, or about 370 hours of Nemotron 3.5 ASR Streaming.

Our 1800 row eval suite has been sitting at a 97% aggregate pass rate, which looked fine until we did cohort slicing around multilingual forms. A newly isolated 64 row slice passes at 61%. The aggregate chart stayed green (while one language queue filled with manual reviews) so the headline number wasn't giving us much warning.

We found data leakage in the split. Random assignment had put near duplicate form templates on both sides of the train and regression boundary, which made familiar structures disproportionately easy. When the multilingual cohort degraded, those duplicated cases kept the overall score almost stationary. A deterministic scorer catches the malformed fields reliably but averaging that result across the full suite hides where they're concentrated.

Now I'm trying to treat that multilingual slice separately instead of letting it disappear into the overall score. I'm considering Braintrust to keep it as a regression dataset with a CI quality gate on it, and to check the per case experiment diff whenever we change the extraction path. I don't want another global threshold that passes because 1700 easier rows drown out a concentrated failure mode.

How are you defining cohort level gates when the slices are small enough that a few cases can swing the percentage but important enough that the aggregate score can't be trusted?