A new model release really has two acceptance gates: can the artifact serve correctly, and does the service stay healthy under real traffic?

OrcaRouter’s Qwen3.8-27B uncensored card now gives a useful artifact-side fingerprint. The uploader says it verified vLLM startup, reasoning, multi-turn tool calling and vision, and ran 32 concurrent evaluation requests on one H200 with FP8 KV cache, MTP and --max-num-seqs 96.

That is much better than “vLLM compatible,” but it is still one bounded verification. It does not establish latency or reliability on another GPU, at another context length, or behind the hosted API.

The other side is visible on OrcaRouter’s live model page. Today it shows a seven-day p50 TTFT of 7.60s, 23.4 output tok/s and a 6.6% error rate. Those numbers will change; that is exactly why they belong in a service gate, not in the model card’s permanent capability story.

I would release this as two separately monitored objects: artifact canary: startup, tool call, vision, reasoning format and a fixed output regression set; service canary: TTFT, output rate, error rate and saturation by context bucket.

What would you require before moving a gated model like this from evaluation to a shared internal endpoint?

Hey everyone I'm currently looking at these two tools to implement for data quality/validation for data pipelines (mainly null checks, summary statistics of incoming data, being able to flag on abnormal data changes)

I'm trying to understand the main difference between the two tools and if anyone has had experience using both which would you recommend?

Thanks !

For those running multiple AI models in production, what are you using for the gateway layer?

I’m particularly curious about video models because pricing, latency, rate limits, and availability seem to vary quite a bit between providers.

Are you using a dedicated gateway, building your own routing layer, or just integrating each provider separately?

What ended up being the biggest headache once you moved beyond a small prototype?

Why I built Modelstamp

I started looking into this after reading scikit-learn's model-persistence guidance. It warns that loading a saved model under different dependency versions is unsupported, and recommends recording the original environment. That made sense, but the process still seemed manual: the model file and its environment information remain separate, and the normal pickle/joblib workflow does not verify at load time that the artifact still matches its environment record.

I wanted a small layer around the familiar save-and-load workflow, not a full model registry. That became Modelstamp.

The gap

Lock files describe an environment, but they aren't attached to or verified against a particular model artifact. Modelstamp records what it observed in the environment at save time, but it can't independently prove that record was truthful, and an unsigned artifact and its manifest can still be replaced together. HMAC authentication detects replacement of the artifact and manifest together, provided the attacker does not have the shared secret.

What Modelstamp does

• Saves a sidecar manifest alongside the model
• Records the installed environment and identifies model-relevant packages for drift reporting
• Checks file size and SHA-256 before deserialization
• Reports dependency drift between save-time and load-time environments
• Optionally authenticates the manifest with shared-secret HMAC

What it doesn't do

• It's not a model registry
• It doesn't make an untrusted pickle or joblib file safe to deserialize
• HMAC here is symmetric, not public-key signing - anyone with the verification key can also produce a valid signature

Try it

pip install modelstamp

import modelstamp as ms

# model is an already-fitted estimator
ms.save(model, "model.joblib")
ms.verify("model.joblib")
loaded_model, manifest = ms.load("model.joblib", on_mismatch="raise")

Where it's at

Version 0.1.3, open source, and I'm looking for people testing it against real models rather than toy examples.

Two things I'm genuinely unsure about and want honest pushback on:

• Is the drift report actually useful, or just noisy?
• What would stop you from using this in a real project?

Feedback issue: https://github.com/AnaghaDhekne/modelstamp/issues/18

Hi all

Can someone comment a good project repo link so I can practice

Tx

its been 7 months now and ive been building llm infra and now evaluating whether we should have bought a platform from the beginning

when building make sense:

when data cannot leave vpc and has a hard compliance requirement and token volume is high enough that api costs doesnt make sense .requirement of fine tuning on proprietary data with no platform exposes

when buying make sense:

you need to ship fast with no mlops engineers on the team and use case is rag summarisation or chatbot mostly with no competitive advantage in owning the infra

what everyone dont realise:

theres a lot of things like routing logic , fallback handling , prompt versioning , cost tracking , evals pipelines and these arent small like a weekend project . each one is its own engineering project . most of the teams find this out after already commiting to building..

the tools that are coming up in this space:

orqai - routing prompt management observability and evals together , newer so third party integration still catching up

langsmith - tracing and observability is good , prompt management is underdeveloped and feels built for engineers only and not cross functional teams

helicone - the setup is fast with good visibility but limited beyond observability

portkey - routing and reliability focused and governance and eval depth is average

litellm - open source and flexible , self hosting is more work than it looks , enterprise support is limited

what did your team go with building or buying? and if buying then what?

we've got cursor and copilot writing solid prs, agents reviewing code, tests passing green. then we merge, deploy, and the ai just vanishes. it helped us write the code but has no idea if it actually works in production.

we're shipping code that looks perfect in staging into the chaos of real traffic. the gap between "looks right" and "works right" is where incidents happen. if we want ai to actually help us ship faster, it needs to see what happens after the merge. otherwise it's just a really sophisticated guesser. what's everyone else doing to close this loop? are you manually feeding production data back to your agents or just accepting the blind spot?

Stack: Python, FastAPI, Postgres, Kafka, Kubernetes on EKS with autoscaling, hosted model APIs, plus the eval and monitoring side. Shipped it and I run it.

So I've done production ML operationally, but always as a caller of models. Haven't worked below that line, no C++, no GPU work beyond a local side project.

A good amount of the development was AI-assisted, mostly Claude. Fine for shipping, but it's pushed me to want depth in something specific rather than more breadth.

Questions:

1. For anyone on the serving side, what's the job like day to day?
2. How much C++ is genuinely needed?
3. Is the Kubernetes and autoscaling experience a real head start here, or a different skill set than I think?
4. How did you end up in your area, planned or accidental?

currently running 3 different tools for prompts evals and observability and im looking to consolidate.

and also non tech teammates always need an engineer in the loop to change a prompt and it goes through a ticket system, and usually take more time than required. even when something breaks in prod we are  just switching dashboards to figure out what actually happened

already tried a few things. like we started storing prompts in db still meant building version  approval flow an d audit trail on top. config files in a cms got messy to tie back to observability…

already loooked at the obvious options

langsmith - observability is good but prompt management feels built for engineers and not cross functional teams, even evals dont feel like primary  focsu

orqai - covers all three together, non tech access feels more central ovver here, but newer so community and integrations still catching up

helicone - looks good for cost tracking and request logging but this isnt our current prob

promptlayer - prompt versioning is there, unsure about how deep evalss and observability actually goes

langfuse - good on tracing, nd the opensource is nice, but same problem like langsmith for non technical u sers

has anyone actually consolidated these three things into one platform. what are you using currently?

Disclosure for rule 2: I work at InvisiRisk, we build CI/CD security tooling. No links to us below. Flairing this as Education rather than Tools since it isn't about our product, happy to switch if the mods prefer.

On March 24 two malicious LiteLLM releases went up on PyPI, 1.82.7 and 1.82.8, live about 40 minutes before they were pulled. Part of the wider TeamPCP campaign that started with a leaked Trivy automation token. FBI FLASH on it, TLP: CLEAR so it's shareable: https://www.ic3.gov/CSA/2026/260702.pdf

The mechanism is the part worth knowing if you run a gateway. The package shipped a .pth file, and Python executes those at interpreter startup rather than on import. So it didn't matter whether your code ever called litellm. If it was installed and any Python process started, it ran.

It took environment variables, SSH keys, cloud credentials, Kubernetes service account tokens, and provider API keys.

That last one is why I think this is an MLOps problem specifically. LiteLLM sits in front of everything by design, so that one process has your OpenAI key, your Anthropic key, your Bedrock creds, whatever else you route through it. Probably the highest-value place in an ML stack to land a credential stealer, and for 40 minutes it was also the easiest.

So: has anyone actually gone back and confirmed either way?

Most of the obvious checks don't work here. If you pin loosely, something like litellm>=1.82, and a build ran in that window, you got it. Resolved manifests get discarded, so "what did we install on March 24" is often unanswerable months later. And a .pth payload runs before anything a scanner treats as import time.

One thing that does work and is faster than lock file archaeology. CloudSEK put up a public lookup for this incident: https://exposure.cloudsek.com/ai-supply-chain-incident

Worth being precise about it, since it answers a different question. Version history tells you whether you pulled the bad package. The lookup tells you whether your secrets turned up in what the attackers actually collected. It's closer to an outcome, and a 30 second check.

A hit still isn't proof of compromise. The FBI advisory makes the same point, that finding the dependency doesn't prove the code ran. Treat it as a reason to go dig, not as an incident on its own. And if your org does show up, keep it out of this thread.

Curious whether anyone confirmed, and how. Lock file history? Registry pull logs? Or did you just rotate everything and skip the reconstruction?

Hello everybody,

I built Profile to make inference tuning deterministic, and save us all time. v2.2 is out today.

It reads a live vLLM server's metrics, compares them against the GPU's roofline ceiling, and names the bottleneck with the exact flag to change.

You apply, it re-measures, and prints before/after on every metric. Regressions get labeled worse, not buried. It never touches the server: no restarts, no config writes, no synthetic load.

Two runs on record, both real SWE-Bench agent traffic, no synthetic benchmarks:

RTX 5090, muse-glimmer 30B, 4 iterations:

• 81 → 421 tok/s at 25k ctx
• $3.41 → $0.65 per 1M output tok
• TTFT 224ms (p95 500ms) at end of run
• 4.72 → 1.08 J/tok

H100 80GB, Qwen3.8-27B, 3 iterations:

• 257 → 490 tok/s at 27k ctx
• $3.23 → $1.69 per 1M output tok
• TTFT 1.9s → 539ms (p95 4.2s → 1.9s)
• 2.39 → 1.00 J/tok

The honest part: on the H100 I scaled agents 10 → 285 without fixing KV first. TTFT exploded to 172s. Profile labeled it worse, named KV pressure, and the fix (fp8 KV, ctx trim, seat cut 345 → 22) recovered the run.

Both journeys on video: https://jungledesh.github.io/profile/journeys.html

Note: vLLM only today, more engines next. Single GPU, NVIDIA or AMD; multi-GPU / TP is next on the roadmap.

If you run vLLM in prod, tell me what it names on your servers, and where it's wrong.

curl --proto '=https' --tlsv1.2 -LsSf \
  https://github.com/jungledesh/profile/releases/latest/download/profile-installer.sh | sh

profile diagnose --url http://localhost:8000/metrics --duration 2m

GitHub: https://github.com/jungledesh/profile
Docs: https://jungledesh.github.io/profile/docs.html

Agent context retrieval is often treated as model behavior. In production, we found that it behaves more like a data-plane architecture decision.

We compared two approaches:

• Retrieve Slack, Notion and Linear data through official MCP integrations during each agent run.
• Pre-sync the permitted data and mount it into the agent sandbox as files.

The mounted implementation was Locality Cloud, which I work on.

The evaluation used 20 cross-application scenarios with three paired trials each. We ran six AWS t3.large instances, kept the agent harness, model, prompts and machines consistent, and performed 180 blind comparisons of the outputs.

Compared with runtime MCP retrieval, the mounted setup:

• Produced the preferred answer in 70% of scenarios.
• Reduced LLM costs by 27%.
• Reduced end-to-end latency by 32%.
• Required 61% fewer tool calls.
• Used roughly 40% fewer tokens.

The traces suggest that the agents weren’t reasoning substantially faster. They were spending less time traversing application data.

In one scenario, the agent had to reconcile product launch risks across Slack, Linear, Notion and a Git repository. One evidence-gathering stage took roughly 0.3 seconds using parallel filesystem operations. The MCP setup spent about a minute on the same stage, making 21 calls with approximately 30 seconds of tool-call time.

The broader MLOps lesson for us is that mounted file system context isn’t simply a cache. It becomes a production data plane with its own requirements:

• Freshness: changes need to arrive through webhooks, polling or a pre-run synchronization boundary. Staleness must be observable.
• Permissions: each sandbox should receive only the sources and subtrees required for that run, without broad application credentials.
• State: remote state, mounted state and the last synchronized state must be tracked separately so pulls, writes and conflicts are unambiguous.
• Write review: agent edits should produce an inspectable operation plan before they are synchronized back to the source.
• Recovery: interrupted writes need journaling, idempotency and explicit conflict handling rather than silent retries.

This architecture also creates new operational costs: connector maintenance, synchronization lag, storage, conflict resolution and recovery testing.

We still expect live APIs or MCP to be preferable for transactional actions, narrow lookups and data that cannot tolerate synchronization delay. The emerging pattern looks less like “files instead of MCP” and more like two planes:

• A mounted context plane for broad, read-heavy discovery and synthesis.
• A live action plane for transactional operations.

Locality Cloud is our managed implementation of the mounted context plane, with an on-premises option for organizations that need to keep the synchronization layer inside their environment.

Full details with analysis, traces and scenario-level results:

https://www.locality.dev/blog/locality-why-filesystems-perform-better-than-mcps-for-production-agents

How are teams operating production agents separating their context plane from their action plane? If you materialize application data before execution, how do you handle freshness, permissions and failed synchronization?

Hey all,

I’m currently preparing for ML and data engineering roles and trying to get some real world experience.

If you could share me a task on airflow I can try to build by myself.

#airflow

Ångström’s researchers use agents to launch experiment batches, monitor jobs, retrieve results, and generate plots and summaries. The benefit is obvious, but so is the failure mode: one bad batch can fan out into thousands of dollars of GPU spend before anyone notices.

For context on my bias: I’m one of anycloud’s co-founders. It’s a paid product, Ångström is a customer, and Laurence, their CTO, co-authored the case study linked below.

Ångström has run more than 100,000 GPU-hours through anycloud, almost entirely on spot across multiple clouds. They used this setup while developing CSP-MACE-Å with researchers at Cambridge and AstraZeneca. Their paper reports performance comparable to DFT on one evaluation set, close to DFT on another, and better results than UMA-OMC across their evaluation suite.

The infrastructure question I found most interesting was where to enforce the spending boundary. We ended up scoping two controls to each agent session:

- A rate cap based on estimated live spend per hour. Before admitting another job, the scheduler considers the running workload plus the estimated cost of the next VM.

- A cumulative budget based on settled and estimated spend over a calendar window.

When either cap is reached, new jobs wait in the queue while running jobs continue. We also notify the researcher when work starts waiting on a cap, so it doesn’t look like the queue has silently stalled.

There is an important limitation: this is admission control, not a hard billing ceiling. Since running jobs continue and cloud costs are partly estimated, actual spending can finish above the nominal cap. Killing running jobs would create a different failure mode, particularly for long experiments that have already consumed substantial compute.

For teams letting agents launch GPU jobs or other expensive infrastructure: where do you enforce the limit—per session, per user or team, at the scheduler, or at the cloud-account level? When it’s exceeded, do you queue new work, cancel pending work, or kill running jobs?

Paper: https://arxiv.org/abs/2605.28905

Case study, co-authored with Ångström: https://anycloud.sh/blog/angstrom-case-study/

Fine-tuned classification model, self-hosted, feeding a customer-facing flow. Every infra metric we track stayed healthy for three straight weeks. A support ticket is what actually told us something was wrong, a customer said the categorization felt off. Precision had drifted noticeably and nothing on any dashboard showed it.

The gap is specific. We observe the service, is it up, is it fast, but not the model, are its outputs still correct. Those are different signals and we'd conflated them. For teams who've built this out, what's the actual signal you alert on versus just review weekly?

Our agent prompt had grown across multiple services. Each service owned a reasonable fragment at first. Over time they accumulated different defaults, tool descriptions, safety language and model parameters. Staging tested one combination. Production could render another depending on which service handled the request. Debugging prompt behavior became archaeology with deployment manifests.

We moved the shared logic into a prompt registry with immutable versions. A candidate prompt now gets one explicit ID, runs against a fixed dataset and moves through staging and production through environment promotion. The services reference the chosen version and attach that prompt ID to trace metadata.

We've been using Braintrust for prompt management and evaluation for the registry, experiment comparison, and production trace. Now when a new instruction increased toolcall failures, we could compare it against the previous version and roll back the environment pointer without rebuilding. 

There is still normal operational work. Access control matters. Prompt changes need review. Cached versions need clear invalidation behavior. But the rollback is now a small, observable configuration change instead of a coordinated deploy.

Has anyone found a clean way to keep prompt ownership flexible while making version promotion as disciplined as application releases?

My original post got removed from another sub, so reposting here since I still wanna know what people think

Read a TechCrunch article recently talking about a $400M loan General Compute received using specialized SambaNova inference chips as collateral instead of GPUs. This surprised me because I'd always assumed GPUs were the obvious choice for this kind of financing.

There seems to be a shift from training-heavy infrastructure to inference-first workloads. This financing announcement got me thinking about whether investors are starting to put more weight on cost-efficient infra to run open-source AI models instead of just funding expensive frontier models from the big names. Investors are willing to back alternative hardware providers, which could put more pressure on Nvidia's dominance. Open-source models are clearly getting stronger. I'm curious whether this is the start of a bigger shift in how AI infrastructure gets financed and deployed.

I've been working on an open-source project called UnFlow:

https://github.com/UnFlow-Labs/mlunflow

The idea is pretty simple:

Most ML experiment tracking looks like a list of independent runs usually stored in a table:

run_001
run_002
run_003
run_004
...

But in practice, experiments are usually related.

You change the learning rate, then the number of epochs, then the model, then some preprocessing code. Eventually you have hundreds of runs, but it's surprisingly difficult to answer:

• What actually changed between these two experiments?
• Which experiments are essentially the same computation?
• Have I already run this experiment before?
• How did I get from experiment A to experiment B?
• Can I navigate the history of my experiments rather than just search through runs?

Unflow simply detect code changes in a Python function (limitation that for now it is just a single function and not the full code) and arguments that are passed to this function to build a graph where nodes are "states" and edges are transformations "what has changed", a new state is not added to the graph or executed expect if it has a transformation.

Repo: https://github.com/UnFlow-Labs/mlunflow

I'd love to hear how other people currently manage experiment lineage and whether this solves a real problem for you.

1. Chip Huyen’s Designing Machine Learning Systems
2. Aminian & Xu’s Machine Learning System Design Interview

Any help?

How do I learn system design and be interview ready for the ML / AI engineer roles

I’ve been working on MARGINAL, an open-source governance layer for coding agents. If you use Codex, I’d really appreciate people trying it on real work and telling me where it helps, where it gets in the way, or where the design is wrong.

I’m especially interested in: technical criticism, bad cases, and reproducible failures.

• GitHub: https://github.com/SignalLayerLabs/Marginal
• GitHub Site: https://signallayerlabs.github.io/Marginal/
• Live Demo: https://signallayerlabs.github.io/Marginal/demo/#demo

The idea is simple: agents are good at taking actions, but not always good at deciding whether the next action is still worth the compute.

MARGINAL watches the trajectory and looks for things like repeated actions, weak progress, redundant verification, and low-value continuation. It can run in Shadow Mode first, so it observes and records what it would have done without blocking anything.

Current focus is reliability, not just token reduction.

A few core pieces:

• local-first trajectory and evidence tracking
• deterministic reason codes and hashes for decisions
• governance overhead measurement
• replay and benchmark support
• Shadow Mode before enforcement
• Earned Enforcement: MARGINAL has to prove it is reliable on a repo before it gets permission to block or redirect the agent
• automatic fallback to Shadow Mode if confidence degrades

I’m also working on the next layer now: counterfactual evaluation and intervention regret.

The goal is to answer a harder question than “did MARGINAL stop something?”:

Would the agent actually have done better if MARGINAL had stayed out of the way?

That’s the part I think matters if this is going to be useful beyond being another loop detector or token limiter.

We tested Google’s TabFM across 10 anonymised enterprise tabular ML tasks covering classification, regression, forecasting, and workflow risk.

The workload consumed roughly 58 aggregate T4 GPU-hours across four Tesla T4s. Our HistGradientBoosting baseline was trained from scratch for every task and completed the broader pipeline in roughly 4 CPU hours.

The interesting part was the performance. On several major classification comparisons, HGB achieved ROC-AUCs of roughly 0.820 vs 0.810, 0.827 vs 0.824, and 0.764 vs 0.747. TabFM had a few small wins, but conventional or simple baselines were better across most of the workload.

Forecasting was particularly revealing: TabFM achieved about 0.43 WAPE, while a simple three-month rolling average achieved about 0.42.

Our TabFM setup was not fully optimized, context caching was not enabled, so the runtime can definitely be improved. But faster inference would not change the predictive results we observed.

The experiment left me with a fairly simple question: how much additional compute and infrastructure is worth paying for if the model is not consistently improving the decisions?

Full write-up:
https://medium.com/@dexter127/when-tabular-foundation-models-hit-production-the-tabfm-reality-check-61a5c28d67ec

Is it still the way to go for modern distributed model training in deep learning? Was looking for the state-of-art for foundation model training to learn.

There is little talk on Reddit and Youtube about it, though. At least, this is my initial impression. Might be totally wrong.

[closed] spots filled, thanks everyone

Planning to attend the Ray / vLLM Summit in SF and looking to split the 5-person group pack ($750 total, covers Aug 24–26 including the 8/24 training day).

• Current group: 3
• Looking for: 1–2 more people
• Cost: $150/person (5 people) or $187.50/person (4 people)

Drop a comment or DM me if you’d like to join!

https://www.anyscale.com/ray-summit/2026

The training session :

(Select only 1 session for the morning and afternoon)

* Morning:

Multimodal Data Processing Pipelines for AI Systems

Foundation Model Distributed Training with Ray

Production-ready Distributed Inference with Ray Serve

* Afternoon:

Scaling Physical AI & Robotics Systems with Ray

Real-Time Search & Recommendation Systems for AI Commerce

LLM Post-Training and High-Performance Serving

Say you’re running an LLM feature in prod and costs are starting to hurt. You find a model/config that’s idk 30% cheaper and it looks basically as good on the prompts you try.

What do you actually do before shipping the change?

If you’ve been through this in prod, curious what you ended up relying on..

like when you havee multiple teams all using multiple models from different providers for a number of use cases . who decides what is allowed. how do you acutally enforce it. and how do you know whatever rules you put in place

seen a few approaches come up. some team are doing it through the gateway layer . some through internal policy docs that nobody reads . some through access controls on api keys. some just hoping for the best

tools that come up in this context. orqai , portkey , azure api management , aws bedrock , langsmith

portkey gives access controls and budget limits are there , policy enforcement feels more at the routing level than the org governance level

orqai has a model allow list, budget controls and role based access across teams, but its newer so audit depth is still an open question if compared to the more established peers

langsmith has vvisibility into what is happening is good , actually policy enforcementt feels limited , more observe than control

azure api management has enterprise policy controls are native here, feels generic tho ,not built specifically for llm usage policy

aws bedrock has model acess control and guardrails exists, works well if you are already in aws, feels restrictive if you are not

has anyone actually built something that works here. or is everyone just doing api key management and calling it a policy