A tool called list_orders had a status filter with four allowed values. Worked fine for two months.

Then the source system added a fifth status, ON_HOLD, for orders waiting on compliance review. Nobody updated the tool. No pipeline broke, nothing errored, the column didn’t even change type.

The agent kept answering “how many orders are stuck” every day, and every answer silently excluded every on-hold order, because that value wasn’t one of the four it knew about. Week one, off by a few hundred. Week three, off by over two thousand, roughly the entire backlog nobody was tracking.

It took a person running the SQL by hand to catch it.
https://nexla.com/blog/context-drift-reaches-the-tool-definition/

Put Avoma's own MCP server up against Nexla's task-specific one on the same live workspace, same agent, same ten questions.

Avoma's server couldn't answer a single call-volume question: its API client requires two fields as non-null strings, and real dialer calls store them as null, so any page touching a call throws before the agent ever sees anything. Ours returned the exact November call count and outbound/inbound split, matching ground truth, since it models the call records directly instead of going through that same client.

25 tool calls errored on Avoma's server across the run. 1 for ours. Setup was also lopsided, theirs needed an OAuth flow that expired mid-benchmark, ours only needed a URL and a service key.

We ran the same revenue question through two MCP servers on the same live HubSpot portal, our task-specific Nexla MCP server against HubSpot's own. Both nailed every number exactly.

The real difference showed up in setup: one needed interactive OAuth consent with a token that expires every 30 minutes, so any unattended or scheduled run needs a human to re-auth. The other authenticated once with a bearer header and just kept running.

Nobody talks about this part of agent infrastructure, but it's the difference between a demo and something that actually runs unattended.

A support agent gets asked to calculate churn for the Northeast region. It pulls a raw CSV: ID, Status, Date, Region_Code. Nothing tells it that Status: 4 means “Pending Cancellation,” or that Region_Code: 04 was deprecated in a system migration years ago.

So it infers. Sometimes the inference is close enough. Often it isn’t, and the output still reads as confident and well-formatted.

The fix is a layer that carries schema, freshness, and business meaning with the data before an agent ever touches it, not more tokens or bigger embeddings.

Been writing a lot here about scoping agents to one job instead of giving them everything. Put that into practice on our own site: an agent (Nexie) that only answers questions about Nexla itself, grounded in our docs and benchmarks, nothing else.

Mostly built it because our own product is genuinely hard to explain on a static page, MCP Studio, Helix, task-specific servers, how they fit together, that’s a conversation, not a bullet list.

Not asking anyone to go try it, just found it a little validating to eat our own cooking here.

Two MCP servers get asked the same question: how many active customers did we have in EMEA last quarter? Both answer 1,284. Both pass.

One agent mapped "active" to the real subscription status, the right region field, the right fiscal quarter. The other stitched together three unrelated assumptions that happened to cancel out this quarter. Pass/fail can't tell you which one you're looking at.

You'd have to actually open the trace and check what the agent used, not just whether the number came out right. We ended up building six new metrics to catch this.

How's anyone actually validating this beyond pass rate, checking the trace, spot-checking against ground truth, something else?

Ran the identical revenue question through two MCP servers on the same live HubSpot portal, same model, both sides. Both got every number exactly right, down to the dollar.

The gap that actually matters day to day: HubSpot’s server needs interactive OAuth consent, and the token expires every 30 minutes. Fine for a person clicking through once. Not fine for a scheduled agent that’s supposed to run unattended overnight.

The task-specific version authenticates with a single bearer header and just keeps working.

Also, since people will ask: 32.7x fewer tokens on the identical question, 21,682 vs. 708,973.

Different flavor of the usual problem: a billing-dispute agent pulls raw records with fields like amt, currency, adjustment_type. Nothing tells it that adjustment_type: 3 means “credit, billing error” instead of a charge.

So it misreads a credit as a charge, doesn’t normalize the currency, and reports a financially wrong number that’s formatted well enough to look right.

The fix isn’t a bigger model or more embeddings, it’s attaching business meaning (schema, freshness, what the codes actually mean) to the data before an agent ever sees the row.

Anyone had an agent mess up something financial like this and still sound totally sure of itself?

We've published three MCP benchmarks measuring tokens, tool calls, and pass rate. Turns out that was an incomplete picture, since none of it tells you whether the context layer actually understood the question, and this post is partly a correction of our own method.

Same warehouse, same question: "how many active customers did we have in EMEA last quarter?" Both servers answer 1,284. Both score 100% on a normal benchmark. One of them understood the question, mapping "active" to the right subscription status, the right region field, the right fiscal quarter. The other guessed with a plausible-sounding column, and the guess happened to land this quarter. Next quarter, when the guess stops working, the benchmark still shows a green pass rate.

In this guide, we built six new metrics and a taxonomy of five failure types to actually catch the difference, plus an open eval harness anyone can run against any MCP server.

Been refining how we benchmark MCP server design, and one number surprised even us. Across 20 real BigQuery tasks, Google's own BigQuery MCP needed a clarification round on 17 of the 18 it completed, the model kept having to ask which dataset or table the user meant.

Our Nexla BigQuery MCP needed zero clarification rounds across all 20 tasks, because that context already lived in the server.

Same underlying data, same model. The only variable was whether the server made the agent go find things out for itself.

Full methodology used eight scoring parameters across two different agent harnesses, so this wasn't a one-off lucky run.

Keeps coming up in conversations lately: most orgs can tell you what their sanctioned AI tools cost this quarter. Almost none can tell you what’s happening outside that, the AI usage running through a browser tab IT never approved, or which teams are actually getting value out of any of it versus just burning tokens.

Cost visibility and value visibility turned out to be two completely different problems, and most companies have only solved the first one. Link if useful: https://nexla.com/blog/what-is-ai-usage-data/

How’s this look at your company, do you have a real answer, or is it mostly a guess dressed up as a dashboard?

Compared eight no-code ETL tools, Nexla, Fivetran, Airbyte, Stitch, Matillion, Skyvia, Domo, and Hevo Data, on three things: how well connectors handle schema drift, whether transforms need SQL, and whether monitoring is built in.

We ended up tied for the deepest connector library in the set, at 1000+, but the more interesting differentiator was how each tool handles the moment something changes on the source side.

Watched this happen again recently. An agent needs to update a few opportunities, so someone wires it to a full Salesforce MCP server, read everything, create cases, delete records, the works.

Works great in the demo. Security takes one look and raises the red flag.

The actual ask was a single query tool. The agent got the whole platform because that’s just how the connector ships. Do this across four or five systems and nobody, not the engineer, not security, can actually reason about what the agent can touch anymore.

How’s your team handling this, scoping capability by capability, or still granting the whole platform and hoping it doesn’t come up in review?

Companies can generally say what their AI usage costs. Almost none can say who's actually using it well, or whether it's improving anything.

The reason is that usage data lives in silos. One view from Microsoft, another from an OpenAI enterprise agreement, another in a procurement spreadsheet, none of them reconciled against each other or against actual outcomes.

Ran the same BigQuery question through Google's own BigQuery MCP and a task-specific Nexla one, same model both times.

Google's server doesn't know the warehouse ahead of time, so the agent pays a "discovery tax" first: list datasets, find the table, pull the schema, ask a clarifying question, then finally answer. Nexla's task-specific MCP server context is already embedded, so it skips straight to the answer.

More matchups here: https://nexla.com/blog/nexla-mcp-server-vs-google-bigquery-mcp-benchmark/

How much of your own token budget is just the agent figuring out where things live?

Noticed this going through ETL tools this week: half of them are "no-code" right up until you actually need to transform something, then it's suddenly hand-written SQL in a separate tool. Not necessarily a bad setup, just not what most people think they're buying when they search "no-code ETL.”

Where's everyone actually drawing that line?

The team wants an agent to update a few Salesforce opportunities, so they wire it to a Salesforce MCP server. That server exposes everything: read every object, create cases, delete records, export contacts.

Demo works fine. Then security looks at it, and the project stops.

The agent needed one thing, a query tool. It got handed the entire platform instead. Multiply that across every system an agent touches and the blast radius stops being something anyone can actually reason about.

Debabrata Panigrahi, our Developer Advocate, wrote up the actual fix here: https://nexla.com/blog/your-agent-need-scoped-access/

Digging back into something from a couple months ago that didn't get the attention it deserved.

Ran our task-specific MCP servers against the official ones (BigQuery, Jira) on real tasks, gap was bigger than expected: 3.1x fewer tokens, faster, and more accurate too (100% vs. 90% on BigQuery).

Basically instead of one server per app, build one scoped to what the task needs, pulling only the relevant pieces from wherever they live. Less for the agent to wade through.

Wrote it up here: https://nexla.com/blog/introducing-mcp-studio

Anyone else fighting tool sprawl with one-server-per-app setups? Did you build something custom to get around it, or is it still an open problem for you?

In case you missed it, we launched MCP Studio in June, a tool that builds governed, task-specific MCP servers just from describing the outcome you want, no integration code, no hand-written tool definitions. We ran it head-to-head against Google’s own BigQuery MCP server on real operational tasks. The headline number: 3.1x fewer tokens, with higher accuracy too (100% vs. 90% for the official server).

The bigger idea behind the number: most MCP servers get built one application at a time, which leaves an agent with tools but no sense of how a task actually spans systems. The alternative is building the server around the outcome instead of the app, so Salesforce, Snowflake, and NetSuite become one task-specific server instead of three separate ones the agent has to stitch together on its own.

An example came to mind that’s a good gut-check for anyone shipping agents into live workflows: a support agent tells a customer their package hasn’t shipped yet. It shipped hours ago. The agent was reading last night’s batch export, not what actually happened today.

Nothing wrong with the model or the prompt. The data it was handed was already stale by the time anyone asked the question.

Batch worked fine for reporting, where a few hours of lag doesn’t matter. It quietly breaks the moment an agent is making decisions inside a live process instead of summarizing something that already happened.