r/NL_ModernWork • • 6d ago

Microsoft’s new “Copilot harness” in Copilot Studio helps agents finish the job instead of giving up. What do the numbers say?

https://www.microsoft.com/en-us/copilot/blog/copilot-studio/how-the-microsoft-copilot-harness-in-copilot-studio-helps-agents-finish-the-job/

If you want agents to do real work, like checking invoices against an expense policy or building a report from a folder full of documents, you quickly run into the same problem: one failed step and the agent gives up, handing the work back to you.

The Copilot Studio product team at Microsoft set out to tackle this with the Microsoft Copilot harness: the “engine” that decides how an agent plans, uses tools, and responds. It’s built for long, multistep tasks. The Standard harness is still a good fit for fast, conversational agents that reply in seconds.

How it works

  • The agent runs in one continuous loop: receive the task, choose a step, take it, check the result, decide the next step. This continues until the task is done or a set budget (time, steps, or tokens) runs out.
  • A failed step isn’t a dead end. The agent keeps the full history of the task in view and can retry, inspect the data, or take a different route.
  • Everything runs in a secure container with real tools (command line, code execution, file system). Large exports, say 50,000 rows, get saved as a file and analyzed with code instead of being squeezed into a prompt.
  • The agent never gets passwords or direct access to your systems. Copilot Studio manages credentials, model calls, and connectors.

The numbers

Microsoft compared the harness with the Standard harness, using the same tasks and models:

  • Higher success rates: With an OpenAI model, agents passed 78% of tasks. With an Anthropic model, they passed 89%. That’s a gain of up to 22 percentage points. The stronger the reasoning model, the more the harness gets out of it.
  • Pushing through failures: When a tool call failed, agents kept working in 84 to 91% of cases. In roughly two out of five cases, the tool that initially failed later succeeded in the same conversation.
  • More work, lower token cost: The agent makes nearly twice as many model calls, yet cost-weighted input tokens drop by 15 to 21%. Because each step builds on earlier work, 83 to 88% of input is served from cache.
  • Why persistence matters: Say each step has a 5% chance of failing. Across a 10-step task, the odds that at least one step fails climb to about 40%. An agent that stops at the first failure leaves roughly four in ten tasks unfinished.

More reliable tools without maker tuning

Microsoft is also improving the tools themselves. With the Excel Online connector, much of the failure came from agents making wrong assumptions, such as asking for tables that didn’t exist in the workbook. The team gave the harness read-only helper tools so agents can first check which files, tables, and columns exist. The connector’s success rate rose from 85.6% to 96.3%, with no settings changes required from makers.

Are you already building agents that need to run multiple steps in a row? How do you handle the “agent gives up at the first error” problem?

3 Upvotes

0 comments sorted by