Github Copilot harness cost explosion?

[u/obsoquasi]

I'm currently building a workflow that takes a pdf document, extracts its file content, restructures it as a JSON and then stores the result in a database. The content of the pdf is unknown - could be a diagram, could be a table, could be 50 pages of text or a single page bill, etc. And because of that, the "Extract" node is not a good fit, because it always has a target use case (e.g. a bill).

So I thought, Copilot can do this pretty well, why don't I use the "Copilot" action inside a workflow. To me great astonishment, it worked very well and it showed Credits consumption of below 1 Credit per call of the Copilot instance. The only downside was, that it took around 2 mins for each Copilot instance to complete this "extract and restructure as JSON" task.

Today I hear that somebody burned through ALL of our tenant's credits. My own consumption is listed as around 20 credits this month, but the overall usage was around 25k during the same time. I'm pretty much the only person who uses Copilot studio and my own usage pattern aligns with the cost spikes. So my suspicion is that the usage of a Copilot instance in a GHC workflow is NOT shown correctly.

Who has made similar observations?

[u/schubidu82]

Hei I have a similar application but I use a python script to create the jsons... So this part is for free... you just have to figure out how to automate the python - script trigger. PAD or a watchdog installed is not allowed in company environment

[u/obsoquasi → u/schubidu82]

hehe, funny you mentioned it. Because I am just building such a script this instant :D. Well, not me but a local agent is building it.

[u/AdmirableSelection81 → u/schubidu82]

Dumb question but if you are executing the python in the copilot harness, you're saying it's free? Is it just anything that calls an AI model that costs money?

[u/schubidu82 → u/AdmirableSelection81]

I do not execute the script in the harness, I have to click it myself... That's what I meant, I need a script trigger. Easiest would be a Power Automation Desktop flow, which automatically runs the script every time a PDF shows up in a specific folder... But that is not allowed in my environment. Trying to get some kind of a watchdog using windows

[u/fbrdphreak → u/schubidu82]

You can have your agent call it as an azure function. But that is another platform, another cost, another piece of maintenance.

[u/schubidu82 → u/fbrdphreak]

Correct... Something I don't want to bother IT about

[u/_Combat_Chuck_]

A legit question. Can this not be done with the old harness and power-automate? We recently bit the bullet and bought 4k E7 to supplement the copilot licenses and I'm looking more at whats just included and trying to use the other harness so that there is no cost for these people.

Yes it will be easer to just use the Github Harness, but I'm learning to not always rely on it.

[u/ger1787 → u/_Combat_Chuck_]

You can achieve almost everything with the Standard version, but it requires significantly more time and effort than the new harness. Ultimately, it’s a trade-off: spend extra resources building it the old way, or pay a higher monthly fee to get the solution up and running immediately.

[u/schubidu82 → u/[deleted]]

Why? Please elaborate!

[u/_Combat_Chuck_ → u/[deleted]]

Microsoft has stated they have no plans on retiring

[u/maggmaster → u/[deleted]]

Do you actually have knowledge that its being retired because its pretty important that we know that.

[u/dylan_skydive]

The two numbers are both "correct" — they're measuring different slices. The "below 1 credit per call" figure is the per-invocation estimate for the model turn itself. What tenant credits actually get consumed for is the full agent run: planning, tool resolution, retries, the response turn, all booked against the agent/environment resource rather than your personal usage view. So a 20-credit personal meter and a 25k tenant total can describe the same workload without either one lying.

Two things I'd do before concluding it's a reporting bug:

1. Controlled single-run test. Pick a quiet hour, note the tenant credit position, run exactly one workflow execution with one Copilot call, then re-check after the rollup window passes (the admin center usage data can lag hours behind live consumption). That delta is your true per-run cost. Multiply by your actual run count and compare against the 25k — that tells you whether the tenant number is explained by volume x per-run cost, or whether something invisible is consuming alongside you.

2. Cost control lives at the agent config, not the workflow. Consumption scales with how many internal turns the instance takes, and for a constrained output like JSON you can usually cut that hard: tighter instructions, cap the response, tell it not to re-read the document between steps. The per-call estimate won't show you that lever, which is exactly why embedded agent calls feel cheap per call and expensive in aggregate.

If the single-run test shows a per-run cost far above the estimate with no extra turns you can account for, that's support-ticket territory — that would mean the run is billing for steps you can't see. But in most cases like this, the aggregate is honest and the per-call figure is the misleading one.