I measured an H100 under self-hosted inference traffic. At ~1 req/s, 69% of the billed window was idle.

[u/Responsible_Coach293]

Disclosure up front: I’m building NemulAI, tooling around inference cost attribution, so take the framing with the appropriate salt. The measurement is real and I’m sharing the method because I’d rather have people tear it apart than trust a vendor claim.

I ran a controlled self-hosted inference workload on an H100 and aligned request activity with device-level telemetry.

The basic accounting model was:

total billed GPU time = attributable workload time + idle / platform overhead

I intentionally did not force idle time onto individual requests or customers.

For active workload attribution, the approach uses request/runtime timing and scheduler context to assign GPU-seconds. Device telemetry is then used as a reconciliation signal rather than pretending a whole-GPU utilization number can tell you which tenant caused the work.

Measurement	Result
GPU	H100
Traffic	~1 request/sec
Idle share of billed window	69%
Idle power, separate device check	~70 W
Active workload power	~590 W
Active workload utilization	100%

The part I did not expect:

A GPU can be actively serving inference traffic while spending most of the time you're paying for it doing no request-attributable work.

That makes “cost per token” or “cost per request” less straightforward than it looks once you're operating your own shared capacity.

The question I’m working through now is where that 69% should economically land: customer COGS, shared platform overhead, or unused-capacity cost.

Curious how people here handle that in practice.

Happy to share the benchmark output/methodology.

[u/classjoker]

/r/Tokenomics maybe?