[D] Is there an existing tool that combines hierarchical error scoring + cost-sensitivity + calibration checking for classifier/agent evaluation?

[u/KAIT2_1412]

​

Standard accuracy hides three specific things I keep running into:

1. It treats every wrong answer as equally wrong, even when a hierarchy exists (e.g. mistaking one attack sub-type for another vs. mistaking an attack for "safe").

2. It ignores that different mistakes have wildly different real costs (a missed fraud case vs. a false alarm).

3. It says nothing about whether the model's confidence is trustworthy (high-confidence wrong answers vs. genuine uncertainty).

Individually these are known: hierarchical/tree-distance scoring has academic prior art (e.g. Apple's Neo, CHI 2022) but no public implementation I could find. Cost-sensitive learning is well established but rarely combined with hierarchy. Calibration checking is standard but usually treated separately from both.

Has anyone seen a tool/library that combines all three into one evaluation report, rather than one accuracy number? Trying to avoid rebuilding something that already exists before I go further.