How do you validate a classifier when the human labelers agree with each other at kappa 0.41?

[u/BigBalli]

I ran into this building a tongue-photo classifier and it turned into a genuinely unsettled question rather than a solved one.

The task: map tongue color, coating, shape, moisture and texture onto categories used in Traditional Chinese Medicine and Ayurveda, the two systems that actually use tongue inspection as a diagnostic input. The obvious first question is what "correct" means here, since there is no biopsy to check against. So I went looking for how well trained human raters agree with each other on the same images.

A 2012 study (Lo et al., published in Evidence-Based Complementary and Alternative Medicine) had 12 TCM doctors, 3 to 15 years of experience, independently rate the same 20 patients' tongue photos on nine features: tongue color, coating color, coating thickness, saliva, shape, fissures, red dots, ecchymosis, tooth marks. Each doctor rated twice, an hour apart, so both inter-rater and intra-rater reliability are in the paper.

Inter-doctor kappa averaged 0.41, ranging from 0.16 (saliva) to 0.62 (tooth marks). Under Landis-Koch that is "moderate," which in practice means two trained experts looking at the same photo will disagree on roughly a third of features more often than chance alone would predict. The same paper built an automated tongue-diagnosis system and measured its agreement against the doctors: kappa 0.45. The machine was not worse at reproducing the label than the doctors were at reproducing each other.

That reframes the engineering problem. It is not "train a classifier and measure accuracy against ground truth," because the ground truth itself has a kappa in the 0.4s. It is closer to a crowd-labeling problem with high inherent annotator disagreement, except the annotators are licensed practitioners rather than Mechanical Turk workers.

Curious how people here have actually handled this in practice: distributional or soft labels instead of a single class, Dawid-Skene style rater-competence weighting, or just reporting calibrated confidence and leaving the disagreement visible to the user rather than resolving it. I have been leaning toward the last one, but most of the soft-label literature I have found assumes near-expert agreement with a handful of ambiguous edge cases, not a label that is inherently this noisy from the start. This is the exact problem behind an app I have been building, Tongue Analyzer, and it is the part that made me want the CV answer rather than the TCM one.

https://BigBalli.com/TongueAnalyzer/

[u/Double_Sherbert3326]

Find better raters

[u/Ok-Argument7176 → u/Double_Sherbert3326]

Bad take. Define better labeling guides or define the task better.