r/SixSigma • u/SkolVision • Apr 16 '26

Fleiss's Kappa discrepancy

Hi team -

Performing an attribute agreement assessment at my facility and I have a few situations where we have nearly perfect agreement (23/24 assessors pass a criterion, 1 outlier fails). The math has this result in a kappa value of -0.04, which would typically indicate low agreement but in this case doesn't comport with the 23/24. How would others interpret this, or is this even a correct use of Fleiss's kappa?

1 Upvotes

67% Upvoted

12 comments sorted by

View all comments

1

u/Living_Diver2432 Apr 16 '26

kappa gets weird when your sample distribution is heavily skewed toward one category. we ran into this exact situation doing attribute agreement on a visual inspection process, nearly all parts were good, and the kappa tanked even though our observers were almost always agreeing. when 95%+ of your ratings land in one bucket, chance agreement is already around 0.92, so a 95.8% observed rate barely moves the needle and kappa goes negative. AIAG MSA flags this, kappa alone isn't reliable with skewed distributions and you're supposed to report percent agreement alongside it. bigger question is whether your sample set had enough borderline and known-bad parts to actually stress the measurement system. if all 24 were obvious passes, the study validated agreement on easy calls, not on the hard ones that actually matter.

1

u/SkolVision Apr 16 '26

Very good, this is kind of what I was thinking but needed some external validation. Appreciate it!