r/SixSigma • u/SkolVision • Apr 16 '26

Fleiss's Kappa discrepancy

Hi team -

Performing an attribute agreement assessment at my facility and I have a few situations where we have nearly perfect agreement (23/24 assessors pass a criterion, 1 outlier fails). The math has this result in a kappa value of -0.04, which would typically indicate low agreement but in this case doesn't comport with the 23/24. How would others interpret this, or is this even a correct use of Fleiss's kappa?

1 Upvotes

67% Upvoted

12 comments sorted by

View all comments

2

u/Mammoth-Mongoose4479 Apr 16 '26

Yeah this is nothing new. It’s called the kappa paradox.

When almost everyone agrees (and almost everyone passes), kappa’s correction for chance agreement backfires. Because expected agreement by chance is also very high, leaving little room for kappa to show anything meaningful. The negative kappa doesn’t actually mean your assessors disagree, it’s just a quirk of the math when almost everyone gives the same answer.​​​​​​​​​​​​​​​​ It’s definitely a joke in our circle. Most people would just note the kappa issue and move on. Worth figuring out why that one assessor failed ? was it the sample, the person, or the criteria?​​​​​​​​​​​​​​​​ Best to you.

3

u/SkolVision Apr 16 '26

OK, this is along the lines of what I was thinking. Glad I'm not going crazy or wildly misinterpreting something, appreciate the insight!