Cohen’s kappa calculator

How to use it

Enter the agreement table: rater A’s categories as rows, rater B’s as columns, in the same order, with the number of cases in each cell. Use weights only for ordered categories (for example severity grades).

Formulas

• Observed agreement p_o = Σ w_ij p_ij; chance agreement p_e = Σ w_ij p_i· p_·j; κ = (p_o − p_e) / (1 − p_e).
• Weights: 1 on the diagonal and 0 elsewhere (unweighted); 1 − |i − j|/(k − 1) (linear); 1 − (i − j)²/(k − 1)² (quadratic).
• Standard error for the interval: Fleiss, Cohen and Everitt (1969); the z-test of κ = 0 uses their null standard error.

Assumptions and limits

• The same cases rated independently by two raters into the same set of categories. For more than two raters use Fleiss’ kappa.
• κ depends on how common each category is: with very unbalanced marginals κ can be low despite high raw agreement (the maximum attainable κ is shown).
• Landis and Koch’s labels (0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial) are conventions, not tests.

Worked example

Two raters classify 50 cases: both yes 20, A yes and B no 5, A no and B yes 10, both no 15. p_o = 0.70, p_e = 0.50, κ = 0.40 (95% CI 0.15 to 0.65), fair agreement. The calculator also reproduces vcd::Kappa for R’s SexualFun table: 0.1293 (ASE 0.0686) unweighted and 0.2374 (ASE 0.0783) with linear weights.

References

• Cohen (1960), A coefficient of agreement for nominal scales, Educational and Psychological Measurement 20(1)
• Cohen (1968), Weighted kappa, Psychological Bulletin 70(4)
• Fleiss, Cohen and Everitt (1969), Large sample standard errors of kappa and weighted kappa, Psychological Bulletin 72(5)
• Landis and Koch (1977), The measurement of observer agreement for categorical data, Biometrics 33(1)