Cohen's Kappa Calculator

Figran

Paste two raters' labels or an agreement table to measure agreement beyond chance.

  • Ratings or agreement table
  • Weighted and Fleiss kappa
  • Agreement matrix figure

Data

Data

Columns

SelectNameRole
Pathologist A
Pathologist B

Categories

Negative
Atypical hyperplasia
Carcinoma in situ
Invasive carcinoma

Weighted kappa uses this category order.

Weights
Cells

Your data stay in this browser.

Weighted κ (quadratic) = .78

95% CI [.71, .86]

Substantial agreement · Landis & Koch 1977

Unweighted κ = .49 [.38, .60]

n = 118

Pathologist A / Pathologist BPathologist BPathologist ANegativeNegativeAtypicalhyperplasiaAtypicalhyperplasiaCarcinoma insituCarcinoma insituInvasivecarcinomaInvasivecarcinoma2222057140023600117102626382827126910118036

Statistics

Weighted κ (quadratic).78
Confidence interval.71, .86
Unweighted κ.49
Unweighted κ Confidence interval.38, .60
Percent agreement63.56%
Chance agreement28.12%
Standard error.04
z8.61
p value< .001
Maximum kappa.62
PABAK.51
Gwet’s AC1.53
AC1 CI.41, .64

Specific agreement

CategoryRater 1 nRater 2 nSpecific agreement
Negative2627.83
Atypical hyperplasia2612.37
Carcinoma in situ3869.67
Invasive carcinoma2810.53

Disagreements (43)

IDPathologist APathologist B
1Invasive carcinomaCarcinoma in situ
9Atypical hyperplasiaCarcinoma in situ
12Atypical hyperplasiaCarcinoma in situ
13Invasive carcinomaCarcinoma in situ
14NegativeAtypical hyperplasia
17Atypical hyperplasiaNegative
20Atypical hyperplasiaCarcinoma in situ
21Invasive carcinomaCarcinoma in situ
23Atypical hyperplasiaNegative
24Atypical hyperplasiaCarcinoma in situ

Report sentence

Two raters agreed on 63.6% of 118 records (quadratic-weighted Cohen's κ = .78, 95% CI [.71, .86]), indicating substantial agreement (Landis & Koch, 1977). Unweighted κ = .49.

Cohen's kappa calculator examples

Start with a dataset similar to yours.

Pathologists grading cervical slides1 Negative · 2 Atypical hyperplasia · 3 Carcinoma in situ · 4 Invasive carcinoma

Pathologists grading cervical slides

Two pathologists classify 118 slides into four ordered categories.

Source: Holmquist et al. 1967, via Agresti, Categorical Data Analysis

Abstract screening, two reviewers

Abstract screening, two reviewers

Compare inclusion decisions and inspect disagreements.

Simulated screening · 200 records · 2 reviewers

Same agreement, lower kappa

Same agreement, lower kappa

Examine the effect of unbalanced categories.

Source: Feinstein & Cicchetti 1990

Multiple sclerosis diagnosis, two neurologists1 Certain · 2 Probable · 3 Possible · 4 Doubtful

Multiple sclerosis diagnosis, two neurologists

Compare ordered diagnoses for patients in Winnipeg.

Source: Westlund & Kurland 1953, via Landis & Koch 1977

Psychiatric diagnoses, six ratersR1–R6 = Rater 1–6

Psychiatric diagnoses, six raters

Fleiss reported category counts for changing raters; the fixed-rater columns here are simulated.

Rater assignment: simulated allocation based on Fleiss 1971 counts

Interview coding, two coders1 Access · 2 Cost · 3 Trust · 4 Workload · 5 Other

Interview coding, two coders

Compare five nominal codes across interview segments.

Simulated coding · 150 segments · 5 codes

How to calculate Cohen's kappa

Go from ratings to a report-ready result in four steps.

  1. 1Paste ratings or an agreement table

    Paste two or more columns, or enter a square table.

  2. 2Select raters and order categories

    Choose the rater columns and move ordered categories into place.

  3. 3Choose weights and read kappa

    Select the weighting method and confidence level.

  4. 4Copy or download your results

    Download a figure or copy your report sentence.

Interpreting Cohen's kappa

Kappa compares observed agreement with agreement expected from the raters’ category frequencies. Interpretation depends on the task and the consequences of disagreement.

κ rangeLandis & Koch 1977McHugh 2012
< 0Poor agreement—
0–.20Slight agreementNone
.21–.39Fair agreementMinimal
.40Fair agreementWeak
.41–.59Moderate agreementWeak
.60Moderate agreementModerate
.61–.79Substantial agreementModerate
.80Substantial agreementStrong
.81–.90Almost perfect agreementStrong
> .90–1Almost perfect agreementAlmost perfect

High agreement, low kappa

The same 85% agreement can yield different kappa values when the category distributions differ. Examine prevalence and report PABAK or AC1 alongside kappa.

Rater 1 / Rater 2Rater 2Rater 1PositivePositiveNegativeNegative40964549514654100045

κ = .70 · PI = −.05

Rater 1 / Rater 2Rater 2Rater 1PositivePositiveNegativeNegative80105590108515100080

κ = .32 · PI = .75

Both examples: 85% agreement · PABAK = .70

Weighted kappa: linear or quadratic

Linear weights treat each step of disagreement equally; quadratic weights penalize larger differences more strongly. Choose weights before analysis and verify the category order.

Linear

1.000.670.330.00
0.671.000.670.33
0.330.671.000.67
0.000.330.671.00

Quadratic

1.000.890.560.00
0.891.000.890.56
0.560.891.000.89
0.000.560.891.00

References: Landis & Koch (1977) · McHugh (2012) · Feinstein & Cicchetti (1990)

Cohen's kappa formula

Observed agreement is pₒ and expected agreement from the marginal proportions is pₑ. Weighted kappa applies agreement weights to each cell; Fleiss kappa extends agreement to multiple ratings per subject.

po=∑niin,pe=∑pi·p·ipow=∑wijpij,pew=∑wijpi·p·jai=∑wijp·j,bj=∑wijpi·S=∑pij[wij−(1−κw)(ai+bj)]2SE(κw)=[S−(κw−pew(1−κw))2n(1−pew)2]1/2κ=po−pe1−pe,κw=pow−pew1−pewwij=1−(|i−j|k−1)q,q∈{1, 2}CI=κ±z1−α/2SEκF=P̄−Pe1−PePABAK=kpo−1k−1

For the default data, pₒ = 75/118, pₑ = .281, and κ = .493.

References: Cohen (1960) · Cohen (1968) · Fleiss, Cohen & Everitt (1969) · Fleiss (1971)

Cohen's kappa calculator FAQ

What is Cohen's kappa?

Cohen’s kappa measures agreement between two raters after accounting for agreement expected by chance. A value of 1 is perfect, 0 equals chance and a negative value is below chance. For internal consistency of questionnaire items, use the Cronbach’s alpha calculator.

What is a good kappa value?

There is no universal cutoff. Landis and Koch describe .61–.80 as substantial agreement, but the context and consequences of disagreement matter.

Why can agreement be high but kappa low?

An imbalance in categories can make expected chance agreement high, reducing kappa even when observed agreement is high. Consider reporting PABAK and AC1 as well. If one rater is a reference standard, consider the sensitivity and specificity calculator.

When should I use weighted kappa?

Use weighted kappa for ordered categories when some disagreements are more serious than others. Select linear or quadratic weights before analysis.

When should I use Fleiss' kappa?

Use Fleiss’ kappa for categorical ratings with a fixed number of ratings per subject from three or more raters. Krippendorff’s alpha supports a broader range of missing-data designs.

Should I use kappa or ICC?

Use kappa for categorical ratings and consider an intraclass correlation coefficient (ICC) for continuous measurements.

How is the confidence interval calculated?

The calculator uses a large-sample standard error for Cohen’s kappa and a subject-level variance with a t distribution for Fleiss’ kappa.

Can I calculate kappa in SPSS or R?

Yes. SPSS provides kappa in Crosstabs; R packages including irr, psych and vcd also calculate agreement statistics.

How should I report kappa?

Report the coefficient, confidence interval, sample size, number of raters, weighting method when used, and the observed agreement.

Check agreement between your raters

Calculate kappa and export your results.

Calculate kappa