Significance Tests for the Measure of Raw Agreement
by Alexander von Eye, Michael Schauerhuber, Patrick Mair.
Abstract:
Significance tests for the measure of raw agreement are proposed. First, it is shown that
the measure of raw agreement can be expressed as a proportionate reduction-in-error measure,
sharing this characteristic with Cohen's kappa and Brennan and Prediger's kappan. Second, it is
shown that the coefficient of raw agreement is linearly related to Brennan and Prediger's kappa_n.
Therefore, using the same base model for the estimation of expected cell frequencies as Brennan
and Prediger's kappa_n, one can devise significance tests for the measure of raw agreement. Two
tests are proposed. The first uses Stouffer's Z, a probability pooler. The second test is the
binomial test. A data example analyzes the agreement between two psychiatrists' diagnoses. The
covariance structure of the agreement cells in a rater by rater table is described. Simulation studies
show the performance and power functions of the test statistics. Key Words:
rater agreement, raw agreement coefficient, Stouffer's Z, binomial test, Monte Carlo
Simulation
Authors:
Alexander von Eye, voneye@msu.edu
Michael Schauerhuber, Michael.Schauerhuber@wu-wien.ac.at
Patrick Mair, Patrick.Mair@wu-wien.ac.at
Editor:
R. G. Graf, rgraf@sunstroke.sdsu.edu
READING THE ARTICLE: You can read the article in
portable document (.pdf) format (332812 bytes.)
NOTE: The content of this article is the intellectual property of the authors, who retains all rights to future publication.
This page has been accessed 509 times since January 18, 2007.
Return to the 
Home Page.