Significance Tests for the Measure of Raw Agreement

by Alexander von Eye, Michael Schauerhuber, Patrick Mair.

Abstract: Significance tests for the measure of raw agreement are proposed. First, it is shown that the measure of raw agreement can be expressed as a proportionate reduction-in-error measure, sharing this characteristic with Cohen's kappa and Brennan and Prediger's kappan. Second, it is shown that the coefficient of raw agreement is linearly related to Brennan and Prediger's kappa_n. Therefore, using the same base model for the estimation of expected cell frequencies as Brennan and Prediger's kappa_n, one can devise significance tests for the measure of raw agreement. Two tests are proposed. The first uses Stouffer's Z, a probability pooler. The second test is the binomial test. A data example analyzes the agreement between two psychiatrists' diagnoses. The covariance structure of the agreement cells in a rater by rater table is described. Simulation studies show the performance and power functions of the test statistics.

Key Words: rater agreement, raw agreement coefficient, Stouffer's Z, binomial test, Monte Carlo Simulation

Authors:
Alexander von Eye, voneye@msu.edu
Michael Schauerhuber, Michael.Schauerhuber@wu-wien.ac.at
Patrick Mair, Patrick.Mair@wu-wien.ac.at

Editor: R. G. Graf, rgraf@sunstroke.sdsu.edu

READING THE ARTICLE: You can read the article in portable document (.pdf) format (332812 bytes.)

NOTE: The content of this article is the intellectual property of the authors, who retains all rights to future publication.

This page has been accessed 4387 times since January 18, 2007.


Return to the InterStat Home Page.