Lindell, M. K., and Brandt, C. J. (1997). The measurement of interratist concordance for ratings of a single objective. Appl. Psychol. Meas. 21, 271-278 doi: 10.1177/01466216970213006 Cohens kappa (κ) calculates the Inter Observer agreement taking into account the expected agreement as follows: The field in which you work determines the acceptable level of agreement. If it is a sports competition, you can accept a 60% evaluator agreement to designate a winner.

However, if you look at the data from cancer specialists who opt for treatment, you want much higher support – more than 90%. In general, more than 75% are considered acceptable in most areas. It is often interesting to know whether measurements made by two (sometimes more than two) different observers or by two different techniques give similar results. This is called concordance or concordance or reproducibility between measurements. Such an analysis considers the pairs of measurements, either categorical or both numerically, each pair having been made on an individual (or a pathology slide or an X-ray). Cohen, A., Doveh, E., and Nahum-Shani, I (2009). Test agreement for multi-coin scales with the indices rwg (j) and ADm (j). From the organ. Methods 12, 148-164.

doi: 10.1177/1094428107300365 Several formulas can be used to calculate compliance limits. The simple formula given in the previous paragraph, which works well for sample sizes greater than 60,[14] is Rousseau, D.M (1985). “Issues of level in organizational research: multilevel and cross level perspectives”, in Research in Organizational Behavior, Vol. 7, eds L. Cummings and B. Saw (Greenwich, CT: JAI Press), 1-37. Compliance limits = mean difference observed ± 1.96 standard deviation × of the observed differences. κ = (observed agreement [Po] – expected agreement [Pe]/(1-expected [Pe]). To calculate pe (the probability of a random match), we find that: Kappa is a way to measure compliance or reliability and correct the number of times evaluations can coincide by chance. Cohens Kappa,[5] which works for two evaluators, and Fleiss`Kappa,[6] an adaptation that works for any fixed number of evaluators, improve the common probability by taking into account the amount of concordance that one might expect by chance. The original versions had the same problem as the common probability, as they treat the data as nominal and assume that the evaluations are not natural; If the data do have a rank (ordinary measurement level), this information is not fully taken into account in the measurements.

Kozlowski, S. W. J., and Hattrup, K. (1992). A disagreement on the group agreement: separating the issues of coherence and consensus. J. Appl. Psychol. 77, 161-167. doi: 10.1037/0021-9010.77.2.161 Bland and Altman[15] expanded this idea by graphically showing the difference of each point, the mean difference and the limits of the vertical concordance against the average of the two horizontal evaluations.

. . .