{"id":10310,"date":"2019-11-07T03:23:32","date_gmt":"2019-11-07T01:23:32","guid":{"rendered":"https:\/\/www.datanovia.com\/en\/?post_type=dt_lessons&#038;p=10310"},"modified":"2019-11-07T03:23:32","modified_gmt":"2019-11-07T01:23:32","slug":"inter-rater-reliability-analyses-quick-r-codes","status":"publish","type":"dt_lessons","link":"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-reliability-analyses-quick-r-codes\/","title":{"rendered":"Inter-Rater Reliability Analyses: Quick R Codes"},"content":{"rendered":"<div id=\"rdoc\">\n<p>This chapter provides a quick start R code to compute the different statistical measures for analyzing the inter-rater reliability or agreement. These include:<\/p>\n<ul>\n<li><em>Cohen\u2019s Kappa<\/em>: It can be used for either two nominal or two ordinal variables. It accounts for strict agreements between observers. It is most appropriate for two nominal variables.<\/li>\n<li><em>Weighted Kappa<\/em>: It should be considered for two ordinal variables only. It allows partial agreement.<\/li>\n<li><em>Light\u2019s Kappa<\/em>, which is the average of Cohen\u2019s Kappa if using more than two categorical variables.<\/li>\n<li><em>Fleiss Kappa<\/em>: for two or more categorical variables (nominal or ordinal)<\/li>\n<li><em>Intraclass correlation coefficient<\/em> (ICC) for continuous or ordinal data<\/li>\n<\/ul>\n<p>Contents:<\/p>\n<div id=\"TOC\">\n<ul>\n<li><a href=\"#r-packages\">R packages<\/a><\/li>\n<li><a href=\"#prerequisites\">Prerequisites<\/a><\/li>\n<li><a href=\"#examples-data\">Examples data<\/a><\/li>\n<li><a href=\"#cohens-kappa-two-raters\">Cohen\u2019s Kappa: two raters<\/a><\/li>\n<li><a href=\"#weighed-kappa-ordinal-scales\">Weighed kappa: ordinal scales<\/a><\/li>\n<li><a href=\"#lights-kappa-multiple-raters\">Light\u2019s kappa: multiple raters<\/a><\/li>\n<li><a href=\"#fleiss-kappa-multiple-raters\">Fleiss\u2019 kappa: multiple raters<\/a><\/li>\n<li><a href=\"#intraclass-correlation-coefficients-continuous-scales\">Intraclass correlation coefficients: continuous scales<\/a><\/li>\n<li><a href=\"#summary\">Summary<\/a><\/li>\n<li><a href=\"#references\">References<\/a><\/li>\n<\/ul>\n<\/div>\n<div class='dt-sc-hr-invisible-medium  '><\/div>\n<div class='dt-sc-ico-content type1'><div class='custom-icon' ><a href='https:\/\/www.datanovia.com\/en\/product\/inter-rater-reliability-essentials-practical-guide-in-r\/' target='_blank'><span class='fa fa-book'><\/span><\/a><\/div><h4><a href='https:\/\/www.datanovia.com\/en\/product\/inter-rater-reliability-essentials-practical-guide-in-r\/' target='_blank'> Related Book <\/a><\/h4>Inter-Rater Reliability Essentials: Practical Guide in R<\/div>\n<div class='dt-sc-hr-invisible-medium  '><\/div>\n<div id=\"r-packages\" class=\"section level2\">\n<h2>R packages<\/h2>\n<p>There are many R packages and functions for inter-rater agreement analyses, including:<\/p>\n<table>\n<thead>\n<tr class=\"header\">\n<th>Measures<\/th>\n<th>R function [package]<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr class=\"odd\">\n<td>Cohen\u2019s kappa<\/td>\n<td>Kappa() [vcd], kappa2() [irr]<\/td>\n<\/tr>\n<tr class=\"even\">\n<td>Weighted kappa<\/td>\n<td>Kappa() [vcd], kappa2() [irr]<\/td>\n<\/tr>\n<tr class=\"odd\">\n<td>Light\u2019s kappa<\/td>\n<td>kappam.light() [irr]<\/td>\n<\/tr>\n<tr class=\"even\">\n<td>Fleiss Kappa<\/td>\n<td>kappam.fleiss() [irr]<\/td>\n<\/tr>\n<tr class=\"odd\">\n<td>ICC<\/td>\n<td>icc() [irr], ICC() [psych]<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<div id=\"prerequisites\" class=\"section level2\">\n<h2>Prerequisites<\/h2>\n<p>In the next sections, we\u2019ll use only the functions from the <code>irr<\/code> package. Make sure you have installed it.<\/p>\n<p>Load the package:<\/p>\n<pre class=\"r\"><code># install.packages(\"irrr\")\r\nlibrary(irr)<\/code><\/pre>\n<\/div>\n<div id=\"examples-data\" class=\"section level2\">\n<h2>Examples data<\/h2>\n<ul>\n<li>psychiatric <strong>diagnoses<\/strong> data provided by 6 raters [irr package]. A total of 30 patients were enrolled and classified by each of the raters into 5 nominal categories <span class=\"citation\">(Fleiss and others 1971)<\/span>: 1. Depression, 2. Personality Disorder, 3. Schizophrenia, 4. Neurosis, 5. Other.<\/li>\n<li><strong>anxiety<\/strong> data [irr package], which contains the anxiety ratings of 20 subjects, rated by 3 raters on ordinal scales. Values are ranging from 1 (not anxious at all) to 6 (extremely anxious).<\/li>\n<\/ul>\n<p>Inspect the data:<\/p>\n<pre class=\"r\"><code># Diagnoses data\r\ndata(\"diagnoses\", package = \"irr\")\r\nhead(diagnoses[, 1:3])<\/code><\/pre>\n<pre><code>##                    rater1                  rater2                  rater3\r\n## 1             4. Neurosis             4. Neurosis             4. Neurosis\r\n## 2 2. Personality Disorder 2. Personality Disorder 2. Personality Disorder\r\n## 3 2. Personality Disorder        3. Schizophrenia        3. Schizophrenia\r\n## 4                5. Other                5. Other                5. Other\r\n## 5 2. Personality Disorder 2. Personality Disorder 2. Personality Disorder\r\n## 6           1. Depression           1. Depression        3. Schizophrenia<\/code><\/pre>\n<pre class=\"r\"><code># Anxiety data\r\ndata(\"anxiety\", package = \"irr\")\r\nhead(anxiety, 4)<\/code><\/pre>\n<pre><code>##   rater1 rater2 rater3\r\n## 1      3      3      2\r\n## 2      3      6      1\r\n## 3      3      4      4\r\n## 4      4      6      4<\/code><\/pre>\n<\/div>\n<div id=\"cohens-kappa-two-raters\" class=\"section level2\">\n<h2>Cohen\u2019s Kappa: two raters<\/h2>\n<p>The Cohen\u2019s kappa corresponds to the unweighted kappa. It can be used for two nominal or two ordinal categorical variables<\/p>\n<pre class=\"r\"><code>kappa2(diagnoses[, c(\"rater1\", \"rater2\")], weight = \"unweighted\")<\/code><\/pre>\n<pre><code>##  Cohen's Kappa for 2 Raters (Weights: unweighted)\r\n## \r\n##  Subjects = 30 \r\n##    Raters = 2 \r\n##     Kappa = 0.651 \r\n## \r\n##         z = 7 \r\n##   p-value = 2.63e-12<\/code><\/pre>\n<\/div>\n<div id=\"weighed-kappa-ordinal-scales\" class=\"section level2\">\n<h2>Weighed kappa: ordinal scales<\/h2>\n<p>Weighted kappa should be considered only when ratings are performed in ordinal scale as in the following example.<\/p>\n<pre class=\"r\"><code>kappa2(anxiety[, c(\"rater1\", \"rater2\")], weight = \"equal\")<\/code><\/pre>\n<\/div>\n<div id=\"lights-kappa-multiple-raters\" class=\"section level2\">\n<h2>Light\u2019s kappa: multiple raters<\/h2>\n<p>It returns the average Cohen\u2019s kappa when you have multiple raters<\/p>\n<pre class=\"r\"><code>kappam.light(diagnoses[, 1:3])<\/code><\/pre>\n<pre><code>##  Light's Kappa for m Raters\r\n## \r\n##  Subjects = 30 \r\n##    Raters = 3 \r\n##     Kappa = 0.555 \r\n## \r\n##         z = NaN \r\n##   p-value = NaN<\/code><\/pre>\n<\/div>\n<div id=\"fleiss-kappa-multiple-raters\" class=\"section level2\">\n<h2>Fleiss\u2019 kappa: multiple raters<\/h2>\n<p>The raters are not assumed to be the same for all subjects.<\/p>\n<pre class=\"r\"><code>kappam.fleiss(diagnoses[, 1:3])<\/code><\/pre>\n<pre><code>##  Fleiss' Kappa for m Raters\r\n## \r\n##  Subjects = 30 \r\n##    Raters = 3 \r\n##     Kappa = 0.534 \r\n## \r\n##         z = 9.89 \r\n##   p-value = 0<\/code><\/pre>\n<\/div>\n<div id=\"intraclass-correlation-coefficients-continuous-scales\" class=\"section level2\">\n<h2>Intraclass correlation coefficients: continuous scales<\/h2>\n<p>Read more in Chapter @ref(intraclass-correlation-coefficient):<\/p>\n<pre class=\"r\"><code>icc(\r\n  anxiety, model = \"twoway\", \r\n  type = \"agreement\", unit = \"single\"\r\n  )<\/code><\/pre>\n<pre><code>##  Single Score Intraclass Correlation\r\n## \r\n##    Model: twoway \r\n##    Type : agreement \r\n## \r\n##    Subjects = 20 \r\n##      Raters = 3 \r\n##    ICC(A,1) = 0.198\r\n## \r\n##  F-Test, H0: r0 = 0 ; H1: r0 &gt; 0 \r\n##  F(19,39.7) = 1.83 , p = 0.0543 \r\n## \r\n##  95%-Confidence Interval for ICC Population Values:\r\n##   -0.039 &lt; ICC &lt; 0.494<\/code><\/pre>\n<\/div>\n<div id=\"summary\" class=\"section level2\">\n<h2>Summary<\/h2>\n<p>This article describes how to compute the different inter-rater agreement measures using the <code>irr<\/code> packages.<\/p>\n<\/div>\n<div id=\"references\" class=\"section level2 unnumbered\">\n<h2>References<\/h2>\n<div id=\"refs\" class=\"references\">\n<div id=\"ref-Fleiss1971\">\n<p>Fleiss, J.L., and others. 1971. \u201cMeasuring Nominal Scale Agreement Among Many Raters.\u201d <em>Psychological Bulletin<\/em> 76 (5): 378\u201382.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<p><!--end rdoc--><\/p>\n","protected":false},"excerpt":{"rendered":"<p>This article describes how to compute the different inter-rater agreement measures using the irr packages.<\/p>\n","protected":false},"author":1,"featured_media":9135,"parent":0,"menu_order":0,"comment_status":"open","ping_status":"closed","template":"","class_list":["post-10310","dt_lessons","type-dt_lessons","status-publish","has-post-thumbnail","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v25.2 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Inter-Rater Reliability Analyses: Quick R Codes - Datanovia<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-reliability-analyses-quick-r-codes\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Inter-Rater Reliability Analyses: Quick R Codes - Datanovia\" \/>\n<meta property=\"og:description\" content=\"This article describes how to compute the different inter-rater agreement measures using the irr packages.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-reliability-analyses-quick-r-codes\/\" \/>\n<meta property=\"og:site_name\" content=\"Datanovia\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.datanovia.com\/en\/wp-content\/uploads\/2019\/05\/P1040336.JPG.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1024\" \/>\n\t<meta property=\"og:image:height\" content=\"512\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-reliability-analyses-quick-r-codes\/\",\"url\":\"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-reliability-analyses-quick-r-codes\/\",\"name\":\"Inter-Rater Reliability Analyses: Quick R Codes - Datanovia\",\"isPartOf\":{\"@id\":\"https:\/\/www.datanovia.com\/en\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-reliability-analyses-quick-r-codes\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-reliability-analyses-quick-r-codes\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.datanovia.com\/en\/wp-content\/uploads\/2019\/05\/P1040336.JPG.jpg\",\"datePublished\":\"2019-11-07T01:23:32+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-reliability-analyses-quick-r-codes\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-reliability-analyses-quick-r-codes\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-reliability-analyses-quick-r-codes\/#primaryimage\",\"url\":\"https:\/\/www.datanovia.com\/en\/wp-content\/uploads\/2019\/05\/P1040336.JPG.jpg\",\"contentUrl\":\"https:\/\/www.datanovia.com\/en\/wp-content\/uploads\/2019\/05\/P1040336.JPG.jpg\",\"width\":1024,\"height\":512},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-reliability-analyses-quick-r-codes\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.datanovia.com\/en\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Lessons\",\"item\":\"https:\/\/www.datanovia.com\/en\/lessons\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Inter-Rater Reliability Analyses: Quick R Codes\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.datanovia.com\/en\/#website\",\"url\":\"https:\/\/www.datanovia.com\/en\/\",\"name\":\"Datanovia\",\"description\":\"Data Mining and Statistics for Decision Support\",\"publisher\":{\"@id\":\"https:\/\/www.datanovia.com\/en\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.datanovia.com\/en\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.datanovia.com\/en\/#organization\",\"name\":\"Datanovia\",\"url\":\"https:\/\/www.datanovia.com\/en\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.datanovia.com\/en\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.datanovia.com\/en\/wp-content\/uploads\/2018\/09\/datanovia-logo.png\",\"contentUrl\":\"https:\/\/www.datanovia.com\/en\/wp-content\/uploads\/2018\/09\/datanovia-logo.png\",\"width\":98,\"height\":99,\"caption\":\"Datanovia\"},\"image\":{\"@id\":\"https:\/\/www.datanovia.com\/en\/#\/schema\/logo\/image\/\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Inter-Rater Reliability Analyses: Quick R Codes - Datanovia","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-reliability-analyses-quick-r-codes\/","og_locale":"en_US","og_type":"article","og_title":"Inter-Rater Reliability Analyses: Quick R Codes - Datanovia","og_description":"This article describes how to compute the different inter-rater agreement measures using the irr packages.","og_url":"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-reliability-analyses-quick-r-codes\/","og_site_name":"Datanovia","og_image":[{"width":1024,"height":512,"url":"https:\/\/www.datanovia.com\/en\/wp-content\/uploads\/2019\/05\/P1040336.JPG.jpg","type":"image\/jpeg"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-reliability-analyses-quick-r-codes\/","url":"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-reliability-analyses-quick-r-codes\/","name":"Inter-Rater Reliability Analyses: Quick R Codes - Datanovia","isPartOf":{"@id":"https:\/\/www.datanovia.com\/en\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-reliability-analyses-quick-r-codes\/#primaryimage"},"image":{"@id":"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-reliability-analyses-quick-r-codes\/#primaryimage"},"thumbnailUrl":"https:\/\/www.datanovia.com\/en\/wp-content\/uploads\/2019\/05\/P1040336.JPG.jpg","datePublished":"2019-11-07T01:23:32+00:00","breadcrumb":{"@id":"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-reliability-analyses-quick-r-codes\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-reliability-analyses-quick-r-codes\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-reliability-analyses-quick-r-codes\/#primaryimage","url":"https:\/\/www.datanovia.com\/en\/wp-content\/uploads\/2019\/05\/P1040336.JPG.jpg","contentUrl":"https:\/\/www.datanovia.com\/en\/wp-content\/uploads\/2019\/05\/P1040336.JPG.jpg","width":1024,"height":512},{"@type":"BreadcrumbList","@id":"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-reliability-analyses-quick-r-codes\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.datanovia.com\/en\/"},{"@type":"ListItem","position":2,"name":"Lessons","item":"https:\/\/www.datanovia.com\/en\/lessons\/"},{"@type":"ListItem","position":3,"name":"Inter-Rater Reliability Analyses: Quick R Codes"}]},{"@type":"WebSite","@id":"https:\/\/www.datanovia.com\/en\/#website","url":"https:\/\/www.datanovia.com\/en\/","name":"Datanovia","description":"Data Mining and Statistics for Decision Support","publisher":{"@id":"https:\/\/www.datanovia.com\/en\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.datanovia.com\/en\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.datanovia.com\/en\/#organization","name":"Datanovia","url":"https:\/\/www.datanovia.com\/en\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.datanovia.com\/en\/#\/schema\/logo\/image\/","url":"https:\/\/www.datanovia.com\/en\/wp-content\/uploads\/2018\/09\/datanovia-logo.png","contentUrl":"https:\/\/www.datanovia.com\/en\/wp-content\/uploads\/2018\/09\/datanovia-logo.png","width":98,"height":99,"caption":"Datanovia"},"image":{"@id":"https:\/\/www.datanovia.com\/en\/#\/schema\/logo\/image\/"}}]}},"multi-rating":{"mr_rating_results":[]},"_links":{"self":[{"href":"https:\/\/www.datanovia.com\/en\/wp-json\/wp\/v2\/dt_lessons\/10310","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.datanovia.com\/en\/wp-json\/wp\/v2\/dt_lessons"}],"about":[{"href":"https:\/\/www.datanovia.com\/en\/wp-json\/wp\/v2\/types\/dt_lessons"}],"author":[{"embeddable":true,"href":"https:\/\/www.datanovia.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.datanovia.com\/en\/wp-json\/wp\/v2\/comments?post=10310"}],"version-history":[{"count":0,"href":"https:\/\/www.datanovia.com\/en\/wp-json\/wp\/v2\/dt_lessons\/10310\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.datanovia.com\/en\/wp-json\/wp\/v2\/media\/9135"}],"wp:attachment":[{"href":"https:\/\/www.datanovia.com\/en\/wp-json\/wp\/v2\/media?parent=10310"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}