{"id":10309,"date":"2019-11-07T03:18:58","date_gmt":"2019-11-07T01:18:58","guid":{"rendered":"https:\/\/www.datanovia.com\/en\/?post_type=dt_lessons&#038;p=10309"},"modified":"2019-11-07T03:18:58","modified_gmt":"2019-11-07T01:18:58","slug":"inter-rater-agreement-chart-in-r","status":"publish","type":"dt_lessons","link":"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-agreement-chart-in-r\/","title":{"rendered":"Inter-Rater Agreement Chart in R"},"content":{"rendered":"<div id=\"rdoc\">\n<p>Previously, we describe many statistical metrics, such as the Cohen\u2019s Kappa @ref(cohen-s-kappa) and weighted Kappa @ref(weighted-kappa), for assessing the agreement or the concordance between two raters (judges, observers, clinicians) or two methods of measurement.<\/p>\n<p>This chapter describes the <strong>agreement chart<\/strong> <span class=\"citation\">(S. I. Bangdiwala 1985)<\/span>, which provides a solution for visualizing the strength of the agreement between two methods measuring on ordinal scale. For example, the agreement chart can be used to visually compare two diagnostic or classification methods. Note that, the agreement plot is generally recommended for ordinal categorical variables.<\/p>\n<p>You will learn:<\/p>\n<ul>\n<li><strong>The basics for constructing an agreement chart<\/strong><\/li>\n<li><strong>R codes for creating an agreement chart<\/strong><\/li>\n<\/ul>\n<p>Contents:<\/p>\n<div id=\"TOC\">\n<ul>\n<li><a href=\"#prerequisites\">Prerequisites<\/a><\/li>\n<li><a href=\"#construction-of-the-agreement-chart\">Construction of the agreement chart<\/a><\/li>\n<li><a href=\"#example-of-data\">Example of data<\/a><\/li>\n<li><a href=\"#create-an-agreement-chart-in-r\">Create an agreement chart in R<\/a><\/li>\n<li><a href=\"#summary\">Summary<\/a><\/li>\n<li><a href=\"#references\">References<\/a><\/li>\n<\/ul>\n<\/div>\n<div class='dt-sc-hr-invisible-medium  '><\/div>\n<div class='dt-sc-ico-content type1'><div class='custom-icon' ><a href='https:\/\/www.datanovia.com\/en\/product\/inter-rater-reliability-essentials-practical-guide-in-r\/' target='_blank'><span class='fa fa-book'><\/span><\/a><\/div><h4><a href='https:\/\/www.datanovia.com\/en\/product\/inter-rater-reliability-essentials-practical-guide-in-r\/' target='_blank'> Related Book <\/a><\/h4>Inter-Rater Reliability Essentials: Practical Guide in R<\/div>\n<div class='dt-sc-hr-invisible-medium  '><\/div>\n<div id=\"prerequisites\" class=\"section level2\">\n<h2>Prerequisites<\/h2>\n<p>Make sure you have installed the <code>vcd<\/code> package for visualizing categorical data.<\/p>\n<pre class=\"r\"><code>install.packages(\"vcd\")<\/code><\/pre>\n<\/div>\n<div id=\"construction-of-the-agreement-chart\" class=\"section level2\">\n<h2>Construction of the agreement chart<\/h2>\n<p><strong>Data structure<\/strong>. Let\u2019s considere the following k x k contingency table, where k is the number of categories:<\/p>\n<pre><code>##           rater2\r\n## rater1     Level.1 Level.2 Level... Level.k Total\r\n##   Level.1  n11     n12     ...      n1k     n1+  \r\n##   Level.2  n21     n22     ...      n2k     n2+  \r\n##   Level... ...     ...     ...      ...     ...  \r\n##   Level.k  nk1     nk2     ...      nkk     nk+  \r\n##   Total    n+1     n+2     ...      n+k     N<\/code><\/pre>\n<p><strong>Terminologies<\/strong>:<\/p>\n<ul>\n<li>The column \u201cTotal\u201d (<code>n1+, n2+, ..., nk+<\/code>) indicates the sum of each row, known as <strong>row margins<\/strong> or marginal counts. Here, the total sum of a given row <code>i<\/code> is named <code>ni+<\/code>.<\/li>\n<li>The row \u201cTotal\u201d (<code>n+1, n+2, ..., n+k<\/code>) indicates the sum of each column, known as <strong>column margins<\/strong>. Here, the total sum of a given column <code>i<\/code> is named <code>n+i<\/code><\/li>\n<li>N is the total sum of all table cells<\/li>\n<\/ul>\n<p><strong>Agreement chart<\/strong>. The agreement chart is a visual representation of a k \u00d7 k square contingency table. It is constructed with the following steps <span class=\"citation\">(S. Bangdiwala and Shankar 2013, <span class=\"citation\">Friendly, Meyer, and Zeileis (2015)<\/span>)<\/span>:<\/p>\n<ol style=\"list-style-type: decimal;\">\n<li>Draw an N \u00d7 N square, where N is the total sample size<\/li>\n<li>Draw k rectangles, each of dimensions row x column marginal totals, placed inside the N \u00d7 N square<\/li>\n<li>Draw k black squares of dimensions based on the diagonal cell frequencies (<code>nii<\/code>), placed inside the corresponding rectangle. These black squares represents to the observed agreement.<\/li>\n<li>\u201cPartial agreement\u201d areas can be similarly placed within the rectangles, with gray color for cells further away from the diagonal cells.<\/li>\n<\/ol>\n<\/div>\n<div id=\"example-of-data\" class=\"section level2\">\n<h2>Example of data<\/h2>\n<p>We\u2019ll use the <code>anxiety<\/code> demo dataset where two clinical doctors classify 50 individuals into 4 ordered anxiety levels: \u201cnormal\u201d (no anxiety), \u201cmoderate\u201d, \u201chigh\u201d, \u201cvery high\u201d.<\/p>\n<p>The data is organized in the following 3x3 contingency table:<\/p>\n<pre class=\"r\"><code>anxiety &lt;- as.table(\r\n  rbind(\r\n    c(11, 3, 1, 0), c(1, 9, 0, 1),\r\n    c(0, 1, 10, 0 ), c(1, 2, 0, 10)\r\n  )\r\n)\r\ndimnames(anxiety) &lt;- list(\r\n  Doctor1 = c(\"Normal\", \"Moderate\", \"High\", \"Very high\"),\r\n  Doctor2 = c(\"Normal\", \"Moderate\", \"High\", \"Very high\")\r\n)\r\nanxiety<\/code><\/pre>\n<pre><code>##            Doctor2\r\n## Doctor1     Normal Moderate High Very high\r\n##   Normal        11        3    1         0\r\n##   Moderate       1        9    0         1\r\n##   High           0        1   10         0\r\n##   Very high      1        2    0        10<\/code><\/pre>\n<div class=\"warning\">\n<p>Note that the factor levels must be in the correct order, otherwise the results will be wrong.<\/p>\n<\/div>\n<\/div>\n<div id=\"create-an-agreement-chart-in-r\" class=\"section level2\">\n<h2>Create an agreement chart in R<\/h2>\n<p>We want to visually compare the concordance of the two doctors diagnoses.<\/p>\n<pre class=\"r\"><code>par(mar = c(4, 2, 2,2))\r\nlibrary(vcd)\r\n# Create the plot\r\np &lt;- agreementplot(anxiety)<\/code><\/pre>\n<p><img decoding=\"async\" src=\"https:\/\/www.datanovia.com\/en\/wp-content\/uploads\/dn-tutorials\/inter-rater-reliability\/figures\/205-inter-rater-agreement-chart-agreement-plot-1.png\" width=\"432\" \/><\/p>\n<pre class=\"r\"><code># Show the Bangdiwala agreement strength statistics\r\nunlist(p)[1 : 2]<\/code><\/pre>\n<pre><code>##          Bangdiwala Bangdiwala_Weighted \r\n##               0.644               0.795<\/code><\/pre>\n<p>Legends for the plot:<\/p>\n<ul>\n<li>Black square: exact observed agreement<\/li>\n<li>Gray square: partial agreement by including a weighted contribution from off-diagonal cells. By default, the weight w = 1 is assigned to all diagonal cells (exact agreement); a quadratic weight (<code>1 - 1\/(k - 1)^2<\/code>) is assigned to the one-step disagreements (or to the one-step off-diagnal cells, partial agreement). If you don\u2019t want to show partial agreement, specify the argument <code>weights<\/code> as follow: <code>agreementplot(anxiety, weights = 1)<\/code><\/li>\n<\/ul>\n<div class=\"warning\">\n<p>Note that,<\/p>\n<ul>\n<li>The visual image is affected if the order of the categories is permuted, and thus the use of agreement chart is recommended only for ordinal level variables (Bangdiwala et al., 2013).<\/li>\n<li>In the case of perfect agreement, the k rectangles determined by the marginal totals are all perfect squares and the shaded squares determined by the diagonal cell entries are exactly equal to the rectangles, producing a B-statistic value of 1 (Bangdiwala et al., 2013).<\/li>\n<li>Lesser agreement is visualized by comparing the area of the blackened squares to the area of the rectangles (Bangdiwala et al., 2013).<\/li>\n<li>It may happen that one rater consistently tends to classify the subject into higher or lower categories than the other, perhaps due to using stricter thresholds for the boundaries between adjacent categories (Friendly et al., 2015). This observers or raters bias is visualized by examining the deviation of the dark squares from the 45\u00b0 diagonal line within the larger N x N square (Bangdiwala et al., 2013; Friendly et al., 2015].<\/li>\n<\/ul>\n<\/div>\n<\/div>\n<div id=\"summary\" class=\"section level2\">\n<h2>Summary<\/h2>\n<p>This article describes how to create an agreement chart in R.<\/p>\n<\/div>\n<div id=\"references\" class=\"section level2 unnumbered\">\n<h2>References<\/h2>\n<div id=\"refs\" class=\"references\">\n<div id=\"ref-Bangdiwala1985\">\n<p>Bangdiwala, Shrikant I. 1985. \u201cA Graphical Test for Observer Agreement.\u201d <em>Amsterdam: International Statistical Institute<\/em>, 307\u20138.<\/p>\n<\/div>\n<div id=\"ref-Bangdiwala2013\">\n<p>Bangdiwala, Shrikant, and Viswanathan Shankar. 2013. \u201cThe Agreement Chart.\u201d <em>BMC Medical Research Methodology<\/em> 13 (July): 97. doi:<a href=\"https:\/\/doi.org\/10.1186\/1471-2288-13-97\">10.1186\/1471-2288-13-97<\/a>.<\/p>\n<\/div>\n<div id=\"ref-Friendly2015\">\n<p>Friendly, Michael, D. Meyer, and A. Zeileis. 2015. <em>Discrete Data Analysis with R: Visualization and Modeling Techniques for Categorical and Count Data<\/em>. 1st ed. Chapman; Hall\/CRC.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<p><!--end rdoc--><\/p>\n","protected":false},"excerpt":{"rendered":"<p>This article describes how to create an agreement chart in R.<\/p>\n","protected":false},"author":1,"featured_media":9209,"parent":0,"menu_order":0,"comment_status":"open","ping_status":"closed","template":"","class_list":["post-10309","dt_lessons","type-dt_lessons","status-publish","has-post-thumbnail","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v25.2 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Inter-Rater Agreement Chart in R : Best Reference- Datanovia<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-agreement-chart-in-r\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Inter-Rater Agreement Chart in R : Best Reference- Datanovia\" \/>\n<meta property=\"og:description\" content=\"This article describes how to create an agreement chart in R.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-agreement-chart-in-r\/\" \/>\n<meta property=\"og:site_name\" content=\"Datanovia\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.datanovia.com\/en\/wp-content\/uploads\/2019\/05\/Anserelle.naine_.2.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1024\" \/>\n\t<meta property=\"og:image:height\" content=\"512\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-agreement-chart-in-r\/\",\"url\":\"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-agreement-chart-in-r\/\",\"name\":\"Inter-Rater Agreement Chart in R : Best Reference- Datanovia\",\"isPartOf\":{\"@id\":\"https:\/\/www.datanovia.com\/en\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-agreement-chart-in-r\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-agreement-chart-in-r\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.datanovia.com\/en\/wp-content\/uploads\/2019\/05\/Anserelle.naine_.2.jpg\",\"datePublished\":\"2019-11-07T01:18:58+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-agreement-chart-in-r\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-agreement-chart-in-r\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-agreement-chart-in-r\/#primaryimage\",\"url\":\"https:\/\/www.datanovia.com\/en\/wp-content\/uploads\/2019\/05\/Anserelle.naine_.2.jpg\",\"contentUrl\":\"https:\/\/www.datanovia.com\/en\/wp-content\/uploads\/2019\/05\/Anserelle.naine_.2.jpg\",\"width\":1024,\"height\":512,\"caption\":\"SONY DSC\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-agreement-chart-in-r\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.datanovia.com\/en\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Lessons\",\"item\":\"https:\/\/www.datanovia.com\/en\/lessons\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Inter-Rater Agreement Chart in R\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.datanovia.com\/en\/#website\",\"url\":\"https:\/\/www.datanovia.com\/en\/\",\"name\":\"Datanovia\",\"description\":\"Data Mining and Statistics for Decision Support\",\"publisher\":{\"@id\":\"https:\/\/www.datanovia.com\/en\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.datanovia.com\/en\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.datanovia.com\/en\/#organization\",\"name\":\"Datanovia\",\"url\":\"https:\/\/www.datanovia.com\/en\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.datanovia.com\/en\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.datanovia.com\/en\/wp-content\/uploads\/2018\/09\/datanovia-logo.png\",\"contentUrl\":\"https:\/\/www.datanovia.com\/en\/wp-content\/uploads\/2018\/09\/datanovia-logo.png\",\"width\":98,\"height\":99,\"caption\":\"Datanovia\"},\"image\":{\"@id\":\"https:\/\/www.datanovia.com\/en\/#\/schema\/logo\/image\/\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Inter-Rater Agreement Chart in R : Best Reference- Datanovia","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-agreement-chart-in-r\/","og_locale":"en_US","og_type":"article","og_title":"Inter-Rater Agreement Chart in R : Best Reference- Datanovia","og_description":"This article describes how to create an agreement chart in R.","og_url":"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-agreement-chart-in-r\/","og_site_name":"Datanovia","og_image":[{"width":1024,"height":512,"url":"https:\/\/www.datanovia.com\/en\/wp-content\/uploads\/2019\/05\/Anserelle.naine_.2.jpg","type":"image\/jpeg"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-agreement-chart-in-r\/","url":"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-agreement-chart-in-r\/","name":"Inter-Rater Agreement Chart in R : Best Reference- Datanovia","isPartOf":{"@id":"https:\/\/www.datanovia.com\/en\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-agreement-chart-in-r\/#primaryimage"},"image":{"@id":"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-agreement-chart-in-r\/#primaryimage"},"thumbnailUrl":"https:\/\/www.datanovia.com\/en\/wp-content\/uploads\/2019\/05\/Anserelle.naine_.2.jpg","datePublished":"2019-11-07T01:18:58+00:00","breadcrumb":{"@id":"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-agreement-chart-in-r\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-agreement-chart-in-r\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-agreement-chart-in-r\/#primaryimage","url":"https:\/\/www.datanovia.com\/en\/wp-content\/uploads\/2019\/05\/Anserelle.naine_.2.jpg","contentUrl":"https:\/\/www.datanovia.com\/en\/wp-content\/uploads\/2019\/05\/Anserelle.naine_.2.jpg","width":1024,"height":512,"caption":"SONY DSC"},{"@type":"BreadcrumbList","@id":"https:\/\/www.datanovia.com\/en\/lessons\/inter-rater-agreement-chart-in-r\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.datanovia.com\/en\/"},{"@type":"ListItem","position":2,"name":"Lessons","item":"https:\/\/www.datanovia.com\/en\/lessons\/"},{"@type":"ListItem","position":3,"name":"Inter-Rater Agreement Chart in R"}]},{"@type":"WebSite","@id":"https:\/\/www.datanovia.com\/en\/#website","url":"https:\/\/www.datanovia.com\/en\/","name":"Datanovia","description":"Data Mining and Statistics for Decision Support","publisher":{"@id":"https:\/\/www.datanovia.com\/en\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.datanovia.com\/en\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.datanovia.com\/en\/#organization","name":"Datanovia","url":"https:\/\/www.datanovia.com\/en\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.datanovia.com\/en\/#\/schema\/logo\/image\/","url":"https:\/\/www.datanovia.com\/en\/wp-content\/uploads\/2018\/09\/datanovia-logo.png","contentUrl":"https:\/\/www.datanovia.com\/en\/wp-content\/uploads\/2018\/09\/datanovia-logo.png","width":98,"height":99,"caption":"Datanovia"},"image":{"@id":"https:\/\/www.datanovia.com\/en\/#\/schema\/logo\/image\/"}}]}},"multi-rating":{"mr_rating_results":[]},"_links":{"self":[{"href":"https:\/\/www.datanovia.com\/en\/wp-json\/wp\/v2\/dt_lessons\/10309","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.datanovia.com\/en\/wp-json\/wp\/v2\/dt_lessons"}],"about":[{"href":"https:\/\/www.datanovia.com\/en\/wp-json\/wp\/v2\/types\/dt_lessons"}],"author":[{"embeddable":true,"href":"https:\/\/www.datanovia.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.datanovia.com\/en\/wp-json\/wp\/v2\/comments?post=10309"}],"version-history":[{"count":0,"href":"https:\/\/www.datanovia.com\/en\/wp-json\/wp\/v2\/dt_lessons\/10309\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.datanovia.com\/en\/wp-json\/wp\/v2\/media\/9209"}],"wp:attachment":[{"href":"https:\/\/www.datanovia.com\/en\/wp-json\/wp\/v2\/media?parent=10309"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}