Posted on Sunday, April 30, 2017

Sampling: An Irrefutable Law in Rater-based Assessments, or is it?

RBC Convention Centre, York Ballroom


Moderator: Timothy J. Wood Ph.D., University of Ottawa

Discussant: Kevin W. Eva, Ph.D., Hon. FAcadMEd, University of British Columbia

Chairs: Ryan Brydges Ph.D., Assistant Professor, Department of Medicine, University of Toronto & Scientist and Associate Director, The Wilson Centre, UHN and Dr. Marguerite Roy, Ottawa (MCC)


The advent of competency-based education and its accompanying broadening of focus into areas like programmatic assessment and entrustable professional activities creates an increased reliance on rater- based assessments. Such assessments are not new to medical education despite their use always being accompanied by concerns about rater-driven variability. Increasing the number of independent samples of performance either through an increase in the number of cases or raters is commonly promoted as the best solution for reducing variance attributed to rater-induced measurement error. Recent rater cognition research, however, has begun to uncover findings that suggest boundaries to the argument that increased sampling is sufficient and should cause the profession  to re-think its strategies for designing defensible assessments.  In  this symposium, we will  explore the notion  of sampling  as a panacea for rater-based assessments along with ways in which rater-variability might better inform decision-making.


Learning Objectives: As a result of this session, the audience members will be able to:

  • Discuss the role of multiple independent samples in rater-based assessment from psychometric and post-psychometric perspectives;
  • Critically examine assumptions related to sampling in the context of rater-based assessments, including how rater perspectives should or could be integrated;
  • Discuss the implications of assumptions related to sampling in rater-based assessment and potential future directions.




Timothy J. Wood received his PhD in Cognitive Psychology from McMaster University in 1999 and then spent 12 years with the Medical Council of Canada (MCC), where he collaborated on a number of MCC projects including computer and web-based examinations, a national OSCE, result reporting, and  scoring support to the examinations. Dr. Wood has been a member of the Department of Innovation in Medical Education at uOttawa since 2011. His research interests include quality assurance measures for assessments and developing tools to assess the impact of educational interventions, and the assessment of clinical skills. Of particular interest is the role of rater cognition, especially around first impressions. Dr. Wood is an associate editor of Advances in Health Sciences Education, served two terms on the executive board  for the Canadian Association for Medical Education, is the Chair of the CCME Abstract selection committee and is the Director of DIME’s Healthcare Education Scholars Program.

Back to top