Collecting Validity Evidence: A Hands-on Workshop for Medical Education Assessment Instruments

Caroline R. Paul, Michael S. Ryan, Gary L.Beck Dallaghan, Thanakorn Jirasevijinda, Patricia D. Quigley, Janice L. Hanson, Amal M. Khidir, Jean Petershack, Joseph Jackson, Linda Tewksbury, Mary Esther M. Rocha

Research output: Contribution to journalArticle

Abstract

Introduction: There is an increasing call for developing validity evidence in medical education assessment. The literature lacks a practical resource regarding an actual development process. Our workshop teaches how to apply principles of validity evidence to existing assessment instruments and how to develop new instruments that will yield valid data. Methods: The literature, consensus findings of curricula and content experts, and principles of adult learning guided the content and methodology of the workshop. The workshop underwent stringent peer review prior to presentation at one international and three national academic conferences. In the interactive workshop, selected domains of validity evidence were taught with sequential cycles of didactics, demonstration, and deliberate practice with facilitated feedback. An exercise guide steered participants through a stepwise approach. Using Likert-scale items and open-response questions, an evaluation form rated the workshop's effectiveness, captured details of how learners reached the objectives, and determined participants' plans for future work. Results: The workshop demonstrated generalizability with successful implementation in diverse settings. Sixty-five learners, the majority being clinician-educators, completed evaluations. Learners rated the workshop favorably for each prompt. Qualitative comments corroborated the workshop's effectiveness. The active application and facilitated feedback components allowed learners to reflect in real time as to how they were meeting a particular objective. Discussion: This feasible and practical educational intervention fills a literature gap by showing the medical educator how to apply validity evidence to both existing and in-development assessment instruments. Thus, it holds the potential to significantly impact learner and, subsequently, patient outcomes.

Original languageEnglish (US)
Number of pages1
JournalMedEdPORTAL : the journal of teaching and learning resources
Volume15
DOIs
StatePublished - Apr 12 2019

Keywords

  • Assessment
  • Curriculum Development
  • Editor's Choice
  • Evaluation
  • Quantitative Research
  • Validity Evidence

Fingerprint Dive into the research topics of 'Collecting Validity Evidence: A Hands-on Workshop for Medical Education Assessment Instruments'. Together they form a unique fingerprint.

  • Cite this

    Paul, C. R., Ryan, M. S., Dallaghan, G. L. B., Jirasevijinda, T., Quigley, P. D., Hanson, J. L., Khidir, A. M., Petershack, J., Jackson, J., Tewksbury, L., & Rocha, M. E. M. (2019). Collecting Validity Evidence: A Hands-on Workshop for Medical Education Assessment Instruments. MedEdPORTAL : the journal of teaching and learning resources, 15. https://doi.org/10.15766/mep_2374-8265.10817