Drug information quality assurance program used to appraise students' performance

A quality assurance program developed for a drug information service was used to objectively measure the performance of doctor of pharmacy degree students on rotation in the service. Two five-point rating scales based on objective criteria were developed so that judgmental and nonjudgmental drug information responses could be evaluated separately. Preceptors used the scales to evaluate 30 randomly selected responses from each of 22 Pharm.D. students (15 responses from mid-rotation and 15 from the end of the rotation). Written responses were also evaluated on a five-point scale by the requesters. In all of the evaluations, a score of 5 was excellent and a score of 1 was unacceptable. The preceptors assigned a score of 5 to 52.3% of the 660 responses evaluated, 4 to 29.1%, 3 to 12.9%, 2 to 4.5%, and 1 to 1.2%. No significant difference was found based on the academic quarter during which the rotation took place. The difference in scores between judgmental and nonjudgmental responses was not significant. End-of-rotation scores tended to be higher than midrotation scores, although the difference was not significant. Testing for interrater reliability revealed moderate agreement between raters. Requesters gave mean scores higher than 4 for accuracy, completeness, objectivity, and usefulness of the written responses. Timeliness scores were significantly lower but still acceptable. A drug information quality assurance program provided a structured, objective means of evaluating student performance.