Metrics Working Group Meeting at MedBiquitous Annual Conference
May 15, 2008
Attending: Linda Casebeer and Francis Kwakwa, co-chairs; Andrew Fawcett, Christy Gaughan, Don Holmes, Tao Le, Rachel Makleff, Leisa McMullen, Andy Rabin, Pesha Rubenstein, Valerie Smothers (staff), David Topps.
Francis began the meeting with an overview of the working group and its accomplishments. The Metrics working group is developing technical specifications for evaluation data. The specification would facilitate a single organization compiling their evaluation data from across the enterprise. It would also enable exchange of evaluation data between organizations. The specification includes information about learner demographics, the course evaluated, participation metrics, and reaction data. We have been focusing on Kirkpatrick level 1 evaluation data, the learner's reaction to a course, which is captured as the results of learner surveys. We have a draft specification and an XML schema for this functionality. In addition, we have announced the medical education metrics work to ANSI as a standards development activity. This means that ultimately, the specification will become a national standard.
Linda added that one thing the group couldn't agree on were the questions within these surveys. As a result, specific questions have been taken out of the specifications and instead there is a reference to an external set of questions. With this approach, there needs to be a way to catalog evaluation items, the survey questions, in addition to the other specification. The Alliance for Continuing Medical Education is developing an open access web site for survey items in order to facilitate evaluation and research. They have received final board approval for this web site.
Linda clarified that in some cases, it does not make sense to use a standardized survey item. For example, questions about individual learning objectives will likely differ from course to course, but a question regarding commercial bias will likely be the same across many courses. Therefore, it makes sense to collect data on these items to enable validation of tools and items. The alliance web site will have a form that allows evaluators to add their survey items and indicate whether or not they have been validated.
The group went over the sample survey item and metrics report distributed as handouts. Valerie then went over the survey item diagrams which illustrate the data model of the survey items schema. Valerie commented that the format of the Comparison element may need to be modified. When evaluators compare questions results from different points in time, the questions asked at those different points in time, while related, may not be the same. Andy agreed there may need to be some modifications and offered to review the data model in detail and provide suggestions.
Linda commented that now that Healthcare LOM is a standard, it will be important for other in the industry to adopt this standard, particularly the Alliance for Continuing Medical Education. Valerie agreed to follow up with them. Linda also asked that Valerie get in touch with everyone after the meeting to provide links to the wiki and listserv and encouraged participation and pilot projects.