March 10, 2014
10 CDT/11 EDT/15 GMT/16 CET (note DST clock change!)
Attending: Linda Lewin and Alan Schwartz, Co-Chairs; Susan Albright, Carol Carraccio, Bob Englander, Kimberly Hoffman, Scott Kroyer, Vijay Rajput, Sandhya Samavedam, Valerie Smothers, Hugh Stoddard, and Lori Troy
1 Review minutes of last meeting
The meeting began with a brief introduction of Hugh Stoddard, who is the Assistant Dean for Medical Education at Emory. He has been involved with the Curriculum Inventory for several years and he wanted to find out how this work will impact individual students.
The minutes from the last meeting were approved as submitted.
Alan reviewed what happened since the last call. He made a point of explaining instead of having a new way of describing entrusted scale, we need to define a set of levels of entrustment for many EPAs using the existing Performance Framework specification. The group is tasked with implementing that and to show how it would fit into the schema with examples in XML using a mock up entrustment scale for levels of entrustment and encode using the Performance Framework. Slide two shows the metadata identifier and title of “Nosuch University Entrustment Levels”. Slide three provides general information about the scale overall, the ID assigned to it, and the performance scale range from least competent to most competent, 5 being most competency and 1 being least competent. Slide 4 is the Component of Performance Framework. The ID and title given are the identical title of the Performance Framework. There is one performance scale and one component for Nosuch University and it refers back to the performance scale. Slide 5 shows the first performance level again as a fictional example of how it would be coded. Valerie mentioned that Carol came up with nicer descriptions. On the first one a score of 1 label was called observation, and the related indicator was “entrusted to observe only”. She noted there are multiple indicators in a regular Performance Framework. Level two 2 would be entrusted to perform under heavy supervision, to level 5 which is entrusted to perform without supervision. Valerie reminded the group that these are only for illustrative purposes.
Valerie briefly presented the specification document changes beginning on page twenty-five where entrustment scales were removed. It describes the Performance Level or Entrustment levels, and the performance framework definition has been expanded to incorporate a new way of using the Performance Framework. On pages 48 and 49 points back to the Performance Framework specification. Page 53 level of entrustment has changed, indicating level of entrustment decision in two parts. The first part is the score as an integer, 1, 2, 3, 4, or 5, and it aligns directly with one of those levels, and then there are references that will point to Performance Framework. This lets you interpret the score. This sample XML and the code link to the sample XML shows what all of this looks like. It lets you see the line number, it’s line 554. For example, the care of a healthy newborn, points to entrustment scale and the XML defines entrustment levels.
This document was presented to Competency working group and Susan was on that call. The document was received favorably by the group and they like how everything was integrated. They liked the use of Performance Framework to represent entrustment scales with one caution; UCSF has the potential for enormous need to have guidelines to make it simple. Sashca Cohen suggested the use of the EPA developed by the AAMC as an example in our work. The group supported that idea and it can be incorporated when guidelines are developed. Susan asked whether there is an opportunity for an overall score of 4.5. Valerie informed her there was not for entrustment level and she further explained that if you looked at the entrustment scale, you wouldn’t entrust somewhere between heavy supervision and light supervision. Susan was thinking in terms of if there were seven things under consideration, could you have a non-integer result as part of the framework. Valerie noted when describing a score of assessment, or summary of assessment results, you can have those kinds of integers. In professional competency twelve different assessment scores could be 3.25 but not for entrustment. Linda commented she thought it was great, clear and simple instead of making a whole new set of guidelines. Woman said it aligns well with the literature and commended Valerie on her hard work. Alan agreed. Valerie asked Alan about the data model Performance Framework. Alan thought it was fine and is happy with the way it came out.
3 Review Feb 19 Competency working group discussion - favorable review of performance framework integration
Valerie mentioned to the group there is a link to the minutes if they want more details about the Competency Working Group call, item two. She noted the third paragraph on the end is a good summary to review. Tim asked the group for approval, adding Sascha’s concerns. Scott Kroyer was also on the call, and agreed with the advice for flexibility and guidance on how to implement this.
4 Next steps
Valerie mentioned if the group was happy with the specification; the next step is that the Performance Framework specification has to be balloted as a MedBiquitous standard. There is one institution we are still waiting on to make sure we can accommodate them. Afterwards we will put the Performance Framework specification through the Standards Committee, then a public review, and any changes go back to the Competency group. Then the Educational Achievement specification can be balloted. Linda asked Valerie if there was anything they needed to be doing right now. Valerie commented it is a hurry up and wait situation. Someone asked what Pubic Comment meant. Valerie announces the specification on the MedBiquitous website and newsletter. A public review is as much for other standards development organizations as our community health professional educators. When you get to that point, people pay attention and they find things they want to change which is a good thing. Any sustentative changes you have to extend the ballot for six weeks, and that adds time to the process. It’s human nature when you vote on something that is when you find the problems. Typically there are not a lot of comments coming in if any. There are two things we are still looking for technical reviews from e-value and myevaluations.com. Valerie advised them to keep in touch with the group and do a review if you see anything and send a note out to the working group. Valerie will contact David Melamed, from Myevaluations.com. Sandhya commented their group used Myevaluation, and they do have some scales for EPAs 1-5 and different grades. Valerie asked them to review standard. Sandhya will contact them. Scott will keep in touch about the technical review and send comments to the group. Linda asked Valerie if the group will go on sabbatical until Valerie calls us out of retirement. Valerie noted there is a lot going on in this space, and to call us regarding any new developments on competency based assessments.
5 Open discussion