January 30, 2014
8 PST/9 MST/10 CST/11 EST/16 GMT/17 CET
Attendees: Linda Lewin, Alan Schwartz, Co-Chairs; Bob Englander, Rachel Glassford, Scott Kroyer, Vijay Rajput, Sandhya Samavedam, Scott Smith, and Valerie Smothers.
Rachel introduced herself to the group. She serves on the Board of Trustees as their education physician fellow, for the American Medical Student Association.
1 Update on recent progress
Valerie began with an update that the Competency working group has developed a performance framework specification and she provided a link to the overview. The last time we discussed that was in December of 2012. We looked at the technical comments from the University of Maastricht who made decisions on what changes we needed to make to the specification to accommodate their work in competency-based portfolios. It was decided that this working group needed to wait until that performance framework specification was developed before we could move forward. We wanted to describe achievements before milestones and the existing specification couldn’t do it. It is just about complete so this is a good time to stitch the specification together and make sure it fits well and see if there are any changes that need to be made before this marriage.
2 Overview of Performance framework
Valerie continued with the conceptual overview of the performance framework slide deck. She would like to create a voice over Power Point some time but for now there is a script in the notes to follow along. Slide 2 shows how we are living in a world where competencies are very important for health professions education and are being applied across education. On slide three it talks about an important component of competency-based learning is concept of educational milestones which are expected levels of performance related to a competency. We are calling these performance levels together a performance framework. Usually it’s that novice to mastery spectrum. Slide 4 shows how the performance framework provides a consistent measuring stick for conveying how competent a learner is with regard to a particular competency.
Vijay asked about the terms used on this scale such as mastery. Valerie mentioned there were comments on the previous call reacting to the term “mastery”. They edited this particular slide extensively. Vijay suggested editing slide 3 to match slide 4. Alan reminded the group that the standard profives a flexible framework; the details of the content are not our issue. Linda mentioned if it’s easy to change the term in the previous slide she suggested to make the change for the sake of consistency. Valerie will take that suggestion to the competency working group. Scott Smith recommended making it more explicit that you are not picking one framework but providing a flexible way to encode the framework of your choice. Valerie agreed with his comment.
Slide 5 displays the benefits of standardizing performance framework; you have standardization across a discipline or specialty, across educational context, and standards make it easier to compile data within and across institutions. Slide 6 is goes through the example in Internal Medicine. The Metadata slide shows general information, the title, publishers, authors, and date. Slide 9 addresses supporting information for milestone reporting; high level descriptions can be included that point to something on the internet. Slide 10 is one page from the Internal Medicine milestones that gives you an example set of performance levels. We call that whole chunk a component. Often component is a set of performance level in a competency, but there may be nested components. It has the capability to accept that. Slide 11, highlights the competency and describes what it is that we are measuring. This competency object specification represents that. Slide 13 shows arrows pointing to each performance level. This competency has five levels of performance. On slide 14 performance framework defines the scale, what level represents the least competent and most competent. Woman asked if component is bigger than a competency. Valerie answered no not really. But a competency may be broken into facets or sub-competencies, with those facets or sub-competencies having different levels of performance.
Vijay asked how ready for unsupervised practice and competent were different. Valerie commented they were trying to define a numeric scale from 1-5. Most competent represents the best performance and least competent represents the worst performance. Slide 15 shows we are creating a scale of performance. The number of performance levels doesn’t have to correspond to the numbers on the scale. You can have a scale of 1-5 with three performance levels. Slide 16 displays the scores associated with each level of performance. It can also be a score range. When you get a learner’s score you can use the performance framework to interpret that score. The Educational Achievement spec would allow you to put the learner between a performance level, such as 3.4. The performance framework itself uses single numbers to define performance level; however, the score has the capability of being a decimal point.
On slide 17 each of those paragraphs is an indicator, a behavioral marker of what the performance level looks like. This allows people to track indicators separately if they want to but it is not required. Slide 18 is a different example of Transformation of Medical Education, in the University of Texas system. Slide 19 displays a set of performance levels that are used across all competencies; it’s a generic set of performance levels. There are seven levels that go from 1-6, including a 2 and a 2+.
Alan stated this was very elegant and flexible but asked what the specification doesn’t do. Is performance measurement tied to an ordinal scale of some kind? Valerie commented everything comes down to numbers. Alan gave an example of a more narrative assessment technique: given the examples of behavior, which example do you most resemble. It doesn’t necessary line up as an ordinal number. For some traits, like talkativeness, being in the middle of the spectrum is preferable to being at either end. Valerie confirmed you couldn’t do that with this framework. We haven’t seen any example of milestones or performance measurement scales where optimal performance is in the middle.
Valerie noted item three on the agenda links to a presentation. She noted the slides with the red titles are the ones that have changed. Slide 9 describes what the data looks like for an individual learner, using ACGME derived competency, on the right there is a set of generic performance levels that go from 1 being a beginner, to 4 advanced competent and 5 is mastered. The line lets you see where the learner is. The addition was adding “learner” to notes. This has the capability to include learner notes on the assessment and the learner can see their notes about this assessment, so they can improve. Slide 10 shows the heart of the change to the educational achievement specification, individual scores on a single assessment or competency conveyed may be somewhere in the middle 2.5 or 2.7; the educational achievement specification now supports that. Linda asked if this was an average of several scores. Valerie answered yes; the evaluator could say this learner is between two levels. She commented this is the point where the educational achievement and performance framework come together. Alan asked if that score can be any number. Valerie answered it can be any decimal point number. She continued with the next change on slide 13 where there is more detailed information about this learner’s performance on this competency and on this particular assessment event. Valerie reiterated the changes are in red. It can now say the site which it occurred, University of Maryland Medical Center and faculty notes, what we were calling sequence block. Slide 16 shows another big change to the specification related to entrustment. The Maastricht comments mentioned they don’t just have binary entrustment; they have levels of entrustment. We needed to come up with a way to have level of entrustment. On Slide 17, Valerie noted the entrustment scale that is being used to describe entrustment for this particular learner.
Alan asked what the entrustment scale was. Valerie answered the performance framework specification had so many things in there that were not appropriate; she created a different mini specification that is now a part of the educational achievement specification that allows one to represent entrustment scales. This is similar to the performance framework but the semantics are different. Alan recommended using the existing performance framework specification to represent entrustment scales. Linda agreed.
Valerie will modify the specification to see what that would look like and develop an XML example to show how that might work. She will take that as an action item for the next call. Either way we will have a way of conveying levels of entrustment. In this case there would not be decimal points, 2.5 wouldn’t make any sense. In slide 16 the ad hoc element was added so you can indicate an entrustment was made on an ad hoc basis. There is an EPA related to patient resuscitation that is the kind of thing ad hoc basis where a resident stepped in, they weren’t entrusted to do that. There is a flag to say this was done ad hoc. Linda asked if there were other examples. Valerie clarified the reason for the discussion we had the last time on ad hoc, was people wanted a way to distinguish between ad hoc and formal decisions. Alan provided the example of a family conference; he was wondering if they are really entrusted. Linda asked if it wouldn’t be a committee who would ultimately award entrustment. Valerie commented this is for representing entrustment with regard to a single incident. Alan mentioned they are not yet entrusted but they did it, so potentially the resident could be entrusted for that activity. Linda stated its’ evidence going forward. Do we have a source of entrustment by committee or person?
Valerie replied that entrustment has a title, description, an ad hoc indicator, a reference to a competency, and then there is the source, the date it was awarded, expiration date, and evidence that was used in making the entrustment decision. Linda commented the capability was there, it either comes from a committee or ad hoc.
Valerie continued with how we incorporate the scale used for entrustment. She noted on pages 55-56, the level of entrustment is described using an embedded reference or an external scale reference. With the external references we are using a new data type from RDF, Resource Description Framework. RDF is the basis for semantic web technologies. There is now a section called references in this document that describes how to use RDF to create references to competency frameworks, performance frameworks, and other external documents.
Valerie mentioned there are many other changes but there isn’t time to go over all of them. The event duration is now optional, and we can link to an external competency framework. Instead of having a notes field, we have faculty notes and learner notes for assessment results. A framework score may have a decimal point and references a performance framework.
Linda asked Valerie what she needed from the group next. Valerie commented if there are people who wish to develop the technical systems to measure competencies, or if we have anybody that is working with someone that could do a technical review, hand the specification and schema to your programmer and find out if this will work for your system. Or if you have a vendor you are working with you would like to review the specification that would be helpful.
Linda commented E-value is trying to deal with milestones. Scott Kroyer commented that e-Value’s milestone work is part of the reason he is participating in this group. He mentioned they have some new things that are going on right now. Valerie asked Scott if he could do a technical review from their system. Scott agreed to the review and asked if there was a timeline. Valerie mentioned the next call is on March 7th. Scott commented that would give him enough time.
Valerie offered to also ask David Melamed and his team to review document and suggested the group think about that and if any ideas come to mind we can follow up on that. She commented for next time, we’ll look at the technical reviews and also ask for a review of the work on using performance framework instead of separate entrustment scale specification. Valerie is going to do a lot of work to see how that fits. Linda mentioned she and Bob Englander will be at the AAMC on the date of the next call. Valerie suggested rescheduling the call for later in March.
4 Open discussion
Suggestions for performance framework voice over powerpoint
- Change Mastery to Expert on slide 3.
- Make it more explicit that you are not picking one framework but providing a flexible way to encode the framework of your choice
- Valerie will revise the specification to use the Performance framework to represent levels of entrustment and develop sample XML to show what that might look like.
- Scott Kroyer will work with his staff to do a technical review of the specification.
- Valerie will contact David Melamed regarding his company doing a technical review.