February 19, 2014
12 PM PST/3 PM EST/8 PM GMT
7 AM Feb 20, AEDT (Melbourne time)
Attending: Tim Willett, Co-Chair; Chris Andrews, Terri Cameron, Mary Jo Clark, Stephen Clyman, Sascha Cohen, Bob Englander, Ian Graham, Linda Gwinn, Kevin Krane, Scott Kroyer, Deborah Larimer, Steve Lieberman, Karen Macauley, Paul Schilling
1 Review minutes of last call
Tim began with a recap of the issues from the last meeting. The first issue discussed was about supporting information element. The recommendation from the Technical Steering Committee was to stay with XHTML. There were a number of suggestions for the voice-over powerpoint script and Valerie will address those later on the call. Mary Jo had a minor revision to the minutes; the Benner model was misspelled in a paragraph on slide four. Mary Jo moved to approve the minutes as amended; it was seconded and unanimously approved.
Valerie explained that the Educational Trajectory Working Group is group that is working to develop the Educational Achievement specification that describes the individual learners achievement’s. One component described within the specification is competency using performance framework as a basis for measurement. There was a placeholder for the performance framework in the Educational Achievement specification; now it’s time to stich things together. On January 30 the Educational Trajectory working group looked at the XML Educational Achievement sample XML to see how that was integrated. The spec uses the Performance Framework when describing assessment results in a Framework score. The learners score could be a number like 1 or 2 or it could be a decimal like 1.2 etc. There is a reference that points to a component in the Performance Framework. The working group thought that looked fine. Alan Schwartz, who co-chairs the Educational Achievement working group, thought this was very elegant.
One of the changes to the Educational Achievement specification resulting from the technical review from the University of Maastricht was incorporating levels of entrustment. They award different levels of entrustment, such as performing activity under light supervision, heavy supervision or without supervision or indirect supervision (as in the US) and these levels of entrustment needed to be described. The Educational Trajectory Working Group asked that the Performance Framework be used to define levels of entrustment in an entrustment scale. The Mockup of Entrustment scale shows what that looks like.
Tim questioned whether this is a generic framework for entrustment that could be applied to any EPA. Valerie answered yes and no, she commented emphasized that some people define entrustment levels differently. At Maastricht they have a level indicating no supervision. In the US people react differently and feel some level of supervision is needed even if it’s indirect. The scales could be a little different but overall it’s generic. Tim commented that en EPA would also need to be referenced for this to make sense. Valerie confirmed that the Educational Achievement specification provided a space for that reference.
Tim asked whether the Educational Achievement specification is meant to represent a single assessment event or a score that is the product of multiple assessment events. Valerie answered it has the capability for both.
Mary Jo noted this was something similar to what they did when they redesigned their clinical evaluation tool. She offered to share that tool. Valerie asked if it would be ok to share with this working group and other working groups. Mary Jo answered yes. Valerie commented that would help to make sure we are not missing anything.
Tim asked whether the working groups were OK with the Performance Framework specification being used in this way. He thought that was a good approach. Valerie asked if anyone had any concerns from a technical standpoint. Sascha commented he didn’t have any concerns right now but that the use of the specification has the potential for enormous complexity. He encouraged documentation on best practices and guidelines for those who want to use the specification. Scott agreed with Sascha and mentioned the overall specification may need a great deal of complexity and flexibility to encompass everything it needs to accomplish. Prototypical implementations or usage scenarios would be helpful. Valerie thought that was a great idea. Valerie suggested maybe working with specific groups that are trying to move in that direction, like the PIVIO team. If the ACGME would be interested in collecting data using our standards, maybe there is a subset ACGME needs.
Mary Jo noted that CCNE would be interested in aggregate data. They can pull aggregate data out of that. Valerie will talk to Mary Jo offline.
Chris asked if the group gave any thought to mapping frameworks to one another so that data can be transferred across institutions. Valerie will follow up with that as well. Karen commented that inter-rater reliability is important. Tim mentioned the information element would still be available. He asked the group to send a message to Valerie or the group to express their ideas or concerns.
Valerie continued with a brief discussion on the voice over power point changes made on January 9th. The changes came from the minutes and Mary Jo’s emailed comments. In addition there were two suggestions from the Educational Achievement working group. They commented that the difference in terminology (mastery vs expert) on slides 3 and 4 was confusing. They recommended changing slide 3 to make it the same as slide 4. Mary Jo had difficulty accepting that suggestion; she preferred mastery verses expert. Deborah & Ian agreed. The group agreed to change slide 4 from expert to mastery. Tim suggested to keep the (e.g.) in front of it to indicate it’s an example not a prescribed term.
Valerie continued with the next comment from the Educational Achievement working group. They wanted to make it more explicit that you are not picking one framework but providing a flexible way to encode the framework of your choice. Tim suggested including a slide between slide 5 and 6 that talks about the generic nature of the spec. Mary Jo agreed with adding an extra slide, but she put it between slides 2 and 3. Tim thought putting it there may interrupt the flow. He suggested putting it after 5 or between 3 and 4, saying so that we call it “x” but you may call it something else. The terminology is generic but you might use something else. Valerie will take a shot at adding a slide and send it back to the group for review. Tim thought it made sense to declare impartiality up front.
Valerie walked through the changes based on Mary Jo’s comments. The group agreed with the changes. On slide 5 it says compilation of performance data across courses/units/systems. Is that an evaluation of progression over time or does progression over time reference the learner. Tim thought Valerie was thinking program. Tim suggested progression over time be added. Mary Jo noted when CCNE looks at accreditation of programs involved in continuing quality improvements they add the terminology continuous program improvement.
Valerie mentioned the next step was to actually record the voice over power point. She will take that as an action item. She commented the Educational Achievement working group needs to review items this group has reviewed today and give their blessing. If we get the Educational Achievement specification nailed down and this specification gets nailed down then we can transition them to the Standards Committee. The Educational Achievement working group is dependent on the work of this group.
The only change we have outstanding is moving competence reference to performance level indicator. We are doing that to support the work out of Tulane. Deborah mentioned they have been working to revise their model. By the next call Deborah may have something to report regarding their changes. Valerie commented we will work to schedule the next call and put that on the agenda.
If the Standards Committee takes issue with anything during their review, it would come back to this working group, and the working group would have to send the proposed revision back to the Standards Committee. The whole process takes several months. Valerie will speak with Deborah about a reasonable time line to see where Tulane is in terms of scheduling the next call.
4 Updates from working group members
5 Open discussion
- Mary Jo and Karen will share the performance framework used in nursing at USD
- Valerie will add a slide clarifying the generic nature of the specification.
- On slide 5 Valerie will edit “compilation of performance data across courses/units/systems” and add “progression over time.”
- Valerie will record the voice-over powerpoint.