September 11, 2012
8 PDT/9 MDT/10 CDT/11 EDT/16 BST
Attendees: Linda Lewin and Alan Schwartz, Co-Chairs; Susan Albright, Carol Carraccio, Kelly Caverzagie, Robyn Herring, Larry Hurtubise, Valerie Smothers, Kevin Souza, Janet Trial, and Lori Troy
1 Review minutes of last meeting
The minutes were accepted as submitted.
2 Discuss feedback received on technical review
Valerie began by stating a few people have communicated and promised feedback but have not sent it yet. Jan Carline from University of Washington reviewed specification and sent it to others at UW. They are not on competency based curriculum yet, but are implementing new learning management system. They compared the document to their requirements. Jan liked the spec but raised a question regarding ho you indicate if a learner has repeated an activity (ie retaking a course or examination). An example from the MSPE was included.
Alan commented that the current document would indicate if a learner took a course more than once. It could show one instance of the course where the learner did not pass and another where they did. The same would be true of examinations. Lori commented that a reviewer would have to drill down to know if courses were repeated, if they graduated in more than 4 years, etc. Alan replied that was more of a display issue. Valerie added that if you look at Educational Achievement data together with Educational Trajectory, you can get the big picture of learner’s path through medical school.
Valerie asked if mapping the MSPE to the specification would be helpful. Linda commented it’s a moving target and agreed with Alan that it was a display issue. Applications could call out that data. Lori commented that the importance of such features is really dictated by one’s educational view. In competency-based systems, you want to know that the learner achieved a particular competency. Others want to know if a person is at the top or bottom of their class.
Valerie moved onto the next set of comments from the Medical College of Wisconsin, where she spoke with Deborah Simpson, Pamela Holt, Dawn Bragg, and Kay Harbach. Their comments weren’t about the specification, but more about the way competencies were implemented. Deborah wants a more granular approach to reporting competency assessment information. For NBME subject matter exams, they have that information, the NBME has that information, they just don’t share it. Valerie commented it was a policy issue, not a specification issue.
Valerie had a call with Moises Eisenberg at Stony Brook and they talked about underlying technical use of standard and he strongly recommending the use of RDF instead of XML. She agreed to pass on his comments. No one else has argued for a change to RDF, so there is not enough of an opinion to change the specification to RDF at this time. Linda asked for an explanation of RDF verses XML. Valerie answered XML uses a hierarchical structure. RDF uses triples, a subject-verb-object like construction. Instead of saying date, is September 11, 2012, you would have to have a whole sentence around that. It goes into the relationships around data. There are advantages to using RDF, but it’s a higher technical bar and a little harder to do. Valerie expressed concern that if we move to RDF we’ll lose adoption. Kevin agreed we should move forward with XML.
Valerie commented there is more to come from Jean Berlo and Jeroen Donkers at the University of Maastricht. They created the ePASS system used by medical schools in the Netherlands. They are in the process of mapping data in ePASS to elements in the standard. The pictures we have in the John Doe slides are very similar to what their system is doing, so their feedback will be important. As soon as Valerie receives feedback she will share it with the group.
The next set of comments was from Larry Hurtubise at Ohio State University. he provided somments on the definitions beginning on page 15. The definition of event is a single educational or assessment activity documented in a curriculum. Examples of events included: lecture, problem-based learning activity, virtual patient activity, simulation activity, seminar, lab, and bedside learning activity, to name a few. Larry recommended that the examples line up with the instructional methods defined in the curriculum inventory. Valerie agreed to make that change. Next, with regard to the definition of Entrustable Professional Activity, he recommended changing the explanation to “Competency is characteristic of individual.” The group agreed with the change.
For the definition of Performance framework, Larry recommended including conditions, ie Is the competency performed with a standardized patient, supervised, unsupervised with real patients, etc. Valerie commented that in her experience, the performance frameworks exclude conditions. Kelly commented that the context in which you perform that competency is unique and independent for each observation. In the Internal Medicine milestones, they don’t call out conditions; they allow programs to have flexibility.
Larry also had a comment on the definition of statement of awarded responsibility (STAR). A STAR is the point at which entrustment for a specific EPA has been awarded, to be certified or competent to perform in a clinical setting without supervision. Alan commented that there are EPAs that are performed without patients and outside of the clinical setting. Valerie commented that we don’t actually use the word STAR, we use the word entrustment. The group agreed to remove the definition of STAR and add a definition of entrustment that includes “without direct supervision.”
Larry also recommended minor edits to page 25 section 8.3.1 under Event. The group agreed with the language. Valerie thanked the group for providing good feedback. She has had ongoing conversations with some people at the American Board of Pediatrics and will be setting up a time to talk through some of their questions.
3 Next steps
Valerie is waiting for the performance framework specification to be written by the competency working group. Thanks to accelerated funding from the AAMC we should have specification by the end of March. That will allow us to finalize this specification, so we’re in a holding pattern until then. We can send this sepecification to the Standards Committee for review and ballot once the Standards Committee once the Performance framework specification is ballotted. At that time we may need a call to go through comments, but that is months away.
At the AAMC annual meeting, Terri Cameron is speaking Nov 5th at 1:30 in a session describing curriculum performance, e-folio connector, and MedBiquitous. Valerie will send details to the group. Linda requested details of related sessions be sent to the group as well.
- The specification can accommodate the remediation example and no changes are necessary to describe remediation (the same course can occur multiple times and the same exam can occur multiple times)
- We will move forward with XML as the basis for the specification
Valerie will make the following changes to the specification:
- Change the definition of Event to use the instructional methods in the curriculum inventory spec to describe the type of activity.
- Add the word characteristic to the definition of EPA (competency is characteristic of an individual)
- Replace the STAR definition with a definition of entrustment that uses the language “without direct supervision.”
- Make edits to page 25; use the word activity in the definition of Event and remove unnecessary language. (see hurtubise edits)
Valerie will send a list of related AAMC sessions to the working group & work on a happy hour schedule.