July 10, 2012
8 PDT/9 MDT/10 CDT/11 EDT/16 BST
Attendees: Linda Lewin, Alan Schwartz, Co-Chairs; Susan Albright, PJ Kania, David Melamed, Morgan Passiment, Howard Silverman, Valerie Smothers, Lori Troy.
1 Review minutes of last meeting
The minutes were approved as submitted.
2 Discuss possibility of linking events to milestones
Valerie reviewed the last discussion of the group on having portfolios reviewed by peers and being evaluated on something other than clinical performance. Valerie had a discussion with Kelly Caverzagie regarding how milestones may be used with EPAs in Internal Medicine. Based on the conversation with Kelly, Valerie didn’t think anything needed to be changed in the specification. She further that what medicine is calling milestones would be a competency framework in the MedBiquitous world. Milestones in internal medicine are constructed a little differently than in pediatrics. They are developmental in progression, but they are not functioning as levels of competence. Internal Medicine proposed something called narratives that would be used for that.
3 Review specification updates
Valerie continued with the review of specification updates.
She explained that summary element now refers to assessment results, not summary score. The reason for making the change was that assessment results have built into them the notion of learner notes and portfolio references to be included at the summary level. Valerie noted other changes beginning on page forty. She added “without direct supervision” to the definition of entrustment based on Carol Carraccio’s comment on the last call. On page forty-one and forty-two, we also changed the model for describing entrustment, added an element describing the source of the entrustment decision. The source has to be indicated. There can be multiple sources, such as listing each member of a committee. Or you can point to a URL describing the decision makers. Valerie asked the group if they had any questions or concerns.
Linda commented it looks like everything is included but it is hard to know how it will work when nobody is using it yet. She asked if anyone had started documenting this. David said they were pretty close to doing this but not exactly there yet. The part they are still trying to figure out is the interface. They are trying to define what EPA’s are entrustable and what are milestones. Don Bordley at Rochester is using EPA’s for points in time. Others are expecting EPA’s are gradually improving and are entrustable at higher levels as time goes on. The challenge is to understand how folks truly want to apply EPA’s.
Alan commented that the specification can accommodate either approach. Howard noted there is more gradation tracking the medical student.
Valerie explained we can do it both ways. There is element called entrustment that describes a binary thing, an entrustment decision regarding a specific activity, and details about that. There are also assessment results which can be a part of the summary. That allows you to report gradations related to competency. If you had an EPA modeled as competency, using the competency framework and assessment results, you could say this learner has this score related to this milestone or competency. The score could be characterized as specific levels.
Howard commented that their scale is observed, performed beginner novice and performed mastery. Valerie commented in MedBiquitous view that would be performance framework. Howard explained that it is a self report; the learner is reporting. Usually that is routed to a supervisor and then they don’t respond. It is really not assessment information but more like a self reported piece. He added it implies they are capable of preceding independently. Howard added that page forty-one says the summary score or entrustment is required. It should say assessment results or entrustment is required. Valerie agreed to correct. She clarified it could be a competency based score, task based score, or there could be entrustment data. Or you can have both assessment results and entrustment data. That is the current definition. She asked Howard what he would change. Howard commented that entrustment would not apply to medical students.
4 Discuss questions:
Portfolio reference instead of learner notes
Alan noted we are trying to cram a lot of portfolio information into the specification instead of allowing the learner to reference out to a portfolio. Based on Alan’s comments, Valerie recommended changing learner notes to portfolio reference within assessment results. Howard thought that data would fit here. He suggested adding portfolio reference and not taking out notes. Not everything will point to a portfolio reference. The group agreed.
Integrating different assessment types in a single display
Valerie commented that her next question related to using different scales and types of grades and integrating them into a single display. Alan commented you could use standardized scores and percentiles. Valerie asked if there were any concerns or thoughts about that. Alan added that you could use radar plots, where the same scales could be normalized. Valerie commented that it sounded like this is a non-issue.
5 Next steps
Valerie went on to discuss next steps. She commented that the specification is pretty strong at this point. She will take on as an action item to add XML examples in the specification for reference and ask some technical people to see how they would implement it. She added that if there are specific places or people using this development system we could share the specification with them and get their feedback.
David commented that they are a month away from having constructive feedback. People are now putting outcomes results together and there haven’t been any issues with the specification per say.
Valerie asked the group if we should we contact the boards? David commented that the ACGME newsletter indicated that Internal Medicine was expected to use the Next Accreditation System and apply milestones as of July 2013. The ACGME has said not to worry about subspecialties for July 2013. They don’t have milestones or EPA’s for subspecialties yet. A lot of evaluation structures will need to be updated.
Linda asked how to use frameworks at the medical school levels. Howard asked how it relates to the AAMC’s MedAPS work. Valerie commented the MedAPs effort of the AAMC is to develop a suite of tools for medical schools to evaluate their own performance. The Curriculum Inventory system is part of that: it will gather curriculum data from medical schools in the US and Canada and facilitates preparing for accreditation. Linda asked if the data was on the level of the school an is not collective learner data. Valerie replied that was correct; it is school level data. She didn’t know if they have plans to create aggregate data; however there are other sources that will go with curriculum data. The Educational Achievement standard is based on the Curriculum Inventory standard; it will look like data in Curriculum Inventory, but with learner specific results added.
Linda suggested having many IT folks looking at the spec. Howard suggested working with Morgan through the AAMC Group on Information Resources (GIR). Valerie will follow-up with Morgan. Morgan shared she may be able to help with the Residency programs implementation as well. Valerie will connect with the GIR on undergraduate and residency programs and connect with the boards since they aren’t on the call. Linda agreed. Valerie suggested if there are other residency programs that come to mind, let her know so we can get in touch with those people. Alan confirmed we are skipping the next meeting on July 24th so the next call will be August 7th. Linda commented if the review process takes too long we can just put it off till the next call.
- We will add portfolio reference to the data model for assessment results.
- The July 24 call is cancelled.
- Valerie will correct pg 41 typo, Summary Score instead of Assessment results.
- Valerie will add sample XML to the specification.
- Valerie will work with Morgan to have the GIR review the specification.
- Valerie will contact specialty boards and request their review as well.
- Valerie will also forward the specification to University of Maastricht as per correspondence with them.