June 5, 2012
8 PDT/9 MDT/10 CDT/11 EDT/16 BST
Attending: Linda Lewin and Alan Schwartz; Co-Chairs: Carol Carraccio, Mike Dugan, Kelly Caverzagie, Reid Kallman, Sandhya Samavedam, Scott Smith, and Valerie Smothers
1 Review minutes of last meeting
Minutes were approved as submitted.
updates to score structure
Valerie continued with an update on the specification. The changes started on page fourteen with changes to the terminology. New definitions include educational trajectory, event, sequence block, and integration block. These concepts were taken from other specifications. Alan recommended referencing the source of the definitions. Valerie agreed.
Valerie continued describing the changes. On page twenty-three the note indicates that the PESC specification is being modified to include more contextual data about grades. On page twenty-eight the language used was modified to be more general as assessment results may be summary results across a course or curriculum. On page twenty-nine and thirty, what we had described related to performance levels milestones wasn’t quite right; it did not allow for the capability to have a composite score that might be a decimal 9ie 3.2 on a five point scale where each number denotes a different performance level or milestone). She created a new element called framework score to correct that problem. On page thirty-six you can see what score framework looks like. There is decimal point score, reference to a competency, and a reference to the scale being used to generate the score so you have a means to interpret the score.
Valerie asked the group if that seemed like something that would work for implementation of milestones. Linda thought it made sense to her. Scott commented it made sense for Internal Medicine. Alan agreed it makes sense to get framework described, and competency attached.
Valerie continued with changes to page thirty-eight where she tweaked the portfolio reference description so that it is applicable for sequence blocks as well events. She added language indicating that the URI may point to supporting evidence. Page forty included additional documentation. Valerie noted we have something fairly complete, although changes will certainly continue as we vet the specification with sample data.
3 Open questions
Does awarding of responsibility for an EPA/skill happen as part of an assessment or after an assessment? Do you want to track who does the awarding?
Valerie asked the group when you award a learner responsibility for an EPA: is that happening as part of assessment or after the assessment? Linda asked if had to be either or. She mentioned in certain procedures you do get credentialed/certified after you do the assessment. There might be other things that are based on being observed over a period of time. Carol commented that in Olle ten Cate’s example, awarding of entrustment happens when two people signed off, based on some discussion and comparison of notes. Where the ACGME is going will require a competency committee to make that decision. We need to have the capacity to have more than one voice involved in the judgment.
Scott commented at the last program directors meeting, they felt strongly that committees make better judgments than individuals. There are objective evaluations and subjective elements indicating that globally the learner is ready, able, and has accomplished enough to do this. Linda commented that the discussion speaks to the fact that we need both ways. Valerie asked Jan about the undergraduate medical education Skills and Thrills curriculum and the way entrustment occurs. Jan shared that the basic clinical skills ready the person to be at the patient’s bedside; there is no assessment other than they moved through the practice. They are not signed off on particularly, they’ve move through the curriculum and done the practice. Valerie thought that it sounded like there is another piece missing related to entrustment. It could be part of an assessment or independent. Valerie will take as an action item to add piece on entrustment to specification.
Valerie asked the group if they had a chance to review the note sent by David Melamed. She asked for any reaction to his note and wondered if that kind of maintenance of an EPA is something many people are moving forward with. Scott commented most systems are thinking about the need to do this. Most are granting time limited privileges because of that. Alan commented that it is a decision-making issue as to whether credentials need to be reassessed. Valerie asked how it is being implemented. Scott commented that the hospital uses it for privileging. They use a simulation to assess the ability to do an intubation, and that privilege is time limited. Linda mentioned the specification may need to have a field that explains how long entrustment is good for or an expiration date.
Carol commented that some EPAs don’t have to do with a procedural skill but are more about deliberate practice. If one of your EPA’s is caring for a normal newborn, how do you demonstrate that skill to achieve entrustment? Is there a way for an individual to track what they consider to be deliberate practice in that area? The learner could attended “x” amount of deliveries; you could enter or not use the field. Kelly commented if you put it in there people will tend to fill it out. He added that certain things like critical judgment don’t expire. As trainees move towards expert and master, they should not have to go back and be recertified in those EPAs. Valerie commented we would need guidance around the implementation of EPA’s in general. Linda added that the concept is so new that we should leave the specification open to that ability.
Scott commented that an EPA might be the ability to work in interdisciplinary team. That is more of a threshold developmentally; once you pass you don’t need to retest. Some tests will be subjective and some objective. Carol commented that we want to drive expertise; development doesn’t stop with entrustment. The way you assess is important. Scott commented in the MOC process they want to see something like a graduate thesis or portfolio where a group of your peers would judge individual abilities over time. Carol agreed that was a good idea; it is meaningful to learn by doing something that is important to your work and have peers judge that. Scott commented it is hard for institutions to do. PhD defense portfolio is based on presentation to peers and experts. Linda shared that in the educational trajectory, we incorporated the notion of a testimonial, where another person verifies and describes your achievement. Valerie agreed to investigate the extent to which we could support a peer driven portfolio review in the standard.
Should we allow for pointers to external performance frameworks and competency frameworks, or is the descriptive data in those important for research? Will those frameworks be accessible and be maintained?
Valerie asked the group how important it is to have data embedded in a single file altogether. For example, the curriculum inventory specification is designed to support medical schools sending curriculum data to the AAMC to a central repository to be used for research. It pays to have competencies embedded in that data, so you can search for specific competency structure. Another way to structure the data would be to allow the data to point to a set of competencies maintained on an external website. Valerie explained the data will travel on demand, and things will be inside a single document. But research is one important use case. Alan commented there could be benefits to having greater standardization that comes from everyone referencing a set of external competencies that remain consistent. He added that he had no strong opinion. Valerie asked Reid to comment on how Stanford made decisions regarding what to include in the transcript and what to reference. He commented that they flip flopped on this one; there is no clear indicator. Valerie was glad to hear it is a struggle for other groups too. She will take this as an action item to bring up to the TSC and report back to the group.
4 Open discussion
EPAs described for learners should have the ability to include expiration dates.
- Valerie will add references to the source of terminology definitions
- Valerie will add a piece on entrustment to specification
- Valerie will investigate the extent to which we could support a peer driven portfolio review in the standard
- Valerie will ask the Technical Steering Committee for guidance as to what data to include and what to reference.