June 13, 2012
8 PDT/10 CDT/11 EDT/16 BST
Attending: Rosalyn Scott and Tim Willet, co-chairs; Susan Albright, Mary Pat Aust, Dana Bostrom, Rod Campbell, Kelly Caverzagie, Steve Clyman, Bob Englander, Josh Jacobs, Valerie Smothers (staff).
Tim reviewed that we discussed definitions. We are coming together on the concepts and how they relate to one another. Valerie and Rosalyn provided a demonstration of how EPAs may be captured using the competency framework; there may be some pieces missing. At the conference we had a conversation that provided a lot of background on the use cases; from quality assurance, to undergrad, post grad, nursing, medicine, etc.
Valerie asked the group to email her any revisions for the minutes.
Kelly described how they got to the point of developing EPAs. Their work began in 2007 with defining Internal Medicine milestones. At the time, they were responding to the ACGME & ABIM. The article and long list of milestones resulted. The milestones are a developmental, observable, stepwise progression through a competency. The next step is to have the community apply the milestones in assessment. They ran into a problem: the milestones are reductionist and had the potential to deconstruct into a checklist. A resident could meet the checklist but not the broader goal. They began to learn more about Entrustable Professional Activities (EPAs). Over the past 18months they have looked at EPAs. The slide presentation on EPAs is an abbreviated copy of a presentation to a program directors group. Page 3 shows the roadmap. That describes the process of how we are accomplishing competency based training.
There are narratives, but everyone is calling them different things. That’s the synthetic activity of taking assessments and reorganizing them back into the ACGME competency domain framework. That meets the anticipated needs of the ACGME and ABIM. Narratives are the bridge. Page 6 shows a definition of EPAs, criteria, etc.
Slide 18 is how he envisions EPAs. Each circle is a window. One window won’t show you competence. You need to look at them together to see competence in a resident. You have to: 1) define the EPA, what it means, and 2) define each window. What does the resident do to show competence? Carol and Bob have described widows as factors. 3) You have to link the windows back to milestones. If we define the EPA, we don’t need to define every step to get there. We need a shared mental model of the end goal. The next step, what EPAs do I need to have? What are the correct EPAs? They are defining a series of EPAs that a resident should be entrusted with at the end of training. Thirteen EPAs are in draft form. They are collecting feedback from stakeholders within internal medicine. They may seek feedback beyond internal medicine, too.
Tim asked Kelly to expand on the link between milestones and EPAs. Kelly provided the example of “Lead and work in interprofessional teams.” Ideally there would be several sentences describing that activity. Then you define each window, what a resident should demonstrate. This is program specific. The program director creates the windows then links that back to milestones. Thirteen milestones inform the assessment of this EPA. That allows them to report out domains of competence.
Tim asked at what point do we go from generic statements of competency to a more finite description of abilities, or performance levels? Kelly commented that they have not defined levels the way pediatrics has done.
Tim commented that for each milestone, like historical data gathering, it looks like a progression.
Kelly replied that there is a progression. When applying it to this example, ICS-A8, helps to inform working in team dynamics. They have not had to do 1-7 to get to this level.
Rosalyn commented that at a recent NBME meeting, a number of people from internal medicine programs were present. All were concerned about what they will do at the individual level. They were hoping this would be more crisply defined. It’s at odds with diminishing resources, and many don’t know how to do this.
Kelly commented that they started with 142 milestones. There was perpetual concern for reporting each and every one. This EPA has only 13 milestones. Carol Caraccio has described the EPA as the big ticket items. If we focus assessment on that, you don’t need to look at every detail. They are starting to build dissemination throught. Most faculty embrace the idea. If you ask faculty to assess can this resident work in an interprofessional team, faculty embrace. If the epa is leading a family meeting, different criteria go in. Hospitalist, oncology, and palliative care may focus on that. They will fully recognize that this necessitates a change in how we work. There is more observation, less proxy. That is where we need to go.
Rosalyn commented that as we look at how to design the standard, how can programs efficiently collect data to address these requirements, to minimize the work?
Kelly commented that he thinks of milestone as a source of data; epa is the data collection. The narrative is how they will report. We will need to have competency committees: groups of people that can work together to interpret data and report through a narrative descriptive document. The reporting mechanism should be stable.
Rosalyn asked would it be possible for faculty to use smart phones to mark what a resident has done?
Kelly replied there are many possibilities. We need to set expectations that there are one or two things to look at. He would like to move to a resident portfolio. He would like a resident to come to him and say, “no one has watched me run a family meeting. Would you observe and give me feedback?” That would be welcome to the faculty.
Tim asked if faculty would be rating the EPA globally and if one would infer progression from that. Or would they assess more granularly?
Kelly commented that it could evolve in many ways. There are ngoing studies on the level of granularity. You can take milestones and apply them as anchors on the mini cex and collect that data. It will take years of work to get there. As part of a multi-institutional project, they created a tool that incorporates 8 milestones to assess whether a resident has met an EPA. Global assessments will be there. But there may be discreet activities focusing on other things. If faculty can start assessing about interprofessional teams and family meetings as opposed to systems based practice, that is a great step forward.
Susan commented there are at least 2 ways to assess. For discrete milestones, either you meet it or not. Another is to say that there are levels for each milestone and that the learner is at some point along the way. She asked which characterization was accurate.
Kelly commented that it is more of a sampling strategy. You don’t have to assess everything all of the time. You collect a sample and make an assumption. To assess ICS A 8 (role model effective communication skills in challenging situations), you don’t have to have assessed 1-7. ICS-8 is more met/not met. They don’t have to achieve all 13 milestones as long as they accomplish the outcome of what the public entrusts physicians to do as practicing professionals. Milestones provide a context and shared understanding. If the resident struggles, faculty can say, here is what they are struggling with. Generic feedback is not helpful. You need to say here is what you are struggling with. Over time they may define which milestones are crucial.
Tim asked if the idea was to capture that a facilitator has seen the resident do this EPA and thinks they are meeting milestones x, y, and z but needs to work on A and B.
Kelly replied yes. They will collect data based on milestones through the lens of the EPA. Data is rooted in and based on the milestones. If you train faculty to look for competence through the EPA, don’t anticipate them to do every milestone all the time. The EPA provides the context.
Susan asked if some milestones are mandatory.
Kelly commented that was a realistic possibility. You can have faculty define what they want to look for. At the workshop he was asked when do you entrust a resident to discharge a patient from ambulatory without confirming the findings? What does the resident have to do? He gave them an abbreviated list of milestones and had them do a game to prioritize. The top milestones were synergistic.
Simon questioned whether we were talking about two different concepts – competence and assessment. He asked if there was a way to separate them logically. Sampling doesn’t directly affect how people conceive of the competence itself.
Kelly commented that an individual faculty member can’t assess readiness for independent practice; that is achieved over a long period of time. Individual milestones are assessing competence in a particular skill. Assessment is informed by milestones. From program director’s perspective, different EPAs inform competence. The same concepts apply. But the windows don’t look the same.
Rosalyn replied that is why we separated competency framework and assessment. Our current task is to look at assessment. We have to understand the competence to understand how it will be assessed.
Susan commented that most of the milestones are met/not met. She asked if there would be levels in others.
Bob commented that for an entrustment decision, you are looking at met/not met criteria. It’s very specific criteria. There are clearly milestones that are more than a single level in progression to an entrustment decision. In pediatrics, you may be at a novice level. The feedback around an EPA would be “you are at this level; you have 2 stages to get to a level of entrustment.”
Susan commented that some milestones are for novices, some for moderate, and some for experts.
Bob commented that if you look at ICS-6 (Engage patients/advocates in shared decision making for difficult, ambiguous, or controversial scenarios), the 5 before are a series from lower to higher stages of performance.
Kelly added that you don’t have to check off 1-5. They will have to assess novice stuff at earlier stages.
Bob added that if faculty make the decision not to entrust, then they have as a framework a series of milestones to go back to. You should be at ICS-7, but you are at a 4.
Josh described what NUS is rolling out for UME. They have adopted the pediatric framework, using levels 1-5. At 5, a student can teach; 4 is the level of entrustment. It’s not a go - no go qualification.
Tim thanked Kelly for his explanation. He added that if other questions come up, please use the listserv.
3 Review revised definitions
4 Discuss use cases
5 Open discussion