June 19, 2012
8 PDT/9 MDT/10 CDT/11 EDT/16 BST
Attending: Linda Lewin and Alan Schwartz, Co-Chairs; Susan Albright, Carol Carraccio, Erick Emde, Patricia Hicks, PJ Kania, David Melamed, Scott Smith, Valerie Smothers, Janet Trial, Lori Troy.
1 Review minutes of last meeting
The minutes were approved as submitted.
Valerie described changes to the specification document. On page fourteen, sources were added for definitions in the terminology section. There are references for every term except for performance ones. The definitions come from a page on the competency working group wiki, but we may want to wait until we have something more final to reference there. On page fifty-six, references to competency framework and educational trajectory were added; we will add more as we go along.
On page thirty-nine the summary scores summary element was changed to just summary and modified to include entrustment data. Data about the learner’s performance in a particular content, (i.e., throughout residency, medical school, clinical practice, etc.), would be available in this section. The model for entrustment information follows. So when you capture information about entrustment there is a title, description which is optional and there can be a reference to a competency object. The competency references have a way of pointing to EPA. The model also includes the award date and the date the entrustment decision was made, and an optional expiration was date added. There is also entrustment evidence, which describes or points to evidence that was used in making entrustment decisions. You can point to specific portfolio items or things in the educational achievement record.
Alan commented the source of the entrustment decision is missing. Valerie asked the group if everyone agrees with identifying the source. Lori asked if it would be an optional field. Linda agreed it was good idea to have it as an optional field. Carol commented that in Ten Cate’s original work, he had two people to make a decision; she agreed it was good to have the capability to enter several names (one or two people or a committee of four. Carol mentioned the description of entrustment really means being able to entrust the learner to perform certain activities without direct supervision and she suggested that be added to the description. Scott agreed.
Valerie commented that the summary speaks to performance in a specific context. In the John Doe example, the first context is the MD program at University of Toronto ; the second is the University of Maryland Pediatrics Residency Program. Entrustment data would live within one of those contexts, is that sufficient?
Patty said that the challenge is people will have to look at performance in a given context; then they’ll have to make their own predictions about a different context. If you are looking for someone who graduated from residency program that can do activity a, b, and c in one setting, entrustment may give you false reassurance that the person is ready to perform those activities in another context. It is hard to transfer entrustment from one setting to another, is that implied? Valerie shared that we are making context explicit, but we can’t dictate the assumptions people will make.
Susan commented that medical students may never get to entrustment, but they may meet milestones. Will the summary have milestone data? Scott commented he hopes medical students have identified entrustment doing simple things like blood drawing, just so that he knows this set of skills has already been entrusted and achieved. Valerie clarified Susan’s question by stating in activities where they aren’t given entrustment, but they have reached certain milestones, how would that data be captured. Lori mentioned LCME standards stipulate medical students must be supervised. Many technical skills and procedures that are required upon graduation perhaps are not under the summary here.
Valerie commented that at the event level, we capture that they are reaching particular level of performance within the context of an event. Linda commented the idea of entrustment is vague; when ten Cate talks about it, entrustable activities are not skills. The definition would include big things like physical exam on a newborn baby. Carol mentioned it is supposed to be big integrated skills. At one of their meetings, ten Cate presented smaller skill sets and the concept of nested EPA’s. Within big ticket items, there are smaller activities you might expect of trainees, and he calls them nested EPA’s. Is there something we could think about as an entrusted student activity, such as taking history from a new mom, that would that be an entrustable student activity? Some of the work Kelly has done in Internal Medicine EPAS mirror what they have done in Pediatrics; they could come up with some entrustable activities that cross disciplinary lines.
Susan mentioned a presentation Kelly gave in the competencies working group. He showed a cluster of milestones to achieve an EPA. Scott acknowledged confusion on what to call them. Linda commented that a student may not be entrusted to put in an IV, but they may be checked off to take history on children. The specification needs to be able to do it both ways. David commented that the terminology needs to be flexible. Family medicine hasn’t absorbed the EPA’s terminology yet. In pediatrics, they use sub-competencies instead of EPAs, nested milestone are discussed in no different manner than sub-competencies. Alan disagreed, commenting that the terminology needs to be less flexible. EPAs’ are an entrustment issue; not an assessment, but a decision. No one would entrust medical students without supervision, so we shouldn’t expect EPA’s for medical students. We should be expecting achievement of milestones related to competencies.
Carol commented the proposed language is entrustment without direct supervision. A medical student entrustment could imply that the medical student can do a medical history but doesn’t need direct supervision for that. It is important to think of EPA’s and milestones as different perspective. Milestones are related to demonstrating a certain behavior; EPA’s are the 30,000 foot view, can they put together milestones to perform an activity. It’s not an either or; both are important. Some people can check all boxes but can’t perform a meaningful professional activity. Linda asked Valerie if she had a sense of what to do to the specification. Valerie clarified there may be cases we have entrustment for medical students and others where we don’t. She asked the group whether they thought we needed to have something about milestones at the summary level, noting distinction between milestones and EPA’s. Linda commented that milestones would be related to the summary of competency data. Valerie noted in that case you would have that data within summary score of competency. If you look at summary score, sub-scores, you have the ability to provide a reference to competency and provide a score within the performance framework. Here is where the learner is. That is where the milestones come in.
Valerie mentioned a lot of the terms come from Tim Willett, Bob Englander and Kelly Caverzagie. She added that we don’t want to adopt the exact language used in medicine because MedBiquitous’ scope is broader than just medicine; we are trying to be generic but map well with what is happening in medicine. Susan wants to consider milestones as assessments where the learner is in relation to competency that would be captured at the summary level. She asked if that model is the same in Internal Medicine. Scott thought that would integrate fairly well in Internal Medicine. Some of the discussion has been helpful; milestones are arranged in Internal Medicine and give a developmental idea as to where you should be in your training, whereas EPAs are professional judgments based on competencies and observable behaviors. They are based on a more global judgment. There are two choices to become a group that standardizes terminology or leave it understood that terminology is evolving and this is what we mean by all these things.
3 Discuss peer driven portfolio review
This subject came from a question Carol asked about peer driven portfolio review for physicians so they may be assessed on things they really care about, and assessed by their peers. Carol asked if the Educational Achievement spec could enable this; Valerie’s response is in some cases yes, but not all cases. She talked about the need for clinical logs, but noted it was out of scope. The next question is can we track other kinds of work, like educational program development. The specification is designed for end results and not individual judgments. It could communicate judgements rolled up into a single score. Valerie continued with an example of residency program director implementing milestones. She wants to be assessed on her work developing a new assessment program at her institution and have her peers judge her on that. Slide five shows what that would look like, one of the events could be a residency reform project that would have assessment methods and the event would have to map to milestones. The work Kelly presented to the Competencies working group showed EPA’s linking to milestones rather than competencies. Valerie noted right now we don’t have milestones related to assessment judgments; she would have to make a change to accommodate that. The reference to portfolio or external document is important, but it’s currently not in the spec. You want to be able to point to that as supporting evidence. Using milestones as framework for scoring her, (novice to expert) you would be able to say what the score is and have a score like 4.2 based on judgment of peers. The data would point to the assessment scale that was the basis of that score.
Valerie commented that learner notes are supported in the specification; assessor notes are not. Valerie noted changing summary scores and using assessment results would enable us to point to portfolios or external documents. Valerie asked if it would be valuable to include assessor notes in the specification as well.
Linda questioned whether we we want that level of comments there. You want the comments for yourself to exist elsewhere. If you were using the data for maintenance of certification, would you want the board to see the comments as well as the score? Carol agreed. It seems like formative feedback doesn’t have a place in this document. Scott commented when you publish the final document, you don’t credit formative feedback, if it gets too public it won’t be useful.
Valerie asked Scott if the event needs to link to milestones in addition to competencies to accommodate uses in internal medicine. Carol stated a lot of boards are linking MOC’s to competency domains. They are not thinking about linking to milestones but it could be a vision for the future. Milestones are too new for people to have thought that far, but it would be a good place holder. Valerie agreed with building in the capacity to link to milestones would be good for the future, and if we’re not sure we can go back to Kelly and ask him that specifically.
Carol questioned the linkage to entrustable professional activities. There may be instances where the expectation is for the graduate to have reached advanced beginner, but they are not necessary entrusted. The learner will continue and finally build that skill set in a practice setting. Ultimately as part of MOC they would engage in individual learning activities that would take them from beginner to savvy with practice management would now be entrusted. Valerie commented right now one of the principles we are operating under is that you would be able to point to EPA’s the same way as you point to competencies. We still have to see if that holds up, but using that model, you could indicate the learner’s performance for that EPA without indicating entrustment.
Patty had a question regarding the time sensitive nature of the normative score data. She asked if in the normative group, the data reported reflected status of that group in that point in time. Valerie clarified there is a date associated with the assessment, the normative should be as of that single point in time. Valerie asked if it was sufficient to assume normative data is as of that point of time. Patty agreed it was.
Valerie commented when she applied data to the specification she noticed there is not capability to reference portfolios or learner notes at the summary level. She suggested changing the specification to address that.
4 Open discussion
- Add an element identifying the source of an entrustment decision. It should be repeatable so that multiple people can be referenced.
- “without direct supervision” will be added to the description of entrustment.
- Change summary score to assessment results (to enable portfolio references and learner notes)
Valerie will discuss linking events to milestones with Kelly.