December 7, 2012
9 PST/10 MST/11 CST/12 EST/17 GMT/18 CET
Attendees: Linda Lewin and Alan Schwartz, Co-Chairs: Carol Carraccio, David Melamed, Howard Silverman, Scott Smith, Valerie Smothers, and Jan Trial.
1 Review minutes of last meeting
The minutes were approved as submitted.
2 Review updates to specification
Larry Hurtubise, on the last call, suggested changing the definition of entrusted. Valerie added to the definition of entrustment “without direct supervision.” She added that may need to be revised based on comments from Maastricht. The definition of event was also changed to reflect instructional methods developed in the Curriculum Inventory group. On page 16 the definition of statement of awarded responsibility was deleted. On page 25, the definition of events was updated. PESC updates are referenced in the specification, so the section saying that change is pending has been removed. On page 35, the definition of Norm Reference Values now includes a reference to PESC Educational Test Score reporting version 1.7.0, which has updates to accommodate reporting of score distribution histograms. The PESC schema has been updated to include those changes, but the human readable document has not been updated. Valerie suggested coming back to the entrustment decision later and Alan agreed.
Valerie began the discussion and commented some of the points Maastricht brings up are simple and easy to clarify. One of the questions to discuss regards AchievementInContext. They ask if that would be used to represent different internships at different hospitals within the same program. Would that be a separate achievement in context?
Linda asked Valerie to explain further. Valerie explained we have an organizing principal in the specification called Achievement In Context that describes the learner’s achievements within a specific context. For example, the University of Maryland pediatric residency program is a context, and AchievementInContext could describe a learner’s achievements within that residency program. Maastricht asked should AchievementInContext represent the curriculum of the learner (the entire residency program) or should it represent different parts of the same program, like an internship at a specific hospital. So if resident starts education at academic hospital that has internships at other hospitals, do we want to break apart the data about achievements at each hospital or keep it together?
Linda commented that if someone is in a residency program, they spend time elsewhere but get credit for it in the program; it’s still the same program. Howard thinks they made clinical learning part of the larger program. Carol commented she thinks being able to define context is going to be very important. The University of Maryland could do inpatient ward experience with an affiliate; they are very different experiences and levels of skills. For us to be able to say where a rotation takes place as well as where is it, whether inpatient or outpatient, is important. Assessment has to be context based. You can’t transfer skills from one context to another.
Linda asked if we had another place to define. Valerie replied we do not have that data anywhere else. Valerie asked if it is related to sequence blocks, ie courses. Would we put a field in sequence blocks and indicate location? Linda agreed that made sense. Carol commented it has to be tied to assessment. She suggested adding site to the event element. Valerie suggested have it at event and sequence block, macro level or micro level. She added those changes would not get to the entrustment that Carol brought up. Linda commented maybe you could be entrusted to attend delivery at the University hospital or the community Hospital. Carol mentioned that both ad hoc entrustment and more global entrustment would be useful. Carol asked Valerie if ad hoc entrustment is documented. How do we deal with ad hoc entrustment? Valerie suggested adding site to event and Linda thought site would be useful. Howard agreed, they have specific courses for specific sites. Sites matter in terms of experience and assessment.
DECISION: Add site to Event and SequenceBlock elements.
Valerie addressed the comment about program. Maastricht mentions that in cases of super specialization there may be more than one program. Currently the program element is optional and may only occur once. We could change that and make it so program can be there multiple times. Howard commented if this is going to be a broad standard it will have to apply to many programs. Carol asked for clarification, are they saying you might be in more than one program at the same time. Howard stated fellowships would be sequential not simultaneous, only dual degree programs would eb simultaneous. Carol asked how about combined residency programs. Linda commented it would be med/peds residency, you wouldn’t want to separate parts of it. That is how they are accredited. Howard said med/peds is an accredited combined program. Other combined programs are separate, like IM and emergency medicine, and they have two separate accreditation processes. Valerie will ask for specific examples and details and ask whether combining programs is sufficient.
Valerie continued with the next set of comments under events. She explained that right now events in the curriculum are part of the learner’s experience. We require event duration, and Maastricht doesn’t use that. Should we make that optional? Linda said yes, if we’re moving away from time-based curriculum. The group agreed.
DECISION: Make Event Duration optional.
She continued with the competency object reference; they say this works fine. She asked whether the group wanted to expand to allow the achievement record to just point to a competency on the Internet instead of including all the information about that competency in the XML. Valerie asked how people feel about using a reference to point to things on the internet. Howard stated those links could change over time, and he asked if they would be maintained at that location? Valerie commented within implementation guidelines for the competency framework, it recommends that Retired Competency Framework documents should remain published in an accessible location for a minimum of 5 years. We can’t enforce that, but people will see benefit in maintaining published frameworks. Linda thought it made sense to do that, unless you thinks it‘s too big of a gamble. Howard commented the benefits outweigh the risk.
DECISION: Allow references to competencies published on the Internet
Carol mentioned the comments on EPA’s and entrustment awarded on an ad hoc basis. There may be cases where the supervisor comes in the next morning and learns what happened and signs off on something ad hoc. This would be different than a structural entrustment decision. Howard asked who the evaluator is, and who is assessing the student. There is often less consistent observation over time. Carol mentioned there is different weighting to it, and that distinction needs to be made. Valerie commented that we could put ad hoc entrustment data under specific events or include it in the summary section and say this was an ad hoc decision. Carol wanted clarification, by including it in summary data, is there still opportunity for somebody that achieves the professional activity to award them a star at that point, so they can collect their merit badges along the way. Valerie answered you can have multiple entrustments for the same activity, you could have ad hoc for performing a newborn exam, and you could have more formal entrustment that is not ad hoc for the same EPA, but the document source would be different. A man asked if there were two different performance frameworks. Valerie thought that was a good question and Linda seemed to think that it was the same framework but different evidence? Carol agreed that made sense. Valerie asked if she should add an ad hoc flag. Linda, Carol and Howard agreed.
DECISION: Add an Ad Hoc flag to the Entrustment element.
David commented he was not clear on who could trigger that or how it would be triggered. In a clinical setting a faculty person entrusts a resident to perform something, how will it actually be documented? Howard mentioned in many cases procedure logs list details of procedure and who supervised them. As a program director, I have to certify competency at a higher level, and that is done all the time. Almost all of those observations are ad hoc.
Valerie mentioned the comment from Maastricht different levels of entrustment they have. We could accommodate that by adding a performance framework level reference to entrustment. We don’t want to code their language; somebody might come up with something different in the future. Alan suggested leaving option open to have different levels, but not specifying what they have to be. Jan agreed.
DECISION: Add support for levels of entrustment.
Valerie commented she doesn’t understand comments about score on page two of Maastricht’s comments on assessment results. They comment they have two types of scores, competency domain scores and overall performance scores. One is not a sub score of another. She was unsure if we needed to change anything in our specification. Howard commented it was hard to understand without a specific example. Valerie did receive an example of a narrative score, which was circulated to the group (form a/b for assessing residents). It is for annual assessment of residents and provides free text fields for assessing different competencies. There are also yes/no questions regarding the resident’s portfolio, patient care, and courses. She is not sure how to implement it.
Patty commented that the portfolio text fields may fall short in being used in assessment. Linda asked what it adds that we are missing. Valerie commented the notes section under assessment results could capture those comments. Linda commented she can’t envision a system that protects score data. She asked if that interpretation needs to be documented in the specification. Does the synthesis function need to be captured or does it happen outside of the specification? Valerie answered they would want to capture it. Howard agreed they want some sort of qualitative to subjective part. Valerie commented we have a notes section for assessment results, she could add something that says faculty notes, or qualitative assessment. Howard liked the notes term better. Valerie will add optional faculty note. Should we change plan vanilla notes to learner notes? Howard said that would be helpful.
DECISION: Add Faculty Notes to Assessment Results; change Notes to Learner Notes.
Valerie offered to see if she can do the rest by email since we didn’t get to themes. Valerie will make changes and find out more information on our open questions.
4 Open discussion
Add site element to Sequence Block and Event
Email Maastricht asking for more details on situation where there are multiple programs.
Make event duration optional
Add the capability to link to a competency framework published on the Internet and point to that from CompetencyObjectReference.
Add Ad Hoc flag to entrustment element. “for the special purpose or end presently under consideration” doesn’t seem sufficient.
Add levels to entrustment
Add Faculty Notes element to assessment results; change notes to Learner Notes.