February 21, 2012
8 PST/9 MST/10CST/11 EST/16 GMT
Attending: Linda Lewin and Alan Schwartz, Co-Chairs; Dana Bostrom, Carol Carraccio, Kelly Caverzagie, Simon Grant, Patricia Hicks, Kimberly Hoffman, David Melamed, Howard Silverman, C. Scott Smith, Valerie Smothers, Janet Trial, Loreen Troy, Sandra Waters
1 Review minutes
Linda began the meeting with a review of the minutes. The minutes were accepted as submitted.
2 Discuss further medical student feedback (data pending)
Valerie mentioned there was no further feedback. Linda asked if the group was finished with that phase and Valerie answered yes. Valerie told the student contacts she would be in touch with them again when a specification was ready for their review. The group can continue to ask for their feedback in the standards development process.
3 Review updates to data analysis
Valerie updated the group on the data analysis changes based on the last call. The changes are marked with a red asterisk. The first slide has several changes. The critiques mentioned it wasn’t showing how data could be used for different purposes. This slide was intended to be a landing page that better illustrates the different uses of the data.
Linda asked if the colors were important. Valerie replied that blue indicates a link. Linda commented it seems like there are two sections, competencies and events, and then actions all in one list and in different formats. She was not sure how you would delineate that content. Linda offered to work with Valerie on clarifying the first slide. If anyone has a better taxonomies or better terms for transcripts let Valerie know.
Linda commented that the slide set is something we can take to show someone else, we want it to be clear. David asked for clarification on what event based is referencing. Valerie answered it is referencing the results of specific tests and course grades. Valerie continued with a discussion of other changes. She noted the comments reflected the importance of identifying the source of all data. In slide two wherever possible the source is indicated and dates are associated with competency data. The next change had to do with having private comments and the need to be able to mark something private, which may be a function of the application. Private notes have been added as an example. There was a prior discussion on the ability to see longitudinal assessment data for a particular sub-competency; that has been added. Slide nine now includes a link to information about the assessment methodology. Patty asked whether we anticipate a common list of methodologies in the specification. Valerie thought that was a good question that would be addressed as we go through the specification development process. She suggested group members post any lists they use to the wiki and we can consider them when we get to that point.
Valerie continued with changes to slide fourteen. the context is now shown as well as the date. On slide twenty-five where the USMLE scores are shown, the two digit scores have been removed and just the three digit scores remain. The same is true of slide twenty-six. Slide twenty-nine provides a link to undergraduate portfolio & transcript to indicate we can link out to that and don’t need to describe it in detail in this specification. Linda asked the group if there was anything glaringly missing that anybody wants to note.
David mentioned slide one and asked if there was a reason why event-based activities were indented to the left; he sees them as part of competencies. Valerie commented the event based data shown has no ties to competencies; USMLE hasn’t characterized the specific tests associated with competency. The group decided that data can only be shown as related to a competency if the assessing organization makes that connection explicit. In addition to seeing data in a competency context, we need the ability to see the data independent of the competency framework. Linda shared the same data may need to be linked to more than one place. What we have now should allow that.
Valerie mentioned if we identify something we want to change we can continue to change the slide set. It may be useful for presenting our work at meetings and we can certainly keep it up to date.
Lori began with credit to Rachel Yudkowsky, who shared these reports borrowed from the clinical performance center. These reports are examples of the way data is presented both to the learner and the curriculum committees on how the class is doing as a whole. One was a report concerning resident assessment. They started down this path through procedural competencies for graduate medical students and worked with the clinical performance center to simplify the evaluation. It was determined the best students gain competency with real live patients at performance centers where simulations are available. They found that students just reporting procedures with supervision had difficulty practicing these skills. Sometimes there were other medical students and new residents that found when students logged procedures on their own; they depended on the supervisor to approve the procedure. The faculty would review and approve student activity. Using student center simplified that quite a bit. The student can repeat procedures until the supervisor says they have done it correctly and verified they have completed the task. It is part of the M4 exam. Lori asked the group if anybody had questions or comments.
Valerie asked if the UIC report was pass/fail. Lori answered yes; they have to complete all protocols properly. The rating list for students indicates if they do or do not do critical points. When they do not, they have to correct any deficiency in their performance. The initial assessment is based on what they do the first time through the procedure. Valerie commented it’s more for student feedback to correct what they’ve done wrong. Howard asked what percent of the class got it right, or does it represent what percent of the steps everybody got right? Would the score be less than 100? Lori said she would ask Rachel and get back to the group. Valerie was wondering about the current way we represent skills. Is it enough to say that the learner passed the skill and they possess this skill? Lori explained the first pass would say the student had satisfied the procedural competency. If someone wanted more information they could drill deeper to find the evidence. Here is the mean class performance or where this student is. Linda commented on slide fourteen some of that is covered under additional information. Slide fifteen implies there is the ability to input different data. We could include multiple points of assessment.
Linda mentioned it is at the discretion of the institution as to what they want to display. Howard said students may want to see that level of detail; it may or may not need to be added to their permanent record. Valerie asked if we assume there is a local system to get some of the formative data, would you want that data to be included in a national centralized system? Howard thought it would be hard to know what that means. Valerie mentioned as long as we have options available to people using it that would be sufficient and Linda agreed. Howard was thinking about how one would roll this up nationally. If the learner attempted three attempts across every procedure, that is potentially damaging information. Schools should be able to have flexibility in differentiating what is useful. Patty commented progression through roles may be useful in reporting.
Linda asked if we need more clarification about this. Valerie asked what is being including in assessment of procedures; we may want to have a link to methodology or description when describing skills. Linda agrees and people can use it if they want to. Valerie commented as far as Patty’s comment on progression of roles that would probably need to be another standard. We talked about clinical logs and how encounters should be incorporated into this specification. They are very complex, as are the roles and we may need to sleight that for another project because it is so complex. Simon agreed with Valerie. He that the emphasis on visual representation of tools is confusing. You can put lots of things in the standard, but it works much better if they are simpler and limited in scope, and useable.
Patty had a comment regarding slide thirteen, representation of entrustment, the details of which would be critical. Entrustment might mean different things to different people. Valerie suggested adding description and methodology fields that will address that without the full complexity of a log. Adding description of skills and methodology will say more what learners are going through without too much detail. If there is an interest in providing data with multiple points of assessment, there is an opportunity to do that too.
5 Discuss next steps
Valerie asked if the group was ready to write the specification. Alan, Linda and Valerie talked earlier in the week about doing a standards research document that takes a look at our requirements and what some of the existing standards and specifications are out there to determine if there are components we can leverage that would cover part of our needs. The next step is to do the standards research and then get on with writing the specification. Bob mentioned one of the issues of being thorough is trying to get the spec as clean as possible. He asked Valerie to talk to the group about the mechanism of changing the specification if it’s not perfect. Valerie continued with a discussion on the ANSI process for creating standards. After the specification is reviewed and approved by the working group it goes to the standards committee then for public review. The standards committee ballots the proposed standard, and once it’s published ANSI allows us a required five years to revise or reaffirm the standard. We will begin work on Version two not too long after we submit version one. The nice thing about that is you can obtain information from version one and clear up any points of confusion. Valerie will take on the action item of working on a standards research project for the next call.
The group agreed to close the data analysis portion of the project. We will begin research of existing standards and development of a new or integrating standard on the next call.
- Valerie and Linda will revise slide 1
- Group members will post any lists of assessment methodologies they use
- Include links to methodology or description when describing skills and entrusted activities
- Valerie will begin research on existing standards addressing our requirements.