Sep 13, 2011
8 PDT/9 MDT/10 CDT/11 EDT/16 BST
Attendees: Linda Lewin and Alan Schwartz, Co-Chairs; Susan Albright, Dana Bostrom, Rachel Brown, Robert Englander, Maureen Garrity, Simon Grant, Patricia Hicks, William Iobst, Howard Silverman, Kirk Smith, Scott Smith, Valerie Smothers, Andria Thomas, and Lori Troy.
1. Review minutes of last meeting
The minutes were approved as submitted.
2. Review champion volunteers
Valerie continued and thanked the users who volunteered for the list and asked if there was anyone who volunteered who didn’t see their name listed, to let her know and she will add them right now. She continued with an update on communicating with the medical student organizations. She has emailed all of them and spoken with AMSA, AMA Medical Student Section, and the SNMA; all three are ready to participate. She has not heard from OSR. We will connect when we want their input and have something to share: finalizing data analysis and the specification. We’ll write up a summary to show our work to date and present it to them for feedback.
3. Review of educational achievement data in mspe (if Maureen available)
Alan asked Lori to talk about a couple of Medical Student Performance Evaluations (MSPE’s) she sent around. Valerie commented that Maureen had been out of the country and would not be able to give a full update as to where MSPE is headed but if anything stands out in her mind, to jump in and comment. Lori continued and mentioned as she looked at the sample this morning, she was not sure that it was as useful as she imagined because there is a lot of information arranged more as a narrative, rather than a record of educational achievement. She also looked at their academic educational profile and didn’t submit that sample because that has all individual grades that students have earned since completing course work. She thought that was excessive. If the student achieved honors, that was worth noting but otherwise, they wondered about providing too much information. Each school is using different metrics.
Maureen commented it was helpful to have the AAMC document Guidelines for Preparation of MSPE. The documents Lori provided are good examples of what is recommended. The majority of documents in the country would look similar to this. The task force has been working on the MSPE and would love to see how the program directors are using the MSPE.
Scott shared he was in the middle of sorting through 800 of these and he said they have to be efficient. Like the AAMC guidelines, most do look similar. They have to view the important information fairly fast in a standardized way. He added it is difficult to judge when schools don’t use any grades. At Chicago they use a capstone course report that indicates they can do a catheter, etc. He would like an outcome or competency component.
Linda commented there is no consensus; she noted there is no list of procedures every student needs to learn. Maureen worked on the MSPE task force and said it was frustrating to have 130 US schools submitting, close to 10,000 program directors reading them, not to get more consensus information on what they want to see. She hopes the MSPE becomes obsolete because of our work. Senior associate deans spend 500-600 hours working on them, but their use is very limited. Rachel agreed. She added that MSPE has the potential to give detail about the individual and their achievements, and she does not want to lose that. She is in favor of a more efficient process that maintains special characteristics of the way we do things now. Scott agreed those things are useful to find people who struggle. Narratives provide hints about struggling students, but sometimes struggles are not captured.
Maureen that the writer of the letter is under pressure to make sure their students match; they risk disclosing negative information. There has to be a way to identify students at risk and working with them up front instead of waiting 9 months until the student is in trouble.
Alan asked Valerie what the next steps would be. Valerie asked the group if there are specific parts of the letter that anyone would want to call out, discuss separately in more detail when Maureen provides an update when the task force meets. Linda participated in the Educational Trajectory development that this group did previously. That defines when the learner starts and stops their primary degree and what else is going on during that period of time, including enrichment activities that could capture experience as opposed to metrics. Are there other parts that are important?
Scott shared he look at the MSPE for unique characteristics. He reads the summary and a histogram describing academic performance of the class and where this student falls in that histogram. If there are honors he wants to know how special was the honors at this place. He would also want to ability to drill down and comments related to specific rotations.
Linda agreed she does not read the details. She would like to know if there were any interruptions in the person’s medical school, their basic grade, what quartile of class they are in, the grades for the 4th year structured clinical exam, and to know that this person can graduate. She loves the idea of drilling down.
Bob Englander commented the organizational framework of MSPE is predominantly rotational. Having something that speaks to competencies across the rotations is another opportunity. Selection committees are frustrated by the work involved to figure out deficiencies in the current model.
Scott commented that having both sets of data would be helpful. There is huge redundancy in the department letters MSPE and a lot of people are spending a lot of effort that doesn’t get us a lot of mileage. Bill agreed with Bob; graduate medical education has implemented a competency based framework using six general competencies. The joint commission is going with 6 competency and competency based assessments are becoming more prevalent across professional careers. Competencies are becoming the coin of the realm.
Linda mentioned the group might consider the framework we used in reporting on educational achievement be put into a competency based format. Valerie summarized the group’s comments regarding an educational achievement data format:
- Needs to have capability to support competency based framework
- Needs to have capability to support rotation or course based format as well
- Needs to have a descriptive narrative summary and unique capability section
- Histogram capability to see where the learner fell in relation to their classmates
- The ability to drill down to specific comments
4. Other examples of educational achievement data
Alan mentioned there were several things posted on the wiki: one article, some transcripts from Simon, a performance evaluation from Bob Englander, and a description of different types of achievement data from Linda.
Simon questioned how our work would integrate with the UK work he cited. The UK Higher Education Achievement report is organized by what the person has to do to make this achievement. Linda asked if MedBiquitous has transcript capabilities and do we want to provide information like that. Valerie commented that there were existing transcript specifications and that we don’t want to recreate what others have done. If there are parts of other standards we can use, that is ok. We are trying not to recreate transcript, but we can try to integrate it.
Simon commented he can’t see a hard and fast dividing line between transcripts and MSPE’s. Valerie shared maybe in the UK there is not a hard and fast dividing line; however in the US they are very different. A transcript is an official record of courses and grades, whereas MSPE contains more than that. Linda commented the MSPE are more integrated; a transcript doesn’t tell you how anybody else did in the course. We want to make sure they can do the job not just what grade they got.
Simon mentioned the vocational qualifications. Scott shared they are getting pressure to have vocational qualifications that demonstrate minimal competency. Simon shared the letters on the site, described the general skills that are required but you can’t tell whether the individual acquired them or not. His concern was the challenge of how these things integrate with each other.
Linda commented that there is no consensus on the use of competencies in medical school. She had previously worked on the Doctor Development Project, which created a framework for medical school through practice. She added that most medical schools are proud of being different and don’t want to adopt one thing and look the same.
Bob commented that there is increasing momentum around the 6 core competencies due to the maintenance of certification process. Maintenance of licensure is trying to use the same language. Bill commented that the ABIM is transitioning their reporting mechanism to competency based and ABMS is investigating the same approach. Linda asked what happens to the other kinds of education information that was submitted. Bob pointed to Figure 3 and 4 in the Pediatric Milestones Assessment article. Linda commented that all the achievement types she documented could be linked to competencies and used as evidence.
Bob described the Division of Hospital Medicine Profile. The document was used for ongoing professional performance to document competency on an every six month basis. It shows practitioner specific data and where it came from.
Alan asked Valerie to summarize the next steps. Valerie suggested going through the data submitted ad examples and start developing a synthesis as to how this all fits together and what that means for us. There is a lot of commonality across institutions and also important differences we may need to capture. Alan agreed with that plan and suggested looking at the boundaries between achievement and the evidence for that achievement which might include various kinds of assessments and how we describe that. Alan mentioned once we come to settlement about that, the next steps are applications of use cases to see what would need to be included in the specification.
5. Open discussion
Scott commented that in internal medicine and pediatrics they are looking at the competencies, what were the milestones, and how do we make judgments about competence. They want to know when a learner can see a patient in clinic without having to be reviewed, when they have skills to run the clinic. Those elements need to be tied to competencies and milestones. If you have six to eight observations over time, some knowledge based, you have good defensible opinion of the learner’s overall competency.
Valerie will work on drafting a synthesis of example data submitted and the group’s discussion for the next call.