September 11, 2009
9 AM CDT/10 AM EDT/4 PM CEST
Attending: Kimberly Hoffman, Chair; Brownie Anderson, Carol Carraccio, Elaine Dannefer, Dennis Donohue, Bob Galbraith, Simon Grant, Linda Lewin, Neil Mehta, Amber Montañano, John Norcini
Staff: Jody Poet, Valerie Smothers
1 Review minutes of last meeting
Kim began with a review of the minutes from 8/28/09. Simon noted that he will not be here on the 18th and that was the only correction to the minutes.
2 Review Doodle poll
Kim emphasized the need for everyone to respond to the doodle schedule link to find a suitable time for these conference calls. Valerie sent out a link to the doodle poll prior to the call. Kim mentioned that the call is scheduled for September 18, but no calls have been scheduled after that. A new call schedule will be distributed in the near future.
3 Discuss survey results
Kim asked Valerie to discuss the survey results which she sent in a PDF version at the start of the call. The purpose of the survey was to identify what types of educational trajectory data schools are collecting and how they are using it. Of six respondents, all six collect data on courses, academic difficulties, and gaps in study. Five collect data on extracurricular learning, and three collect summary data. Data is stored in a variety of systems even within single institutions:
- Student affairs
- CV database
- Patient logs (for clinical rotations)
- SAP Student Life-cycle management (SLM)
- Blackboard Portfolio(s): narrative only
Respondents indicated a variety of uses for the data:
- Document graduation requirements and transcript
- Use to write MSPE (Medical Student Performance Evaluation)/dean's letter
- Track remediation
- Reports for regulatory bodies
- Advise students
- Promotions and progress committee
50% of respondents currently share the data with others, including:
- Residency programs
- Regulatory bodies
- Royal Colleges of medicine (professional associations)
- Department of health
- Postgraduate deaneries
- University registrar
- Promotions committee
Two of three respondents send data via paper or survey; only one is currently sending the data electronically. When asked if they send summary data, detailed data, or both, respondents answers were evenly distributed. Respondents indicated that recipient use data to:
- Evaluate student application
- Provide formative and summative assessment
- Regulatory purposes/quality assurance
- Create transcript
- Write MSPEs
Valerie thought it would be helpful to see how the data was used to see if it would shape our use cases. Kim mentioned she thought it would be worthwhile to establish the meaning of the extracurricular learning. Bob noted having a standardized set of data in a specific format would help to compare data between schools.
4 Discuss data analysis spreadsheet
On the last call Neil urged the group to catalog their experiences and provided a sample concentrating on elective courses. Valerie merged Neil's work with existing work done by the EII and developed a spreadsheet to gather information from participants. The group discussed the terminology used for classifying courses and recommended the term "Core requirement." Kim asked Brownie if the data was easily obtainable from Currmit. Brownie commented that Required courses would be downloadable, electives not so much, dependent on the school entering into the data base.
Simon ask what "collected" meant? Kim suggested maybe "tracked" was a better word. Neil and Kim suggested that a column for dates/length of rotation be added. Kim volunteered to send a note to medical schools not listed in Currmit to find out about their required data. Kim was worried about how much data we were asking medical schools to contribute. Brownie suggested seeing how much information we have and then see what extra reports would be required.
A discussion followed regarding extracurricular learning or non-curricular learning and how that fits into the portfolio. Linda commented that it is difficult to verify the time spent in a non-curricular activity. Simon commented that if you are clear about what is self-reported and what is verified by the institution, that helps the recipient of the data to use it appropriately. Valerie asked if non-curricular data was currently captured in student portfolios. Elaine commented that it was captured as self-reported data by the student. It is up to the student to organize and upload data, such as pictures from a trip to Peru. Students put together information for the person writing the MSPE letter.
Kim commented that the group could learn from AMCAS (American Medical College Application Service), which does a nice job of categorizing student activities. Carol mentioned that from her perspective as a residency program director, community service and advocacy are very important. Neil added that we need to know how data will be used. If the college sends the data, that adds a level of validity.
Simon asked for samples of what people submitted in their application. Carol mentioned her experience with pediatrics they are looking for initiative and advocacy. She recommended having a self-reported checkbox and an institution verified checkbox. Linda added that it should be clear who entered the information. Kim commented that we are looking at a more basic level - what buckets of information are important to consider?
Valerie noted that the main task is to create a vehicle that will be useful for residency program directors in making decisions. Kim asked if there was anything codified as to what program directors are looking for. Carol commented that they are looking for initiative and advocacy. It also makes a difference if a person's work was paid or volunteer.
Neil commented that theirs is a five year program. The ability to work together is important, as are writing skills. At the fellowship level, they are looking at how well they did in residency and did they publish.
Carol was concerned about how comprehensive these things need to be reported. She prefers highlights. Simon commented that hearing an account of the selection process would be helpful. Carol offered to describe her experiences.
Kim asked if Elaine and Neil would write a paragraph about the lessons learned from the first class who graduated with using the e-folio. They agreed. Neil asked if there was a place for a web form where the student uploads what they want to send, and there is no need for the college to validate the data. Kim agreed that was an idea for exploration.
5 Review scenario for inclusiveness
6 Open discussion
- Valerie and Kim will make modifications to spreadsheet.
- Carol will write a paragraph about the selection process and factors considered
- Neil and Elaine will write a paragraph about lessons learned using e-folio.