February 12, 2010
8 PST/11 EST/16 GMT
Please note: the conferencing service will ask you to enter the pound sign. Press # for pound.
Attending: Kim Hoffman, chair; Michael Barbouche, Carol Caraccio, Maria Esquela, Bob Galbraith, Kevin Souza, Gwen Garrison, Simon Grant, Amber Montañano, Pat O'Sullivan, Morgan Passiment, Valerie Smothers
1 Review minutes of last meeting
The minutes were approved.
2 Small group reports
Kevin explained that the technical group worked on the realities of pulling data together. UCSF will take the specification, pull data together, and describe the reality of that situation. They have almost all data in electronic resources, including some competency data. The data is spread across four different databases. He estimates 4-5 people will be necessary to provide the data. Data is housed in the registrar's system, in the medical school owned student information system (including extracurricular activity data and some self reported data), and the curriculum management tool if they are to provide information on coursework. One of 4 individuals will go through the process of creating an xml file from the data so they can describe what that is like and then share the information with others.
The group also discussed looking at commercial products to see what tools they may have to help export the data, and engaging commercial groups like Advanced Informatics to discuss what would it take to build export tools.
Kim asked if he could monitor time it takes individuals to obtain the data and get it into the appropriate format. Kevin agreed.
Valerie summarized that the group recommended providing a title and link to a paragraph describing coursework leading to MD or primary degree. The paragraph would provide a summary of the program in very general terms. So there would be a paragraph describing the regular curriculum and a different paragraph describing the regular curriculum plus pathways to discovery, without getting into the details of the learner's specific project. The same paragraph could be used for several students.
Carol commented that a link to that information sounded good. Pat replied it sounded OK. She asked if there would be a field where the learner can supply information. Valerie replied that it would be easy to add a link to more information.
Kevin added that the technical group has talked about how the spec can carry an attachment or link to legacy work from a scholarly pathway. If the learner produced a document, whether published in the literature or in a portfolio, you could provide a link, or include a document in the package.
Valerie added that the small group recommended omitting student publications from the trajectory for the time being. It would still be possible to provide links to publications that were the outcome of a particular activity.
Pat commented that they try to keep up with publications, and it is a challenge. There is often a lack of synchronization between the activity and the publication, which comes out months later. The ability to link to publications is relevant to the medical school experience.
Kim asked Valerie to provide more insight on the group's recommendation. Valerie commented that one complicating factor cited was first vs. fourth author and the difficulty of distinguishing between posters and papers. Simon added that a concern was how you would represent it visually. If a red dot is a publication, what consistiutes a dot and what doesn't? He added that a list of publications seems to be a more natural format.
Pat added that publications are important when looking at productivity in a sustained way. The first vs fourth author distinction seems inappropriate in today's collaborative research environment. Kim asked Pat what that might look like. Pat proposed dots that glow or expand based on contributions. Maria commented that it seems the group wants to track information and let others interpret the information.
Valerie commented that publications could be linked from the entry for the program.
Carol commented that she liked the idea of an asterisk that one could click on to lead you to the output of a particular activity.
Kevin added that he was puzzled by the recommendation not to include publications. Schools have the ability to vet information that is presented. There could be some sort of attribute describing the intensity of the experience or role. Pat added that journals require people to describe the role that they had.
Simon commented that his recollection was that it was more in terms of how the information was presented. If there are a lot of dots on a bar for less intense work, that may look better than fewer dots which actually required a larger role. There is no reason why you cannot look for further details and have that alongside a list of publications.
Kim recommended taking that feedback to the extracurricular small group: the larger group proposes links alongside the paragraph describing the program that point to outputs in addition to a list of publications. She recommended getting the small group's thoughts on that feedback.
Valerie summarized the group's proposal regarding verification of enrichment activities:
- Requiring a reference contact that could verify the learner's involvement and role.
- Allowing an optional link to evidence of the enrichment activity.
Pat commented that she would not support requiring a reference. Contact information can be quickly outdated. The attestation provided by the learner and the resulting product is more robust. Kim agreed adding that schools could crumble under the weight of trying to verify learner involvement. Carol added that program directors would not take the time to follow-up with contacts.
Gwen commented that the AAMC undertook the study to look at publications submitted during the ERAS process. In 2001 applicants submitted 42,000 publications. In 2009 applicants submitted 169,000 publications. Trying to manage that amount of data is a huge task. There is no good software for an automated check. The AAMC performed a complex random sampling and confirmed 46% of the publications listed second. They could not confirm the rest. Checks on the sixth and 11th listed publications eroded further. Some things were misidentified, some authors listed themselves as second author when they were third. The biggest challenge was the number of people that said the publication was pending. Most publications do not comment on the status of the manuscripts until it has been published. They recommended that if you wanted information, the applicant should submit the PDF from the Journal, a JPEG of the poster, something that was easy for the director to look at. Kim asked Valerie to pull together another small group meeting and added that she would love to be part of the discussion.
3 Review an early draft of the specification
Valerie explained that each activity is described as an entry within a collection of entries called a feed. Page 9 lists possible elements within a feed. That table is followed by a description of possible elements within an entry, which would describe it activity. She had a question as to who would be considered the author of the feed i.e. the educational trajectory document. Is it the author or the institution?
Kim commented that the group had been very clear that the learner is at the center of the educational trajectory and that no data moves without their authorization. Therefore the learner should be the author of the feed.
The group commented that the specification document was difficult to understand without Valerie's explanation of the underlying concepts and architecture. Kim recommended that Valerie provide an explanation of the big picture on the architecture prior to the next call. She recommended using the Pat scenario to show how the decisions made us far would impact the visual display.
Maria commented that the student would not be the only author of the fee. Certain entries would have to come from the school.
Simon clarified that when we say the learner is the author of the fee, we are saying that is the learner who selected everything, not that they offered everything.
Valerie agreed to keep working on the specification, orientation materials, and a diagram of Pat scenario that reflects the decisions to date.
4 Open discussion
- Valerie and Kim will share the suggestions of the larger group with the extracurricular small group.
- Valerie will provide orientation materials and some descriptions of the larger technical architecture.
- Kevin will proceed with UCSF's work to compile data and share lessons learned on the next call.