Meeting Information
Date: |
March 20, 2012 |
Time: |
8PDT/11 EDT/15 GMT |
Attending: Linda Lewin and Alan Schwartz, Co-Chairs; Susan Albright, Dana Bostrom, Carol Carraccio, Maureen Garrity, Simon Grant, Kimberly Hoffman, David Melamed, Morgan Passiment, Howard Silverman, C. Scott Smith, Valerie Smothers, Kevin Souza, Janet Trial, and Loreen Tory
Agenda Items
1. Review minutes
The minutes were approved as submitted
2. Review updated Standards Research and system requirements (see spreadsheet)
Valerie continued with a review of the system requirements. You now have the ability to mark something as private; no other charges were made to the prior document. The Standards Research Report was updated yesterday and sent out. Valerie described the meeting at the AAMC last Friday that brought together the Co-Chairs from the Competencies, Educational Trajectory, Curriculum Inventory working groups, Terri Cameron form AAMC, and Sasha Cohen, a key implementer from UCSF. The group discussed the various MedBiquitou specifications and how they relate to one another. One of the key insights was the relevance of the Curriculum Inventory specification in describing Educational Achievement data. Alan noted it was a productive meeting. Using the Competency Framework and Curriculum Inventory specifications for describing curriculum pathways from a medical school standpoint was a key insight. He mentioned leveraging that information could be powerful to see what the learner achievement is. There would be rich linkages between the kind of data the AAMC and LCME have on curriculum and educational achievement data to characterize the learner’s experience. Susan affirmed that we’re all talking about the same thing; the richness that the Curriculum Inventory could give to the Educational Achievement work could be useful in many use cases. It is worthwhile to continue to work together.
Valerie suggested the next step would be to walk through the PowerPoint presentation sent yesterday. She noted the Curriculum Inventory working group, chaired by Susan Albright from Tufts and Marc Triola from NYU was launched in 2010. The AAMC was developing a Curriculum Inventory portal and wanted local curriculum management systems to be able to upload data with the push of a button. There needed to be a standard for that exchange to take place. Valerie mentioned it was intended to be different than CurrMIT, which helps schools manage data. Instead, the new portal would support benchmarking and educational research. In addition, the Curriculum Inventory Portal would make data available to the ASSET system, a system designed to help schools prepare for LCME accreditation.
Valerie continued with the power point presentation on the data structure of Curriculum Inventory. Slide four pertains to the curriculum being made up of many educational and assessment events. The events include a description of assessment methods, instructional methods, and resources, all of which have vocabularies maintained by the AAMC. The blue dots are events, (lecture, test, specific discreet event). Expectations on slide five are represented as purple dots. Expectations reference competencies, learning outcomes, objectives – any statement of expectation about the learner. Milestones are excluded for now. Slide seven shows sequence blocks, which show how the curriculum is organized. Sequence blocks can be associated with expectations and can be nested (sequence block for the year, multiple modules, multiple courses, etc). Sequence blocks may contain references to events in the curriculum, learning objectives, and competencies as well. There is metadata about these sequence blocks, (how long it lasts, when it occurs). Slide eight shows the integration blocks, which represent themes that cut through the curriculum. For example, a theme like anatomy can show up in different sequence blocks. Once you define an integration block it can be referenced in events and sequence blocks. Slide nine shows sequence blocks being put together to show the structure of the curriculum.
Slide ten displays details about the data model, Expectations consist of individual competency objects (ie objectives, competencies, etc) and competency frameworks (how a set of competencies/objectives relate to one another). Slide eleven shows the data model for an event, which includes a unique identifier, title, duration in hours (so you can calculate how many hours spent on certain topics), keywords (such as MeSH). There is an indication that it is a professional event. There is a container for continuing education event credits associated with that. There is a reference to any competency objects associated with the event. You can indicate resources, like virtual patient, wet lab, and any other special resources used in instruction. There is a recommended evolving vocabulary for that. And either instructional method or assessment method has to be present. AAMC vocabularies are strongly recommended but not required.
Slide twelve is the data model for sequence block. There are a lot of attributes for indicating details about the structure of the curriculum: whether the course is required, if nested courses have to happen in a particular order, if the nested courses are selectives, the minimum and maximum number of courses required. There is an indicator if this sequence block is a track. It includes a title description, dates of the course, academic level. The clerkship element indicates whether this is a rotation or integrated clerkship model. There is a reference to competency objects, expectations for the course can be referenced there. If it is a course that has events you can reference that. Valerie paused and asked the group if they had any questions.
Susan mentioned when this group first started the time line to complete the work was only missed by a few months, and that it produced a good solid piece of work that everybody contributed to. Linda agreed the work is critical to what we’re doing and defines places we are going to get information achievement data from; each event can be linked to outcome measures which would be added. The sequence of events are key. She thought it clarified a great deal. Susan commented the use cases coming out of this will be the standard to guide data at the medical school level. Linda mentioned if the information is going to be required, then we all should be able to use the same standards. The question becomes what we need to develop to allow outcomes to connect what we want to provide. Alan thought we still need to connect things that happen outside the medical school framework, probably some summary competency scores, high level view. Valerie asked about weighted assessments, and Alan commented that would not be in the specification. We should allow the ability to use summary scores.
Simon thought it was worth considering the experience from Europe. They have an MLO specification, Metadata for Learning Opportunities. There is also the EuroLMAI specification for achievement information. With MLO there is an opportunity, just one kind of thing as opposed to events and sequence blocks. He recommended looking at the way in which EuroLMAI uses MLO. Do you want to do a profile of Curriculum Inventory within achievement data? Valerie mentioned the Curriculum Inventory has a slightly different purpose; it is not modeling individual learners. There may be things in Curriculum Inventory that are not needed and also some great things we want to use. Linda asked if we took our power point we may be able to use the terms and mark the overlap. Valerie noted grades aren’t in there and anything about individual learner is not there. Alan commented we could do different things that produce our radar charts and identify which learners had similar experience. Valerie asked Alan whether or not calculations should be done by the receiving or sending system; should the summary scores should be something in the specification or something that gets calculated when receiving data. Alan commented that was a difficult question. He noted if you have a situation where the learner is releasing data and only able to release their own data, not other learners, than you have to verify the source within each learner’s data or else you will have coordination of access problems.
Lori commented about the MSPE performance evaluation. If we took some information from MSPE relative to other students (ie this student was in the top quartile) that would give some indication of relative performance. Linda shared in the medical school realm; you would have to give the school the opportunity to give a summary. There should be a way to transmit summary scores and subscores. Lori commented that the subject scores, whether it was the first or last rotation would be useful information to make available that allows someone to put the data in perspective. Linda suggested going through the power point presentation again to highlight connections and she volunteered to help Valerie with that. Valerie commented they would work on that for the next call.
Linda asked if there were other items in the standards research report that needed to be highlighted. Valerie said one thing to consider in leveraging other specifications is how easily you can take the part out of the whole, or whether you just let that existing work inform your work. Several things that are unique require development of a new specification. Valerie stated medicine is unique in that you want to know the information behind the story; that didn’t show up in HR XML. The whole concept of entrustment is fairly unique as well. Simon mentioned medical education is the leading light in this area bur that he would be less sure the requirements are unique to medicine. Engineering and teaching do similar things. Alan agreed things may be more common than we think.
3. Next steps
Valerie will take a look at the power point presentation through new lenses and revise that accordingly. She agreed to come up with a draft data model specification with some general architecture for the group to kick around. Simon mentioned it may be useful to look at the relationship between EuroLMAI and MLO. Simon will go over that as a whole and look though it with a lens of integration. Valerie will follow-up with Simon on that. David asked Valerie for assistance in retrieving the vocabularies form the AAMC website and Valerie agreed to follow up with him on that.
4. Open discussion
Decisions
Action Items
- Valerie and Linda will revise the powerpoint presentation to show the alignment with the Curriculum Inventory
- Valerie will develop a straw man architecture for educational achievement data
- Valerie will follow up with Simon on lessons learned from EuroLMAI