May 22, 2012
8 PDT/9 MDT/10 CDT/11 EDT/16 BST/17 CEST
Attending: Linda Lewin, Co-Chair; Susan Albright, Kelly Caverzagie, Bob Galbraith, Maureen Garrity, Reid Kallman, Howard Silverman, Scott Smith, Valerie Smothers, Jan Trial, and Lori Troy
1 Review minutes of last meeting
The minutes were accepted as submitted.
2 Review of PESC proposal re: norm data
Valerie began the discussion on the PESC (post secondary electronic standards council) proposal for adding mean, median, standard deviation, and frequency distribution (ie the range of scores for others in the cohort) to PESC schemas for norm-based scores. That data is currently missing from their standards describing test scores. PESC standards are in use in the higher education community for transmitting transcript data and standardized test scores among other things. They have a lot that we can build on, and they are very open to collaboration.
Valerie has been working with Michael Morris of ACT, who is the chair of the PESC working group for test scores. He commented that PESC could make the revised schema available quickly, but that a formally approved version of the PESC specification would take much longer.
Valerie provided an example of the kind of score ranges that could be represented with the PESC proposal: you could have one group of learners scoring 0 to 50% and another scoring over 50% and associate the number of learners scoring in each range. The approach would also work for scores like “proficient” and “superior.” Valerie asked the group if they thought that was sufficient to describe contextual data in educational achievement standard.
Linda commented that covered everything that we need; she can’t think of anything that is missing. She thought it made sense. Susan asked where this fits into the educational achievement spec. Valerie answered it would be an optional element. It came up in earlier discussions. Currently this type of data is included in the Medical Student Performance Evaluation (MSPE) for describing performance levels in the clerkships. We may just use for clerkship data. Before the May 8 call Valerie forwarded a document from Helen Chen at Stanford describing the trend to provide more contextual data about student performance. There may be an interest in using this data structure more broadly. Valerie asked Reid to introduce himself and comment.
Reid Kallman joins the group from Stanford as the Associate Registrar for Records. He discussed how some schools are showing the breakdown of student’s grades compared to other students. Susan asked if it works for pass-fail schools, and Reid wasn’t sure. He noted Stanford is still in the early stages of implementing enhanced transcripts. Linda asked whether the whole university was included or just medicine. Reid commented they were looking at this from the university level. Valerie commented that the PESC format would work for pass-fail; the percent of students passing could indicate 75% of students passed the course.
David asked about the format whether the group was going to stick to what the intent of the project is which is cumulative grade, or do we want to track per assessment tool? Valerie mentioned the model in the current schema is very general. The way we describe summary score for the course and the curriculum is using the same data structure. The amount of detail you include is up to the person who is implementing. Valerie strongly suggested developing implementation guidelines. David mentioned they are currently tracking each assessment tool, formative and summative as well. They are making some label modifications for consistency and normative data is calculated as needed. He asked if the intent was to communicate that data across systems. Does every institution opt in? If certain items are excluded, it may be hard to compare. Valerie the intent with norm-based data structure is to communicate data that is now communicated on student performance data. The data is already being exchanged, however; it is currently being exchanged in a narrative format. This provides a structured language for that data. Other applications are possible and could open new doors and new possibilities. She emphasized the need for policies in addition to data standards.
Linda clarified we are building the standard to create opportunities to do things that we don’t have data for yet. We may not use it right now. Valerie commented it sounded like we’re ok with what PESC has proposed. She asked the group to send her an email with any concerns.
3 Review updated specification
- Do you want to know if a course is required? or part of a track?
- Is it ok to include start and end date for courses and omit duration?
Valerie began on page thirteen with the terminology that has been updated and changed from the last time. It still needs to map for common terms. Several of the terms are new. Others that were not needed were taken out. The pediatrics milestones project is used as an example of a performance framework to make it easier to understand. Kelly mentioned the differences in medicine and pediatrics regarding the application of milestones. The application of the framework is different. Rather than saying pediatric milestones are an example of performance framework, he recommended saying the pediatric application of milestones is one example of a performance framework. Scott commented he is going to start equating milestones to performance framework; they are moving toward performance based. It may also include things like narrative information.
Linda mentioned the amount of information is good and we don’t want it to get more confusing. David mentioned the pediatrics project is an example of performance framework and he thought we should leave it the way it is. Linda agreed. Valerie suggested if other terms are necessary, please email her. She continued describing the changes on page twenty-two; she will edit the note to reflect the addition of the PESC offering to include score ranges and incorporate the revised schema. On page twenty-seven there were a couple of things removed to eliminate redundancy. Competency object reference is already there under events, as is title. On page thirty, evidence has been changed to include a portfolio reference. It is highly likely evidence in support of the learner may be outside the document or transcript. Portfolio Reference provides the ability to link to data inside the achievement document or on an external site. There is a sub-score element now. There are portfolio references on page thirty-five, with the ability to reference things inside and outside of the document. Page thirty-six and thirty-seven are examples of the ability to assess scores and report scores across curriculum. Susan suggested the evidence could point to external dissertation rather than a portfolio entry. Valerie agreed to change the language to supporting document or evidence.
The sequence block describes blocks of instruction or assessment in a curriculum. Valerie asked the group if it is important to know when a course is required or not, or whether a sequence block represents a track? Howard thought both were important. Linda asked for examples. Howard mentioned they have separate scores or outcome in residency. Pediatrics is going towards individualized tracks. Valerie stated we will keep the required and track attributes and she will document them for the next call. We need to make clear the course is required, with an option to describe if the course is representing a track. Erica Friedman at Mt. Sinai described an ambulatory medicine track in their clerkship; the learner takes different clerkships than others in a non-ambulatory track. Valerie added that the required attribute has three values: required, optional, and required track.
Valerie commented that in the Curriculum Inventory there are start dates and end dates as well as duration for sequence blocks. She asked Susan (Co-chair of the Curriculum Inventory working group) if it ok to have just start dates and end dates and omit duration. When describing curriculum, we need to be flexible. But educational achievement describes one learner passing through the curriculum. Not every residency has same length block of time, some do, and they can miss a month. Linda commented the duration isn’t always the same in many programs. A resident may take vacation during one block. Valerie asked Linda if the program tracks that data. Linda answered yes; every resident has to do actual time. Susan thought using start date and end date sounded fine.
Valerie went on to discuss the integration blocks. Susan suggested looking at pictures of integrating blocks, if the group wants to look at learners achievement based on curriculum theme. Anatomy would be an integration block that could reference events. Linda asked if the term integration block came from the Curriculum Inventory working group. Valerie replied it did: it describes themes that integrate the curriculum. Linda recommended adding that and some of the other curriculum inventory terms to the definitions.
Linda asked Valerie how close she thought this specification is to being final. Valerie thought at least a few more calls would be necessary, we are 85% there. The devil is always in the details. When people start applying it or mapping to current data, we will get good feedback. If there are points of friction, we can change things to make it easier. She noted the other factor is this specification is dependent on the competency working group to develop a performance framework specification that will take more than two months to develop; we’ll be in holding pattern until then.
4 Open discussion
- The working group accepted the PESC proposal for encoding cohort data (mean, median, standard deviation, and score range frequency).
- Attributes describing whether a sequence block/course is required or whether a sequence block represents a track will be maintained.
- Elements describing the duration of sequence blocks will be omitted as dates are sufficient.
- Valerie will change the language for portfolio reference to include non-portfolio references, such as dissertations.
- Valerie will add terms from the curriculum inventory to the Terminology section.
- Valerie will continue adding documentation of the schema.