March 6, 2012
8 PST/11 EST/16 GMT
Attending: Linda Lewin and Alan Schwartz, Co-Chairs; Carol Carraccio, Kelly Caverzagie, Maureen Garrity, Simon Grant, Patty Hicks, Bob Galbraith, Rosalyn Scott, and Valerie Smothers.
1 Review minutes
The minutes were approved as submitted.
2 Review standards requirements
Alan referenced the link on the agenda for the group to look at. Valerie explained that this is an attempt to summarize the pieces of data in power point illustration. The power point document can be thought of as building a set of requirements for a standard. This document provides a concise format to use as a comparison when looking at other specifications. She asked the group to review the document to see if it matches what we’ve been talking about or if there are other pieces of data that need to be included. Valerie added that this is not a data model, just a summary of requirements.
Rosalyn asked what the difference was between reference to a performance level/milestone and evidence. Valerie commented there could be other types of evidence, assessment details, and attachments. Looking at the powerpoint, there are individual assessment items and in some cases, (slide 12) there is a matrix. Rosalyn suggested listing it as assessment data that includes performance levels. Patty commented that separating them out made sense; you want to know why someone decided this learner was at level 3. Rosalyn questioned whether evidence was a part of the performance level definition. Valerie commented there are two different kinds things: 1) criteria, or generic descriptions that don’t reference a particular learner (a description of what evidence would be necessary to support a learner being at that performance level), and 2) evidence regarding a particular learner and his/her competence. Criteria would be a part of the performance level definition; a learner’s evidence is part of the educational achievement definition.
Patty asked if Valerie anticipated evidence further qualifying the context and activities to help people interpret the data. On a given milestone a learner may go backward when their context changes. Valerie shared we had added the ability to include context when we talked about activities and statements of competence as well. Patty questioned the ability to integrate the competency and activity data. Valerie replied that the dates should provide a mechanism for integrating the data; however, it would be up to the system implementing the standards to do that integration.
Carol noted under benchmark data, there are a number of ways data could be represented. Would the group consider adding representation in a way that shows progression overtime? The Denver Developmental approach is a useful one. At a glance you can look at different milestones and see where someone is. You can tell where 90% of the learners would be in regard to a skill set. Carol offered to send a picture to Valerie. Alan agreed this is exactly the kind of data no one has until implementation starts. This is an argument for standardization. He added that he is not sure that the specification needs to change. It’s a matter of how the data is displayed more than the data itself. Rosalyn agreed data analysis is separate from the specification. Alan asked if we were tracking things like that anywhere else. Valerie replied that she will look on the wiki and if there is not a page for that we’ll create one. Linda suggested sending Valerie anything else that comes to mind before the next call.
3 Review standards research (working draft)
Valerie continued with a discussion on the standards research draft report. The report captures information on existing standards that are in the space in case. The report will compare the specs with our requirements. There is a list of organizations and efforts Valerie intends to research on page two; the orange lettering is the current status. There are several she has not begun to research yet, but she is hopeful by the next call the draft will be complete. The European learner mobility achievement information (EuroLMAI) and Europass are part of an EU effort to enable people to move freely across different countries in Europe. Europass 2.0 is for CVs; it has all kinds of information a CV would have, and a number of tools for creating that CV that makes it easy to send to different systems in different countries. It has XML embedded in that PDF document. It’s interesting but it does not address most of the things we discussed; it’s at a higher level. EuroLMAI addresses the same set of issues; it provides more details about formal educational opportunities like a transcript. She asked Simon if he had anything to add. Simon agreed EuroLMAI is really the transcript, the record of achievement given to all graduates when they graduate. It is linked to the UK higher education achievement report. Simon shared he was happy to answer any detailed questions. Valerie suggested to email the group with any questions for Simon.
Valerie discussed the HR-XML review, which is still in progress. HR-XML develops specifications for Human resources, and it is a different perspective than academic medicine. There are detailed assessment results for employee performance management. Competencies are defined in relation to a specific job, not necessarily abstracted out the way it is in academic medicine. It is not clear the extent to which their specification is what we want to do. Rosalyn questioned how HR-XML might be used in a healthcare workforce. As the workforce is evolving, could this marrying up with competencies needed for a particular job? Valerie answered maybe. She doesn’t know of any organization that is looking at their radiologist opening and weighting different competencies in radiology. Weighted competencies and assessment are possible in other jobs with a history of assessment and a lot of data available.
Patty asked if the question the learner releasing information (the ethics and policies) come up in the standards reviewed. Valerie answered no but added that she had not been looking for information on that. We are operating under the assumption that whatever application implements the standard will have those policies in place. Patty mentioned the slides we looked at last time had data partitioned. Would that influence the specification in terms of having open fields and other fields controlled and released under certain conditions. Valerie suggested putting something in our requirements to make that clearer.
Valerie reviewed information on the IMS global learning consortium. They have a specification that provides a way to say a particular learner has a particular competency; it doesn’t say anything about the level, which is a big gap. It includes transcript data and other kinds of data, too. Valerie noted there is more recent work the IMS is doing that may be relevant to our work but it is only available to IMS members. There is no way of knowing whether it is relevant, and we wouldn’t be able to use that work until it became more open.
For the Post-secondary Educational Standards Council (PESC), Dana Bostrom is our liaison and we can talk to her to clarify. They have a transcript standard that is very comprehensive. European one is lightweight; PESC is being used in the US and Canada for transmitting transcript information, SAT scores immunizations and grades. There are also PESC education test scores for reporting results of tests. There are no references to competencies built in. Finally there is the academic portfolio which is aligned with our thinking; however, it’s not clear if they are working on a draft specification. Valerie agreed to research that further.
Bob gave a quick update on the NBME/AAMC work stating they are getting to the point where they are near a final list of features in version 1 software. Once that is nailed down, it will be easier to disseminate. Valerie agreed it would be nice to see those requirements. Rosalyn commented there is nothing useable on this list. Valerie offered to go through the HR XML standards and see where they overlap to get a sense of how much overlap there is. She will take requirements and put them in a spreadsheet and see what they do. Linda commented the HR perspective is an interesting way to think about it. On a recent retreat about supporting residents, one person pointed out that for many this is their first job. It changes the focus. We can see if the HR standards are a good fit. Valerie is skeptical of the HR ability to support a broader portfolio, basing assessments on only a few things. There are lots of other standards organizations to research. Simon volunteered to help with pointing to relevant things at JISC.
4 Open discussion
Valerie will complete standards research
Valerie will add the capability to mark something as private to the standard requirements.