Medbiq_logo_rtm.gif

 

 

 

 

 

MedBiquitous Accelerated Standards
Educational Achievement Standards Research Report
 

March 13, 2012

 

 


Introduction

As part of the standards development process within the Educational Achievement project, MedBiquitous has researched existing standards and specifications that could be assets in the development of an educational achievement specification. As a matter of best practice, MedBiquitous builds on existing standards when it is practical to do so. MedBiquitous staff researched credible learning technology standards developers and relevant open tools. A summary of the findings for each standard or specification researched is included along with information on relevant vocabularies, best practices, and recommendations on the use of competencies. Comments and suggestions for additional research are welcome.

Organizations Researched

The following organizations were researched:

  • Advanced Distributed Learning (ADL)
  • CEN (The European Committee for Standardization) ( complete )
  • HR-XML (A consortium for interoperability in human resources) ( in progress )
  • IEEE Learning Technologies Standards Committee (IEEE LTSC) ( complete )
  • IMS Global Learning ( complete )
  • The UK Joint Information Systems Committee (JISC) ( complete )
  • Mozilla ( complete )
  • The Post Secondary Education Standards Council (PESC)  ( complete )
  • Schools Interoperability Framework (SIF) ( complete )

Within each organization, a search for assessment and competency related standards, specifications, and systems was conducted.

Findings

While several of the specifications are partially overlapping in scope with the vision set our stated requirements, none of them integrate competencies, professional activities, assessments, and e-portfolios. Even in areas that are well-specified, such as assessment results, the fit is often imperfect. Depending on the IP policies of the specification and standards developing organizations, we may be able to build on existing work.

In addition, several of the specifications referenced are parts of a large architecture and appear to be difficult to untangle from the larger whole and use with other components. The PESC transcript specification and the EuroLMAI specification hold promise as a solution for communicating transcript data. But they do appear to be competing standards, each with significant adoption in their own domain. How we navigate that complexity is not clear.

Overall, health professions education appears distinctive in a few ways.

1)      There is an interest in knowing the methodologies used for assessment and the evidence used to determine a result

2)      There is an interest in having learner provided portfolio data integrated with more formal assessment data

3)      Having the capability to understand the full meaning of a mark is essential

4)      The inclusion of professional activities and the concept of entrustment as an educational achievement are unique.

To accurately communicate educational achievement data with these distinctions will require a new specification.

Existing standards, specifications, and open systems

 

Europass XML Schema v 2.0

http://europass.cedefop.europa.eu/en/home

http://europass.cedefop.europa.eu/TechnicalResources/XML/xsd/europass_XML-schema-v2.0-description.pdf

Europass is a European Union directive that seeks to make skills and qualifications easily understood across Europe. It includes tools, templates, and specifications for cvs and language skill assessment called a Language Passport. Europass includes identifying information about the individual, their occupation, work experience, education, language, and skills.

European Learner Mobility Achievement Information (EuroLMAI)

http://www.cen-wslt.din.de/sixcms_upload/media/3378/CWA16132.pdf

“The EuroLM Achievement Information Model seeks to address directly the European requirement for rapid implementation and dissemination of the institution-owned transparency information tools throughout Europe. This EuroLM Achievement Information (EuroLMAI) Model will be developed as a generic model for representing the institution attested achievements of a learner within a formal learning setting, resulting from the learner's participation in one or more learning opportunity instances and including any available associated results. The Europass institution-owned learner achievement information structures, namely the Europass Diploma Supplement and the Europass Mobility, as well as other official achievement reports (e.g. the learner's Transcript of records), will be specified as application profiles of the EuroLMAI model.”

EuroLMAI is a CEN Workshop Agreement approved by those who participated in its development. The specification build on CEN’s Metadata for Learning Opportunities (MLO) – Advertising (MLO-AD) and its Educational Credit Information Model. It combines information about the learning opportunity taken (MLO-AD), which can be a course or an entire program, and adds information about the learner, the issuer, the results of the learning opportunity, the grading scheme used, language, date, and additional information. The document may indicate achievement of a qualification or credit. The specification may be used as a lightweight transcript. The data is at a fairly course level of granularity, much coarser than the information discussed during the working group’s data analysis.

The Europass Diploma Supplement Application Profile of the EuroLMAI (EuroLMAI Europass DS AP) does add some further details to the data model, including the main field of study, any title conferred, and professional status. There are no detailed reports on competencies obtained.

 

HR-XML Assessment Report

http://ns.hr-xml.org/schemas/org_hr-xml/3_2/Documentation/ComponentDoc/AssessmentReport-noun.php

“A document carrying information about an assessment result. Provides scores and other details to fulfill or respond to an assessment order. AssessmentStatusDetails is part of the data model so that tests, whether completed or pending, can be reported upon.”

The HR-XML assessment result does have a way of indicating the competency assessed, the proficiency level (expressed as a score, point scale, or mark), and additional details determined by the implementer if desired. Scores include a code, interval, minimum, and maximum. Scores may be percentile, raw, stanine, sten, T-score, Z-score, Intelligence Quotient, Scaled, normal curve, or GPA. Scores may be represented as text as well. The specification also permits the indication of a “band,” a place on a normal distribution of scores. Comments may be added, and attachments are supported.

HR-XML EPM

http://ns.hr-xml.org/schemas/org_hr-xml/3_2/Documentation/ComponentDoc/EPMResult-noun.php

A result from an employee performance management (EPM) process or ‘appraisal’. An appraisal typically consists of a number of sections, including an overall rating or score, competency ratings, objective ratings, and other ratings (e.g., ‘core values’ is an example of a section within some appraisals outside of the competency or objectives section). Also includes result metadata, such as appraisal dates, ‘prepared by,’ etc. Note that EPM is an HR-XML approved abbreviation for "Employee Performance Management".”

A note about IP: “Licensee may not facilitate or assist any HR technology Standards Organization other than Licensor in co-opting, copying, distributing, publishing or using the Work without prior written approval of Licensor. Licensee may not create any Derivative Work for ownership by, presentation by, distributing by, publishing by or attribution to any Standards Organization or similar entity other than Licensor, unless specifically approved in writing by Licensor.” If we were interested in leveraging EPM, we would clarify our ability to create derivative works, and provide appropriate attribution to all parties, in writing with HR-XML.

HR-XML Position Competency Model

http://ns.hr-xml.org/schemas/org_hr-xml/3_2/Documentation/ComponentDoc/PositionCompetencyModel-noun.php

“A collection of competency information associated with a position or job. A position competency model includes references to individual competencies and references to groups of competencies that are required or associated with a position or job. Within those groups and individual competency references is information about associated proficiency levels (required or desired proficiency), and weightings of the group or individual competencies among their sibilings.”

Includes desired proficiency, required proficiency, level of difficulty, measurable dimensions of the competence (ie frequency), evidence requirements, and weight among sibling requirements, and related competencies.  This is fairly different than the MedBiquitous notion of a competency framework in that many more details about the importance of competencies and their relation to a specific job are detailed.

The IMS Learner Information Package

http://www.imsglobal.org/profiles/

IMS Learner Information Package (IMS LIP) provides a common format for the following types of learner information: accessibilities; activities; affiliations; competencies; goals; identifications; interests; qualifications, certifications and licences; relationship; security keys; and transcripts. IMS LIP is able to indicate the level of a qualification attained, for example, journeyman pipe fitter. The Competencies section is able to indicate skills a learner has acquired and their relationship to other activities or qualifications captured in the LIP. It also includes transcript data. The structure is abstract and fairly general. While IMS LIP does allow one to state attainment of a competence, it does not provide a way of indicating the level of performance at a particular point in time.

IMS also has a Learning Information Services development project which allows for outcomes management within an institution, but the details of that project are only visible to IMS members. (See: http://www.imsglobal.org/lis.html )

JISC Higher Education Achievement Report

http://www.xcri.org/wiki/index.php/HEAR

“The HEAR is a means for recording student achievement in higher education (HE) and is intended to provide more detailed information about a student’s learning and achievement than the traditional degree classification system. It will be issued to students on graduation and will include and extend the existing record of academic achievement - the academic transcript - and the European Diploma Supplement.”

“The Centre for Recording Achievement, the Higher Education Academy (the Academy), and the Joint Information Systems Committee are developing the HEAR. This is being overseen by the Burgess Implementation Steering Group. A pilot group of 18 institutions, using data from students who graduated in 2008 in four subject areas, have trialled the report, and a second trial took place in 2009-10.”

HEAR provides more information on the diploma, degree, or certificate (collectively known as qualification) including the level, contents, results, and function of the qualification. The broad approach to assessment used may be included in addition to the weighting, attempts, results, and grading scheme. HEAR is related to CEN’s European Learner Mobility Achievement Information.

Leap 2A & MedBiquitous Educational Trajectory

http://www.leapspecs.org/2A/

http://www.medbiq.org/sites/default/files/EducationalTrajectoryProfile.pdf

Leap 2A is an e-portfolio specification supported by CETIS, funded by JISC, and developed by Simon Grant and colleagues. Leap 2A e-portfolios may contain:

  • digital artifacts, including documents, audio, video, etc, made or jointly made by the portfolio holder
  • expressions or text entries by the portfolio holder , including learner reflections
  • information about the portfolio holder, their abilities, achievements, experiences, activities, goals, plans etc.

MedBiquitous has developed a profile of LEAP 2A that creates several vocabularies specific to e-potfolios for health professions education. The result is a profilecapable of describing the path of individual learners through one or more programs of study, including extracurricular learning, dates of study, gaps in study, and academic withdrawal or dismissal. Integration with the Educational Trajectory is a requirement for the Educational Achievement specification we are to build.

Mozilla Badges

https://wiki.mozilla.org/Badges

The Mozilla badges project is creating an open infrastructure that allows learners to collect and display badges. Badges are designed to recognize skills and abilities demonstrated outside of a formal educational environment. With funding from the MacArthur Foundation, Mozilla is developing an open infrastructure for awarding and collecting badges. The Khan Academy (www.khanacademy.org), a non-profit offering free online learning program, awards a variety of badges ranging from “Just Getting Started” ( for listening to 30 minutes of videos) to “Copernicus” (proficiency in any 200 exercises) or the more targeted “Master of Trigonometry.” The value of the badge is entirely subjective; schools and employers may or may not place value on badges a learner has earned. In some communities, peers may award badges. There is no way to connect badges to predefined competency frameworks and no way to include assessment information.

PESC College Transcript

http://www.pesc.org/interior.php?page_id=164

PESC offers specifications for both high school and college level transcripts. The transcript includes academic awards and honors, degree requirements, academic summary (ie GPA and program details), course grades, additional student achievements, immunizations, and test scores. PESc participants are based in the US and Canada; it’s unclear how well the specification would work for colleges outside of North America. There are codes for describing the area of study for a course. It’s not clear whether grades like “proficient” are able to be communicated using this specification. Notes may be included. Peer benchmark data is not included in the specification. There is an Education Record User Group which focuses on utilization and development of standardized, electronic transcripts.

PESC Education Test Score report

http://www.pesc.org/interior.php?page_id=132

“The Education Test Score Reporting standard can be used by any testing agency to fully communicate test score information to high schools, colleges and universities, affiliated third parties (clearinghouses, employers, research facilities, etc.), and local, state and federal agencies.”

PESC Education test Score Result provides a detailed specification for the communication of test scores. It includes capability to provide subtest scores (ie verbal scores on SATs). It also includes different types of scores: raw score, percent correct, scale score, labels describing a score (ie proficient), mastery scores (ie pass fail), academic grade, gpa, norm referenced, and scores using local scoring systems (ie not otherwise defined). Other student characteristics may be included. While there is no explicit reference to competencies, the subtest code could be used to reference competencies.

PESC Academic ePortfolio Workgroup

http://www.pesc.org/interior.php?page_id=209

PESC also has a workgroup around the development of Academic ePortfolios. It’s not clear if these groups have draft specifications that we could build on. A description of the vision behind the work follows. Dana

“Sophisticated ePortfolio software is being utilized today for integrating course assignments with learning outcomes and academic standards. The assignments are scored using well-designed rubrics, and the data from these scores are then aggregated and disaggregated into reports showing how individuals, courses, programs and departments are performing on a wide variety of student learning outcomes. In addition to helping to manage accreditation systems, these reports help to drive positive change in the way teachers teach and students learn.”

“Taking ePortfolios one step further is the notion of "Folio Thinking," a concept proposed by Helen Chen at Stanford. It is a unique teaching approach that is based on supporting students who are creating their own ePortfolios with opportunities to reflect on the experience through coaching and other techniques. This process both enhances and authentically displays distinct levels of student achievement, knowledge and skills that ultimately will help learner's transition into meaningful work and career development after they graduate.”

Schools Interoperability Framework (SIF)

http://specification.sifassociation.org/Implementation/US/2.5/html/

“The Schools Interoperability Framework (SIF) is not a product, but  a technical blueprint for enabling diverse applications to interact and share data related to entities in the pK-12 instructional and administrative environment. SIF is designed to:

  • Facilitate data sharing and reporting between applications without incurring expensive customer development costs;
  • Enhance product functionality efficiently; and
  • Provide best-of-breed solutions to customers easily and seamlessly.”

SIF does have capability for communicating assessment data. They have vocabularies for defining the assessment type (achievement test, attitudinal test, personality test, statewide, diagnostic, etc) and the mechanism to associate the assessment with learning standards identified by a globally unique SIF identifier.  Assessments include performance levels defining score ranges indicating a certain level of achievement. There are numerous elements related to the administration, assignment, psychometrics, and instance details of the assessment. Assessments may have subtests. Measurement results for an item may be included and may detail Mean Square Fit, Weighted Mean Square Fit, Revised Mean Square Fit, Revised Point Biserial measure, Rasch Item Score, Response Correlation, Response Correlation Squared, Z CHI Square, PValue, PointBiserial, Biserial, DiscriminationIndex, Reliability Coefficient, Coefficient Alpha, ItemTestCorrelation, ItemVariance, and ScaleValue. SIF can encode an individual student’s responses to an assessment and a set of scores. Scores are described using a value and a diagnostic statement (ie shows ability to identify symmetry). Metadata can be associated with the assessment including grade level, subject areas, and learning standards (ie competencies) among other things. SIF also provides detailed standards for K-12 academic records.

SIF has announced a collaboration with IMS to “to develop state of the art online assessment capabilities for students across the country.” This appears to mean they are finding ways for SIF and IMS specifications to work together. SIF has worked with the National Center for Education Statistics (NCES), on Common Education Data Standards (CEDS).


Tin Can API

http://www.adlnet.gov/wp-content/uploads/2012/02/TinCanAPI.pdf

“ADL, through a 2011 Broad Area Announcement (BAA) with Rustici Software, researched an implementation approach to Next Generation SCORM as if there were no previous generation of SCORM. That is, if we were to start from scratch with the web infrastructure that exists today, how would we implement distributed learning that is interoperable and reusable?”

“The answer was simple but powerful – ‘I did this’. The idea of describing learning experiences as ‘actor verb object’ can describe traditional learning content or may go out to other forms of content not normally tracked. So whether ‘Jack completed the Chapter 5 Algebra exam’ or ‘Jill experienced Fighter Pilot Training Simulation #17’, what happened can be tracked easily, but can also be re-aggregated with other learning experiences to create something larger and possibly more meaningful.”

The MedBiquitous Technical Steering Committee and the Virtual Patients Working Group received an update on this project from ADL on March 12. The specification does allow for the transmission of score data and the collection of educational analytics by those with appropriate rights (ie an instructor can run queries regarding their class). The beta version of the specification is due for release in a day or two. There is currently no notion of competencies in the specification.

Controlled Vocabularies

Our approach should account for some important controlled vocabularies for healthcare and health professions education.

Instructional methods, Assessment methods, and Resources

https://www.aamc.org/download/273966/data/cistandardizedvocablist.pdf

The MedBiquitous Curriculum Inventory working group has developed a set of vocabularies describing instructional methods, assessment methods, and resources used in health professions education. The vocabularies are being maintained by the AAMC and will be updated on a quarterly basis.

Medical Subject Headings (MeSH)

Developed by the National Library of Medicine for indexing PubMed content, MeSH is used by many schools to index their course offerings.

S ystematized No menclature of Med icine- C linical T erms ( SNOMED-CT)

SNOMED-CT is a comprehensive, multilingual healthcare terminology designed for use in healthcare information systems.   The International Health Terminology Standards Development Organization (IHTSDO) administers SNOMED CT licenses. IHTSDO members manage the release, adaptation, distribution and sub-licensing of SNOMED CT and other products of the Association within their Territory. In the US, the National Library of Medicine distributes SNOMED CT at no cost. IHTSDO includes 15 member countries, including the US and the UK. For full details, see: http://www.ihtsdo.org/members/

Healthcare LOM

Developed by MedBiquitous, Healthcare LOM provides vocabularies and recommended lists of terms for the following:

  • Activity sponsorship(joint/direct)
  • Participation modality (conference/workshop, technology based, on the job, print)
  • Activity delivery (live/not live)
  • Accrediting bodies (recommended list for CE)
  • Activity certification (recommended list for CE)
  • Credit type (CME, CE, CNE, CPE, and CHES, CPD)
  • Credit unit (CECH, CEH, CEU, Cognate, Contact Hour, Credit, Hour, Unit, Credit Hour, and Point)
  • Pacing (learner paced/provider paced)
  • Audience category (general, patient, caregiver, professional)
  • Profession (recommended list)
  • Specialty (recommended list)
  • Reading level (recommended list)
  • Orientation (axial, coronal, horizontal, longitudinal, sagittal, transverse)
  • Medical Image Type (11 terms)
  • Specimen type (cell, organ, organ system, organelle, tissue)
  • Learning resource type (expanded on LOM vocabulary)
  • Educational context (9 terms)
  • Classification purpose (expanded on lom vocabulary)

 

Use of Competencies

Whatever approach adopted should accommodate references to competencies, learning objectives, etc. The MedBiquitous Competency Object and Competency framework specifications provide a mechanism for uniquely identifying a competency statement/outcome/learning objective/milestone and collections of competency statements/outcomes/learning objectives/milestones. A conceptual model of this work is available at: http://groups.medbiq.org/medbiq/display/CWG/Conceptual+Model