Metrics Use Cases

 

 

Document History

Date

By

Changes

22 Jun 2007

Tim Willett

Initial document

 

 

 

 

 

 

 

 

 

 

 

Use cases

 

  1. Internal comparison of activities participation
  2. Collecting and presenting evaluation data for a single activity
  3. Compiling evaluation data for a programme
  4. Communication of activity and evaluation data to Accreditor
  5. Comparison of activities for educational research purposes
  6. An activity evaluator plans the evaluation according to a MedBiq recommendation

 

 

Actors

 

Learner or Participant

The health professionals who undertake continuing education in order to improve competence, performance or patient outcomes, and to satisfy recertification needs.

Health professional recertification body

An organization that monitors health professionals, requiring them to participate in continuing education to maintain licensure or certification.

Provider

An institution that designs and delivers continuing education.

Accreditor

An organization that monitors, evaluates and accredits Providers.  Accreditation gives Providers the right to award CE credits to Learners.

Supporter

An institution that provides funding to Providers.

Researcher

An individual or group of individuals interested in performing education research based on one or more Activities or Programmes.  The Researchers do not necessarily represent the Provider, Accreditor or Supporter.

Electronic Educational System

A computer system used by the Provider to create, manage and/or deliver its educational programme.  Includes, but not limited to, Learning Management Systems (LMSs), Virtual Learning Environments (VLEs), Content Management Systems (CMSs), and Learning Object Repositories (LORs).

 

 

Key Concepts

 

Activity

An educational intervention offered by a Provider to Learners.  May be only one event (e.g. a single lecture or a computer module) or an organized set of events that address one topic.

Programme

All the CME Activities offered by a Provider.

 

 


1.  Internal comparison of activities participation

 

ID

MWG-1

Title

Internal comparison of activities participation

Actors

Learner, Provider, Activity

Trigger event

A Provider wishes to compare the various activities it offers in terms of participation rates.

Success End Condition

The provider is able to determine which of its activities are offered most frequently or accessed by the most Learners.

Assumptions

  • The Provider offers multiple activities
  • The Provider collects participation data on its activities

Description

Someone working for the Provider accesses the Provider’s electronic educational system.

The employee obtains a list of all activities offered by the Provider.

The employee selects which activities to compare and a time frame over which to compare the activities.

The employee submits a query which returns participation metrics (or a subset of it) for each selected activity .

The employee can then use this data to compare the Provider’s activities and make decisions about activities that are rarely employed, or which are often begun but not completed, etc.

Transactions

Provider’s management system accesses database that holds participation statistics for its activities .

Exceptions

No participation metrics are available.

Potential Standards

Participation metrics (included in MEMS).

 


2. Collecting and presenting evaluation data for a single activity

 

ID

MWG-2

Title

Collecting and presenting evaluation data for a single activity

Actors

Providers, Learners, Activities

Trigger event

A Provider is aware that a number of different evaluations have been done for a certain activity it offers.  The Provider wishes to compile and present all evaluation data for that activity in a particular way (perhaps according to an internal standard).

Success End Condition

The Provider produces a report on an activity that compiles all evaluation data and presents it in a certain way.

Assumptions

  • Evaluations have been performed on the activity
  • The purpose, target, method, level (Kirkpatrick?), result and conclusions of each evaluation have been electronically recorded in some database

Description

An employee of the Provider identifies a particular activity that has been the subject of multiple evaluations.

The employee accesses the Provider’s management system and selects the activity for which to compile evaluations.

The system displays a list of the evaluations performed on the selected activity.

The employee selects which data to include in the compilation.

The data may include:

  • Activity title and description and goal (target competencies?)
  • Participation metrics for the activity (see MWG-1)
  • Educational evaluations.  Every evaluation instance might include data on:
  • Purpose of the evaluation
  • Competencies evaluated
  • Target – what is being evaluated? (Activity overall, instructor, eLearning module, educational method, etc.)
  • Methodology
  • Level (Kirkpatrick?)
  • Results
  • Conclusions or planned actions

Where target and methodology are the same, the system may aggregate results from multiple evaluation instances.

The system will compile the evaluation results into a human-readable document in a pre-determined format (perhaps a “impact matrix”, but this is beyond the scope of this specification).

Transactions

Activity may reference published Competencies (domain of MedBiq Competency Working Group).

Provider’s management system accesses database holding data about description and goal of activity.

Provider’s management system accesses database that holds participation statistics for the activity.

Provider’s management system accesses database holding data from evaluation instances for the activity.

Exceptions

Evaluation data is not available.

Potential Standards

Educational evaluation data.

 

 


3. Compiling evaluation data for a programme

 

ID

MWG-3

Title

Compiling evaluation data for a programme

Actors

Provider, Activity, Programme

Trigger event

The Provider wishes to evaluate its programme to determine the effectiveness of the programme overall and to determine which activities included in the programme are most beneficial.

Success End Condition

The Provider produces a report on the effectiveness of a programme and the activities within that programme.

Assumptions

  • A programme consists of multiple activities
  • The effectiveness of the programme may be greater than, less than, or equal to the sum of its parts
  • Evaluations have been performed on the activities and on the programme overall
  • The purpose, target, method, level (Kirkpatrick?), result and conclusions of each evaluation have been electronically recorded in some database

Description

An employee of the Provider identifies a programme for which evaluation data is to be compiled.

The employee accesses the Provider’s management system and selects the programme to be evaluated.

The management system identifies the activities included as part of the selected programme.

The system retrieves description, participation and evaluation data for the selected programme and the activities within the programme.

The data may include:

  • see MWG-2

The system compiles a human-readable report on the programme and its component activities.

The employee analyzes the report to draw conclusions about the effectiveness of the programme or its component activities.

Transactions

As in MWG-2:

Activity (and programme) may reference published Competencies (domain of MedBiq Competency Working Group).

Provider’s management system accesses database holding data about description and goal of activity (and programme).

Provider’s management system accesses database that holds participation statistics for the activity (and programme).

Provider’s management system accesses database holding data from evaluation instances for the activity (and programme).

Additional:

The system identifies which activities are included in the selected activity (beyond Group’s scope).

Exceptions

Data for part of the programme is unavailable.

Potential Standards

Programme report.


4. Communication of activity and evaluation data to Accreditor

 

ID

MWG-4

Title

Communication of activity and evaluation data to Accreditor

Actors

Provider, Activity, Accreditor, Programme

Trigger event

An Accreditor requests metrics and evaluation data from a Provider as part of a regular electronic report.

Success End Condition

Provider compiles and delivers such a report to the Accreditor electronically.

Assumptions

  • Provider has electronically collected information on mission, activities, evaluations and programme revisions
  • The specific evaluation data collected may not be the same for every activity
  • The specific evaluation data collected for an activity may not be the same for every evaluation instance

Description

An employee of the Provider accesses the Provider’s management system.

The employee instructs the system to produce an accreditation report that collects data on all activities offered by the Provider.

Data may include:

  • see MWG-2

Where target and methodology are the same, the system may aggregate results from multiple evaluation instances.

The system may compile a human-readable report for review by the Provider before submission to the Accreditor.

Once approved, the data will be sent electronically to the Accreditor’s system in a machine-readable format.

The Accreditor’s system may re-compile this data into a human-readable report.

Transactions

As in MWG-2:

Activity may reference published Competencies (domain of MedBiq Competency Working Group).

Provider’s management system accesses database holding data about description and goal of activity.

Provider’s management system accesses database that holds participation statistics for the activity.

Provider’s management system accesses database holding data from evaluation instances for the activity.

Additional:

Provider’s management system packages and sends data to Accreditor’s system.

Exceptions

Evaluation data is not electronically compiled and sent to Accreditor.

Potential Standards

Accreditation report.

 


5. Comparison of activities for educational research purposes

 

ID

MWG-5

Title

Comparison of activities for educational research purposes

Actors

Provider, Researcher, Activity, Programme

Trigger event

An independent researcher learns of innovative activities being offered by multiple Providers.  The researcher wishes to compare evaluation data from the various activities to determine educational best practice.

Success End Condition

The researcher is able to compare and contrast educational evaluations of the various activities.

Assumptions

  • The researcher is able to identify the activities of interest and contact their Providers
  • Providers consent to the research proposed and allow researcher to access their data
  • Provider has electronically collected information on mission, activities, evaluations and programme revisions

Description

The researcher identifies one or more Activities of interest.

The researcher contacts the Providers of the Activities, who agree to share data for research purposes.

An employee at each of the Providers accesses their management system and electronically sends evaluation data for the particular activity to the researcher’s system.

The data may include:

  • see MWG-2

The researcher’s system compiles a report from the data received from the three Providers.

The researcher can analyze the report for:

  • differences/similarities in educational approaches
  • evaluation methods used and levels targeted
  • outcomes of the evaluations

The researcher may then draw conclusions about educational best practices.

Transactions

As in MWG-2:

Activity may reference published Competencies (domain of MedBiq Competency Working Group).

Provider’s management system accesses database holding data about description and goal of activity.

Provider’s management system accesses database that holds participation statistics for the activity.

Provider’s management system accesses database holding data from evaluation instances for the activity.

Additional:

Provider’s management system packages and sends data to Researcher’s system.

Exceptions

Evaluations of the various activities cannot be compiled and analyzed for research purposes.

Potential Standards

Educational evaluation.

 


6. An activity evaluator plans the evaluation according to a MedBiq recommendation

 

ID

MWG-6

Title

An activity evaluator plans the evaluation according to a MedBiq recommendation

Actors

Provider, Activity, Accreditor

Trigger event

An employee of a Provider is planning to evaluate a particular activity.  The evaluator wishes to plan the evaluation according to best educational evidence and the Accreditor’s expectations.

Success End Condition

The evaluation of the activity is planned and conducted according to best evidence and meets the Accreditor’s expectations.

Assumptions

  • Best evidence (best practices) for educational evaluation exist and have been identified
  • Accreditors have specified their expectations in terms of specific evaluation methods to be used or outcomes to be measured

Description

A Provider has introduced a new educational activity.

An employee of the Provider is charged with planning and administering the evaluation of the new activity.

The employee reviews the recommendations (or specification or standard) offered by MedBiq and by the Accreditor.

The recommendations offer specific suggestions for methods to follow, tools to use, questions to ask, data to collect, etc.

The employee plans the evaluation using the recommended methods and tools.

The evaluation is conducted and data are collected.

Data collection is organized and input into the Provider’s management system according to MedBiq specifications.

Transactions

The Provider accesses the recommendations in a human-readable format.

The Provider’s system accesses the specification for the data structure in a machine-readable format.

The employee is able to understand the data structure and enter data into it using the Provider’s system.

Exceptions

 

Potential Standards

Standard incorporating best practices and accreditor expectations.