MedBiquitousUseCases-Metrics.doc

Metrics Use Cases

Document History

Date	By	Changes
22 Jun 2007	Tim Willett	Initial document

Use cases

Internal comparison of activities participation
Collecting and presenting evaluation data for a single activity
Compiling evaluation data for a programme
Communication of activity and evaluation data to Accreditor
Comparison of activities for educational research purposes
An activity evaluator plans the evaluation according to a MedBiq recommendation

Actors

Learner or Participant	The health professionals who undertake continuing education in order to improve competence, performance or patient outcomes, and to satisfy recertification needs.
Health professional recertification body	An organization that monitors health professionals, requiring them to participate in continuing education to maintain licensure or certification.
Provider	An institution that designs and delivers continuing education.
Accreditor	An organization that monitors, evaluates and accredits Providers. Accreditation gives Providers the right to award CE credits to Learners.
Supporter	An institution that provides funding to Providers.
Researcher	An individual or group of individuals interested in performing education research based on one or more Activities or Programmes. The Researchers do not necessarily represent the Provider, Accreditor or Supporter.
Electronic Educational System	A computer system used by the Provider to create, manage and/or deliver its educational programme. Includes, but not limited to, Learning Management Systems (LMSs), Virtual Learning Environments (VLEs), Content Management Systems (CMSs), and Learning Object Repositories (LORs).

Key Concepts

Activity	An educational intervention offered by a Provider to Learners. May be only one event (e.g. a single lecture or a computer module) or an organized set of events that address one topic.
Programme	All the CME Activities offered by a Provider.

1. Internal comparison of activities participation

ID	MWG-1
Title	Internal comparison of activities participation
Actors	Learner, Provider, Activity
Trigger event	A Provider wishes to compare the various activities it offers in terms of participation rates.
Success End Condition	The provider is able to determine which of its activities are offered most frequently or accessed by the most Learners.
Assumptions	The Provider offers multiple activities The Provider collects participation data on its activities
Description	Someone working for the Provider accesses the Provider’s electronic educational system. The employee obtains a list of all activities offered by the Provider. The employee selects which activities to compare and a time frame over which to compare the activities. The employee submits a query which returns participation metrics (or a subset of it) for each selected activity . The employee can then use this data to compare the Provider’s activities and make decisions about activities that are rarely employed, or which are often begun but not completed, etc.
Transactions	Provider’s management system accesses database that holds participation statistics for its activities .
Exceptions	No participation metrics are available.
Potential Standards	Participation metrics (included in MEMS).

2. Collecting and presenting evaluation data for a single activity

ID	MWG-2
Title	Collecting and presenting evaluation data for a single activity
Actors	Providers, Learners, Activities
Trigger event	A Provider is aware that a number of different evaluations have been done for a certain activity it offers. The Provider wishes to compile and present all evaluation data for that activity in a particular way (perhaps according to an internal standard).
Success End Condition	The Provider produces a report on an activity that compiles all evaluation data and presents it in a certain way.
Assumptions	Evaluations have been performed on the activity The purpose, target, method, level (Kirkpatrick?), result and conclusions of each evaluation have been electronically recorded in some database
Description	An employee of the Provider identifies a particular activity that has been the subject of multiple evaluations. The employee accesses the Provider’s management system and selects the activity for which to compile evaluations. The system displays a list of the evaluations performed on the selected activity. The employee selects which data to include in the compilation. The data may include: Activity title and description and goal (target competencies?) Participation metrics for the activity (see MWG-1) Educational evaluations. Every evaluation instance might include data on: Purpose of the evaluation Competencies evaluated Target – what is being evaluated? (Activity overall, instructor, eLearning module, educational method, etc.) Methodology Level (Kirkpatrick?) Results Conclusions or planned actions Where target and methodology are the same, the system may aggregate results from multiple evaluation instances. The system will compile the evaluation results into a human-readable document in a pre-determined format (perhaps a “impact matrix”, but this is beyond the scope of this specification).
Transactions	Activity may reference published Competencies (domain of MedBiq Competency Working Group). Provider’s management system accesses database holding data about description and goal of activity. Provider’s management system accesses database that holds participation statistics for the activity. Provider’s management system accesses database holding data from evaluation instances for the activity.
Exceptions	Evaluation data is not available.
Potential Standards	Educational evaluation data.

3. Compiling evaluation data for a programme

ID	MWG-3
Title	Compiling evaluation data for a programme
Actors	Provider, Activity, Programme
Trigger event	The Provider wishes to evaluate its programme to determine the effectiveness of the programme overall and to determine which activities included in the programme are most beneficial.
Success End Condition	The Provider produces a report on the effectiveness of a programme and the activities within that programme.
Assumptions	A programme consists of multiple activities The effectiveness of the programme may be greater than, less than, or equal to the sum of its parts Evaluations have been performed on the activities and on the programme overall The purpose, target, method, level (Kirkpatrick?), result and conclusions of each evaluation have been electronically recorded in some database
Description	An employee of the Provider identifies a programme for which evaluation data is to be compiled. The employee accesses the Provider’s management system and selects the programme to be evaluated. The management system identifies the activities included as part of the selected programme. The system retrieves description, participation and evaluation data for the selected programme and the activities within the programme. The data may include: see MWG-2 The system compiles a human-readable report on the programme and its component activities. The employee analyzes the report to draw conclusions about the effectiveness of the programme or its component activities.
Transactions	As in MWG-2: Activity (and programme) may reference published Competencies (domain of MedBiq Competency Working Group). Provider’s management system accesses database holding data about description and goal of activity (and programme). Provider’s management system accesses database that holds participation statistics for the activity (and programme). Provider’s management system accesses database holding data from evaluation instances for the activity (and programme). Additional: The system identifies which activities are included in the selected activity (beyond Group’s scope).
Exceptions	Data for part of the programme is unavailable.
Potential Standards	Programme report.

4. Communication of activity and evaluation data to Accreditor

ID	MWG-4
Title	Communication of activity and evaluation data to Accreditor
Actors	Provider, Activity, Accreditor, Programme
Trigger event	An Accreditor requests metrics and evaluation data from a Provider as part of a regular electronic report.
Success End Condition	Provider compiles and delivers such a report to the Accreditor electronically.
Assumptions	Provider has electronically collected information on mission, activities, evaluations and programme revisions The specific evaluation data collected may not be the same for every activity The specific evaluation data collected for an activity may not be the same for every evaluation instance
Description	An employee of the Provider accesses the Provider’s management system. The employee instructs the system to produce an accreditation report that collects data on all activities offered by the Provider. Data may include: see MWG-2 Where target and methodology are the same, the system may aggregate results from multiple evaluation instances. The system may compile a human-readable report for review by the Provider before submission to the Accreditor. Once approved, the data will be sent electronically to the Accreditor’s system in a machine-readable format. The Accreditor’s system may re-compile this data into a human-readable report.
Transactions	As in MWG-2: Activity may reference published Competencies (domain of MedBiq Competency Working Group). Provider’s management system accesses database holding data about description and goal of activity. Provider’s management system accesses database that holds participation statistics for the activity. Provider’s management system accesses database holding data from evaluation instances for the activity. Additional: Provider’s management system packages and sends data to Accreditor’s system.
Exceptions	Evaluation data is not electronically compiled and sent to Accreditor.
Potential Standards	Accreditation report.

5. Comparison of activities for educational research purposes

ID	MWG-5
Title	Comparison of activities for educational research purposes
Actors	Provider, Researcher, Activity, Programme
Trigger event	An independent researcher learns of innovative activities being offered by multiple Providers. The researcher wishes to compare evaluation data from the various activities to determine educational best practice.
Success End Condition	The researcher is able to compare and contrast educational evaluations of the various activities.
Assumptions	The researcher is able to identify the activities of interest and contact their Providers Providers consent to the research proposed and allow researcher to access their data Provider has electronically collected information on mission, activities, evaluations and programme revisions
Description	The researcher identifies one or more Activities of interest. The researcher contacts the Providers of the Activities, who agree to share data for research purposes. An employee at each of the Providers accesses their management system and electronically sends evaluation data for the particular activity to the researcher’s system. The data may include: see MWG-2 The researcher’s system compiles a report from the data received from the three Providers. The researcher can analyze the report for: differences/similarities in educational approaches evaluation methods used and levels targeted outcomes of the evaluations The researcher may then draw conclusions about educational best practices.
Transactions	As in MWG-2: Activity may reference published Competencies (domain of MedBiq Competency Working Group). Provider’s management system accesses database holding data about description and goal of activity. Provider’s management system accesses database that holds participation statistics for the activity. Provider’s management system accesses database holding data from evaluation instances for the activity. Additional: Provider’s management system packages and sends data to Researcher’s system.
Exceptions	Evaluations of the various activities cannot be compiled and analyzed for research purposes.
Potential Standards	Educational evaluation.

6. An activity evaluator plans the evaluation according to a MedBiq recommendation

ID	MWG-6
Title	An activity evaluator plans the evaluation according to a MedBiq recommendation
Actors	Provider, Activity, Accreditor
Trigger event	An employee of a Provider is planning to evaluate a particular activity. The evaluator wishes to plan the evaluation according to best educational evidence and the Accreditor’s expectations.
Success End Condition	The evaluation of the activity is planned and conducted according to best evidence and meets the Accreditor’s expectations.
Assumptions	Best evidence (best practices) for educational evaluation exist and have been identified Accreditors have specified their expectations in terms of specific evaluation methods to be used or outcomes to be measured
Description	A Provider has introduced a new educational activity. An employee of the Provider is charged with planning and administering the evaluation of the new activity. The employee reviews the recommendations (or specification or standard) offered by MedBiq and by the Accreditor. The recommendations offer specific suggestions for methods to follow, tools to use, questions to ask, data to collect, etc. The employee plans the evaluation using the recommended methods and tools. The evaluation is conducted and data are collected. Data collection is organized and input into the Provider’s management system according to MedBiq specifications.
Transactions	The Provider accesses the recommendations in a human-readable format. The Provider’s system accesses the specification for the data structure in a machine-readable format. The employee is able to understand the data structure and enter data into it using the Provider’s system.
Exceptions
Potential Standards	Standard incorporating best practices and accreditor expectations.

Space shortcuts

Page tree

Metrics Use Cases