Published ACGME Milestones (http://www.acgme-nas.org/milestones.html)
- They follow a very similar structure
- Preface includes a description of what each level represents
- Some specialties have taken an EPA-like approach, focusing on certain clinical problems and describing the performance levels for the medical knowledge and patient care associated with that clinical problem.
- Some include possible methods of assessment/evaluation.
- Radiology includes suggested educational resources.
- Radiology provides categories for criterion for some competencies.
- Several frameworks provide one or more examples for each performance level of a given competency.
- Pediatrics includes a not Yet Assessable column; the framework differs form the pediatric milestones framework reviewed below.
- Psychiatry numbers each criterion within a performance level.
- Psychiatry provides footnotes for some criteria.
- Psychiatry provides annotations for some competencies and some criteria.
- Surgery has 9 practice domains that exist alongside the competencies and serve as an alternate form of organizing the competencies.
- in urology, there are some competencies that are actually bulleted lists.
Pediatric milestones (see: http://www.acgme.org/acgmeweb/Portals/0/PFAssets/ProgramResources/320_PedsMilestonesProject.pdf )
- Definition of Milestones from Nasca et al: developmentally based, specialty-specific achievements that residents are expected to demonstrate at established intervals as they progress through training.
- Milestones are connected to ACGME competencies.
- Background information related to assessment of the the competency as a whole is included. Background may include subsections, tables, matrices, or diagrams.
- References to the literature are included.
- Developmental milestones are provided that describe a scale of behaviors (generally novice to expert or incompetent to expert). These milestones have no numbering associated with them and no labels associated with them, but they do have an order.
- In at least one case (Prescribe and perform all medical procedures - pg 23-29), the developmental milestones are presented as a table with multiple elements, each with its own spectrum of capability. In some cases, performance is described in relation to test scores and the number of standard deviations above or below the mean. There are anchors associated with the beginning and end of the scale: Beginning of Spectrum, and Verification of Training Complete.
- This could be represented as:
- Multiple sub-competencies (i.e. each 'element', such as "anatomy and physiology" or "indications and benefits") of the main competency ("Prescribe and perform all medical procedures"), with one performance level set linked to each sub-competency
- Multiple performance level sets (i.e. one level set for each 'element') linked to one competency ("Prescribe and perform all medical procedures")
- A hierarchy within a performance level set, where the progression along the elements, from "anatomy and physiology" to "post-procedure management", is represented as a performance level set that is a parent to multiple more granular level sets (one for each element) (I don't think anatomy and post procedure mgt are levels of prescribing and performance of procedures - vs)
- This could be represented as:
- In the case of "Demonstrate trustworthiness..." page 111, each "milestone element" has a description and its own set of milestones.This competency also has a set of examples that show learners with varying levels of performance for different elements of the same competency (for example, solid knowledge and skills but poor discernment).
- See comment above re. how the different elements could be represented. However, the third option would not apply here since these elements seem discreet (i.e. each element is something different; there is no progression among the elements, only within each element).
- Not sure how to capture the examples; perhaps these belong in the background for the milestone.
- Each element has its own abbreviated title and description
- In some cases a single competency has multiple sets of milestones. For example, "Perform complete and accurate physical examinations" has sets of milestones for Psychomotor performance of the physical exam, Approach to the pediatric physical exam, and approach to the focused physical exam. "Provide appropriate supervision" (pg 37) has a set of milestones for the supervisor and another for the supervisee. "Professionalism" (pg 80) has sets of milestones for Professionalization, Professional conduct, Humanism, and Cultural competence.
- This could be represented as:
- Multiple sub-competencies (e.g. one for the supervisor and one for the supervisee), each with one performance level set linked to it.
- Multiple performance levels sets that each link to the same competency (e.g. "provide appropriate supervision")
- This could be represented as:
- In some cases the background references or includes a diagram (in at least one case the diagram follows the developmental milestones).
- In some cases, the developmental milestones (i.e. the granular performance level descriptions) include examples.
Internal Medicine Curricular milestones (see article) (Is this more appropriately represented as a competency framework?)
- "The milestones would explicate the 6 ACGME general competencies by describing a developmental progression of observable behaviors." see reference
- Appears to expand the ACGME competency domains (e.g. "Patient Care") into competencies (e.g. "Clinical skills and reasoning") and then into sub-competencies (e.g. "Manages patients using clinical skills of interviewing and physical examination")
- Each milestone set is linked to one competency (not to the more granular sub-competencies)
- Each competency may have more than one milestone set linked to it
- Milestones have a unique identifier (e.g. PC-A1 to A4; PC-B1 to B4)
Milestones map to a competency domain (IE patient care)- Milestones are grouped by suptopic (ie performing a physical exam, clinical reasoning, etc.) - i.e. each milestone set has a title. Arguably, this title could represent a sub-competency.
- Milestones are developmental in progression
- Milestones do NOT constitute a scale for assessment
- Milestones may be mapped to entrustable professional activities (not in this document)
- (timeframe appeared in the initial article but is intentionally omitted from this list as KC indicated that would not be incorporated into practice)
Internal Medicine Reporting Milestones (see document and publication) (updated Jan 30, 2013)
Milestones are "observable developmental steps that describe progression from a beginning learner to the expected level of proficiency at the completion of training. Accordingly, predefined milestones can be used to assess and document a trainee's developmental progression toward competence."
Milestone sets are mapped 1:1 to a competency - note these are not ACGME competencies; they appear to be custom competencies within the ACGME competency domains. Nor are they the same competencies used in the Curricular Milestones.
- Five levels exist for each competency.
- The levels correspond to the following general schema:
- Critical deficiencies
- Column 2 - early learner (unlabelled)
- Column 3 - demonstrating improvement (unlabelled)
- Ready for unsupervised practice
- Aspirational
- Critical Deficiencies and Aspirational are anchors
- Level 4 is Ready for Unsupervised practice
- Milestones are designed for use in assessment
- Learners may be ranked as in the middle of a particular level or on the border between two different levels. (i.e. there are 9 checkboxes for the 5 performance levels)
- Each subcompetency has a comment space. (more relevant to Educational Achievement)
- Each domain has a sign off and a global rating of YES, NO, or MARGINAL (more relevant to Educational Achievement?)
Internal medicine Tufts (see Meade et al. - From theory to actual practice Creation and application of milestones in an internal medicine residency program, 2004-2010.pdf)
- Extended the ACGME competencies to include a 7th domain, "teamwork."
Categorize milestones as progressive, additive, or descriptive.
In progressive, descriptors move from simple to complex, each of which builds on the previous.
In additive, the milestone lists several independent but related skills, with the descriptors showing a progression to consistent use of of those skills even in complex situations.
In descriptive, a set of complex steps towards mastery are already described in the literature and these are adopted verbatim
Milestones describe the developmental progression of observable behaviours and provide more specific feedback to residents, ensuring the acquisition of necessary knowledge, skills, and attitudes for advancement (Swing 2007; Carraccio et al. 2008; Green et al. 2009; Varney et al. 2009)
"Fundamentals" are a framework for organizing sets of milestones and linking them back to the broader ACGME competencies.
Each fundamental is mapped to one or more ACGME competency domains
- Not clear whether fundamentals are mapped 1:1 or 1:many to milestones
- Could these be represented as sub-competencies, each with one performance level set?
"Milestones" are complex behaviours made up of multiple but complementary specific behaviours, which when consistently present, serve to denote the achievement of competence in a given area.
We would call this a performance level set
- Each milestone has 5 "descriptors," which describe the progress from amateur to expert. Labelled as Failure, Needs work, Competent, Proficient, and Expert.
- We would call these the performance levels
- We would call these the performance levels
- Milestones are sets of behaviors designed to support achievement of competence.
- Milestones are used for curriculum, evaluation, resident feedback sessions, and faculty development work
Emergency Medicine (see document)
- Each Milestone has a title and unique identifier
- Each milestone maps to a competency
- Five performance levels are defined, each describing behaviors associated with that level.
- Learners may be ranked as in the middle of a particular level or on the border between two different levels.
- There are suggested evaluation methods for each milestone.
Core Competencies for Public Health Professionals (see link)
- Competencies have a unique identifier.
- Used for workforce development and assessment
- Organized into 8 domains (Analytical/Assessment skills, Policy development /program planning skills, etc.)
- Presented in three tiers. Tier 1 is entry level, Tier 2 is program management/supervisory level, tier 3 is senior management/executive level.
- Competencies are arranged to show difference across tiers (ie progresive development of competencies)
- Some Tier 3 competencies have no tier 1 and sometimes no tier 2 corresponding competencies (1C13, 2C5, etc).
- Competencies may appear in more than one tier. the identifier differs dpending on the tier. The way competence is demonstrated may differ from tier to tier.
- Has been cross-walked with essential public health services. (link)
Quad Council Competencies for Public Health Nurses (see link)
- The content and structure is based on the Core Competencies for public health professionals but is tailored for public health nurses.
- Organized into 8 domains.
- Presented in three tiers. Tier 1 applies to generalist public health nurses. Tier 2 apply to apply to PHNs with an array of program implementation, management and/or supervisory responsibilities. Tier 3
apply to PHNs at an executive/senior management level and leadership levels in public health organizations.
- Competencies are arranged to show difference across tiers (ie progresive development of competencies)
- Some Tier 3 competencies have no tier 1 and sometimes no tier 2 corresponding competencies (1C13, 2C5, etc).
- A clinical evaluation form used at Hahn School of Nursing.
- 5 different domains are assessed.
- Four of the five domains have a 0 to 4 rating scale. 0 is incompetent; 4 is Advanced or expert.
- The domain of professionalism has a list of characteristics, all of which must be demonstrated to pass that domain.
- Students must receive a rating of 3 in each section and a pass in professionalism in order to pass overall; 3 translates to competent.
- A clinical evaluation form used at Hahn School of Nursing.
- There are a number of statements about learner competency, and the supervisory must indicate if the competency has been met, whether its development is in progress, or if it has not been met.
The AACN Synergy Model for Patient Care
- The model includes patient characteristics and nurse characteristics.
- There are different competencies in each area
- There are different levels of competence for each competency
- Levels included are level 1, Level 3, and Level 5.
- Patient characteristics inlude labels for each level. For example, Level 1, minimally resilient; Level 2, moderately resilient, Level 3, Highly resilient
- Behavioral descriptiions of each level are included
- "Synergy results when the needs and characteristics of a patient, clinical unit or system are matched with a nurse's competencies."
National University of Singapore (see powerpoint - restricted to working group members only)
- NUS has defined a performance schema of 5 levels that describes in general terms the progression to competence with relation to a clinical outcome.
- 1= unable to achieve outcome;
- 2= requires a lot of guidance to achieve outcome;
- 3 = requires moderate amount of guidance to achieve outcome;
- 4 = able to achieve outcome with little or no guidance (entrustment);
- 5 = has the ability to guide/teach others.
- They have defined a set of presenting clinical problems (ex: Abdominal distension)
- They have defined a set of EPAs that describe, in very general terms, a series of professional activities required to address the clinical problem. Educators choose from a standardized list of activities to indicate the appropriate ones for a problem.
- They have defined a set of EPAs that describe, in very general terms, a series of outcomes and activities required to manage a condition.
- The performance schema is mapped to the curriculum (ie years of UME) using one of 3 tables: one to characterize problems that students learn to address early in their education (students start at level 3 for most epas), the second to characterize problems students learn to address in the middle of their education, and a third to characterize problems students learn to address at the end of their education (students begin at level 2 in all epas and end at level 3 in most).
- The performance schema is mapped to the curriculum (ie years of UME) for each condition using one of two tables: one that shows expected outcomes for a typical case, one that shows expected outcomes for atypical/complex/complicated cases.
- Problems are then mapped to one of these tables characterizing timeline of expectations with regards to problems.
- Different specialties may map problems differently (breathlessness may be mastered early in pediatrics and late in geriatrics).
- Problems are mapped to core differential diagnoses.
- Differential diagnoses are mapped to one of the tables characterizing timeline of expectations with regards to clinical conditions.
TIME (University of Texas) - see article
Outlines domains, competencies and transition milestones: levels of competence that students in TIME programs will be required to demonstrate before moving from the general academic ("college") campus to the medical campus.
Transition milestones are linked to a competency.
Different levels are assigned to each transition milestone.
I. Demonstrates decontextualized skills, knowledge, behaviors.
II. Demonstrates application of knowledge, skills, behavior to relevant health care problems
II+. Competency demonstration is required in non‐clinical settings as a transition milestone. The skill/behavior is expected whenever the student is in clinical settings, but its demonstration is not required as a transition milestone.
III. Demonstrates competency in simulated environments.
IV. Consistently demonstrates competency in a clinical setting under direct supervision.
V. Consistently demonstrates competency in a clinical setting with supervision immediately available.
VI. Consistently demonstrates competency in a clinical setting without supervision (“entrustable”).
- Transition milestones have levels II through IV.
- In some cases the level is "Demonstrate independent of setting."
- Some competencies do not have transition milestones.
- Milestones do NOT constitute a scale for assessment
UK Pharmacy professional development frameworks (see article)
- See figure 1 page 17
- The document mainly analyzes existing frameworks, particularly the GLF and ACLF.
- GLF is really a survey asking how often a particular behavior occurrs, with a likert scale of always to never. Each point on the scale has percentages associated (0-24%, 25-49%, etc).
- ACLF combines a performance framework showing behaviors at 3 levels - foundation, excellence, and mastery - with a portfolio tool recording evidence of the competency/performance level.
- It provides a summary of the different career trajectories undertaken by pharmacists following their first appointment in either a hospital or community setting. While this is not presented as a performance framework, it could serve as one.
- Instead of charting a path from novice to expert in any one area, the figure charts different paths toward increasing responsibility in different areas, including clinical service provision, management, research and education.
- The box in the lower left represents an entry level pharmacist. boxes to the right show increasing management responsibilities.
- Boxes higher in the chart represent increasing clinical responsibility and education and research responsibilities.
- The chart could serve as a career development tool for pharmacists early in their careers.
University of Toronto Orthopaedic Surgery Example (restricted to working group members only)
- There are categories and subcategories (similar to competencies) for the procedure evaluation
- Each subcategory has 5 levels, 1 corresponding to poor performance and 5 corresponding to superior performance.
- Levels 1, 3 and 5 have behavioral descriptors
Performance Level Schemas
RIME (See Pengaro Article and construct validity article)
A vocabulary of developing competence that includes four stages: Reporter, Interpreter, Manager, Educator
Each term is defined.
The vocabulary is designed for use in assessment tools.
Dreyfus model (See Caraccio article)
European Qualifications Framework
very general descriptions of 8 levels of knowledge, skill, and competence.