Child pages
  • 2012-04-04
Skip to end of metadata
Go to start of metadata

Meeting Information


April 4, 2012


8 PDT/10 CDT/11 EDT/16 BST

Attending: Valerie Smothers, Tim Willett, Josh Jacobs, Mary Pat Aust, Steve Clyman, Dana Bostrom, Bob Englander, Kelly Caverzagie, Susan Albright.

Agenda Items

1 Review minutes of last meeting

Tim reviewed the minutes. Last time we discussed performance frameworks and milestones. We discussed other people to include in the working group. As a result we have several new people on today’s call. Second, we discussed competency definitions. Bob shared the work he has been doing with the AAMC to standardize this language. We looked at definitions and got feedback. Valerie and Rosalyn were going to look to see if an EPA could be represented as a competency framework, and Tim revised definitions.

The minutes were accepted.

2 Update on working group expansion

Tim commented Valerie has recruited fantastic additions to the group.  The new members introduced themselves.

Josh said hello from Japan. He originally is from the University of Hawaii. He worked quite a bit with Terri Cameron on Currmit at University of Hawaii. He then moved to Japan for a couple of years, moved back to Hawaii, and now he is at National University of Singapore, overseeing the curriculum. He is looking forward to learning with the group.

Steve is at the NBME in Philadelphia. He has a background in medicine and medical informatics, and worked on patient exams, part 3. He works with Bob Galbraith dabbling in assessment formats.

Kelly is from the University of Nebraska medical center. He is working on an alliance for academic medicine. He is working on internal medicine milestones with program directors, looking through lens of Entrustable Professional Activities (EPAs).

Dana is at the AAMC, where she manages the electronic portfolio project, which will help folks manage and track competencies.

Tim provided a quick overview of the group’s work to date and scope moving forward. To date we have specs for competency object and competency framework. A competency object  has a single competency statement and associated metadata. A Competency Framework in currently in review in the standards committee. After a public review and ballot, the specification will be submitted to ANSI for approval. The Competency Framework points to competency objects and gathers them into a hierarchical or non-hierarchical framework. Near the end of discussions, the idea of milestones assessment and performance criteria became prevalent. They made the decision to not include information on assessment of competencies in the competency framework. Now we are examining that issue. We are trying to standardize definitions and doing an environmental scan to understand what people are doing in reference to performance criteria: what they are doing, how that is related to other data, etc.

3 Revised definitions

Tim commented that anything changed is in green. He replaced performance level with performance level schema and performance criteria. A set of levels can be applied to any competency. There may be 4-5 levels, may be novice to master, etc. Others, in nursing had levels 1, 2, 3.

Josh commented that performance criteria has been described as anchor statements. That is another alternative to consider. Susan asked if that is being more broadly used.

Josh commented that he has seen it used when talking about a mini cex. The purpose is to anchor scales being used. Often the anchor provides a clinical context, or qualitative context rather than a numeric score. The Pediatric group refers to it as a milestone anchor.

Tim commented that they have expressed it as measurable behavior.

Bob agreed. They usually use it as a dyad. It’s a behavioral anchor. Tim commented that may be more recognized than performance criteria. Bob replied that performance criteria describes it well. He likes the distinction. There is a schema behind the performance level. He added he may still call it performance level schema and performance level.

Kelly mentioned that some had called performance criteria milestones. That fits. Milestones are a bit different than more discrete behaviors. Performance criteria makes sense from a more global perspective. Milestones in internal medicine are very discreet observable behaviors. You can have developmental levels within milestones. You may have 4 milestones addressing a developmental stage of behavior. That fits with other milestones work. Under performance criteria, we should explicitly describe milestones. They are developing the schema that goes on top of that.

Tim commented that for a given competency, there are a number of behaviors. He asked if for each behavior, there are a number of milestones.

Kelly commented that for internal medicine, under the competency of Interpersonal communication, there are 20 discrete behaviors referred to as milestones. 3-4 may have different performance levels of that behavior.

Bob commented that the definition of milestone has been the most difficult. It has taken on various meanings depending on the context. The most granular is performance level. The milestone Kelly is referring to is making the jump to the next level of performance, discrete measurable behaviors for a specific competency. Level 1 to level 2. Then there are also aggregates of multiple performance levels that make other milestones. The transition from supervised to unsupervised resident for a specific task. That means you have achieved an aggregate of performance levels.

Tim commented that it refers both to a statement and to point in time when achieved. Bob agreed. It is something that represents an achievement step.

Tim asked if we are on the right track to avoid pigeonholing a definition of milestone and using other terms.

Josh commented that the term that jumps to mind is performance threshold. Once you have crossed, you have entered the next step.

Bob asked if the current terminology is adequate to allow us to create the standard.

Tim commented that the approach we took for the competency framework, was that we developed a schema that would accommodate the same principles regardless of what an institution calls them. We could apply the same approach. He agreed to change performance criteria to performance level.

Kelly commented that Level is the threshold. Criteria is the thing to overcome.

Bob commented that is the inverse of what has been done in the glossary. The term is performance level but it is defined by performance criteria. The other reason to do it that way is that you have a schema but nothing that it is a schema for.

Tim commented that level is a description of what beginner is like, etc. That consists of one or more performance criteria, which are discrete behaviors one would expect.

Bob added that performance criteria has also been described as behavioral anchors.

Tim asked if Kelly could share a document on the Internal Medicine milestones. Kelly agreed to send it to Valerie.

Tim commented that another term is performance framework. That is all of the criteria for all the competencies collected together.

Susan asked what milestone would equate to. Tim replied it is equivalent to performance level. Some others refer to it as a point in time when certain levels have been achieved.  Susan commented it would be help if we included milestone in the appropriate place. Tim agreed to add a note saying some groups refer to level as milestone.

Kelly asked if it would be helpful to draw a table that lays out the different aspects. It would be helpful to see medicine, pediatrics, and surgery side by side.

Bob commented that to him, a performance level is just a statement: the expected criteria. The milestone implies achievement. Milestone is that behavior attached to an achievement. For interns, milestone is reaching the third performance level to go on to an unsupervised role.

Susan commented that requires its own definition.

Bob commented he is struggling with the feedback he’s gotten. His sense is attachment of performance level to achievement along the trajectory that makes it different.

Kelly sympathized. For their milestones, it’s a statement of behavior. They build assessment to measure it. In medicine, it is too overstepping to imply an expectation of a performance level. It’s just a statement of the behavior.

Bob commented he sees it specifically attached to the timeframe in Internal Medicine.

Kelly commented that when written the article was written, yes. But they don’t look at the time frame that much anymore. There was a suggested timeframe to begin application. But the timeframe will be varied based on the resident and program. They are still draft milestones. The timeframe in the paper has effectively gone away.

Tim asked if for this conceptualization of milestones there was an expectation of certain level to progress through a program. Is there another label to use?

Josh commented that learners are progressing through performance level schema until they get to a level of entrustability.

Tim commented that it is not necessarily entrustability. You may have completed a course or stage of training. He agreed to put it on the back burner and bring it up again next time.

4 Modeling EPAs as competency frameworks

Valerie commented that she and Rosalyn had attempted to model an Entrustable Professional Activity as a competency framework. The linked document shows how the EPA maps to the concepts or the competency framework.

Bob commented that Carol, Joe and he created a core set of epas, mapped then to critical domains of competency and subcompetency. The document shows a piece of that map. The wording of the competency has changed to acute single system diagnoses. If you look on the left, you see the competencies that map critically to an entrustment decision. As you go across the row, there are performance levels for those competencies. Down columns, you see what a novice looks like across integrated competencies for this EPA. 

Tim commented it would be helpful to see a corresponding milestone to see how an epa frameworks information differently than a milestone. Is the row the same row that would appear under a milestone?

B ob commented that physical exam is broken into three things. First milestone, performs …. Second, perform basic physical exam maneuvers, etc.  Each of those boxes is a milestone: Level 1-5 in the performance schema.  

Susan clarified that in this table, performance criteria would be the same as a milestone. Bob agreed.

Tim commented that we could conceptualize for a given competency a number of milestones or performance levels, each of which have criteria. How is an EPA different?

Bob replied it’s the integrated piece. On this sheet, all you see is one competency. The table goes on for 2-3 pages with other competencies critical to the entrustment decision. Each column represents integrated competencies at a particular performance level. Bob will send an example. You take the columns, create a paragraph of the expected behaviors across the competencies, and turn into a vignette.

Susan asked if to get to an epa, do you have to be expert in every competency listed?

Bob commented that the definition of EPA is an activity that you are entrusted to perform without direct supervision. One piece of work to do is decide level at which one is entrusted. That will be different for each epa, but not very. For most, 3 or 4. In most cases, entrustment will happen at the third column.

Tim commented that an EPA is binary. Either it has been achieved or has not. You need to hit the required level for each subcompetency. Bob commented that the table is a diagnostic for the learner and supervisor when the learner is not entrusted.

Tim commented that the relationship between the epa and each of the subcompetencies is required. There is also the relationship to each of the performance levels.

Tim commented that using the competency framework, you can’t capture the notion of which levels are required in order to pass the epa. There are other potential limitations. The ACGME publishes competency framework, but the specialties or programs define epas and milestones. This epa framework is related to a generic acgme competency framework; each subcompetency relates to milestones for pediatrics.  How do we connect those pieces?  Do we extend the Competency Framework, so that if representing as EPA, you can capture additional data,or do we release a similar extended framework?

Valerie agreed to research and come back with a proposal.

5 Review/brainstorm usecases


6 Open discussion 


Tim asked if there were other people that should be invited to participate.

Valerie commented she was hoping to get the American Board of Surgery.

Bob agreed it would be important to have a procedure based specialty represented. Surgery is the other group that has completed their milestones.

Tim asked about Emergency medicine. Bob replied they are in the works.

Tim commented there had been mention before of the royal college of nursing.

Mary Pat commented that everyone is struggling with this. She didn’t know of anyone that had made huge progress. There is a credentialing group for OB work on continued competency; they have released new programs. They may have more information.

Tim asked if Mary Pat could find out where they are in their work. If they have made progress, their input would be helpful.

Bob commented that so far we don’t have Pas represented. Nancy Davis is a PA and has colleagues in their organization. Olle ten Cate has done epas for PAs. They are another possibility. Valerie agreed to follow up with nancy on getting PAs involved.

Mary Pat added that the AACN has an advanced practice nursing consensus model. They are defining levels of advanced practice nursing, competencies and requirements for those specific groups. Representatives from AACN would be of value.

The working group will meet in person May 2. Valerie will research the feasibility of a teleconference for those unable to participate.


Action Items

  • Tim will update the terminology.
  • Kelly will send a document on the internal medicine milestones.
  • Valerie will research mechanisms for connecting performance levels and EPAs.
  • Mary Pat will research the credentialing group in OB to see if they have made progress in this area.
  • Valerie will follow up with Nancy Davis on getting Pas involved in our competency work.
  • Valerie will follow up with Mary Pat on AACN work.
  • Valerie will research the feasibility of a teleconference for May 2.
  • No labels