August 15, 2012
8 PDT/9 MDT/10 CDT11 EDT/16 BST
Attending: Tim Willett, Co-Chair; Susan Albright, Mary Pat Aust, Stephen Clyman, Matthew Cownie, Robert Englander, Joshua Jacobs and Valerie Smothers.
1 Review minutes of last meeting
Tim began the meeting with a review of the minutes. Susan made a motion to accept the minutes and the motion was seconded by Mary Pat. The minutes were approved as submitted.
Tim commented that on the last call Kelly gave us examples of EPA’s and how they will be used in Internal Medicine. It became clear during the discussion that we have sets of data called milestones, EPA’s, and narratives, and they are all similar and related but not exactly the same. We will see that there is variability and we should keep flexibility moving forward.
2 Discuss potential use of narratives in internal medicine
Valerie announced that the AAMC agreed to accelerate the standards development process for the performance framework standard. The AAMC will provide funding to MedBiquitous, accelerating the frequency of the calls to twice a month. Jody will work with Tim and Rosalyn to set up a regular call schedule. Valerie is writing a project plan and will share it with the working group. Susan asked how the AAMC will be using a performance framework standard. Valerie answered that their interest relates to the efolio connector project. They want to be able to include performance data in a learner’s portfolio in future versions of the tool. Tim noted there was nobody from the AAMC on the call; however, Bob Englander is going to be our liaison. Tim asked Valerie what impact that has on our mandate to be agnostic and to meet the needs of multiple professions. Valerie commented it will help the AAMC if the standard is broadly applicable and can be used in other health professions and other countries. The earliest target date to send the standard to the Standards Committee is February 15. She noted we have a good conceptual framework to build on. Tim agreed. Valerie will be expanding the working group to allow for broader representation.
Tim commented that the liked document came from Kelly Caverzagie and represents a way of describing the current ability level of a resident. The narrative would be used in sending data to the board of internal medicine. The narratives are informed by assessments of resident’s achievement of milestones. Tim mentioned the power point slide is a draft and confidential. It is an example of one narrative for the competency patient care. There is a list of behaviors and spectrum from unacceptable to aspirational. There are nine boxes at the bottom, five are in line with columns and the others are in between. The statements in each column aren’t exactly the same as statements from their milestone document and are certainly more granular than any competencies listed.
Josh asked how this document would be used. Tim replied that Kelly had explained the Program Director or Program Assessment Committee would look at information provided by assessors and locate where on the continuum the resident is. It is not as mathematical as trying to add up the sum of where the resident is relative to each milestone, but more of an analytic activity. Bob commented that narratives are closer to pediatric milestones; there is a developmental progression of behaviors.
Josh commented that it looked like a combination of milestones and a mini CEX form with a nine-point scale. You would expect medical student to reach level three by the time they graduate, but that doesn’t leave a whole lot in residency to develop.
Mary Pat suggested taking it out of medical model and putting it in the continuum. Nursing has the educational piece, then in the clinical setting you begin the next phase of demonstrating competency. The two phases are separate and distinct. In nursing school, competencies are defined by the state board. In the clinical setting, the hospital or healthcare setting has separate competencies that must be demonstrated and a continuum from novice to expert that is distinct. Scales may be similar, but they often have different meanings.
Tim asked when nursing will have new competencies developed or expanded. Mary Pat commented you may be able to demonstrate medication as a basic general nursing knowledge; however, if you move into critical care, you are not going to be at that basic level. There are going to be additional competencies based on your current job, like hemodynamic monitoring, managing patients with respiratory distress, etc. Valerie clarified there are many competency and performance layers.
Josh commented that he is presenting the Singapore model in a 10 minute slot at AMEE. He offered to share with the group on a future call. Mary Pat requested he bring to our attention any specific examples in nursing. Susan added that the physician assistant program is using TUSK; it would be good to get feedback from that group.
3 Review revised definitions
4 Discuss use cases
Valerie began the discussion on use cases. Use cases describe a person who is acting in a specific role to accomplish a specific goal. The people and systems involved in the use cases are actors. She gave the example of going to an ATM and using your bank card to get money out. You are the actor interacting to get money, and if all goes well you have money. Sometimes the systems can be actors too. Use cases allow us to be clear about our goals for the standards we are creating.
The definitions of actors will be fleshed out. The group recommended breaking out the definition of faculty, which could entail several roles, including evaluator, decision maker, and mentor. There are also faculty that develop performance frameworks (framework developer). The group also recommended changing learner assessment system to add portfolio, curriculum management or performance management.
Mary Pat asked if the learner can include a clinician, anybody anywhere that is taking on the role of the learner. Valerie answered yes. The same person could be both faculty and administrator. The person might take on different roles at different times.
Valerie walked through the first use case and asked the group if there were any questions. Tim commented that sometimes people will want local copies of the framework, but in other cases they may want to reference a published document on the web. Valerie agreed and stated she would update the use cases. Tim added that publishing the framework should be its own use case. Tim asked the group to brainstormhow they might use performance frameworks in their context. Tim stated on the next call we will finish Valerie’s list and then open the discussion for suggestions from others.
5 Open discussion
- Valerie will provide the project plan to the working group when it is approved
- Valerie will update the use cases
- Adding definitions for actors
- Adding the following actors: evaluator, decision maker, mentor, framework developer (in place of faculty)
- Changing learner assessment system to add portfolio, curriculum management or performance management
- Adding a use case for publishing a framework
- Revise the use cases to account for referencing a framework published on the web.