2013-10-11

Meeting Information

Attending: Tim Willett, Co-Chair; Kirstin Cirulis, Mary Jo Clark, Robert Englander, Jason Ladicos, Deborah Larimer, Dan Nelson, Paul Schilling, Valerie Smothers and Julie Watson.

Agenda Items

1. Review minutes of last call

Tim recapped the previous brief call.  Deborah had questions about the Performance Framework mapping to multiple things such as the AAMC milestones, ACGME competencies, etc. Tim noted there were a couple different models for doing that.  The next steps were identified as updating the Educational Achievement specification.  The colleagues from Tufts and Tulane spent time trying out the framework.  Deborah moved to accept the minutes as submitted and Dan second the motion.  Minutes were approved. 

2.  Discuss results of Tulane analysis (Tulane Milestones excerpt, Tulane objectives, and MedBiq diagram)

Deborah Larimer, from Tulane, continued with a detailed discussion on the specification.  A draft copy of the document was circulated to the group.  They looked at the milestones for interpersonal and communication skills and at the overall educational program objectives.  She explained that phase one objectives were to be accomplished by end of the second year; phase two milestones were to be accomplished by the time they graduate. Objectives are linked to the milestones; this is shown as little numbers and letters in bold following a description of each milestone. Year one milestone describes rapport building and references an attitude/behavior objective, treating the patient as a person.  The skill objective is establishing rapport with patients. She commented the objectives were set years ago by their curriculum committee. 

Jason asked about the skill S5, how it appears in multiple years, and asked if they have thresholds that span components and how that would be represented in the schema?  Deborah commented the developer would be better able to answer that question, however; she thinks the answer is yes they are going to be able to do that, and she hasn’t seen a problem representing that yet. She mentioned the developer is going to make some changes from the draft he sent by email last night.  It is a work in progress, as they build it out they find places where changes are needed.  Valerie also commented to Jason that we will be revising the Curriculum Inventory specification in January to integrate milestones.

Jason asked would there be a score applied to the skill?  Deborah answered they are planning to build that in and it will be an issue.  Valerie asked how these things work together for assessment.  Deb mentioned they are currently working on the assessment piece and haven’t determined that yet; they are anticipating assessing these milestones as much as possible with things that are already happening. There are two OSCEs they know they will be able to use.  They will also be looking at the existing evaluations that are completed, particularly those in the third and fourth years.  One of questions will be how to assess these things in the 1^st and 2^nd year; scientific faculty generally indicate that they don’t deal with any of these competencies. 

Valerie asked how the OSCEs will relate to milestones.  Deborah answered in terms of this particular competency interpersonal communication skills are rolled up in a unit.  All of the basic communication skills will be rolled up.  Mary Jo commented what they are trying to do in developing the assessment process is to use Standardized Patient Encounters (SPE) and map each SPE to a specific milestone, so that one SPE would address milestones XYZ, and subsequent ones would address additional ones. Over the course of the program every one of the milestones would be incorporated. 

Valerie asked if SPE scores feed back to related milestones. Mary Jo answered yes; SPE’s will feed into that.  There is a range of scores is 0-9, then for a specific competency there are specific behaviors that need to be demonstrated. The composite becomes a score for assessment.  Valerie noted they are scoring each competency. Bob thinks it would accommodate EPA’s as well.  Tim summarized that there are multiple measurement instances that map to different competencies, all loading together to indicate how they are progressing with respect to competency development. 

Valerie asked Deborah if assessment would feed into a score related to the attitude skills and knowledge.  Deborah answered they haven’t been measuring those using any scoring rubric.  They are trying to connect everything they’ve done in the past with previous objectives and ACGME competencies.  Tim asked Deborah when they do an assessment do they see themselves granularly assessing attitude and skills on that list or assessing the milestones and then inferring performance on the attitude and behaviors. Deborah answered the second one is the case. 

Valerie continued with discussion on the slide deck interpretation of the Tulane frameworks.  There are five different pieces, ACGME competencies, Tulane core competencies, Tulane global educational program objectives, AAMC competencies, and Tulane milestones. Valerie’s understanding was all but Tulane milestones would be represented as competency frameworks. Tulane milestones would be represented as a Performance Framework, which would allow you to see the progression of different levels of performance related to that dimension.  Slide three illustrates how the different pieces are connected. 

Valerie and Deborah spoke about a column missing in the Tulane milestones. If you look at row one of the interpersonal milestones, there is an implied sub-competency of rapport building.  The best practice would be to create a competency called rapport building and have the first row odf milestones as performance levels of that competency. 

Tim asked in this example rows 1-5 speak to goal A, could those be represented as five sub-competencies of single component, or presented as 5 individual components.  Valerie commented in some ways it’s the same thing; you could do both.  She commented that what is missing is linking a Performance Level to a Competency.  That is represented as a dashed line on slide 3. 

Tim asked about slide four.  Valerie commented slide four represents if Tulane milestones were not a Performance Framework but a Competency Framework.  The way competency framework was written is up to interpretation.  If you were to represent milestones as a Competency Framework, then it wouldn’t require any new code, however you wouldn’t be able to see developmental progression.  You would be able to see all connections among various frameworks.  Tim was leaning toward slide three being the better way. Tim asked Valerie if she took a stab at changing specification.  There is a link in the agenda item four where Valerie’s word document and XML schema show the changes.  Valerie added a competency reference element to Performance Level.  Mary Jo agreed with Deborah that slide three example will be more useful from an educational perspective. 

The group recommended creating a voice over powerpoint providing an overview of the specification.

3.  Questions and comments from the group

4.  Potential revisions to the specification (word document, XML schema)

Valerie continued with revisions to the specification with minor changes beginning on page 41 and then on the table that follows on page 43.  Performance Level has optional competency element.  It doesn’t describe the nature of the relationship of that competency.  Do we want to leave it as is?  Valerie asked the group if they felt this was a positive change.  Tim asked Jason if this addressed his uncertainty from his earlier questions.  Jason commented he would have to see it in practice.  Mary Jo asked if competency can incorporate sub-competencies. Valerie commented she would suggest representing competency at the component level. She gave the analogy of competency representing a tree branch, and the set of Performance Levels relates to specific competency.  A complex competency may have sub-competencies and milestones relate to that subcompetency. 

Deborah asked if the sub-elements and sub-competencies are the same.  Valerie explained in XML they have a notion of an element (like a noun) and an attribute (like an adjective).  The XML is hierarchical, and may break elements down into smaller pieces.  That is what a sub-element is.  Deberah asked where within the schema would competencies and subcompetencies be represented?  Valerie answered the information would be in the Competency Framework specification.  Deborah will have the developer chat with Valerie later today or next week. 

Jason asked about score and range values only accepting integers, and was there a reason why they can’t accept decimal values. Valerie commented learning performance data can be a decimal, but Performance Levels were defined as integers. Tim asked if there was a risk to allowing integer values there. Valerie commented the concern would be confusing the user of the specification.  We’ve not seen any frameworks to date that use decimals. Jason mentioned use of the DIV element and the supporting information element; he recommended using something with more sematic meaning like article.  Valerie will explore that further with Jason.  Valerie used the DIV element because of the mark up allowed.  She asked everyone to continue to make these suggestions to the entire group. Jason will do that. 

Jason asked why performance scales live at the top of the document and are not defined inside in the Performance Level set.  Valerie commented they have been taken out to reduce redundancy in the code; usually there are one or two performance scales defined that are used throughout the framework. For example, the Internal Medicine reporting milestones have two kinds of scales. There is a 1-5 scale for the competencies like gather essential and accurate information that goes from critical deficiencies to aspirational. There is also a summative 1-3 performance scale for competency domains like professionalism scale, indicating of the learner meets expectations: the scale provides yes, no, and marginal.  The three point scale gets referenced in six different places. By making it it its own thing, you don’t have to re-state that information. There is an advantage in terms of displaying data, knowing these competencies are using the same scale and may be displayed using a radar plot.  Jason thought that made sense. 

Tim asked if the group will hear back from Tufts, and Valerie thought since the AAMC meeting is coming up, she was unsure if they will have time to review.  Deborah thanked Valerie for her help and guidance for putting example together. Valerie thanked Deborah and her team for taking the time to review the specification.  Mary Jo stated it makes more sense to her seeing it in Tulane.  Valerie noted we will be scheduling the next call soon. 

5.  Open discussion

Decisions

Action Items

• Valerie will follow up with Tulane and draft XMl representing their framework.
• Valerie will work to create a voice over powerpoint providing an overview of the specification.
• Valerie will follow up with Jason on the XHTML datatype used for SupportingInformation.