IGNITE SESSIONS

Disruption in the Credentialing Space: Why Is It Happening and What Should We Do About it? Read more >

Disruption has invaded the credentialing space. MOOCs, nano-degrees, microcredentials, and badging—all unheard of a few years ago — have quickly become part of the credentialing vernacular. Most likely, there are more disruptions to come that cannot even be anticipated. This is the new environment in which professional credentialing programs must operate, and it is here to stay. How do those in the credentialing industry adapt and continue to thrive in this environment? In this Ignite session, the presenters will explore three critical questions: (1) Why are is there disruption in the credentialing space, (2) what opportunities and threats does disruption pose for professional credentialing programs, and (3) what should (or can) anyone do about it?

• Lenora Knapp, Knapp & Associates International, Inc.

Four Innovative Ways to Lead and Facilitate SME Groups
in the Assessment Industry Read more >

When developing assessments, focus groups composed of qualified subject matter experts (SMEs) are used very often to validate the assessment. While focus groups are often necessary, professional development opportunities on leading and facilitating groups are rarely offered or discussed in the assessment industry. Often, the industry hires technically savvy individuals who are expected to lead groups without any formal training on facilitating, leading, and managing any type of group.

In this ignite session, the presenter will provide the audience with a professional development opportunity to learn about four innovative ways to lead and facilitate SME groups. These four ways are based on the presenter's experience and knowledge of leading therapy groups while employed as a social therapist and on the presenter's many years of leading focus groups in the assessment industry. The presenter will discuss many of the focus groups typically used across the lifecycle of developing an assessment such as planning, job analysis, standard setting, and item development and audit meetings.

Finally, the presenter will explain and review the four innovative approaches using examples and case studies from the assessment industry. An outline of the basic four innovative approaches when leading and facilitating focus groups are: (1) manage expectations, (2) manage attention, (3) manage the dynamics of resistance, power, and fears, and (4) manage yourself as a leader/facilitator (e.g., emotions, posture, non-verbals, genuineness, disclosure, and humor).

• Manfred Straehle, Assessment, Education, and Research Experts

Rating Using Paired Comparisons Read more >

Rating is an important aspect that is crucial to the quality of assessment. Often, assessments are designed so that the rating is manageable, and this could result in a limitation of what and how constructs are assessed. It is also well known that rater effects could challenge the validity of assessment results due to lenience or stringency on the part of the rater or due to halo effects. An alternative to the rating of assessments is an evaluation based on paired comparisons. In this approach the only evaluation that a rater makes is to be given a pair of candidates and judge which candidate performed better.

Based on multiple comparisons, the test takers can be ranked and ordered based on their performance. It is also possible to grade performance based on this approach if some of aspects of performance are defined as standard of a typical performance level. Borders between performance levels can be defined based on the standard and the rank order of all individuals being evaluated. The psychometric theory behind this approach dates back to the first half of the 20th century but the application in assessment is relatively new. Quite a substantial amount of research has been done within the field of writing assessment, but the potential of this approach generalizes to a broader field of judgments in both performance assessment and in evaluation of non-cognitive traits.

In this ignite session, presenters will describe the evaluation using paired comparisons. Presenters will then discuss how a study using this approach can be set up both efficiently and effectively. The session will also explore practical considerations that could influence the applicability of this approach with different types of assessment and evaluation. The goal of this session is to provide both theoretical and practical insight in rating based on paired comparisons. Test professionals and those responsible for program design could get ideas for innovative assessments and alternative rating procedures. The general public will get a practical overview of the implications of using evaluation based on paired comparisons.

• Anton Beguin, CITO

Structured Blueprint Review: Setting the Foundation for Test Development Read more >

An essential element of any testing program is assuring the relevance and appropriateness of examination content. The starting point for content is a well-defined blueprint or a set of test specifications, aligned to the requirements for the credential offered. To meet that goal, the presenters of this session developed a structured exam blueprint review. The purpose of this session will be to illustrate how the structured blueprint review informs the many components of test development (e.g., item selection and writing new content) and provides validity evidence for a medical certification exam program.

The presenters gathered information on content relevance by surveying practitioners and asking them to rate blueprint content areas and associated tasks in terms of importance and frequency seen in practice. For example, respondents rated the overall frequency of a particular disease state (content area) and independently rated the importance of tasks related to that disease state (e.g., diagnosis, treatment, risk assessment). The exam committee used the resulting data to adjust the blueprint at a high level and to identify content areas of low relevance that should not be tested. This data was then used to inform the Automated Test Assembly for the next exam. Items in content areas with tasks that had low importance and low frequency ratings (i.e., low relevance) were not selected for the exam, and strong emphasis was placed on selecting items that had both high frequency and high importance ratings (i.e., high relevance). Some items in content seen less frequently but still important to know were also selected.

Critically, the presenters were able to avoid emphasizing exam content that was deemed less relevant to practitioners. They subsequently published the updated blueprint and included associated task relevance data in an effort to increase transparency and to better inform test takers about the exam. Going forward, exam committees will be able to evaluate items against both statistical performance and relevance rating criteria. By using ratings from practitioners, the presenters were able to more closely align the content of the exam with what examinees do in practice. This process sets a foundation for future validation arguments since the basis for validity in credentialing must start with identifying the knowledge and skills that are important for examinees to use in practice.

• Andrew Jones, American Board of Internal Medicine
• Rebecca Baranowski, American Board of Internal Medicine

The ABC's of Digital Badges Read more >

The use of digital badges to recognize various professional achievements such as certifications, awards, and other continuing education experiences has gained tremendous momentum over the past few years. However, many organizations are still unsure about what this technology is, how it works, and how it can be used to support their goals. This often prompts the question, “I have heard of digital badging, but how can it help my organization?”

This ignite session will explore the basics of the digital badging trend including:

1. how digital badges help organizations avoid the risk associated with issuing paper certificates,
2. how organizations can leverage digital badges to brand and market their programs, and
3. how digital badges provide a huge convenience for professionals. Attendees will leave this session better educated about the trend and armed with the considerations necessary to potentially leverage digital badge technology for their own organizations.

• Ginger Malin, BadgeCert

Text Mining for Assessment of Writing and Social Positioning Read more >

Computational models of discourse reveal evidence of cognitive and social processes and competencies that are valuable both for assessment and for triggering real-time support for learning. In this session, the presenter will offer a brief overview of the field of text mining as it is represented in application areas such as assessment of writing and of social positioning in collaborative discourse. Then the presenter will zoom in on the topic of analysis of role-taking behavior in collaborative groups. In these groups, computational models of social interaction in textual form reveal layer upon layer of insight about student orientation towards one another and their joint endeavors.

In this session, attendees will observe successful and unsuccessful teams and notice unexpected behavior profiles that either do not align with assigned roles or do not align with expectations of effective role taking. In this session, the presenter will introduce a novel role-based behavior-modeling approach that enables identification of behavior profiles that are associated with specific success measures. This enables dynamic support for students to reflect on their team’s performance and guidance to opportunities for improved engagement.

• Carolyn Penstein Rose, Carnegie Mellon University