FEATURED SPEAKERS

Collaborative Assessments

This session will feature a discussion of the development, design, and delivery of assessments for collaborative problem-solving (CPS) skills. Assessments for CPS tasks have become increasingly important. The session will primarily focus on three topics: what makes a good design for a collaborative assessment, what defines a good platform for a collaborative assessment, and how these can be applied to language testing. “Collaborative discussions” as part of language assessments addresses one of the needs highlighted in college and career readiness standards. The goal of this session will be to engage attendees in a discussion of the advantages and challenges surrounding collaborative assessments.

Alina von Davier, Educational Testing Service
Cindy Hmelo-Silver, Indiana University
Mikyung K. Wolf, Educational Testing Service

Evidence-Based Assessment

The trend in psychological and educational assessment has increasingly been to follow other areas of professional practice, with a reasonable demand for evidence that supports practice. Evidence-based medicine and evidence-based psychological interventions have moved the respective fields forward by requiring accountability in practice. Physicians and psychotherapists are increasingly expected to document the efficacy of their treatments based on research and evidence.

Education has moved in this direction as well, with an increased emphasis on educational standards and core curricula. Psychological and educational assessment has incrementally progressed along this path as well, with emphases on “direct assessment,” “assessment to intervention linkages,” “authentic assessment,” and eschewing construct-based assessments in favor of a direct, developmental, or behavioral approaches. This session will examine this trend in psychoeducational assessment and implications for future assessment paradigms.

Bruce Bracken, The College of William and Mary

Evolution and Revolution in Individual Clinical Intelligence Testing

The field of clinical intelligence testing has been impacted by major developments in intelligence research and theory (CHC theory), related research and theory in cognitive psychology (information processing and complexity sciences) and the brain sciences (neural efficiency and brain networks), and psychometric methods (e.g., planned incomplete data collection and bootstrap resampling methods). This session will provide an overview and examples of these developments and how they are impacting clinical intelligence test development, data gathering and analysis, and test interpretation.

Kevin McGrew, Institute for Applied Psychometrics

Text Mining for Assessment of Writing and Social Positioning

Computational models of discourse reveal evidence of cognitive and social processes and competencies that are valuable both for assessment and for triggering real-time support for learning. In this session, the presenter will offer a brief overview of the field of text mining as it is represented in application areas such as assessment of writing and of social positioning in collaborative discourse. Then the presenter will zoom in on the topic of analysis of role-taking behavior in collaborative groups. In these groups, computational models of social interaction in textual form reveal layer upon layer of insight about student orientation towards one another and their joint endeavors.

In this session, attendees will observe successful and unsuccessful teams and notice unexpected behavior profiles that either do not align with assigned roles or do not align with expectations of effective role taking. In this session, the presenter will introduce a novel role-based behavior-modeling approach that enables identification of behavior profiles that are associated with specific success measures. This enables dynamic support for students to reflect on their team’s performance and guidance to opportunities for improved engagement.

Carolyn Penstein Rose, Carnegie Mellon University

Theory and Practice of Principled Assessment in Licensure

How can test professionals ensure that the items on their tests do what they need them to do? In recent years, principled assessment frameworks such as Assessment Engineering (AE) have been defining a methodology that tightly links a construct to the items on the test and the scores received from them. In this session, Ric Luecht, Ph.D. will provide an overview of AE and its tools (i.e., construct maps, task models, and templates). Ric will discuss the theoretical basics of AE and its potential to improve all parts of the test life cycle, including such innovations as automated item generation. Then, Joshua Stopek, CPA, will discuss the AICPA’s experience in beginning to implement the tenets of AE for its current practice analysis. Josh will explain what they have done; how AE has changed the conversation with its stakeholders; and the practical impacts, improvements, obstacles, and issues from an operational perspective. To do so, the presenters will provide samples from the proposed test specifications and show how they relate to simulation development via templates.

Joshua Stopek, AICPA
Ric Luecht, University of North Carolina, Greensboro

Support Personalized Learning by Accelerating the Transition to e-Assessment via a Highly Interoperable Assessment Ecosystem

Over the last several years, there has been aggressive movement in the US and around the world toward digital assessment that is being spurred by state and national programs to better measure student learning. Building an online assessment system that meets the high-stakes, summative testing requirements across multiple states or regions is a massive undertaking. Critical to the success of such an undertaking are open technology standards enabling interoperability of systems and platforms to support the wide variety of student accessibility needs and preferences in each state.

IMS Global Learning Consortium, a non-profit organization, has been leading a worldwide collaboration to create and drive adoption of the IMS APIP and QTI standards to accelerate the transition to e-Assessment. IMS Global has seen great progress in the market over the last three years because of industry adoption of these standards, as demonstrated by PARCC delivering millions of online tests to students across 11 states in its first year of online testing. With the accelerated adoption of e-Assessment around the world, it is critical that the assessment industry advance the usability and end-to-end interoperability of e-Assessment solutions by driving adoption of IMS open standards to improve content portability, make outcomes data interoperable, enable deeper integration with assistive technologies, and remove barriers to accelerate industry adoption of assessment standards.

In this session, attendees will learn what IMS members are doing now to shape the future of assessment interoperability with the convergence of the QTI and APIP specifications into aQTI v3. The goal of aQTI v3 is to establish a highly interoperable end-to-end assessment architecture to enable growth of e-Assessment content, applications, and innovation. These standards are continually evolving and presenters will encourage attendees’ input during this session to help shape the future of e-Assessment.

Sandra DeCastro, IMS Global Learning Consortium
Rob Abel, IMS Global Learning
Marc Oswald, Open Assessment Technologies
Chloe Torres, PARCC, Inc.
Gary Driscoll, ETS

Bad News and Good News About Global Workforce Skills Credentialing...News and a Report

In this session, presenters will travel the planet, from the USA to India, providing some real life success stories of how developing and assessing people for the correct demonstrated workforce skill credentials is paying off in changed lives and changed businesses. Attendees will see firsthand through real-life examples how a set of newly verified job skills can impact and change lives. Presenters will discuss applied best practices with global examples. They will cover both hard and soft skills credentials in the cases discussed, and attendees will leave with new ideas and insights.

Dan Rinn, NTT Training
Madan Padaki, Head Held High

Highlighting Proctoring Best Practices

“Proctoring Best Practices” is a publication developed jointly by the National College Testing Association and the Association of Test Publishers Security Committee. Compiled by experts, this document captures proctoring best practices in paper-based, computer-based, and online testing contexts. It can serve as an incredibly helpful guide for test sponsors and publishers when drafting their test administration policies or evaluating their proctoring practices. In this session, testing experts will share the industry’s current thinking around proctor responsibilities and best practices. They will also address how proctor responsibilities compare and differ across delivery methods.

James Wollack, University of Wisconsin-Madison
Joe Brutsche, Pearson VUE
Jarret Dyer, College of DuPage
Rory McCorkle, PSI Services
Steve Saladin, University of Idaho
Rachel Schoenig, ACT, Inc.

Pulse of Security in Testing: ATP Survey of Security Practices

Security is a critical consideration for any credentialing organization. This session will present the results of the 2015 survey of security practices, conducted by the ATP Security Committee. These results include responses from credentialing organizations and vendors and show the practices being used by credentialing organizations to address security prevention, detection, enforcement, mitigation, and litigation. The presenters will also discuss how vendors and credentialing bodies see the division of security responsibilities, as well as perceptions related to the effectiveness of various security activities. Finally, the presenters will make recommendations for how certification examination sponsors can address these gaps.

Chuck Friedman, Professional Examination Services
Benjamin Hunter, Pharmacy Technician Certification Board
Rory McCorkle, PSI Services

Talent Analytics in the Reputation Economy

Over two decades have passed since McKinsey first introduced the notion of a war for talent, which highlighted the importance of talent identification and retention as key determinants of organizational effectiveness. Despite much research and data to support the premise, the war for talent has unfortunately become the war on talent: 2/3 of the global workforce is disengaged, 70% of employees are looking for better jobs, and self-employment and entrepreneurship rates are increasing steadily, mostly because people are fed up with incompetent management. This session will examine the clear opportunities for the assessment industry that this context creates, as demand for accurate potential metrics continues to soar.

Tomas Chamorro-Premuzic, Hogan Assessment Systems

The Complete Equation for Workforce Skills Credentialing

The global need for workforce skills credentials continues to grow. However, for medium- to high-stakes summative credentials, the integrity of the assessment tool is not the complete equation for valid, reliable results. This session will review efforts by the Security subcommittee of the new Workforce Skills Credentialing Division to develop a common framework that can be used to align the expectations of those involved with workforce credentialing (e.g., test publishers, examinees, and employers) and make transparent the level of security and trustworthiness of a credential within that framework. In short, the Security subcommittee is helping the industry provide the missing test security variable to the workforce skills credentialing equation. Attendees will learn about the potential benefits, challenges, and outcomes associated with this work.

Rachel Schoenig, ACT Inc.
Jamie Mulkey, Caveon
Jennifer Geraets, ACT Inc.

The Transition from Computer to Mobile and Tablet Based Testing: Misinterpretations, Implications and Pitfalls

The increasing use of smartphones and tablets seems to urge test providers to make their tests accessible via these devices. The question does not seem to be whether test providers are going to use these mobile devices for testing. There seems to exist, however, some controversy about the purposes and the conditions of the meaningful use of these devices. In this session, presenters will address some questions on testing via mobile devices.

Is there a difference in data quality gathered by computers and mobile devices, and if so, what are the consequences? If this is true, can selection decisions be based on data gathered via mobile devices? Should they be complemented? Should they not be used for selection and rather for other purposes such as realistic job previews?

What criteria should be the basis for the decision of whether to include mobile devices in the selection process? A first criterion might be the constructs to be measured. Computers might be more appropriate for some constructs, mobile devices for others. A second criterion might be practicability. When confronted with a significant number of candidates who have a tablet or a smartphone but no computer, it might be a good idea to provide the test on a mobile device. A third criterion might be the stage of the selection process. Maybe mobile devices should be used solely in the beginning of the process.

Do mobile devices enable the measurement of different psychological constructs and skills from regular computers? What is the difference between mobile devices and computers in relation to the constructs being measured? For instance, is the fact that it is easier to confront the candidates with more text relevant for the measurement of constructs, or is this only a minor difference?

Is the impact of the shift towards mobile devices as fundamental as the shift from paper-and-pencil testing to computer testing? The shift toward computers has confronted the testing industry with important challenges. The tests had to be adapted to be used on computers, for example. Similarly, test providers can ask themselves how their tools need to be adapted to mobile devices. Test taking on the computer has also brought about new, interesting possibilities. Adaptive testing and the use of videos, for instance, were not possible when using paper-and-pencil tests. What opportunities does the shift towards mobile devices offer?

Rostislav Benak, Assessment Systems International
David Barrett, cut-e Group
Michael Tack, BDO Belgium
Robert McHenry, OPP

The World Out There - HR Management and Tools in Emerging Countries

Testing and HR management has been present in the developed world for decades. Groups of emerging countries around the globe like Central Eastern Europe, Southeast Asia, or West Africa became involved at later stages, but caught up quickly. In countries where a regular phone line was a luxury, mobile phones achieved a much quicker and larger market penetration than in older economies. Is this analogy right for the testing industry as well? Are methods that are valid for the US and the UK fully applicable in developing economies? Up to what point are they are replicable, and what should be customized to local needs? Are the norms and validity standards showing extremely different outcomes?

Are business leaders requiring professionals to speak a different “language”? How educated is the market, and are the expectations different? This session will focus on countries in which paper-and-pencil testing has quickly become obsolete in most cases and in which business standards might differ from what other countries experience on a day-to-day basis. This session will be presented by Assessment Systems International and will focus on emerging markets. The presenters will provide interesting examples to showcase how it is done.

Rostislav Benak, Assessment Systems International

Which Test is Best? Exploring New Ways to Assess Continuing Competence

Going to a secure testing center and spending long hours answering tough questions is a universally dreaded event, even if it happens only once every few years. A more challenging and more important issue to address than individual convenience and comfort, though, is whether these examinations are the most effective tools for evaluating continuing competency. This is just one question that the American Board of Pediatrics discussed in 2015 at its Future of Testing Conference. This gathering brought together pediatricians, medical education experts, testing experts, and representatives from other key certification organizations to explore innovative ways that can be used to measure knowledge and competency.

Increasingly, physicians have raised concerns about the maintenance of certification examinations. Some consider leaving their practice and patients to go to a testing center to be burdensome, particularly if they practice in a rural setting. Others question the relevance of the closed-book examination format when doctors today use technology and tools to inform the practice of medicine. Doctors today can quickly access the medical knowledge necessary to make clinical decisions rather than relying on memory. These advancements have prompted many physicians to advocate for an examination that is both more relevant to their practice and that provides a learning opportunity.

This session will present the general outcomes of this conference. Particular focus will be placed on a new model of continuous assessment that the American Board of Anesthesiology presented. Adopting this model would ultimately eliminate the need for a separate examination, resulting in a philosophical shift in the link between high-stakes testing and the demonstration of continuing competency. This model also represents the next generation of assessment and continuous professional development by combining technology with real-time feedback in an environment that allows for ongoing assessment and learning through smartphones and tablets. This testing experience becomes both more convenient and relevant to real-life situations, while still allowing summative decision making. Both medical boards will discuss their experiences with their pilot studies as variations of the model are implemented.

Ann Harman, American Board of Anesthesiology
Linda Althouse, American Board of Pediatrics

Walking the Fine Line Between Sufficient and Excessive Testing

The popular media has recently criticized the testing industry for testing that has been viewed as excessive. Although issues surrounding frequency and the fine line between sufficient and excessive testing may vary across practice areas, test publishers face many common challenges in sufficiently testing populations of interest. This session will have three goals. First, presenters will use real-world examples to discuss sufficient versus excessive testing in their areas. Second, they will compare and contrast these issues across practice areas. Finally, they will demonstrate how different practice areas can work with and learn from one another to mutually benefit all parties.

Blaine Gaddis, Hogan Assessment Systems
Kimberly Nei, Hogan Assessment Systems
Rory McCorkle, PSI Services, LLC
Jesse Markow, University of Wisconsin - Madison
Mark Ledbetter, Houghton Mifflin Harcourt Assessment Studio
Dan Rinn, NTT Training