Peas in a Pod Discussions

Would you like to participate in face-to-face conversations with fellow attendees who share common interests? Do you prefer direct engagement and exploration of ideas with other industry professionals?

If so, the Peas in a Pod Discussions are perfect for you! These are informal discussions without presentations -- just an open discussion on exciting topics. Check out our Peas in a Pod Discussions below.

A Credentialing Birth Story - How Two Programs Launched their Professions' First Certification

Facilitators: Wendy Davis, Postpartum Support International; Amy McKee, HumRRO

Follow two sponsors along their recent journeys in conceiving of, vetting, shaping, designing, and launching a brand new certification program. This session will cover almost all aspects of the business from member input to exam development to candidate communications to exam delivery to infrastructure and program management. Learn how they funded their ideas, sought appropriate expert support, identified staff and resources within their organizations, and found marketing partners to grow their fledgling program. Of course, they will also provide insights to lessons learned the hard way, and cautionary tales of paths to avoid. Whether you're implementing a certification program from scratch with no experience or you're on track to launch a new exam within an existing program, this session will be inspirational and educational!


Engaging Stakeholders in a Credentialing Exam Restructure

Facilitators: Brooke Midkiff, National Board of Examiners in Optometry; Brianne Hobbes, National Board of Examiners in Optometry

This session will provide discussion of the work to engage critical stakeholder groups in the early stages of the restructure process of a clinical skills examination. A forced-choice survey was designed and delivered to stakeholders using innovative survey design features to boost response rates. A task force comprised of stakeholders was formed to review the analysis of the survey results, determine the broad categories of skills and knowledge to be assessed by the restructured exam, synthesize the ways in which clinical skills would be embedded within patient scenarios alongside clinical decision-making, critical thinking, and patient documentation. This session will focus on the challenges faced in reaching out to stakeholders, the innovative survey design features used, and the positive outcomes from engaging stakeholders early in the process.


Global Technical Certifications: From Startup to Build Out

Facilitators: Katie Schmidt, Splunk; Middy Davis, Splunk

Building an exam-based certification program in the technology sector is no small task, but it is a critical one that can breathe significant life into the business, evangelize a service or product, and build the brand around the globe. There are great challenges and valuable lessons to be learned when it comes to taking content built in a startup environment and shaping it into a global certification program where candidates achieve professional validation and customers benefit greatly. We will discuss the many components required to build, launch, and maintain a fresh certification program, along with lessons learned and steps for continuous improvement. Technology is only as good as the people who use it, and exam-based certification takes center stage from this perspective.


Innovative Strategic Solutions to Low-Volume Credential Programs - A Case Studies and Business Guidance

Facilitators: Marc Lukasik, KeyW, a wholly owned subsidiary of Jacobs; Dawn Gibas, American Society of Agronomy

Adherence to industry best practices can be challenging in areas for which there is little prescriptive guidance in the literature. When faced with such challenges, exam developers must supplement interpretation of industry guidelines with additional sources. Several less salient areas in the field of test development will be addressed specific to low-volume assessments.

Case studies will be presented highlighting the development of a several certification efforts.

Strategic business guidance pertinent to low-volume testing will be offered. Presenters will attend to financials (budgeting and pricing), use of vendors to perform certain functions instead of staff, marketing techniques, choice of acceptable volumes, alignment to the mission and vision of the program and organization, accreditation concerns, and staffing challenges.


Machine Learning Algorithms in the Time of Fairness and Transparency

Facilitator: Brandon Ferrell, Hogan Assessment Systems

Are artificial intelligence and machine learning algorithms inherently biased against minority groups? How are test publishers handling calls for algorithm transparency? This Peas-in-a-Pod discussion will give people the opportunity to discuss their experiences implementing AI/ML algorithms in the testing industry, including how they handle issues concerning bias/fairness and potential regulation (right-to-explanation/algorithm transparency).


Social Media and Open Source Intelligence Tools – How to use them in an investigation

Facilitators: Brent Morris, Cisco; Bryan Friess, Pearson VUE

Personal data and social media are continuously evolving and becoming more and more complex. Social media and Open Source Intelligence Tools (OSINT) are becoming more widely used by test publishing and sponsor organizations as they conduct test security and integrity investigations. In this session, exam security experts will facilitate a discussion to highlight where different organizations are trending and how to acquire online information.

Be prepared to discuss methods and techniques used by security teams that focus on search terms, professional databases, social media platforms, and research tools. These methods and techniques are intended for security professionals and those responsible for overall program integrity and company branding.

You will learn how to conduct these types of investigations in an ever-changing environment for the governance of personal data and how to utilize these strategies to drive a variety of outcomes that will support programs of any size.


Standard Setting for Performance Examinations: Practical Experiences and Lessons Learned

Facilitators: Fang Tian, Medical Council of Canada; Andrea Gotzmann, Medical Council of Canada

Standard setting for performance assessments is complex, due to their multi-dimensional nature. Examinee-centered methods (e.g., Contrasting Groups or Borderline Group) are most appropriate where expert judges review the performance of a group of examinees and provide global judgments as to the adequate level of performance.

In this session, we will share our organization’s practical experiences and lessons learned in setting standard(s) for performance examinations in a medical licensure context. Discussions will include: (1) selection of methods; (2) selection of judges; (3) sampling of examinee performances; (4) performance level definitions; (5) training of judges; (6) collecting judgment data; (7) technical issues in calculating cut score(s); and (8) gathering procedural, internal, and external evidence to validate the resulting standard(s). Participants will be encouraged to share their own thoughts and experiences in setting standards for performance examinations.


The Big Picture - Understanding and Leveraging Your Program's Online Trends

Facilitator: Jen Baldwin, Caveon Test Security

During this discussion you will:

  • Become familiar with all areas of the internet; know the various channels of where digital information is exchanged (websites, blogs, social media, forums, chatrooms, and more).
  • Develop an understanding of the various ways proprietary information is exchanged.
  • Know how and when to react to an exam’s exposure on the internet.
  • Learn which important strategies you should implement if your content cannot be removed.
  • Know where to draw the line between fair use and infringement.
  • Properly prepare Digital Millennium Copyright Act (DMCA) notifications.
  • Understand the challenges and successes of monitoring internationally.


View the Schedule at a Glance
Special thanks to our
Platinum Sponsors