2021 Graduate Student Research Award Presentations
Please join us for our Graduate Student Research Award presentations. This is the third year in a row we have offered this program, and we are excited to announce we were able to select two recipients this year.
The theme of the 2021 ATP Graduate Student Research Award was Innovations in Testing: Disruptive Technology in Large-Scale Assessments. Graduate students conducting research regarding how technology may be used to enhance or support large-scale assessments were encouraged to apply.
Please review the two presentations from our two winning graduate students below and join us on Tuesday, April 27, 2021 2:00 PM to 3:00 PM to hear their presentations live.
Conducting DIF Analysis Based on the Summarized Datasets
Differential item functioning (DIF) analysis is a crucial component in test construction and evaluation. It is a widely used method to provide validity evidence that assessments perform in the same way for different groups of people. Researchers have developed many different methods for determining whether DIF is present. Almost all methods are using examinees’ raw item responses to perform the analysis. With current computing power, it is not a challenge to run DIF analysis for a very large dataset. But saving and transforming the large dataset is still time and space consuming. This study proposed a simple way to summarize a large raw item response file. It put examinees’ proficiency estimates into bins to get the frequency table of correct responses. This study performed a few DIF analyses for an operational widely used assessment on both summarized and raw data files. For the summarized dataset, the frequency tables were used as the dependent variable of logistic regression instead of raw item scores. It showed DIF analysis based on summarized datasets that provided equivalent results to raw data files. This method allows more flexibility to perform DIF research for large-scale assessments.
Speaker: Siyu Wan, Ph.D student at University of Massachusetts Amherst, majored in Research, Educational Measurement, & Psychometrics (REMP)
Using Hypothesis Test Method to Determine Reportable Subscores in the Medical Certification Exam
In practice, it is meaningful to obtain and understand the examinees’ performance of each specified sub-domain within the overall test. Meanwhile, there is always a need for feedback on subtest performance. The purpose of this study is to explore the psychometric evidence to determine whether there are reportable subscores within the medical certification exam to best communicate useful subtest feedback to examinees by adopting Sinharay’s (2018) hypothesis method of subscores. The analysis has been conducted across 4 classification schemas on two forms of the 2017 medical certification exam. Across each of the four classification schemas and both forms of the exam, there were no subscores that provided added value which is not unexpected given that this exam is constructed to be unidimensional. However, these results do provide some additional support for reporting augmented subscores if the goal of reporting subscores is to provide examinees with a category structure for feedback. This study illustrates a more data-driven and psychometrically robust approach to score reporting.
Speaker: Chen Qiu, M.S. & M.A., Department of Educational, School, and Counseling Psychology, University of Kentucky
Thank You to Our Sponsors
The American Board of Pediatrics (ABP) certifies general pediatricians and pediatric subspecialists based on standards of excellence that lead to high quality health care during infancy, childhood, adolescence, and the transition into adulthood. The ABP certification provides assurance to the public that a general pediatrician or pediatric subspecialist has successfully completed accredited training and fulfills the continuous evaluation requirements.
NCCPA is the only certifying organization for PAs in the U.S, and all U.S. states, the District of Columbia and the U.S. territories rely on NCCPA certification as one of the criteria for initial licensure or regulation of PAs. More than 175,000 PAs have been certified by NCCPA since 1975.