GRADUATE STUDENT RESEARCH AWARD SESSION

Graduate Student Research Award Presentation 1

Using Process Data for Motivation Filtering in Remote, Low-Stakes Assessment

In remote, low-stakes assessments, examinees’ true performance may be confounded with low motivation. Several analytic methods exist to remove the effect of low effort from observed data by filtering examinees who exhibit low effort based on total test-taking time. Other methods filter responses identified as non-effortful using the effort-moderated IRT model. Both types of methods require the selection of a threshold to separate effortful and non-effortful responses. For comparison, six combinations of methods (examinee-based filtering methods or response-based filtering methods) and thresholds (conservative or liberal) were applied to estimate the score means of examinees assessed remotely in 2021. The score means, estimated six ways, were compared to score means of examinees assessed in person in 2016-2020. The results support the use of motivation filtering regardless of which method is used. However, the selection of specific methods and thresholds may depend on the assessment context.

Speaker: Sarah Alahmadi, James Madison University

Graduate Student Research Award Presentation 2

A Comparison of Quantitative Methods for Detecting Test Cheating

Test cheating has a great impact on the test validity of high-stake assessments and is a significant source of concern for many testing companies. While many studies develop statistical methods to identify fraud in testing, the comparison of their applications to empirical data is rarely discussed. The current study was designed to apply four quantitative methods to exam results data and compare their ability to identify known cheaters and exposed items. Preliminary results demonstrated some implications and limitations for current psychometric methods. Further analysis to improve the method performance is discussed.

Speaker: Jiaying Xiao, University of Washington

Thank You to Our Sponsors