High-risk students talking low-stakes assessments: Do the data reflect ability or effort?

Deborah Reed; S. Ryan  Hall; David E. Houchins

doi:10.21697/fp.2024.2.1.2

Published: 2024-12-31

Vol. 14 No. 2.1 (2024)

High-risk students talking low-stakes assessments: Do the data reflect ability or effort?

Deborah Reed

, S. Ryan Hall , David E. Houchins

Pedagogical Forum

Section: Topic

https://doi.org/10.21697/fp.2024.2.1.2

Abstract

This exploratory study examined whether test-taking effort (TTE) might be a concern in monthly low-stakes testing of juvenile offenders’ (n = 50) reading abilities. Among the graphs of 10 randomly selected students’ scores, 6 showed large fluctuations in performance from administration to administration, and another 2 showed precipitous declines across time. For the full sample, most of the average changes in scores from month-to-month far exceeded the standard error of measurement and equated to a 1- to 3-grade-level difference in how students’ reading performance could be interpreted. These changes could be positive or negative and varied within and across students. Most of the average testing times were below the expected minimum of 5 min, but total testing time generally was not correlated with scores. Given the response validity concerns, recommendations are made for supporting TTE.

Keywords:

test-taking effort, low-stakes tests, reading, juvenile justice

Download files

pdf

Citation rules

ACM
ACS
APA
ABNT
Chicago
Harvard
IEEE
MLA
Turabian
Vancouver
Download Citation
Endnote/Zotero/Mendeley (RIS)
BibTeX

Reed, D., Hall, S. R., & Houchins, D. E. (2024). High-risk students talking low-stakes assessments: Do the data reflect ability or effort? . Pedagogical Forum, 14(2.1), 17–31. https://doi.org/10.21697/fp.2024.2.1.2

Cited by / Share

Licence

This work is licensed under a Creative Commons Attribution-NoDerivatives 4.0 International License.

References

Charmtech Labs. (2020). Capti Assess with ETS ReadBasix. https://www.captivoice.com
Google Scholar

Christ, T. J., Monaghen, B. D., Zopluoglu, C., & Van Norman, E. R. (2013). Curriculum-based measurement of oral reading: Evaluation of growth estimates derived with pre–post assessment methods. Assessment for Effective Intervention, 38(3), 139-153. https://doi.org/10.1177/1534508412456417 DOI: https://doi.org/10.1177/1534508412456417
Google Scholar

Colón, E. P., & Kranzler, J. H. (2006). Effect of instructions on curriculum-based measurement of reading. Journal of Psychoeducational Assessment, 24(4), 318–328. http://doi.org/10.1177/0734282906287830 DOI: https://doi.org/10.1177/0734282906287830
Google Scholar

Cummings, K. D., Biancarosa, G., Schaper, A., & Reed, D. K. (2014). Examiner error in curriculum-based measurement of oral reading. Journal of School Psychology, 52(4), 361-375. http://doi.org/10.1016/j.jsp.2014.05.007 DOI: https://doi.org/10.1016/j.jsp.2014.05.007
Google Scholar

Duckworth, A. L., Quinn, P. D., Lynam, D. R., Loeber, R., & Stouthamer-Loeber, M. (2011). Role of test motivation in intelligence testing. Proceedings of the National Academy of Sciences, 108(19), 7716–7720. https://doi.org/10.1073/pnas.1018601108 DOI: https://doi.org/10.1073/pnas.1018601108
Google Scholar

Erdodi, L. A., Abeare, C. A., Lichtenstein, J. D., Tyson, B. T., Kucharski, B., Zuccato, B. G., & Roth, R. M. (2017). Wechsler Adult Intelligence Scale-Fourth Edition (WAIS-IV) processing speed scores as measures of noncredible responding: The third generation of embedded performance validity indicators. Psychological Assessment, 29(2), 148–157. https://psycnet.apa.org/doi/10.1037/pas0000319 DOI: https://doi.org/10.1037/pas0000319
Google Scholar

Finney, S. J., Sundre, D. L., Swain, M. S., & Williams, L. M. (2016). The validity of value-added estimates from low-stakes testing contexts: The impact of change in test-taking motivation and test consequences. Educational Assessment, 21(1), 60–87. https://doi.org/10.1080/10627197.2015.1127753 DOI: https://doi.org/10.1080/10627197.2015.1127753
Google Scholar

Gagnon, J. C., Barber, B. R., VanLoan, C., & Leone, P. E. (2009). Juvenile correctional schools: Characteristics and approaches to curriculum. Education & Treatment of Children, 32(4), 673– 696. https://psycnet.apa.org/doi/10.1353/etc.0.0068 DOI: https://doi.org/10.1353/etc.0.0068
Google Scholar

Kilgus, S. P., Methe, S. A., Maggin, D. M., & Tomasula, J. L. (2014). Curriculum-based measurement of oral reading (R-CBM): A diagnostic test accuracy meta-analysis of evidence supporting use in universal screening. Journal of School Psychology, 52(4), 377-405. https://doi.org/10.1016/j.jsp.2014.06.002 DOI: https://doi.org/10.1016/j.jsp.2014.06.002
Google Scholar

Penk, C., & Schipolowski, S. (2015). Is it all about value? Bringing back the expectancy component to the assessment of test-taking motivation. Learning and Individual Differences, 42, 27–35. https://doi.org/10.1016/j.lindif.2015.08.002 DOI: https://doi.org/10.1016/j.lindif.2015.08.002
Google Scholar

Puzzanchera, C., Hockenberry, S., & Sickmund. (2022). Youth and the juvenile justice system: 2022 national report. National Center for Juvenile Justice.
Google Scholar

Reed, D. K. (2018). Education for U.S. youth in secure care: The sum of the parts is not whole. In D. Gallard, K. Evans, & J. Millington (Eds.), Children and their education in secure accommodation: Interdisciplinary perspectives of education, health, and youth justice (pp. 117-127). Routledge Books.
Google Scholar

Reed, D. K., & Petscher, Y. (2012). The influence of testing prompt and condition on middle school students’ retell performance. Reading Psychology, 33(6), 562-585. http://doi.org/10.1080/02702711.2011.557333 DOI: https://doi.org/10.1080/02702711.2011.557333
Google Scholar

Reed, D. K., Cummings, K. D., Schaper, A., *Lynn, D., & Biancarosa, G. (2019). Accuracy and reliability in identifying miscues during oral reading. Reading and Writing: An Interdisciplinary Journal, 32, 1009-1035. https://doi.org/10.1007/s11145-018-9899-5 DOI: https://doi.org/10.1007/s11145-018-9899-5
Google Scholar

Reed, D. K., & Wexler, J. (2014). “Our teachers…don’t give us no help, no nothin’”: Juvenile offenders’ perceptions of academic support. Residential Treatment for Children and Youth, 31(3), 188-218. http://doi.org/10.1080/0886571X.2014.943568 DOI: https://doi.org/10.1080/0886571X.2014.943568
Google Scholar

Reed, D. K., Wexler, J., & Vaughn, S. (2012). RTI for reading at the secondary level: Recommended literacy practices and remaining questions. Guilford Press.
Google Scholar

Rios, J. (2021). Improving test-taking effort in low-stakes group-based educational testing: A meta-analysis of interventions. Applied Measurement in Education, 34(2), 85-106. https://doi.org/10.1080/08957347.2021.1890741 DOI: https://doi.org/10.1080/08957347.2021.1890741
Google Scholar

Ritchey, K. D., Silverman, R. D., Schatschneider, C., & Speece, D. L. (2015). Prediction and stability of reading problems in middle childhood. Journal of Learning Disabilities, 48(3), 298-309. https://doi.org/10.1177/0022219413498116 DOI: https://doi.org/10.1177/0022219413498116
Google Scholar

Sabatini, J., Weeks, J., O’Reilly, T., Bruce, K., Steinburg, J., & Chao, S-F. (2019). SARA reading components tests, RISE forms: Technical adequacy and test design (3rd ed.). ETS Research Report 19-36. ETS. DOI: https://doi.org/10.1002/ets2.12269
Google Scholar

Sanders, S., Jolivette, K., & Harris, C. (2021). Improving the reading comprehension skills of systems-involved youth: A preliminary investigation of an underserved population. Learning Disabilities Research & Practice, 36(3), 201-212. https://doi.org/10.1111/ldrp.12254 DOI: https://doi.org/10.1111/ldrp.12254
Google Scholar

Silm, G., Pedaste, M., & Täht, K. (2020). The relationship between performance and test-taking effort when measured with self-report or time-based instruments: A meta-analytic review. Educational Research Review, 31(100335), 1-22. https://doi.org/10.1016/j.edurev.2020.100335 DOI: https://doi.org/10.1016/j.edurev.2020.100335
Google Scholar

Soland, J. (2018). The achievement gap or the engagement gap? Investigating the sensitivity of gaps estimates to test motivation. Applied Measurement in Education, 31(4), 312–323. https://doi.org/10.1080/08957347.2018.1495213 DOI: https://doi.org/10.1080/08957347.2018.1495213
Google Scholar

Van Norman, E. R., & Parker, D. C. (2016). An evaluation of the linearity of curriculum-based measurement of oral reading (CBM-R) progress monitoring data: Idiographic considerations. Learning Disabilities Research & Practice, 31(4), 199-207. https://doi.org/10.1111/ldrp.12108 DOI: https://doi.org/10.1111/ldrp.12108
Google Scholar

Wise, S. L., & DeMars, C. E. (2005). Examinee motivation in low-stakes assessment: Problems and potential solutions. Educational Assessment, 10(1), 1–18. https://doi.org/10.1207/s15326977ea1001_1 DOI: https://doi.org/10.1207/s15326977ea1001_1
Google Scholar

Wise, S. L., & DeMars, C. E. (2010). Examinee non-effort and the validity of program assessment results. Educational Assessment, 15(1), 27–41. https://doi.org/10.1080/10627191003673216 DOI: https://doi.org/10.1080/10627191003673216
Google Scholar

Deborah Reed

dreed20@utk.edu
https://orcid.org/0000-0003-0874-1412

Affiliation:

Tennessee Reading Research Center, University of Tennessee United States