Assessing computational thinking skills of science and mathematics upper-secondary school students

Eli Rohaeti 1, Nur Huda 1 *
More Detail
1 Universitas Negeri Yogyakarta, Sleman, Yogyakarta, INDONESIA
* Corresponding Author
EUR J SCI MATH ED, Volume 13, Issue 4, pp. 289-303. https://doi.org/10.30935/scimath/17248
Published: 08 October 2025
OPEN ACCESS   33 Views   9 Downloads
Download Full Text (PDF)

ABSTRACT

Computational thinking (CT) is a thinking skill developed and integrated into curricula worldwide in recent years. However, limited assessment is one of the challenges in integrating CT skills into the educational curriculum of developing countries such as Indonesia. This study aimed to develop and validate a CT assessment instrument tailored for upper-secondary school students majoring in science and mathematics in Indonesia. The cross-cultural assessment adaptation method was adopted, comprising six stages: translation, synthesis, back-translation, expert committee review, pretesting, and research audit. Twelve experts were involved in the content validation stage to assess the feasibility of the instrument adapted in Indonesia. The validation process was followed by a pilot test with 501 upper-secondary students majoring in science and mathematics (220 female and 281 male). The data collected were analyzed using the Rasch model measurement. The findings showed that all adapted items met the fit based on the Rasch model measurement, except one spatial question item. The instrument demonstrated high item reliability, although person reliability was relatively low, indicating variation in student responses. The average upper-secondary school students majoring in science have good CT skills. Based on the differential item function value, there are two gender-biased items and four age-biased items. This study hopes to contribute to the literature on CT assessment by providing references and alternative tests for researchers and teachers to use in assessing CT in upper-secondary school students.
Keywords: assessment, computational thinking, Rasch model, science and mathematics student, test adaptation

CITATION

Rohaeti, E., & Huda, N. (2025). Assessing computational thinking skills of science and mathematics upper-secondary school students. European Journal of Science and Mathematics Education, 13(4), 289-303. https://doi.org/10.30935/scimath/17248

REFERENCES

  • Adams, C., Cutumisu, M., Yuen, C., Hackman, Lu, C., & Samuel, M. (2019). Callysto computational thinking test (CCTt) teacher version. Callysto. https://callysto.ca/computational-thinking-tests/callysto-computational-thinking-test-cctt-teacher-version/Resource_Callysto-CTt_Instrument_Teacher_version-1.pdf
  • Aiken, L. R. (1980). Content validity and reliability of single items or questionnaires. Educational and Psychological Measurement, 40(4), 955–959. https://doi.org/10.1177/001316448004000419
  • Andrich, D. (1988). Rasch models for measurement. SAGE. https://doi.org/10.4135/9781412985598
  • Andrich, D., & Styles, I. (2004). Final report on the psychometric analysis of the early development instrument (EDI) using the Rasch model. AEDI. https://api.semanticscholar.org/CorpusID:141656000
  • Angeli, C., & Giannakos, M. (2020). Computational thinking education: Issues and challenges. Computers in Human Behavior, 105, Article 106185. https://doi.org/10.1016/j.chb.2019.106185
  • Beaton, D. E., Bombardier, C., Guillemin, F., & Ferraz, M. B. (2000). Guidelines for the process of cross-cultural adaptation of self-report measures. Spine, 25(24), 3186–3191. https://doi.org/10.1097/00007632-200012150-00014
  • Bellettini, C., Lonati, V., Malchiodi, D., Monga, M., Morpurgo, A., & Torelli, M. (2015). How challenging are Bebras tasks?: An IRT analysis based on the performance of Italian students. In Proceedings of the 2015 ACM Conference on Innovation and Technology in Computer Science Education (pp. 27–32). ACM. https://doi.org/10.1145/2729094.2742603
  • Bond, T. G., & Fox, C. M. (2007). Applying the Rasch model: Fundamental measurement in the human sciences. Lawrence Erlbaum Associates Publishers.
  • Boone, W. J. (2016). Rasch analysis for instrument development: Why, when, and how? CBE—Life Sciences Education, 15(4), Article rm4. https://doi.org/10.1187/cbe.16-04-0148
  • Boone, W. J., Staver, J. R., & Yale, M. S. (2014). Rasch analysis in the human sciences. Springer. https://doi.org/10.1007/978-94-007-6857-4
  • Cansu, F. K., & Cansu, S. K. (2019). An overview of computational thinking. International Journal of Computer Science Education in Schools, 3(1), 17–30. https://doi.org/10.21585/ijcses.v3i1.53
  • Cantlon, J. F., Becker, K. T., & DeLong, C. M. (2024). Computational thinking during a short, authentic, interdisciplinary STEM experience for elementary students. Journal for STEM Education Research, 7, 425–443. https://doi.org/10.1007/s41979-024-00117-0
  • Chagas, D., & Furtado, E. (2019). Computational thinking in basic education in a developing country perspective. In A. Visvizi, & M. D. Lytras (Eds.), Research & innovation forum 2019 (pp. 135–150). Springer. https://doi.org/10.1007/978-3-030-30809-4_14
  • Chytas, C., Van Borkulo, S. P., Drijvers, P., Barendsen, E., & Tolboom, J. L. J. (2024). Computational thinking in secondary mathematics education with GeoGebra: Insights from an intervention in calculus lessons. Digital Experiences in Mathematics Education, 10, 228–259. https://doi.org/10.1007/s40751-024-00141-0
  • Cutumisu, M., Adams, C., Glanfield, F., Yuen, C., & Lu, C. (2022). Using structural equation modeling to examine the relationship between preservice teachers’ computational thinking attitudes and skills. IEEE Transactions on Education, 65(2), 177–183. https://doi.org/10.1109/TE.2021.3105938
  • Cutumisu, M., Adams, C., Yuen, C., Hackman, L., Lu, C., & Samuel, M. (2019a). Callysto computational thinking test (CCTt) student version. Callysto. https://callysto.ca/computational-thinking-tests/callysto-computational-thinking-test-cctt-student-version/Resource_Callysto-CTt_Instrument_Student_version-1.pdf
  • Cutumisu, M., Adams, C., Yuen, C., Hackman, M., Lu, C., & Samuel, M. (2019b). Computational thinking test. Callysto. https://callysto.ca/computational-thinking-tests/
  • Dagiene, V., & Stupuriene, G. (2016). Informatics concepts and computational thinking in K-12 education: A Lithuanian perspective. Journal of Information Processing, 24(4), 732–739. https://doi.org/10.2197/ipsjjip.24.732
  • De Jong, I., & Jeuring, J. (2022). Developing a self-efficacy scale for computational thinking (CT-SES). In Proceedings of the 22nd Koli Calling International Conference on Computing Education Research (pp. 1–2). https://doi.org/10.1145/3564721.3565954
  • El-Hamamsy, L., Zapata-Cáceres, M., Martín-Barroso, E., Mondada, F., Zufferey, J. D., Bruno, B., & Román-González, M. (2025). The competent computational thinking test (cCTt): A valid, reliable and gender-fair test for longitudinal CT studies in grades 3–6. Technology, Knowledge and Learning. https://doi.org/10.1007/s10758-024-09777-8
  • Ertugrul-Akyol, B. (2019). Development of computational thinking scale: Validity and reliability study. International Journal of Educational Methodology, 5(3), 421–432. https://doi.org/10.12973/ijem.5.3.421
  • ESTE, & CSTA. (2011). Operational definition of computational thinking for K-12 education. ISTE. https://cdn.iste.org/www-root/Computational_Thinking_Operational_Definition_ISTE.pdf
  • Ezeamuzie, N. O., & Leung, J. S. C. (2022). Computational thinking through an empirical lens: A systematic review of literature. Journal of Educational Computing Research, 60(2), 481–511. https://doi.org/10.1177/07356331211033158
  • Güven, I., & Gulbahar, Y. (2020). Integrating computational thinking into social studies. The Social Studies, 111(5), 234–248. https://doi.org/10.1080/00377996.2020.1749017
  • Harangus, K., & Kátai, Z. (2020). Computational thinking in secondary and higher education. Procedia Manufacturing, 46, 615–622. https://doi.org/10.1016/j.promfg.2020.03.088
  • Herrmann-Abell, C. F., Hardcastle, J., & DeBoer, G. E. (2018). Comparability of computer-based and paper-based science assessments. In Proceedings of the NARST Annual International Conference.
  • Hsu, T.-C., & Liang, Y.-S. (2021). Simultaneously improving computational thinking and foreign language learning: Interdisciplinary media with plugged and unplugged approaches. Journal of Educational Computing Research, 59(6), 1184–1207. https://doi.org/10.1177/0735633121992480
  • Huda, N., & Rohaeti, E. (2024). Computational thinking skill level of senior high school students majoring in natural science. International Journal of Learning, Teaching and Educational Research, 23(1), 339–359. https://doi.org/10.26803/ijlter.23.1.17
  • Hurt, T., Greenwald, E., Allan, S., Cannady, M. A., Krakowski, A., Brodsky, L., Collins, M. A., Montgomery, R., & Dorph, R. (2023). The computational thinking for science (CT-S) framework: Operationalizing CT-S for K-12 science education researchers and educators. International Journal of STEM Education, 10(1), Article 1. https://doi.org/10.1186/s40594-022-00391-7
  • Jin, H.-Y., & Cutumisu, M. (2023). Predicting pre-service teachers’ computational thinking skills using machine learning classifiers. Education and Information Technologies 28, 11447–11467. https://doi.org/10.1007/s10639-023-11642-7
  • Kakavas, P., & Ugolini, F. C. (2019). Computational thinking in primary education: A systematic literature review. Research on Education and Media, 11(2), 64–94. https://doi.org/10.2478/rem-2019-0023
  • Kamak, L. P., & Mago, V. (2023). Assessing the impact of using Python to teach computational thinking for remote schools in a blended learning environment. In P. Zaphiris, & A. Ioannou (Eds.), Learning and collaboration technologies (vol. 14041, pp. 482–500). Springer. https://doi.org/10.1007/978-3-031-34550-0_35
  • Kelderman, H. (1989). Item bias detection using loglinear IRT. Psychometrika, 54(4), 681–697. https://doi.org/10.1007/BF02296403
  • Korkmaz, Ö., Çakir, R., & Özden, M. Y. (2017). A validity and reliability study of the computational thinking scales (CTS). Computers in Human Behavior, 72, 558–569. https://doi.org/10.1016/j.chb.2017.01.005
  • Krejcie, R. V., & Morgan, D. W. (1970). Determining sample size for research activities. Educational and Psychological Measurement, 30(3), 607–610. https://doi.org/10.1177/001316447003000308
  • Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28(4), 563–575. https://doi.org/10.1111/j.1744-6570.1975.tb01393.x
  • Li, X., Sang, G., Valcke, M., & Van Braak, J. (2024). The development of an assessment scale for computational thinking competence of in-service primary school teachers. Journal of Educational Computing Research, 62(6), 1318–1347. https://doi.org/10.1177/07356331241254575
  • Li, Y., Schoenfeld, A. H., diSessa, A. A., Graesser, A. C., Benson, L. C., English, L. D., & Duschl, R. A. (2020). Computational thinking is more about thinking than computing. Journal for STEM Education Research, 3(1), 1–18. https://doi.org/10.1007/s41979-020-00030-2
  • Mohaghegh, M., & McCauley, M. (2016). Computational thinking: The skill set of the 21st century. International Journal of Computer Science and Information Technologies, 7(3), 1524–1530.
  • Moreno-Leon, J., Roman-Gonzalez, M., & Robles, G. (2018). On computational thinking as a universal skill: A review of the latest research on this ability. In Proceedings of the 2018 IEEE Global Engineering Education Conference (pp. 1684–1689). IEEE. https://doi.org/10.1109/EDUCON.2018.8363437
  • Niousha, R., Saito, D., Washizaki, H., & Fukazawa, Y. (2023). Investigating the effect of binary gender preferences on computational thinking skills. Education Sciences, 13(5), Article 433. https://doi.org/10.3390/educsci13050433
  • Noyes, J. M., & Garland, K. J. (2008). Computer- vs. paper-based tasks: Are they equivalent? Ergonomics, 51(9), 1352–1375. https://doi.org/10.1080/00140130802170387
  • Nunnally, J. C. (1978). Psychometric theory (2nd ed.). McGraw-Hill.
  • Ocampo, L. M., Corrales-Álvarez, M., Cardona-Torres, S. A., & Zapata-Cáceres, M. (2024). Systematic review of instruments to assess computational thinking in early years of schooling. Education Sciences, 14(10), Article 1124. https://doi.org/10.3390/educsci14101124
  • Ogegbo, A. A., & Ramnarain, U. (2022). A systematic review of computational thinking in science classrooms. Studies in Science Education, 58(2), 203–230. https://doi.org/10.1080/03057267.2021.1963580
  • Papert, S. (1980). Mindstorms: Children, computers, and powerful ideas. Basic Books.
  • Popat, S., & Starkey, L. (2019). Learning to code or coding to learn? A systematic review. Computers & Education, 128, 365–376. https://doi.org/10.1016/j.compedu.2018.10.005
  • Putra, Z. H., Ramiati, Zufriady, Hidayat, R., Jismulatif, Hermita, N., & Sulistiyo, U. (2022). Development of computational thinking tasks based on Riau Malay culture: A study of fifth-grade public school students in Pekanbaru, Indonesia. Education 3–13, 52(8), 1387–1397. https://doi.org/10.1080/03004279.2022.2150063
  • Rahimi, A. R., & Sevilla-Pavón, A. (2025). Scaling up computational thinking skills in computer-assisted language learning (CTsCALL) and its fitness with language learners’ intentions to use virtual exchange: A bi-symmetric approach. Computers in Human Behavior Reports, 17, Article 100607. https://doi.org/10.1016/j.chbr.2025.100607
  • Revana, G., Kavita, K., & Madhavi, V. (2021). Exploring the concept of computational thinking in STEM education. In Proceedings of the 2021 World Engineering Education Forum/Global Engineering Deans Council (pp. 375–380). https://doi.org/10.1109/WEEF/GEDC53299.2021.9657230
  • Romero, M., Lepage, A., & Lille, B. (2017). Computational thinking development through creative programming in higher education. International Journal of Educational Technology in Higher Education, 14(1), Article 42. https://doi.org/10.1186/s41239-017-0080-z
  • Rottenhofer, M., Kuka, L., Leitner, S., & Sabitzer, S. (2022). Using computational thinking to facilitate language learning: A survey of students’ strategy use in Austrian secondary schools. IAFOR Journal of Education, 10(2), 51–70. https://doi.org/10.22492/ije.10.2.03
  • Saidin, N. D., Khalid, F., Martin, R., Kuppusamy, Y., & Munusamy, N. A. (2021). Benefits and challenges of applying computational thinking in education. International Journal of Information and Education Technology, 11(5), 248–254. https://doi.org/10.18178/ijiet.2021.11.5.1519
  • Sari, U., Ulusoy, A., & Pektaş, H. M. (2025). Computational thinking in science laboratories based on the flipped classroom model: Computational thinking, laboratory entrepreneurial and attitude. Journal of Science Education and Technology. https://doi.org/10.1007/s10956-024-10192-y
  • Snyder, S., & Sheehan, R. (1992). The Rasch measurement model: An introduction. Journal of Early Intervention, 16(1), 87–95. https://doi.org/10.1177/105381519201600108
  • So, H.-J., Jong, M. S.-Y., & Liu, C.-C. (2020). Computational thinking education in the Asian Pacific Region. The Asia-Pacific Education Researcher, 29(1), 1–8. https://doi.org/10.1007/s40299-019-00494-w
  • Sovey, S., Osman, K., & Mohd Matore, M. E. E. (2022). Rasch analysis for disposition levels of computational thinking instrument among secondary school students. Eurasia Journal of Mathematics, Science and Technology Education, 18(3), Article em2088. https://doi.org/10.29333/ejmste/11794
  • Sumintono, B., & Widhiarso, W. (2015). Aplikasi pemodelan Rasch pada assessment pendidikan [Application of Rasch modelling to educational assessment]. Trim Komunikata Publishing House.
  • Taber, K. S. (2018). The use of Cronbach’s alpha when developing and reporting research instruments in science education. Research in Science Education, 48(6), 1273–1296. https://doi.org/10.1007/s11165-016-9602-2
  • Tabesh, Y. (2017). Computational thinking: A 21st century skill. Olympiads in Informatics, 11(2), 65–70. https://doi.org/10.15388/ioi.2017.special.10
  • Taherdoost, H. (2017). Determining sample size: How to calculate survey sample size. International Journal of Economics and Management Systems, 2, 237–239.
  • Tang, X., Yin, Y., Lin, Q., Hadad, R., & Zhai, X. (2020). Assessing computational thinking: A systematic review of empirical studies. Computers & Education, 148, Article 103798. https://doi.org/10.1016/j.compedu.2019.103798
  • Tariq, R., Aponte Babines, B. M., Ramirez, J., Alvarez-Icaza, I., & Naseer, F. (2025). Computational thinking in STEM education: Current state-of-the-art and future research directions. Frontiers in Computer Science, 6. https://doi.org/10.3389/fcomp.2024.1480404
  • Tripon, C. (2022). Supporting future teachers to promote computational thinking skills in teaching STEM–A case study. Sustainability, 14(19), Article 12663. https://doi.org/10.3390/su141912663
  • Tsai, M.-J., Liang, J.-C., & Hsu, C.-Y. (2021). The computational thinking scale for computer literacy education. Journal of Educational Computing Research, 59(4), 579–602. https://doi.org/10.1177/0735633120972356
  • Voogt, J., Fisser, P., Good, J., Mishra, P., & Yadav, A. (2015). Computational thinking in compulsory education: Towards an agenda for research and practice. Education and Information Technologies, 20(4), 715–728. https://doi.org/10.1007/s10639-015-9412-6
  • Voskoglou, M. G., & Buckley, S. (2012). Problem solving and computational thinking in a learning environment. Egyptian Computer Science Journal, 36(4), 28–46.
  • Waterman, K. P., Goldsmith, L., & Pasquale, M. (2020). Integrating computational thinking into elementary science curriculum: An examination of activities that support students’ computational thinking in the service of disciplinary learning. Journal of Science Education and Technology, 29(1), 53–64. https://doi.org/10.1007/s10956-019-09801-y
  • Weintrop, D., Morehouse, S., & Subramaniam, M. (2021). Assessing computational thinking in libraries. Computer Science Education, 31(2), 290–311. https://doi.org/10.1080/08993408.2021.1874229
  • Wing, J. M. (2006). Computational thinking. Communications of the ACM, 49(3), 33–35. https://doi.org/10.1145/1118178.1118215
  • Wu, M., & Adams, R. J. (2007). Applying the Rasch model to psycho-social measurement: A practical approach. Educational Measurement Solutions.
  • Wu, T.-T., Asmara, A., Huang, Y.-M., & Permata Hapsari, I. (2024). Identification of problem-solving techniques in computational thinking studies: Systematic literature review. SAGE Open, 14(2). https://doi.org/10.1177/21582440241249897
  • Xu, Z., & Zhang, J. (2021). Computational thinking: A perspective on computer science (1st ed.). Springer. https://doi.org/10.1007/978-981-16-3848-0
  • Yamashita, T. (2022). Analyzing Likert scale surveys with Rasch models. Research Methods in Applied Linguistics, 1(3), Article 100022. https://doi.org/10.1016/j.rmal.2022.100022
  • Yang, D., Snelson, C., & Feng, S. (2023). Identifying computational thinking in students through project-based problem-solving activities. Information Discovery and Delivery, 51(3), 293–305. https://doi.org/10.1108/IDD-09-2022-0091
  • Yang, T.-C., & Lin, Z.-S. (2024). Enhancing elementary school students’ computational thinking and programming learning with graphic organizers. Computers & Education, 209, Article 104962. https://doi.org/10.1016/j.compedu.2023.104962
  • Zamzami, E. M., Tarigan, J. T., Zendrato, N., Muis, A., Yoga, A. P., & Faisal, M. (2020). Exercising the students computational thinking ability using Bebras challenge. Journal of Physics: Conference Series, 1566(1), Article 012113. https://doi.org/10.1088/1742-6596/1566/1/012113
  • Zapata, J. H., Gutiérrez Posada, J. E., & Diago, P. D. (2024). Design and validation of a computational thinking test for children in the first grades of elementary education. Multimodal Technologies and Interaction, 8(5), Article 39. https://doi.org/10.3390/mti8050039