English Education Students' Perceptions of Automated vs Human Assessment in Spoken English Proficiency

Nur Aeni; Muhalim Muhalim; Hasriani Ganteng; Muhammad Tahir; Ahmad Talib

doi:10.35445/alishlah.v17i3.7655

English Education Students' Perceptions of Automated vs Human Assessment in Spoken English Proficiency

Nur Aeni, Muhalim Muhalim, Hasriani Ganteng, Muhammad Tahir, Ahmad Talib

Abstract

The increasing use of automated evaluation systems in language assessment raises questions about their acceptance and perceived fairness compared to human evaluation. This study examines how English Education students perceive automated and human assessment of spoken English proficiency, focusing on factors influencing acceptance and preferences for hybrid models. A mixed-methods design was employed with 120 English Education students (80 female, 40 male) from Universitas Negeri Makassar. Quantitative data were collected using a 20-item Likert-scale questionnaire (Cronbach’s α = .87) covering six dimensions: Perceived Ease of Use, Perceived Usefulness, Attitude Toward Technology, Self-Efficacy, Behavioral Intention, and Personal Innovativeness. Qualitative data from semi-structured interviews explored students’ experiences and preferences regarding automated and human evaluation. Descriptive statistics indicated generally positive perceptions of automated evaluation, with the highest mean scores for “Automated feedback helps improve pronunciation and fluency” (M = 3.9, SD = 0.928) and “I enjoy playing with new technology in language acquisition” (M = 4.0, SD = 1.071). However, the lowest score for “I plan to use automated evaluation frequently” (M = 2.7, SD = 1.071) reflected hesitancy toward regular use. Thematic analysis revealed three main themes: appreciation of efficiency but skepticism about accuracy, preference for human empathy and contextual understanding, and concerns about algorithmic bias, particularly for non-standard accents. Students strongly favored a hybrid approach, endorsing AI for preliminary feedback and routine practice while valuing human evaluation for comprehensive assessment and motivational support. These findings suggest the need for transparent, inclusive AI tools integrated with human oversight to achieve balanced, pedagogically sound evaluation frameworks in English language education.

Keywords

automated evaluation, human raters, spoken English proficiency, student perceptions

Full Text:

PDF

References

Aeni, N., Khang, A., Al Yakin, A., Yunus, M., & Cardoso, L. (2024). Revolutionized teaching by incorporating artificial intelligence chatbot for higher education ecosystem. In AI-centric modeling and analytics (pp. 43–76). CRC Press.

Aeni, N., Muthmainnah, M., Nurfadhilah, A. S., & Inayah, F. (2025). AI-Driven Classroom Conversations: Revolutionizing Education 5.0 for Enhanced Student Engagement in Speaking Class. In Innovations in Educational Robotics: Advancing AI for Sustainable Development (pp. 173-192). IGI Global Scientific Publishing. https://doi.org/10.4018/979-8-3693-6165-8.ch009

Bandura, A. (1997). Self-efficacy: The exercise of control. W. H. Freeman.

Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa

Burston, J. (2021). Intelligent computer-assisted language learning: A review of the field. Computer Assisted Language Learning, 34(5–6), 429–449. https://doi.org/10.1080/09588221.2021.1901745

Chapelle, C. A., & Chung, Y.-R. (2021). The promise of NLP and speech processing in language assessment. Language Testing, 38(2), 189–200. https://doi.org/10.1177/0265532220925731

Dai, H., Ai, H., & Lin, C. (2023). Generative AI as a feedback provider in second language writing: Comparing ChatGPT and human instructors. Computers & Education, 201, 104785. https://doi.org/10.1016/j.compedu.2023.104785

Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319–340. https://doi.org/10.2307/249008

Fryer, L. K., & Carpenter, R. (2022). Chatbot language learning: Moving beyond the hype. Computer Assisted Language Learning, 35(3), 203–218. https://doi.org/10.1080/09588221.2022.2032184

Godwin-Jones, R. (2020). Dealing with complexity in language learning: Language learning analytics and AI. Language Learning & Technology, 24(1), 1–9. https://doi.org/10125/44707

He, Y., Chen, J., & Liu, Z. (2020). Learner perceptions of automated essay scoring and feedback: A mixed-methods study. System, 92, 102279. https://doi.org/10.1016/j.system.2020.102279

Hummel, S., & Donner, M.-T. (Eds.). (2023). Student assessment in digital and hybrid learning environments. Springer.

Hwang, G. J., Xie, H., Wah, B. W., & Gašević, D. (2021). Artificial intelligence applications in education. International Journal of Emerging Technologies in Learning, 16(11), 196–213. https://doi.org/10.3991/ijet.v16i11.19657

Johnson, K., & Valente, M. (2023). Artificial intelligence in second language acquisition: Enhancing self-regulated learning through adaptive scaffolding. Language Learning & Technology, 27(1), 1–18. https://doi.org/10.1017/LLT.2023.105

Kim, G.-m., & Lee, S.-j. (2016). Korean students' intentions to use mobile-assisted language learning: Applying the technology acceptance model. International Journal of Contents, 12(3), 47–53. https://doi.org/10.5392/IJoC.2016.12.3.047

Knoch, U., & Macqueen, S. (2020). Assessing English proficiency in the age of automation: A critical perspective on AI-driven testing. Routledge.

LeMay, D. J., Morin, M. M., Bazelais, P., & Doleck, T. (2018). Modeling students' perceptions of simulation-based learning using the technology acceptance model. Clinical Simulation in Nursing, 20, 28–37. https://doi.org/10.1016/j.ecns.2018.04.004

Li, J., Link, S., & Hegelheimer, V. (2015). Rethinking the role of automated writing evaluation (AWE) feedback in ESL writing instruction. Journal of Second Language Writing, 27, 1–18. https://doi.org/10.1016/j.jslw.2014.10.004

Lu, X., & Zhang, W. (2023). AI and human teachers in language education: Bridging efficiency and adaptability. Journal of Educational Technology, 40(3), 215–230. https://doi.org/10.1080/09588221.2023.2256789

Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103). American Council on Education.

Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook (2nd ed.). Sage Publications.

Pereira, F. D., Oliveira, E., Rodrigues, L., Cabral, L., Oliveira, D., Carvalho, L., Silva, I., Brandão, A., Isotani, S., & Mello, R. F. (2023). Evaluation of a hybrid AI-human recommender for CS1 programming assignments. In European Conference on Technology Enhanced Learning (pp. 289–303). Springer.

Rahayu, T. (2021). AI-assisted language learning for non-native English speakers. Journal of English Educators Society, 6(1), 33–49. https://doi.org/10.21070/jees.v6i1.849

Selwyn, N. (2022). Education and technology: Key issues and debates (3rd ed.). Bloomsbury Academic.

Sun, Y., Wang, J., & Liu, C. (2022). The impact of artificial intelligence on second language acquisition: A systematic review. Frontiers in Psychology, 13, 1049139. https://doi.org/10.3389/fpsyg.2022.1049139

Wang, T., & Liu, Y. (2023). Advancing pragmatic competence assessment through AI-driven discourse analysis. Computer-Assisted Language Learning, 36(2), 189–210. https://doi.org/10.1080/09588221.2023.2184967

Wang, Y., & Lin, C. (2023). Bridging AI and human interaction in language learning: Challenges and future directions. Computer Assisted Language Learning, 36(2), 145–163. https://doi.org/10.1080/09588221.2023.1874563

Yastibas, A. E., & Yastibas, G. C. (2021). Learners' opinions on the use of AI-based tools in EFL classrooms. Education and Information Technologies, 26, 3587–3606. https://doi.org/10.1007/s10639-021-10439-1

Zou, D., Huang, Y., & Xie, H. (2022). A review of research on AI-supported language learning and teaching. Computer Assisted Language Learning, 35(1–2), 1–25. https://doi.org/10.1080/09588221.2022.2054844

DOI: https://doi.org/10.35445/alishlah.v17i3.7655