Supporting Generative AI Literacy: Exploring the Pedagogical Roles Students Assign ChatGPT and Impact on Course Grades
DOI:
https://doi.org/10.5281/zenodo.15993999Keywords:
ChatGPT, Generative AI, Higher Education, Pedagogical Centered AI, Students, Course PerformanceAbstract
This research examines generative AI (GenAI) use in a university course that encouraged ChatGPT for specific assignments. Using the Pedagogical Centered AI (PCAI) framework, we explore how students perceive, use, and position ChatGPT, and how usage patterns influenced performance. Students utilized ChatGPT during the latter half of the Spring 2024 semester. Comparisons were made with the first half of the course and prior iterations (2022 and 2023) without GenAI. All students in the 2024 cohort — 40 students — were invited to participate in the study. Data includes 18 student interviews from the 2024 cohort and student work from all iterations. Interviews underwent qualitative deductive thematic analysis using PCAI’s predefined codes PCAI frames AI in education through six learning theories: behaviorism, cognitivism, constructivism, social constructivism, experiential learning, and communities of practice. Class materials and academic records were analyzed to assess performance quantitatively using inferential statistics. Findings reveal students predominantly view AI from a behaviorist perspective: as a tool for completing tasks. Some aligned usage with cognitive learning theory by using AI to reduce cognitive load, while others adopted social constructivist or constructivist perspectives, using AI to build understanding through feedback and exam preparation functions. Overuse of ChatGPT correlated with lower grades, though only one student acknowledged its negative impact on learning. We discuss implications for higher education and highlight how ChatGPT supports diverse teaching and learning approaches tailored to students’ needs. In particular, strategies aligned with constructivism, social constructivism, and communities of practice approaches seem to enhance student learning. However, behaviorist approaches to AI use could hinder learning outcomes. Although most students were aware of the negative impact of AI overuse, they also mentioned that minimal training and explanation were provided in other classes, highlighting the need for a more comprehensive program to support AI literacy in higher education.
References
Adeshola, I., & Adepoju, A. P. (2023). The opportunities and challenges of ChatGPT in education. Interactive Learning Environments, 32(10), 6159-6172. https://doi.org/10.1080/10494820.2023.2253858
Alfredo, R., Echeverria, V., Jin, Y., Yan, L., Swiecki, Z., Gaševi?, D., et al. (2024). Human-centred learning analytics and AI in education: A systematic literature review. Computers and Education: Artificial Intelligence, 6, 100215. https://doi.org/10.1016/j.caeai.2024.100215
Alkaissi, H., & McFarlane, S. I. (2023). Artificial Hallucinations in ChatGPT: Implications in Scientific Writing. Cureus, 15(2), e35179. https://doi.org/10.7759/cureus.35179
Baber, H., Nair, K., Gupta, R., & Gurjar, K. (2023). The beginning of ChatGPT – a systematic and bibliometric review of the literature. Information and Learning Sciences, 125(7/8), 587-614. https://doi.org/10.1108/ILS-04-2023-0035
Bingley, W. J., Haslam, S. A., Steffens, N. K., Gillespie, N., Worthy, P., Curtis, C., et al. (2023). Enlarging the model of the human at the heart of human-centered AI: A social self-determination model of AI system impact. New Ideas in Psychology, 70, 101025. https://doi.org/10.1016/j.newideapsych.2023.101025
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101. https://doi.org/10.1191/1478088706qp063oa
Bruner, J. (1966). Toward a Theory of Instruction. Harvard University Press. https://www.hup.harvard.edu/books/9780674897014
Cardona, M. A., Rodríguez, R. J., & Ishmael, K. (2023). Artificial Intelligence and the Future of Teaching and Learning: Insights and Recommendations. U.S. Department of Education, Office of Educational Technology. https://digital.library.unt.edu/ark:/67531/metadc2114121
Ch, D. R., & Saha, S. K. (2019). RemedialTutor: A blended learning platform for weak students and study its efficiency in social science learning of middle school students in India. Education and Information Technologies, 24(3), 1925-1941. https://doi.org/10.1007/s10639-018-9813-4
Chen, G., Alsharef, A., Ovid, A., Albert, A., & Jaselskis, E. (2025). Meet2Mitigate: An LLM-powered framework for real-time issue identification and mitigation from construction meeting discourse. Advanced Engineering Informatics, 64, 103068. https://doi.org/10.1016/j.aei.2024.103068
Chiu, T. K. F., Xia, Q., Zhou, X., Chai, C. S., & Cheng, M. (2023). Systematic literature review on opportunities, challenges, and future research recommendations of artificial intelligence in education. Computers and Education: Artificial Intelligence, 4, 100118. https://doi.org/10.1016/j.caeai.2022.100118
Chomsky, N., Roberts, I., & Watumull, J. (2023, March 8). Noam Chomsky: The False Promise of ChatGPT. The New York Times. https://www.nytimes.com/2023/03/08/opinion/noam-chomsky-chatgpt-ai.html
Crompton, H., & Burke, D. (2023). Artificial intelligence in higher education: the state of the field. International Journal of Educational Technology in Higher Education, 20(1), 22. https://doi.org/10.1186/s41239-023-00392-8
Dempere, J., Modugu, K., Hesham, A., & Ramasamy, L. K. (2023). The impact of ChatGPT on higher education. Frontiers in Education, 8, 1206936. https://doi.org/10.3389/feduc.2023.1206936
Díaz, B., & Delgado, C. (2024). Artificial intelligence: Tool or teammate? Journal of Research in Science Teaching, 61(10), 2575-2584. https://doi.org/10.1002/tea.21993
Díaz, B., Delgado, C., Han, K., & Lynch, C. (2024). Using communities of practice to investigate work-integrated learning in engineering education: a grounded theory approach. Higher Education, 88(6), 2419-2443. https://doi.org/10.1007/s10734-024-01225-x
Díaz, B., & Nussbaum, M. (2024). Artificial intelligence for teaching and learning in schools: The need for pedagogical intelligence. Computers & Education, 217, 105071. https://doi.org/10.1016/j.compedu.2024.105071
EDUCAUSE. (2019). Horizon Report: 2019 Higher Education Edition. Louisville, CO: EDUCAUSE. https://library.educause.edu/-/media/files/library/2019/4/2019horizonreport.pdf
European Commission Directorate. (2022). Ethical Guidelines on the Use of Artificial Intelligence (AI) and Data in Teaching and Learning for Educators. Publications Office of the European Union. https://doi.org/10.2766/153756
Farhi, F., Jeljeli, R., Aburezeq, I., Dweikat, F. F., Al-shami, S. A., & Slamene, R. (2023). Analyzing the students’ views, concerns, and perceived ethics about chat GPT usage. Computers and Education: Artificial Intelligence, 5, 100180. https://doi.org/10.1016/j. caeai.2023.100180
Gardner, H. E. (2003). Multiple Intelligences After Twenty Years (Vol. 21). Chicago, Illinois: American Educational Research Association.
Gardner, H. E. (2011). Frames of Mind: The Theory of Multiple Intelligences. Basic Books.
Gillani, N., Eynon, R., Chiabaut, C., & Finkel, K. (2023). Unpacking the “Black Box” of AI in Education. Educational Technology & Society, 26(1), 99-111. https://www.jstor.org/stable/48707970
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., et al. (2023). How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment. JMIR Medical Education, 9, e45312. https://doi.org/10.2196/45312
Guest, G., MacQueen, K. M., & Namey, E. E. (2012). Applied Thematic Analysis. SAGE Publications, Inc. https://doi.org/10.4135/9781483384436
Hsu, F.-H., Lin, I. H., Yeh, H.-C., & Chen, N.-S. (2022). Effect of Socratic Reflection Prompts via video-based learning system on elementary school students’ critical thinking skills. Computers & Education, 183, 104497. https://doi.org/10.1016/j.compedu.2022.104497
Hu, K. (2023, February 2). ChatGPT sets record for fastest-growing user base - analyst note. Reuters. https://www.reuters.com/technology/chatgpt-sets-record-fastest-growing-user-base-analyst-note-2023-02-01
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., et al. (2023). Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 55(12), 1-38. https://doi.org/10.1145/3571730
Katz, D. M., Bommarito, M. J., Gao, S., & Arredondo, P. (2024). GPT-4 passes the bar exam. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 382(2270), 20230254. https://doi.org/10.1098/rsta.2023.0254
Kim, W. J., & Rachmatullah, A. (2025). Science Teachers’ Approaches to Artificial Intelligence Integrated Science Teaching. Research in Science Education. https://doi.org/10.1007/s11165-025-10233-5
Kolb, D. A. (1984). Experiential Learning: Experience as the Source of Learning and Development. Prentice-Hall.
Lave, J., & Wenger, E. (1991). Situated Learning: Legitimate Peripheral Participation (1st ed.). Cambridge University Press. https://doi.org/10.1017/CBO9780511815355
Lee, B. C., & Chung, J. (2024). An empirical investigation of the impact of ChatGPT on creativity. Nature Human Behaviour, 8(10), 1906-1914. https://doi.org/10.1038/s41562-024-01953-1
Li, S., & Gu, X. (2023). A Risk Framework for Human-centered Artificial Intelligence in Education: Based on Literature Review and Delphi–AHP Method. Educational Technology & Society, 26(1), 187-202. https://doi.org/10.30191/ETS.202301_26(1).0014
Lo, C. K. (2023). What Is the Impact of ChatGPT on Education? A Rapid Review of the Literature. Education Sciences, 13(4), 410. https://doi.org/10.3390/educsci13040410
Lo, C. K., Hew, K. F., & Jong, M. S.-y. (2024). The influence of ChatGPT on student engagement: A systematic review and future research agenda. Computers & Education, 219, 105100. https://doi.org/10.1016/j.compedu.2024.105100
Mollick, E., & Mollick, L. (2023). Assigning AI: Seven Approaches for Students, with Prompts. arXiv preprint arXiv:2306.10052. https://doi.org/10.48550/arXiv.2306.10052
Montenegro-Rueda, M., Fernández-Cerero, J., Fernández-Batanero, J. M., & López-Meneses, E. (2023). Impact of the Implementation of ChatGPT in Education: A Systematic Review. Computers, 12(8), 153. https://doi.org/10.3390/computers12080153
Nasrullah, N., & Al Wahyu, T. (2024). The Application of Chat GPT in English Language Evaluation: A Systematic Literature Review. Futurity Education, 4(3), 217-235. https://doi.org/10.57125/FED.2024.09.25.13
Ovadia, O., Brief, M., Mishaeli, M., & Elisha, O. (2023). Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs. arXiv preprint arXiv:2312.05934. https://doi.org/10.48550/arXiv.2312.05934
Ozmen Garibay, O., Winslow, B., Andolina, S., Antona, M., Bodenschatz, A., Coursaris, C., et al. (2023). Six Human-Centered Artificial Intelligence Grand Challenges. International Journal of Human–Computer Interaction, 39(3), 391-437. https://doi.org/10.1080/10447318.2022.2153320
Piaget, J. (1973). To Understand is to Invent: The Future of Education. Grossman. https://unesdoc.unesco.org/ark:/48223/pf0000006133
Rahm, L., & Rahm-Skågeby, J. (2023). Imaginaries and problematisations: A heuristic lens in the age of artificial intelligence in education. British Journal of Educational Technology, 54(5), 1147-1159. https://doi.org/10.1111/bjet.13319
Shneiderman, B. (2020). Human-Centered Artificial Intelligence: Three Fresh Ideas. AIS Transactions on Human-Computer Interaction, 12(3), 109-124. https://doi.org/10.17705/1thci.00131
Shneiderman, B. (2022). Human-Centered AI. Oxford University Press. https://doi.org/10.1093/oso/9780192845290.001.0001
Sun, D., Boudouaia, A., Zhu, C., & Li, Y. (2024). Would ChatGPT-facilitated programming mode impact college students’ programming behaviors, performances, and perceptions? An empirical study. International Journal of Educational Technology in Higher Education, 21(1), 14. https://doi.org/10.1186/s41239-024-00446-5
Tai, T.-Y., & Chen, H. H.-J. (2024). Improving elementary EFL speaking skills with generative AI chatbots: Exploring individual and paired interactions. Computers & Education, 220, 105112. https://doi.org/10.1016/j.compedu.2024.105112
Talha Junaid, M., Barakat, S., Awad, R., & Anwar, N. (2024). Adopting the Power of AI Chatbots for Enriching Students Learning in Civil Engineering Education: A Study on Capabilities and Limitations. In A. Al-Marzouqi, S. A. Salloum, M. Al-Saidat, A. Aburayya, & B. Gupta (Eds.), Artificial Intelligence in Education: The Power and Dangers of ChatGPT in the Classroom (pp. 25-47). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-52280-2_3
Uddin, S. M. J., Albert, A., Ovid, A., & Alsharef, A. (2023). Leveraging ChatGPT to Aid Construction Hazard Recognition and Support Safety Education and Training. Sustainability, 15(9), 7121. https://doi.org/10.3390/su15097121
Uddin, S. M. J., Albert, A., Tamanna, M., Ovid, A., & Alsharef, A. (2024). ChatGPT as an educational resource for civil engineering students. Computer Applications in Engineering Education, 32(4), e22747. https://doi.org/10.1002/cae.22747
Usmani, U. A., Happonen, A., & Watada, J. (2023). Human-Centered Artificial Intelligence: Designing for User Empowerment and Ethical Considerations. In 2023 5th International Congress on Human-Computer Interaction, Optimization and Robotic Applications (HORA) (pp. 1-7). IEEE. https://doi.org/10.1109/HORA58378.2023.10156761
van den Berg, G., & du Plessis, E. (2023). ChatGPT and Generative AI: Possibilities for Its Contribution to Lesson Planning, Critical Thinking and Openness in Teacher Education. Education Sciences, 13(10), 998. https://doi.org/10.3390/educsci13100998
Vygotsky, L. S. (1978). Mind in Society: Development of Higher Psychological Processes. Harvard University Press. https://www.hup.harvard.edu/books/9780674576292
Watson, J. B. (2017). Behaviorism. Routledge. https://doi.org/10.4324/9781351314329
Whalen, J., & Mouza, C. (2023). Editorial: ChatGPT: Challenges, Opportunities, and Implications for Teacher Education. Contemporary Issues in Technology and Teacher Education, 23(1), 1-23. https://www.learntechlib.org/p/222408
Wood, D. A., Achhpilia, M. P., Adams, M. T., Aghazadeh, S., Akinyele, K., Akpan, M., et al. (2023). The ChatGPT Artificial Intelligence Chatbot: How Well Does It Answer Accounting Assessment Questions? Issues in Accounting Education, 38(4), 81-108. https://doi.org/10.2308/ISSUES-2023-013
Xia, Q., Chiu, T. K. F., Chai, C. S., & Xie, K. (2023). The mediating effects of needs satisfaction on the relationships between prior knowledge and self-regulated learning through artificial intelligence chatbot. British Journal of Educational Technology, 54(4), 967-986. https://doi.org/10.1111/bjet.13305
Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education – where are the educators? International Journal of Educational Technology in Higher Education, 16(1), 39. https://doi.org/10.1186/s41239-019-0171-0
Zhang, P., & Tur, G. (2023). A systematic review of ChatGPT use in K-12 education. European Journal of Education, 59(2), e12599. https://doi.org/10.1111/ejed.12599
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Comunicar
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
