Raising University Students’ Critical Awareness of the Linguistic Limitations and the Potential Invalid Knowledge of ChatGPT Responses to Academic Writing Prompts

1. INTRODUCTION

Academic writing helps students navigate critical thinking. It challenges them to respond, analyse, synthesise and evaluate information acquired from multiple sources. Academic writing, prominent within higher education, serves as one of the fundamental practices for creating knowledge. It is through writing that students can start to clarify their thoughts and express nuanced ideas effectively. McCarthy (2011) argues that academic writing improves lifetime learning, prepares students for professional activity, and satisfies professional membership requirements.  It also improves students' ability to communicate and effectively express complicated ideas, equipping them for academic and professional endeavours in their professions (Kennelly, 2017). In addition, García Díaz (2019) reported that academic writing develops students’ ability to produce knowledge, interact with disciplinary cultures, and coordinate theoretical sources with empirical facts, all of which improve their communication skills and overall learning process. Moreover, writing for academic purposes improves students’ communication, intellectual, and reflective abilities, which aid in a deeper comprehension of the subjects (Burbano, 2023). Academic writing is crucial for success in higher education since it affects tenure, promotions, and hiring (Woloshyn et al., 2024). Successful academic writing requires fundamental potential in every discipline, such as critical analysis, synthesis, and material evaluation (Zhang, 2024). Academic writing is essentially integrated into higher education since it fosters critical thinking skills, aids in understanding the academic field, and calls for data analysis, synthesis, and evaluation. Also, it provides students with insights into how to construct arguments on particular subjects, which is essential for academic and professional success.

 In higher education as well, academic writing is a major assessment method for measuring student learning. Academic writing can be defined as one of the forms of examination: it helps the teacher to check whether the student is aware of the content of the course and whether they can present ideas in a meaningful way (Baker, 2021). Classroom assignments provide spaces for students to interact with the existing literature and participate in conversations that are already happening. Such activity contributes to the higher standards of a student's writing by teaching how to reference existing work correctly and prepares students for future employment by practising analytical thinking and effective communication skills (Smith & Jones, 2020). Based on the foregoing, one can conclude that assessment of students' work through academic writing is of great importance in education as it helps in the development of skills that would be used in their future workplace.

OpenAI has played a crucial role in advancing natural language processing (NLP) and machine learning (ML) through numerous innovative initiatives and models. These efforts have improved the ability of AI systems to comprehend and produce human language (Pokhrel & Banjade, 2023). Artificial intelligence (AI) technologies that can produce text for various jobs are developing at an exponential rate, and their widespread use is expected to have profound and far-reaching effects across numerous industries. Recently, Academic writing and academic environments have been significantly impacted by artificial intelligence (AI). A considerable body of research has looked at the implications, advantages, and difficulties of artificial intelligence (AI) in academic writing, which is becoming a bigger part of higher education. In addition to writing content on any subject and including pertinent citations, these tools can, for instance, summarise, translate, or paraphrase text. Many students and scholars are beginning to adopt AI technologies such as Chatbots as writing helpers because of these affordances, which are helpful for academic writing. Chatbots are computer programs that use artificial intelligence (AI) and natural language processing (NLP) to simulate conversations with humans. One such chatbot is ChatGPT, which uses the third-generation generative pre-trained transformer (GPT-3) developed by OpenAI.

Although ChatGPT has been praised for its ability to generate text, concerns have been raised about its accuracy and precision in generating data, as well as legal issues related to references (Athaluri et al., 2023). ChatGPT's drawbacks for academic writing are numerous and include problems with dependability, correctness, and ethics. Even though ChatGPT has potential advantages like increasing productivity and offering language assistance, there are important issues that need to be resolved when integrating it into academic settings. While many researchers such as Derga et al. (2023) noted that ChatGPT and other related NLP technologies could be useful in improving technical writing and research productivity, the current research advocates for further and more comprehensive debates about the tools’ possible applications, threats, and scope and the need to emphasise ethical and scholarly standards, with the researching activities being dominated by the human intellect and evaluation.

In addition, Conde et al. (2024) found that the employment of AI technologies raises several possible problems, from ethical questions to the veracity of the material produced, which may contain inaccurate statements. Also, Athaluri et al. (2023) referred to ChatGPT's possible shortcomings in producing trustworthy references for research ideas. The issue of AI hallucinations has the potential to impair judgment and lead to moral and legal dilemmas. The model's high degree of non-determinism erodes reproducibility and trust by generating different results for the same stimuli (Ouyang et al., 2024). According to Costa et al. (2024), ChatGPT's output may be inaccurate, which could result in inaccurate information in academic writing. Biases in ChatGPT's training data may be reflected in its outputs, which raises questions about accountability and fairness. The model's trustworthiness is further limited by its inability to be explained, which makes it difficult to comprehend how it makes decisions (Patil et al., 2024).

Despite being a major development in natural language processing, ChatGPT has several linguistic, credibility and reliability flaws that reduce its usefulness. Its architecture, training data, and innate biases are the sources of these shortcomings, which may make it difficult to produce precise and contextually relevant responses in academic writing responses. Recently, many universities have modified their policies, including permission for the use of generative artificial intelligence (GenAI) tools. As noted by Ullah et al. (2024), 41 out of the top 50 universities have published their guidelines, allowing the use of AI under certain conditions. Policies differ greatly, as some universities have established extensive frameworks to encourage AI literacy, whereas others provide very limited guidance (Atkinson-Toal & Guo, 2024). Nevertheless, the potential for generative artificial intelligence (GenAI) to write scholarly works on par with human ability raises questions about its potential abuse in the classroom by students. While Tang & Chaw (2024) clarify such abuse that students might turn in AI-generated writing assignments as their own since ChatGPT can generate a full text, jeopardising academic integrity, Mbwambo & Kaaya (2024) point out that ChatGPT's accessibility and usability raise the possibility that students will become overly dependent on it. Such potential abuse could happen due to the lack of students’ awareness of the ChatGPT limitations. Therefore, this study targets exposing university students to critical thinking and evaluation practices while using ChatGPT in academic writing to cultivate their critical awareness of ChatGPT limitations.

2. LITERATURE REVIEW

2.1 Students’ Use of ChatGPT in Academic Writing

Since the launch of ChatGPT in November 2022, there has been a lot of interest in using generative AI technologies in higher education. In Im’s (2023) study, students expressed their satisfaction and self-development in academic writing confidence due to the use of ChatGPT in their academic submissions; however, it confirms the urgent need for organised instruction that promotes critical thinking and cooperative learning settings. This dual strategy could improve students' language proficiency as well as their capability for meaningful peer interaction and critical interpersonal skills. Although students value ChatGPT's instant support, their reliance on it could highly impede their ability to think critically and solve problems (Juan et al., 2023). Therefore, Guleria et al. (2023) emphasised the significance of cultivating critical thinking among students and researchers to raise awareness about the associated privacy and ethical risks.

Moreover, Ho (2024) investigated students’ behaviour and attitudes while using ChatGPT in their academic writing assignments. Although Ho acknowledged ChatGPT's effectiveness for vocabulary acquisition, grammar checking and paraphrasing, the study results revealed that students mostly used ChatGPT to get immediate answers to their writing prompts and structure. Therefore, he recommends the necessity of teaching students how to use ChatGPT properly and ethically while adopting a critical thinking mind towards ChatGPT responses. Yazid & Dzulfikri (2024) also stated that students often struggle to comprehend the output generated by ChatGPT. This lack of understanding can lead to confusion and misinterpretation of the information provided by the AI, which may hinder their linguistic potential and writing process. Вobk & Kryvoshyia (2024) also highlight in their study that ChatGPT students’ overreliance may reduce their mental capabilities and stress the need to establish a framework for using ChatGPT in academic writing.

According to a recent study by Shrivastava (2024), these cutting-edge technologies are altering educational processes and increasing student and teacher productivity. However, there are drawbacks to the quick uptake of these instruments. Shrivastava’s study emphasizes important worries about the consequences of bias, the possibility of plagiarism, and difficulties with assessment procedures. Shrivastava points out the need for a diversified strategy to address these issues, including the creation of strong policies and instructional frameworks that encourage moral application and optimize the advantages of generative artificial intelligence in educational contexts. Nepal (2024) highlights the importance of training students and researchers to critically evaluate the outputs generated by ChatGPT and emphasises the need for responsible awareness and deployment of ChatGPT biases to ensure fair and equitable use of AI technologies in academia. Warr et al. (2024) also added that it is critical to develop an understanding and critical awareness of the bias embedded in these models as ChatGPT.

            In addition, Li (2024) calls for ethical considerations in ChatGPT for academic writing and advocates for developing the students’ responsible use of ChatGPT in academic writing integrity. While Gultom et al. (2024) study illustrates that using ChatGPT aids and improves the academic writing process in higher education, it advocates the need to avoid students’ irresponsible use and ChatGPT’s potential limitations. Also, Asad et al. (2024) refer in their study to the potential enhancement of using ChatGPT in personalised learning in writing strategies; however, they pointed out its negative impact on students’ creativity and critical skills in academic writing. Rensburg & Reedy’s (2024) qualitative study explored the literature on the use of ChatGPT by students in higher education, including its applications, challenges and ethical concerns. The study pinpoints the limitations and ethical concerns of ChatGPT use in academic writing, focusing on bias, reliability, accuracy, and accountability. It also concludes that clear and transparent guidelines should be offered for students and integrated into the teaching of academic writing. Although a large body of studies argues against the misuse of ChatGPT in academic writing, a few studies advocate for utilising ChatGPT in the academic writing process.

2.2 ChatGPT Credibility & Reliability Limitations

AI Hallucinations & Inaccuracies

“Hallucinations” is a term that refers to false or misleading information created by ChatGPT, which can be problematic in academic contexts where precision is essential. The frequency of hallucinations (inaccurate or fabricated information) in ChatGPT, particularly in academic writing, raises serious questions regarding the credibility and reliability of ChatGPT-generated information. ChatGPT credibility issues have been thoroughly investigated in different contexts, highlighting notable inaccuracies of the content generated by ChatGPT due to its "hallucinations". George & Stuhlmüller (2023) evaluated hallucinations in ChatGPT's academic summaries using Factored Verification for detection, which proved effective detection of hallucinations in academic summaries. The findings show an average of 0.62 hallucinations per summary, highlighting the need for caution in using language models for accurate academic writing. Hwang et al. (2024) asserted that even the improved model ChatGPT-4 showed a hallucination rate of 1.13 items per abstract in the studied scholarly work.

ChatGPT’s hallucinations dramatically impact its accuracy, validity and credibility. Mahyoo et al. (2023) found that ChatGPT’s hallucination produces nonfactual inferences, repetition, and illogical reasoning in academic writing. In addition, Alkaissi & McFarlane (February 19, 2023) identified instances where it generated confident yet inaccurate information in academic writing due to its 'artificial hallucinations', arguing that ChatGPT's ability to generate factually correct scientific writing is questionable. Hallucinations can compromise the integrity of research proposals in academic writing by causing the inclusion of false information, incorrect predictions, redundant insufficient information and unreliable references, which leads to misleading users by providing inaccurate, unreliable information (Costa et al., 2024; Jančařík & Dušek, 2024; Mondal & Sancheti, 2024; Cabuquin et al., 2024 and Kolesnikov et al., 2024). Other studies, such as Gupta et al. (April 1, 2024), Kobak et al. (2024), Royani & Sihombing (2024), and (Akhoon et al., 2024) argued the lack of domain-specific knowledge, inaccuracies in scientific data and limited ability to interpret results, thus dramatically impacting the validity of academic writing and research work.

Moreover, research has reported instances where ChatGPT’s hallucination provided citations to nonexistent sources, which raises questions about its reliability. Buchanan & Shapoval (2024) found that over 30% of the references produced by GPT-3.5 were fake due to AI hallucinations creating nonexistent data, with a slight improvement in GPT-4, which still exhibited high error rates. Jan (2024) confirmed ChatGPT exhibits serious trouble correctly citing scholarly material that has been withdrawn; the model's applicability for academic applications is doubtable since it relies on predictive reasoning rather than validated evidence While several studies have pointed out that ChatGPT could enhance productivity and language mechanics, it is urgent to consider its integration into academic writing with caution to eliminate the risks related to AI hallucinations fabricating information and references. These limitations have been evident in academic and assessment environments, where accurate, credible, and reliable information is essential.

Language Model Inherent Bias

Big language models like ChatGPT in educational contexts face a threat of bias, potentially reinforcing existing disparities rather than reducing them. Recent studies such as Koçyiğit & Zhaksylyk, 2023 Kleebayoon & Wiwanitkit, 2023 and Kolesnikov et al. 2024, indicate that biases in training data reflected in AI models compromise academic justice and honesty. Wen and Younes’s study (2023) found that the ChatGPT model revealed biases such as hate speech rather than racial and gender biases. However, Motoki et al., 2023 pinpointed biases in political orientation, gender, race, and religion in the ChatGPT model. Fulgu & Capraro’s (2024) research has shown that ChatGPT-4 gender biases often attribute masculine stereotypes to female writers more frequently than the reverse; for example, bias favouring male characters in violent scenarios in moral dilemmas. In addition, Warr et al. (2024) found racial biases very implicit in ChatGPT when demographic information is indicated rather than directly presented. In an educational task, the model's responses can differ based on a student's perceived race or socioeconomic status, suggesting a deeper bias reflecting societal prejudices. Georgiou's research (2024) also highlighted ChatGPT's bias in favour of industrialised nations over developing ones, with more favourable opinions of high HDI nations than low HDI ones. On the other hand, Nepal (2024) identified confirmation and cultural bias in ChatGPT models, which generate responses that ignore opposing views or minority ideas in favour of dominant or popular narratives, limiting the diversity of viewpoints in academics. Inherent biases in ChatGPT outputs raise concerns about its use in sensitive academic discussions.

Racial and cultural bias in AI models like ChatGPT may have a substantial effect on academic writing, affecting representation, research focus, stereotyping, collaboration, and ethical standards. Hanna et al. (2023) argue that cultural biases in academic literature could result in the reinforcement of stereotypes and false beliefs about certain cultures and spread false beliefs, undermining the authority of scholarly writing and producing a lack of diversity in academic research. Additionally, this lack of representation can alienate students from other backgrounds, who may feel their viewpoints and experiences are not recognised. For example, language models like ChatGPT favour some narratives or frameworks over others when used in academic contexts (Özer, 2024) and exclude non-Western voices when they produce text that primarily focuses on Western narratives, which perpetuates the power structures already in place in academics (Nationality Bias in Text Generation, 2023 & Nepal, 2024). ChatGPT cultural biases could lead to the silence of voices that offer insightful contributions to the academic community, homogenising ideas, eroding the depth of scientific discourse, and inhibiting creativity and critical thinking.

2.3 ChatGPT Linguistic Limitations

Formulaic Language & Monotonous Writing Style

One of the linguistic limitations of ChatGPT has been identified as producing formulaic language, raising concerns of scholars unique voice and expression. Baziyad et al. (2023) argue that ChatGPT's autoregressive mechanism produces formulaic language, lacking genuineness and nuance, which hinders the development of critical thinking and creative skills in students. This results in poor emotional depth and engagement in academic writing. While Bhatia (2023) indicated that ChatGPT produces “plausible-sounding” answers based on its automated use of vocabulary and formulaic language, this raises concerns that AI tools may compromise critical thinking and individual expression, suppressing unique thought and expression, which are essential for competent writing (Audichy & Saini, 2023). In addition, Alam (2024) opined that ChatGPT has trouble with discourse particles, conversational cues, and formulaic genre structures, impacting the subtle flow and clarity and leading to decreased coherence. Moreover, Markey et al. (2024) referred to the documents produced using ChatGPT as dialogically closed, "empty," and "fluffy," missing the complexity and involvement that characterise academic writing. Yazid & Dzulfikri (2024) found that the informative density and coherence of ChatGPT academic writing responses are diminished when comparing ChatGPT outputs to writings produced by published and inexperienced authors, the informative density and coherence are diminished (Yazid & Dzulfikri, 2024).

With ChatGPT formulaic language, another linguistic limitation tends to dominate the model writing style to be monotonous. Baziyad et al. (2023) concluded that ChatGPT’s machine-patterned language, resulting from its autoregression strategy, contributes to its monotonous writing style, detected through their experimental analysis of its generated texts. However, Silva et al. (2023) argued that the ChatGPT production of monotonous writing styles is due to its reliance on extensive training data. Additionally, Basmo et al. (2023) identified monotonicity as one of ChatGPT entailments, highlighting the demolishment risk of originality and engagement of scientific writing (Mahyoob et al., 2023). Ningsih & Rohmah (2024) criticised ChatGPT’s repetitive writing styles, including direct sentences, as one of the monotonous style flaws (Fiialka et al., 2024). Nepal (2024) concluded that ChatGPT’s monotonous writing style could hinder maintaining diverse and engaging language in academic contexts.

The limitations of ChatGPT in producing formulaic language and monotonous writing styles, which have been highlighted across various studies, could significantly impact academic context quality and creativity. Bhatia (2023) declared that students in academic contexts have become more prone to adopting ways of synthesis information uncritically due to the increasing use of ChatGPT, impairing their critical thinking and analytical skills—two things that are essential for scientific academic writing (Bhatia, 2023). Baziyad et al. (2023) confirmed that the spread of AI-generated content in academic contexts makes it challenging to distinguish between unique concepts and machine-generated language, compromising the essential qualities of depth and originality (Dergaa et al., 2023). On the other hand, some researchers argue that ChatGPT could still serve as a valuable tool for generating ideas and assisting with editorial tasks despite its limitations if used judiciously. Therefore, becoming aware of the limitations of ChatGPT is crucial for understanding the model's capabilities and shortcomings and for critically evaluating its content.

Decontextualization & Linguistic Ambiguity

Another linguistic limitation that impacts the model's performance is decontextualisation, being incapable of comprehending context. As stated by Sinha et al. (2022), specific assessments reveal that ChatGPT frequently displays instability in response to various contextual cues, especially those that closely correspond with grammatical infractions. In addition, Qiu et al. (2023) found that ChatGPT encounters difficulties digesting pragmatic implicatures with the same human-like flexibility, which might result in miscommunications in complex discussions. Basmov et al. (2023) further found that ChatGPT lacks the level of contextual knowledge required to produce complex discussions, which can lead to a shallow interaction with the content. Liu et al. (2023) confirmed that ChatGPT encounters difficulties digesting pragmatic implicatures with the same human-like flexibility, which might result in miscommunications in complex discussions. While Markey et al. (2024) pointed out that ChatGPT can produce text rapidly, such lack of contextual understanding hinders the production of nuanced arguments and emotional resonance, which are vital in persuasive writing and arguments (Jiang & Hyland, 2024). In addition to decontextualization, Arif et al. (2023) found that ChatGPT produced ambiguous response output in academic writing responses. Ortega-Martín et al. (2023) further argued that ChatGPT’s general weaknesses in handling linguistic ambiguity can negatively affect the coherence and clarity of its generated texts. Due to this linguistic ambiguity limitation embedded in the ChatGPT model, Bhatia (2023) noted that ChatGPT’s ambiguous linguistic outputs risk its generated content clarity, coherence and reliability. When ChatGPT-generated content is compared to texts created by humans, these constraints may result in academic text that is less contextually relevant and incomprehensible.

The decontextualisation and lack of clarity limitation in ChatGPT affect its generated content precision, unity, and credibility. The contextual misunderstanding limitation is particularly pronounced in complex academic disciplines, where the subtleties of language can significantly alter meaning. Elhag et al. (2023) pointed out that the ChatGPT’s outputs stemming from misunderstanding its input could jeopardise the integrity of academic communication, particularly when researchers rely on ChatGPT to write or evaluate sophisticated arguments without sufficient human supervision. For example, ChatGPT can produce believable medical advice or instructional content; however, its incapability to appropriately contextualise information or validate facts could negatively affect users who depend on it to make important decisions (Chowdhury & Haque, 2023). Some studies confirmed that ChatGPT's performance has been shown to vary dramatically based on the specificity and complexity of the prompts it receives, leading to the risk of several misinterpretations (Suárez et al., 2023 & Yıldırım & Tellou, 2024). Although some literature declares that AI models like ChatGPT can facilitate certain writing tasks, their rigid, ambiguous output may deter students from developing a meaningful, fluid and adaptive writing style essential for higher-level academic success (Jiang & Hyland, 2024). Datskiv et al. (2024) concluded that ChatGPT's limitations in replicating nuanced academic language and complex argumentation structures can develop ambiguity, lead to reduced coherence in generated texts, and impact the clarity and depth of academic writing.

In conclusion, ChatGPT offers potential benefits, such as enhancing the writing pace and providing language support; however, its integration into academic writing presents significant limitations that must be addressed. ChatGPT’s multifaceted limitations extend from linguistic limitations to issues about the accuracy, reliability, and credibility of its output. Although experts and researchers in the field believe that ChatGPT limitations raise significant concerns about its impact on students’ critical and analytical thinking, the integrity of students’ academic work, and students’ irresponsible misuse of technology, they overlooked the techniques of how to mitigate those limitations to leverage its use while maintaining academic integrity and data validity. In addition, despite the prior observations and statements in research about the need for guidelines to refine students’ utilisation of ChatGPT, it remains unclear what those guidelines are and how they can be integrated into academic writing instruction. Moreover, the presented literature extensively studied students’ behaviour and attitudes while using ChatGPT in their academic writing assessments; however, none of the previous research has examined students’ awareness of ChatGPT limitations to be considered one of the factors behind their potential misuse and irresponsible integration of ChatGPT in their academic writing. Therefore, this study intends to measure the effectiveness of the researcher-designed critical review tasks on students’ critical awareness of the limitations of ChatGPT.  The research questions are “What are the ChatGPT limitation areas required to be recognised and considered by students?”, “What is the design of the critical review activity required to raise students’ critical awareness of ChatGPT limitations?” and “How effectively could the designed critical review activity, integrated into the academic writing course, develop students’ critical awareness of the ChatGPT limitations? The research uses the quasi-experimental method of one experimental group. The research participants practice the critical review activity integrated into the academic writing course for a semester. Using the pre- and post-questionnaire for data collection, results are discussed in terms of the development of students’ critical awareness of ChatGPT limitations and students’ recognition of ChatGPT limitations.

3. RESEARCH METHODOLOGY

3.1 Research Design

Quasi-experimental research was conducted to measure the effectiveness of using a researcher-designed five-step critical review activity while reading ChatGPT responses in raising university students’ critical awareness of ChatGPT limitations in academic writing. For data collection, a quantitative approach was adopted in the study using a pre-post ChatGPT Limitations Critical Awareness Questionnaire to assess university students’ level of awareness and recognition of ChatGPT model limitations in Academic Writing.

The quasi-experimental research design was conducted in three phases: the pre-administration of the ChatGPT Limitations Critical Awareness questionnaire, the integration of critical review activities while using ChatGPT for in-class academic writing tasks for a semester (three months), and the post-administration of the Students’ Critical Awareness of ChatGPT Limitations questionnaire.

3.2 Statistical Analysis

Data was collected, cleaned and organised by the researcher. The data was run by the SSPS (Statistical Package for the Social Sciences) software to compute the statistical analysis tests and obtain results. Descriptive and comparative analysis methods were employed to evaluate the data collected on students’ critical awareness of ChatGPT Limitations. The descriptive analysis tests included calculating mode, means, median and standard deviations to present the data. The comparative analysis included calculating the T-value and the P-value using a t-test to measure the difference between pre-questionnaire and post-questionnaire scores. The p-value was compared to a set significance level (usually 0.05) to decide whether the results were statistically significant or non-significant. To measure the effect of the proposed treatment "researcher-designed critical review activity", the effect size of the suggested treatment was calculated by using the Eta square formula.

3.3 Research Setting

The research was conducted for four months in the second semester of the 2023/2024 academic year, from February 2024 to May 2024, at the University of Prince Edward Island, Cairo Campus, hosted by the University of Canada in Egypt (UPEI-UCE). The study limited itself to the type of international campus universities in Egypt, which educate both Egyptian and international students.

3.4 Research Participants

The participants of the study were selected based on two criteria. Firstly, they had to be enrolled in the Academic Writing Subject (Writing Studies) and use ChatGPT in their process of in-class academic writing tasks. Secondly, they had to come from the first or the second year in the three schools: Computer Science, Engineering, and Business School in the UPEI-UCE. The number of participants was 120 students in total, randomly enrolled in three sections of a mix of the three majors and a blend of first- and second-year students. Seventy-five students were from business majors, 27 students were from computer science majors, and 18 students were from engineering majors. The questionnaire in a Microsoft Outlook online form was sent to all the participants via email and Outlook Teams. The questionnaire yielded responses from 120 students, among 44 second-year and 76 first-year students.

3.5 Research Framework

As educators grapple with the implications of integrating tools like ChatGPT into their curricula, a focus on fostering critical thinking becomes paramount; after all, student essays should reflect more nuanced epistemic stances that are essential for persuasive argumentation (Markey et al., 2024). The challenge lies not only in leveraging these technologies effectively but also in ensuring that they enhance—rather than diminish—the development of vital rhetorical skills among learners. As researchers continue to explore these limitations, it becomes increasingly clear that a collaborative approach involving human expertise is essential to mitigate potential biases and enhance the utility of language models like ChatGPT in specialised domains. Therefore, fostering a critical analysis approach towards the use of such technologies is essential, ensuring that human oversight remains central to the research process and that any generated material undergoes rigorous scrutiny before publication (Osama & Maaz, 2023).

The critical analysis approach of a text involves analysis and evaluation rather than passively absorbing information. It encourages students to analyse text features and composition structure, question assumptions and views, analyse logic, reasoning and arguments, evaluate sources, and recognise biases and fallacies through interaction with the text. Thus, based on the critical analysis approach, the researcher designed a five-step critical review activity for students to practice doing while using ChatGPT in their in-class academic writing tasks. The critical review activity steps are 1) annotation, 2) comparative analysis, 3) argument analysis, 4) bias identification, and 5) language analysis.

In the academic semester of 13 teaching weeks, students attended two lectures each week. In the first week of the academic writing course, students were introduced to the course content orientation in the first lecture, and they filled in two online forms in the second lecture: the Pre-ChatGPT Limitations Awareness Questionnaire and the consent form to participate in the research study. During the second week of lectures, students were introduced to the ChatGPT limitations forms and the researcher-designed five-step critical review activity procedures. For the next ten weeks, students attended lectures to write five in-class writing tasks. Those in-class academic writing tasks included a short personal narrative, a reflective essay, an analytical essay of arguments, a problem-solution essay focusing on solutions, and a summary essay of an article.

Students spent four lectures to write each in-class academic writing task. Students were divided to work in pairs to finish each in-class writing task; pairs were shuffled for each in-class writing task to enhance intellectual cooperation and scaffold writing skills development. Each in-class writing task was monitored by the instructor (the researcher of this study). Students were instructed to use ChatGPT during their writing process in class. They were encouraged to use it to brainstorm ideas, outline their writing tasks, gain knowledge and information, and find research evidence and sources while maintaining academic integrity. Students did the five-step critical review activity every time ChatGPT provided them with a written text response for their prompts. Meanwhile, the instructor provided guidance, clarification, help and support to facilitate the activity and ensure doing the activity steps correctly. In the last week of the semester, students filled in the online form of the Post ChatGPT Limitations Awareness Questionnaire during the lecture while monitored by the instructor to ensure receiving the participants’ responses.

4. RESEARCH INSTRUMENT

4.1 Questionnaire Design

The study utilised a researcher-designed survey questionnaire, which underwent reliability and validity testing. The researcher developed a questionnaire based on four main ChatGPT limitation forms in Academic Writing, including “Formulaic Language and Monotonous Writing Style”, “Decontextualisation & Linguistic Ambiguity”, “Hallucinations & Inaccuracies”, and “Model Inherent Biases”, which were identified in the study literature. The online questionnaire form has two parts: the first part is Demographic information, and the second part is the Students’ Critical Awareness Questionnaire of ChatGPT Limitations in Academic Writing. The second part (Closed Questionnaire) includes 16 statements covering the four main ChatGPT limitations mentioned above. Each statement begins with “I can recognise ….” to assess students’ critical awareness of the four mentioned limitations. The questionnaire method for collecting responses was categorical, allowing students to choose one response of three categories of responses: “Yes”, “No”, and “Not Sure”.  The questionnaire scores were recorded as Yes = 2 Points, No = 0 Points, and Not Sure = 0 Points. Each student’s responses were scored with a total score of 32 Points. See Appendix A.

• Questionnaire Reliability

The questionnaire was piloted, followed by a test of reliability and internal consistency. The reliability test of data was examined by Cronbach’s alpha coefficient. The subscales of the four limitation forms were above 0.7, and the Cronbach’s alpha coefficient value of the total was 0.937, which indicated that the reliability of the scale was relatively high. The level of reliability was 0.893 for the limitation “Formulaic Language & Monotonous Writing Style”, 0.935 for the limitation “Decontextualization & Language Ambiguity”, 0.967 for the limitation “Hallucinations Inaccuracy”, and 0.953 for the final limitation “Inherent Model’s Biases”.

4.3 Data Collection

To ensure that equal questionnaire responses were collected, both the pre- and post-questionnaire administrations were done through an online Microsoft Outlook form while being monitored in class. At the beginning of the semester, the research participants filled in the ChatGPT Limitations Critical Awareness pre-questionnaire form online while being monitored in class. The research participants, enrolled in the academic writing subject, practised the five-step critical review activity during the semester while using ChatGPT to do five in-class academic writing tasks. Finally, they filled in the ChatGPT Limitations Critical Awareness post-questionnaire form online while being monitored in class.

5. RESULTS

Students’ critical awareness of ChatGPT limitations was assessed by a researcher-designed pre- and post-questionnaire, based on the literature on ChatGPT limitations in academic writing. The researcher collected, cleaned and organised the data to run a descriptive and comparative analysis and reach conclusions.

The “t” test formula below was employed to difference between the experimental group mean scores on the Pre and Post-Administrations of the Students’ Critical Awareness Questionnaire of ChatGPT Limitations.

.

Where: 

"t" =is the calculated value of the difference between mean scores, 

Dm = the general mean score of the differences between pairs of scores, and 

Dd = the deviation of the differences from their general mean score. 

See the table in the next page: 

Table 5.1: "t" Value of the difference between the experimental group mean scores on the Pre and Post-Administrations of the Students’ Critical Awareness Questionnaire of ChatGPT Limitations

The above table indicates the following: 

1- The "t" calculated value of the overall Students’ Critical Awareness Questionnaire of ChatGPT Limitations (136.86) was higher than the “t” tabled Value (2.457) with (34) degree of freedom at the (0.01) level of significance. So, there was a statistically significant difference between the participants' mean scores on the pre and post administrations of the Students’ Critical Awareness Questionnaire of ChatGPT Limitations.

2- The mean score of the participants on the post-administration of the Students’ Critical Awareness Questionnaire of ChatGPT Limitations (122.71) was higher than their mean score on the pre-administration (50.14). Thus, the overall development of the Critical Awareness of ChatGPT Limitations of the participants was due to the researcher-designed critical review activity during the experiment.

Further, the “t” test formula was used to decide the relative extent of differences between the mean scores of the participants in each of the four ChatGPT limitations, nurtured by the implementation of the researcher-designed critical review activity.

See the table below: 

Table 5.2: "t" Value of the difference between Mean Scores of the Participants on the Pre and Post-Administrations in each of the four ChatGPT limitations.

The table above confirms the following: 

a- There is a statistically significant difference between the mean score of the participants in the post-questionnaire administration on the Hallucinations and Inaccuracies Limitation at its basic level (21.71) and the mean score of the pre-questionnaire administration (9.54) at the level of (0.1). 

b- There is a statistical difference between the mean score of the participants in the post-questionnaire administration on the Model’s Inherent Biases at its basic level (25.20) and the mean score of the pre-questionnaire administration (12.83) at the level of (0.1). 

c- There is a statistical difference between the mean score of the participants in the post-questionnaire administration on Formulaic Language and Monotonous Writing Style Limitation at its basic level (25.09) and the mean score of the pre-questionnaire administration (8.57) at the level of (0.1).

d- There is a statistical difference between the mean score of the participants in the post-questionnaire administration on Decontextualization Language Ambiguity Limitation at its basic level (25.06) and the mean score of the pre-questionnaire administration (8.06) at the level of (0.1).

To measure the effect of the proposed treatment "researcher-designed critical review activity", the effect size of the suggested treatment was calculated by using the Eta square formula (η²) as follows,

See the table below: 

Table 5.3: The Calculated (η²) and the Effect Size for the Students’ Critical Awareness Questionnaire of ChatGPT Limitations

As indicated in Table (5.3), the calculated (η²) value of the researcher-designed critical review activity on the experimental group’s overall critical awareness of ChatGPT Limitations was 0.998. Thus, a large size of effect, on the development of the critical awareness of ChatGPT Limitations of the experimental group was evidenced, due to utilizing the researcher-designed critical review activity during the experiment.

To measure the effect of the proposed "researcher-designed critical review activity", the effect size of the proposed treatment on students' critical awareness of each of the four ChatGPT limitations was calculated by using the Eta square (η2) formula as follows:

See the table below: 

Table 5.4: The calculated Eta square and the Effect Size of the proposed researcher-designed critical review activity on students' critical awareness of each of the four ChatGPT limitations.

As indicated in Table (5.4) the calculated Eta square values of the proposed treatment "researcher-designed critical review activity" on students’ critical awareness of each of the four ChatGPT limitations were the Hallucinations and Inaccuracies Limitation (0.992), the Inherent Model’s Biases Limitation (0.989), the Formulaic Language and Monotonous Writing Style Limitation (0.995), and the Decontextualization and Language Ambiguity Limitation (0.988), respectively. Thus, a large size of effect on the development of the students’ critical awareness in the four ChatGPT Limitations was evidenced due to implementing the proposed treatment "researcher-designed critical review activity" used during the experiment.

6. CONCLUSION

University students' growing dependence on ChatGPT for academic writing requires a thorough examination of its limitations. This study aims to help students develop a critical awareness and deep understanding of the linguistic and factual flaws in ChatGPT responses. Through the use of a critical review activity created by the researcher, the study successfully involved participants in assessing the appropriateness and reliability of information produced by ChatGPT. The necessity of including critical thinking activities in academic writing courses is highlighted by the significant improvement in students' critical awareness that was revealed by analysing the data collected by the students’ critical awareness of ChatGPT Limitations pre and post-questionnaire.

These results support integrating educational materials that develop students’ critical attitude towards AI-generated content, give them the right critical tools they need to evaluate academic content generated by ChatGPT and enhance their academic writing potential. The findings also emphasise the importance of fostering critical active engagement over passive dependence behaviour, which could be unintentionally developed due to the impressive plausible display of the AI tools.  This research contributes to the ongoing discourse on the role of artificial intelligence in education while preparing students intellectually and critically to willingly adopt mindful perspectives towards the invasive wave of AI technology.

Putting these ideas into practice will be crucial as educational institutions develop further to prepare a generation of students who are not just skilled writers but also information evaluators. Educators can enable students to take an active role in their education by including these analytical tools into the curriculum. This will allow students to successfully question, analyse, and synthesise material and become proactive critical users of AI technology models. By encouraging student cooperation, this transformative approach creates a classroom climate that values technological development while fostering critical thinking. In the end, such thorough research may help educators understand how to best incorporate AI technologies into the curriculum, creating an atmosphere that encourages critical use of technology while improving academic results.

Future studies should examine the effects of including critical review exercises for ChatGPT replies in academic subjects other than academic writing courses. Examining how these exercises affect students' general critical thinking abilities, research potentials, and academic achievement may yield insightful information. Studies could also look at how well various teaching strategies and resources work to raise students' critical awareness of AI-generated content. To evaluate the generalizability of the results, it would also be advantageous to include a wide range of participants from different educational institutions and backgrounds. Lastly, to ensure that teaching methods stay applicable and successful in a learning environment that is becoming more and more digital, longitudinal research could monitor how students' attitudes and abilities toward AI technologies like ChatGPT change over time.