At a Glance

Generative AI has the potential to transform higher education—but it’s not without its pitfalls. These technology tools can generate content that’s skewed or misleading (Generative AI Working Group, n.d.; Cano et al., 2023). They’ve been shown to produce images and text that perpetuate biases related to gender, race (Nicoletti & Bass, 2023), political affiliation (Heikkilä, 2023), and more. As generative AI becomes further ingrained into higher education, it’s important to be intentional about how we navigate its complexities.

Biased Content

Problems with bias in AI systems predate generative AI tools. For example, in the Gender Shades project, Buolamwini (2017) tested AI-based commercial gender classification systems and found significant disparities in accuracy across different genders and skin types. These systems performed better on male and lighter-skinned faces than others. The largest disparity was found in darker-skinned females, where error rates were notably high.

Generative AI tools present similar problems. For example, a 2023 analysis of more than 5,000 images created with the generative AI tool Stable Diffusion found that it simultaneously amplifies both gender and racial stereotypes (Nicoletti & Bass, 2023). These generative AI biases can have real-world consequences. For instance, adding biased generative AI to “virtual sketch artist” software used by police departments could “put already over-targeted populations at an even increased risk of harm ranging from physical injury to unlawful imprisonment” (Luccioni et al., 2023). There’s also a risk that the veneer of objectivity that comes with technology tools could make people less willing to acknowledge the problem of biased outputs (Nicoletti & Bass, 2023). These issues aren’t unique to image generators, either; researchers and users have found that text generators like ChatGPT may also produce harmful and biased content (Germain, 2023).

AI bias research has continued to expand since 2023, with growing evidence that LLMs exhibit systematic bias not just in image generation but in text, including in tasks directly relevant to teaching. Studies have shown that language models produce different quality feedback, grading assessments, and professional recommendations depending on demographic cues embedded in names or writing styles. Stanford HAI’s AI Index Report 2025 documents ongoing bias concerns across multiple domains and model families (Stanford HAI, 2025). A 2025 study in Scientific Reports found that Stable Diffusion homogenized depictions of Middle Eastern men, assigning them traditional cultural attributes regardless of professional context (AlDahoul et al., 2025). And a UNESCO analysis of major LLMs found that women were described in domestic roles four times more often than men (UNESCO & IRCAI, 2024). These findings reinforce the importance of reviewing AI outputs for bias, particularly before sharing with students.

Inaccurate Content

Generative AI tools can produce fabricated information that appears authentic—a problem widely known as “hallucination” (Generative AI Working Group, n.d.). A Stanford HAI study found that general-purpose AI chatbots hallucinated on 58–82% of legal research queries when tested on 2023-era models; even specialized legal AI tools built on retrieval-augmented generation (RAG), a technique that grounds AI responses in a curated document database, hallucinated more than 17% of the time (Magesh et al., 2025).

Since then, the picture has grown more complicated. In a 2025 Vals AI report, researchers found that ChatGPT with search achieved roughly 80% accuracy, comparable to specialized legal AI tools (Vals AI, 2025). Despite this increased accuracy, recent evaluations have indicated that advanced reasoning models still do not consistently eliminate hallucinations; during multi-step reasoning, the models make “strategic guesses” where they generate plausible but false statements when uncertain (OpenAI, 2025). Furthermore, holistic performance audits have found that even as reasoning capabilities advance, model reliability remains inconsistent across different knowledge domains, failing to provide a guaranteed safeguard against factual inaccuracies (Stanford Center for Research on Foundation Models, 2025).

For an example of how AI hallucinations can play out in the real world, consider the legal case of Mata v. Avianca. In this case, a New York attorney representing a client’s injury claim relied on ChatGPT to conduct his legal research. The federal judge overseeing the suit noted that the opinion contained internal citations and quotes that were nonexistent. Not only did the chatbot make them up, it even stipulated they were available in major legal databases (Weiser, 2023). This case was among the first widely publicized examples of AI hallucination in legal proceedings, but far from the last. Between 2023 to 2025, judges worldwide issued hundreds of decisions addressing hallucinations in court filings, with roughly 90% (790 of 863) recorded in that 2025 alone (Charlotin, n.d.). Beyond the courtroom, a 2025 analysis found hallucinated citations in dozens of papers accepted at NeurIPS, one of the most competitive venues in AI research itself (GPTZero, 2025). Hallucination is now considered a serious, cross-domain challenge, and verifying AI-generated citations is non-negotiable regardless of context.

Why is AI Flawed?

Generative AI systems can produce inaccurate and biased content for several reasons:

  1. Training Data Sources: Generative AI models are trained on vast amounts of internet data. This data, while rich in information, contains both accurate and inaccurate content, as well as societal and cultural biases. Since these models mimic patterns in their training data without discerning truth, they can reproduce any falsehoods or biases present in that data (Weise & Metz, 2023).
  2. Limitations of Generative Models: Generative AI models function like advanced autocomplete tools: They’re designed to predict the next word or sequence based on observed patterns. Their goal is to generate plausible content, not to verify its truth. That means any accuracy in their outputs is often coincidental. As a result, they might produce content that sounds reasonable but is inaccurate (O’Brien, 2023).
  3. Inherent Challenges in AI Design: The technology behind generative AI tools isn’t designed to differentiate between what’s true and what’s not true. Even if generative AI models were trained solely on accurate data, their generative nature would mean they could still produce new, potentially inaccurate content by combining patterns in unexpected ways (Weise & Metz, 2023).

In short, the “hallucinations” and biases in generative AI outputs result from the nature of their training data, the tools’ design focus on pattern-based content generation, and the inherent limitations of AI technology. Acknowledging and addressing these challenges will be essential as generative AI systems become more integrated into decision-making processes across various sectors.

Navigate AI’s Pitfalls

Consider these strategies to help mitigate generative AI tools’ issues with hallucination and bias.

  1. Critically Evaluate AI Outputs: Unlike humans, AI systems do not have the ability to think or form beliefs. They operate algorithmically based on their training data, without any inherent capacity for reasoning or reflection. Given this context, users must approach AI outputs with a critical eye and evaluate them with human judgement (Silberg & Manyika, 2019).
  2. Diversify Your Sources: Always double check the accuracy of AI-generated content. This could mean consulting with experts or cross-referencing with peer-reviewed publications that you access through the MIT Libraries.
  3. Use Retrieval-Based Tools: Some generative AI tools are built with Retrieval-Augmented Generation (RAG) architectures. This means they’ll retrieve relevant information from trusted sources—such as your syllabus, research article, or case PDF—before generating output. Research has shown that RAG improves both factual accuracy and user trust in AI-generated answers (Li et al., 2024).
  4. Use Clear and Structured Prompts: The quality of AI output is closely tied to how specific your input is. Vague prompts often lead to vague—or even inaccurate—answers. You can reduce this risk by setting clear expectations and giving the model a structure to follow. For example, prompting the AI to explain its reasoning step-by-step can expose logical gaps or unsupported claims. This technique, known as Chain-of-Thought Prompting, has been shown to improve transparency and accuracy in complex tasks (Wei et al., 2022).
  5. Adjust the Tool’s Temperature: Temperature is a setting that controls how random or creative the model’s responses are. In tools that allow you to adjust it, using a low temperature (e.g., 0–0.3) produces more focused, consistent, and factual outputs—especially for well-defined prompts. A higher temperature (e.g., 0.7–1.0) encourages more varied and imaginative responses, making it better suited for open-ended tasks like brainstorming or storytelling.

Conclusion

Generative AI offers great potential to improve how we teach, research, and operate. However, it’s essential to remember that AI tools can produce falsehoods and amplify harmful biases. While AI is a powerful tool, the human touch remains crucial. By working together, we can make the most of what AI offers while mitigating its known limitations.

References

AlDahoul, N., Rahwan, T., & Zaki, Y. (2025). AI-generated faces influence gender stereotypes and racial homogenization. Scientific Reports, 15(1), Article 14449. https://doi.org/10.1038/s41598-025-99623-3

Ayala, O., & Béchard, P. (2024). Reducing hallucination in structured outputs via retrieval-augmented generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track) (pp. 228–238). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.naacl-industry.19

Buolamwini, J. (2017). Gender shades: Intersectional phenotypic and demographic evaluation of face datasets and gender classifiers. DSpace@MIT. https://dspace.mit.edu/handle/1721.1/114068

Cano, Y. M., Venuti, F., & Martinez, R. H. (2023). ChatGPT and AI text generators: Should academia adapt or resist? Harvard Business Publishing. https://hbsp.harvard.edu/inspiring-minds/chatgpt-and-ai-text-generators-should-academia-adapt-or-resist

Charlotin, D. (n.d.). AI hallucination cases database. https://www.damiencharlotin.com/hallucinations/

Generative AI Working Group. (n.d.) How can we counteract generative AI’s hallucinations? Digital, Data, and Design Institute at Harvard. https://d3.harvard.edu/how-can-we-counteract-generative-ais-hallucinations

Germain, T. (2023, April 13). ‘They’re all so dirty and smelly:’ study unlocks ChatGPT’s inner racist. Gizmodo. https://gizmodo.com/chatgpt-ai-openai-study-frees-chat-gpt-inner-racist-1850333646

GPTZero. (2025). Hallucinated citations in NeurIPS papers. https://gptzero.me/news/neurips/

Heikkilä, M. (2023, August 8). AI language models are rife with different political biases. MIT Technology Review. https://www.technologyreview.com/2023/08/07/1077324/ai-language-models-are-rife-with-political-biases

Li, J., Yuan, Y., & Zhang, Z. (2024). Enhancing LLM factual accuracy with RAG to counter hallucinations: A case study on domain-specific queries in private knowledge-bases. arXiv. https://arxiv.org/abs/2403.10446

Magesh, V., Surani, F., Dahl, M., Suzgun, M., Manning, C. D., & Ho, D. E. (2025). Hallucination‐free? Assessing the reliability of leading AI legal research tools. Journal of Empirical Legal Studies, 22(2), 216–242. https://doi.org/10.1111/jels.12413

Nicoletti, L., & Bass, D. (2023, June 14). Humans are biased. Generative AI is even worse. Bloomberg Technology + Equality. https://www.bloomberg.com/graphics/2023-generative-ai-bias

O’Brien, M. (2023, August 1.) Chatbots sometimes make things up. Is AI’s hallucination problem fixable? AP News. https://apnews.com/article/artificial-intelligence-hallucination-chatbots-chatgpt-falsehoods-ac4672c5b06e6f91050aa46ee731bcf4

OpenAI. (2025, September 10). Why language models hallucinate. https://openai.com/index/why-language-models-hallucinate/

Silberg, J., & Manyika, J. (2019, June 6). Tackling bias in artificial intelligence (and in humans). McKinsey & Company. https://www.mckinsey.com/featured-insights/artificial-intelligence/tackling-bias-in-artificial-intelligence-and-in-humans

Stanford Center for Research on Foundation Models. (2025, March 20). HELM Capabilities: Evaluating LMs capability by capability. https://crfm.stanford.edu/2025/03/20/helm-capabilities.html

Stanford HAI. (2025). AI Index Report 2025. https://aiindex.stanford.edu/report/

UNESCO & IRCAI. (2024). Challenging systematic prejudices: An Investigation into Gender in Large Language Models. United Nations Educational, Scientific and Cultural Organization. https://discovery.ucl.ac.uk/10188772/1/Unesco_results.pdf

Vals AI. (2025, October 14). Vals Legal AI Report: An investigation into legal research accuracy. https://www.vals.ai/industry-reports/vlair-10-14-25

Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., et al. (2022). Chain of thought prompting elicits reasoning in large language models. arXiv. https://arxiv.org/abs/2201.11903

Weise, K., & Metz, C. (2023, May 1). When A.I. chatbots hallucinate. The New York Times. https://www.nytimes.com/2023/05/01/business/ai-chatbots-hallucination.html

Weiser, B. (2023, May 27). Here’s what happens when your lawyer uses ChatGPT. The New York Times. https://www.nytimes.com/2023/05/27/nyregion/avianca-airline-lawsuit-chatgpt.html