Limitations and responsible use of AI
Day 2 - 1:00 PM - 1:30 PM
2026-07-14
Responsible use and limitations of AI
With great power, comes great responsibility!
- What do you think ‘responsible use’ entails?
- What do you think are some limitations of AI?
They actually go hand in hand with each other. Understanding the limits of AI will allow you to responsibly use it.
First, let’s review and also learn a new term
If you remember from yesterday morning, LMs are trained to predict the next token or sequence of token, based on its training data.
Throughout this presentation, I will be using the terms comfortability and limitation zone. Comfortability zone is just like your own comfort zone! As you get near the edge of your comfort zone, you become less certain. For AI, it transitions into the limitation zone. AI models work the same way.
Search engine? More like autocomplete
AI models are trained to predict the next token, known as next token prediction. They are designed to predict what comes next based on training data.
What sorts of tasks or knowledge would you say would be within AI’s comfortability zone?
Examples:
- Knowledge contained within their training data
- Language refining, reformatting
- Code development for well-known coding languages
Next token prediction risks
Hallucination/frabication occur in AI models when it generates confident, well-structured responses that are actually false, misleading, or entirely made up. Fabrication is frequent in…
- Niche or rare information
- Novel tasks or information
- Obsecure task requests
Next token prediction risks (cont)
How can you protect yourself from the risks posed by next token prediction?
- You are the one interacting with the AI model, you are responsible for what you do with the response. The best thing you can do is VERIFY the information AI is providing you.
- Consider all output from AI as a draft that must be reviewed.
- If you ask for sources, ensure they are real. A link that works, does not mean the information on that site is relevant to what you asked.
- AI models DO NOT know if they are providing you with fabricated/hallucinated information.
Next token prediction risks (cont)
Let’s go through an example to test our knowledge of the comfortability and limitation zone.
From the two prompts below, which prompt would recieve the better response? Where would these two prompts fall on the comfortability/limitation circle?
- Summarize the second chapter of How to Kill a Mockingbird. Provide me key details I would need to know for a pop quiz.
- Provide the citations to the three most recent publications in the Plant Disease journal. Provide the affiliations of all of the authors from each citation.
Knowledge gaps in AI models
AI models are trained on datasets much larger than any human could comprehend. However, these datasets are finite, and training has a start and an end date. This leads to models that may not be up to date on current events.
How can you protect yourself
- Test the model to determine event accuracy if time sensitive
- Use tools, such as retrival-augmented generation, to gather recent data
What is working memory?
The working memory is basically what your AI model is paying attention to right now. Let’s go over what happens when AI is developing a response to your newest prompt.
- Previous prompts + responses + input data = context window
- AI looks at context window to create new response
So, why does working memory matter?
Similar to human memory, old prompts, responses and data is ‘forgotten’ or removed from the context window when conversations get lengthy.
- AI will not inform you when the context window begins to omit previous material.
- Working memory is not a spectrum. Previous content is either apart of the content window or its not.
Additional grey areas of AI
- Sycophany: The phenomenon where AI tells you what you want to hear. The actual definition of confirmation bias. Be aware of this!
- Verbosity: AI models are trained to give long, thorough responses by default. You may be able to recieve the same answer with less information.
- Confidence: Confidence is challenging to train, and remember, AI models cannot differentiate what it fabricates and what it fact. Take AI’s confidence with a grain of salt.
- The middle area: Research has shown content towards the beginning and end of a prompt or document will recieve higher weight/attention than content in the middle.
Financial/token responsibility
While you as a student or staff member may not consider the financial aspect of using AI, your usage could impact you or your PI’s wallet.
AI companies will impose limits on your usage. Take Github Copilot inline suggestions for example (right). This is a very common practice across all AI companies. Asking too many prompts or asking AI to complete a challenging can lead to limited access depending on the platform.
