ChatGPT 4o System Prompt Leak
Yesterday, LLMrefs leaked the "rule book" that sits behind ChatGPT-4o. These are the built-in instructions, injected by OpenAI, right at the top of the context window for any conversation with ChatGPT.
These system instructions are normally hidden in public-facing UIs and protected from simple prompt injection attacks. We're now able to get a detailed look at how ChatGPT operates.
Here's a copy of the full system prompt:
So... let's break it down 👇
How does ChatGPT find citations?
ChatGPT has access to a built-in web() tool "to access up-to-date information from the web or when responding to the user requires information about their location."
Once a user sends a message to ChatGPT, it must first decide whether or not to use this web search tool. You'll notice here there are four reasons which would cause it to use web search; local context, very fresh data, niche knowledge, or extra verification for accuracy.
The system prompt shows that ChatGPT spends almost all of its time answering from memory. It only goes out to the live web in a few clearly-defined situations, and when it does, it follows a very precise playbook.
| Scenario | Examples | Breakdown |
|---|---|---|
| Real-time facts | "What's the score right now?", "Tomorrow's weather" | The information changes minute-to-minute. |
| Location-specific requests | "Find me cafés near the Eiffel Tower" | Needs your location or fresh local listings. |
| Niche or obscure topics | A brand-new research paper, a local club's fixtures | May never have appeared in training data. |
| Out-of-date info could be harmful | Installing the current version of a software library | Outdated advice could break things or cause harm. |
Anything that doesn't fall into those buckets is answered from the model's knowledge, built up from it's training data. Learn more about how large language models work.
Infact the system prompt explicity states "Avoid using the `web` tool for information already known to be available internally unless the user explicitly indicates preference for public sources."
How does the web search tool work?
Up to five parallel queries are fired off at once, so ChatGPT can skim different angles in a single pass.
Some imformation from the important words get a "+" boost in the query string so the search engine focuses on them.
Additionally each recieves a "Query Deserved Freshness" score, deciding how aggressively to favour very recent pages:
- --QDF=0 = information from 5+ years ago (eg. "the radius of the earth").
- --QDF=5 = imformation from past 30 days (eg. "weather in london today")
The system prompt also mentions if "the user's question is not in English, you must issue the above queries in both English and also translate the queries into the user's original language" to widen the chance of finding relevant sources.
We built a ChatGPT search query extractor Chrome extension to enable you to view the underlying fan-out queries sent by the web search tool.
No links in ChatGPT answers, why?
Ever noticed how you rarely see links in ChatGPT responses? This is because ChatGPT answered "offline", it decided to not use the web tool. Therefore proving that ChatGPT does not keep its own index of URLs.
If no search was triggered, any URL the model gives is guessed token-by-token from memory and that's why AI-generated links are often pure hallucination and lead to a 404 page.
An independent study by SISTRIX put the share of ChatGPT answers that contain a live, working link at ~6% this is far lower than Gemini (23%) or other chatbots.
Generative AI search optimization
To increase you brand's visibility in AI search it's very important to ensure that your content is indexed by Bing! Bing is the search engine behind the web tool. If not, when ChatGPT decides to trigger search, the web search tool has no chance of finding or using it as a source.
Is your content non-English? Publish an additional version in English to increase your chances of being cited.
Bonus: Track your citations in AI search and setup 301 redirects for commonly hallucinated URLs.
It's clear that most of the time ChatGPT isn't "online". It's quoting the huge snapshot of the web it learned during training. You should be seeding your brand name where LLMs are crawling new datasets.
A quick overview
The system prompt reveals that ChatGPT will only crawl the web when it must, such as live scores, current prices, local places, breaking news, etc. That's why you sometimes get an instant answer, sometimes see it "thinking" or "searching" while it fetches sources, and sometimes get you'll see no links at all.
James Berry
Founder & CEO at LLMrefs
llmrefs.com
Many thanks to Hanns Kronenberg for the initial analysis of the leak.