Overview
Threat actors are conducting a mass disinformation and phishing campaign by deliberately poisoning the web content ingested by major AI chatbots — including ChatGPT, Gemini, and Google AI Overviews. By seeding the internet with malicious links and optimised content, attackers are manipulating AI-generated answers to serve fabricated information or direct users toward phishing infrastructure. The campaign illustrates a systemic vulnerability in how retrieval-augmented and web-connected AI systems trust external content.
Technical Analysis
The attack methodology exploits the retrieval pipeline underpinning modern AI assistants. When AI systems like Google AI Overviews or ChatGPT’s browsing-enabled mode query the web to supplement their responses, they surface content ranked highly by search-engine-like signals. Attackers are abusing this by:
- Web seeding: Publishing large volumes of attacker-controlled pages containing targeted disinformation or phishing lures.
- Content optimisation: Applying SEO-style techniques to boost the visibility and apparent authority of malicious pages, increasing the likelihood that AI retrieval systems index and surface them.
- Link manipulation: Embedding malicious URLs within content that AI systems may reproduce verbatim in their responses, directing users to phishing sites.
This is a practical instantiation of RAG Poisoning — an attack class where adversaries corrupt the external knowledge sources that retrieval-augmented generation systems depend upon. Unlike traditional prompt injection, this attack requires no access to the model itself; the attack surface is the open web.
Framework Mapping
MITRE ATLAS:
- AML.T0070 (RAG Poisoning) and AML.T0071 (False RAG Entry Injection) are the primary techniques — attackers are injecting false content into the retrieval corpus.
- AML.T0066 (Retrieval Content Crafting) describes the SEO optimisation used to elevate malicious content.
- AML.T0067 (LLM Trusted Output Components Manipulation) applies where AI outputs reproduce attacker-controlled links as trusted recommendations.
OWASP LLM Top 10:
- LLM03 (Training Data Poisoning) covers the web-content poisoning vector.
- LLM09 (Overreliance) is the user-side risk — victims trusting AI-generated answers without verification.
- LLM02 (Insecure Output Handling) applies where malicious URLs are rendered as actionable links.
Impact Assessment
The campaign affects any user querying ChatGPT, Gemini, or Google AI Overviews on topics targeted by attackers. The scale of potential harm is significant: AI chatbots are increasingly used as primary information sources, meaning a successfully poisoned answer can reach millions of users rapidly. Phishing links surfaced within AI-generated responses carry elevated credibility because users associate AI output with authority, reducing scepticism. Organisations relying on AI-assisted research workflows face elevated risk of staff being exposed to disinformation or credential-harvesting pages.
Mitigation & Recommendations
- For end users: Treat AI-generated answers referencing external URLs with the same scrutiny as unsolicited email links. Verify claims via primary sources.
- For security teams: Monitor for AI-tool-sourced URLs in browser telemetry and flag domains associated with known malicious infrastructure.
- For AI providers: Strengthen retrieval pipeline integrity checks, implement domain reputation scoring for cited sources, and increase transparency around which sources inform AI answers.
- For enterprises: Restrict use of web-browsing AI features in sensitive research workflows until provider-side mitigations are confirmed.