Overreliance

7 reports

All LLM Security Agentic AI Industry News Research Supply Chain Prompt Injection First Look: Security Regulatory Adversarial ML Jailbreaks Model Theft Data Poisoning AI Security Tools Security Operations

Y Combinator Ships Agentic Code Generation at 37K Lines Daily

FIRST LOOK ATLAS OWASP MEDIUM ▲ 5.8 HN AI Security Jul 08, 2026

Y Combinator CEO Garry Tan has publicly claimed to ship approximately 37,000 lines of AI-generated code per day using agentic coding tools, and an independent developer analysis has revealed the underlying mechanics of this workflow. This level of AI-assisted code velocity introduces meaningful security concerns around code provenance, supply chain integrity, and the reduced human review time per line of shipped code. Defenders should treat high-velocity AI code pipelines as a new supply chain risk category requiring dedicated SAST/DAST tooling and policy controls.

Phantom Squatting: LLM Hallucinations Enable Domain Takeover

ATLAS OWASP HIGH ▲ 8.2 Dark Reading Jul 02, 2026

Researchers have identified a novel attack vector dubbed 'Phantom Squatting', in which LLMs consistently hallucinate plausible but non-existent web domains for legitimate brands, which attackers can then register and weaponise. Unlike traditional typosquatting, these hallucinated domains carry implicit trust because they originate from AI-generated outputs that users and developers may act upon without verification. The technique is difficult to detect because the domains are not misspellings but plausible inventions, making automated defences less effective.

Dragos Launches EmberAI, an OT-Specific AI Platform

FIRST LOOK ATLAS OWASP HIGH ▲ 7.2 SecurityWeek Jun 24, 2026

Dragos has launched EmberAI, an AI module embedded within its OT security platform that allows analysts to query threat intelligence, asset data, and network activity in plain language, grounded in a decade of proprietary OT-specific data. The system introduces new attack surface considerations because it aggregates highly sensitive OT network telemetry, vulnerability data, and adversary intelligence into a single AI-queryable layer — making the platform itself a high-value target. Defenders must weigh the risks of prompt injection, over-reliance on AI-generated recommendations in safety-critical environments, and the intelligence value this consolidated dataset represents to nation-state adversaries.

OpenAI Ships GPT-5.5 Instant with Health Intelligence

FIRST LOOK ATLAS OWASP MEDIUM ▲ 5.8 OpenAI Blog Jun 21, 2026

OpenAI has upgraded ChatGPT's health and wellness response capabilities via GPT-5.5 Instant, incorporating stronger reasoning, physician-informed evaluations, and improved contextual understanding for medical queries. This expansion into high-stakes health guidance raises meaningful concerns for defenders, as improved fluency and authority in medical responses increases the risk of user overreliance and lowers the perceived threshold for trusting AI-generated health advice. Security and trust-safety teams should evaluate how this capability interacts with prompt injection, social engineering chains, and the broader risk of AI-mediated medical misinformation at scale.

Constraint Decay: LLM Code Agents Fail at Scale

ATLAS OWASP HIGH ▲ 7.2 HN AI Security May 25, 2026

A systematic study of LLM agents performing backend code generation reveals a 'constraint decay' phenomenon where agents lose up to 30 assertion pass-rate points as structural requirements accumulate, approaching complete failure in some configurations. This fragility has direct security implications: production deployments relying on LLM-generated code may silently violate architectural constraints such as ORM patterns, database access controls, and API contracts. The findings expose a critical gap between functional correctness and structural safety in agentic coding systems.

LLM Agents Exploit Human Over-Trust in Strategic Games

ATLAS OWASP MEDIUM ▲ 6.2 Schneier on Security Apr 17, 2026

Research published via Schneier on Security reveals that humans systematically over-trust LLMs in strategic game environments, defaulting to Nash-equilibrium rational play based on assumptions of LLM rationality and cooperation. This behavioural bias has direct security implications for mixed human-LLM systems, where adversaries could exploit predictable human over-trust to manipulate decision outcomes. The findings underscore systemic risks in deploying LLMs as agents in high-stakes economic or security-relevant decision loops.

Claude Code Source Leak Exposes 512K Lines of Code

ATLAS OWASP MEDIUM ▲ 6.2 HN AI Security Apr 16, 2026

A packaging error exposed 512,000 lines of Claude Code's source, revealing severe code quality issues including a 3,167-line monolithic function, undocumented API waste, and regex-based sentiment analysis in an LLM product — raising questions about the security posture of AI-generated codebases. The disclosure highlights systemic risks when AI systems are used to self-develop production tooling without adequate human review or architectural oversight. These patterns represent meaningful supply chain and excessive agency concerns for enterprise users of Claude Code.