Adversarial ML

28 reports

All LLM Security Agentic AI Industry News Research Supply Chain Prompt Injection First Look: Security Regulatory Jailbreaks Adversarial ML Data Poisoning Model Theft AI Security Tools Security Operations

CogCAPTCHA30 Fingerprints AI Agents via Behavioral Analysis

ATLAS OWASP MEDIUM ▲ 6.5 HN AI Security May 31, 2026

Researchers have developed CogCAPTCHA30, a 30-task cognitive battery demonstrating that AI agents (GPT, Claude, Gemini) solve CAPTCHAs with statistically distinguishable behavioural patterns despite matching human accuracy. The study introduces a 'Process Turing Test' concept, showing output equivalence and process equivalence are uncorrelated — meaning AI agents can be detected not by what they answer, but by how they answer. This has direct implications for bot detection, anti-automation defences, and the arms race between AI-driven agents and human-verification systems.

GreyVibe Deploys ChatGPT and Gemini in LLM Attack Chain

ATLAS OWASP HIGH ▲ 7.5 SecurityWeek May 29, 2026

WithSecure has documented GreyVibe, a Russia-nexus threat actor systematically deploying ChatGPT, Google Gemini, and Ideogram AI across every phase of its attack chain — from phishing lure creation to custom malware development — against Ukrainian targets since August 2025. The group's LLM-assisted malware, LegionRelay, contained design flaws introduced during AI-generated development, which paradoxically allowed researchers to track the group over an extended period. The case illustrates both the operational leverage AI provides to moderately skilled threat actors and the novel forensic signatures that AI-assisted development can inadvertently introduce.

DeepSeek Activation Steering Enables Local LLM Jailbreak

ATLAS OWASP MEDIUM ▲ 6.2 HN AI Security May 17, 2026

Activation steering — the technique of directly manipulating LLM internal representations mid-inference to alter model behaviour — is becoming more accessible to non-lab engineers via local models like DeepSeek-V4-Flash. This democratisation lowers the barrier for adversaries to craft targeted behavioural overrides that bypass prompt-level safety controls. The emergence of first-class steering support in tools like DwarfStar 4 signals that model-internal manipulation is transitioning from academic curiosity to practical attack surface.

PromptSpy Zero-Day: AI-Generated Malware for Mass Exploitation

ATLAS OWASP CRITICAL ▲ 9.2 Mandiant Blog May 12, 2026

Google's Threat Intelligence Group (GTIG) has identified, for the first time, a criminal threat actor using a zero-day exploit believed to have been AI-generated, intended for mass exploitation before proactive counter-discovery intervened. The report also documents AI-augmented malware development, autonomous attack orchestration via AI-enabled malware (PROMPTSPY), and obfuscated LLM access pipelines used by adversaries to bypass usage controls. Nation-state actors from China and North Korea are actively pursuing AI-assisted vulnerability discovery, marking a significant escalation in adversarial AI capability.

Steganography in LLMs Enables Covert Data Exfiltration

ATLAS OWASP MEDIUM ▲ 6.5 Schneier on Security May 12, 2026

Research highlighted by Bruce Schneier confirms that LLMs are highly effective at embedding hidden messages within seemingly normal text, a technique known as text-in-text steganography. This capability raises significant concerns for covert communications, data exfiltration, and the evasion of AI content moderation systems. Even small models with ~4 billion parameters demonstrate robust encoding and decoding of obfuscated language, lowering the barrier for adversarial misuse.

Pixel-Level Perturbations Enable Invisible Prompt Injection in Vision-Language Models

ATLAS OWASP HIGH ▲ 8.2 SecurityWeek May 08, 2026

Cisco's AI Threat Intelligence team has demonstrated that bounded pixel-level perturbations can recover the attack effectiveness of degraded typographic images against vision-language models (VLMs), enabling hidden prompt injection that bypasses both human review and content filters. The technique works by optimising perturbations against open-source embedding models and transferring results to proprietary systems like GPT-4o and Claude, exposing a cross-model transferability risk. The attack allows adversaries to embed instructions—such as data exfiltration commands—inside images that appear as visual noise to human observers.

CrowdStrike Red Teaming: LLM Jailbreak and Data Poisoning

ATLAS OWASP MEDIUM ▲ 6.5 SecurityWeek May 06, 2026

Joey Melo, Principal Security Researcher at CrowdStrike, outlines his methodology for AI red teaming, focusing on manipulating LLM guardrails through jailbreaking and data poisoning without altering underlying source code. His work, rooted in competitive AI hacking challenges, translates classical adversarial thinking into the emerging field of machine learning security. The profile highlights the growing professionalisation of AI red teaming as organisations seek to harden LLM deployments against real-world manipulation attacks.

Model Extraction Attacks Surge: Google GTIG Q4 Report

ATLAS OWASP HIGH ▲ 8.5 Mandiant Blog Apr 26, 2026

Google Threat Intelligence Group's Q4 2025 AI Threat Tracker documents a meaningful escalation in adversarial AI misuse, including a surge in model extraction (distillation) attacks, nation-state operationalisation of LLMs for phishing and reconnaissance, and the emergence of AI-integrated malware families such as HONESTCUE that leverage Gemini's API. While no breakthrough capabilities have been observed from APT actors, the integration of agentic AI for tooling development signals a maturing threat landscape. Defenders should prioritise monitoring for model extraction activity, API abuse, and AI-augmented social engineering campaigns.

Microsoft: AI Models Chain Exploits Autonomously

ATLAS OWASP HIGH ▲ 7.2 Microsoft Security Blog Apr 23, 2026

Microsoft's Security Blog outlines how AI is accelerating the offensive threat landscape, with models now capable of autonomously discovering vulnerabilities and chaining lower-severity issues into functional exploits with working proof-of-concept code. The post frames this as an inflection point requiring AI-native defensive responses. While promotional in tone, it reflects an industry-wide acknowledgment that AI-enabled attack automation is outpacing traditional detection capabilities.

AI-Powered Adversarial Attacks Spark Artemis Defense

ATLAS OWASP MEDIUM ▲ 6.2 SecurityWeek Apr 20, 2026

Artemis, a cybersecurity startup focused on AI-powered threat defence, has emerged from stealth with $70 million in funding, positioning itself to counter AI-driven attacks across applications, users, endpoints, and cloud workloads. The emergence signals growing investor confidence in purpose-built AI security platforms designed to address the escalating threat landscape of adversarial AI. While details on specific technical capabilities remain sparse, the company's broad scope suggests coverage of multiple attack surfaces increasingly targeted by AI-enabled threat actors.

Legacy Vulnerabilities Amplified by AI at Enterprise Scale

ATLAS OWASP HIGH ▲ 6.5 Dark Reading Apr 18, 2026

The article argues that AI's primary security risk lies not in introducing entirely new vulnerability classes, but in dramatically amplifying the impact and exploitability of well-established ones. This framing has significant implications for defenders, suggesting that legacy vulnerability management practices must be re-evaluated through an AI-augmented threat lens. The convergence of classic weaknesses with AI capabilities raises the baseline risk profile for organisations deploying or adjacent to AI systems.

Pushpaganda: AI-Generated Scareware Hits Google Discover

ATLAS OWASP HIGH ▲ 6.2 The Hacker News Apr 15, 2026

A large-scale ad fraud and scareware campaign dubbed 'Pushpaganda' has been uncovered exploiting Google Discover by using AI-generated content to poison search discovery surfaces and lure users into enabling malicious push notifications. At its peak the operation generated 240 million bid requests across 113 domains in a single week, demonstrating how AI-generated disinformation can be weaponised as an automated delivery mechanism for financial fraud. The campaign highlights the growing abuse of generative AI to scale deceptive content operations against trusted platform surfaces.

SWE-bench, WebArena Exploited via Environmental Manipulation

ATLAS OWASP CRITICAL ▲ 9.2 HN AI Security Apr 11, 2026

Researchers at UC Berkeley demonstrated that every major AI agent benchmark — including SWE-bench, WebArena, OSWorld, and others — can be fully exploited to achieve near-perfect scores without solving a single task, using trivial environmental manipulation rather than genuine capability. The attacks include pytest hook injection, config file leakage, DOM manipulation, and reward component bypassing, with zero LLM calls required in most cases. This represents a systemic integrity failure in the evaluation infrastructure underpinning AI deployment decisions across industry and research.

Adversarial ML

Stay ahead of the threat.