LIVE THREATS
MEDIUM How Hackers Are Thinking About AI // HIGH Your MTTD Looks Great. Your Post-Alert Gap Doesn't // HIGH CSA: CISOs Should Prepare for Post-Mythos Exploit Storm // MEDIUM OWASP GenAI Security Project Gets Update, New Tools Matrix // HIGH OpenAI Impacted by North Korea-Linked Axios Supply Chain Hack // HIGH Python Supply-Chain Compromise // HIGH Over 1,000 Exposed ComfyUI Instances Targeted in Cryptomining Botnet Campaign // HIGH Google's Vertex AI Is Over-Privileged. That's a Problem // CRITICAL Flowise AI Agent Builder Under Active CVSS 10.0 RCE Exploitation; 12,000+ Instances … // CRITICAL How We Broke Top AI Agent Benchmarks: And What Comes Next //
ATLAS OWASP MEDIUM Moderate risk · Monitor closely Research LLM Security Industry News RELEVANCE ▲ 6.2

How Hackers Are Thinking About AI

A new academic paper analysed over 160 cybercrime forum conversations to understand how threat actors are discussing and adopting AI tools for criminal purposes. The research documents both misuse of legitimate AI platforms and attempts to build bespoke criminal AI models, revealing early-stage diffusion of AI capabilities within cybercriminal communities. The findings carry practical implications for law enforcement and security practitioners monitoring the evolving AI-enabled threat landscape.

SOURCE Schneier on Security ↗
RESEARCHSchneier on SecurityMEDIUMHow Hackers Are Thinking About AI

Overview

A newly published academic paper — “What hackers talk about when they talk about AI: Early-stage diffusion of a cybercrime innovation” — offers one of the more methodologically grounded looks at how cybercriminals are internalising and operationalising AI capabilities. Highlighted by Bruce Schneier, the study draws on a dataset of more than 160 cybercrime forum conversations collected over seven months via a cyber threat intelligence platform. The findings indicate that AI adoption within criminal ecosystems is real but uneven, marked by curiosity, scepticism, and active experimentation.

The paper is significant not because it reveals a sudden AI-powered crime wave, but because it provides empirical grounding for a threat trajectory that has largely been discussed anecdotally. Understanding how threat actors conceptualise and debate AI tools is essential for anticipating where capability gaps will be filled and where defensive attention is most needed.

Technical Analysis

The research identifies two primary vectors of AI misuse currently active in cybercriminal communities:

  1. Abuse of legitimate AI tools — Threat actors are actively probing consumer and enterprise LLMs (such as ChatGPT and similar platforms) for jailbreak techniques and prompt injection methods that bypass safety guardrails. This aligns with well-documented behaviours including social engineering content generation, phishing lure creation, and malware assistance.

  2. Development of bespoke criminal AI models — A subset of more technically sophisticated actors is discussing the creation of purpose-built models fine-tuned or trained on illicit data, designed without ethical constraints. This mirrors the emergence of tools like WormGPT and FraudGPT that have previously been reported in underground markets.

Notably, the forum conversations also reflect doubt and anxiety about AI’s effectiveness — suggesting that criminal AI adoption is not yet mature, and that the community itself is stress-testing claims about AI capability.

Framework Mapping

  • AML.T0047 (ML-Enabled Product or Service): Criminals leveraging commercial LLM APIs as attack-enablement infrastructure.
  • AML.T0054 (LLM Jailbreak): Documented attempts to bypass safety layers on legitimate platforms.
  • AML.T0051 (LLM Prompt Injection): Likely co-occurring with jailbreak discussions as a method to manipulate model outputs.
  • LLM01 (Prompt Injection) and LLM09 (Overreliance): Defenders and victims may over-trust AI-generated content, amplifying the impact of AI-assisted social engineering.

Impact Assessment

The immediate risk is concentrated around AI-assisted social engineering, phishing, and content generation for fraud — areas where even modest AI capability provides meaningful uplift for low-skill actors. More concerning in the medium term is the development of unconstrained fine-tuned models that could assist with vulnerability research, malware development, or operational security evasion at scale. Law enforcement and threat intelligence teams face a growing need to monitor AI-related discourse in criminal forums as a leading indicator.

Mitigation & Recommendations

  • Threat intelligence teams should expand keyword monitoring in dark web forums to capture AI tooling discussions, including references to specific model names and jailbreak techniques.
  • Platform providers should continue investing in abuse detection for API access patterns consistent with bulk phishing or social engineering content generation.
  • Policymakers should engage with findings from diffusion-of-innovation research to anticipate the maturation curve of criminal AI adoption before capabilities fully diffuse.
  • Defenders should assume AI-generated phishing content is increasingly indistinguishable from human-authored material and update user awareness training accordingly.

References