Overview
A report endorsed by security veteran Bruce Schneier argues that AI security measurement is fundamentally broken — and that simply maximising benchmark scores provides no meaningful security guarantee. The piece, published on Schneier on Security in May 2026, draws a direct parallel to the 30-year evolution of software security engineering: from black-box penetration testing, through white-box code analysis, to process-driven standards like the Building Security In Maturity Model (BSIMM). The central claim is that a similar maturity arc is needed for AI, but that organisations must not wait for a ‘security meter’ that may never exist.
Technical Analysis
Commenter Clive Robinson expands on the measurement problem with a theoretical framing worth examining. LLMs encode knowledge as high-dimensional continuous weight spaces — spectrums rather than discrete countable objects. Classical security metrics assume measurable, bounded properties. The ‘continuum hypothesis’ problem surfaces here: the cardinality of real-valued spectrums is undecidable in standard set theory. Robinson argues this means the information encoded in LLM weights is not only difficult to measure in practice but may be structurally unmeasurable in principle.
This has direct security consequences. If training data shifts the encoded spectrum in ways that cannot be fully characterised, then:
- Backdoors or poisoned behaviours may be embedded in weight space without being detectable by output-layer testing
- Benchmark evaluations probe a tiny slice of the input distribution and cannot guarantee generalised safety properties
- Trust in model outputs cannot be grounded in formal verification at scale
The report’s conclusion — that we must manage AI security through assurance processes rather than metrics — mirrors how mature software security organisations operate today.
Framework Mapping
MITRE ATLAS AML.T0031 (Erode ML Model Integrity) is directly relevant: the argument that weight-space properties are unmeasurable implies integrity erosion could go undetected. AML.T0047 (ML-Enabled Product or Service) applies because the failure mode discussed affects any organisation deploying AI in production. AML.T0044 (Full ML Model Access) is relevant to the white-box analysis strand of the argument — even with full access, the encoded spectrum resists meaningful audit.
OWASP LLM09 (Overreliance) is the primary OWASP mapping: the article is fundamentally a warning against overrelying on benchmark outputs as a security signal.
Impact Assessment
The impact is systemic rather than tied to a specific exploit. Organisations that have structured compliance or procurement requirements around AI security benchmarks are at risk of a false assurance gap. Regulated sectors — finance, healthcare, critical infrastructure — are particularly exposed if benchmark compliance is accepted as a substitute for genuine security architecture review. The risk is compounded as AI is deployed in higher-stakes decision pipelines.
Mitigation & Recommendations
- Adopt process-driven assurance frameworks: Model AI security programmes on BSIMM or equivalent, emphasising repeatable processes over point-in-time scores.
- Conduct architectural risk analysis: Treat LLM integration points as architectural attack surfaces requiring threat modelling, not just benchmark evaluation.
- Invest in continuous red-teaming: Automated and human adversarial testing should be ongoing, not a pre-deployment gate.
- Be explicit about measurement limits: Communicate to stakeholders that no current benchmark certifies AI security; governance documents should reflect this uncertainty.
- Track the WHAT pile: As the report notes, cataloguing known unknowns in your AI deployment is a concrete near-term action with real risk reduction value.