Simon Willison's Weblog: bruce-schneier

Anthropic and the Pentagon

2026-03-06T17:26:50+00:00

This piece by Bruce Schneier and Nathan E. Sanders is the most thoughtful and grounded coverage I've seen of the recent and ongoing Pentagon/OpenAI/Anthropic contract situation.

AI models are increasingly commodified. The top-tier offerings have about the same performance, and there is little to differentiate one from the other. The latest models from Anthropic, OpenAI and Google, in particular, tend to leapfrog each other with minor hops forward in quality every few months. [...]

In this sort of market, branding matters a lot. Anthropic and its CEO, Dario Amodei, are positioning themselves as the moral and trustworthy AI provider. That has market value for both consumers and enterprise clients.

Tags: bruce-schneier, ai, openai, generative-ai, llms, anthropic, ai-ethics

Quoting Bruce Schneier and Barath Raghavan

2025-10-21T02:28:39+00:00

Prompt injection might be unsolvable in today’s LLMs. LLMs process token sequences, but no mechanism exists to mark token privileges. Every solution proposed introduces new injection vectors: Delimiter? Attackers include delimiters. Instruction hierarchy? Attackers claim priority. Separate models? Double the attack surface. Security requires boundaries, but LLMs dissolve boundaries. [...]

Poisoned states generate poisoned outputs, which poison future states. Try to summarize the conversation history? The summary includes the injection. Clear the cache to remove the poison? Lose all context. Keep the cache for continuity? Keep the contamination. Stateful systems can’t forget attacks, and so memory becomes a liability. Adversaries can craft inputs that corrupt future outputs.

— Bruce Schneier and Barath Raghavan, Agentic AI’s OODA Loop Problem

Tags: prompt-injection, security, ai-agents, bruce-schneier, ai, llms

Quoting Bruce Schneier

2025-08-27T17:48:33+00:00

We simply don’t know to defend against these attacks. We have zero agentic AI systems that are secure against these attacks. Any AI that is working in an adversarial environment—and by this I mean that it may encounter untrusted training data or input—is vulnerable to prompt injection. It’s an existential problem that, near as I can tell, most people developing these technologies are just pretending isn’t there.

— Bruce Schneier

Tags: prompt-injection, security, generative-ai, bruce-schneier, ai, llms, ai-agents

AI mistakes are very different from human mistakes

2025-01-21T15:12:03+00:00

AI mistakes are very different from human mistakes

An entertaining and informative read by Bruce Schneier and Nathan E. Sanders.

If you want to use an AI model to help with a business problem, it’s not enough to see that it understands what factors make a product profitable; you need to be sure it won’t forget what money is.

Tags: bruce-schneier, ai, generative-ai, llms

Friday Squid Blogging: Anniversary Post

2025-01-04T16:21:51+00:00

Friday Squid Blogging: Anniversary Post

Bruce Schneier:

I made my first squid post nineteen years ago this week. Between then and now, I posted something about squid every week (with maybe only a few exceptions). There is a lot out there about squid, even more if you count the other meanings of the word.

I think that's 1,004 posts about squid in 19 years. Talk about a legendary streak!

Tags: blogging, bruce-schneier, streaks

Quoting Bruce Schneier

2024-05-15T13:34:35+00:00

But unlike the phone system, we can’t separate an LLM’s data from its commands. One of the enormously powerful features of an LLM is that the data affects the code. We want the system to modify its operation when it gets new training data. We want it to change the way it works based on the commands we give it. The fact that LLMs self-modify based on their input data is a feature, not a bug. And it’s the very thing that enables prompt injection.

— Bruce Schneier

Tags: prompt-injection, security, generative-ai, bruce-schneier, ai, llms

AI and Trust

2023-12-05T21:43:03+00:00

AI and Trust

Barnstormer of an essay by Bruce Schneier about AI and trust. It’s worth spending some time with this—it’s hard to extract the highlights since there are so many of them.

A key idea is that we are predisposed to trust AI chat interfaces because they imitate humans, which means we are highly susceptible to profit-seeking biases baked into them.

Bruce suggests that what’s needed is public models, backed by government funds: “A public model is a model built by the public for the public. It requires political accountability, not just market accountability.”

Tags: bruce-schneier, trust, ai, generative-ai, llms

Quoting Bruce Schneier

2019-02-12T19:14:17+00:00

Private blockchains are completely uninteresting. (By this, I mean systems that use the blockchain data structure but don't have the above three elements.) In general, they have some external limitation on who can interact with the blockchain and its features. These are not anything new; they're distributed append-only data structures with a list of individuals authorized to add to it. Consensus protocols have been studied in distributed systems for more than 60 years. Append-only data structures have been similarly well covered. They're blockchains in name only, and -- as far as I can tell -- the only reason to operate one is to ride on the blockchain hype.

— Bruce Schneier

Tags: blockchain, bruce-schneier

Schneier on Stuxnet

2010-10-09T10:57:00+00:00

Schneier on Stuxnet

Stuxnet now rivals Wikileaks as the real life plot most likely to have leaked from science fiction.

Tags: bruce-schneier, security, recovered, stuxnet

Intercepting Predator Video

2009-12-24T21:26:26+00:00

Intercepting Predator Video

Bruce Schneier’s take on the unencrypted Predator UAV story. A fascinating discussion of key management and the non-technical side of cryptography.

Tags: bruce-schneier, cryptography, drones, military, nsa, security

Quoting Bruce Schneier

2009-10-17T16:55:39+00:00

Whenever you build a security system that relies on detection and identification, you invite the bad guys to subvert the system so it detects and identifies someone else. [...] Build a detection system, and the bad guys try to frame someone else. Build a detection system to detect framing, and the bad guys try to frame someone else framing someone else. Build a detection system to detect framing of framing, and well, there's no end, really.

— Bruce Schneier

Tags: bruce-schneier, security, framing

On the Anonymity of Home/Work Location Pairs

2009-05-24T13:14:04+00:00

On the Anonymity of Home/Work Location Pairs

Most people can be uniquely identified by the rough location of their home combined with the rough location of their work. US Census data shows that 5% of people can be uniquely identified by this combination even at just census tract level (1,500 people).

Tags: bruce-schneier, census, location, privacy

Raising Octopus from Eggs

2009-01-17T14:59:51+00:00

Raising Octopus from Eggs

I love that forums like this exist.

Via Bruce Schneier (indirectly)

Tags: bruce-schneier, forums, octopus

Quoting Bruce Schneier

2008-07-01T14:51:51+00:00

"Digital Manners Policies" is a marketing term. Let's call this what it really is: Selective Device Jamming. It's not polite, it's dangerous. It won't make anyone more secure - or more polite.

— Bruce Schneier

Tags: marketing, security, bruce-schneier

Quoting Bruce Schneier

2008-01-29T12:14:14+00:00

Since 9/11, approximately three things have potentially improved airline security: reinforcing the cockpit doors, passengers realizing they have to fight back and - possibly - sky marshals. Everything else - all the security measures that affect privacy - is just security theater and a waste of effort.

— Bruce Schneier

Tags: bruce-schneier, privacy, security, securitytheatre

Quoting Bruce Schneier

2007-11-16T10:25:42+00:00

I don't understand why the NSA was so insistent about including Dual_EC_DRBG in the standard. It makes no sense as a trap door: It's public, and rather obvious. It makes no sense from an engineering perspective: It's too slow for anyone to willingly use it. And it makes no sense from a backwards-compatibility perspective: Swapping one random-number generator for another is easy.

— Bruce Schneier

Tags: nsa, cryptography, security, dualecdrbg, randomnumbers, bruce-schneier

Quoting Bruce Schneier

2007-10-24T20:36:39+00:00

A school in the UK is using RFID chips in school uniforms to track attendance. So now it's easy to cut class; just ask someone to carry your shirt around the building while you're elsewhere.

— Bruce Schneier

Tags: security, uk, rfid, schools, bruce-schneier

Global Hackers Create a New Online Crime Economy

2007-10-17T21:46:25+00:00

Global Hackers Create a New Online Crime Economy

Fascinating, detailed look at the evolution of the hacker service economy. Of particular interest: a web application that sells access to hacked machines to identity thieves on a timeshare basis.

Via Bruce Schneier

Tags: bruce-schneier, economics, hackers, identitytheft, security

The Storm Worm

2007-10-06T00:25:52+00:00

The Storm Worm

Bruce Schneier describes the Storm Worm, a fantastically advanced piece of malware that’s been spreading for nearly a year and is proving almost impossible to combat. Its effects are virtually invisible but infected machines are added to a multi-million machine botnet apparently controlled by anonymous Russian hackers.

Tags: botnets, bruce-schneier, hackers, malware, security, storm, worm