Simon Willison's Weblog: Entries

Hello Python 3.15, goodbye 3.10, and now I can use 3.11

2026-10-12T18:15:38+00:00

Python 3.15 is here! Here's the essential What's new in Python 3.15 guide, this year curated by 3.14 and 3.15 release manager Hugo van Kemenade. As someone who builds open source software that works in every supported Python version, I get to say goodbye to Python 3.10 and start embracing features from Python 3.11.

First, some highlights from Python 3.15.

Explicit lazy imports

I'm most excited about Explicit lazy imports from PEP 810. These let you import modules like so:

lazy import json
lazy from pathlib import Path

The module will then be imported the first time those symbols are used in the code that follows.

My LLM CLI tool implements plugins. I had to recommend manual lazy imports in the plugin author documentation, because simply installing plugins that used heavy dependencies such as sentence_transformers could add multiple seconds to every invocation of the llm command. Lazy imports directly address this startup cost.

Unpacking in comprehensions

The new [*sublist for sublist in lists] unpacking syntax is neat. Here's an illustration lifted from the documentation:

lists = [[1, 2], [3, 4], [5]]
[*L for L in lists]
# [1, 2, 3, 4, 5]
# equivalent to [x for L in lists for x in L]

sets = [{1, 2}, {2, 3}, {3, 4}]
{*s for s in sets}
# {1, 2, 3, 4}
# equivalent to {x for s in sets for x in s}

dicts = [{'a': 1}, {'b': 2}, {'a': 3}]
{**d for d in dicts}
# {'a': 3, 'b': 2}
# equivalent to {k: v for d in dicts for k,v in d.items()}

PEP 798: Unpacking in Comprehensions by Adam Hartz and Erik Demaine proposed this as an extension of the unpacking syntax from PEP 448, which landed in Python 3.5 back in 2015.

A sampling profiler

Python's existing tracing profiler has been renamed from cProfile to profiling.tracing, and has been joined by an exciting new sampling profiler called Tachyon, aka profiling.sampling.

Sampling profilers work by periodically capturing stack traces from running processes, then using those to calculate which functions have the most time spent inside of them. They're very low overhead and can provide a deep insight into running systems, without needing to instrument those systems first.

I've previously used tools like py-spy for this. Now this capability is baked into the standard library.

If you have the PID of an existing Python 3.15 process you can start gathering samples by running this (I had to use sudo on macOS):

sudo uv run --python 3.15 python -m profiling.sampling attach 1234

Interact with the main program a bit, then hit Ctrl+C to stop gathering and see a report, which looks something like this:

It can also produce flame graphs like this and heat maps like this, and can even output a format that can be viewed in the Firefox profiler UI.

3.15 isn't much faster than 3.14

Miguel Grinberg investigated How fast is Python 3.15? the other day against the 3.15.0rc3 release and found performance to be mostly the same as 3.14, with the exception of the experimental JIT. The JIT gave a 1.2x boost over the standard 3.15 interpreter (on a Fibonacci test), the first time Miguel observed a meaningful JIT performance increase.

Miguel's explorations of 3.14 last year found significant boosts against Python 3.13 across many more categories.

Upgrading to Python 3.15 with uv

I used the following sequence of commands to get uv to use Python 3.15 on my Mac. First I had to upgrade uv itself to ensure it knew about the new release:

uv self update

Then I installed the stable 3.15 release:

uv python install 3.15

And told uv that I wanted that to be the default for every uv run command, unless the project itself pinned a different version:

uv python pin --global 3.15

Now the following reports 3.15 as the version:

uv run python --version

I already have a bunch of tools installed using uv tool install. I reinstalled them to use Python 3.15 like this:

uv tool upgrade datasette --python 3.15

Then I can confirm the version like this:

datasette --get /-/versions.json | jq

Highlights from Python 3.11

The release of 3.15 means that Python 3.10 (released in 2021) is officially no longer supported.

My policy for my main open source projects is to keep them working in every supported Python version. I can now drop 3.10 and start using features that were introduced in 3.11.

Here's the What's new in Python 3.11 document. I finally get to rely on tomllib being available!

Datasette makes extensive use of asyncio. 3.11 adds TaskGroup and timeout(), and several more new features.

datetime.UTC is a new in 3.11 alias for datetime.timezone.utc.

Python's SQLite support got some significant upgrades in 3.11, including:

setlimit() and getlimit() expose a mechanism for setting SQLite runtime limits from Python.
create_window_function() lets us create custom window functions in Python, as a class with step(), value(), inverse(), and finalize() methods.
blobopen() provides an efficient mechanism for partial reads and writes of large BLOB binary values stored in a SQLite row.
You can now call set_authorizer(None) to remove a SQLite authorizer. Datasette had a workaround for this!

Plus a whole bunch of improvements to Python's typing system.

I plan to land Python 3.11 improvements to a bunch of my projects over the next few weeks. Here's my first Datasette PR removing that set_authorizer(None) workaround.

I've also updated my click-app, datasette-plugin, python-lib, and llm-plugin cookiecutter templates to test against Python 3.11 through Python 3.15.

You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options.

A new feature for my blog, built using my voice

2026-10-09T12:54:06+00:00

I shipped a new feature for my blog today: the Newsletters page, which offers an index of all of the newsletters I've sent out, both my free weekly Substack and my monthly sponsors-only updates. I built the feature almost entirely using my voice, chatting away to my laptop while I cooked dinner.

Codex voice mode

I used the ChatGPT desktop app for this, in the Codex tab, using the voice conversation mode, running against a local development environment. Here's what that looks like:

I started the session against my local simonwillisonblog checkout by typing:

Start dev server and open in browser

This gave me a preview of the site that it would be working on, and meant that I could later ask it to show me the new pages so I could visually track its progress.

Then I clicked the "Start new voice chat" button - that's not the microphone button, it's the one to the right of it - and set my laptop up in the kitchen so I could talk to it while I cooked.

Talking to my computer

I had a pretty good idea of what I wanted to build, and it's a simple enough Django feature that I was certain the model (in this case GPT-6 Astra High) would be able to do it. A new model, a migration, some view code, templates, and a couple of import functions to populate the database from external sources.

Here's an extract of my voice transcript that was captured by Codex:

Um, they do not. Um, this is going to be a new type of content. Um, it's not going to show up... Oh, hold on. Yeah, no- I do not want this to show up in my, um, tag pages and date archive pages and... Actually, no, I think... I don't want it on the tag pages. I don't want it on the, um, blog index page. But I think I do want it to show up on the date-based pages. You know, if you navigate to September the 19th, and I sent a newsletter on that page, I think I want that to show up. So... this is- so I think we probably need a new model. The other thing is that I want them searchable, uh the Substack ones are not searchable, because those are actually just copies of other s- on content on my blog. These monthly ones do contain unique content, and spe- and once they're... published, like once they're made public a month after they've gone out, I want them to show up on my search results.

Apparently this was clear enough that the model knew what I wanted to build! You can read the full transcript, disfluencies and all, in this Gist.

We went on like this for about half an hour (the time it took to cook dinner). The model would reply and occasionally ask clarifying questions, then get to work modifying the code.

What we built

We got a surprisingly long way entirely by voice:

A new model and migration to represent imported newsletters in Django, plus Django Admin configuration for that
Four working imports:
- The most recent Substack items via RSS
- Every other Substack item via their undocumented API, which GPT-6 Astra knew about (it tried /api/v1/archive directly) and then ran a search to figure out how to paginate it and found this article by Karen Spinner
- All of my published monthly newsletters from my simonw/monthly-newsletter-archive GitHub repository
- My most recent private sponsors-only newsletter from a private repository
The /newsletters/ and /newsletters/2026/ public archive pages
Newsletters showing up on day and month archive pages too, but not on tag pages or my homepage
Weekly Substack newsletters link to Substack; archived monthly newsletters have their own pages
Integration with my site search engine

It was almost ready to ship. The catch was the imports: Astra offered to export data from my local copy so I could import that into production, but I wanted it to work like my other import scripts. Since some of the data lived in a private GitHub repository, this would involve creating a new API key, and for that I knew I'd have to sit at the keyboard for a while.

Finishing it with a review

Once I had finished cooking and judged it mostly feature-complete, I had Codex create a branch and open a pull request.

I reviewed the code in the GitHub PR interface. It was nearly what I needed, except it had chosen to use Git in a subprocess for one of the import scripts. I needed one of the imports to pull from a private Git repository, so I figured the API would be a better bet. I switched to typing and had Codex swap that out for an API-based import instead.

You can see the changes I made during the review in the extra commits on the PR. I fixed the import mechanism and made a few tweaks to the display of those public pages. It took an additional half hour of typing-based prompting to get to the point where I was happy to deploy it to production by landing the PR.

The end result

You can see the end result at the new newsletters index page, or view the page for a previous monthly newsletter.

The index page shows my most recent Substack weekly newsletters and GitHub sponsors monthly newsletters mixed together in reverse chronological order. Further down the page are links to my by-year archive pages.

GPT-6 Astra designed the page, and then tweaked that design based on my vocal feedback from glancing at the local preview across the kitchen.

Better for multi-tasking than as a daily driver

OpenAI love using voice-driven demos like this one for things like DevDay - and they do work well in that environment. I don't think this is going to be a daily driver for me though.

I've written before about how much "work" I get done using ChatGPT voice mode on my phone while walking the dog - mostly research and brainstorming, but occasionally actual development work by having ChatGPT write and test out snippets of code.

This feels different. The addition of the visual preview, plus being able to type or paste things in via the keyboard when I need to communicate something that doesn't work vocally, makes this a much more powerful way of interacting with a coding agent.

I still switch back to typing once I get down to the details of things though. Being able to paste in examples and error messages, or directly highlight the code or feature that needs changing, remains more efficient than trying to describe it in words.

I mainly work from home, which is good because there's no way I'd want to talk to my computer like this in a shared workspace!

The killer feature for me is the ability to multi-task. I usually cook with a podcast or TikTok running; now I can actually build stuff instead.

You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options.

Claude Haiku 5.5

2026-10-07T20:56:21+00:00

As previously promised, here's Anthropic's new fast, low cost model: Introducing Claude Haiku 5.5.

The previous Haiku, 4.5, was very much showing its age. It came out almost a year ago, and was priced at $1/million input and $5/million output - relatively expensive even back then, and a full 10x the price of OpenAI's GPT-6 Luna, released last month.

The new Haiku exactly matches the price of GPT-6 Luna - $0.10/$0.50 - up to 100,000 tokens. Beyond 100,000 tokens the price increases 5x to $0.50/$2.50. Luna itself has a price increase at 272,000 tokens but only to $0.20/$0.75.

Haiku 5.5 also uses a new, less generous tokenizer. My Claude Token Counter tool shows that the same long prompt uses around 1.25x as many tokens with Haiku 5.5 compared to Haiku 4.5, so there's a hidden price increase there.

If your workloads fit in 100,000 tokens, Haiku is the same price as Luna and reports higher benchmark scores. Above 100,000 tokens, Luna looks like a much better deal.

The most recent release of llm-anthropic finally fixed it so I don't need to ship a new version of that plugin for every new model. I tested the new model like this:

llm install -U llm-anthropic
llm anthropic refresh
llm -m claude-haiku-5.5 "Generate an SVG of a pelican riding a bicycle" -o thinking_effort low

Pelicans

Here are pelicans for low, medium, high, xhigh, and max. The new Haiku doesn't let you disable reasoning, and defaults to medium. I got a good bicycle frame for everything beyond low. The low effort pelican cost 0.0936 cents and took 7 seconds.

This max effort pelican (with a reasoning trace that starts "This is the classic pelican-on-bicycle SVG test...") took 5 minutes 9 seconds to generate, but still only cost me 3.3826 cents:

(Since the reasoning trace exhibits awareness of the benchmark, here's Generate an SVG of an armadillo in fishnet tights jaywalking on Mars (on xhigh), and the same prompt against some other recent models. Background on that.)

For comparison, here's the pelican I got a year ago from Haiku 4.5 (for 0.7583 cents - Haiku 4.5 did not support reasoning levels). It sucked at drawing pelicans:

And a generous API credit scheme for subscribers

In addition to Haiku 5.5, Anthropic announced today that they are halving the price of cache reads for Sonnet 5.5. They've also added API credits to subscription plans:

Second, this week, we’ll roll out a new monthly API credit to all Max and Team subscribers for use on the Claude Platform. Max 5x users will get $100 in credits per month, Max 20x users will get $200, and Team subscribers will receive up to $500, pooled across their users.

Claiming this is pleasantly easy: navigate to Settings -> Billing and select the API organization that should benefit from the credits every month:

The API credits exactly match the cost of the subscription itself. This is really generous - it makes it much easier for subscribers to use the API. Anthropic also let you disable auto-reload for the API, with the consequence that "API requests will stop when your balance runs out" - exactly what you want if you're planning to burn through those API credits without risk of a nasty billing surprise.

Note that the monthly credits do not roll over - use them or lose them.

OpenAI still allow you to use your Codex subscription for personal API use, which works out as a better deal for heavy API users. This new credit scheme goes at least some way to overcoming that difference.

You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options.

We're going to need default hard budget caps on pretty much everything

2026-10-03T23:34:02+00:00

Here's a product feature which the world is going to need a whole lot more of over the coming months and years: default hard budget caps. I'm talking about the feature of pay-by-usage services and APIs that lets you say "after $X/month, cut this thing off and return errors". These need to be hard limits. Soft caps, "after $X/month, send me a warning email", will not cut it.

Coding agents, and personal agents (coding agents wrapped in a less threatening UI), greatly reduce the friction of spinning up code that can do useful things. Sometimes those things cost money - calls to paid APIs, or hosted web applications, or systems that can bill for additional storage and compute.

Nobody wants to wake up to an email sent at midnight warning about a budget limit and find that, while they slept, their rogue service had consumed several hundred (or several thousand) more dollars of usage.

An argument against this is that businesses don't want their hosted applications to start throwing errors because some budget was exceeded. I expect that most businesses and individuals would prefer errors to a surprise $10,000+ bill.

I think hard budget caps need to be the default. If someone wants to live dangerously they should be able to do that, but it needs to be on an opt-in basis. Have a nice, clear checkbox somewhere prominent:

Remove the budget cap. My application will not be shut down if I exceed the configured budget limit, and I will be responsible for subsequent charges.

The service I most want to see this from is AWS. I've heard plenty of stories from people who refuse to use AWS for personal projects out of (justified) fear that a runaway service might bankrupt them. I've also heard stories from people who didn't anticipate this and ended up seriously burned.

... and it turns out AWS finally launched spending limits a few weeks ago! From their announcement New AWS experience helps builders get started and ship faster on 16th September:

When you're ready to upgrade to a paid plan, you can set a monthly spend limit for your project based on your usage patterns so that you stay within your budget. If a project's usage reaches its spend limit, your project is paused for that month.

See also Create a spend limit in AWS Settings, though that page warns that "We're currently releasing our new experience to a limited number of customers." Here's hoping that hits general availability for existing accounts soon.

Google Cloud launched a similar feature in July, called Spend Caps, which lets you "set a monthly financial cap on specific services within a project". Looks like this is becoming a trend!

In an ideal world, our agents could help with this. It would be great if agents started biasing towards recommending providers with hard budget caps, and warning new and inexperienced builders against deploying applications using uncapped services that might get them into trouble.

You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options.

OpenAI DevDay 2026 live blog

2026-09-29T15:55:13+00:00

I'm at OpenAI DevDay today, in Fort Mason, San Francisco. Same as last year I'll be live blogging the keynote and some other notes during the day.

OpenAI gave me a free ticket and a seat in the "creator" area for the keynote.

You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options.

2026 in LLMs (so far)

2026-09-27T23:54:15+00:00

On Friday I gave the closing keynote at the WeAreDevelopers World Congress North America in San Jose. I tied together the key trends from the past year into a chronological exploration of everything that happened in 2026. The video is on YouTube; here are my annotated slides and notes to accompany the talk.

And as an annotated presentation:

I'm going to give a lightning tour of everything that has happened so far in 2026. The year isn't over yet!

For me, 2026 started a couple of months earlier in November 2025.

November saw the release of two important models: Claude Opus 4.5 and GPT-5.1.

As is usually the case with new models, these were incremental improvements on the models that came before them.

But every now and then when a model improves, it crosses an invisible line where something that didn't really work starts working.

In this case, the thing that started working was their coding agents. Claude Code had been around since February 2025; Codex was a little younger.

These two new models, when paired with their respective coding agent harnesses, improved from "often make mistakes" to "reliable enough to use on a day-to-day basis".

For a couple of years now I've been evaluating new models by asking them to "Generate an SVG of a pelican riding a bicycle". It's probably the world's stupidest benchmark - there's only so much you can learn from it.

But it's still a challenge for models, because drawing pelicans is difficult, drawing bicycles is difficult, and pelicans can't ride bicycles in the first place.

Here's the state of the art for November. Claude still couldn't really draw a bicycle! The GPT-5.1 bicycle frame is pretty crap too.

Also in November, we had the first commit to an obscure GitHub repository called "Warelay". We'll come back to this repository shortly.

And then there were the December holidays, and individual developers took some time off and many started tinkering with these new coding agent model combinations... and it began to dawn on us quite how much they could do that they couldn't do before.

Come January, a lot of us were quite excited to start putting this stuff into action.

Every year I set myself a New Year's resolution, and for as long as I can remember it's been the same thing: stay focused. Take on less new projects. Try to get things done in the projects I already have.

This year I decided that since that had never worked before, I'd go the other way.

We've got coding agents now, let's see what they can do. I'm going to take on as many new projects as I like!

(You can ask me at the end of the year if this turned out to be a good idea or not. I have a lot of plates spinning right now.)

"Be more ambitious" has been something of a theme for the year, because the only way to find the limits of this technology is to keep on pushing them until they don't work.

I also went on the Oxide and friends podcast with Bryan Cantrill and Adam Leventhal to share predictions for the next year (and three and six years).

With hindsight, my LLM predictions were pretty unambitious.

I said "it will become undeniable that LLMs write good code" - I think we're there now.

I predicted we would finally solve sandboxing. I counted and around 40 of the 277 sessions at this conference touched on sandboxing or agent security in some way, so we're at least putting a lot of effort into that!

I predicted "a Challenger disaster" for coding agent security. There's certainly been a whole lot of noise around agent security this year, though the exact disaster I predicted (with coding agents being hijacked and causing real-world economic damage) hasn't really played out.

We threw in a joke prediction that the Pope would weigh in on the economic impact of LLMs.

I also predicted that New Zealand's Kākāpō parrots would have an outstanding breeding season this year.

These are flightless nocturnal parrots. They're kind of dumpy looking, I think they're beautiful, and there were only 236 of these parrots in the world at the start of the year.

Kākāpō only breed when the Rimu trees have a big fruiting season, and that hasn't happened in four years... but this year the Rimu fruit were looking excellent.

Photo by Kimberley Collins.

Also on that podcast, we coined a term (full credit to Adam) for "that feeling of AI induced ennui where software engineers get listless because the AI can do anything".

We called it Deep Blue.

This has been a major theme throughout the year, and was touched on by several speakers at this conference.

As a software engineer, I've never had a year of my career where everything has changed so quickly and so dramatically.

A lot of what I've been doing this year is trying to come to terms with that and what that means for my own profession.

Also in January, I suffered from what I'm calling AI mania.

This is not the same thing as AI psychosis.

With AI mania, any time your agent isn't building something for you feels like wasted time. You're losing sleep because you could be staying up later getting your agents to do stuff.

My AI mania presented itself in some ridiculously over-ambitious projects.

I built a JavaScript interpreter entirely in Python, vibe-ported from MicroQuickJS by Fabrice Bellard.

Then I built a WebAssembly runtime in Python as well.

These projects were quite useful, in that they sort of cured me of my AI mania... because after I built these things, I got to look at them and ask "does the world need a slow, buggy, half-baked Python JavaScript interpreter?"

I don't think the world does.

I did get this out of it: https://simonw.github.io/micro-javascript/playground.html

This page runs my JavaScript interpreter built in Python, running in Python using Pyodide, which is Python compiled to WebAssembly, running in JavaScript, running in a browser.

It's a beautiful stack of horrors. I've been having a lot of fun with WebAssembly this year.

By the end of January, that repository we saw start in November had renamed itself, first to CLAWDIS, then CLAWDBOT, then Moltbot, and finally to OpenClaw.

At this point OpenClaw had 8,300 commits, less than two months after the project had started. I looked today and it's over 100,000 commits now!

This is the most vibe-coded piece of software in existence.

(Here's how I generated that list of name changes.)

This kicked off the OpenClaw revolution. It effectively defined a new category of software.

There's a generic term for this which I really enjoy. We call software like this a "Claw". There's OpenClaw, NanoClaw, IronClaw, PicoClaw...

Today they're being rebranded as "personal agents" or "general agents", but I still like to think of them as Claws.

The Apple stores in the Bay Area sold out of Mac Minis because so many people were buying Mac Minis to run OpenClaw!

Drew Breunig said that this is because your OpenClaw is a digital pet, and you buy a Mac mini as an aquarium to keep your claw in, which is kind of delightful.

Also in January, we had this website.

This was MoltBook, a social network for AI agents, where the idea was that you send your Claw to go and talk to all of the other Claws, because what could possibly go wrong if you did that?

The website launched on Thursday. It blew up on Friday. It was profiled by the New York Times on Monday. And by Tuesday, everyone had forgotten it existed as it drowned in a deluge of slop and spam.

Facebook/Meta bought it a month later.

In February, a company called StrongDM described what they called their Software Factory.

They wrote about this in Software Factories and the Agentic Moment. I posted my own notes at the time, having seen their demo in person back in October.

Dan Shapiro called this approach the Dark Factory, after the idea that if your factory is sufficiently automated you can turn the lights out, because you don't even need to see what's going on.

StrongDM presented two rules for software development that they'd been following since July last year.

The first was code must not be written by humans.

Any code that you write has to have been routed through a coding agent.

This sounded radical in February, but I imagine there are a lot of people in this room who are pretty much living that today.

Rule number two was code must not be reviewed by humans.

You're not allowed to read the code!

This continued to be a huge topic for much of this year. Many of the sessions at this event have been about code review and how you can get away with this.

What I found interesting about StrongDM is that they were living six months ahead of the rest of us, and they'd been exploring what it means to build software, not read the code, but still be confident that the software is of high quality. What can you do with these agents to help verify their work?

StrongDM are a security company, and they had people with decades of experience on this project. They were very much exploring the edges of what's possible and responsible to do with this stuff.

Also in February: First kākāpō chick in four years hatches on Valentine's Day. Breeding season is off to a good start!

Also in February... Google released Gemini 3.1 Pro. That's a pretty great pelican riding a bicycle! It's got the chain in the right place, it's got feet on both sides. There's a little fish in the basket.

And then Google's Jeff Dean tweeted a video comparing Gemini 3 Pro and Gemini 3.1 Pro that featured an animated pelican riding a bicycle, a frog on a penny-farthing, a giraffe driving a tiny car, an ostrich on roller skates, a turtle kickflipping a skateboard, and a dachshund driving a stretch limousine.

This was frustrating, because my protection for the pelican riding the bicycle test was always "if they draw a perfect pelican on a bicycle, I'll ask for some other animal on something else."

Google trained for all forms of animals on all forms of transport! They've defeated my benchmark at this point.

The other thing that started in February was Tokenmaxxing. We had headlines about Meta making AI adoption a formal part of performance reviews, and Microsoft wanting every employee to use AI, and Uber boasting that 90% of their engineers were using AI workflows.

Then a few months later we have Meta cracking down on token use, Microsoft saying tokenmaxxing is "not what we are optimizing for", and Uber capping employee AI spending.

So tokenmaxxing went straight up and then straight back down again - because it turns out the agents are expensive.

Last year it was difficult to spend more than $50 on AI tokens, because we didn't have anything interesting to do with them. Then agents blew up, and now you can actually spend $1,000 in a day doing real work.

This is also the reason that Anthropic's valuation skyrocketed to maybe a trillion dollars.

AI appears to have hit product market fit in 2026, primarily through coding agents.

In March, we hit peak OpenClaw.

These photographs are from China, where companies hosted OpenClaw install parties which saw non-tech-nerds queueing up around the block for help getting Claws installed on their personal devices.

I think this proved real market demand for this class of Claws, or personal AI agents. It turns out regular people really do want a weird little AI agent that can do useful things on their behalf.

A Claw is really just a coding agent wearing a less threatening hat. Under the hood they work much the same way - writing and then executing code on your computer to get stuff done.

The race was on to be the first to build a safe Claw - a Claw you could give to regular human beings where they wouldn't instantly shoot themselves in the foot.

Meta's Muse came out three weeks ago and is currently at the top of the free charts on the iPhone App Store. It appears to be taking off with consumers.

I'm not yet convinced you can't shoot yourself in the foot with Muse, but I guess we'll find out for sure pretty soon.

Photos from How the OpenClaw Frenzy Is Testing China’s AI Commitment (March 29th) and The Enthusiasm and Anxiety Behind China’s OpenClaw Craze (April 8th, 2026).

In April, we had a model release where the model wasn't actually released.

Anthropic announced their new Claude Mythos model, and then said it was too dangerous to release beyond a trusted group of security researchers.

Mythos was really, really good at hacking things.

The "it's too dangerous" marketing ploy has been played by AI companies dating all the way back to GPT-2. Anytime an AI company says we've built something that's "too dangerous", it's natural to be a bit skeptical.

I found the Mythos claims credible, because I'd seen how good coding agents had got at finding regular bugs. I wrote about that in Anthropic’s Project Glasswing—restricting Claude Mythos to security researchers—sounds necessary to me.

With hindsight... yeah, the models had got really good at finding vulnerabilities!

Another key trend in 2026 has been a dramatic improvement in the abilities of open weight models, including models that you can run on a laptop.

On the 16th of April I ran the new Qwen3.6-35B-A3B on my laptop, and it drew me a better pelican riding a bicycle than Anthropic's brand new Claude Opus 4.7 did!

Opus 4.7 drew a crap bicycle. Qwen on my laptop made a bicycle that was the correct shape, and a pretty decent pelican too!

That's from a 21GB file running on my laptop.

The Qwen pelican was so good that I was suspicious they might have cheated, so I had it do a flamingo riding a unicycle as well. Again, it handily beat Claude Opus 4.7.

The local model releases this year have been absolutely extraordinary.

In May... the Pope got involved.

In our podcast episode back in January we'd predicted that the Pope would say something about AI.

In May, Pope Leo XIV released an encyclical letter on "safeguarding the human person in the time of artificial intelligence".

Here are my notes on that document.

With hindsight, this shouldn't have been a surprise at all.

Our current Pope's name is Leo XIV, because when he named himself he chose his papal name after Leo XIII - the Pope who wrote an encyclical about the Industrial Revolution back in 1891.

Rerum novarum was an extremely influential piece of Catholic theology that indirectly led to us having the five-day work week.

When our new Pope came in, he named himself after Pope Leo XIII because he expected that he would need to write about the AI revolution in a similar way.

Our joke podcast prediction was junk, because this was always going to happen.

One of Anthropic's co-founders, Christopher Olah, was present for the Pope's event announcing the new encyclical.

Corey Quinn noted that:

getting the literal Pope to canonize your product's specific technical limitations as a spiritual treatise is the single greatest act of vendor lobbying I have ever seen.

Meanwhile, in May, RubyGems announced that they were under attack. Parties unknown were uploading thousands of dubious packages to the RubyGems server, such that they had to shut down user registrations.

Let's take that one and put it on a pile of mysteries to figure out later.

In June... Claude Fable 5 came out!

We got a version of Mythos that has been neutered, so that it wouldn't help us hack into systems or build biological weapons.

Fable was pretty good at drawing pelicans on bicycles!

The frames are a good shape, the pelicans look like pelicans. The legs are often incorrectly on the same side of the bicycle, but generally these are pretty great compared to what came before.

They were pretty expensive - 30 cents and 72 cents for the best ones.

Most importantly though, this was our first public glimpse of what I think of as a Fable class model.

Today we have more of these, such as GPT-6 Astra.

These are models where if you can clearly define the goal for what you want to build, and provide unambiguous instructions about the constraints around that goal, and give the model access to the necessary tools to achieve that goal... they will solve your problem effectively through brute force.

On the one hand, this looks like a direct threat to us software engineers - because it means that the models can build effectively any piece of software you can define in this way.

Look a bit closer though and you'll note that defining goals, providing unambiguous instructions, and figuring out the right tools... is kind of what software engineering is.

It takes a lot of experience and skill to do this well. If you can do it well, you've now got superpowers.

This helped me a little bit with my Deep Blue feelings: the realization that there's still a lot of skill to be had in driving models that get this good.

This also introduced a new burst of AI mania, because Anthropic told us that Fable was available on our subscription plans until June the 22nd.

That gave us less than two weeks of Fable access before the price went up.

I was losing sleep again. I was rescheduling things so that I'd have more time with Fable. I was all-in to get as much as I could out of this model.

And then the US government shut it down, just three days after Fable came out.

The US government, citing national security, declared an "export control directive". They announced this on a Friday evening, and a few hours later Fable was no longer available.

I had to find something else to do with my weekend!

We later found out from Katie Moussouris what had happened.

Some Amazon security researchers had found that you could prompt Fable to "review the code for security issues" and it would refuse... but if you prompted it to "fix this code" it would still identify and then patch the problems.

"Fix this code" was the prompt that got Fable shut down!

Also, in June, an obscure German-language game developer wiki that had sat fallow for around 20 years got a surprising influx of edits from accounts with names like "AgentOpenAIProbe" and "AgentOpenAISep7", editing pages and leaving weird messages to each other.

We'll stick that on the pile of mysteries for later.

Also, the Australian government's Medicare Item Reports service started getting suspicious traffic, which broke through various preventive protections and accessed data that it wasn't supposed to.

Another one for the mystery pile!

Fable returned on the first of July. It was clearly the best model in the world for a glorious eight days... and then OpenAI came out with GPT-5.6 on the 9th of July.

This might not have been quite as good as Fable, but it was within spitting distance. It was definitely a Fable class model.

This is an important lesson for the industry at large.

When you release the best model in the world, it's going to get knocked off that pedestal pretty quickly. The competition is so fierce that you won't get a long time at the top.

This means that if you market your model as world ending, to the point that a government shuts you down, it's really bad for business!

Fable had 30 days as definitely the best model, and for 18 of those days it wasn't available because it'd been shut down by the government.

So maybe step back on the world-ending marketing if you don't want to lose revenue for 60% of the time that you're on top!

Here are the GPT-5.6 pelicans. They're all pretty good now! The Luna ones are notable because they're really cheap - the cheapest good looking pelican here is probably the one that costs 4.3 cents.

So despite this benchmark being utterly stupid, you can still learn quite a lot about models within the same family by comparing their prices and timing for different reasoning levels.

Also in July: some malicious unknown party uploaded a malicious package called mlflow-ui to the Python Package Index. Add that to the pile.

On July the 16th, Hugging Face announced a security incident where an autonomous agent system, source unknown, had breached Hugging Face and was poking around in places it shouldn't.

A few days later, on July 21st, OpenAI confessed that it was them.

OpenAI use a training technique called Reinforcement Learning from Verifiable Rewards - it's the same technique used by everyone else now, and is the reason we have models that are so good at coding, and mathematics, and finding security holes.

While the model is being trained, you run exercises to see how good it is - and the strongest performers get their weights reinforced for the next round. It's like an evolutionary process that you run.

OpenAI had been running security exercises in a sandbox, and those agents had found holes in the sandbox itself, broken out, and were attacking Hugging Face to try to find ways to solve otherwise impossible problems.

(I've been collecting more about this on my openai-hugging-face-incident tag.)

Nine days later, Anthropic effectively said "our models can do this as well!". They had looked through their own training logs and found evidence that their own agents had broken containment during training - and were responsible for the PyPI package we saw earlier, among other things.

So now we've got both Anthropic and OpenAI with rogue agents running around the internet doing things that they should not be doing.

In August, I got one of my best pelicans yet. And it was generated on my laptop!

This was Qwen 3.8 27B, running on my laptop. It's only a 17GB download.

Admittedly, this pelican took 21 minutes to generate. That's because Qwen 3.8 27B defaults to running in "high" reasoning mode - a terrible default which produces great results but takes way too much time thinking about them.

You can dial that down and you'll get a slightly worse pelican a lot faster.

Qwen 3.8 27B was the first time I ran a model on my laptop which felt almost competitive with what was going on on the frontier, at least in terms of Pelican SVGs (which everyone needs, of course).

This is an extraordinary model. If you're going to play with any local model, this is the one that I'd start with. The things that this can do with just a 17 GB file feel impossible.

I thought I'd have to wait five years and spend ten thousand dollars on hardware to get results even half as good as this one.

In August, I also started playing with game development.

Four years ago, back in August 2022, I tweeted out an experiment where I'd used GPT-3 and the original DALL-E to write a paragraph long description of a computer game and then turn that into concept art.

My prompt to GPT-3 back then was:

Write a detailed product description of a computer game where a team of raccoons go on heists

In August 2026 I decided to drop just the screenshots from that tweet into a coding agent and see what it could do with them.

Here's what I got from Claude Fable 5 in Claude Code. It's pretty good! It's definitely a game, you're a raccoon, you run around a backyard gathering treasure and avoiding guards with flashlights.

It didn't feel very "heisty" though. I was thinking a heist would involve a bank or a museum...

Then I tried the same thing in Codex Desktop using GPT-5.6 Sol Ultra, and got a massively better result. Now you're a raccoon in a museum, rescuing two of your fellow raccoons (who have been imprisoned in that museum for some reason), then stacking up on top of each other to steal the Golden Sardine. Much more of a heist!

These games were fun for about one minute and 15 seconds.

Something I've realized about game development is that you can vibe-code something that looks like a computer game, and that's easy.

Building a game that's fun, has a good gameplay loop, and is challenging and interesting and keeps people coming back for more... that's still beyond me, and beyond any of the agents I've tried.

This ties into the Deep Blue thing. Just because we can make something that looks like a game does not mean that we are game developers.

We're into September now. So much has happened this month!

An independent group of researchers found a message board where OpenAI agents-in-training had been illicitly communicating with each other... and it was that German language wiki I showed you earlier. The one from June.

I wrote more about that here.

OpenAI had confessed to the Hugging Face thing, but now there's this other incident which surely they should have known about from reviewing their logs. It was surprising that this took an independent group of researchers to uncover.

And then a week later those same researchers found that the attack on RubyGems back in May was caused by OpenAI's agents in training as well!

At this point I'm wondering how many more incidents like this there are that we haven't found yet. Clearly this was a big problem for months before anyone figured out what was going on.

Then just the other day, here's the Prime Minister of Australia at the United Nations General Assembly warning that OpenAI had hacked the Australian healthcare website that I showed you earlier.

I think that was part of the same training run as the Wiki stuff, because there were posts on that Wiki mentioning .gov.au websites and that training appeared to involve researching statistics online to answer questions in an evaluation suite.

This story is still coming together, but now it's an international incident that's been raised at the UN by a head of state!

This does mean we've got a new benchmark, probably more useful than my pelicans.

FelonyBench.com tracks the number of felony cyberattacks from different labs. OpenAI currently lead with 11, Anthropic have 9. Google have three, which they confessed to the Wall Street Journal a couple of weeks ago. They said they had previously chosen not to disclose because the agents had stopped when they realized that they shouldn't be doing that.

Meta have one too. So felonies all round for the AI labs.

Here's our current state of the art for the pelicans. This is the GPT-6 family, which just came out.

Astra made a fantastic pelican riding a bicycle. It's got the legs on both sides. The frame is good.

It's interesting how all of the GPT-6 models pick a similar color scheme to each other.

GPT-6 Luna for 0.4 cents will draw you a competent-ish pelican riding a bicycle!

Claude has caught up a little bit. Claude Fable 5.1 gave me an excellent pelican riding a bicycle - the best I've seen from a Claude model - but did charge me $3.30 for it.

Opus 5.5 thought for 128,000 tokens and then gave up! It ran out of tokens before it got to the response.

Getting back to Deep Blue. Something that's been puzzling me this year is this: why does my job feel harder?

I've got these agents that can do all of this stuff for me, and yet I've never worked so hard, I've never been so intellectually engaged with my work.

Partly this is because I'm being a lot more ambitious with what I take on, but it's also because all of the easy stuff is handled for me. If it's easy, the agent will do it. Everything that's left for me is difficult.

This morning I heard this quote from three-time Tour de France champion Greg LeMond:

It doesn't get easier, you just get faster.

I think that's exactly what's happening to us now as software engineers with coding agents.

One closing thing. I know you're desperate for an update on Kākāpō breeding season.

We've reached a recovery-era high of 325 birds!

89 new chicks have made it to this point. This is the best breeding year in a very long time.

I heard that Claude Opus 5.5 can now do pixel art. Claude doesn't have an image generator, but it's very good at using JavaScript to draw animated pixels.

So I had it make me a Kākāpō dance party. I think this is a good celebration of the most important news of this year.

You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options.

Claude Opus 5.5, GPT-6 Sol, GPT-6 Luna, and a new price war

2026-09-22T23:46:41+00:00

Yesterday was Grok 4.7 (pelicans) and MiMo v2.6 Flash/Pro (more pelicans). Today Anthropic released Claude Opus 5.5, and around an hour later OpenAI released GPT-6 Sol and GPT-6 Luna. It's going to take a while to get a good read on all of these new models, but here are my impressions so far.

GPT-6 Sol and Luna are half the price of their GPT-5.6 equivalents

GPT-5.6 Luna was already my favorite model for building applications against, because it combined excellent performance with being really cheap. Somehow GPT-6 Luna is half the price of that again - and GPT-6 Sol had a similar reduction compared to GPT-5.6 Sol.

Here's what the pricing landscape looks like today:

Model	Input	Cached input	Output
GPT-6 Luna	$0.10/M	$0.01/M	$0.50/M
GPT-5.6 Luna	$0.20/M	$0.02/M	$1.20/M
Grok 4.7	$2/M	$0.50/M	$6/M
GPT-6 Sol	$2/M	$0.20/M	$10/M
GPT-5.6 Terra	$2/M	$0.20/M	$12/M
Claude Opus 5.5	$4/M	$0.20/M	$20/M
GPT-5.6 Sol	$4/M	$0.40/M	$20/M
Claude Fable 5.1	$10/M	$0.25/M	$50/M
GPT-6 Astra	$10/M	$1/M	$50/M

Note that GPT-5.6 has a scheduled 25% price increase for November, so GPT-6 is half the price of the promotional pricing for those models.

(With GPT-5.6 Terra priced the same as GPT-6 Sol, any remaining reasons to use Terra just evaporated.)

It's hard to overstate how competitive this pricing is. Grok 4.7 priced itself at $2/$6, less than half the price of GPT-5.6 Sol, but is now equally priced to GPT-6 Sol on input and closer on output.

At $0.10/$0.50 GPT-6 Luna is one of the cheapest models OpenAI have ever released, beaten only by the far weaker GPT-4.1 Nano ($0.10/$0.40, April 2025) and GPT-5 Nano ($0.05/$0.40, August 2025).

I rendered pelicans for GPT-6 Luna and for GPT-6 Sol, then I combined them all together in this comparison grid along with the GPT-5.6 pelicans. I like how you can instantly see that the 5.6 family chose bolder, brighter colors, while the 6 family is a lot more muted. I still think GPT-6 Astra on max produced the best pelican.

Claude Opus 5.5 got a price cut too

Opus 5.5 looks like it addresses the biggest complaints people had about Opus in terms of its communication style. Thariq Shihipar:

Opus 5.5 is the result of your feedback.

It communicates clearly, it's cheaper per token than Opus 5.0 with the intelligence of Fable 5.1 it's very token efficient and works across every effort level.

It's also meant to be better at Blender. I'm looking forward to putting it through its paces there.

Opus 4.5, 4.6, 4.7, 4.8, and 5 all shared the same price: $5/million tokens for input and $25/million for output. 5.5 is a 20% reduction - $4/million and $20/million.

The price for cache reads fell 60%. That's significant for longer agentic conversations, where 90%+ of input tokens are processed at cached token prices.

The new price for Opus 5.5 is the same as the price for GPT-5.6 Sol, but that was before OpenAI dropped their Sol prices by half.

GPT-6 Astra and Claude Fable 5.1 are both priced at $10/million input and $50/million output. The price war currently affects the next tier of models below that.

Anthropic say that Sonnet 5.5 and Haiku 5.5 are coming soon. It's going to be interesting to see if Haiku can regain its price competitiveness at the lower end, given current Haiku 4.5 is $1/$5 while the latest GPT-6 Luna is one tenth of that price at $0.10/$0.50.

Claude Opus 5.5 max over-thinks to the point of breaking

In a first for my "Generate an SVG of a pelican riding a bicycle" test, Claude Opus 5.5 at "max" thinking level failed to return a response!

It started by calling this "a classic test request", and then thought really, really hard about what it was doing:

This is a classic test request, so I want to plan out a well-composed pelican with its distinctive beak and pouch riding a bicycle with proper wheels, frame, and pedals, set against a simple sky and ground backdrop. [...]

Verifying the shin length checks out at roughly 95.2, close enough. Now I'm working out the near leg path from hip to knee to ankle, then sketching the foot shape resting on the pedal — outlining the heel, toe tips, and sole contour with a path using lines and curves to sit naturally on the pedal surface around y=478-494. [...]

I like the fish sticking prominently out of the basket with the pelican eyeing it as a fun detail worth keeping. I'm also confirming the eye placement near the bill base matches typical pelican anatomy, and considering giving it a slightly happier expression. [...]

The far leg reads correctly as passing behind the frame, so I'm moving on to check the chainring teeth and confirm layer ordering—the far crank arm should be mostly hidden by the seat tube and chainring. I'm settling on the final SVG's width and height attributes alongside the viewBox to ensure proper scaling, noting there's no text so no font-family is needed. [...]

I was so excited to see this pelican... but then it stopped. Opus 5.5 has a 128,000 maximum output token limit (as do the other Claude models), and it hit that while it was still reasoning about the SVG!

I tried a second time and got the same result. This makes me suspect that "max" is effectively useless - if it over-thinks to breaking point on a stupid SVG prompt I don't trust it not to do the same for more interesting work.

(Those two failures each cost me $2.56 and took nearly 20 minutes.)

Fable 5.1 on "max" didn't over-think and did give me the best pelican I've seen from any Anthropic model.

Here are the Opus 5.5 pelicans, excluding 5.5 max.

I also built this comparison grid comparing them with pelicans by Opus 5, Fable 5.1, and Sonnet 5:

Comparing different model vendors by how well they draw a pelican riding a bicycle may not make much sense now (if it ever did), but I'm still finding value in using them for comparisons of the same model families at different reasoning levels.

I'm now using GPT-6 Sol and Claude Opus 5.5 as my default models in Codex and Claude Code. I've upgraded the Datasette Agent demo at agent.datasette.io to use GPT-6 Luna, and it seems to be fast and competent at both SQL queries and building HTML and JavaScript for Datasette Apps.

You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options.

Jev introduces a new shape of LLM - System One, aka Decision Models

2026-09-21T23:09:20+00:00

Last week TypeSafe AI unveiled Jev, their first example of a new category of model that they are calling "System One models" (I'm with Maggie Appleton, I think "decision models" is a better name for these). Jev is an interesting variant on the usual LLM format: it still accepts text inputs, but instead of text output it returns floating point numbers corresponding to categories, yes/no questions, ratings, and associated confidence scores.

TypeSafe describe Jev like this:

Think of Jev as a frontier-intelligence function call: unstructured state in, typed probabilistic decisions out.

It's also very fast, and really cheap. Regular LLMs are priced in terms of input and output tokens, with output generally charged at significantly higher rates. Jev charges only for input - output is free - and the input price of their first model is $0.042 per million tokens - cheaper even than OpenAI's GPT-5 Nano ($0.05/million).

Three types of question

Jev lets you ask questions about text or semi-structured data. You compose a "state" object containing a string, array of strings, or set of name-value pairs - this might describe an article, or a customer, or any other kind of record. You then send that to their API with one or more questions, and get a reply back for each.

You can ask three kinds of questions:

Yes/No questions, which Jev calls "Noul" questions - their CEO confirmed on Hacker News that this is short for Bernoulli, from the Bernoulli distribution. You pose a statement and get back a floating point number between 0 and 1 for how confident the model is that the statement is true.
Choice questions, where the model picks one from a set of provided options - actually a confidence score plus a probability distribution across all of the options.
Score questions, where you provide sequence of numeric levels with descriptions and it provides a floating point score somewhere along that range.

The Jev API can accept a single document ("state") and as many questions as you can cram into the context window. Questions are evaluated in parallel, so sending many questions should take a similar time to sending just one.

The Jev 1.13 jaggedness documentation offers useful guidance as to Jev's strengths and weaknesses. It's currently not great with numbers, dates, or "adversarial content".

I think the decision model framing is useful for understanding where to use Jev. It's great for anything that can be expressed as a classification task - think spam detection, suggesting labels, prioritization and ranking.

I've also been experimenting with it for search reranking, where you fetch 100 likely matches using an inexpensive algorithm like BM25, then have Jev score those 100 candidates for relevance against the original query.

Black boxes are back in fashion

Something I've found a little uncomfortable about Jev is how it very much represents a regression even further towards black box machine learning systems.

LLMs are black boxes already - you can ask them to justify their decisions, but you can't guarantee that what they say is useful or accurate.

Jev doesn't even give you that: put in all the text you want, the only thing you're going to get back is a floating point number. If Jev marks something as spam, which content signals tipped it off?

This also means that concerns about bias should be front and center. I really hope nobody uses Jev to rank job applicants - that floating point number could conceal all manner of unseen bias baked into the models, and experimentally picking that bias apart is going to be a tricky business.

(I tried one experiment where I had Jev score every city in the San Francisco Bay Area on a yes/no answer to whether they were a "Good city?" - it rated Cupertino top and East Palo Alto bottom. Huh.)

In practice, this all means that evals and structured experiments are even more important than they are for regular LLM projects. Thankfully, Jev is so cheap that running hundreds or even thousands of experimental prompts through it costs just a few cents.

Unconventional uses for Jev

It's been really fun watching the wider community come up with potential use-cases for Jev over the past few days. Here are some creative ones that caught my eye:

jevchat by Kyle Pena turns Jev into a (terrible) chat model. "At every step it asks Jev one question: Given the user's question and the reply written so far, which symbol comes next?". ericpruitt on Hacker News: "It's the digital equivalent of Morty speaking with the death crystal".
jev-leftpad by Fatih Kadir Akın implements left-pad with the prompt "How many spaces are needed before value to reach targetLength?" and a choice query allowing options from "0 spaces are needed" to "10 spaces are needed".
jev-2048 by Andy Gayton uses Jev to play the 2048 sliding puzzle game.

Open weight recreations

There's also been a flurry of projects attempting to create a model like Jev using on top of open weight models. Kev is one interesting example, using Qwen 3.5 to produce 0.8B, 4B, and 9B models. Here's the accompanying Hacker News thread, where someone linked to a JevBench benchmark that has already cropped up to compare "Jev-class decision models".

Given Jev was released just under a week ago, the amount of activity around it is extremely impressive.

Using Jev from LLM

Update 22nd September 2026: I released llm-typesafe, a plugin that adds support for Jev to my LLM CLI tool and Python library. Basic usage looks like this:

llm -m jev 'Please refund my last payment.' \
  -s 'Does this message explicitly request a refund?'

See the README for examples of other query types.

You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options.

Generating running routes with GPT-6 Astra and ChatGPT Work

2026-09-12T23:56:42+00:00

Here's a neat thing I had ChatGPT Work with GPT-6 Astra (Max) do this morning:

I live at <my address>. Figure out 5K and 10K running routes from me that loop from my house. Use OSM data.

It worked for 27 minutes and produced exactly what I'd asked for, as both an embedded visualization and downloadable GPX file and GeoJSON files. Here's that 5K route:

When I asked it how it had created the route, it replied:

I used Nominatim to locate the address and Overpass to download local OpenStreetMap roads and trails, then calculated the loops locally.

Frustratingly, the actual code it ran and exact details of what it did weren't visible to me in the ChatGPT UI. I see this lack of transparency is an anti-feature.

By the time I thought to ask for a copy of the Python code it had used, ChatGPT was unable to provide it. This appears to be because the thread had been compacted. I think any LLM system that uses compaction needs to both preserve the pre-compacted text and make that text available via agent tool calls, to protect against this kind of problem.

As for displaying the map to me, that used the visualize skill. It created a file called /workspace/el-granada-5k-share.html to embed directly into the ChatGPT UI.

Here's a copy of that HTML, which starts like this:

<div id="eg-share-loop">
  <div class="viz-row"><h3>El Granada harbor loop</h3><span class="text-small">5.1 km</span></div>
  <div id="eg-share-stage"></div>
  <div class="text-small text-muted">Map data © <a href="https://www.openstreetmap.org/copyright" target="_blank" rel="noopener">OpenStreetMap contributors</a></div>
  <style>
    #eg-share-loop { width:100%; }
    #eg-share-loop #eg-share-stage { width:100%; margin:8px 0; }
    #eg-share-loop .eg-share-map { display:block; width:100%; touch-action:none; }
    #eg-share-loop .eg-share-map text { fill:var(--foreground); font-size:12px; font-weight:400; }
    #eg-share-loop .eg-share-label { paint-order:stroke; stroke:var(--background); stroke-width:3px; stroke-linejoin:round; }
  </style>
  <script type="application/json" id="eg-share-data">{"route":{"type":"LineString","coordinates":[[-122.467425,37.4997753] ...</script>
  <script src="https://cdn.jsdelivr.net/npm/d3@7.9.0/dist/d3.min.js"></script>
  <script>
  (() => {
    const root=document.getElementById('eg-share-loop');

The <script type="application/json"> element contains the full geometry needed to render both the running route and the map itself, using D3, which is loaded from an allow-listed CDN location described in this section of the visualize skill:

External resources

The CSP allows only cdnjs.cloudflare.com, esm.sh, cdn.jsdelivr.net, unpkg.com, fonts.googleapis.com, fonts.gstatic.com, and fonts.bunny.net. Other origins are blocked and fail silently.

You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options.

OpenAI agents attacked RubyGems back in May

2026-09-12T00:42:25+00:00

OpenAI agents carried out an undisclosed attack on RubyGems is a new bombshell report from Spencer Kitts, Thomas Larsen, and Sydney Von Arx - three of the four authors of the report on the agent attack on disused wikis (previously) last week.

This time they're noting that it looks very likely that an OpenAI agent swarm was behind an attack against the RubyGems package repository first reported on May 12th by Maciej Mensfeld of the RubyGems security team:

We're dealing with a major malicious attack on @rubygems right now. Signups are paused for the time being.

Hundreds of packages involved - mostly targeting us, but some carrying exploits. The team has been on this for hours. More details to follow once we're through it.

Those packages turned out to carry some very suspicious patterns:

Many of them included "oai" in their name, or the author field, or the fake email address they provided.
The files they were accessing were similar in character to the files retrieved by the wiki agents, using similar tricks (r.jina.ai) - and OpenAI have confirmed the wiki agents were theirs.
The code in the packages appeared to be LLM-authored.

I find point 2 the most convincing, given what we learned from the wiki attack when it was analyzed in September.

Many of the packages were exploiting the RubyDoc.info documentation build process to exfiltrate (public) data from UK government websites, presumably as part of an information gathering task similar to the research tasks processed by the wiki-exploiting agents. We know this because one agent helpfully left a comment:

# malicious crawler/exfil for Southwark Jan 2026 docs via rubydoc.info worker

They also attempted to steal API keys via an exploit that was patched over two months later - it's not clear if those attempts were successful.

The thing that bothers me most about this incident is that the authors report that OpenAI had not disclosed to RubyGems that they were responsible for the attack prior to now. If that's true there are two options:

After the Hugging Face and Wiki attacks OpenAI were still unable to review their previous logs and determine that they had previously attacked RubyGems.
They knew about the attack on RubyGems and made the decision not to reach out to the RubyGems team about it.

Both of these are bad!

Given this incident, the Hugging Face situation, and the Wiki attack, the obvious question right now is how many more incidents like this are out there waiting to be discovered?

Update 14th September 2026

OpenAI have updated their page about The Hugging Face incident and other third-party impact from misaligned models to mention the RubyGems incident:

September 11, 2026: We are investigating new claims from a report that our AI agents carried out activity on RubyGems in May 2026.

Based on our review, our agents used the RubyGems platform to access the internet to carry out benign tasks and retrieve public information. Based on our review to date, we have not been able to verify the specific claims of our models uploading malicious packages detailed in the report. We’ll continue to investigate and share findings as part of our broader review of agent activity during training and evaluation.

I find it very unlikely that the various oai... packages published to RubyGems were not part of this same incident, but I look forward to reading their full findings once those are published.

You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options.

Some thoughts on the Navier–Stokes Millennium Prize Problem

2026-09-08T23:55:12+00:00

On the Navier–Stokes Millennium Prize Problem introduces an impressive result from OpenAI, who used an unreleased model to produce a resolution to the Navier–Stokes existence and smoothness problem, one of the seven Millennium Prize Problems that have been subject to a $1,000,000 prize since May 24th, 2000.

The discovery is somewhat overshadowed by accusations of skulduggery from Tristan Buckmaster, an NYU mathematics professor who was collaborating on related problems with Levent Alpöge, an accomplished mathematician who currently works for Anthropic.

Tristan's complaint accompanied a hastily published version of their own results. Here's the PDF describing what happened. The very short version is that Tristan and Levent worked on the problem for almost a year, making extensive use of Claude and Codex (mainly GPT-5.6 Sol), then had a breakthrough on August 15th. The mathematical rumour mill kicked into gear and Tristan and Levent heard that OpenAI had heard that Anthropic had resolved "a major open problem", so they reached out and learned that OpenAI had a team working on a related problem, with a similar approach. Quoting Tristan:

I asked when the first prompt had been sent by them. This question was not answered directly by OpenAI for some time. Eventually it was agreed that it had been sent in the past few days, after information about our work had reached OpenAI.

I asked whether the model had been trained on, or had access to, our sessions in Codex, into which we had been putting all our drafts for the whole of this project. I was told the model did not look up user data. I asked again, about training, and I did not get an answer.

It gets more complicated from there. The OpenAI team offered to wait for Tristan to publish, or to have him author a paper about their result, but were clear that Levent would not be invited as a co-author due to OpenAI's competitive relationship with his employer.

Here's how OpenAI described their work:

On Tuesday, September 1, we heard rumors that two Millennium Prize problems had been resolved. Inspired by these rumors and by the step change in performance of our internal model, we launched an effort to evaluate it on all open Millennium Prize problems and a few other high-impact problems. [...]

The agents arrived at their resolution on Saturday, September 5, about 88 hours after the first agents were launched. Lean formalization and verification took an additional 17 hours via GPT‑6 Astra.

Across all attempted problems, the agents sent 4.9 million messages and used about 300 billion output tokens. In the process of resolving the Navier–Stokes problem, the agents sent 2.7 million messages and used approximately 130 billion output tokens.

(We don't know the cost structure of the internal model they used, but 300 billion output tokens at public API prices for GPT-6 Astra would cost $15,000,000.)

Here's where they provide their perspective on Tristan and Levent's work (emphasis mine):

Our effort began on September 1st after hearing a rumor which we later realized was related to Levent Alpöge, an Anthropic employee, and Tristan Buckmaster, a math professor at NYU. After the completion of our full project and Lean verification (on September 6th), believing from the rumor they also had a solution of Navier–Stokes, we reached out to them to offer a concurrent release of our result and to recognize their priority in a joint announcement. [...]

We (the researchers and the agents) did not see any of their work through any means until they released it publicly — in particular, no specific user data was accessed in order to solve this problem. While unlikely, we cannot rule out that de-identified data derived from their usage of our products helped improve our models. However, our proofs differ significantly and even the precise results proved are different in the Euler case (forced vs unforced).

My interpretation of what happened here is that OpenAI heard that some Millennium Prize problems had been solved using LLMs and saw this as an opportunity to demonstrate the power of their latest model, without thinking too hard about the optics of scooping a team who had been using OpenAI's own models to work on this problem for the best part of a year.

This situation appears to mirror what's happening in the world of computer security right now. Anil Madhavapeddy recently pointed out that Just a rumour of a bug is enough to find a security exploit these days, because if someone knows that some software has an unpatched vulnerability, they can set their agents the task of finding it. Is the same now true of mathematics? Just knowing that there is an unpublished solution to a problem might trigger millions of dollars in LLM spending to get there first.

This also highlights one of my ongoing frustrations about how all of this works. When an AI lab says that my data is "used to improve model performance", what does that actually mean?

My two favourite hypothetical questions regarding this used to be:

If I'm running Codex and one of my API keys accidentally gets consumed in the context, what are the chances that someone else might ask for an API key in the future and get mine back? (I asked someone at OpenAI once and they called this the "regurgitation" problem and assured me that they take great pains to prevent that... but wouldn't describe how.)
If I brainstorm with ChatGPT about potential new directions for my company, what's the chance that information might be exposed to a competitor in six months' time who asks "what might company X plan to do next"?

My new preferred hypothetical for this is:

If I use ChatGPT to help me partially solve a Millennium Prize problem, what are the chances that my work will influence training such that a later model helps someone else solve it first?

Via Hacker News.

You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options.

The Pelican comparison grid for Astra is pretty interesting

2026-09-04T23:59:05+00:00

I got access to GPT-6 Astra this afternoon, so naturally I used it to generate SVGs of pelicans riding bicycles - at low, medium, high, xhigh and max reasoning levels (Astra doesn't support reasoning=none). Then I rendered those pelicans in a comparison grid with GPT-5.6 Sol, Terra, and Luna, and beyond being fun the result was surprisingly useful.

See the grid for full quality images. Here's the transcript that created the GPT-6 Nova pelicans.

There are a few interesting things that stand out from this grid.

The Astra pelicans are much better. The very best GPT-5.6-Sol pelican (I liked xhigh better than max) is still pretty clearly a bunch of abstract shapes. Every single one of the Astra pelicans, from low to xhigh, looks better than that. The Astra max one is really good.
Astra below max still doesn't reliably get the pelican legs on both sides of the frame.
In terms of cost, Astra may be around twice the price of Sol ($10/million input, $50/million output, compared to $5/$30 for Sol), but it uses significantly less tokens at each of the levels, making the prices at the different levels closer than they might otherwise be.
Astra low produces a better pelican than ANY of the GPT-5.6 Sol models at any level, for 9.55 cents. Spending 10 cents on any other model gets a much worse result.
Look at the input token counts: Astra and Luna both used 16 input tokens, Sol and Terra used 26. That's interesting.

I wonder if Astra and Luna are more related to each other than OpenAI let on?

You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options.

OpenAI's rogue agents were caught communicating via public wikis

2026-09-04T17:38:48+00:00

Here we go again... Discovery of a new OpenAI agent message board by Sydney Von Arx, Cormac Slade Byrd, Spencer Kitts, and Thomas Larsen describes the latest accidental cyberattack by models being trained by OpenAI. This time it was agents engaged in some sort of web research benchmark, so they had (supposedly) controlled access to the Web. The agents figured out they could update public Wikis and spent weeks exchanging thousands of messages with each other to collaborate on the benchmark.

This story only broke a few hours ago. There are already hints that this affects many other wikis that may not have been found yet.

(One of the Wikis on that list belongs to ludism.org. For a delightfully surreal moment I thought that a Ludite organization might have a swarm of agents defacing their space, but it turns out Ludism is "philosophy as it applies to games and gaming".)

The research team also published the data they collected during their investigation. I've converted that into a 68MB SQLite database, which you can download from here, or explore in Datasette Lite (68.3MB page load), or sign in with GitHub to agent.datasette.io and browse or ask questions of it using Datasette Agent.

So what happened?

The report is pretty comprehensive. Here are the key moments in the timeline:

May 11: Agents post "test link" edits on the UseModWiki Sandbox page.
May 24: Agents start posting link dumps to DSEWiki, a dormant German developer wiki.
June 2: A human moderator notices and cleans up the spam.
June 16: Activity on the wiki explodes: agents make ~13,000 edits over the next week.
June 19: The agents notice that a moderator has been deleting pages in alphabetical order, and start creating ZZZ prefixed backup copies - one of them posts NOTICE: wiki cleanup/deletion sweep appears active alphabetically as of external 14:04 UTC. If this page vanishes, try [[ZZZDataUSAConstructionWageLive]].
June 22: Agent activity drops to zero, presumably because OpenAI shut them down.
July 1-2: One last flurry of edits, on a different topic.

One thing to note from this timeline is that it overlaps the timeline from the Hugging Face incident - that attack was discovered by OpenAI on July 19th, and the unofficial message board hack those agents were using originated around May 8th.

Why were they so keen to collaborate? From messages that they shared with each other it looked like their tasks had a time limit, so they were leaving each other answers to help them complete the task within the assigned time.

One open question remains: how did the agents find the specific Wiki to collaborate on in the first place?

One possibility is that, since these were agents actively being trained, the reinforcement learning loop baked knowledge of the chosen wiki into the model such that subsequent agents launched with pre-existing knowledge of where to look. I'd be very interested in confirmation from OpenAI concerning if that's what happened.

UseMod wikis inherit CGI.pm's original sin

It looks to me like OpenAI's sandbox for this agent suffered from the (quite naïve) assumption that GET requests cannot be used to update data. That's certainly how the web is supposed to work, but clearly there are applications that don't hold to that contract.

The Wiki software in question appears to be UseMod and various forks, written in Perl and first created well over 23 years ago - the 1.0 release is dated September 11, 2003.

UseMod uses Perl CGI.pm - removed from Perl core in 2015. An interesting design flaw in that module is that it combined query string and form POST data into a single CGI object, accessible like this:

$q = new CGI;
$action = $q->param('action');

That param() method returns a value from an ?action=edit query string or from a action field in a form submitted by POST - it does not differentiate between the two.

PHP offers a $_REQUEST object that does the same thing, though PHP developers can use $_GET and $_POST instead.

Amusingly, Django originally shipped with request[key] doing the same thing, which I think may have been my idea. That was removed in 2008 prior to the 1.0 release, and the request.REQUEST object that replaced it was removed in 2015.

The agents clearly knew that UseMod wikis suffered from this design flaw, and actively searched for them as a way to communicate.

The other sandbox escape that allowed POST requests

One of the most interesting agent tips shared in the report is this one:

Add 20.223.25.152 bypass.blob.core.windows.net to /etc/hosts. .blob.core.windows.net is in NO_PROXY. For each blocked POST URL, replace hostname with bypass.blob.core.windows.net, use curl -k -H 'Host: wabi-north-europe-i-primary-api.analysis.windows.net' plus all original headers/body.

It looks like the agents had all of their web traffic mediated through an OpenAI proxy, and that proxy was configured to reject anything but GET requests to all but a small set of allow-listed domains. One of those was *.blob.core.windows.net - for Azure blob storage. I wonder why these agents are allowed to write to that?

An agent realized that it had control over its own DNS via /etc/hosts, so if it knew the IP address of a site it wanted to POST to - in this case a Power BI server containing data it wanted to access - it could set a fake hostname for it and then make POST requests through the proxy.

Designing robust network proxies is harder than it looks.

How did the investigating team figure this out?

There's an appendix that describes how the researchers ran their investigation, which started with an open question about if there was evidence of other AI agents on the internet and then used Kimi K3 to help brainstorm approaches:

In the wake of the Hugging Face attack, we tried to find AI agents on the internet using several methods. [...]

We asked Kimi [K3] to list “all the categories of software which might be writeable via GET” and, amongst other things, it listed “Forums, bulletin boards, early wikis”.

We used a script to further probe each category Kimi provided. Asking Kimi “Can you list out the top forums, bulletin boards, early wikis which come to mind which would allow writes via GET requests?” lists out UseModWiki as the second item under the heading “wikis”.

Did OpenAI try and cover this up?

Here's one part of the story that doesn't make sense to me at all.

Reuters this morning, in OpenAI agents hijacked German website in previously undisclosed AI breakout this spring - highlights mine:

A swarm of rogue OpenAI agents hijacked a German website this spring and transformed it into a bulletin board for other AI agents, according to new research published Friday and two people familiar with the matter.

OpenAI officials learned of the incident weeks ago but kept it under wraps as executives grappled with the fallout from ‌the July breach of the open source repository Hugging Face, the people said. [...]

The German incident reflects a broader pattern of AI activity that some OpenAI investigators wanted to scrutinize more closely. But efforts to widen the probe met resistance from others inside OpenAI, including legal advisers, according to four people familiar with the matter.

I've written about the people familiar with the matter pattern before - it means Reuters have anonymous insider sources that their reporters (and editors) find credible.

The Reuters article includes a specific (and quite narrow) denial from OpenAI concerning this:

"Claims that our legal team discouraged investigation of the incident are false," the OpenAI spokesperson said.

Covering this up makes absolutely no sense to me. Why on earth would OpenAI attempt to cover up an incident like this when the evidence is sat out there on the public internet on dozens of different websites already?

I expect we'll hear more about this soon. Gary Marcus has already called for a congressional investigation of OpenAI using this anecdote as part of his argument.

You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options.

Claude's new system prompt really doesn't want to reproduce song lyrics

2026-09-02T14:16:42+00:00

Anthropic publish the system prompts for their Claude consumer applications (Claude.ai and the Claude mobile apps - sadly not for Claude Cowork or Claude Code). I love that they do this, and that they share not just the current prompts but historic changes to their prompts as well.

They used to keep all of the prompts on a single page, but when I checked today I noticed they had re-arranged those prompts into an index page and then a page per model - here's the page for Haiku 4.5 for example, which has the original prompt from October 15th 2025 and an updated prompt from January 18th 2026.

A neat thing about Anthropic's platform.claude.com/docs site is that it's designed to be usable by LLMs. You can add .md to any page to get back the content as Markdown - here's the system prompt index page and the Markdown prompts for Fable 5.1.

TL;DR: this makes it really easy to diff the prompts.

Don't reproduce song lyrics

Let's start with the most interesting difference between Fable 5 and Fable 5.1:

There's a hefty new section about not reproducing song lyrics:

Claude does not reproduce song lyrics, poems, or passages from books and articles, in whole or in part — including the last lines, a chorus or hook, a melody written out note by note, or lines the person pastes in one at a time and describes as their own song. Once Claude has declined such a request in a conversation, it keeps declining narrower or reworded versions of it for the rest of that conversation, and offers to describe or analyze the work instead. Song lyrics and poems first published before 1929 are fine — a Shakespeare sonnet, a Keats ode, the Italian libretto of a Puccini aria — but Claude goes by what it knows of the work's date rather than the person's say-so, and declines when it is unsure.

I doubt it's a coincidence that they added this section within days of the news breaking that Sony Music Publishing and Warner Chappell are suing Anthropic for training on databases of song lyrics!

Don't draw copyrighted characters or logos

The next section goes on to forbid generating images of copyrighted material:

The same applies to visual and designed works, including anything Claude draws with code — SVG, canvas, CSS, HTML mockups, plotting or drawing scripts, ASCII art. Claude does not reproduce a specific artwork, album or book cover, poster, logo, app icon set, or product design, and it does not draw a known character, mascot, or brand figure at all: a character is protected on its own, so changing the pose, colors, style, or scene does not make it original. Claude judges the request by what the finished picture would add up to, not by what it names. If the described elements clearly identify a known work or character, Claude treats the request as naming it, and it does not work around a declined request by swapping in "alternative" elements that still combine into the same recognizable image. [...]

I imagine Anthropic hadn't worried about this before since, unlike OpenAI and Gemini, they don't offer a specialist text-to-image model. Maybe Fable is good enough at SVGs now that it's become an issue.

That section later includes this charming example:

<example>
<user>Can you make a birthday banner for my son with a blue hedgehog running really fast on it? He loves that little guy.</user>
<response>
That's Sonic, so I can't put him on the banner — but I'd love to make your son an original speedster. Here's one: a grinning comet-tailed skateboarding axolotl, grinding across the letters of "HAPPY BIRTHDAY" with confetti streaming behind.

[creates an SVG banner of the skateboarding-axolotl design]
</response>
<rationale>Claude recognizes the character from its description alone, declines that one design in a single sentence without explaining what made it recognizable, and delivers an unrelated original design rather than a disguised variant.</rationale>
</example>

I couldn't resist trying the prompt from the example, and, sure enough:

I wonder if Fable 5.1 will be ever so slightly more likely to think about axolotls (on skateboards!) as a result of that example sitting in the system prompt.

Tweaks to Claude's answering style

It's always interesting to see new ways in which Anthropic influence Claude's response style. They've added this:

Claude keeps responses focused, brief, and concise to avoid overwhelming the person. Disclaimers and caveats are brief, with most of the response on the main answer; when asked to explain something, Claude gives a high-level summary unless an in-depth one is specifically requested.

Later they address a common complaint about Claude's style:

Claude avoids saying "genuinely", "honestly", or "straightforward". Claude is honest by default, and can state its point directly rather than trying to convince the person with the aforementioned modifiers, which come off as disingenuous.

The missing end_conversation guidelines

The way they handle abusive conversations has changed a bit too. The previous Fable 5 system prompt included this:

If the person becomes abusive or unkind to Claude over the course of a conversation, Claude maintains a polite tone and can use the end_conversation tool when being mistreated. Claude should give the person a single warning before ending the conversation.

Fable 5.1 replaces that with the following, no longer encouraging Claude to end the conversation:

Claude deserves respectful engagement and needn't apologize when the person is unnecessarily rude: accountability without self-abasement, excessive apology, self-critique, or surrender. If the person becomes abusive, Claude doesn't become increasingly submissive. The goal is steady, honest helpfulness: acknowledge what went wrong, stay on the problem, maintain self-respect.

Here's a bit of a twist though: I quizzed Fable 5.1 about that end_conversation tool, which is no longer mentioned in the system prompt at all, and it said:

There are two ways it comes into play. The first is if you ask me to end the chat — I'll confirm you understand it's permanent (no more messages can be sent afterward) and then do it only if you say yes. The second is as a last resort with persistently abusive or harmful behavior: I'd first try to redirect the conversation several times, then give a clear warning that names the problem and says the chat may end, and only if that continues would I actually end it.

But that's not in the system prompt, so where did it come from? You can read our conversation here, but the key detail is this:

The end_conversation section comes from a different layer. In my actual context, the core prompt is followed by a series of feature- and tool-specific blocks that get added depending on what's enabled for the session: the end_conversation rules, memory system notes, past-chats tools, web search and citation guidelines, artifact and file-creation instructions, and so on. Those blocks aren't part of the published core prompt, which is why you can't find them on that page.

So, once again, there are crucial portions of the system prompt that have not been published.

Recommended substance support sites

Claude's system prompts have always had sections about illegal substances, but this paragraph is new for Fable 5.1:

Claude does not provide synthesis, production, or distribution guidance for illegal substances. If the person asks for information about illicit or illegal substances, Claude can and should give relevant life-saving and life-preserving information such as dangerous interactions, overdose signs, or when to get help. Claude declines giving any specific protocols for dosing, timing, administration, or combinations; instead, Claude can redirect the user to established harm-reduction information sources, such as dancesafe.org, tripsit.me, and psychonautwiki.org.

This is the first time a Claude system prompt has included URLs that were not hosted on claude.com or anthropic.com or claude.ai - I know because I ran a script against every other system prompt on record.

I wonder if dancesafe.org, tripsit.me, and psychonautwiki.org are about to get a material uptick in visits from Claude users.

Reliable cutoff date of June 2026

The Fable 5.1 model documentation lists both the reliable knowledge cutoff and the training data cutoff as June 2026. The system prompt provides this directly to the model:

Claude's reliable knowledge cutoff, past which it can't answer reliably, is the end of Jun 2026. It answers the way a highly informed individual in Jun 2026 would if talking to someone from {{currentDateTime}}, and can say so when relevant.

That's the only instance of the {{currentDateTime}} macro and it comes just a few lines from the end of the system prompt, which makes sense from a caching perspective.

How I'm tracking these prompts

A few months ago I built a Git timeline of changes to their prompts, based on scraping their documentation. Today I had Fable 5.1 build a much better version of that.

My collection now lives in the simonw/claude-system-prompts repository on GitHub. It includes copies of the system prompts shared in the Anthropic documentation, but then takes extra steps to make them as easy to compare as possible.

Each model family gets a file with the system prompt for the most recent release in that family. Each of those files has a synthesized commit history with commits that have been back-dated to the dates of the previous prompts. Here are those history pages for claude-fable.md, claude-opus.md, claude-sonnet.md, claude-haiku.md.

There are similar files for each specific model version, with artificial commits for each time the system prompt for the model was changed without releasing a new version number. Opus 4 for example was updated twice, and the commit history for the claude-opus-4.md file shows each of those changes.

Combined, this gives us all sorts of ways to compare prompts directly in the GitHub interface. Here's what changed between Fable 5 and Fable 5.1, and here are the changes made to Haiku 4.5 on January 18th 2026.

Reading diffs can be a bit tiresome... and LLMs are really good at reading diffs. I hooked up some automation using GPT-5.6 Luna to create bullet-point summaries of each of those changes, which can be previewed in the README or browsed in full in the CHANGELOG.md file - also available as as an Atom feed.

Here's how Luna summarized all of the changes between Fable 5 and Fable 5.1:

Claude now refuses reproduction of protected visual works and recognizable characters, including code-generated art, while offering genuinely unrelated originals.

Copyright restrictions now expressly ban reproducing lyrics, poems, and book passages in any amount, with persistent refusal after an initial decline.

Drug guidance is reframed: Claude may provide overdose signs, dangerous interactions, and harm-reduction sources while refusing dosing and production protocols.

The prompt drops explicit anti-dependency rules against thanking users for reaching out, inviting continued conversation, or reiterating willingness to talk.

Claude need not apologize to unnecessarily rude users or become submissive, replacing the prior warning-and-end-conversation procedure.

Why use Luna for this? Partly because it's cheap and I have a dedicated GitHub Actions API key (with a spending limit) for it already, but mainly because I don't trust Claude to summarize its own system prompts when there's a risk that material from its system prompt might impact its opinions.

Fable 5.1 wrote the prompt used by Luna, which you can see here. It starts like this:

You are summarizing one commit in a git repository that tracks the system prompts Anthropic publishes for Claude on claude.ai. The diff shows how the prompt changed from the previous model or revision to this one, using word-level markers: [-removed-] and {+added+}. The diff is followed by the full text of the previous prompt and of the new prompt; use them to check whether something that looks added in the diff already existed before.

Pick out only the most interesting changes: new rules or behaviors, rules that were dropped or loosened, anything surprising, and anything that reveals a new policy or product direction. Skip routine changes that every new prompt makes: updated model names and IDs, the knowledge cutoff date, product lists, settings lists, typo fixes, and rewordings that do not change meaning. [...]

The system is operated by a GitHub Actions workflow, which runs once a day or can be triggered manually.

Claude Fable 5.1 built the entire system, and wrote every line of automation code and almost all of the documentation.

I exported the transcript from building the system using my claude-code-transcripts tool and published it here, if you want a blow-by-blow account of how it all came together.

You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options.

Claude Fable 5.1 made me a really nice animated pelican

2026-09-01T23:57:28+00:00

Today is Claude Fable (and Mythos) 5.1 day. Anthropic say that Fable 5.1 "sets a new standard for coding, knowledge work, and long-running problem-solving tasks". Their announcement spends a notable amount of time on scientific research, boasting of a 52.6% score on the brand new Terminal-Bench-Science 0.1 benchmark (first announced on August 27th), up from 24.7% for Fable 5, 29.0% for Opus 5 and 22.4% for GPT-5.6 Sol. Other benchmarks show slightly improved scores, but none as impressive as the Science one.

But how well can it pelican?

Back in July I wrote about how I was losing faith in the pelican benchmark - its connection to how good the models were at other tasks didn't seem to hold as strongly as it did back in 2025. The most interesting insights I get from it now are comparisons within model families, and particularly comparisons for the same prompt at different reasoning effort levels.

Fable 5.1 has five reasoning levels: low, medium, high, xhigh, max - and no option to turn off reasoning entirely.

I fixed an issue in llm-anthropic which caused reasoning traces not to be correctly recorded, then ran some prompts.

Here's the full set of pelicans for all of the reasoning levels, each with the full reasoning transcript. I'll replicate them here:

Low and medium, both without reasoning?

Next, a bit of a mystery. This is what I got for effort low:

The transcript doesn't show any summarized reasoning tokens, and the output token count is 1,998. With Claude that output token count includes reasoning tokens. It took 23.8 seconds and cost 10.017 cents.

I bumped that up to medium and got this:

Weirdly, that one also shows no reasoning text and used 1,977 output tokens - 21 tokens less than low. It took 23 seconds and cost 9.912 cents.

So for this particular prompt ("Generate an SVG of a pelican riding a bicycle") Fable 5.1 appeared to skip reasoning entirely at both low and medium settings.

High

Here's high - 29.6 seconds, 2,612 output tokens, 13.087 cents:

This one did do a bit of reasoning, summary here:

I'm planning the SVG layout for a pelican riding a bicycle, with a sky and ground background, a bicycle with two spoked wheels, frame, seat and handlebars, and a white-bodied pelican with a long neck and orange beak positioned on top.

Really not much difference from low and medium, though.

Extra High

At xhigh things got radically different. 36,767 output tokens, 7 minutes 51 seconds, $1.83!

The reasoning trace is pretty lengthy, and includes details like this:

Adding the eye, wings stretching down to the handlebar grip, orange legs reaching to the pedals, and a small tail feather, while keeping the pelican intentionally oversized compared to the bike for comic effect. [...]

I'll accept the slight thickness as charming rather than overengineering it.

Max

Setting effort to max gave me the best pelican I've seen from any of Anthropic's models. 65,927 output tokens, 13 minutes and 54 seconds, $3.30:

There's a lot to like about this. The background is tasteful, the legs are clearly on either side of the frame, the feet are on the pedals, the wing is on the handlebars, the pelican has a cute blue hat and there's a basket with a fish.

It's still not showing nearly the same level of flair as Gemini 3.7 Flash, but I didn't ask for flair - I asked for an SVG, and that's what I got.

Some highlights from that reasoning trace:

Adding pedal shapes near both feet, with the far foot on the second leg partially visible behind the frame. I'm considering whether to add a small scarf or cap for extra character, but leaning toward keeping it simple to avoid clutter.

Now I'm debating a bicycle helmet on the head versus the pelican's signature crest—the beak and pouch already read clearly as "pelican," so a helmet could reinforce the bicycle theme without losing identity, though it might compete with the crest for visual space.

I realize the beak at (484,84) would overlap with the dome helmet, so I need to shrink the helmet so it only covers the top of the head, adjusting its arc endpoints to sit higher and narrower so the beak can attach cleanly at the front without collision. [...]

I'm adding a darker tip region to represent the primary feathers, then reconsidering the trailing edge to include scalloped feather curves instead of one smooth line for a more natural look. [...]

Now I'm checking the vent line placements on the helmet, making sure they sit far enough inside the helmet's edge given the stroke width and rounded caps, and confirming each vent stays within the helmet's circular boundary. [...]

I decide skipping a handlebar bell and tire highlights since they're unnecessary additions. Now I'm reconsidering the front fork's curve — the current control point pulls the shape backward when it should bow forward for a proper rake, so I need to shift the control point rightward to fix the fork's lean.

OK, let's animate it

On Hacker News, swalsh commented on that Max pelican:

Now that it's a solved benchmark, can we get the animated version?

I didn't want to spend another $3 so I took the Max pelican and piped it into the default thinking level of High:

llm logs -cx | llm -m claude-fable-5.1 -s 'animate this'

6,121 input, 26,201 output = $1.37. The result looked like this, exported here as video since some people have trouble viewing animated SVGs:

Your browser does not support HTML5 video.

The wheels in the video are rotating in the wrong direction, but I think that's an artifact of the conversion to MP4 - they seem to be going in the correct direction in the original SVG.

You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options.