Simon Willison's Weblog: til

Adding TILs, releases, museums, tools and research to my blog

2026-02-20T23:47:10+00:00

I've been wanting to add indications of my various other online activities to my blog for a while now. I just turned on a new feature I'm calling "beats" (after story beats, naming this was hard!) which adds five new types of content to my site, all corresponding to activity elsewhere.

Here's what beats look like:

Those three are from the 30th December 2025 archive page.

Beats are little inline links with badges that fit into different content timeline views around my site, including the homepage, search and archive pages.

There are currently five types of beats:

Releases are GitHub releases of my many different open source projects, imported from this JSON file that was constructed by GitHub Actions.
TILs are the posts from my TIL blog, imported using a SQL query over JSON and HTTP against the Datasette instance powering that site.
Museums are new posts on my niche-museums.com blog, imported from this custom JSON feed.
Tools are HTML and JavaScript tools I've vibe-coded on my tools.simonwillison.net site, as described in Useful patterns for building HTML tools.
Research is for AI-generated research projects, hosted in my simonw/research repo and described in Code research projects with async coding agents like Claude Code and Codex.

That's five different custom integrations to pull in all of that data. The good news is that this kind of integration project is the kind of thing that coding agents really excel at. I knocked most of the feature out in a single morning while working in parallel on various other things.

I didn't have a useful structured feed of my Research projects, and it didn't matter because I gave Claude Code a link to the raw Markdown README that lists them all and it spun up a parser regex. Since I'm responsible for both the source and the destination I'm fine with a brittle solution that would be too risky against a source that I don't control myself.

Claude also handled all of the potentially tedious UI integration work with my site, making sure the new content worked on all of my different page types and was handled correctly by my faceted search engine.

Prototyping with Claude Artifacts

I actually prototyped the initial concept for beats in regular Claude - not Claude Code - taking advantage of the fact that it can clone public repos from GitHub these days. I started with:

Clone simonw/simonwillisonblog and tell me about the models and views

And then later in the brainstorming session said:

use the templates and CSS in this repo to create a new artifact with all HTML and CSS inline that shows me my homepage with some of those inline content types mixed in

After some iteration we got to this artifact mockup, which was enough to convince me that the concept had legs and was worth handing over to full Claude Code for web to implement.

If you want to see how the rest of the build played out the most interesting PRs are Beats #592 which implemented the core feature and Add Museums Beat importer #595 which added the Museums content type.

Tags: blogging, museums, ai, til, generative-ai, llms, ai-assisted-programming, claude-artifacts, claude-code, site-upgrades

TIL: Running OpenClaw in Docker

2026-02-01T23:59:13+00:00

TIL: Running OpenClaw in Docker

I've been running OpenClaw using Docker on my Mac. Here are the first in my ongoing notes on how I set that up and the commands I'm using to administer it.

Here's a screenshot of the web UI that this serves on localhost:

Tags: ai, docker, til, generative-ai, llms, ai-agents, openclaw

TIL from taking Neon I at the Crucible

2026-01-11T17:35:57+00:00

TIL from taking Neon I at the Crucible

Things I learned about making neon signs after a week long intensive evening class at the Crucible in Oakland.

Tags: art, til

TIL: Downloading archived Git repositories from archive.softwareheritage.org

2025-12-30T23:51:33+00:00

TIL: Downloading archived Git repositories from archive.softwareheritage.org

Back in February I blogged about a neat Python library called sqlite-s3vfs for accessing SQLite databases hosted in an S3 bucket, released as MIT licensed open source by the UK government's Department for Business and Trade.

I went looking for it today and found that the github.com/uktrade/sqlite-s3vfs repository is now a 404.

Since this is taxpayer-funded open source software I saw it as my moral duty to try and restore access! It turns out a full copy had been captured by the Software Heritage archive, so I was able to restore the repository from there. My copy is now archived at simonw/sqlite-s3vfs.

The process for retrieving an archive was non-obvious, so I've written up a TIL and also published a new Software Heritage Repository Retriever tool which takes advantage of the CORS-enabled APIs provided by Software Heritage. Here's the Claude Code transcript from building that.

Via Hacker News comment

Tags: archives, git, github, open-source, tools, ai, til, generative-ai, llms, ai-assisted-programming, claude-code

TIL: Subtests in pytest 9.0.0+

2025-12-05T06:03:29+00:00

TIL: Subtests in pytest 9.0.0+

I spotted an interesting new feature in the release notes for pytest 9.0.0: subtests.

I'm a big user of the pytest.mark.parametrize decorator - see Documentation unit tests from 2018 - so I thought it would be interesting to try out subtests and see if they're a useful alternative.

Short version: this parameterized test:

@pytest.mark.parametrize("setting", app.SETTINGS)
def test_settings_are_documented(settings_headings, setting):
    assert setting.name in settings_headings

Becomes this using subtests instead:

def test_settings_are_documented(settings_headings, subtests):
    for setting in app.SETTINGS:
        with subtests.test(setting=setting.name):
            assert setting.name in settings_headings

Why is this better? Two reasons:

It appears to run a bit faster
Subtests can be created programatically after running some setup code first

I had Claude Code port several tests to the new pattern. I like it.

Tags: python, testing, ai, pytest, til, generative-ai, llms, ai-assisted-programming, coding-agents, claude-code

TIL: Dependency groups and uv run

2025-12-03T05:55:23+00:00

TIL: Dependency groups and uv run

I wrote up the new pattern I'm using for my various Python project repos to make them as easy to hack on with uv as possible. The trick is to use a PEP 735 dependency group called dev, declared in pyproject.toml like this:

[dependency-groups]
dev = ["pytest"]

With that in place, running uv run pytest will automatically install that development dependency into a new virtual environment and use it to run your tests.

This means you can get started hacking on one of my projects (here datasette-extract) with just these steps:

git clone https://github.com/datasette/datasette-extract
cd datasette-extract
uv run pytest

I also split my uv TILs out into a separate folder. This meant I had to setup redirects for the old paths, so I had Claude Code help build me a new plugin called datasette-redirects and then apply it to my TIL site, including updating the build script to correctly track the creation date of files that had since been renamed.

Tags: packaging, python, ai, til, generative-ai, llms, ai-assisted-programming, uv, coding-agents, claude-code

Using Codex CLI with gpt-oss:120b on an NVIDIA DGX Spark via Tailscale

2025-11-07T07:23:12+00:00

Using Codex CLI with gpt-oss:120b on an NVIDIA DGX Spark via Tailscale

Inspired by a YouTube comment I wrote up how I run OpenAI's Codex CLI coding agent against the gpt-oss:120b model running in Ollama on my NVIDIA DGX Spark via a Tailscale network.

It takes a little bit of work to configure but the result is I can now use Codex CLI on my laptop anywhere in the world against a self-hosted model.

I used it to build this space invaders clone.

Tags: ai, tailscale, til, generative-ai, local-llms, llms, nvidia, coding-agents, space-invaders, codex-cli, nvidia-spark

Curiosity-driven blogging

2025-10-31T17:09:56+00:00

My piece this morning about the Marimo acquisition is an example of a variant of a TIL - I didn't know much about CoreWeave, the acquiring company, so I poked around to answer my own questions and then wrote up what I learned as a short post. Curiosity-driven blogging if you like.

Tags: blogging, til

TIL: Testing different Python versions with uv with-editable and uv-test

2025-10-09T03:37:06+00:00

TIL: Testing different Python versions with uv with-editable and uv-test

While tinkering with upgrading various projects to handle Python 3.14 I finally figured out a universal uv recipe for running the tests for the current project in any specified version of Python:

uv run --python 3.14 --isolated --with-editable '.[test]' pytest

This should work in any directory with a pyproject.toml (or even a setup.py) that defines a test set of extra dependencies and uses pytest.

The --with-editable '.[test]' bit ensures that changes you make to that directory will be picked up by future test runs. The --isolated flag ensures no other environments will affect your test run.

I like this pattern so much I built a little shell script that uses it, shown here. Now I can change to any Python project directory and run:

uv-test

Or for a different Python version:

uv-test -p 3.11

I can pass additional pytest options too:

uv-test -p 3.11 -k permissions

Tags: python, testing, pytest, til, uv

TIL: Running a gpt-oss eval suite against LM Studio on a Mac

2025-08-17T03:46:21+00:00

TIL: Running a gpt-oss eval suite against LM Studio on a Mac

The other day I learned that OpenAI published a set of evals as part of their gpt-oss model release, described in their cookbook on Verifying gpt-oss implementations.

I decided to try and run that eval suite on my own MacBook Pro, against gpt-oss-20b running inside of LM Studio.

TLDR: once I had the model running inside LM Studio with a longer than default context limit, the following incantation ran an eval suite in around 3.5 hours:

mkdir /tmp/aime25_openai
OPENAI_API_KEY=x \
  uv run --python 3.13 --with 'gpt-oss[eval]' \
  python -m gpt_oss.evals \
  --base-url http://localhost:1234/v1 \
  --eval aime25 \
  --sampler chat_completions \
  --model openai/gpt-oss-20b \
  --reasoning-effort low \
  --n-threads 2

My new TIL breaks that command down in detail and walks through the underlying eval - AIME 2025, which asks 30 questions (8 times each) that are defined using the following format:

{"question": "Find the sum of all integer bases $b>9$ for which $17_{b}$ is a divisor of $97_{b}$.", "answer": "70"}

Tags: python, ai, til, openai, generative-ai, local-llms, llms, evals, uv, lm-studio, gpt-oss

simonw/codespaces-llm

2025-08-13T05:39:07+00:00

simonw/codespaces-llm

GitHub Codespaces provides full development environments in your browser, and is free to use with anyone with a GitHub account. Each environment has a full Linux container and a browser-based UI using VS Code.

I found out today that GitHub Codespaces come with a GITHUB_TOKEN environment variable... and that token works as an API key for accessing LLMs in the GitHub Models collection, which includes dozens of models from OpenAI, Microsoft, Mistral, xAI, DeepSeek, Meta and more.

Anthony Shaw's llm-github-models plugin for my LLM tool allows it to talk directly to GitHub Models. I filed a suggestion that it could pick up that GITHUB_TOKEN variable automatically and Anthony shipped v0.18.0 with that feature a few hours later.

... which means you can now run the following in any Python-enabled Codespaces container and get a working llm command:

pip install llm
llm install llm-github-models
llm models default github/gpt-4.1
llm "Fun facts about pelicans"

Setting the default model to github/gpt-4.1 means you get free (albeit rate-limited) access to that OpenAI model.

To save you from needing to even run that sequence of commands I've created a new GitHub repository, simonw/codespaces-llm, which pre-installs and runs those commands for you.

Anyone with a GitHub account can use this URL to launch a new Codespaces instance with a configured llm terminal command ready to use:

codespaces.new/simonw/codespaces-llm?quickstart=1

While putting this together I wrote up what I've learned about devcontainers so far as a TIL: Configuring GitHub Codespaces using devcontainers.

Tags: github, projects, python, ai, til, openai, generative-ai, llms, llm, github-codespaces, anthony-shaw

TIL: Rate limiting by IP using Cloudflare's rate limiting rules

2025-07-03T21:16:51+00:00

TIL: Rate limiting by IP using Cloudflare's rate limiting rules

My blog started timing out on some requests a few days ago, and it turned out there were misbehaving crawlers that were spidering my /search/ page even though it's restricted by robots.txt.

I run this site behind Cloudflare and it turns out Cloudflare's WAF (Web Application Firewall) has a rate limiting tool that I could use to restrict requests to /search/* by a specific IP to a maximum of 5 every 10 seconds.

Tags: rate-limiting, security, cloudflare, til

TIL: Using Playwright MCP with Claude Code

2025-07-01T23:55:09+00:00

TIL: Using Playwright MCP with Claude Code

Inspired by Armin ("I personally use only one MCP - I only use Playwright") I decided to figure out how to use the official Playwright MCP server with Claude Code.

It turns out it's easy:

claude mcp add playwright npx '@playwright/mcp@latest'
claude

The claude mcp add command only affects the current directory by default - it gets persisted in the ~/.claude.json file.

Now Claude can use Playwright to automate a Chrome browser! Tell it to "Use playwright mcp to open a browser to example.com" and watch it go - it can navigate pages, submit forms, execute custom JavaScript and take screenshots to feed back into the LLM.

The browser window stays visible which means you can interact with it too, including signing into websites so Claude can act on your behalf.

Tags: armin-ronacher, til, playwright, ai-assisted-programming, anthropic, claude, claude-code

TIL: SQLite triggers

2025-05-10T05:20:45+00:00

TIL: SQLite triggers

I've been doing some work with SQLite triggers recently while working on sqlite-chronicle, and I decided I needed a single reference to exactly which triggers are executed for which SQLite actions and what data is available within those triggers.

I wrote this triggers.py script to output as much information about triggers as possible, then wired it into a TIL article using Cog. The Cog-powered source code for the TIL article can be seen here.

Tags: python, sql, sqlite, til

TIL: Styling an HTML dialog modal to take the full height of the viewport

2025-03-14T23:13:55+00:00

TIL: Styling an HTML dialog modal to take the full height of the viewport

I spent some time today trying to figure out how to have a modal <dialog> element present as a full height side panel that animates in from the side. The full height bit was hard, until Natalie helped me figure out that browsers apply a default max-height: calc(100% - 6px - 2em); rule which needs to be over-ridden.

Also included: some spelunking through the HTML spec to figure out where that calc() expression was first introduced. The answer was November 2020.

Tags: css, html, natalie-downe, web-standards, til

Using a Tailscale exit node with GitHub Actions

2025-02-23T02:49:32+00:00

Using a Tailscale exit node with GitHub Actions

New TIL. I started running a git scraper against doge.gov to track changes made to that website over time. The DOGE site runs behind Cloudflare which was blocking requests from the GitHub Actions IP range, but I figured out how to run a Tailscale exit node on my Apple TV and use that to proxy my shot-scraper requests.

The scraper is running in simonw/scrape-doge-gov. It uses the new shot-scraper har command I added in shot-scraper 1.6 (and improved in shot-scraper 1.7).

Tags: github, scraping, github-actions, tailscale, til, git-scraping, shot-scraper

TIL: Downloading every video for a TikTok account

2025-01-19T02:05:44+00:00

TIL: Downloading every video for a TikTok account

TikTok may or may not be banned in the USA within the next 24 hours or so. I figured out a gnarly pattern for downloading every video from a specified account, using browser console JavaScript to scrape the video URLs and yt-dlp to fetch each video. As a bonus, I included a recipe for generating a Whisper transcript of every video with mlx-whisper and a hacky way to show a progress bar for the downloads.

Tags: til, whisper, tiktok, speech-to-text

TIL: Using uv to develop Python command-line applications

2024-10-24T05:56:21+00:00

TIL: Using uv to develop Python command-line applications

I've been increasingly using uv to try out new software (via uvx) and experiment with new ideas, but I hadn't quite figured out the right way to use it for developing my own projects.

It turns out I was missing a few things - in particular the fact that there's no need to use uv pip at all when working with a local development environment, you can get by entirely on uv run (and maybe uv sync --extra test to install test dependencies) with no direct invocations of uv pip at all.

I bounced a few questions off Charlie Marsh and filled in the missing gaps - this TIL shows my new uv-powered process for hacking on Python CLI apps built using Click and my simonw/click-app cookecutter template.

Tags: cli, packaging, pip, python, til, cookiecutter, uv, astral, charlie-marsh

Julia Evans: TIL

2024-10-24T05:52:10+00:00

Julia Evans: TIL

I've always loved how Julia Evans emphasizes the joy of learning and how you should celebrate every new thing you learn and never be ashamed to admit that you haven't figured something out yet. That attitude was part of my inspiration when I started writing TILs a few years ago.

Julia just started publishing TILs too, and I'm delighted to learn that this was partially inspired by my own efforts!

Tags: blogging, julia-evans, til

Write about what you learn. It pushes you to understand topics better.

2023-08-14T14:50:48+00:00

Write about what you learn. It pushes you to understand topics better.

Addy Osmani clearly articulates why writing frequently is such a powerful tool for learning more effectively. This post doesn’t mention TILs but it perfectly encapsulates the value I get from publishing them.

Via Hacker News

Tags: blogging, writing, til, addy-osmani

Annotated explanation of David Beazley's dataklasses

2021-12-20T05:05:04+00:00

Annotated explanation of David Beazley's dataklasses

David Beazley released a self-described “deliciously evil spin on dataclasses” that uses some deep Python trickery to implement a dataclass style decorator which creates classes that import 15-20 times faster than the original. I put together a heavily annotated version of his code while trying to figure out how all of the different Python tricks in it work.

Via @simonw

Tags: david-beazley, python, til

One year of TILs

2021-05-02T18:01:44+00:00

Just over a year ago I started tracking TILs, inspired by Josh Branchaud's collection. I've since published 148 TILs across 43 different topics. It's a great format!

TIL stands for Today I Learned. The thing I like most about TILs is that they drop the barrier to publishing something online to almost nothing.

If I'm writing a blog entry, I feel like it needs to say something new. This pressure for originality leads to vast numbers of incomplete, draft posts and a sporadic publishing schedule that trends towards not publishing anything at all.

(Establishing a weeknotes habit has helped enormously here too.)

The bar for a TIL is literally "did I just learn something?" - they effectively act as a public notebook.

They also reflect my values as a software engineer. The thing I love most about this career is that the opportunities to learn new things never reduce - there will always be new sub-disciplines to explore, and I aspire to learn something new every single working day.

My hope is that by publishing a constant stream of TILs I can reinforce the idea that even if you've been working in this industry for twenty years there will always be new things to learn, and learning any new trick - even the most basic thing - should be celebrated.

Tags: blogging, til

Weeknotes: Working on my screenplay

2020-05-14T04:53:46+00:00

I'm taking an Introduction to Screenwriting course with Adam Tobin at Stanford, and my partial screenplay is due this week. I'm pulling together some scenes that tell the story of the Russian 1917 February Revolution and the fall of the Tsar through the lens of the craftsmen working on the Tsar's last Fabergé egg. So I've not been spending much time on anything else.

Some brief bullet points for this week's software projects:

Released version 0.1 of datasette-media, a new plugin that allows Datasette to serve files from disk based on executing a SQL query to find the file to return. I'm building it to help make photos-to-sqlite more immediately useful.
Released Datasette 0.42 with improved (and now documented) internal methods to allow plugins to execute read-only SQL queries. I needed these for datasette-media.
Released sqlite-utils 2.9 with new CLI commands sqlite-utils drop-table and sqlite-utils drop-view.
Released sqlite-utils 2.9.1 with a tiny cosmetic improvement: the PyPI project page now shows project links! See this TIL for details.

I've also started adding changelog badges to various projects, showing the latest release version according to GitHub and linking to that project's changelog. Datasette, photos-to-sqlite, sqlite-utils all have these now.

TIL this week

Tags: projects, stanford, screen-writing, datasette, weeknotes, til

Restricting SSH connections to devices within a Tailscale network

2020-04-23T18:28:05+00:00

Restricting SSH connections to devices within a Tailscale network

TIL how to run SSH on a VPS instance (in this case Amazon Lightsail) such that it can only be SSHd to by devices connected to a private Tailscale VPN.

Tags: security, ssh, tailscale, til

Using a self-rewriting README powered by GitHub Actions to track TILs

2020-04-20T01:38:15+00:00

I've started tracking TILs - Today I Learneds - inspired by this five-year-and-counting collection by Josh Branchaud on GitHub (found via Hacker News). I'm keeping mine in GitHub too, and using GitHub Actions to automatically generate an index page README in the repository and a SQLite-backed search engine.

TILs

Josh describes his TILs like this:

A collection of concise write-ups on small things I learn day to day across a variety of languages and technologies. These are things that don't really warrant a full blog post.

This really resonated with me. I have five main places for writing at the moment.

This blog, for long-form content and weeknotes.
Twitter, for tweets - though Twitter threads are tending towards a long-form medium these days. My Spider-Verse behind-the-scenes thread ran for nearly a year!
My blogmarks - links plus short form commentary.
Niche Museums - effectively a blog about visits to tiny museums. It's on hiatus during the pandemic though.
GitHub issues. I've formed the habit of thinking out loud in issues, replying to myself with comments as I figure things out.

What's missing is exactly what TILs provide: somewhere to dump a couple of paragraphs about a new trick I've learned, with chronological order being less important than just getting them written down somewhere.

I've intermittently used gists for things like this in the past, but having them in an organized repo feels like a much less ad-hoc solution.

So I've started my own collection of TILs in my simonw/til GitHub repository.

Automating the README index page with GitHub Actions

The biggest feature I miss from reStructuredText when I'm working in Markdown is automatic tables of content.

For my TILs I wanted the index page on GitHub to display all of them. But I didn't want to have to update that page by hand every time I added one - especially since I'll often be creating them through the GitHub web interface which doesn't support editing multiple files in a single commit.

I've been getting a lot done with GitHub Actions recently. This felt like an opportunity to put them to more use.

So I wrote a GitHub Actions workflow that automatically updates the README page in the repo every time a new TIL markdown file is added or updated!

Here's an outline of how it works:

It runs on pushes to the master branch (no-one else can trigger it by sending me a pull request). It ignores commits that include the README.md file itself - otherwise commits to that file made by the workflow could trigger further runs of the same workflow. UPDATE: Apparently this isn't necessary.
It checks out the full repo history using actions/checkout@v2 with the fetch-depth: 0 option. This is needed because my script derives created/updated dates for each TIL by inspecting the git history. I learned a few days ago that this mechanism breaks if you only do a shallow check-out of the most recent commit!
It sets up Python, configures pip caching and installs dependencies from my requirements.txt.
It runs my build_database.py script, which uses GitPython to scan for all */*.md files and find their created and updated dates, then uses sqlite-utils to write the results to a SQLite database on the GitHub Actions temporary disk.
It runs update_readme.py which reads from that SQLite database and uses it to generate the markdown index section for the README. Then it opens the README and replaces the section between the  and  with the newly generated index.
It uses git diff to detect if the README has changed, then if it has it runs git commit and git push to commit those changes. See my TIL Commit a file if it changed for details on that pattern.

I really like this pattern.

I'm a big fan of keeping content in a git repository. Every CMS I've ever worked on has eventually evolved a desire to provide revision tracking, and building that into a regular database schema is never particularly pleasant. Git solves content versioning extremely effectively.

Having a GitHub repository that can update itself to maintain things like index pages feels like a technique that could be applied to all kinds of other content-related problems.

I'm also keen on the idea of using SQLite databases as intermediary storage as part of an Actions workflow. It's a simple but powerful way for one step in an action to generate structured data that can then be consumed by subsequent steps.

Implementing search with Datasette

Unsurprisingly, the other reason I'm using SQLite here is so I can deploy a database using Datasette. The last two steps of the workflow look like this:

- name: Setup Node.js
  uses: actions/setup-node@v1
  with:
    node-version: '12.x'
- name: Deploy Datasette using Zeit Now
  env:
    NOW_TOKEN: ${{ secrets.NOW_TOKEN }}
  run: |-
    datasette publish now2 til.db \
      --token $NOW_TOKEN \
      --project simon-til \
      --metadata metadata.json \
      --install py-gfm \
      --install datasette-render-markdown \
      --install datasette-template-sql \
      --template-dir templates

This installs Node.js, then uses Zeit Now (via datasette-publish-now) to publish the generated til.db SQLite database file to a Datasette instance accessible at til.simonwillison.net.

I'm reusing a bunch of tricks from my Niche Museums website here. The site is a standard Datasette instance with a custom index.html template that uses datasette-template-sql to display the TILs. Here's that template section in full:

{% for row in sql("select distinct topic from til order by topic") %}
    <h2>{{ row.topic }}</h2>
    <ul>
        {% for til in sql("select * from til where topic = '" + row.topic + "'") %}
            <li><a href="{{ til.url }}">{{ til.title }}</a> - {{ til.created[:10] }}</li>
        {% endfor %}
    </ul>
{% endfor %}

The search interface is powered by a custom SQL query in metadata.json that looks like this:

select
    til_fts.rank,
    til.*
from til
join til_fts on til.rowid = til_fts.rowid
where
    til_fts match case
        :q
        when '' then '*'
        else escape_fts(:q)
    end
order by
    til_fts.rank limit 20

A custom query-til-search.html template then renders the search results.

A powerful combination

I'm pretty happy with what I have here - it's definitely good enough to solve my TIL publishing needs. I'll probably add an Atom feed using datasette-atom at some point.

I hope this helps illustrate how powerful the combination of GitHub Actions, Datasette and Zeit Now or Cloud Run can be. I'm running an increasing number of projects on that combination, and the price, performance and ease of implementation continue to impress.

Tags: github, projects, markdown, datasette, github-actions, til