Simon Willison's Weblog: django

Quoting Tim Schilling

2026-03-17T16:13:37+00:00

If you do not understand the ticket, if you do not understand the solution, or if you do not understand the feedback on your PR, then your use of LLM is hurting Django as a whole. [...]

For a reviewer, it’s demoralizing to communicate with a facade of a human.

This is because contributing to open source, especially Django, is a communal endeavor. Removing your humanity from that experience makes that endeavor more difficult. If you use an LLM to contribute to Django, it needs to be as a complementary tool, not as your vehicle.

— Tim Schilling, Give Django your time and money, not your tokens

Tags: django, open-source, ai, generative-ai, llms, ai-ethics

Shopify/liquid: Performance: 53% faster parse+render, 61% fewer allocations

2026-03-13T03:44:34+00:00

Shopify/liquid: Performance: 53% faster parse+render, 61% fewer allocations

PR from Shopify CEO Tobias Lütke against Liquid, Shopify's open source Ruby template engine that was somewhat inspired by Django when Tobi first created it back in 2005.

Tobi found dozens of new performance micro-optimizations using a variant of autoresearch, Andrej Karpathy's new system for having a coding agent run hundreds of semi-autonomous experiments to find new effective techniques for training nanochat.

Tobi's implementation started two days ago with this autoresearch.md prompt file and an autoresearch.sh script for the agent to run to execute the test suite and report on benchmark scores.

The PR now lists 93 commits from around 120 automated experiments. The PR description lists what worked in detail - some examples:

Replaced StringScanner tokenizer with String#byteindex. Single-byte byteindex searching is ~40% faster than regex-based skip_until. This alone reduced parse time by ~12%.

Pure-byte parse_tag_token. Eliminated the costly StringScanner#string= reset that was called for every {% %} token (878 times). Manual byte scanning for tag name + markup extraction is faster than resetting and re-scanning via StringScanner. [...]

Cached small integer to_s. Pre-computed frozen strings for 0-999 avoid 267 Integer#to_s allocations per render.

This all added up to a 53% improvement on benchmarks - truly impressive for a codebase that's been tweaked by hundreds of contributors over 20 years.

I think this illustrates a number of interesting ideas:

Having a robust test suite - in this case 974 unit tests - is a massive unlock for working with coding agents. This kind of research effort would not be possible without first having a tried and tested suite of tests.
The autoresearch pattern - where an agent brainstorms a multitude of potential improvements and then experiments with them one at a time - is really effective.
If you provide an agent with a benchmarking script "make it faster" becomes an actionable goal.
CEOs can code again! Tobi has always been more hands-on than most, but this is a much more significant contribution than anyone would expect from the leader of a company with 7,500+ employees. I've seen this pattern play out a lot over the past few months: coding agents make it feasible for people in high-interruption roles to productively work with code again.

Here's Tobi's GitHub contribution graph for the past year, showing a significant uptick following that November 2025 inflection point when coding agents got really good.

He used Pi as the coding agent and released a new pi-autoresearch plugin in collaboration with David Cortés, which maintains state in an autoresearch.jsonl file like this one.

Via @tobi

Tags: django, performance, rails, ruby, ai, andrej-karpathy, generative-ai, llms, ai-assisted-programming, coding-agents, agentic-engineering, november-2025-inflection, tobias-lutke, autoresearch, pi

SWE-bench February 2026 leaderboard update

2026-02-19T04:48:47+00:00

SWE-bench February 2026 leaderboard update

SWE-bench is one of the benchmarks that the labs love to list in their model releases. The official leaderboard is infrequently updated but they just did a full run of it against the current generation of models, which is notable because it's always good to see benchmark results like this that weren't self-reported by the labs.

The fresh results are for their "Bash Only" benchmark, which runs their mini-swe-bench agent (~9,000 lines of Python, here are the prompts they use) against the SWE-bench dataset of coding problems - 2,294 real-world examples pulled from 12 open source repos: django/django (850), sympy/sympy (386), scikit-learn/scikit-learn (229), sphinx-doc/sphinx (187), matplotlib/matplotlib (184), pytest-dev/pytest (119), pydata/xarray (110), astropy/astropy (95), pylint-dev/pylint (57), psf/requests (44), mwaskom/seaborn (22), pallets/flask (11).

Correction: The Bash only benchmark runs against SWE-bench Verified, not original SWE-bench. Verified is a manually curated subset of 500 samples described here, funded by OpenAI. Here's SWE-bench Verified on Hugging Face - since it's just 2.1MB of Parquet it's easy to browse using Datasette Lite, which cuts those numbers down to django/django (231), sympy/sympy (75), sphinx-doc/sphinx (44), matplotlib/matplotlib (34), scikit-learn/scikit-learn (32), astropy/astropy (22), pydata/xarray (22), pytest-dev/pytest (19), pylint-dev/pylint (10), psf/requests (8), mwaskom/seaborn (2), pallets/flask (1).

Here's how the top ten models performed:

It's interesting to see Claude Opus 4.5 beat Opus 4.6, though only by about a percentage point. 4.5 Opus is top, then Gemini 3 Flash, then MiniMax M2.5 - a 229B model released last week by Chinese lab MiniMax. GLM-5, Kimi K2.5 and DeepSeek V3.2 are three more Chinese models that make the top ten as well.

OpenAI's GPT-5.2 is their highest performing model at position 6, but it's worth noting that their best coding model, GPT-5.3-Codex, is not represented - maybe because it's not yet available in the OpenAI API.

This benchmark uses the same system prompt for every model, which is important for a fair comparison but does mean that the quality of the different harnesses or optimized prompts is not being measured here.

The chart above is a screenshot from the SWE-bench website, but their charts don't include the actual percentage values visible on the bars. I successfully used Claude for Chrome to add these - transcript here. My prompt sequence included:

Use claude in chrome to open https://www.swebench.com/

Click on "Compare results" and then select "Select top 10"

See those bar charts? I want them to display the percentage on each bar so I can take a better screenshot, modify the page like that

I'm impressed at how well this worked - Claude injected custom JavaScript into the page to draw additional labels on top of the existing chart.

Update: If you look at the transcript Claude claims to have switched to Playwright, which is confusing because I didn't think I had that configured.

Via @KLieret

Tags: benchmarks, django, ai, openai, generative-ai, llms, anthropic, claude, coding-agents, ai-in-china, browser-agents, minimax

Adding dynamic features to an aggressively cached website

2026-01-28T22:10:08+00:00

My blog uses aggressive caching: it sits behind Cloudflare with a 15 minute cache header, which guarantees it can survive even the largest traffic spike to any given page. I've recently added a couple of dynamic features that work in spite of that full-page caching. Here's how those work.

Edit links that are visible only to me

This is a Django site and I manage it through the Django admin.

I have four types of content - entries, link posts (aka blogmarks), quotations and notes. Each of those has a different model and hence a different Django admin area.

I wanted an "edit" link on the public pages that was only visible to me.

The button looks like this:

I solved conditional display of this button with localStorage. I have a tiny bit of JavaScript which checks to see if the localStorage key ADMIN is set and, if it is, displays an edit link based on a data attribute:

document.addEventListener('DOMContentLoaded', () => {
  if (window.localStorage.getItem('ADMIN')) {
    document.querySelectorAll('.edit-page-link').forEach(el => {
      const url = el.getAttribute('data-admin-url');
      if (url) {
        const a = document.createElement('a');
        a.href = url;
        a.className = 'edit-link';
        a.innerHTML = '<svg>...</svg> Edit';
        el.appendChild(a);
        el.style.display = 'block';
      }
    });
  }
});

If you want to see my edit links you can run this snippet of JavaScript:

localStorage.setItem('ADMIN', '1');

My Django admin dashboard has a custom checkbox I can click to turn this option on and off in my own browser:

Those admin edit links are a very simple pattern. A more interesting one is a feature I added recently for navigating randomly within a tag.

Here's an animated GIF showing those random tag navigations in action (try it here):

On any of my blog's tag pages you can click the "Random" button to bounce to a random post with that tag. That random button then persists in the header of the page and you can click it to continue bouncing to random items in that same tag.

A post can have multiple tags, so there needs to be a little bit of persistent magic to remember which tag you are navigating and display the relevant button in the header.

Once again, this uses localStorage. Any click to a random button records both the tag and the current timestamp to the random_tag key in localStorage before redirecting the user to the /random/name-of-tag/ page, which selects a random post and redirects them there.

Any time a new page loads, JavaScript checks if that random_tag key has a value that was recorded within the past 5 seconds. If so, that random button is appended to the header.

This means that, provided the page loads within 5 seconds of the user clicking the button, the random tag navigation will persist on the page.

You can see the code for that here.

And the prompts

I built the random tag feature entirely using Claude Code for web, prompted from my iPhone. I started with the /random/TAG/ endpoint (full transcript):

Build /random/TAG/ - a page which picks a random post (could be an entry or blogmark or note or quote) that has that tag and sends a 302 redirect to it, marked as no-cache so Cloudflare does not cache it

Use a union to build a list of every content type (a string representing the table out of the four types) and primary key for every item tagged with that tag, then order by random and return the first one

Then inflate the type and ID into an object and load it and redirect to the URL

Include tests - it should work by setting up a tag with one of each of the content types and then running in a loop calling that endpoint until it has either returned one of each of the four types or it hits 1000 loops at which point fail with an error

Then:

I do not like that solution, some of my tags have thousands of items

Can we do something clever with a CTE?

Here's the something clever with a CTE solution we ended up with.

For the "Random post" button (transcript):

Look at most recent commit, then modify the /tags/xxx/ page to have a "Random post" button which looks good and links to the /random/xxx/ page

Then:

Put it before not after the feed icon. It should only display if a tag has more than 5 posts

And finally, the localStorage implementation that persists a random tag button in the header (transcript):

Review the last two commits. Make it so clicking the Random button on a tag page sets a localStorage value for random_tag with that tag and a timestamp. On any other page view that uses the base item template add JS that checks for that localStorage value and makes sure the timestamp is within 5 seconds. If it is within 5 seconds it adds a "Random name-of-tag" button to the little top navigation bar, styled like the original Random button, which bumps the localStorage timestamp and then sends the user to /random/name-of-tag/ when they click it. In this way clicking "Random" on a tag page will send the user into an experience where they can keep clicking to keep surfing randomly in that topic.

Tags: caching, django, javascript, localstorage, ai, cloudflare, generative-ai, llms, ai-assisted-programming

Django 6.0 released

2025-12-04T23:57:34+00:00

Django 6.0 released

Django 6.0 includes a flurry of neat features, but the two that most caught my eye are background workers and template partials.

Background workers started out as DEP (Django Enhancement Proposal) 14, proposed and shepherded by Jake Howard. Jake prototyped the feature in django-tasks and wrote this extensive background on the feature when it landed in core just in time for the 6.0 feature freeze back in September.

Kevin Wetzels published a useful first look at Django's background tasks based on the earlier RC, including notes on building a custom database-backed worker implementation.

Template Partials were implemented as a Google Summer of Code project by Farhan Ali Raza. I really like the design of this. Here's an example from the documentation showing the neat inline attribute which lets you both use and define a partial at the same time:

{# Define and render immediately. #}
{% partialdef user-info inline %}
    <div id="user-info-{{ user.username }}">
        <h3>{{ user.name }}</h3>
        <p>{{ user.bio }}</p>
    </div>
{% endpartialdef %}

{# Other page content here. #}

{# Reuse later elsewhere in the template. #}
<section class="featured-authors">
    <h2>Featured Authors</h2>
    {% for user in featured %}
        {% partial user-info %}
    {% endfor %}
</section>

You can also render just a named partial from a template directly in Python code like this:

return render(request, "authors.html#user-info", {"user": user})

I'm looking forward to trying this out in combination with HTMX.

I asked Claude Code to dig around in my blog's source code looking for places that could benefit from a template partial. Here's the resulting commit that uses them to de-duplicate the display of dates and tags from pages that list multiple types of content, such as my tag pages.

Tags: django, python, ai, generative-ai, llms, ai-assisted-programming, htmx, coding-agents, claude-code

YouTube embeds fail with a 153 error

2025-12-01T05:26:23+00:00

YouTube embeds fail with a 153 error

I just fixed this bug on my blog. I was getting an annoying "Error 153: Video player configuration error" on some of the YouTube video embeds (like this one) on this site. After some digging it turns out the culprit was this HTTP header, which Django's SecurityMiddleware was sending by default:

Referrer-Policy: same-origin

YouTube's embedded player terms documentation explains why this broke:

API Clients that use the YouTube embedded player (including the YouTube IFrame Player API) must provide identification through the HTTP Referer request header. In some environments, the browser will automatically set HTTP Referer, and API Clients need only ensure they are not setting the Referrer-Policy in a way that suppresses the Referer value. YouTube recommends using strict-origin-when-cross-origin Referrer-Policy, which is already the default in many browsers.

The fix, which I outsourced to GitHub Copilot agent since I was on my phone, was to add this to my settings.py:

SECURE_REFERRER_POLICY = "strict-origin-when-cross-origin"

This explainer on the Chrome blog describes what the header means:

strict-origin-when-cross-origin offers more privacy. With this policy, only the origin is sent in the Referer header of cross-origin requests.

This prevents leaks of private data that may be accessible from other parts of the full URL such as the path and query string.

Effectively it means that any time you follow a link from my site to somewhere else they'll see this in the incoming HTTP headers even if you followed the link from a page other than my homepage:

Referer: https://simonwillison.net/

The previous header, same-origin, is explained by MDN here:

Send the origin, path, and query string for same-origin requests. Don't send the Referer header for cross-origin requests.

This meant that previously traffic from my site wasn't sending any HTTP referer at all!

Tags: django, http, privacy, youtube

Highlights from my appearance on the Data Renegades podcast with CL Kao and Dori Wilson

2025-11-26T00:29:11+00:00

I talked with CL Kao and Dori Wilson for an episode of their new Data Renegades podcast titled Data Journalism Unleashed with Simon Willison.

I fed the transcript into Claude Opus 4.5 to extract this list of topics with timestamps and illustrative quotes. It did such a good job I'm using what it produced almost verbatim here - I tidied it up a tiny bit and added a bunch of supporting links.

What is data journalism and why it's the most interesting application of data analytics [02:03]

"There's this whole field of data journalism, which is using data and databases to try and figure out stories about the world. It's effectively data analytics, but applied to the world of news gathering. And I think it's fascinating. I think it is the single most interesting way to apply this stuff because everything is in scope for a journalist."
The origin story of Django at a small Kansas newspaper [02:31]

"We had a year's paid internship from university where we went to work for this local newspaper in Kansas with this chap Adrian Holovaty. And at the time we thought we were building a content management system."
Building the "Downloads Page" - a dynamic radio player of local bands [03:24]

"Adrian built a feature of the site called the Downloads Page. And what it did is it said, okay, who are the bands playing at venues this week? And then we'll construct a little radio player of MP3s of music of bands who are playing in Lawrence in this week."
Working at The Guardian on data-driven reporting projects [04:44]

"I just love that challenge of building tools that journalists can use to investigate stories and then that you can use to help tell those stories. Like if you give your audience a searchable database to back up the story that you're presenting, I just feel that's a great way of building more credibility in the reporting process."
Washington Post's opioid crisis data project and sharing with local newspapers [05:22]

"Something the Washington Post did that I thought was extremely forward thinking is that they shared [the opioid files] with other newspapers. They said, 'Okay, we're a big national newspaper, but these stories are at a local level. So what can we do so that the local newspaper and different towns can dive into that data for us?'"
NICAR conference and the collaborative, non-competitive nature of data journalism [07:00]

"It's all about trying to figure out what is the most value we can get out of this technology as an industry as a whole."

NICAR 2026
ProPublica and the Baltimore Banner as examples of nonprofit newsrooms [09:02]

"The Baltimore Banner are a nonprofit newsroom. They have a hundred employees now for the city of Baltimore. This is an enormously, it's a very healthy newsroom. They do amazing data reporting... And I believe they're almost breaking even on subscription revenue [correction, not yet], which is astonishing."
The "shower revelation" that led to Datasette - SQLite on serverless hosting [10:31]

"It was literally a shower revelation. I was in the shower thinking about serverless and I thought, 'hang on a second. So you can't use Postgres on serverless hosting, but if it's a read-only database, could you use SQLite? Could you just take that data, bake it into a blob of a SQLite file, ship that as part of the application just as another asset, and then serve things on top of that?'"
Datasette's plugin ecosystem and the vision of solving data publishing [12:36]

"In the past I've thought about it like how Pinterest solved scrapbooking and WordPress solved blogging, who's going to solve data like publishing tables full of data on the internet? So that was my original goal."
Unexpected Datasette use cases: Copenhagen electricity grid, Brooklyn Cemetery [13:59]

"Somebody was doing research on the Brooklyn Cemetery and they got hold of the original paper files of who was buried in the Brooklyn Cemetery. They digitized those, loaded the results into Datasette and now it tells the story of immigration to New York."
Bellingcat using Datasette to investigate leaked Russian food delivery data [14:40]

"It turns out the Russian FSB, their secret police, have an office that's not near any restaurants and they order food all the time. And so this database could tell you what nights were the FSB working late and what were the names and phone numbers of the FSB agents who ordered food... And I'm like, 'Wow, that's going to get me thrown out of a window.'"

Bellingcat: Food Delivery Leak Unmasks Russian Security Agents
The frustration of open source: no feedback on how people use your software [16:14]

"An endless frustration in open source is that you really don't get the feedback on what people are actually doing with it."
Open office hours on Fridays to learn how people use Datasette [16:49]

"I have an open office hours Calendly, where the invitation is, if you use my software or want to use my software, grab 25 minutes to talk to me about it. And that's been a revelation. I've had hundreds of conversations in the past few years with people."
Data cleaning as the universal complaint - 95% of time spent cleaning [17:34]

"I know every single person I talk to in data complains about the cleaning that everyone says, 'I spend 95% of my time cleaning the data and I hate it.'"
Version control problems in data teams - Python scripts on laptops without Git [17:43]

"I used to work for a large company that had a whole separate data division and I learned at one point that they weren't using Git for their scripts. They had Python scripts, littering laptops left, right and center and lots of notebooks and very little version control, which upset me greatly."
The Carpentries organization teaching scientists Git and software fundamentals [18:12]

"There's an organization called The Carpentries. Basically they teach scientists to use Git. Their entire thing is scientists are all writing code these days. Nobody ever sat them down and showed them how to use the UNIX terminal or Git or version control or write tests. We should do that."
Data documentation as an API contract problem [21:11]

"A coworker of mine said, you do realize that this should be a documented API interface, right? Your data warehouse view of your project is something that you should be responsible for communicating to the rest of the organization and we weren't doing it."
The importance of "view source" on business reports [23:21]

"If you show somebody a report, you need to have view source on those reports... somebody would say 25% of our users did this thing. And I'm thinking I need to see the query because I knew where all of the skeletons were buried and often that 25% was actually a 50%."
Fact-checking process for data reporting [24:16]

"Their stories are fact checked, no story goes out the door without someone else fact checking it and without an editor approving it. And it's the same for data. If they do a piece of data reporting, a separate data reporter has to audit those numbers and maybe even produce those numbers themselves in a separate way before they're confident enough to publish them."
Queries as first-class citizens with version history and comments [27:16]

"I think the queries themselves need to be first class citizens where like I want to see a library of queries that my team are using and each one I want to know who built it and when it was built. And I want to see how that's changed over time and be able to post comments on it."
Two types of documentation: official docs vs. temporal/timestamped notes [29:46]

"There's another type of documentation which I call temporal documentation where effectively it's stuff where you say, 'Okay, it's Friday, the 31st of October and this worked.' But the timestamp is very prominent and if somebody looks that in six months time, there's no promise that it's still going to be valid to them."
Starting an internal blog without permission - instant credibility [30:24]

"The key thing is you need to start one of these without having to ask permission first. You just one day start, you can do it in a Google Doc, right?... It gives you so much credibility really quickly because nobody else is doing it."
Building a search engine across seven documentation systems [31:35]

"It turns out, once you get a search engine over the top, it's good documentation. You just have to know where to look for it. And if you are the person who builds the search engine, you secretly control the company."
The TIL (Today I Learned) blog approach - celebrating learning basics [33:05]

"I've done TILs about 'for loops' in Bash, right? Because okay, everyone else knows how to do that. I didn't... It's a value statement where I'm saying that if you've been a professional software engineer for 25 years, you still don't know everything. You should still celebrate figuring out how to learn 'for loops' in Bash."
Coding agents like Claude Code and their unexpected general-purpose power [34:53]

"They pretend to be programming tools but actually they're basically a sort of general agent because they can do anything that you can do by typing commands into a Unix shell, which is everything."
Skills for Claude - markdown files for census data, visualization, newsroom standards [36:16]

"Imagine a markdown file for census data. Here's where to get census data from. Here's what all of the columns mean. Here's how to derive useful things from that. And then you have another skill for here's how to visualize things on a map using D3... At the Washington Post, our data standards are this and this and this."

Claude Skills are awesome, maybe a bigger deal than MCP
The absurd 2025 reality: cutting-edge AI tools use 1980s terminal interfaces [38:22]

"The terminal is now accessible to people who never learned the terminal before 'cause you don't have to remember all the commands because the LLM knows the commands for you. But isn't that fascinating that the cutting edge software right now is it's like 1980s style— I love that. It's not going to last. That's a current absurdity for 2025."
Cursor for data? Generic agent loops vs. data-specific IDEs [38:18]

"More of a notebook interface makes a lot more sense than a Claude Code style terminal 'cause a Jupyter Notebook is effectively a terminal, it's just in your browser and it can show you charts."
Future of BI tools: prompt-driven, instant dashboard creation [39:54]

"You can copy and paste a big chunk of JSON data from somewhere into [an LLM] and say build me a dashboard. And they do such a good job. Like they will just decide, oh this is a time element so we'll do a bar chart over time and these numbers feel big so we'll put those in a big green box."
Three exciting LLM applications: text-to-SQL, data extraction, data enrichment [43:06]

"LLMs are stunningly good at outputting SQL queries. Especially if you give them extra metadata about the columns. Maybe a couple of example queries and stuff."
LLMs extracting structured data from scanned PDFs at 95-98% accuracy [43:36]

"You file a freedom of information request and you get back horrifying scanned PDFs with slightly wonky angles and you have to get the data out of those. LLMs for a couple of years now have been so good at, 'here's a page of a police report, give me back JSON with the name of the arresting officer and the date of the incident and the description,' and they just do it."
Data enrichment: running cheap models in loops against thousands of records [44:36]

"There's something really exciting about the cheaper models, Gemini Flash 2.5 Lite, things like that. Being able to run those in a loop against thousands of records feels very valuable to me as well."

datasette-enrichments
Multimodal LLMs for images, audio transcription, and video processing [45:42]

"At one point I calculated that using Google's least expensive model, if I wanted to generate captions for like 70,000 photographs in my personal photo library, it would cost me like $13 or something. Wildly inexpensive."

Correction: with Gemini 1.5 Flash 8B it would cost 173.25 cents
First programming language: hated C++, loved PHP and Commodore 64 BASIC [46:54]

"I hated C++ 'cause I got my parents to buy me a book on it when I was like 15 and I did not make any progress with Borland C++ compiler... Actually, my first program language was Commodore 64 BASIC. And I did love that. Like I tried to build a database in Commodore 64 BASIC back when I was like six years old or something."
Biggest production bug: crashing The Guardian's MPs expenses site with a progress bar [47:46]

"I tweeted a screenshot of that progress bar and said, 'Hey, look, we have a progress bar.' And 30 seconds later the site crashed because I was using SQL queries to count all 17,000 documents just for this one progress bar."

Crowdsourced document analysis and MP expenses
Favorite test dataset: San Francisco's tree list, updated several times a week [48:44]

"There's 195,000 trees in this CSV file and it's got latitude and longitude and species and age when it was planted... and get this, it's updated several times a week... most working days, somebody at San Francisco City Hall updates their database of trees, and I can't figure out who."
Showrunning TV shows as a management model - transferring vision to lieutenants [50:07]

"Your job is to transfer your vision into their heads so they can go and have the meetings with the props department and the set design and all of those kinds of things... I used to sniff at the idea of a vision when I was young and stupid. And now I'm like, no, the vision really is everything because if everyone understands the vision, they can make decisions you delegate to them."

The Eleven Laws of Showrunning by Javier Grillo-Marxuach
Hot take: all executable code with business value must be in version control [52:21]

"I think it's inexcusable to have executable code that has business value that is not in version control somewhere."
Hacker News automation: GitHub Actions scraping for notifications [52:45]

"I've got a GitHub actions thing that runs a piece of software I wrote called shot-scraper that runs Playwright, that loads up a browser in GitHub actions to scrape that webpage and turn the results into JSON, which then get turned into an atom feed, which I subscribe to in NetNewsWire."
Dream project: whale detection camera with Gemini AI [53:47]

"I want to point a camera at the ocean and take a snapshot every minute and feed it into Google Gemini or something and just say, is there a whale yes or no? That would be incredible. I want push notifications when there's a whale."
Favorite podcast: Mark Steel's in Town (hyperlocal British comedy) [54:23]

"Every episode he goes to a small town in England and he does a comedy set in a local venue about the history of the town. And so he does very deep research... I love that sort of like hyperlocal, like comedy, that sort of British culture thing."

Mark Steel's in Town available episodes
Favorite fiction genre: British wizards caught up in bureaucracy [55:06]

"My favorite genre of fiction is British wizards who get caught up in bureaucracy... I just really like that contrast of like magical realism and very clearly researched government paperwork and filings."

The Laundry Files, Rivers of London, The Rook

Colophon

I used a Claude Project for the initial analysis, pasting in the HTML of the transcript since that included <span data-timestamp="425"> elements. The project uses the following custom instructions

You will be given a transcript of a podcast episode. Find the most interesting quotes in that transcript - quotes that best illustrate the overall themes, and quotes that introduce surprising ideas or express things in a particularly clear or engaging or spicy way. Answer just with those quotes - long quotes are fine.

I then added a follow-up prompt saying:

Now construct a bullet point list of key topics where each item includes the mm:ss in square braces at the end

Then suggest a very comprehensive list of supporting links I could find

Then one more follow-up:

Add an illustrative quote to every one of those key topics you identified

Here's the full Claude transcript of the analysis.

Tags: data, data-journalism, django, ai, datasette, podcast-appearances

How I automate my Substack newsletter with content from my blog

2025-11-19T22:00:34+00:00

I sent out my weekly-ish Substack newsletter this morning and took the opportunity to record a YouTube video demonstrating my process and describing the different components that make it work. There's a lot of digital duct tape involved, taking the content from Django+Heroku+PostgreSQL to GitHub Actions to SQLite+Datasette+Fly.io to JavaScript+Observable and finally to Substack.

The core process is the same as I described back in 2023. I have an Observable notebook called blog-to-newsletter which fetches content from my blog's database, filters out anything that has been in the newsletter before, formats what's left as HTML and offers a big "Copy rich text newsletter to clipboard" button.

I click that button, paste the result into the Substack editor, tweak a few things and hit send. The whole process usually takes just a few minutes.

I make very minor edits:

I set the title and the subheading for the newsletter. This is often a direct copy of the title of the featured blog post.
Substack turns YouTube URLs into embeds, which often isn't what I want - especially if I have a YouTube URL inside a code example.
Blocks of preformatted text often have an extra blank line at the end, which I remove.
Occasionally I'll make a content edit - removing a piece of content that doesn't fit the newsletter, or fixing a time reference like "yesterday" that doesn't make sense any more.
I pick the featured image for the newsletter and add some tags.

That's the whole process!

The Observable notebook

The most important cell in the Observable notebook is this one:

raw_content = {
  return await (
    await fetch(
      `https://datasette.simonwillison.net/simonwillisonblog.json?sql=${encodeURIComponent(
        sql
      )}&_shape=array&numdays=${numDays}`
    )
  ).json();
}

This uses the JavaScript fetch() function to pull data from my blog's Datasette instance, using a very complex SQL query that is composed elsewhere in the notebook.

Here's a link to see and execute that query directly in Datasette. It's 143 lines of convoluted SQL that assembles most of the HTML for the newsletter using SQLite string concatenation! An illustrative snippet:

with content as (
  select
    id,
    'entry' as type,
    title,
    created,
    slug,
    '<h3><a href="' || 'https://simonwillison.net/' || strftime('%Y/', created)
      || substr('JanFebMarAprMayJunJulAugSepOctNovDec', (strftime('%m', created) - 1) * 3 + 1, 3) 
      || '/' || cast(strftime('%d', created) as integer) || '/' || slug || '/' || '">' 
      || title || '</a> - ' || date(created) || '</h3>' || body
      as html,
    'null' as json,
    '' as external_url
  from blog_entry
  union all
  # ...

My blog's URLs look like /2025/Nov/18/gemini-3/ - this SQL constructs that three letter month abbreviation from the month number using a substring operation.

This is a terrible way to assemble HTML, but I've stuck with it because it amuses me.

The rest of the Observable notebook takes that data, filters out anything that links to content mentioned in the previous newsletters and composes it into a block of HTML that can be copied using that big button.

Here's the recipe it uses to turn HTML into rich text content on a clipboard suitable for Substack. I can't remember how I figured this out but it's very effective:

Object.assign(
  html`<button style="font-size: 1.4em; padding: 0.3em 1em; font-weight: bold;">Copy rich text newsletter to clipboard`,
  {
    onclick: () => {
      const htmlContent = newsletterHTML;
      // Create a temporary element to hold the HTML content
      const tempElement = document.createElement("div");
      tempElement.innerHTML = htmlContent;
      document.body.appendChild(tempElement);
      // Select the HTML content
      const range = document.createRange();
      range.selectNode(tempElement);
      // Copy the selected HTML content to the clipboard
      const selection = window.getSelection();
      selection.removeAllRanges();
      selection.addRange(range);
      document.execCommand("copy");
      selection.removeAllRanges();
      document.body.removeChild(tempElement);
    }
  }
)

From Django+Postgresql to Datasette+SQLite

My blog itself is a Django application hosted on Heroku, with data stored in Heroku PostgreSQL. Here's the source code for that Django application. I use the Django admin as my CMS.

Datasette provides a JSON API over a SQLite database... which means something needs to convert that PostgreSQL database into a SQLite database that Datasette can use.

My system for doing that lives in the simonw/simonwillisonblog-backup GitHub repository. It uses GitHub Actions on a schedule that executes every two hours, fetching the latest data from PostgreSQL and converting that to SQLite.

My db-to-sqlite tool is responsible for that conversion. I call it like this:

db-to-sqlite \
  $(heroku config:get DATABASE_URL -a simonwillisonblog | sed s/postgres:/postgresql+psycopg2:/) \
  simonwillisonblog.db \
  --table auth_permission \
  --table auth_user \
  --table blog_blogmark \
  --table blog_blogmark_tags \
  --table blog_entry \
  --table blog_entry_tags \
  --table blog_quotation \
  --table blog_quotation_tags \
  --table blog_note \
  --table blog_note_tags \
  --table blog_tag \
  --table blog_previoustagname \
  --table blog_series \
  --table django_content_type \
  --table redirects_redirect

That heroku config:get DATABASE_URL command uses Heroku credentials in an environment variable to fetch the database connection URL for my blog's PostgreSQL database (and fixes a small difference in the URL scheme).

db-to-sqlite can then export that data and write it to a SQLite database file called simonwillisonblog.db.

The --table options specify the tables that should be included in the export.

The repository does more than just that conversion: it also exports the resulting data to JSON files that live in the repository, which gives me a commit history of changes I make to my content. This is a cheap way to get a revision history of my blog content without having to mess around with detailed history tracking inside the Django application itself.

At the end of my GitHub Actions workflow is this code that publishes the resulting database to Datasette running on Fly.io using the datasette publish fly plugin:

datasette publish fly simonwillisonblog.db \
  -m metadata.yml \
  --app simonwillisonblog-backup \
  --branch 1.0a2 \
  --extra-options "--setting sql_time_limit_ms 15000 --setting truncate_cells_html 10000 --setting allow_facet off" \
  --install datasette-block-robots \
  # ... more plugins

As you can see, there are a lot of moving parts! Surprisingly it all mostly just works - I rarely have to intervene in the process, and the cost of those different components is pleasantly low.

Tags: blogging, django, javascript, postgresql, sql, sqlite, youtube, heroku, datasette, observable, github-actions, fly, newsletter, substack, site-upgrades

Talk Python: Celebrating Django's 20th Birthday With Its Creators

2025-08-29T20:02:50+00:00

Talk Python: Celebrating Django's 20th Birthday With Its Creators

I recorded this podcast episode recently to celebrate Django's 20th birthday with Adrian Holovaty, Will Vincent, Jeff Triplet, and Thibaud Colas.

We didn’t know that it was a web framework. We thought it was a tool for building local newspaper websites. [...]

Django’s original tagline was ‘Web development on journalism deadlines’. That’s always been my favorite description of the project.

Tags: adrian-holovaty, django, python, podcast-appearances

Happy 20th birthday Django! Here's my talk on Django Origins from Django's 10th

2025-07-13T18:47:13+00:00

Today is the 20th anniversary of the first commit to the public Django repository!

Ten years ago we threw a multi-day 10th birthday party for Django back in its birthtown of Lawrence, Kansas. As a personal celebration of the 20th, I'm revisiting the talk I gave at that event and writing it up here.

Here's the YouTube video. Below is a full transcript, plus my slides and some present-day annotations.

Django Origins (and some things I have built with Django)

Presented 11th July 2015 at Django Birthday in Lawrence, Kansas

My original talk title, as you'll see on your programs, was "Some Things I've Built with Django." But then I realized that we're here in the birthplace of Django, celebrating the 10th birthday of the framework, and nobody's told the origin story yet. So, I've switched things around a little bit. I'm going to talk about the origin story of Django, and then if I have time, I'll do the self-indulgent bit and talk about some of the projects I've shipped since then.

I think Jacob's introduction hit on something I've never really realized about myself. I do love shipping things. The follow-up and the long-term thing I'm not quite so strong on. And that came to focus when I was putting together this talk and realized that basically every project I'm going to show you, I had to dig out of the Internet Archive.

Ten years on from writing this talk I'm proud that I've managed to overcome my weakness in following-up - I'm now actively maintaining a bewildering array of projects, having finally figured out how to maintain things in addition to creating them!

But that said, I will tell you the origin story of Django.

For me, the story starts very much like Jacob's. I was reading RSS feeds back in 2003, and I saw this entry on Adrian's blog, talking about a job opportunity for a web programmer or developer in Lawrence, Kansas.

Now, I was in England. I was at university. But my university had just given me the opportunity to take a year abroad, to take a year out to do an internship year in industry. My girlfriend at the time was off to Germany to do her year in industry. So I was like, well, you know, do I stay at university? And then this comes along.

So I got in touch with Adrian and said, you know, could this work as a year-long internship instead? And he was reading my blog and I was reading his blog, and we knew that we aligned on a bunch of things. So we thought we'd give it a go.

Now, if you look through this job ad, you'll see that this is all about expert knowledge of PHP and experience designing and maintaining databases, particularly MySQL. So this was a PHP and MySQL gig.

But when I arrived in Kansas, we quickly realized that we were both kind of over PHP. You know, we'd both built substantial systems in PHP, and we were running up against the limits of what you can do in PHP and have your code still be clean and maintainable.

And at the same time, we were both reading Mark Pilgrim's blog (archive link). Mark Pilgrim had been publishing Dive into Python and making a really strong case for why Python was a great web language.

So we decided that this was the thing we wanted to do. But we both had very strong opinions about how you should build websites. Things like URL design matters, and we care about the difference between get and post, and we want to use this new thing called CSS to lay out our web pages. And none of the existing Python web frameworks really seemed to let us do what we wanted to do.

Now, before I talk more about that, I'll back up and talk about the organization we're working for, the Lawrence Journal World.

David gave a great introduction to why this is an interesting organization. Now, we're talking about a newspaper with a circulation of about 10,000, like a tiny newspaper, but with a world-class newsroom, huge amounts of money being funneled into it, and like employing full-time software developers to work at a local newspaper in Kansas.

And part of what was going on here was this guy. This is Rob Curley. He's been mentioned a few times before already.

And yeah, Rob Curley set this unofficial mission statement that we "build cool shit". This is something that Adrian would certainly never say. It's not really something I'd say. But this is Rob through and through. He was a fantastic showman.

And this was really the appeal of coming out to Lawrence, seeing the stuff they'd already built and the ambitions they had.

This is Lawrence.com. This is actually the Lawrence.com written in PHP that Adrian had built as the sole programmer at the Lawrence Journal World. And you should check this out. Like, even today, this is the best local entertainment website I have ever seen. This has everything that is happening in the town of Lawrence, Kansas population, 150,000 people. Every gig, every venue, all of the stuff that's going on.

And it was all written in PHP. And it was a very clean PHP code base, but it was really stretching the limits of what it's sensible to do using PHP 4 back in 2003.

So we had this goal when we started using Python. We wanted to eventually rebuild Lawrence.com using Python. But in order to get there, we had to first build -- we didn't even know it was a web framework. We called it the CMS.

And so when we started working on Django, the first thing that we shipped was actually this website. We had a lot of the six-news Lawrence. This is the six-news Lawrence -- six-news is the TV channel here -- six-news Lawrence weather page.

And I think this is pretty cool. So Dan Cox, the designer, was a fantastic illustrator. We actually have this illustration of the famous Lawrence skyline with each panel could be displayed with different weather conditions depending on the weather.

And in case you're not from Kansas, you might not have realized that the weather is a big deal here. You know, you have never seen more excited weathermen than when there's a tornado warning and they get to go on local news 24 hours a day giving people updates.

So we put the site live first. This was the first ever Django website. We then did the rest of the 6 News Lawrence website.

And this -- Adrian reminded me this morning -- the launch of this was actually delayed by a week because the most important feature on the website, which is clearly the photograph of the news people who are on TV, they didn't like their hairdos. They literally told us we couldn't launch the website until they'd had their hair redone, had the headshots retaken, had a new image put together. But, you know, image is important for these things.

So anyway, we did that. We did six-news Lawrence. And by the end of my year in Kansas, Adrian had rewritten all of Lawrence.com as well.

So this is the Lawrence.com powered by Django. And one thing I think is interesting about this is when you talk to like David Heinemeier Hansson about Rails, he'll tell you that Rails is a framework that was extracted from Basecamp. They built Basecamp and then they pulled out the framework that they used and open sourced it.

I see Django the other way around. Django is a framework that was built up to create Lawrence.com. Lawrence.com already existed. So we knew what the web framework needed to be able to do. And we just kept on iterating on Django or the CMS until it was ready to produce this site here.

And for me, the moment I realized that we were onto something special was actually when we built this thing. This is a classic Rob Curley project. So Rob was the boss. He had the crazy ideas and he didn't care how you implemented them. He just wanted this thing done.

And he came to us one day and said, you know, the kids' little league season is about to start. Like kids playing softball or baseball. Whatever the American kids with bats thing is. So he said, kids' little league season is about to start. And we are going to go all out.

I want to treat these kids like they're the New York Yankees. We're going to have player profiles and schedules and photos and results.

And, you know, we're going to have the ability for parents to get SMS notifications whenever their kid scores.

And we're going to have 360 degree, like, interactive photos of all of the pitches in Lawrence, Kansas, that these kids are playing games on.

They actually did send a couple of interns out with a rig to take 360 degree virtual panoramas of Fenway Park and Lawrence High School and all of these different places.

And he said -- and it starts in three days. You've got three days to put this all together.

And we pulled it off because Django, even at that very early stage, had all of the primitives you needed to build 360 degree interactives. That was all down to the interns. But we had all of the pieces we needed to pull this together.

So when we were working on it back then, we called it the CMS.

A few years ago, Jacob found a wiki page with some of the names that were being brainstormed for the open source release. And some of these are great. There's Brazos -- I don't know where that came from -- Webbing, Physique, Anson.

This is my favorite name. I think this is what I proposed -- is the Tornado Publishing System.

And the reason is that I was a really big fan of Office Space. And if we had the Tornado, we could produce TPS reports, which I thought would be amazing.

But unfortunately, this being Kansas, the association of Tornadoes isn't actually a positive one.

Private Dancer, Physgig, Lavalia, Pithy -- yeah. I'm very, very pleased that they picked the name that they did.

So one of our philosophies was build cool shit. The other philosophy we had was what we called "Wouldn't it be cool if?"

So there were no user stories or careful specs or anything. We'd all sit around in the basement and then somebody would go "Wouldn't it be cool if...", and they'd say something. And if we thought it was a cool idea, we'd build it and we'd ship it that day.

And my favorite example of "Wouldn't it be cool if?" -- this is a classic Adrian one -- is "Wouldn't it be cool if the downloads page on Lawrence.com featured MP3s you could download of local bands?" And seeing as we've also got the schedule of when the bands are playing, why don't we feature the audio from bands who you can go and see that week?

So this page will say, "OK Jones are playing on Thursday at the Bottleneck. Get their MP3. Listen to the radio station." We had a little MP3 widget in there. Go and look at their band profile. All of this stuff.

Really, these kinds of features are what you get when you take 1970s relational database technology and use it to power websites, which -- back in 2003, in the news industry -- still felt incredibly cutting edge. But, you know, it worked.

And that philosophy followed me through the rest of my career, which is sometimes a good idea and often means that you're left maintaining features that seemed like a good idea at the time and quickly become a massive pain!

After I finished my internship, I finished my degree in England and then ended up joining up with Yahoo. I was actually working out of the Yahoo UK office but for a R&D team in the States. I was there for about a year and a half.

One of the things I learned is that you should never go and work for an R&D team, because the problem with R&D teams is you never ship. I was there for a year and a half and I basically have nothing to show for it in terms of actual shipped features.

We built some very cool prototypes. And actually, after I left, one of the projects I worked on, Yahoo FireEagle, did end up getting spun out and turned into a real product.

But there is one project -- the first project I built at Yahoo using Django that I wanted to demonstrate. This was for Yahoo's internal hack day. And so Tom Coates and myself, who were working together, we decided that we were going to build a mashup, because it was 2005 and mashups were the cutting edge of computer science.

So we figured, OK, let's take the two most unlikely Yahoo products and combine them together and see what happens. My original suggestion was that we take Yahoo Dating and Yahoo Pets. But I was told that actually there was this thing called Dogster and this other thing called Catster, which already existed and did exactly that.

So the next best thing, we went for Yahoo News and Yahoo Horoscopes. And what we ended up building -- and again, this is the first Django application within Yahoo -- was Yahoo Astronewsology.

And the idea was you take the news feed from Yahoo News, you pull out anything that looks like it's a celebrity's name, look up their birth date, use that to look up their horoscope, and then combine them on the page. And in a massive stroke of luck, we built this the week that Dick Cheney shot his friend in the face while out hunting.

Dick Cheney's horoscope for that week says, "A very close friend who means a great deal to you has found it necessary to go out of their way to tick you off. You're not just angry, you're furious. Before you let go and let them have it, be sure you're right. Feeling righteous is far better than feeling guilty."

And so if Dick Cheney had only had only been reading his horoscopes, maybe that whole situation would have ended very differently.

So after Yahoo, I spent a while doing consulting and things, mainly around OpenID because I was determined to make OpenID work. I was absolutely convinced that if OpenID didn't take off, just one company would end up owning single sign-on for the entire internet, and that would be a total disaster.

And with hindsight, it didn't quite happen. Facebook login looked like it was going to do that a few years ago, but these days there's enough variety out there that I don't feel like we all have to submit to our Facebook masters.

But, you know, I was enjoying freelancing and consulting and so on. And then I ended up going for coffee with somebody who worked for The Guardian.

I'm sure you've all heard of The Guardian. It's one of England's most internationally focused newspapers. It's a very fine publication. And I realized that I really missed working in a newsroom environment. And I was incredibly jealous of people like Adrian, who'd gone off to the Washington Post and was doing data journalism there, and Derek Willis as well, who bounced from the Post and The New York Times. There was all of this cool data journalism stuff going on.

And The Guardian's pitch was basically, we've been building a CMS from scratch in Java with a giant team of engineers, and we've built it and it's really cool, but we're not shipping things quickly. We want to start exploring this idea of building things much faster to fit in with the news cycle.

And that was a very, very tempting thing for me to get involved with. So I went to work for The Guardian.

And The Guardian have a really interesting way of doing onboarding of new staff. The way they do it is they set you up on coffee dates with people from all over the organization. So one day you'll be having coffee with somebody who sells ads, and the next day it'll be the deputy editor of the newsroom, and the next day it'll be a journalist somewhere. And each of these people will talk to you and then they'll suggest other people for you to meet up with. So over the first few weeks that you're there, you meet a huge variety of people.

And time and time again, as I was talking to people, they were saying, "You know what? You should go and talk to Simon Rogers, this journalist in the newsroom."

This is Simon Rogers. I went down to talk to him, and we had this fascinating conversation. So Simon is a journalist. He worked in the newsroom, and his speciality was gathering data for The Guardian's infographics. Because they are in the paper. They post, they have graphs and charts and all sorts of things like that that they publish.

It turns out that Simon was the journalist who knew how to get that data out of basically any source you can imagine. If you wanted data, he would make some phone calls, dig into some government contacts and things, and he'd get those raw numbers. And all of the other journalists thought he was a bit weird, because he liked hanging out and editing Excel spreadsheets and stuff.

So I said to him halfway through this conversation, "Just out of interest, what do you do with those Excel spreadsheets?" And he's like, "Oh, I keep them all on my hard drive." And showed me this folder with hundreds and hundreds of meticulously researched, properly citable news quality spreadsheets full of data about everything you could imagine. And they lived on his hard drive and nowhere else.

And I was like, "Have you ever talked to anyone in the engineering department upstairs?" And we made this connection.

And so from then on, we had this collaboration going where he would get data and he'd funnel it to me and see if we could, see if I or someone else in the engineering department at Guardian could do something fun with it.

And so that was some of the most rewarding work of my career, because it's journalism, you know, it's news, it's stuff that matters. The deadlines are ridiculous. If a news story breaks and it takes you three weeks to turn around a piece of data journalism around it, why did you even bother? And it's perfect for applying Django to.

So the first story I got to work on at the Guardian was actually one of the early WikiLeaks things. This is before WikiLeaks was like massively high profile. But quite early on, WikiLeaks leaked a list of all of the members of the British National Party, basically the British Nazis. They leaked a list of all of their names and addresses.

And the Guardian is an ethical newspaper, so we're not going to just publish 18,000 people's names and addresses. But we wanted to figure out if there was something we could do that would make use of that data but wouldn't be violating anyone's individual privacy.

And so what we did is we took all of the addresses, geocoded them, figured out which parliamentary constituency they lived in, and used that to generate a heat map that's actually called a choropleth map, I think, of the UK showing where the hotspots of BNP activity were.

And this works because in the UK a parliamentary constituency is, they're designed to all have around about the same population. So if you just like make the color denser for the larger numbers of BNP members, you get this really interesting heat map of the country.

And what was really cool about this is that I did this using SVG, because we have an infographics department with Illustrator who are good at working with SVG. And it's very easy with an SVG file with the right class names on things to set colors on different regions.

And because we produced it in SVG, we could then hand it over to the print department, and the next day it was out in the paper. It was like a printed thing on paper, on like dead trees distributed all over the country, which I thought was super cool.

So that was the first data journalism project that we did at The Guardian. And it really helped prove that given the right data sets and like the right tools and a bit of freedom, you can do some really cool things.

The first few times I did this, I did it by hand. Then we had The Guardian's first hack day and I was like, well okay, I'm going to build a little self-service tool for our infographics journalists to like dump in a bunch of CSV numbers and get one of these maps out of it.

So I built this tool. I didn't have anywhere official to deploy it, so I just ran it on my Linux desktop underneath my desk. And they started using it and putting things in the paper and I kind of forgot about it. And every now and then I get a little feature request.

A few years after I left The Guardian, I ran into someone who worked there. And he was like, yeah, you know that thing that you built? So we had to keep your desktop running for six months after you left. And then we had to like convert it into a VMware instance. And as far as I know, my desktop is still running as a VMware instance somewhere in The Guardian.

Which ties into the Simon database, I guess. The hard thing is building stuff is easy. Keeping it going it turns out is surprisingly difficult.

This was my favorite project at The Guardian. There was a scandal in the UK a few years ago where it turned out that UK members of parliament had all been fiddling their expenses.

And actually the background on this is that they're the lowest paid MPs anywhere in Europe. And it seems like the culture had become that you become an MP and on your first day somebody takes you aside and goes, look, I know the salary is terrible. But here's how to fill your expenses and make up for it.

This was a scandal that was brewing for several years. The Guardian had actually filed freedom of information requests to try and get these expense reports. Because they were pretty sure something dodgy was going on. The government had dragged their heels in releasing the documents.

And then just when they were a month before they finally released the documents, a rival newspaper, the Telegraph, managed to get hold of a leaked copy of all of these expenses. And so the Telegraph had 30 days lead on all of the other newspapers to dig through and try and find the dirt.

So when they did release the expenses 30 days later, we had a race on our hands because we needed to analyze 20,000 odd pages of documents. Actually, here it says 450,000 pages of documents in order to try and find anything left that was newsworthy.

And so we tackled this with crowdsourcing. We stuck up a website. We told people, we told Guardian readers, come to this website, hit the button, we'll show you a random page from someone's expenses. And then you can tell us if you think it's not interesting, interesting, or we should seek an investigative reporter on it.

And one of the smartest things we did with this is we added a feature where you could put in your postcode, we'd figure out who your MP was, and then we would show you their smug press photo. You know, their smug face next to all of their expense claims that they'd filed.

And this was incredibly effective. People were like, "Ooh, you look so smug. I'm going to get you." And once we put this up, and within 18 hours, our community had burned through hundreds of thousands of pages of expense documents trying to find this stuff.

And again, this was built in Django. We had, I think, five days warning that these documents are coming out. And so it was a total, like, I think I built a proof of concept on day one. That was enough to show that it was possible. So I got a team with a designer and a couple of other people to help out. And we had it ready to go when the document dump came out on that Friday.

And it was pretty successful. We dug up some pretty interesting stories from it. And it was also just a fantastic interactive way of engaging our community. And, you know, the whole crowdsourcing side of it was super fun.

So I guess the thing I've learned from that is that, oh, my goodness, it's fun working for newspapers. And actually, if you -- the Lawrence Journal world, sadly, no longer has its own technology team. But there was a period a few years ago where they were doing some cracking data journalism work. Things like tracking what the University of Kansas had been using its private jet for, and letting people explore the data around that and so on.

The other thing we did at the Guardian, this is going back to Simon Rogers, is he had all of these spreadsheets on his hard drive. And we're like, okay, we should really try and publish this stuff as raw data. Because living on your hard drive under your head is a crying shame.

And the idea we came up with was essentially to start something we called the Data blog and publish them as Google spreadsheets. You know, we spent a while thinking, well, you know, what's the best format to publish these things in? And we're like, well, they're in Excel. Google spreadsheets exists and it's pretty good. Let's just put a few of them up as Google sheets and see what people do with them.

And it turns out that was enough to build this really fun community of data nerds around the Guardian's data blog who would build their own visualizations. They'd dig into the data. And it meant that we could get all sorts of -- like, we could get so much extra value from the work that we were already doing to gather these numbers for the newspaper. That stuff was super fun.

Now, while I was working at the Guardian, I also got into the habit of building some projects with my girlfriend at the time, now my wife Natalie. So Natalie and I have skill sets that fit together very nicely. She's a front-end web developer. I do back-end Django stuff. I do just enough ops to be dangerous. And so between the two of us, we can build websites.

The first things we worked on together is a site which I think some people here should be familiar with, called Django People. The idea was just, you know, the Django community appears to be quite big now. Let's try and get people to stick a pin on a map and tell us where they are.

Django People still exists today. It's online thanks to a large number of people constantly bugging me at Django Cons and saying, look, just give us the code and the data and we'll get it set up somewhere so it can continue to work. And that's great. I'm really glad I did that because this is the one project that I'm showing you today which is still available on the web somewhere. (2025 update: the site is no longer online.)

But Django People was really fun. And the thing we learned from this, my wife and I, is that we can work together really well on things.

The other side project we did was much more of a collaborative effort. Again, this no longer exists, or at least it's no longer up on the web. And I'm deeply sad about this because it's my favorite thing I'm going to show you.

But before I show you the project, I'll show you how we built it. We were at a BarCamp in London with a bunch of our new friends and somebody was showing photographs of this Napoleonic sea fortress that they had rented out for the weekend from an organization in the UK called the Landmark Trust, who basically take historic buildings and turn them into vacation rentals as part of the work to restore them.

And we were like, "Oh, wouldn't it be funny if we rented a castle for a week and all of us went out there and we built stuff together?" And then we were like, "That wouldn't be funny. That would be freaking amazing."

So we rented this place. This is called Fort Clonque. It's in the Channel Islands, halfway between England and France. And I think it cost something like $2,000 for the week, but you split that between a dozen people and it's like youth hostel prices to stay in a freaking fortress.

So we got a bunch of people together and we went out there and we just spent a week. We called it /dev/fort. We spent a week just building something together.

And the thing we ended up building was called Wildlife Near You. And what Wildlife Near You does is it solves the eternal question, "Where is my nearest llama?"

Once again, this is a crowdsourcing system. The idea is that you go to wildlifenearyou.com and you've just been on a trip to like a nature park or a zoo or something. And so you create a trip report saying, "I went to the Red Kite feeding station and I saw a common raven and a common buzzard and a red kite." And you import any of your photos from Flickr and so forth.

And you build up this profile saying, "Here are all the places I've been and my favorite animals and things I've seen."

And then once we've got that data set, we can solve the problem. You can say, "Search for llamas near Brighton." And it'll say, "Your nearest llama is 18 miles away and it'll show you pictures of llamas and all of the llama things."

And we have species pages. So here's the red panda page. 17 people love red pandas. You can see them at Taronga Zoo.

And then our most viral feature was we had all of these photos of red pandas, but how do we know which is the best photo of a red panda that we should highlight on the red panda page? So we basically built Hot or Not for photographs of wildlife.

So it's like, "Which marmot photo is better?" And you say, "Well, clearly the one on the right." And it's like, "Okay, which skunk photo is better?"

I was looking at the logs and people would go through hundreds and hundreds of photos. And you'd get scores and you can see, "Oh, wow, my marmot photo is the second best marmot photo on the whole website."

So that was really fun. And then we eventually took it a step further and said, "Okay, well, this is really fun, but this is a website that you have to type on, right?" And meanwhile, mobile phones are now getting HTML5 geolocation and stuff. So can we go a step further?

So we built owlsnearyou.com. And what owlsnearyou.com does is you type in the location, and it says, "Your nearest owl is 49 miles away." It's a spectacle owl at London Zoo. It was spotted one year ago by Natalie.

And if you went here on a mobile phone-- If you went here on a device that supported geolocation, it doesn't even ask you where you live. It's just like, "Oh, okay, here's your nearest owl."

And I think we shipped lions near you and monkeys near you and a couple of other domains, but owlsnearyou.com was always my favorite.

So looking at this now, we should really get this stuff up and running again. It was freaking amazing. Like, this for me is the killer app of all killer apps.

(We did eventually bring this idea back as www.owlsnearme.com, using data from iNaturalist - that's online today.)

So there have actually been a bunch of Devforts since then. One of the things we learned from Devfort is that building applications-- If you want to do a side project, doing one with user accounts and logins and so on, it's a freaking nightmare. It actually took us almost a year after we finished on the fort to finally ship Wildlife Near You because there were so many complexities. And then we had to moderate it and keep an eye on it and so on.

So if you look at the more recent Devforts, they've taken that to heart. And now they try and ship things which just work and don't require ongoing users logging in and all of that kind of rubbish.

But one of the other projects I wanted to show you that came out of a Devfort was something called Bugle. And the idea of Bugle is Bugle is a Twitter-like application for groups of hackers collaborating in a castle, fort, or other defensive structure who don't have an internet connection.

This was basically to deal with Twitter withdrawal when we were all on the fort together and we had an internal network. So Bugle, looking at it now, we could have been Slack! We could have been valued at $2 billion.

Yeah, Bugle is like an internal Twitter clone with a bunch of extra features like it's got a paste bin and to-do lists and all sorts of stuff like that.

And does anyone here know Ben Firshman? I think quite a few people do. Excellent. So Ben Firshman was out on a Devfort and I did a "Wouldn't it be cool if" on him. I said, "Wouldn't it be cool if all of our Twitter apps and our phones talked to Bugle instead on the network?"

And so if you go and look on GitHub, I bet this doesn't work anymore. But he did add magic Twitter support where you could run a local DNS server, redirect Twitter to Bugle and we cloned, he cloned enough of the Twitter API that like Twitter apps would work and it would be able to Bugle instead.

We wanted to do a Devfort in America. You don't really have castles and forts that you can rent for the most part. If anyone knows of one, please come and talk to me because there's a distinct lack of defensible structures at least of the kind that we are used to back in Europe.

So I'm running out of time, but that's OK because the most recent project, Lanyrd, is something which most people here have probably encountered.

I will tell a little bit of the backstory of Lanyrd because it's kind of fun.

Lanyrd was a honeymoon project.

Natalie and I got married. The wildlife near you influence affected our wedding - it was a freaking awesome wedding! You know, in England, you can get a man with a golden eagle and a barn owl and various other birds to show up for about $400 for the day. And then you get to take photos like this.

So anyway, we got married, we quit our jobs, I had to leave the Guardian because we wanted to spend the next year or two of our lives just traveling around the world, doing freelancing work on our laptops and so on.

We got as far as Morocco, we were six months in, when we contracted food poisoning in Casablanca and we were too sick to keep on travelling, so we figured we needed to like, you know, and it was also Ramadan, so it was really hard to get food and stuff. So we rented an apartment for two weeks and said, "Okay, well, since we're stuck for two weeks, let's like finish that side project we've been talking about and ship it and see if anyone's interested."

So we shipped Lanyrd, which was built around the idea of helping people who use Twitter find conferences and events to go to. What we hadn't realised is that if you build something around Twitter, especially back in 2010, it instantly goes viral amongst people who use Twitter.

So that ended up cutting our honeymoon short, and we actually applied for Y Combinator from Egypt and ended up spending three years building a startup and like hiring people and doing that whole thing.

(Natalie wrote more about our startup in Lanyrd: from idea to exit - the story of our startup.)

The only thing I'll say about that is everything in the... Startups have to give the impression that everything's super easy and fun and cool all the time, because people say, "How's your startup going?" And the only correct answer is, "Oh man, it's amazing. It's doing so well." Because everyone has to lie about the misery, pain, anguish and stress that's happening behind the scenes.

So it was a very interesting three years, and we built some cool stuff and we learnt a lot, and I don't regret it, but do not take startups lightly.

So a year and a half ago, we ended up selling Lanyrd to Eventbrite and moving out to San Francisco. And at Eventbrite, I've been mostly on the management team building side of things, but occasionally managing to sneak some code out as well.

The one thing I want to show you from Eventbrite, because I really want to open source this thing, is again at Hack Day, we built a tool called the Tikibar, which is essentially like the Django debug toolbar, but it's designed to be run in production. Because the really tough things to debug don't happen in your dev environment. They happen in production when you're hitting a hundred million row database or whatever.

And so the Tikibar is designed to add as little overhead as possible, but to still give you detailed timelines of SQL queries that are executing and service calls and all of that kind of stuff. It's called the Tikibar because I really like Tikibars.

And the best feature is if a page takes over 500 milliseconds to load, the eyes on the Tiki God glow red in disapproval at you.

If anyone wants a demo of that, come and talk to me. I would love to get a few more instrumentation hooks into Django to make this stuff easier.

(The Tikibar was eventually open sourced as eventbrite/tikibar on GitHub.)

This has been a whistle-stop tour of the highlights of my career working with Django.

And actually, in putting this presentation together, I realized that really it's that Rob Curley influence from all the way back in 2003. The reason I love Django is it makes it really easy to build cool shit and to ship it. And, you know, swearing aside, I think that's a reasonable moral to take away from this.

Colophon

I put this annotated version of my 10 year old talk together using a few different tools.

I fetched the audio from YouTube using yt-dlp:

yt-dlp -x --audio-format mp3 \
  "https://youtube.com/watch?v=wqii_iX0RTs"

I then ran the mp3 through MacWhisper to generate an initial transcript. I cleaned that up by pasting it into Claude Opus 4 with this prompt:

Take this audio transcript of a talk and clean it up very slightly - I want paragraph breaks and tiny edits like removing ums or "sort of" or things like that, but other than that the content should be exactly as presented.

I converted a PDF of the slides into a JPEG per page using this command (found with the llm-cmd plugin):

pdftoppm -jpeg -jpegopt quality=70 django-birthday.pdf django-birthday

Then I used my annotated presentations tool (described here) to combine the slides and transcript, making minor edits and adding links using Markdown in that interface.

Tags: adrian-holovaty, devfort, django, history, jacob-kaplan-moss, lawrence, lawrence-com, lawrence-journal-world, python, my-talks, the-guardian, annotated-talks

Quoting Django’s security policies

2025-07-11T16:51:08+00:00

Following the widespread availability of large language models (LLMs), the Django Security Team has received a growing number of security reports generated partially or entirely using such tools. Many of these contain inaccurate, misleading, or fictitious content. While AI tools can help draft or analyze reports, they must not replace human understanding and review.

If you use AI tools to help prepare a report, you must:

Disclose which AI tools were used and specify what they were used for (analysis, writing the description, writing the exploit, etc).

Verify that the issue describes a real, reproducible vulnerability that otherwise meets these reporting guidelines.

Avoid fabricated code, placeholder text, or references to non-existent Django features.

Reports that appear to be unverified AI output will be closed without response. Repeated low-quality submissions may result in a ban from future reporting

— Django’s security policies, on AI-Assisted Reports

Tags: django, open-source, security, ai, generative-ai, llms, ai-ethics, ai-misuse, ai-security-research

PR #537: Fix Markdown in og descriptions

2025-06-03T23:58:34+00:00

PR #537: Fix Markdown in og descriptions

Since OpenAI Codex is now available to us ChatGPT Plus subscribers I decided to try it out against my blog.

It's a very nice implementation of the GitHub-connected coding "agent" pattern, as also seen in Google's Jules and Microsoft's Copilot Coding Agent.

First I had to configure an environment for it. My Django blog uses PostgreSQL which isn't part of the default Codex container, so I had Claude Sonnet 4 help me come up with a startup recipe to get PostgreSQL working.

I attached my simonw/simonwillisonblog GitHub repo and used the following as the "setup script" for the environment:

# Install PostgreSQL
apt-get update && apt-get install -y postgresql postgresql-contrib

# Start PostgreSQL service
service postgresql start

# Create a test database and user
sudo -u postgres createdb simonwillisonblog
sudo -u postgres psql -c "CREATE USER testuser WITH PASSWORD 'testpass';"
sudo -u postgres psql -c "GRANT ALL PRIVILEGES ON DATABASE simonwillisonblog TO testuser;"
sudo -u postgres psql -c "ALTER USER testuser CREATEDB;"

pip install -r requirements.txt

I left "Agent internet access" off for reasons described previously.

Then I prompted Codex with the following (after one previous experimental task to check that it could run my tests):

Notes and blogmarks can both use Markdown.

They serve meta property="og:description" content=" tags on the page, but those tags include that raw Markdown which looks bad on social media previews.

Fix it so they instead use just the text with markdown stripped - so probably render it to HTML and then strip the HTML tags.

Include passing tests.

Try to run the tests, the postgresql details are:

database = simonwillisonblog username = testuser password = testpass

Put those in the DATABASE_URL environment variable.

I left it to churn away for a few minutes (4m12s, to be precise) and it came back with a fix that edited two templates and added one more (passing) test. Here's that change in full.

And sure enough, the social media cards for my posts now look like this - no visible Markdown any more:

Tags: django, github, postgresql, testing, ai, openai, generative-ai, chatgpt, llms, ai-assisted-programming, ai-agents, coding-agents, async-coding-agents, jules, codex

django-simple-deploy

2025-05-17T12:49:52+00:00

django-simple-deploy

Eric Matthes presented a lightning talk about this project at PyCon US this morning. "Django has a deploy command now". You can run it like this:

pip install django-simple-deploy[fly_io]
# Add django_simple_deploy to INSTALLED_APPS.
python manage.py deploy --automate-all

It's plugin-based (inspired by Datasette!) and the project has stable plugins for three hosting platforms: dsd-flyio, dsd-heroku and dsd-platformsh.

Currently in development: dsd-vps - a plugin that should work with any VPS provider, using Paramiko to connect to a newly created instance and run all of the commands needed to start serving a Django application.

Tags: django, paramiko, plugins, python, heroku, datasette, fly

Django: what’s new in 5.2

2025-04-10T16:27:27+00:00

Django: what’s new in 5.2

Adam Johnson provides extremely detailed unofficial annotated release notes for the latest Django.

I found his explanation and example of Form BoundField customization particularly useful - here's the new pattern for customizing the class= attribute on the label associated with a CharField:

from django import forms

class WideLabelBoundField(forms.BoundField):
    def label_tag(self, contents=None, attrs=None, label_suffix=None):
        if attrs is None:
            attrs = {}
        attrs["class"] = "wide"
        return super().label_tag(contents, attrs, label_suffix)

class NebulaForm(forms.Form):
    name = forms.CharField(
        max_length=100,
        label="Nebula Name",
        bound_field_class=WideLabelBoundField,
    )

I'd also missed the new HttpResponse.get_preferred_type() method for implementing HTTP content negotiation:

content_type = request.get_preferred_type(
    ["text/html", "application/json"]
)

Tags: django, python, adam-johnson

Composite primary keys in Django

2025-04-02T14:51:53+00:00

Composite primary keys in Django

Django 5.2 is out today and a big new feature is composite primary keys, which can now be defined like this:

class Release(models.Model):
    pk = models.CompositePrimaryKey(
        "version", "name"
    )
    version = models.IntegerField()
    name = models.CharField(max_length=20)

They don't yet work with the Django admin or as targets for foreign keys.

Other smaller new features include:

All ORM models are now automatically imported into ./manage.py shell - a feature borrowed from ./manage.py shell_plus in django-extensions
Feeds from the Django syndication framework can now specify XSLT stylesheets
response.text now returns the string representation of the body - I'm so happy about this, now I don't have to litter my Django tests with response.content.decode("utf-8") any more
a new simple_block_tag helper making it much easier to create a custom Django template tag that further processes its own inner rendered content
A bunch more in the full release notes

5.2 is also an LTS release, so it will receive security and data loss bug fixes up to April 2028.

Tags: django, python

suitenumerique/docs

2025-03-17T18:51:50+00:00

suitenumerique/docs

New open source (MIT licensed) collaborative text editing web application, similar to Google Docs or Notion, notable because it's a joint effort funded by the French and German governments and "currently onboarding the Netherlands".

It's built using Django and React:

Docs is built on top of Django Rest Framework, Next.js, BlockNote.js, HocusPocus and Yjs.

Deployments currently require Kubernetes, PostgreSQL, memcached, an S3 bucket (or compatible) and an OIDC provider.

Tags: django, open-source, postgresql, s3, react, kubernetes

Smoke test your Django admin site

2025-03-13T15:02:09+00:00

Smoke test your Django admin site

Justin Duke demonstrates a neat pattern for running simple tests against your internal Django admin site: introspect every admin route via django.urls.get_resolver() and loop through them with @pytest.mark.parametrize to check they all return a 200 HTTP status code.

This catches simple mistakes with the admin configuration that trigger exceptions that might otherwise go undetected.

I rarely write automated tests against my own admin sites and often feel guilty about it. I wrote some notes on testing it with pytest-django fixtures a few years ago.

Tags: django, django-admin, python, testing, pytest

My approach to running a link blog

2024-12-22T18:37:16+00:00

I started running a basic link blog on this domain back in November 2003 - publishing links (which I called "blogmarks") with a title, URL, short snippet of commentary and a "via" link where appropriate.

So far I've published 7,607 link blog posts and counting.

In April of this year I finally upgraded my link blog to support Markdown, allowing me to expand my link blog into something with a lot more room.

The way I use my link blog has evolved substantially in the eight months since then. I'm going to describe the informal set of guidelines I've set myself for how I link blog, in the hope that it might encourage other people to give this a try themselves.

Writing about things I've found

Back in November 2022 I wrote What to blog about, which started with this:

You should start a blog. Having your own little corner of the internet is good for the soul!

The point of that article was to emphasize that blogging doesn't have to be about unique insights. The value is in writing frequently and having something to show for it over time - worthwhile even if you don't attract much of an audience (or any audience at all).

In that article I proposed two categories of content that are low stakes and high value: things I learned and descriptions of my projects.

I realize now that link blogging deserves to be included a third category of low stakes, high value writing. We could think of that category as things I've found.

That's the purpose of my link blog: it's an ongoing log of things I've found - effectively a combination of public bookmarks and my own thoughts and commentary on why those things are interesting.

Trying to add something extra

When I first started link blogging I would often post a link with a one sentence summary of the linked content, and maybe a tiny piece of opinionated commentary.

After I upgraded my link blog to support additional markup (links, images, quotations) I decided to be more ambitious. Here are some of the things I try to do:

I always include the names of the people who created the content I am linking to, if I can figure that out. Credit is really important, and it's also useful for myself because I can later search for someone's name and find other interesting things they have created that I linked to in the past. If I've linked to someone's work three or more times I also try to notice and upgrade them to a dedicated tag.
I try to add something extra. My goal with any link blog post is that if you read both my post and the source material you'll have an enhanced experience over if you read just the source material itself.
- Ideally I'd like you to take something useful away even if you don't follow the link itself. This can be a slightly tricky balance: I don't want to steal attention from the authors and plagiarize their message. Generally I'll try to find some key idea that's worth emphasizing. Slightly cynically, I may try to capture that idea as backup against the original source vanishing from the internet. Link rot is real!
- My most basic version of this is trying to provide context as to why I think this particular thing is worth reading - especially important for longer content. A good recent example is my post about Anthropic's Building effective agents essay the other day.
- I might tie it together to other similar concepts, including things I've written about in the past, for example linking Prompt caching with Claude to my coverage of Context caching for Google Gemini.
- If part of the material is a video, I might quote a snippet of the transcript (often extracted using MacWhisper) like I did in this post about Anthropic's Clio.
- A lot of stuff I link to involves programming. I'll often include a direct link to relevant code, using the GitHub feature where I can link to a snippet as-of a particular commit. One example is the fetch-rss.py link in this post.
I'm liberal with quotations. Finding and quoting a paragraph that captures the key theme of a post is a very quick and effective way to summarize it and help people decide if it's worth reading the whole thing. My post on François Chollet's o3 ARC-AGI analysis is an example of that.
If the original author reads my post, I want them to feel good about it. I know from my own experience that often when you publish something online the silence can be deafening. Knowing that someone else read, appreciated, understood and then shared your work can be very pleasant.
A slightly self-involved concern I have is that I like to prove that I've read it. This is more for me than for anyone else: I don't like to recommend something if I've not read that thing myself, and sticking in a detail that shows I read past the first paragraph helps keep me honest about that.
I've started leaning more into screenshots and even short video or audio clips. A screenshot can be considered a visual quotation - I'll sometimes snap these from interesting frames in a YouTube video or live demo associated with the content I'm linking to. I used a screenshot of the Clay debugger in my post about Clay.

There are a lot of great link blogs out there, but the one that has influenced me the most in how I approach my own is John Gruber's Daring Fireball. I really like the way he mixes commentary, quotations and value-added relevant information.

The technology

The technology behind my link blog is probably the least interesting thing about it. It's part of my simonwillisonblog Django application - the main model is called Blogmark and it inherits from a BaseModel defining things like tags and draft modes that are shared across my other types of content (entries and quotations).

I use the Django Admin to create and edit entries, configured here.

The most cumbersome part of link blogging for me right now is images. I convert these into smaller JPEGs using a tiny custom tool I built (with Claude), then upload them to my static.simonwillison.net S3 bucket using Transmit and drop them into my posts using a Markdown image reference. I generate a first draft of the alt text using a Claude Project with these custom instructions, then usually make a few changes before including that in the markup. At some point I'll wire together a UI that makes this process a little smoother.

That static.simonwillison.net bucket is then served via Cloudflare's free tier, which means I effectively never have to think about the cost of serving up those image files.

I wrote up a TIL about Building a blog in Django a while ago which describes a similar setup to the one I'm using for my link blog, including how the RSS feed works (using Django's syndication framework).

The most technically interesting component is my search feature. I wrote about how that works in Implementing faceted search with Django and PostgreSQL - the most recent code for that can be found in blog/search.py on GitHub.

One of the most useful small enhancements I added was draft mode, which lets me assign a URL to an item and preview it in my browser without publishing it to the world. This really helps when I am editing posts on my mobile phone as it gives me a reliable preview so I can check for any markup mistakes.

I also send out an approximately weekly email newsletter version of my blog, for people who want to subscribe in their inbox. This is a straight copy of content from my blog - Substack doesn't have an API for this but their editor does accept copy and paste, so I have a delightful digital duct tape solution for assembling the newsletter which I described in Semi-automating a Substack newsletter with an Observable notebook.

More people should do this

I posted this on Bluesky last night:

I wish people would post more links to interesting things

I feel like Twitter and LinkedIn and Instagram and TikTok have pushed a lot of people out of the habit of doing that, by penalizing shared links in the various "algorithms"

Bluesky doesn't have that misfeature, thankfully!

(In my ideal world everyone would get their own link blog too, but sharing links on Bluesky and Mastodon is almost as good)

Sharing interesting links with commentary is a low effort, high value way to contribute to internet life at large.

Tags: blogging, django, django-admin, john-gruber

Is async Django ready for prime time?

2024-11-24T17:47:27+00:00

Is async Django ready for prime time?

Jonathan Adly reports on his experience using Django to build ColiVara, a hosted RAG API that uses ColQwen2 visual embeddings, inspired by the ColPali paper.

In a breach of Betteridge's law of headlines the answer to the question posed by this headline is “yes”.

We believe async Django is ready for production. In theory, there should be no performance loss when using async Django instead of FastAPI for the same tasks.

The ColiVara application is itself open source, and you can see how it makes use of Django’s relatively new asynchronous ORM features in the api/views.py module.

I also picked up a useful trick from their Dockerfile: if you want uv in a container you can install it with this one-liner:

COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/uv

Via Hacker News

Tags: asynchronous, django, python, embeddings, rag, uv

django-plugin-django-debug-toolbar

2024-11-13T01:14:22+00:00

django-plugin-django-debug-toolbar

Tom Viner built a plugin for my DJP Django plugin system that configures the excellent django-debug-toolbar debugging tool.

You can see everything it sets up for you in this Python code: it configures installed apps, URL patterns and middleware and sets the INTERNAL_IPS and DEBUG settings.

Here are Tom's running notes as he created the plugin.

Via @tomviner

Tags: django, plugins, djp

Quoting Jacob Kaplan-Moss

2024-10-20T16:34:28+00:00

It feels like we’re at a bit of an inflection point for the Django community. [...] One of the places someone could have the most impact is by serving on the DSF Board. Like the community at large, the DSF is at a transition point: we’re outgrowing the “small nonprofit” status, and have the opportunity to really expand our ambition and reach. In all likelihood, the decisions the Board makes over the next year or two will define our direction and strategy for the next decade.

— Jacob Kaplan-Moss

Tags: django, jacob-kaplan-moss, dsf

2025 DSF Board Nominations

2024-10-16T23:01:22+00:00

2025 DSF Board Nominations

The Django Software Foundation board elections are coming up. There are four positions open, seven directors total. Terms last two years, and the deadline for submitting a nomination is October 25th (the date of the election has not yet been decided).

Several community members have shared "DSF initiatives I'd like to see" documents to inspire people who may be considering running for the board:

Sarah Boyce (current Django Fellow) wants a marketing strategy, better community docs, more automation and a refresh of the Django survey.
Tim Schilling wants one big sponsor, more community recognition and a focus on working groups.
Carlton Gibson wants an Executive Director, an updated website and better integration of the community into that website.
Jacob Kaplan-Moss wants effectively all of the above.

There's also a useful FAQ on the Django forum by Thibaud Colas.

Tags: django, jacob-kaplan-moss, dsf, carlton-gibson

jefftriplett/django-startproject

2024-10-12T23:19:01+00:00

jefftriplett/django-startproject

Django's django-admin startproject and startapp commands include a --template option which can be used to specify an alternative template for generating the initial code.

Jeff Triplett actively maintains his own template for new projects, which includes the pattern that I personally prefer of keeping settings and URLs in a config/ folder. It also configures the development environment to run using Docker Compose.

The latest update adds support for Python 3.13, Django 5.1 and uv. It's neat how you can get started without even installing Django using uv run like this:

uv run --with=django django-admin startproject \
  --extension=ini,py,toml,yaml,yml \
  --template=https://github.com/jefftriplett/django-startproject/archive/main.zip \
  example_project

Via @webology

Tags: django, python, docker, jeff-triplett, uv

If we had $1,000,000…

2024-10-08T19:59:39+00:00

If we had $1,000,000…

Jacob Kaplan-Moss gave my favorite talk at DjangoCon this year, imagining what the Django Software Foundation could do if it quadrupled its annual income to $1 million and laying out a realistic path for getting there. Jacob suggests leaning more into large donors than increasing our small donor base:

It’s far easier for me to picture convincing eight or ten or fifteen large companies to make large donations than it is to picture increasing our small donor base tenfold. So I think a major donor strategy is probably the most realistic one for us.

So when I talk about major donors, who am I talking about? I’m talking about four major categories: large corporations, high net worth individuals (very wealthy people), grants from governments (e.g. the Sovereign Tech Fund run out of Germany), and private foundations (e.g. the Chan Zuckerberg Initiative, who’s given grants to the PSF in the past).

Also included: a TIL on Turning a conference talk into an annotated presentation. Jacob used my annotated presentation tool to OCR text from images of keynote slides, extracted a Whisper transcript from the YouTube livestream audio and then cleaned that up a little with LLM and Claude 3.5 Sonnet ("Split the content of this transcript up into paragraphs with logical breaks. Add newlines between each paragraph.") before editing and re-writing it all into the final post.

Tags: django, jacob-kaplan-moss, whisper, llm, claude-3-5-sonnet, dsf

Django Commons

2024-10-08T03:27:40+00:00

Django Commons

Django Commons is a really promising initiative started by Tim Schilling, aimed at the problem of keeping key Django community projects responsibly maintained on a long-term basis.

Django Commons is an organization dedicated to supporting the community's efforts to maintain packages. It seeks to improve the maintenance experience for all contributors; reducing the barrier to entry for new contributors and reducing overhead for existing maintainers.

I’ve stated recently that I’d love to see the Django Software Foundation take on this role - adopting projects and ensuring they are maintained long-term. Django Commons looks like it solves that exact problem, assuring the future of key projects beyond their initial creators.

So far the Commons has taken on responsibility for django-fsm-2, django-tasks-scheduler and, as-of this week, diango-typer.

Here’s Tim introducing the project back in May. Thoughtful governance has been baked in from the start:

Having multiple administrators makes the role more sustainable, lessens the impact of a person stepping away, and shortens response time for administrator requests. It’s important to me that the organization starts with multiple administrators so that collaboration and documentation are at the forefront of all decisions.

Tags: django, open-source

Thoughts on the Treasurer Role at Tech NonProfits

2024-10-07T22:41:53+00:00

Thoughts on the Treasurer Role at Tech NonProfits

Will Vincent, Django Software Foundation treasurer from 2020-2022, explains what’s involved in the non-profit role with the highest level of responsibility and trust.

Tags: django, dsf

Building an automatically updating live blog in Django

2024-10-02T15:42:39+00:00

Building an automatically updating live blog in Django

Here's an extended write-up of how I implemented the live blog feature I used for my coverage of OpenAI DevDay yesterday. I built the first version using Claude while waiting for the keynote to start, then upgraded it during the lunch break with the help of GPT-4o to add sort options and incremental fetching of new updates.

Tags: django, javascript, ai, generative-ai, chatgpt, llms, ai-assisted-programming, claude

Weeknotes: Three podcasts, two trips and a new plugin system

2024-09-30T17:43:22+00:00

I fell behind a bit on my weeknotes. Here's most of what I've been doing in September.

Lisbon, Portugal and Durham, North Carolina

I had two trips this month. The first was a short visit to Lisbon, Portugal for the Python Software Foundation's annual board retreat. This inspired me to write about Things I've learned serving on the board of the Python Software Foundation.

The second was to Durham, North Carolina for DjangoCon US 2024. I wrote about that one in Themes from DjangoCon US 2024.

My talk at DjangoCon was about plugin systems, and in a classic example of conference-driven development I ended up writing and releasing a new plugin system for Django in preparation for that talk. I introduced that in DJP: A plugin system for Django.

Podcasts

I haven't been a podcast guest since January, and then three came along at once! All three appearences involved LLMs in some way but I don't think there was a huge amount of overlap in terms of what I actually said.

I went on The Software Misadventures Podcast to talk about my career to-date.
My appearance on TWIML dug into ways in which I use Claude and ChatGPT to help me write code.
I was the guest for the inaugral episode of Gergely Orosz's Pragmatic Engineer Podcast, which ended up touching on a whole array of different topics relevant to modern software engineering, from the importance of open source to the impact AI tools are likely to have on our industry.

Gergely has been sharing neat edited snippets from our conversation on Twitter. Here's one on RAG and another about how open source has been the the biggest productivity boost of my career.

On the blog

NotebookLM's automatically generated podcasts are surprisingly effective - Sept. 29, 2024
Themes from DjangoCon US 2024 - Sept. 27, 2024
DJP: A plugin system for Django - Sept. 25, 2024
Notes on using LLMs for code - Sept. 20, 2024
Things I've learned serving on the board of the Python Software Foundation - Sept. 18, 2024
Notes on OpenAI's new o1 chain-of-thought models - Sept. 12, 2024
Notes from my appearance on the Software Misadventures Podcast - Sept. 10, 2024
Teresa T is name of the whale in Pillar Point Harbor near Half Moon Bay - Sept. 8, 2024

Museums

The Vincent and Ethel Simonetti Historic Tuba Collection

Releases

shot-scraper 1.5 - 2024-09-27
A command-line utility for taking automated screenshots of websites
django-plugin-datasette 0.2 - 2024-09-26
Django plugin to run Datasette inside of Django
djp 0.3.1 - 2024-09-26
A plugin system for Django
llm-gemini 0.1a5 - 2024-09-24
LLM plugin to access Google's Gemini family of models
django-plugin-blog 0.1.1 - 2024-09-24
A blog for Django as a DJP plugin.
django-plugin-database-url 0.1 - 2024-09-24
Django plugin for reading the DATABASE_URL environment variable
django-plugin-django-header 0.1.1 - 2024-09-23
Add a Django-Compositions HTTP header to a Django app
llm-jina-api 0.1a0 - 2024-09-20
Access Jina AI embeddings via their API
llm 0.16 - 2024-09-12
Access large language models from the command-line
datasette-acl 0.4a4 - 2024-09-10
Advanced permission management for Datasette
llm-cmd 0.2a0 - 2024-09-09
Use LLM to generate and execute commands in your shell
files-to-prompt 0.3 - 2024-09-09
Concatenate a directory full of files into a single prompt for use with LLMs
json-flatten 0.3.1 - 2024-09-07
Python functions for flattening a JSON object to a single dictionary of pairs, and unflattening that dictionary back to a JSON object
csv-diff 1.2 - 2024-09-06
Python CLI tool and library for diffing CSV and JSON files
datasette 1.0a16 - 2024-09-06
An open source multi-tool for exploring and publishing data
datasette-search-all 1.1.4 - 2024-09-06
Datasette plugin for searching all searchable tables at once

TILs

How streaming LLM APIs work - 2024-09-21

Tags: django, podcasts, weeknotes, psf, llms, djp

Ensuring a block is overridden in a Django template

2024-09-29T19:25:43+00:00

Ensuring a block is overridden in a Django template

Neat Django trick by Tom Carrick: implement a Django template tag that raises a custom exception, then you can use this pattern in your templates:

{% block title %}{% ensure_overridden %}{% endblock %}

To ensure you don't accidentally extend a base template but forget to fill out a critical block.

Via Carlton Gibson

Tags: django, python

DjangoTV

2024-09-28T04:48:04+00:00

DjangoTV

Brand new site by Jeff Triplett gathering together videos from Django conferences around the world. Here's Jeff's blog post introducing the project.

Via @webology

Tags: django, jeff-triplett

Simon Willison's Weblog: django

Quoting Tim Schilling

Shopify/liquid: Performance: 53% faster parse+render, 61% fewer allocations

SWE-bench February 2026 leaderboard update

Adding dynamic features to an aggressively cached website

Edit links that are visible only to me

Random navigation within a tag

And the prompts

Django 6.0 released

YouTube embeds fail with a 153 error

Highlights from my appearance on the Data Renegades podcast with CL Kao and Dori Wilson

Colophon

How I automate my Substack newsletter with content from my blog

The Observable notebook

From Django+Postgresql to Datasette+SQLite

Talk Python: Celebrating Django's 20th Birthday With Its Creators

Happy 20th birthday Django! Here's my talk on Django Origins from Django's 10th

Django Origins (and some things I have built with Django)

Colophon

Quoting Django’s security policies

PR #537: Fix Markdown in og descriptions

django-simple-deploy

Django: what’s new in 5.2

Composite primary keys in Django

suitenumerique/docs

Smoke test your Django admin site

My approach to running a link blog

Writing about things I've found

Trying to add something extra

The technology

More people should do this

Is async Django ready for prime time?

django-plugin-django-debug-toolbar

Quoting Jacob Kaplan-Moss

2025 DSF Board Nominations

jefftriplett/django-startproject

If we had $1,000,000…

Django Commons

Thoughts on the Treasurer Role at Tech NonProfits

Building an automatically updating live blog in Django

Weeknotes: Three podcasts, two trips and a new plugin system

Lisbon, Portugal and Durham, North Carolina

Podcasts

On the blog

Museums

Releases

TILs

Ensuring a block is overridden in a Django template

DjangoTV