Simon Willison's Weblog: djp

django-plugin-django-debug-toolbar

2024-11-13T01:14:22+00:00

Tom Viner built a plugin for my DJP Django plugin system that configures the excellent django-debug-toolbar debugging tool.

You can see everything it sets up for you in this Python code: it configures installed apps, URL patterns and middleware and sets the INTERNAL_IPS and DEBUG settings.

Here are Tom's running notes as he created the plugin.

Via @tomviner

Tags: django, plugins, djp

Weeknotes: Three podcasts, two trips and a new plugin system

2024-09-30T17:43:22+00:00

I fell behind a bit on my weeknotes. Here's most of what I've been doing in September.

Lisbon, Portugal and Durham, North Carolina

I had two trips this month. The first was a short visit to Lisbon, Portugal for the Python Software Foundation's annual board retreat. This inspired me to write about Things I've learned serving on the board of the Python Software Foundation.

The second was to Durham, North Carolina for DjangoCon US 2024. I wrote about that one in Themes from DjangoCon US 2024.

My talk at DjangoCon was about plugin systems, and in a classic example of conference-driven development I ended up writing and releasing a new plugin system for Django in preparation for that talk. I introduced that in DJP: A plugin system for Django.

Podcasts

I haven't been a podcast guest since January, and then three came along at once! All three appearences involved LLMs in some way but I don't think there was a huge amount of overlap in terms of what I actually said.

I went on The Software Misadventures Podcast to talk about my career to-date.
My appearance on TWIML dug into ways in which I use Claude and ChatGPT to help me write code.
I was the guest for the inaugral episode of Gergely Orosz's Pragmatic Engineer Podcast, which ended up touching on a whole array of different topics relevant to modern software engineering, from the importance of open source to the impact AI tools are likely to have on our industry.

Gergely has been sharing neat edited snippets from our conversation on Twitter. Here's one on RAG and another about how open source has been the the biggest productivity boost of my career.

On the blog

NotebookLM's automatically generated podcasts are surprisingly effective - Sept. 29, 2024
Themes from DjangoCon US 2024 - Sept. 27, 2024
DJP: A plugin system for Django - Sept. 25, 2024
Notes on using LLMs for code - Sept. 20, 2024
Things I've learned serving on the board of the Python Software Foundation - Sept. 18, 2024
Notes on OpenAI's new o1 chain-of-thought models - Sept. 12, 2024
Notes from my appearance on the Software Misadventures Podcast - Sept. 10, 2024
Teresa T is name of the whale in Pillar Point Harbor near Half Moon Bay - Sept. 8, 2024

Museums

The Vincent and Ethel Simonetti Historic Tuba Collection

Releases

shot-scraper 1.5 - 2024-09-27
A command-line utility for taking automated screenshots of websites
django-plugin-datasette 0.2 - 2024-09-26
Django plugin to run Datasette inside of Django
djp 0.3.1 - 2024-09-26
A plugin system for Django
llm-gemini 0.1a5 - 2024-09-24
LLM plugin to access Google's Gemini family of models
django-plugin-blog 0.1.1 - 2024-09-24
A blog for Django as a DJP plugin.
django-plugin-database-url 0.1 - 2024-09-24
Django plugin for reading the DATABASE_URL environment variable
django-plugin-django-header 0.1.1 - 2024-09-23
Add a Django-Compositions HTTP header to a Django app
llm-jina-api 0.1a0 - 2024-09-20
Access Jina AI embeddings via their API
llm 0.16 - 2024-09-12
Access large language models from the command-line
datasette-acl 0.4a4 - 2024-09-10
Advanced permission management for Datasette
llm-cmd 0.2a0 - 2024-09-09
Use LLM to generate and execute commands in your shell
files-to-prompt 0.3 - 2024-09-09
Concatenate a directory full of files into a single prompt for use with LLMs
json-flatten 0.3.1 - 2024-09-07
Python functions for flattening a JSON object to a single dictionary of pairs, and unflattening that dictionary back to a JSON object
csv-diff 1.2 - 2024-09-06
Python CLI tool and library for diffing CSV and JSON files
datasette 1.0a16 - 2024-09-06
An open source multi-tool for exploring and publishing data
datasette-search-all 1.1.4 - 2024-09-06
Datasette plugin for searching all searchable tables at once

TILs

How streaming LLM APIs work - 2024-09-21

Tags: django, podcasts, weeknotes, psf, llms, djp

django-plugin-datasette

2024-09-26T21:57:52+00:00

django-plugin-datasette

I did some more work on my DJP plugin mechanism for Django at the DjangoCon US sprints today. I added a new plugin hook, asgi_wrapper(), released in DJP 0.3 and inspired by the similar hook in Datasette.

The hook only works for Django apps that are served using ASGI. It allows plugins to add their own wrapping ASGI middleware around the Django app itself, which means they can do things like attach entirely separate ASGI-compatible applications outside of the regular Django request/response cycle.

Datasette is one of those ASGI-compatible applications!

django-plugin-datasette uses that new hook to configure a new URL, /-/datasette/, which serves a full Datasette instance that scans through Django’s settings.DATABASES dictionary and serves an explore interface on top of any SQLite databases it finds there.

It doesn’t support authentication yet, so this will expose your entire database contents - probably best used as a local debugging tool only.

I did borrow some code from the datasette-mask-columns plugin to ensure that the password column in the auth_user column is reliably redacted. That column contains a heavily salted hashed password so exposing it isn’t necessarily a disaster, but I like to default to keeping hashes safe.

Tags: django, plugins, projects, sqlite, datasette, djp

Solving a bug with o1-preview, files-to-prompt and LLM

2024-09-25T18:41:13+00:00

Solving a bug with o1-preview, files-to-prompt and LLM

I added a new feature to DJP this morning: you can now have plugins specify their middleware in terms of how it should be positioned relative to other middleware - inserted directly before or directly after django.middleware.common.CommonMiddleware for example.

At one point I got stuck with a weird test failure, and after ten minutes of head scratching I decided to pipe the entire thing into OpenAI's o1-preview to see if it could spot the problem. I used files-to-prompt to gather the code and LLM to run the prompt:

files-to-prompt **/*.py -c | llm -m o1-preview "
The middleware test is failing showing all of these - why is MiddlewareAfter repeated so many times?

['MiddlewareAfter', 'Middleware3', 'MiddlewareAfter', 'Middleware5', 'MiddlewareAfter', 'Middleware3', 'MiddlewareAfter', 'Middleware2', 'MiddlewareAfter', 'Middleware3', 'MiddlewareAfter', 'Middleware5', 'MiddlewareAfter', 'Middleware3', 'MiddlewareAfter', 'Middleware4', 'MiddlewareAfter', 'Middleware3', 'MiddlewareAfter', 'Middleware5', 'MiddlewareAfter', 'Middleware3', 'MiddlewareAfter', 'Middleware2', 'MiddlewareAfter', 'Middleware3', 'MiddlewareAfter', 'Middleware5', 'MiddlewareAfter', 'Middleware3', 'MiddlewareAfter', 'Middleware', 'MiddlewareBefore']"

The model whirled away for a few seconds and spat out an explanation of the problem - one of my middleware classes was accidentally calling self.get_response(request) in two different places.

I did enjoy how o1 attempted to reference the relevant Django documentation and then half-repeated, half-hallucinated a quote from it:

This took 2,538 input tokens and 4,354 output tokens - by my calculations at $15/million input and $60/million output that prompt cost just under 30 cents.

Tags: ai, openai, generative-ai, llms, ai-assisted-programming, llm, o1, djp, llm-reasoning, files-to-prompt

DJP: A plugin system for Django

2024-09-25T14:00:42+00:00

DJP is a new plugin mechanism for Django, built on top of Pluggy. I announced the first version of DJP during my talk yesterday at DjangoCon US 2024, How to design and implement extensible software with plugins. I'll post a full write-up of that talk once the video becomes available - this post describes DJP and how to use what I've built so far.

Why plugins?

Django already has a thriving ecosystem of third-party apps and extensions. What can a plugin system add here?

If you've ever installed a Django extension - such as django-debug-toolbar or django-extensions - you'll be familiar with the process. You pip install the package, then add it to your list of INSTALLED_APPS in settings.py - and often configure other picees, like adding something to MIDDLEWARE or updating your urls.py with new URL patterns.

This isn't exactly a huge burden, but it's added friction. It's also the exact kind of thing plugin systems are designed to solve.

DJP addresses this. You configure DJP just once, and then any additional DJP-enabled plugins you pip install can automatically register configure themselves within your Django project.

Setting up DJP

There are three steps to adding DJP to an existing Django project:

pip install djp - or add it to your requirements.txt or similar.

Modify your settings.py to add these two lines:

# Can be at the start of the file:
import djp

# This MUST be the last line:
djp.settings(globals())

Modify your urls.py to contain the following:

import djp

urlpatterns = [
    # Your existing URL patterns
] + djp.urlpatterns()

That's everything. The djp.settings(globals()) line is a little bit of magic - it gives djp an opportunity to make any changes it likes to your configured settings.

You can see what that does here. Short version: it adds "djp" and any other apps from plugins to INSTALLED_APPS, modifies MIDDLEWARE for any plugins that need to do that and gives plugins a chance to modify any other settings they need to.

One of my personal rules of plugin system design is that you should never ship a plugin hook (a customization point) without releasing at least one plugin that uses it. This validates the design and provides executable documentation in the form of working code.

I've released three plugins for DJP so far.

django-plugin-django-header

django-plugin-django-header is a very simple initial example. It registers a Django middleware class that adds a Django-Composition: HTTP header to every response with the name of a random Composition by Django Reinhardt (thanks,Wikipedia).

pip install django-plugin-django-header

Then try it out with curl:

curl -I http://localhost:8000/

You should get back something like this:

...
Django-Composition: Nuages
...

I'm running this on my blog right now! Try this command to see it in action:

curl -I https://simonwillison.net/

The plugin is very simple. Its __init__.py registers middleware like this:

import djp

@djp.hookimpl
def middleware():
    return [
        "django_plugin_django_header.middleware.DjangoHeaderMiddleware"
    ]

That string references the middleware class in this file.

django-plugin-blog

django-plugin-blog is a much bigger example. It implements a full blog system for your Django application, with bundled models and templates and views and a URL configuration.

You'll need to have configured auth and the Django admin already (those already there by default in the django-admin startproject template). Now install the plugin:

pip install django-plugin-blog

And run migrations to create the new database tables:

python manage.py migrate

That's all you need to do. Navigating to /blog/ will present the index page of the blog, including a link to a working Atom feed.

You can add entries and tags through the Django admin (configured for you by the plugin) and those will show up on /blog/, get their own URLs at /blog/2024/<slug>/ and be included in the Atom feed, the /blog/archive/ list and the /blog/2024/ year-based index too.

The default design is very basic, but you can customize that by providing your own base template or providing custom templates for each of the pages. There are details on the templates in the README.

The blog implementation is directly adapted from my Building a blog in Django TIL.

The primary goal of this plugin is to demonstrate what a plugin with views, templates, models and a URL configuration looks like. Here's the full __init__.py for the plugin:

from django.urls import path
from django.conf import settings
import djp

@djp.hookimpl
def installed_apps():
    return ["django_plugin_blog"]

@djp.hookimpl
def urlpatterns():
    from .views import index, entry, year, archive, tag, BlogFeed

    blog = getattr(settings, "DJANGO_PLUGIN_BLOG_URL_PREFIX", None) or "blog"
    return [
        path(f"{blog}/", index, name="django_plugin_blog_index"),
        path(f"{blog}/<int:year>/<slug:slug>/", entry, name="django_plugin_blog_entry"),
        path(f"{blog}/archive/", archive, name="django_plugin_blog_archive"),
        path(f"{blog}/<int:year>/", year, name="django_plugin_blog_year"),
        path(f"{blog}/tag/<slug:slug>/", tag, name="django_plugin_blog_tag"),
        path(f"{blog}/feed/", BlogFeed(), name="django_plugin_blog_feed"),
    ]

It still only needs to implement two hooks: one to add django_plugin_blog to the INSTALLED_APPS list and another to add the necessary URL patterns to the project.

The from .views import ... line is nested inside the urlpatterns() hook because I was hitting circular import issues with those imports at the top of the module.

django-plugin-database-url

django-plugin-database-url is the smallest of my example plugins. It exists mainly to exercise the settings() plugin hook, which allows plugins to further manipulate settings in any way they like.

Quoting the README:

Once installed, any DATABASE_URL environment variable will be automatically used to configure your Django database setting, using dj-database-url.

Here's the full implementation of that plugin, most of which is copied straight from the dj-database-url documentation:

import djp
import dj_database_url

@djp.hookimpl
def settings(current_settings):
    current_settings["DATABASES"]["default"] = dj_database_url.config(
        conn_max_age=600,
        conn_health_checks=True,
    )

If DJP gains traction, I expect that a lot of plugins will look like this - thin wrappers around existing libraries where the only added value is that they configure those libraries automatically once the plugin is installed.

Writing a plugin

A plugin is a Python package bundling a module that implements one or more of the DJP plugin hooks.

As I've shown above, the Python code for plugins can be very short. The larger challenge is correctly packaging and distributing the plugin - plugins are discovered using Entry Points which are defined in a pyproject.toml file, and you need to get those exactly right for your plugin to be discovered.

DJP includes documentation on creating a plugin, but to make it as frictionless as possible I've released a new django-plugin cookiecutter template.

This means you can start a new plugin like this:

pip install cookiecutter
cookiecutter gh:simonw/django-plugin

Then answer the questions:

  [1/6] plugin_name (): django-plugin-example
  [2/6] description (): A simple example plugin
  [3/6] hyphenated (django-plugin-example):
  [4/6] underscored (django_plugin_example):
  [5/6] github_username (): simonw
  [6/6] author_name (): Simon Willison

And you'l get a django-plugin-example directory with a fully configured plugin ready to be published to PyPI.

The template includes a .github/workflows directory with actions that can run tests, and an action that publishes your plugin to PyPI any time you create a new release on GitHub.

I've used that pattern myself for hundreds of plugin projects for Datasette and LLM, so I'm confident this is an effective way to release plugins.

The workflows use PyPI's Trusted Publishers mechanism (see my TIL), which means you don't need to worry about API keys or PyPI credentials - configure the GitHub repo once using the PyPI UI and everything should just work.

Writing tests for plugins

Writing tests for plugins can be a little tricky, especially if they need to spin up a full Django environemnt in order to run the tests.

I previously published a TIL about that, showing how to have tests with their own tests/test_project project that can be used by pytest-django.

I've baked that pattern into the simon/django-plugin cookiecutter template as well, plus a single default test which checks that a hit to the / index page returns a 200 status code - still a valuable default test since it confirms the plugin hasn't broken everything!

The tests for django-plugin-django-header and for django-plugin-blog should provide a useful starting point for writing tests for your own plugins.

Why call it DJP?

Because django-plugins already existed on PyPI, and I like my three letter acronyms there!

What's next for DJP?

I presented this at DjangoCon US 2024 yesterday afternoon. Initial response seemed positive, and I'm going to be attending the conference sprints on Thursday morning to see if anyone wants to write their own plugin or help extend the system further.

Is this a good idea? I think so. Plugins have been transformative for both Datasette and LLM, and I think Pluggy provides a mature, well-designed foundation for this kind of system.

I'm optimistic about plugins as a natural extension of Django's existing ecosystem. Let's see where this goes.

Tags: django, plugins, projects, djp