Simon Willison's Weblog: django-admin

Smoke test your Django admin site

2025-03-13T15:02:09+00:00

Justin Duke demonstrates a neat pattern for running simple tests against your internal Django admin site: introspect every admin route via django.urls.get_resolver() and loop through them with @pytest.mark.parametrize to check they all return a 200 HTTP status code.

This catches simple mistakes with the admin configuration that trigger exceptions that might otherwise go undetected.

I rarely write automated tests against my own admin sites and often feel guilty about it. I wrote some notes on testing it with pytest-django fixtures a few years ago.

Tags: django, django-admin, python, testing, pytest

My approach to running a link blog

2024-12-22T18:37:16+00:00

I started running a basic link blog on this domain back in November 2003 - publishing links (which I called "blogmarks") with a title, URL, short snippet of commentary and a "via" link where appropriate.

So far I've published 7,607 link blog posts and counting.

In April of this year I finally upgraded my link blog to support Markdown, allowing me to expand my link blog into something with a lot more room.

The way I use my link blog has evolved substantially in the eight months since then. I'm going to describe the informal set of guidelines I've set myself for how I link blog, in the hope that it might encourage other people to give this a try themselves.

Writing about things I've found

Back in November 2022 I wrote What to blog about, which started with this:

You should start a blog. Having your own little corner of the internet is good for the soul!

The point of that article was to emphasize that blogging doesn't have to be about unique insights. The value is in writing frequently and having something to show for it over time - worthwhile even if you don't attract much of an audience (or any audience at all).

In that article I proposed two categories of content that are low stakes and high value: things I learned and descriptions of my projects.

I realize now that link blogging deserves to be included a third category of low stakes, high value writing. We could think of that category as things I've found.

That's the purpose of my link blog: it's an ongoing log of things I've found - effectively a combination of public bookmarks and my own thoughts and commentary on why those things are interesting.

Trying to add something extra

When I first started link blogging I would often post a link with a one sentence summary of the linked content, and maybe a tiny piece of opinionated commentary.

After I upgraded my link blog to support additional markup (links, images, quotations) I decided to be more ambitious. Here are some of the things I try to do:

I always include the names of the people who created the content I am linking to, if I can figure that out. Credit is really important, and it's also useful for myself because I can later search for someone's name and find other interesting things they have created that I linked to in the past. If I've linked to someone's work three or more times I also try to notice and upgrade them to a dedicated tag.
I try to add something extra. My goal with any link blog post is that if you read both my post and the source material you'll have an enhanced experience over if you read just the source material itself.
- Ideally I'd like you to take something useful away even if you don't follow the link itself. This can be a slightly tricky balance: I don't want to steal attention from the authors and plagiarize their message. Generally I'll try to find some key idea that's worth emphasizing. Slightly cynically, I may try to capture that idea as backup against the original source vanishing from the internet. Link rot is real!
- My most basic version of this is trying to provide context as to why I think this particular thing is worth reading - especially important for longer content. A good recent example is my post about Anthropic's Building effective agents essay the other day.
- I might tie it together to other similar concepts, including things I've written about in the past, for example linking Prompt caching with Claude to my coverage of Context caching for Google Gemini.
- If part of the material is a video, I might quote a snippet of the transcript (often extracted using MacWhisper) like I did in this post about Anthropic's Clio.
- A lot of stuff I link to involves programming. I'll often include a direct link to relevant code, using the GitHub feature where I can link to a snippet as-of a particular commit. One example is the fetch-rss.py link in this post.
I'm liberal with quotations. Finding and quoting a paragraph that captures the key theme of a post is a very quick and effective way to summarize it and help people decide if it's worth reading the whole thing. My post on François Chollet's o3 ARC-AGI analysis is an example of that.
If the original author reads my post, I want them to feel good about it. I know from my own experience that often when you publish something online the silence can be deafening. Knowing that someone else read, appreciated, understood and then shared your work can be very pleasant.
A slightly self-involved concern I have is that I like to prove that I've read it. This is more for me than for anyone else: I don't like to recommend something if I've not read that thing myself, and sticking in a detail that shows I read past the first paragraph helps keep me honest about that.
I've started leaning more into screenshots and even short video or audio clips. A screenshot can be considered a visual quotation - I'll sometimes snap these from interesting frames in a YouTube video or live demo associated with the content I'm linking to. I used a screenshot of the Clay debugger in my post about Clay.

There are a lot of great link blogs out there, but the one that has influenced me the most in how I approach my own is John Gruber's Daring Fireball. I really like the way he mixes commentary, quotations and value-added relevant information.

The technology

The technology behind my link blog is probably the least interesting thing about it. It's part of my simonwillisonblog Django application - the main model is called Blogmark and it inherits from a BaseModel defining things like tags and draft modes that are shared across my other types of content (entries and quotations).

I use the Django Admin to create and edit entries, configured here.

The most cumbersome part of link blogging for me right now is images. I convert these into smaller JPEGs using a tiny custom tool I built (with Claude), then upload them to my static.simonwillison.net S3 bucket using Transmit and drop them into my posts using a Markdown image reference. I generate a first draft of the alt text using a Claude Project with these custom instructions, then usually make a few changes before including that in the markup. At some point I'll wire together a UI that makes this process a little smoother.

That static.simonwillison.net bucket is then served via Cloudflare's free tier, which means I effectively never have to think about the cost of serving up those image files.

I wrote up a TIL about Building a blog in Django a while ago which describes a similar setup to the one I'm using for my link blog, including how the RSS feed works (using Django's syndication framework).

The most technically interesting component is my search feature. I wrote about how that works in Implementing faceted search with Django and PostgreSQL - the most recent code for that can be found in blog/search.py on GitHub.

One of the most useful small enhancements I added was draft mode, which lets me assign a URL to an item and preview it in my browser without publishing it to the world. This really helps when I am editing posts on my mobile phone as it gives me a reliable preview so I can check for any markup mistakes.

I also send out an approximately weekly email newsletter version of my blog, for people who want to subscribe in their inbox. This is a straight copy of content from my blog - Substack doesn't have an API for this but their editor does accept copy and paste, so I have a delightful digital duct tape solution for assembling the newsletter which I described in Semi-automating a Substack newsletter with an Observable notebook.

More people should do this

I posted this on Bluesky last night:

I wish people would post more links to interesting things

I feel like Twitter and LinkedIn and Instagram and TikTok have pushed a lot of people out of the habit of doing that, by penalizing shared links in the various "algorithms"

Bluesky doesn't have that misfeature, thankfully!

(In my ideal world everyone would get their own link blog too, but sharing links on Bluesky and Mastodon is almost as good)

Sharing interesting links with commentary is a low effort, high value way to contribute to internet life at large.

Tags: blogging, django, django-admin, john-gruber

nanodjango

2024-09-24T16:08:44+00:00

nanodjango

Richard Terry demonstrated this in a lightning talk at DjangoCon US today. It's the latest in a long line of attempts to get Django to work with a single file (I had a go at this problem 15 years ago with djng) but this one is really compelling.

I tried nanodjango out just now and it works exactly as advertised. First install it like this:

pip install nanodjango

Create a counter.py file:

from django.db import models
from nanodjango import Django

app = Django()

@app.admin # Registers with the Django admin
class CountLog(models.Model):
    timestamp = models.DateTimeField(auto_now_add=True)

@app.route("/")
def count(request):
    CountLog.objects.create()
    return f"<p>Number of page loads: {CountLog.objects.count()}</p>"

Then run it like this (it will run migrations and create a superuser as part of that first run):

nanodjango run counter.py

That's it! This gave me a fully configured Django application with models, migrations, the Django Admin configured and a bunch of other goodies such as Django Ninja for API endpoints.

Here's the full documentation.

Tags: django, django-admin, python, radiac

django-http-debug, a new Django app mostly written by Claude

2024-08-08T15:26:27+00:00

Yesterday I finally developed something I’ve been casually thinking about building for a long time: django-http-debug. It’s a reusable Django app - something you can pip install into any Django project - which provides tools for quickly setting up a URL that returns a canned HTTP response and logs the full details of any incoming request to a database table.

This is ideal for any time you want to start developing against some external API that sends traffic to your own site - a webhooks provider like Stripe, or an OAuth or OpenID connect integration (my task yesterday morning).

You can install it right now in your own Django app: add django-http-debug to your requirements (or just pip install django-http-debug), then add the following to your settings.py:

INSTALLED_APPS = [
    # ...
    'django_http_debug',
    # ...
]

MIDDLEWARE = [
    # ...
    "django_http_debug.middleware.DebugMiddleware",
    # ...
]

You'll need to have the Django Admin app configured as well. The result will be two new models managed by the admin - one for endpoints:

And a read-only model for viewing logged requests:

It’s possible to disable logging for an endpoint, which means django-http-debug doubles as a tool for adding things like a robots.txt to your site without needing to deploy any additional code.

How it works

The key to how this works is this piece of middleware:

class DebugMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        response = self.get_response(request)
        if response.status_code == 404:
            path = request.path.lstrip("/")
            debug_response = debug_view(request, path)
            if debug_response:
                return debug_response
        return response

This dispatches to the default get_response() function, then intercepts the result and checks if it's a 404. If so, it gives the debug_view() function an opportunity to respond instead - which might return None, in which case that original 404 is returned to the client.

That debug_view() function looks like this:

@csrf_exempt
def debug_view(request, path):
    try:
        endpoint = DebugEndpoint.objects.get(path=path)
    except DebugEndpoint.DoesNotExist:
        return None  # Allow normal 404 handling to continue

    if endpoint.logging_enabled:
        log_entry = RequestLog(
            endpoint=endpoint,
            method=request.method,
            query_string=request.META.get("QUERY_STRING", ""),
            headers=dict(request.headers),
        )
        log_entry.set_body(request.body)
        log_entry.save()

    content = endpoint.content
    if endpoint.is_base64:
        content = base64.b64decode(content)

    response = HttpResponse(
        content=content,
        status=endpoint.status_code,
        content_type=endpoint.content_type,
    )
    for key, value in endpoint.headers.items():
        response[key] = value

    return response

It checks the database for an endpoint matching the incoming path, then logs the response (if the endpoint has logging_enabled set) and returns a canned response based on the endpoint configuration.

Here are the models:

from django.db import models
import base64


class DebugEndpoint(models.Model):
    path = models.CharField(max_length=255, unique=True)
    status_code = models.IntegerField(default=200)
    content_type = models.CharField(max_length=64, default="text/plain; charset=utf-8")
    headers = models.JSONField(default=dict, blank=True)
    content = models.TextField(blank=True)
    is_base64 = models.BooleanField(default=False)
    logging_enabled = models.BooleanField(default=True)

    def __str__(self):
        return self.path

    def get_absolute_url(self):
        return f"/{self.path}"


class RequestLog(models.Model):
    endpoint = models.ForeignKey(DebugEndpoint, on_delete=models.CASCADE)
    method = models.CharField(max_length=10)
    query_string = models.CharField(max_length=255, blank=True)
    headers = models.JSONField()
    body = models.TextField(blank=True)
    is_base64 = models.BooleanField(default=False)
    timestamp = models.DateTimeField(auto_now_add=True)

    def __str__(self):
        return f"{self.method} {self.endpoint.path} at {self.timestamp}"

    def set_body(self, body):
        try:
            # Try to decode as UTF-8
            self.body = body.decode("utf-8")
            self.is_base64 = False
        except UnicodeDecodeError:
            # If that fails, store as base64
            self.body = base64.b64encode(body).decode("ascii")
            self.is_base64 = True

    def get_body(self):
        if self.is_base64:
            return base64.b64decode(self.body.encode("ascii"))
        return self.body

The admin screens are defined in admin.py.

Claude built the first version of this for me

This is a classic example of a project that I couldn’t quite justify building without assistance from an LLM. I wanted it to exist, but I didn't want to spend a whole day building it.

Claude 3.5 Sonnet got me 90% of the way to a working first version. I had to make a few tweaks to how the middleware worked, but having done that I had a working initial prototype within a few minutes of starting the project.

Here’s the full sequence of prompts I used, each linking to the code that was produced for me (as a Claude artifact):

I want a Django app I can use to help create HTTP debugging endpoints. It should let me configure a new path e.g. /webhooks/receive/ that the Django 404 handler then hooks into - if one is configured it can be told which HTTP status code, headers and content to return.

ALL traffic to those endpoints is logged to a Django table - full details of incoming request headers, method and body. Those can be browsed read-only in the Django admin (and deleted)

Produced Claude v1

make it so I don't have to put it in the urlpatterns because it hooks ito Django's 404 handling mechanism instead

Produced Claude v2

Suggestions for how this could handle request bodies that don't cleanly decode to utf-8

Produced Claude v3

don't use a binary field, use a text field but still store base64 data in it if necessary and have a is_base64 boolean column that gets set to true if that happens

Produced Claude v4

I took that code and ran with it - I fired up a new skeleton library using my python-lib cookiecutter template, copied the code into it, made some tiny changes to get it to work and shipped it as an initial alpha release - mainly so I could start exercising it on a couple of sites I manage.

Using it in the wild for a few minutes quickly identified changes I needed to make. I filed those as issues:

Then I worked though fixing each of those one at a time. I did most of this work myself, though GitHub Copilot helped me out be typing some of the code for me.

Adding the base64 preview

There was one slightly tricky feature I wanted to add that didn’t justify spending much time on but was absolutely a nice-to-have.

The logging mechanism supports binary data: if incoming request data doesn’t cleanly encode as UTF-8 it gets stored as Base 64 text instead, with the is_base64 flag set to True (see the set_body() method in the RequestLog model above).

I asked Claude for a curl one-liner to test this and it suggested:

curl -X POST http://localhost:8000/foo/ \
  -H "Content-Type: multipart/form-data" \
  -F "image=@pixel.gif"

I do this a lot - knocking out quick curl commands is an easy prompt, and you can tell it the URL and headers you want to use, saving you from having to edit the command yourself later on.

I decided to have the Django Admin view display a decoded version of that Base 64 data. But how to render that, when things like binary file uploads may not be cleanly renderable as text?

This is what I came up with:

The trick here I'm using here is to display the decoded data as a mix between renderable characters and hex byte pairs, with those pairs rendered using a different font to make it clear that they are part of the binary data.

This is achieved using a body_display() method on the RequestLogAdmin admin class, which is then listed in readonly_fields. The full code is here, this is that method:

    def body_display(self, obj):
        body = obj.get_body()
        if not isinstance(body, bytes):
            return format_html("<pre>{}</pre>", body)

        # Attempt to guess filetype
        suggestion = None
        match = filetype.guess(body[:1000])
        if match:
            suggestion = "{} ({})".format(match.extension, match.mime)

        encoded = repr(body)
        # Ditch the b' and trailing '
        if encoded.startswith("b'") and encoded.endswith("'"):
            encoded = encoded[2:-1]

        # Split it into sequences of octets and characters
        chunks = sequence_re.split(encoded)
        html = []
        if suggestion:
            html.append(
                '<p style="margin-top: 0; font-family: monospace; font-size: 0.8em;">Suggestion: {}</p>'.format(
                    suggestion
                )
            )
        for chunk in chunks:
            if sequence_re.match(chunk):
                octets = octet_re.findall(chunk)
                octets = [o[2:] for o in octets]
                html.append(
                    '<code style="color: #999; font-family: monospace">{}</code>'.format(
                        " ".join(octets).upper()
                    )
                )
            else:
                html.append(chunk.replace("\\\\", "\\"))

        return mark_safe(" ".join(html).strip().replace("\\r\\n", "<br>"))

I got Claude to write that using one of my favourite prompting tricks. I'd solved this problem once before in the past, in my datasette-render-binary project. So I pasted that code into Claude, told it:

With that code as inspiration, modify the following Django Admin code to use that to display decoded base64 data:

And then pasted in my existing Django admin class. You can see my full prompt here.

Claude replied with this code, which almost worked exactly as intended - I had to make one change, swapping out the last line for this:

        return mark_safe(" ".join(html).strip().replace("\\r\\n", "<br>"))

I love this pattern: "here's my existing code, here's some other code I wrote, combine them together to solve this problem". I wrote about this previously when I described how I built my PDF OCR JavaScript tool a few months ago.

Adding automated tests

The final challenge was the hardest: writing automated tests. This was difficult because Django tests need a full Django project configured for them, and I wasn’t confident about the best pattern for doing that in my standalone django-http-debug repository since it wasn’t already part of an existing Django project.

I decided to see if Claude could help me with that too, this time using my files-to-prompt and LLM command-line tools:

files-to-prompt . --ignore LICENSE | \
  llm -m claude-3.5-sonnet -s \
  'step by step advice on how to implement automated tests for this, which is hard because the tests need to work within a temporary Django project that lives in the tests/ directory somehow. Provide all code at the end.'

Here's Claude's full response. It almost worked! It gave me a minimal test project in tests/test_project and an initial set of quite sensible tests.

Sadly it didn’t quite solve the most fiddly problem for me: configuring it so running pytest would correctly set the Python path and DJANGO_SETTINGS_MODULE in order run the tests. I saw this error instead:

django.core.exceptions.ImproperlyConfigured: Requested setting INSTALLED_APPS, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

I spent some time with the relevant pytest-django documentation and figure out a pattern that worked. Short version: I added this to my pyproject.toml file:

[tool.pytest.ini_options]
DJANGO_SETTINGS_MODULE = "tests.test_project.settings"
pythonpath = ["."]

For the longer version, take a look at my full TIL: Using pytest-django with a reusable Django application.

Test-supported cleanup

The great thing about having comprehensive tests in place is it makes iterating on the project much faster. Claude had used some patterns that weren’t necessary. I spent a few minutes seeing if the tests still passed if I deleted various pieces of code, and cleaned things up quite a bit.

Was Claude worth it?

This entire project took about two hours - just within a tolerable amount of time for what was effectively a useful sidequest from my intended activity for the day.

Claude didn't implement the whole project for me. The code it produced didn't quite work - I had to tweak just a few lines of code, but knowing which code to tweak took a development environment and manual testing and benefited greatly from my 20+ years of Django experience!

This is yet another example of how LLMs don't replace human developers: they augment us.

The end result is a tool that I'm already using to solve real-world problems, and a code repository that I'm proud to put my name to. Without LLM assistance this project would have stayed on my ever-growing list of "things I'd love to build one day".

I'm also really happy to have my own documented solution to the challenge of adding automated tests to a standalone reusable Django application. I was tempted to skip this step entirely, but thanks to Claude's assistance I was able to break that problem open and come up with a solution that I'm really happy with.

Last year I wrote about how AI-enhanced development makes me more ambitious with my projects. It's also helping me be more diligent in not taking shortcuts like skipping setting up automated tests.

Tags: django, django-admin, http, projects, python, webhooks, ai, generative-ai, llms, ai-assisted-programming, anthropic, claude, claude-3-5-sonnet

Weeknotes: Vaccinate The States, and how I learned that returning dozens of MB of JSON works just fine these days

2021-04-26T01:02:22+00:00

On Friday VaccinateCA grew in scope, a lot: we launched a new website called Vaccinate The States. Patrick McKenzie wrote more about the project here - the short version is that we're building the most comprehensive possible dataset of vaccine availability in the USA, using a combination of data collation, online research and continuing to make a huge number of phone calls.

VIAL, the Django application I've been working on since late February, had to go through some extensive upgrades to help support this effort!

VIAL has a number of responsibilities. It acts as our central point of truth for the vaccination locations that we are tracking, powers the app used by our callers to serve up locations to call and record the results, and as-of this week it's also a central point for our efforts to combine data from multiple other providers and scrapers.

The data ingestion work is happening in a public repository, CAVaccineInventory/vaccine-feed-ingest. I have yet to write a single line of code there (and I thoroughly enjoy working on that kind of code) because I've been heads down working on VIAL itself to ensure it can support the ingestion efforts.

Matching and concordances

If you're combining data about vaccination locations from a range of different sources, one of the biggest challenges is de-duplicating the data: it's important the same location doesn't show up multiple times (potentially with slightly differing details) due to appearing in multiple sources.

Our first step towards handling this involved the addition of "concordance identifiers" to VIAL.

I first encountered the term "concordance" being used for this in the Who's On First project, which is building a gazetteer of every city/state/country/county/etc on earth.

A concordance is an identifier in another system. Our location ID for RITE AID PHARMACY 05976 in Santa Clara is receu5biMhfN8wH7P - which is e3dfcda1-093f-479a-8bbb-14b80000184c in VaccineFinder and 7537904 in Vaccine Spotter and ChIJZaiURRPKj4ARz5nAXcWosUs in Google Places.

We're storing them in a Django table called ConcordanceIdentifier: each record has an authority (e.g. vaccinespotter_org) and an identifier (7537904) and a many-to-many relationship to our Location model.

Why many-to-many? Surely we only want a single location for any one of these identifiers?

Exactly! That's why it's many-to-many: because if we import the same location twice, then assign concordance identifiers to it, we can instantly spot that it's a duplicate and needs to be merged.

Raw data from scrapers

ConcordanceIdentifier also has a many-to-many relationship with a new table, called SourceLocation. This table is essentially a PostgreSQL JSON column with a few other columns (including latitude and longitude) into which our scrapers and ingesters can dump raw data. This means we can use PostgreSQL queries to perform all kinds of analysis on the unprocessed data before it gets cleaned up, de-duplicated and loaded into our point-of-truth Location table.

How to dedupe and match locations?

Initially I thought we would do the deduping and matching inside of VIAL itself, using the raw data that had been ingested into the SourceLocation table.

Since we were on a tight internal deadline it proved more practical for people to start experimenting with matching code outside of VIAL. But that meant they needed the raw data - 40,000+ location records (and growing rapidly).

A few weeks ago I built a CSV export feature for the VIAL admin screens, using Django's StreamingHttpResponse class combined with keyset pagination for bulk export without sucking the entire table into web server memory - details in this TIL.

Our data ingestion team wanted a GeoJSON export - specifically newline-delimited GeoJSON - which they could then load into GeoPandas to help run matching operations.

So I built a simple "search API" which defaults to returning 20 results at a time, but also has an option to "give me everything" - using the same technique I used for the CSV export: keyset pagination combined with a StreamingHttpResponse.

And it worked! It turns out that if you're running on modern infrastructure (Cloud Run and Cloud SQL in our case) in 2021 getting Django to return 50+MB of JSON in a streaming response works just fine.

Some of these exports are taking 20+ seconds, but for a small audience of trusted clients that's completely fine.

While working on this I realized that my idea of what size of data is appropriate for a dynamic web application to return more or less formed back in 2005. I still think it's rude to serve multiple MBs of JavaScript up to an inexpensive mobile phone on an expensive connection, but for server-to-server or server-to-automation-script situations serving up 50+ MB of JSON in one go turns out to be a perfectly cromulent way of doing things.

Export full results from django-sql-dashboard

django-sql-dashboard is my Datasette-inspired library for adding read-only arbitrary SQL queries to any Django+PostgreSQL application.

I built the first version last month to help compensate for switching VaccinateCA away from Airtable - one of the many benefits of Airtable is that it allows all kinds of arbitrary reporting, and Datasette has shown me that bookmarkable SQL queries can provide a huge amount of that value with very little written code, especially within organizations where SQL is already widely understood.

While it allows people to run any SQL they like (against a read-only PostgreSQL connection with a time limit) it restricts viewing to the first 1,000 records to be returned - because building robust, performant pagination against arbitrary SQL queries is a hard problem to solve.

Today I released django-sql-dashboard 0.10a0 with the ability to export all results for a query as a downloadable CSV or TSV file, using the same StreamingHttpResponse technique as my Django admin CSV export and all-results-at-once search endpoint.

I expect it to be pretty useful! It means I can run any SQL query I like against a Django project and get back the full results - often dozens of MBs - in a form I can import into other tools (including Datasette).

TIL this week

Releases this week

django-sql-dashboard: 0.10a1 - (21 total releases) - 2021-04-25
Django app for building dashboards using raw SQL queries

Tags: csv, django, django-admin, postgresql, projects, vaccines, weeknotes, vaccinate-ca, django-sql-dashboard

Porting VaccinateCA to Django

2021-04-12T05:18:48+00:00

As I mentioned back in February, I've been working with the VaccinateCA project to try to bring the pandemic to an end a little earlier by helping gather as accurate a model as possible of where the Covid vaccine is available in California and how people can get it.

The key activity at VaccinateCA is calling places to check on their availability and eligibility criteria. Up until last night this was powered by a heavily customized Airtable instance, accompanied by a custom JavaScript app for the callers that communicated with the Airtable API via some Netlify functions.

Today, the flow is powered by a new custom Django backend, running on top of PostgreSQL.

The thing you should never do

Here's one that took me fifteen years to learn: "let's build a new thing and replace this" is hideously dangerous: 90% of the time you won't fully replace the old thing, and now you have two problems!
- Simon Willison (@simonw) June 29, 2019

Replacing an existing system with a from-scratch rewrite is risky. Replacing a system that is built on something as flexible as Airtable that is evolving on a daily basis is positively terrifying!

Airtable served us extremely well, but unfortunately there are hard limits to the number of rows Airtable can handle and we've already bounced up against them and had to archive some of our data. To keep scaling the organization we needed to migrate away.

We needed to build a matching relational database with a comprehensive, permission-controlled interface for editing it, plus APIs to drive our website and application. And we needed to do it using the most boring technology possible, so we could focus on solving problems directly rather than researching anything new.

It will never cease to surprise me that Django has attained boring technology status! VaccineCA sits firmly in Django's sweet-spot. So we used that to build our replacement.

The new Django-based system is called VIAL, for "Vaccine Information Archive and Library" - a neat Jesse Vincent bacronym.

We switched things over to VIAL last night, but we still have activity in Airtable as well. I expect we'll keep using Airtable for the lifetime of the organization - there are plenty of ad-hoc data projects for which it's a perfect fit.

The most important thing here is to have a trusted single point of truth for any piece of information. I'm not quite ready to declare victory on that point just yet, but hopefully once things settle down over the next few days.

Data synchronization patterns

The first challenge, before even writing any code, was how to get stuff out of Airtable. I built a tool for this a while ago called airtable-export, and it turned out the VaccinateCA team were using it already before I joined!

airtable-export was already running several times an hour, backing up the data in JSON format to a GitHub repository (a form of Git scraping). This gave us a detailed history of changes to the Airtable data, which occasionally proved extremely useful for answering questions about when a specific record was changed or deleted.

Having the data in a GitHub repository was also useful because it gave us somewhere to pull data from that wasn't governed by Airtable's rate limits.

I iterated through a number of different approaches for writing importers for the data.

Each Airtable table ended up as a single JSON file in our GitHub repository, containing an array of objects - those files got pretty big, topping out at about 80MB.

I started out with Django management commands, which could be passed a file or a URL. A neat thing about using GitHub for this is that you can use the "raw data" link to obtain a URL with a short-lived token, which grants access to that file. So I could create a short-term URL and paste it directly to my import tool.

I don't have a good pattern for running Django management commands on Google Cloud Run, so I started moving to API-based import scripts instead.

The pattern that ended up working best was to provide a /api/importRecords API endpoint which accepts a JSON array of items.

The API expects the input to have a unique primary key in each record - airtable_id in our case. It then uses Django's update_or_create() ORM method to create new records if they were missing, and update existing records otherwise.

One remaining challenge: posting 80MB of JSON to an API in one go would likely run into resource limits. I needed a way to break that input up into smaller batches.

I ended up building a new tool for this called json-post. It has an extremely specific use-case: it's for when you want to POST a big JSON array to an API endpoint but you want to first break it up into batches!

Here's how to break up the JSON in Reports.json into 50 item arrays and send them to that API as separate POSTs:

json-post Reports.json \                              
   "https://example.com/api/importReports" \
   --batch-size 50

Here are some more complex options. Here we need to pass an Authorization: Bearer XXXtokenXXX API key header, run the array in reverse, record our progress (the JSON responses from the API as newline-delimited JSON) to a log file, set a longer HTTP read timeout and filter for just specific items:

% json-post Reports.json \                              
   "https://example.com/api/importReports" \
  -h Authorization 'Bearer XXXtokenXXX' \
  --batch-size 50 \
  --reverse \
  --log /tmp/progress.txt \
  --http-read-timeout 20 \
  --filter 'item.get("is_soft_deleted")'

The --filter option proved particularly useful. As we kicked the tires on VIAL we would spot new bugs - things like the import script failing to correctly record the is_soft_deleted field we were using in Airtable. Being able to filter that input file with a command-line flag meant we could easily re-run the import just for a subset of reports that were affected by a particular bug.

--filter takes a Python expression that gets compiled into a function and passed item as the current item in the list. I borrowed the pattern from my sqlite-transform tool.

The value of API logs

VaccineCA's JavaScript caller application used to send data to Airtable via a Netlify function, which allowed additional authentication to be added built using Auth0.

Back in February, the team had the bright idea to log the API traffic to that function to a separate base in Airtable - including full request and response bodies.

This proved invaluable for debugging. It also meant that when I started building VIAL's alternative implementation of the "submit a call report" API I could replay historic API traffic that had been recorded in that table, giving me a powerful way to exercise the new API with real-world traffic.

This meant that when we turned on VIAL we could switch our existing JavaScript SPA over to talking to it using a fully tested clone of the existing Airtable-backed API.

VIAL implements this logging pattern again, this time using Django and PostgreSQL.

Given that the writable APIs will recieve in the low thousands of requests a day, keeping them in a database table works great. The table has grown to 90MB so far. I'm hoping that the pandemic will be over before we have to worry about logging capacity!

We're using PostgreSQL jsonb columns to store the incoming and returned JSON, via Django's JSONField. This means we can do in-depth API analysis using PostgreSQL's JSON SQL functions! Being able to examine returned JSON error messages or aggregate across incoming request bodies helped enormously when debugging problems with the API import scripts.

Storing the original JSON

Today, almost all of the data stored in VIAL originated in Airtable. One trick that has really helped build the system is that each of the tables that might contain imported data has both an airtable_id nullable column and an import_json JSON field.

Any time we import a record from Airtable, we record both the ID and the full, original Airtable JSON that we used for the import.

This is another powerful tool for debugging: we can view the original Airtable JSON directly in the Django admin interface for a record, and confirm that it matches the ORM fields that we set from that.

I came up with a simple pattern for Pretty-printing all read-only JSON in the Django admin that helps with this too.

Staying as flexible as possible

The thing that worried me most about replacing Airtable with Django was Airtable's incredible flexibility. In the organization's short life it has already solved so many problems by adding new columns in Airtable, or building new views.

Is it possible to switch to custom software without losing that huge cultural advantage?

This is the same reason it's so hard for custom software to compete with spreadsheets.

We've only just made the switch, so we won't know for a while how well we've done at handling this. I have a few mechanisms in place that I'm hoping will help.

The first is django-sql-dashboard. I wrote about this project in previous weeknotes here and here - the goal is to bring some of the ideas from Datasette into the Django/PostgreSQL world, by providing a read-only mechanism for constructing SQL queries, bookmarking and saving the results and outputting simple SQL-driven visualizations.

We have a lot of SQL knowledge at VaccinateCA, so my hope is that people with SQL will be able to solve their own problems, and people who don't know SQL yet will have no trouble finding someone who can help them.

In the boring technology model of things, django-sql-dashboard counts as the main innovation token I'm spending for this project. I'm optimistic that it will pay off.

I'm also leaning heavily on Django's migration system, with the aim of making database migrations common and boring, rather than their usual default of being rare and exciting. We're up to 77 migrations already, in a codebase that is just over two months old!

I think a culture that evolves the database schema quickly and with as little drama as possible is crucial to maintaining the agility that this kind of organization needs.

Aside from the Django Admin providing the editing interface, everything that comes into and goes out of VIAL happens through APIs. These are fully documented: I want people to be able to build against the APIs independently, especially for things like data import.

After seeing significant success with PostgreSQL JSON already, I'm considering using it to add even more API-driven flexbility to VIAL in the future. Allowing our client developers to start collecting a new piece of data from our volunteers in an existing JSON field, then migrating that into a separate column once it has proven its value, is very tempting indeed.

Open source tools we are using

An incomplete list of open source packages we are using for VIAL so far:

pydantic - as a validation layer for some of the API endpoints
social-auth-app-django - to integrate with Auth0
django-cors-headers
python-jose - for JWTs, which were already in use by our Airtable caller app
django-reversion and django-reversion-compare to provide a diffable, revertable history of some of our core models
django-admin-tools - which adds a handy customizable menu to the admin, good for linking to additional custom tools
django-migration-linter - to help avoid accidentally shipping migrations that could cause downtime during a deploy
pytest-django, time-machine and pytest-httpx for our unit tests
sentry-sdk, honeycomb-beeline and prometheus-client for error logging and observability

Want to help out?

VaccinateCA is hiring! It's an interesting gig, because the ultimate goal is to end the pandemic and put this non-profit permanently out of business. So if you want to help end things faster, get in touch.

VaccinateCA is hiring a handful of engineers to help scale our data ingestion and display by more than an order of magnitude.

If you'd like to register interest:https://t.co/BSvi40sW1M

Generalists welcome. Three subprojects; Python backend/pedestrian front-end JS.
- Patrick McKenzie (@patio11) April 7, 2021

TIL this week

Releases this week

json-post: 0.2 - (3 total releases) - 2021-04-11
Tool for posting JSON to an API, broken into pages
airtable-export: 0.7.1 - (10 total releases) - 2021-04-09
Export Airtable data to YAML, JSON or SQLite files on disk
django-sql-dashboard: 0.6a0 - (13 total releases) - 2021-04-09
Django app for building dashboards using raw SQL queries

Tags: django, django-admin, migrations, postgresql, weeknotes, airtable, vaccinate-ca, boring-technology

Weeknotes: Datasette and Git scraping at NICAR, VaccinateCA

2021-03-07T07:29:00+00:00

This week I virtually attended the NICAR data journalism conference and made a ton of progress on the Django backend for VaccinateCA (see last week).

NICAR 2021

NICAR stands for the National Institute for Computer Assisted Reporting - an acronym that reflects the age of the organization, which started teaching journalists data-driven reporting back in 1989, long before the term "data journalism" became commonplace.

This was my third NICAR and it's now firly established itself at the top of the list of my favourite conferences. Every year it attracts over 1,000 of the highest quality data nerds - from data journalism veterans who've been breaking stories for decades to journalists who are just getting started with data and want to start learning Python or polish up their skills with Excel.

I presented an hour long workshop on Datasette, which I'm planning to turn into the first official Datasette tutorial. I also got to pre-record a five minute lightning talk about Git scraping.

I published the video and notes for that yesterday. It really seemed to strike a nerve at the conference: I showed how you can set up a scheduled scraper using GitHub Actions with just a few lines of YAML configuration, and do so entirely through the GitHub web interface without even opening a text editor.

Pretty much every data journalist wants to run scrapers, and understands the friction involved in maintaining your own dedicated server and crontabs and storage and backups for running them. Being able to do this for free on GitHub's infrastructure drops that friction down to almost nothing.

The lightning talk lead to a last-minute GitHub Actions and Git scraping office hours session being added to the schedule, and I was delighted to have Ryan Murphy from the LA Times join that session to demonstrate the incredible things the LA Times have been doing with scrapers and GitHub Actions. You can see some of their scrapers in the datadesk/california-coronavirus-scrapers repo.

VaccinateCA

The race continues to build out a Django backend for the VaccinateCA project, to collect data on vaccine availability from people making calls on that organization's behalf.

The new backend is getting perilously close to launch. I'm leaning heavily on the Django admin for this, refreshing my knowledge of how to customize it with things like admin actions and custom filters.

It's been quite a while since I've done anything sophisticated with the Django admin and it has evolved a LOT. In the past I've advised people to drop the admin for custom view functions the moment they want to do anything out-of-the-ordinary - I don't think that advice holds any more. It's got really good over the years!

A very smart thing the team at VaccinateCA did a month ago is to start logging the full incoming POST bodies for every API request handled by their existing Netlify functions (which then write to Airtable).

This has given me an invaluable tool for testing out the new replacement API: I wrote a script which replays those API logs against my new implementation - allowing me to test that every one of several thousand previously recorded API requests will run without errors against my new code.

Since this is so valuable, I've written code that will log API requests to the new stack directly to the database. Normally I'd shy away from a database table for logging data like this, but the expected traffic is the low thousands of API requests a day - and a few thousand extra database rows per day is a tiny price to pay for having such a high level of visibility into how the API is being used.

(I'm also logging the API requests to PostgreSQL using Django's JSONField, which means I can analyze them in depth later on using PostgreSQL's JSON functionality!)

YouTube subtitles

I decided to add proper subtitles to my lightning talk video, and was delighted to learn that the YouTube subtitle editor pre-populates with an automatically generated transcript, which you can then edit in place to fix up spelling, grammar and remove the various "um" and "so" filler words.

This makes creating high quality captions extremely productive. I've also added them to the 17 minute Introduction to Datasette and sqlite-utils video that's embedded on the datasette.io homepage - editing the transcript for that only took about half an hour.

TIL this week

Tags: data-journalism, django-admin, youtube, datasette, weeknotes, git-scraping, vaccinate-ca, nicar

The simplest possible call queue

2021-03-06T17:00:00+00:00

Originally posted to my internal blog at VaccinateCA

Today I've been working on the queue calling mechanism for the new Django backend.

We want to do a trial run of the system as soon as possible, so I opted to create the simplest possible version of this that could work.

The Django app is going to work a little bit different from our existing Airtable system. The Airtable system defines a number of different filtered views for locations that we want to call, and leaves it to our caller app to load in rows from those views, select a random location from the options and "lock" that location so no-one else is given it to call for at least twenty minutes.

For the new system, we're going to switch to having an explicit "locations to call" list, which our staff can add locations to and manipulate. This will continue to evolve over time - there's already discussion of having multiple queues, and assigning different callers to different queues depending on their status and trainining.

For the moment though, the simplest thing is to have a database table with a list of locations that need to be called in it.

I implemented the first version of this in issue 70. I'm using Django's admin actions mechanism, which allows you to add bulk actions to the admin change list for any model.

The neat thing about admin actions is that you can apply them to multiple rows in the Django admin action interface: by clicking their checkboxes, by shift-clicking a range of checkboxes or with a "select all" option that can apply to the current page or every row that matches the current set of filters.

Combined with a rich set of filters this is a really powerful way of applying bulk actions to items in your database. The admin actions framework then provides a Django ORM queryset to your action code, which can act on it however it wants.

My first version of the code adds a new admin action for each of the "call request reasons" in the database (currently four, but more to come shortly).

Here's an animated demo showing how the new mechanism works:

Next up: implement the /api/requestCall API that the caller app uses to fetch the next location that the user should call! #54

The Django app on Google Cloud

Alex Vandiver has done a superb job getting the Django app in a state where it can run on our Google Cloud infrastructure. Most of today's commit activity was from Alex, and in addition to having continuous deployment running via Google Cloud Build and Cloud Run thanks to Alex we also have improvements like isort, python-dotenv and the (new to me) django-migration-linter.

Tags: django-admin, vaccinate-ca, vaccinate-ca-blog

Django admin customization, JSON in our PostgreSQL

2021-02-25T17:00:00+00:00

Originally posted to my internal blog at VaccinateCA

My progress slowed a bit today as I started digging into some things I'm less familiar with - but I've found some tricks that I think will help us out a lot.

Django admin customization

In the past I've used the Django admin mostly as a database debugging tool, on the basis that once you get deep into building out an interface that's more than just a very basic CMS you're better off rolling something from scratch.

Today my opinion changed. I think the Django admin may be the solution to a LOT of our problems, with very little extra customization.

I was looking at a simple feature request: show a summary of calls made by a specific reporter. It turns out adding a custom templated block of text to an existing Django admin "change item" page is trivial, using the pattern I wrote up here:

The implementation for this is tiny. We can use this pattern to add SO much depth to our admin pages - and Django's default permission system is robust enough that we can give users access to these pages without them being able to make edits.

I also added a column to the reporters table showing the number of calls each reporter has made, and made that column sortable! Implementation here.

Once we're fully on Django we'll be able to turn around features like this in very little time. You don't need to be a Django expert to build these either - a tiny bit of Python and HTML knowledge should be enough to productively modify this interface.

JSON in our PostgreSQL

My main goal for the day was to tighten up the call reports importer script I wrote yesterday.

My big breakthrough on this came after a long and super-valuable conversation with Nicholas Schiefer, who's been heavily involved in the growth of our Airtable schema ever since the project started.

Our Airtable data is complicated, because the way we write data to it has constantly evolved. Before the launch of help.vaccinateca the data was all entered through a custom Airtable app, and many of the fields we are capturing now weren't being captured just a few weeks ago.

Since writing a one-off importer that patches over all of these differences in a single go is virtually impossible, we decided to try an alternative track: my importer now saves the entire original Airtable JSON to a PostgreSQL JSON column (using Django 3.1's brand new JSONField).

I taught the Django Admin to pretty-print the JSON (implementation here):

The most obvious value of this is in debugging - it's much easier to look at a record now and compare it to the Airtable version.

More importantly: if we make a mistake in the importer code today and don't notice for six months, that's fine! We can re-backfill against the new lessons we have learned using the Airtable JSON data that we've already stored.

The cost? ~20,000 database records with a few extra KB of data stored against them. That's totally worth it.

Querying JSON

PostgreSQL has a bunch of features for querying into JSON fields which I haven't really explored before. I decided to try them out.

They're incredible. Here's a query that shows the callers who have made the most calls, based on extracting the {"Reported by": {"name": "NAME"}} nested field from that JSON column:

select
    jsonb_extract_path(
        airtable_json, 'Reported by', 'name'
    ) as name,
    count(*) as n
from call_report
group by name
order by n desc;

Even more exciting... here's a query that counts the keys that have been used in ALL of the JSON returned from Airtable:

SELECT
    jsonb_object_keys(airtable_json) AS key, count(*)
FROM call_report GROUP BY key;

And here's what it outputs:

JSON key	Times used
Affiliation (from Location)	21589
airtable_createdTime	21589
airtable_id	21589
Appointments by phone?	466
Appointment scheduling instructions	2526
auth0_reporter_id	1583
auth0_reporter_name	1583
auth0_reporter_roles	1583
Availability	21589
County (from Location)	21589
Do not call until	3116
external_reports_base_external_report_id	3139
Hour	21589
ID	21589
Internal Notes	18258
is_latest_report_for_location	21589
is_pending_review	18
Location	21589
location_id	21589
location_latest_eva_report_time	13443
location_latest_report_id	21589
location_latest_report_time	21354
Location Type (from Location)	21588
Name (from Location)	21589
Notes	3359
Number of Reports (from Location)	21589
parent_eva_report	958
parent_external_report	2410
Phone	82
Reported by	21589
report_id	21589
Report Type	21589
soft-dropped-column: Vaccines available?	15948
time	21589
tmp_eva_flips	21589
Vaccine demand	733
Vaccine demand notes	4

This is amazingly useful data for the importer script that I'm writing!

The query also executes in about 800ms against the cheapest PostgreSQL database server that Heroku offer - doing a deep full table scan against all 22,000 imported records.

Based on how powerful is, I'm now thinking that we should go all-in on JSON in our database. Imagine if every scraper we were running dumped its full scraped JSON data into PostgreSQL - we could join arbitrary scraped data against our other tables to figure out if there are any new locations.

My biggest concern about replacing Airtable is that we'll lose the amazing flexibility it's given us. I think JSON columns can help bridge that gap.

Call targeting: the most interesting problem

Another topic that came out of my conversation with Nicholas: I had not seen quite how ingenious the way call targeting works is. This is SUCH a smart usage of Airtable!

The short version: call targeting (the logic that decides which number a volunteer should be asked to call) is powered by Airtable views, with really clever application of Airtable's filters to help build up the call lists.

We need to maintain our ability to smartly target where the calls go, and ideally make it even better. This is going to be a really fun problem to solve!

Tags: django, django-admin, postgresql, vaccinate-ca, vaccinate-ca-blog

Importing data from Airtable into Django, plus a search engine for all our code

2021-02-24T17:00:00+00:00

Originally posted to my internal blog at VaccinateCA

I made a bunch of progress on the Django backend prototype-that-soon-won’t-be-a-prototype today.

Importing data from Airtable

My goal for the day was to get some real data into the prototype, imported from Airtable. I’ve now done that with the two most important tables: Locations (aka places that people can go to get the vaccine) and CallReports, created by volunteers making phone calls.

I achieved this by writing two new Django management commands:

./manage.py import_airtable_locations (code, test) populates the Django Locations model by pulling data from the Locations table in Airtable.
./manage.py import_airtable_reports (code, test) populates the Django CallReports model using data from the Reports table in Airtable.

Both of these importers follow the same pattern: you can point them directly at a Locations.json file you’ve already downloaded:

./manage.py import_airtable_locations \
  --json-file=Locations.json

Or you can use a GitHub personal access token to have them load the data directly from our airtable-data-backup repository, provided you have access to that.

./manage.py import_airtable_locations \
  --github-token=xxxx

I built the GitHub token mechanism to make it easy to run this command on a server, without having to mess around uplooading JSON files first. Since the prototype is running on Heroku I can pull a fresh import into it directly by running the following command:

% heroku run bash -a vaccinateca-preview
Running bash on ⬢ vaccinateca-preview... up, run.3026 (Hobby)
~ $ vaccinate/manage.py import_airtable_locations --github-token xxx
Skipping rec0xZ5EaKnnynfDa [name=CVS Pharmacy® & Drug Store at 25272 Marguerite Pkwy, Mission Viejo, CA 92692], reason=No latitude
Skipping rec7nHXCuSYRR61V0 [name=None], reason=No name
Skipping rec8Xk6kn4SvAKeEm [name=None], reason=No name
Skipping recCYZZRJCRlXykun [name=None], reason=No name
Skipping recJt0iQbqmglF0XL [name=Dignity Health (Woodland)], reason=No county

The command outputs a line for each item it fails to import. This turns out to be an extremely useful way to spot invalid data - locations with no name, latitude or county for example.

The script successfully imported 7038 locations, skipping just 17.

My call report importer needs a lot more work. It imported 21,029 call records, but skipped 962. More details in the GitHub issue thread.

Django Admin customization

I also made some tweaks to the Django admin. Here’s a screenshot of the call records list as it stands now.

And here's the locations table:

SSO with Auth0

This isn't fully working yet, but it's nearly there. I've integrated Auth0 SSO with the Django app, with the goal being that any staff member can use Auth0 to sign in to the Django Admin panel and view and modify the data there.

You can try it out on the https://vaccinateca-preview.herokuapp.com/ page - once it's fully working, you'll be able to sign in and then visit https://vaccinateca-preview.herokuapp.com/admin/ to interact with the data.

Here's the ongoing issue thread. I ended up creating a brand new role in Auth0 called "Vaccinate CA Staff" - I've figured out how to access those roles from the Django app when a user signs in, so I can then use membership of that role to control access to the admin panel.

Even more ambitious goals

Since the Django backend work is going at a very healthy pace, I'm extending my ambitions for it a bit.

The goal of the app is to replace Airtable as the point of truth for the data collected by our calling volunteers AND as the data source behind the public-facing https://www.vaccinateca.com/ site.

Once I've made some improvements to the schema informed by the data importing project, I think the next step will be to spin up just enough of an API endpoint that the calling app can start writing to Django in parallel to writing to Airtable.

This can be done inside the Netlify function used by the app. I'd like that function to continue writing to Airtable but also to write to the Django/PostgreSQL stack, wrapped in an error handler so failures there are ignored.

Then we can run the Django app as a silent partner to Airtable for a few days and compare the results gathered by the two, to gain confidence before switching over from one to the other.

Getting that running will be my goal for next week. If all goes well we may find we can make the full switch to the Django backend within a couple of weeks.

Regular expression code search across all of our repos

I love code search. I particularly like being able to search code with regular expressions, and then share links to those searches with other engineers.

The best code search tool I’ve ever used is ripgrep. A few months ago I built a simple web wrapper around ripgrep called datasette-ripgrep. This evening I deployed a copy of it against the source code from nine of our code repos:

You can access the search engine here (nicer URL coming soon):

https://vaccinateca-ripgrep-j7hipcg4aq-uc.a.run.app/

You'll need to sign in with GitHub - I'm protecting the site using datasette-auth-github configured to only allow in members of the CAVaccineInventor GitHub organization.

Once you've signed in, try this example search:

https://vaccinateca-ripgrep-j7hipcg4aq-uc.a.run.app/-/ripgrep?pattern=fetch%5C%28%7CfetchJsonFromEndpoint&glob=*.js

That runs a search for fetch\(|fetchJsonFromEndpoint across all of our *.js files, which shows us everywhere we are making an HTTP reuest using either fetch() or our own fetchJsonFromEndpoint() function.

The repo for the search engine is here.

Tags: django, django-admin, airtable, vaccinate-ca, vaccinate-ca-blog

Spinning up a new Django app to act as a backend for VaccinateCA

2021-02-23T17:00:00+00:00

Originally posted to my internal blog at VaccinateCA

My goal by the end of this week is to have a working proof of concept for a Django + PostgreSQL app that can replace Airtable as the principle backend for the https://www.vaccinateca.com/ site. This proof of concept will allow us to make a go or no-go decision and figure out what else needs to be implemented before we can start using it to track calls.

I'm calling it a "prototype" and a "proof of concept", but my career has taught me that prototypes often end up going into production - so I'm building it with that in mind.

Today I started building that app. The repo is currently https://github.com/CAVaccineInventory/django.vaccinate though we are likely to rename it soon - possibly to VIAL (for Vaccine Information Archive and Library) - Jesse is good at actually relevant codenames!

Here's what I have so far:

Proof of concept for Auth0 SSO - mostly from following their tutorial. You can try that out on the staging site homepage at https://vaccinateca-preview.herokuapp.com/
The very beginnings of a test suite - built using pytest-django. The tests run against PostgreSQL and I had to figure out how to do that inside GitHub Actions - here's my TIL.
The tests run in GitHub Actions! Continous Integration - should work against pull requests too.
... which means we can take the next step and go for Continuous Deployment. Every commit that passes the tests is now deployed instantly to the staging environment.
The staging environment itself is currently on Heroku, because they make it ridiculously easy to setup Continuous Deployment - it's literally a checkbox in their admin panel. I wrote about this a few years ago: How to set up world-class continuous deployment using free hosted tools. It's likely we'll move this to Google Cloud at some point since other VaccinateCA stuff is running there. I know how to run Continous Deployment using Google Cloud Run so that could be a good option here.
Errors now get logged to a new project in the VaccinateCA Sentry instance.

And the biggest thing: I've implemented a set of Django models against the SQL schema that has been coming together in this pull request. These are exposed in the Django Admin (just with default settings, no customization yet) in the staging environment.

You can try those out by visiting https://vaccinateca-preview.herokuapp.com/admin/ and signing in with username demo and password demo (this account will be deleted the second we have any real data in the prototype).

I wrote data migrations to insert states and insert counties - you can see the results in the admin here (counties) and here (states).

Some engineering principles

I've invested a lot of effort today in getting some fundamental things set up: a test suite, continuous integration, continuous deployment, and a detailed and up-to-date README.

The reason I'm investing that effort now is that I know from experience that these things are incredibly valuable, and very easy to implement at the start of a project... but become massively harder the longer you delay them. Adding comprehensive tests, documentation and CI to a six month old project can take weeks. Adding them to a project that is just starting takes just a few hours.

I also plan to lean very heavily on the Django migrations system.

I've worked at companies in the past where database migrations - any kind of schema change - are slow, rare and exciting. This has horrible knock-on effects: engineers will go to great lengths to avoid adding a column to a table, which can lead to a rapid acretion of technical debt.

I want schema changes to be quick, common and boring. Django's migration system - especially against PostgreSQL, which can execute schema changes inside transactions - is ideally suited to this. I want to start using it agressively as early as possible, to ensure we have a culture that says "yes" to schema changes and executes them promptly and frequently.

Next steps

I want to get some real data into the system! I'm going to lock down the security a bit more, then take some exports from Airtable, convert them to the new schema and load them into the prototype. This will allow us to really start kicking the tires on it.

I'm tracking all of the work on the Django app in the issues for that repository.

Tags: continuous-deployment, continuous-integration, django, django-admin, postgresql, vaccinate-ca, vaccinate-ca-blog

PostgreSQL full-text search in the Django Admin

2020-07-25T23:05:33+00:00

PostgreSQL full-text search in the Django Admin

Today I figured out how to use PostgreSQL full-text search in the Django admin for my blog, using the get_search_results method on a subclass of ModelAdmin.

Tags: django, django-admin, postgresql, search

Optimizing Django Admin Paginator

2018-11-06T18:17:42+00:00

Optimizing Django Admin Paginator

The Django admin paginator uses a count(*) to calculate the total number of rows, so it knows how many pages to display. This makes it unpleasantly slow over large datasets. Haki Benita has an ingenious solution: drop in a custom paginator which uses the PostgreSQL “SET LOCAL statement_timeout TO 200” statement first, then if a timeout error is raised returns 9999999999 as the count instead. This means small tables get accurate page counts and giant tables load display in the admin within a reasonable time period.

Tags: django, django-admin, postgresql, haki-benita

Django #8936: Add view (read-only) permission to admin (closed)

2018-05-17T13:40:35+00:00

Django #8936: Add view (read-only) permission to admin (closed)

Opened 10 years ago. Closed 15 hours ago. I apparently filed this issue during the first DjangoCon back in September 2008, when Adrian and Jacob mentioned on-stage that they would like to see a read-only permission for the Django Admin. Thanks to Olivier Dalang from Fiji and Petr Dlouhý from Prague it’s going to be a feature shipping in Django 2.1. Open source is a beautiful thing.

Tags: django, django-admin, djangocon, open-source

Django 2.0 released

2017-12-02T16:49:03+00:00

Django 2.0 released

The first version of Django to drop support for Python 2. I’ve been running the RC on my blog for the past 5 weeks and greatly enjoying the new mobile-optimized Django admin for posting links and quotations from my phone. The new simplified URL routing syntax (an optional alternative to regular expressions) is a very welcome improvement.

Tags: django, django-admin

pythondotorg/admin.py

2017-11-19T06:28:45+00:00

pythondotorg/admin.py

There are some neat tricks in the Django application that powers Python.org (built a few years ago by RevSys). Here’s how their admin app handles creator/last_modified_by user relationships.

Via Jeff Triplet

Tags: django, django-admin

Using “import refs” to iteratively import data into Django

2017-11-04T19:17:00+00:00

I’ve been writing a few scripts to backfill my blog with content I originally posted elsewhere. So far I’ve imported answers I posted on Quora (background), answers I posted on Ask MetaFilter and content I recovered from the Internet Archive.

I started out writing custom import scripts (like this Quora one), but I’ve now built a generalized mechanism for this which I thought was worth writing up.

Any of my content imports now take the form of a JSON document, which looks something like this:

[
  {
    "body": "<p><em>My answer to ...</em></p>",
    "tags": [
      "backpacks",
      "laptops",
      "style",
      "accessories",
      "bags"
    ],
    "title": "I need a new backpack",
    "datetime": "2005-01-16T14:08:00",
    "import_ref": "askmetafilter:14075",
    "type": "entry",
    "slug": "i-need-a-new-backpack"
  }
]

Two larger examples: the missing content I extracted from the Internet Archive, and the answers I scraped from Ask MetaFilter.

The type property can be set to entry, quotation or blogmark and specifies which type of content should be imported. The datetime, slug and tags fields are common across all three types - the other fields differ for each type.

The most interesting field here is import_ref. This is optional, but if provided forms a unique reference associated with that item of content. I then use that reference in a call Django’s update_or_create() method. This means I can run the same import multiple times - the first run will create objects, while subsequent runs update objects in place.

The end result is that I can incrementally improve the scrapers I am writing, re-importing the resulting JSON to update previously imported records in-place. In addition to hacking on my blog, I’ve been using this pattern for some API integrations at work recently and it’s worked out very well.

import_ref is defined on my models as a unique, nullable text field:

    import_ref = models.TextField(max_length=64, null=True, unique=True)

Since the Django admin doesn’t handle nullable fields well by default, I added import_ref to my readonly_fields property in my admin configuration to avoid accidentally setting it to a blank string when editing through the admin interface.

Here’s my completed import_blog_json management command.

My workflow for importing data is now pretty streamlined. I write the scrapers in a Juyter notebook and use that to generate a list of importable items as Python dictionaries. I run open('/tmp/items.json').write(json.dumps(items, indent=2)) to dump the items to a JSON file. Then I can run ./manage.py import_blog_json /tmp/items.json to import them into my local development environment - thanks to the import_ref I can do this as many times as I like until I’m pleased with the result.

Once it’s ready, I run !cat /tmp/blah.json | pbcopy in Jupyter to copy the JSON to my clipboard, then paste the JSON into a new GitHub Gist. I then copy the URL to that raw JSON and execute it against my production instance.

Heroku tip: running heroku run bash will start a bash prompt in a dyno hooked up to your application. You can then run ./manage.py ... commands against your production environment.

So… I just have to run heroku run bash followed by ./manage.py import_blog_json https://gist.github.com/path-to-json --tag_with=askmetafilter and the new content will be live on my site.

The tag_with option allows me to specify a tag to apply to all of that imported content, useful for checking that everything worked as expected.

Tags: django, django-admin, scraping, heroku, jupyter

What's new in Django 1.2 alpha 1

2010-01-07T19:31:50+00:00

What's new in Django 1.2 alpha 1

Multiple database support, improved CSRF prevention, a messages framework (similar to the Rails “flash” feature), model validation, custom e-mail backends, template caching for much faster handling of the include and extends tags, read only fields in the admin, a better if tag and more. Very exciting release.

Via Django Weblog

Tags: alpha, csrf, django, django-admin, python, releases

Django 1.1 release notes

2009-07-29T09:34:04+00:00

Django 1.1 release notes

Django 1.1 is out! Congratulations everyone who worked on this, it’s a fantastic release. New features include aggregate support in the ORM, proxy models, deferred fields and some really nice admin improvements. Oh, and the testing framework is now up to 10 times thanks to smart use of transactions.

Via Django | Weblog | Django 1.1 released

Tags: aggregates, django, django-admin, open-source, orm, python, releases

django-batchadmin

2008-09-15T10:46:50+00:00

django-batchadmin

Seriously classy reusable Django app that adds batch editing (multiple delete by default, with hooks to add your own custom batch actions) to the Django admin changelist screen, using best practice techniques of sub-classing ModelAdmin and hence requiring no patches to Django core itself.

Via Brian Beck

Tags: brian-beck, django, django-admin, djangobatchadmin, modeladmin, python

Django snippets: Orderable inlines using drag and drop with jQuery UI

2008-09-13T12:19:02+00:00

Django snippets: Orderable inlines using drag and drop with jQuery UI

Code example from my PyCon tutorial on customising the Django admin interface.

Tags: django, django-admin, dragndrop, jquery, pyconuk, pyconuk2008, python, snippets, sortable, speaking, my-talks, tutorials

Django: Security fix released

2008-09-03T00:14:00+00:00

Django: Security fix released

The Django admin used to save partially-submitted forms if your session expired, and continue the submission when you logged in. It turns out that’s actually an unblockable CSRF exploit and is hence broken as designed, so it’s now been removed. Thanks Ed Eliot and other GCap colleagues for helping me flesh out the potential attack.

Tags: csrf, django, django-admin, ed-eliot, exploit, gcap, security

Changeset 8266 - Added ModelAdmin.save_model() and ModelAdmin.save_formset() methods

2008-08-10T13:17:27+00:00

Changeset 8266 - Added ModelAdmin.save_model() and ModelAdmin.save_formset() methods

One of those small changes that opens up enormous possibilities—it’s now incredibly easy to customise exactly how a model is saved in the Django admin interface by over-riding the save_model method.

Tags: admin, django, django-admin, python

Django 1.0 alpha release notes

2008-07-22T06:04:29+00:00

Django 1.0 alpha release notes

The big features are newforms-admin, unicode everywhere, the queryset-refactor ORM improvements and auto-escaping in templates.

Tags: alpha, autoescaping, django, django-admin, newformsadmin, orm, python, querysetrefactor, unicode

newforms-admin branch has been merged into trunk

2008-07-20T23:17:11+00:00

newforms-admin branch has been merged into trunk

Congrats to Brian Rosner for the merge. django.newforms has been renamed to django.forms as well—1.0 grows ever closer.

Tags: brian-rosner, django, django-admin, forms, merge, newformsadmin

Django: security fix released

2008-05-14T07:49:22+00:00

Django: security fix released

XSS hole in the Admin application’s login page—updates and patches are available for trunk, 0.96, 0.95 and 0.91.

Tags: django, django-admin, security, xss

Django admin OmniGraffle stencil

2008-05-13T17:58:04+00:00

Django admin OmniGraffle stencil

Alex Lee put together a beautiful stencil for OmniGraffle containing all of the common UI elements seen in the Django admin interface, as a tool for wireframing.

Tags: alex-lee, django, django-admin, omnigraffle

Filtering foreign key choices in newforms-admin

2008-01-06T20:31:52+00:00

Filtering foreign key choices in newforms-admin

A nice introduction to the Django newform-admin branch, including an example of how to easily implement row-level permissions.

Tags: christian-joergensen, django, django-admin, newforms, newformsadmin, python

This Week in Django podcast

2008-01-01T10:44:01+00:00

This Week in Django podcast

Michael Trier’s been doing a really fantastic job putting together a Django podcast. The most recent episode (number 4) includes an update on the newforms-admin branch and a couple of handy tips.

Tags: django, django-admin, michael-trier, podcasts, python, thisweekindjango

Django on the iPhone

2007-08-19T07:58:04+00:00

Django on the iPhone

Jacob got it working. The next image in his photostream shows the Django admin application querying his phone’s local database of calls.

Tags: apple, django, django-admin, iphone, jacob-kaplan-moss, python