Simon Willison's Weblog: djangocon

Themes from DjangoCon US 2024

2024-09-27T23:36:02+00:00

I just arrived home from a trip to Durham, North Carolina for DjangoCon US 2024. I’ve already written about my talk where I announced a new plugin system for Django; here are my notes on some of the other themes that resonated with me during the conference.

Growing the Django Software Foundation (DSF)

Jacob Kaplan-Moss gave my favorite talk of the conference, asking what the Django Software Foundation could do if it quadrupled its annual income from $250,000 to $1 million dollars, and then mapping out a convincing path to get there.

I really liked this diagram Jacob provided summarizing the foundation’s current income and expenditures. It’s pretty cool that $90,000 of annual income comes from individual donors, over a third of the total since corporate donors provide $160,000.

Top priority would be hiring an Executive Director for the foundation, which is currently lead entirely by an elected, volunteer board. I’ve seen how useful a professional ED is from my own experiences on the Python Software Foundation board.

Having someone working full time on the foundation outside of our current fellows - who have more than enough on their plates already - would enable the foundation to both take on more ambitious goals and also raise more money with which to tackle them.

A line that Jacob used repeatedly in his talk about funding the foundation was this: if you or your organization wouldn’t want to sponsor Django, he’d love to know why that is - understanding those blockers right now is almost as valuable as receiving actual cash. You can reach out to him at jacob at djangoproject.com.

Could we fund a Django LTS accessibility audit?

Django fellows and the Django Accessibility Team have been focusing significant effort on the accessibility of the Django admin. I found this very inspiring, and in combination with the talk of more funding for the foundation it put an idea in my head: what if every Django LTS release (once every two years) was backed by a full, professional accessibility audit, run by an agency staffed with developers who use screen readers?

Imagine how much impact it would have if the default Django admin interface had excellent, documented accessibility out of the box. It could improve things for hundreds of thousands of users, and set an excellent precedent for projects (and foundations) in the wider open source community.

This also feels to me like something that should be inherently attractive to sponsors. A lot of agencies use Django for government work, where accessibility is a requirement with teeth. Would one of those agencies like to be the “accessibility sponsor” for a major Django release?

Django fellows continue to provide outstanding value

The DSF’s fellowship program remains one of the most impactful initiatives I’ve seen anywhere for ensuring the ongoing sustainability of a community-driven open source project.

Both of the current fellows, Natalia Bidart and Sarah Boyce, were in attendance and gave talks. It was great getting to meet them in person.

If you’re not familiar with the program, the fellows are contractors who are paid by the DSF to keep the Django project ticking over - handling many of the somewhat less glamorous tasks of responsible open source maintenance such as ticket triage, release management, security fixes and code review.

The fellows program is in its tenth year, and is a key reason that Django continues to release new versions on a regular schedule despite having no single corporate parent with paid developers.

Unsurprisingly there is always more work than fellow capacity, hence Jacob’s desire to further expand the existing program.

The fellows program launched with a policy that fellows should not work on new feature development. I believe this was partly related to interpretation of IRS nonprofit guidelines which have since been reconsidered, and there is a growing consensus now that this policy should be dropped.

Django needs feature champions

Django has a well deserved reputation for stability, reliability and a dependable release process. It has less of a reputation for constantly turning out ground-breaking new features.

Long-time Django contributors who I talked to all had a similar position on this: the challenge here is that big new features need dedicated champions to both lead design and development on them and to push them through to completion.

The pool of community members who are both willing and able to take on these larger projects is currently too small.

There are a number of ways we could address this - most notably through investing financial resources in sponsoring feature development. This has worked well for Django in the past - Django’s migrations work was funded by a Kickstarter campaign back in 2013.

The Django Software Foundation will shortly be announcing details of elections for both the DSF board and the Django Steering Council. These are extremely influential positions for people who want to help solve some of these larger problems.

htmx fits Django really well

htmx is an incredibly good fit for the uncodified Django community philosophy of building for the web. It came up in multiple talks. It feels like it may be a solution that the Django community has been seeking for years, as a very compelling alternative to writing everything in SPA JavaScript and using Django purely as a backend via something like Django REST Framework.

I've been slightly resistant to embracing htmx myself purely because it's such a critical dependency and in the past I wasn't convinced of its staying power. It's now mature, stable and widely-enough used that I'm ready to consider it for my own long-term projects.

Django Ninja has positive buzz

I haven’t paid much attention to Django Ninja but it had a lot of very positive buzz at the conference as well, as a tool for quickly building full-featured, performative API endpoints (thanks to Rust-backed Pydantic for serialization) with interactive API docs powered by OpenAPI.

I respect Django REST Framework a lot, but my personal programming style leans away from Class Based Views, which it uses quite a bit. Django Ninja looks like it might fit my function-view biases better.

I wrote about Richard Terry’s excellent nanodjango single-file Django application tool the other day - Django Ninja comes baked into that project as well.

The three platinum sponsors for DjangoCon this year were REVSYS, Caktus Group and Valkey. Valkey were a late and somewhat surprising addition to the sponsorship lineup.

Valkey is the Linux Foundation backed fork of Redis, created in response to Redis ditching their Open Source license (which I took quite personally, having contributed my own free effort to promoting and improving Redis in the past).

Aside from expressing thanks to them, I usually don’t pay sponsors that much attention. For some reason this one hit differently - the fact that Valkey were ready to step in as a major sponsor despite being only a few months old has caused me to take that project a whole lot more seriously than I did before. I’ll certainly consider them next time I come across a Redis-shaped problem.

Durham has a world-class collection of tubas

My favorite category of Niche Museum is one that's available by appointment only where the person who collected everything is available to show you around.

I always check Atlas Obscura any time I visit a new city, and this time I was delighted to learn about The Vincent and Ethel Simonetti Historic Tuba Collection!

I promoted it in the DjangoCon US #outings Slack channel and got together a group of five conference attendees for a visit on Thursday, shortly before my flight.

It was peak Niche Museum. I’ve posted photos and notes over on my Niche Museums website, the first new article there in quite a while.

Tags: accessibility, conferences, django, djangocon, jacob-kaplan-moss, python, redis, dsf, pydantic, htmx

Announcing our DjangoCon US 2024 Talks!

2024-07-17T03:20:57+00:00

Announcing our DjangoCon US 2024 Talks!

I'm speaking at DjangoCon in Durham, NC in September.

My accepted talk title was How to design and implement extensible software with plugins. Here's my abstract:

Plugins offer a powerful way to extend software packages. Tools that support a plugin architecture include WordPress, Jupyter, VS Code and pytest - each of which benefits from an enormous array of plugins adding all kinds of new features and expanded capabilities.

Adding plugin support to an open source project can greatly reduce the friction involved in attracting new contributors. Users can work independently and even package and publish their work without needing to directly coordinate with the project's core maintainers. As a maintainer this means you can wake up one morning and your software grew new features without you even having to review a pull request!

There's one catch: information on how to design and implement plugin support for a project is scarce.

I now have three major open source projects that support plugins, with over 200 plugins published across those projects. I'll talk about everything I've learned along the way: when and how to use plugins, how to design plugin hooks and how to ensure your plugin authors have as good an experience as possible.

I'm going to be talking about what I've learned integrating Pluggy with Datasette, LLM and sqlite-utils. I've been looking for an excuse to turn this knowledge into a talk for ages, very excited to get to do it at DjangoCon!

Tags: django, djangocon, plugins, python, speaking, datasette, sqlite-utils, llm

Coping strategies for the serial project hoarder

2022-11-26T15:47:02+00:00

I gave a talk at DjangoCon US 2022 in San Diego last month about productivity on personal projects, titled "Massively increase your productivity on personal projects with comprehensive documentation and automated tests".

The alternative title for the talk was Coping strategies for the serial project hoarder.

I'm maintaining a lot of different projects at the moment. Somewhat unintuitively, the way I'm handling this is by scaling down techniques that I've seen working for large engineering teams spread out across multiple continents.

The key trick is to ensure that every project has comprehensive documentation and automated tests. This scales my productivity horizontally, by freeing me up from needing to remember all of the details of all of the different projects I'm working on at the same time.

You can watch the talk on YouTube (25 minutes). Alternatively, I've included a detailed annotated version of the slides and notes below.

This was the title I originally submitted to the conference. But I realized a better title was probably...

Coping strategies for the serial project hoarder

This video is a neat representation of my approach to personal projects: I always have a few on the go, but I can never resist the temptation to add even more.

My PyPI profile (which is only five years old) lists 185 Python packages that I've released. Technically I'm actively maintaining all of them, in that if someone reports a bug I'll push out a fix. Many of them receive new releases at least once a year.

Aside: I took this screenshot using shot-scraper with a little bit of extra JavaScript to hide a notification bar at the top of the page:

shot-scraper 'https://pypi.org/user/simonw/' \
--javascript "
    document.body.style.paddingTop = 0;
    document.querySelector(
        '#sticky-notifications'
    ).style.display = 'none';
  " --height 1000

How can one individual maintain 185 projects?

Surprisingly, I'm using techniques that I've scaled down from working at a company with hundreds of engineers.

I spent seven years at Eventbrite, during which time the engineering team grew to span three different continents. We had major engineering centers in San Francisco, Nashville, Mendoza in Argentina and Madrid in Spain.

Consider timezones: engineers in Madrid and engineers in San Francisco had almost no overlap in their working hours. Good asynchronous communication was essential.

Over time, I noticed that the teams that were most effective at this scale were the teams that had a strong culture of documentation and automated testing.

As I started to work on my own array of smaller personal projects, I found that the same discipline that worked for large teams somehow sped me up, when intuitively I would have expected it to slow me down.

I wrote an extended description of this in The Perfect Commit.

I've started structuring the majority of my work in terms of what I think of as "the perfect commit" - a commit that combines implementation, tests, documentation and a link to an issue thread.

As software engineers, it's important to note that our job generally isn't to write new software: it's to make changes to existing software.

As such, the commit is our unit of work. It's worth us paying attention to how we can make our commits as useful as possible.

Here's a recent example from one of my projects, Datasette.

It's a single commit which bundles together the implementation, some related documentation improvements and the tests that show it works. And it links back to an issue thread from the commit message.

Let's talk about each component in turn.

There's not much to be said about the implementation: your commit should change something!

It should only change one thing, but what that actually means varies on a case by case basis.

It should be a single change that can be documented, tested and explained independently of other changes.

(Being able to cleanly revert it is a useful property too.)

The goals of the tests that accompany a commit are to prove that the new implementation works.

If you apply the implementation the new tests should pass. If you revert it the tests should fail.

I often use git stash to try this out.

If you tell people they need to write tests for every single change they'll often push back that this is too much of a burden, and will harm their productivity.

But I find that the incremental cost of adding a test to an existing test suite keeps getting lower over time.

The hard bit of testing is getting a testing framework setup in the first place - with a test runner, and fixtures, and objects under test and suchlike.

Once that's in place, adding new tests becomes really easy.

So my personal rule is that every new project starts with a test. It doesn't really matter what that test does - what matters is that you can run pytest to run the tests, and you have an obvious place to start building more of them.

I maintain three cookiecutter templates to help with this, for the three kinds of projects I most frequently create:

simonw/python-lib for Python libraries
simonw/click-app for command line tools
simonw/datasette-plugin for Datasette plugins

Each of these templates creates a project with a setup.py file, a README, a test suite and GitHub Actions workflows to run those tests and ship tagged releases to PyPI.

I have a trick for running cookiecutter as part of creating a brand new repository on GitHub. I described that in Dynamic content for GitHub repository templates using cookiecutter and GitHub Actions.

This is a hill that I will die on: your documentation must live in the same repository as your code!

You often see projects keep their documentation somewhere else, like in a wiki.

Inevitably it goes out of date. And my experience is that if your documentation is out of date people will lose trust in it, which means they'll stop reading it and stop contributing to it.

The gold standard of documentation has to be that it's reliably up to date with the code.

The only way you can do that is if the documentation and code are in the same repository.

This gives you versioned snapshots of the documentation that exactly match the code at that time.

More importantly, it means you can enforce it through code review. You can say in a PR "this is great, but don't forget to update this paragraph on this page of the documentation to reflect the change you're making".

If you do this you can finally get documentation that people learn to trust over time.

Another trick I like to use is something I call documentation unit tests.

The idea here is to use unit tests to enforce that concepts introspected from your code are at least mentioned in your documentation.

I wrote more about that in Documentation unit tests.

Here's an example. Datasette has a test that scans through each of the Datasette plugin hooks and checks that there is a heading for each one in the documentation.

The test itself is pretty simple: it uses pytest parametrization to look through every introspected plugin hook name, and for each one checks that it has a matching heading in the documentation.

The final component of my perfect commit is this: every commit must link to an issue thread.

I'll usually have these open in advance but sometimes I'll open an issue thread just so I can close it with a commit a few seconds later!

Here's the issue for the commit I showed earlier. It has 11 comments, and every single one of those comments is by me.

I have literally thousands of issues on GitHub that look like this: issue threads that are effectively me talking to myself about the changes that I'm making.

It turns out this a fantastic form of additional documentation.

What goes in an issue?

Background: the reasons for the change. In six months time you'll want to know why you did this.
State of play before-hand: embed existing code, link to existing docs. I like to start my issues with "I'm going to change this code right here" - that way if I come back the next day I don't have to repeat that little piece of research.
Links to things! Documentation, inspiration, clues found on StackOverflow. The idea is to capture all of the loose information floating around that topic.
Code snippets illustrating potential designs and false-starts.
Decisions. What did you consider? What did you decide? As programmers we make decisions constantly, all day, about everything. That work doesn't have to be invisible. Writing them down also avoids having to re-litigate them several months later when you've forgotten your original reasoning.
Screenshots - of everything! Animated screenshots even better. I even take screenshots of things like the AWS console to remind me what I did there.
When you close it: a link to the updated documentation and demo

The reason I love issues is that they're a form of documentation that I think of as temporal documentation.

Regular documentation comes with a big commitment: you have to keep it up to date in the future.

Issue comments skip that commitment entirely. They're displayed with a timestamp, in the context of the work you were doing at the time.

No-one will be upset or confused if you fail to keep them updated to match future changes.

So it's a commitment-free form of documentation, which I for one find incredibly liberating.

I think of this approach as issue driven development.

Everything you are doing is issue-first, and from that you drive the rest of the development process.

This is how it relates back to maintaining 185 projects at the same time.

With issue driven development you don't have to remember anything about any of these projects at all.

I've had issues where I did a bunch of design work in issue comments, then dropped it, then came back 12 months later and implemented that design - without having to rethink it.

I've had projects where I forgot that the project existed entirely! But I've found it again, and there's been an open issue, and I've been able to pick up work again.

It's a way of working where you treat it like every project is going to be maintained by someone else, and it's the classic cliche here that the somebody else is you in the future.

It horizontally scales you and lets you tackle way more interesting problems.

Programmers always complain when you interrupt them - there's this idea of "flow state" and that interrupting a programmer for a moment costs them half an hour in getting back up to speed.

This fixes that! It's much easier to get back to what you are doing if you have an issue thread that records where you've got to.

Issue driven development is my key productivity hack for taking on much more ambitious projects in much larger quantities.

Another way to think about this is to compare it to laboratory notebooks.

Here's a page from one by Leonardo da Vinci.

Great scientists and great engineers have always kept detailed notes.

We can use GitHub issues as a really quick and easy way to do the same thing!

Another thing I like to use these for is deep research tasks.

Here's an example, from when I was trying to figure out how to run my Python web application in an AWS Lambda function:

Figure out how to deploy Datasette to AWS Lambda using function URLs and Mangum

This took me 65 comments over the course of a few days... but by the end of that thread I'd figured out how to do it!

Here's the follow-up, with another 77 comments, in which I figure out how to serve an AWS Lambda function with a Function URL from a custom subdomain.

I will never have to figure this out ever again! That's a huge win.

https://github.com/simonw/public-notes is a public repository where I keep some of these issue threads, transferred from my private notes repos using this trick.

The last thing I want to encourage you to do is this: if you do project, tell people what it is you did!

This counts for both personal and work projects. It's so easy to skip this step.

Once you've shipped a feature or built a project, it's so tempting to skip the step of spending half an hour or more writing about the work you have done.

But you are missing out on so much of the value of your work if you don't give other people a chance to understand what you did.

I wrote more about this here: What to blog about.

For projects with releases, release notes are a really good way to do this.

I like using GitHub releases for this - they're quick and easy to write, and I have automation setup for my projects such that creating release notes in GitHub triggers a build and release to PyPI.

I've done over 1,000 releases in this way. Having them automated is crucial, and having automation makes it really easy to ship releases more often.

Please make sure your release notes have dates on them. I need to know when your change went out, because if it's only a week old it's unlikely people will have upgraded to it yet, whereas a change from five years ago is probably safe to depend on.

I wrote more about writing better release notes here.

This is a mental trick which works really well for me. "No project of mine is finished until I've told people about it in some way" is a really useful habit to form.

Twitter threads are (or were) a great low-effort way to write about a project. Build a quick thread with some links and images, and maybe even a video.

Get a little unit about your project out into the world, and then you can stop thinking about it.

(I'm trying to do this on Mastodon now instead.)

Even better: get a blog! Having your own corner of the internet to write about the work that you are doing is a small investment that will pay off many times over.

("Nobody blogs anymore" I said in the talk... Phil Gyford disagrees with that meme so much that he launched a new blog directory to show how wrong it is.)

The enemy of projects, especially personal projects, is guilt.

The more projects you have, the more guilty you feel about working on any one of them - because you're not working on the others, and those projects haven't yet achieved their goals.

You have to overcome guilt if you're going to work on 185 projects at once!

This is the most important tip: avoid side projects with user accounts.

If you build something that people can sign into, that's not a side-project, it's an unpaid job. It's a very big responsibility, avoid at all costs!

Almost all of my projects right now are open source things that people can run on their own machines, because that's about as far away from user accounts as I can get.

I still have a responsibility for shipping security updates and things like that, but at least I'm not holding onto other people's data for them.

I feel like if your project is tested and documented, you have nothing to feel guilty about.

You have put a thing out into the world, and it has tests to show that it works, and it has documentation that explains what it is.

This means I can step back and say that it's OK for me to work on other things. That thing there is a unit that makes sense to people.

That's what I tell myself anyway! It's OK to have 185 projects provided they all have documentation and they all have tests.

Do that and the guilt just disappears. You can live guilt free!

You can follow me on Mastodon at @simon@simonwillison.net or on GitHub at github.com/simonw. Or subscribe to my blog at simonwillison.net!

From the Q&A:

You've tweeted about using GitHub Projects. Could you talk about that?
- GitHub Projects V2 is the perfect TODO list for me, because it lets me bring together issues from different repositories. I use a project called "Everything" on a daily basis (it's my browser default window) - I add issues to it that I plan to work on, including personal TODO list items as well as issues from my various public and private repositories. It's kind of like a cross between Trello and Airtable and I absolutely love it.
How did you move notes from the private to the public repo?
- GitHub doesn't let you do this. But there's a trick I use involving a temp repo which I switch between public and private to help transfer notes. More in this TIL.
Question about the perfect commit: do you commit your failing tests?
- I don't: I try to keep the commits that land on my main branch always passing. I'll sometimes write the failing test before the implementation and then commit them together. For larger projects I'll work in a branch and then squash-merge the final result into a perfect commit to main later on.

Tags: djangocon, documentation, productivity, my-talks, testing, annotated-talks, github-issues

Weeknotes: DjangoCon, SQLite in Django, datasette-gunicorn

2022-10-23T19:58:00+00:00

I spent most of this week at DjangoCon in San Diego - my first outside-of-the-Bay-Area conference since the before-times.

It was a most excellent event. I spent a lot of time in the corridor track - actually the sitting-outside-in-the-sunshine track, catching up with people I haven't seen in several years.

I gave a talk titled "Massively increase your productivity on personal projects with comprehensive documentation and automated tests", with the alternative title "Coping strategies for the serial project hoarder". I'll do a full write-up of this once the video is made available in a few weeks time, but in the meantime the talk materials can be found here:

Supporting notes and links
Slides on Speaker Deck
Video for paying DjangoCon attendees (public video coming soon)

I also gave a lightning talk about AI and magic, which was effectively the five minute oral version of my recent blog post Is the AI spell-casting metaphor harmful or helpful?

Benchmarking SQLite in Django

I also hung around for the first day of the DjangoCon sprints.

For over a decade, the Django documentation has warned against using SQLite in production - recommending PostgreSQL or MySQL instead.

I asked Django Fellow Carlton Gibson what it would take to update that advice for 2022. He suggested that what we really needed was a solid idea for how well modern SQLite performs with Django, against a variety of different settings.

So I spent some time running benchmarks, using my new django_sqlite_benchmark repository.

You can follow the full details of my experiments in these issues:

#2: Locust test to exercise /counter/xxx endpoint which runs benchmarks against a simple Django view that increments a counter stored in a SQLite table
#3: Load test for larger writes runs a benchmark using a script that inserts larger JSON objects into a database table. I also tried this against PostgreSQL, getting very similar numbers to SQLite.
#4: Benchmark endpoint that doesn't interact with database benchmarks a simple "hello world" view that doesn't use SQLite at all - as a baseline for comparison

I used Locust for all of these tests, and wrote up a TIL about using it as well.

Here's the TLDR version of the results: SQLite in its default "journal" mode starts returning "database locked" errors pretty quickly as the write load increases. But... if you switch to "wal" mode (here's how) those errors straight up vanish!

I was expecting WAL mode to improve things, but I thought I'd still be able to hit errors even with it enabled. No - it turns out that, at least for the amount of traffic I could generate on may laptop, WAL mode proved easily capable of handling the load.

Even without WAL mode, bumping the SQLite "timeout" option up to 20s solved most of the errors.

Even more interestingly: I tried using Gunicorn (and Uvicorn) to run multiple Django workers at once. I was certain this would lead to problems, as SQLite isn't designed to handle writes from multiple processes at once... or so I thought. It turned out SQLite's use of file locking meant everything worked far better than I expected - and upping the number of worker processes from 1 to 4 resulted in approximately a 4x increase in throughput.

I shouldn't be surprised by this, if only because every time I've tried to push SQLite in a new direction it's impressed me with how much more capable it is than I expected.

But still, these results are very exciting. This problem still needs more thorough testing and more eyes than just mine, but I think this indicates that SQLite should absolutely be considered a viable option for running Django in production in 2022.

datasette-gunicorn

Datasette has always run as a single process. It uses Uvicorn to serve requests, but it hard-codes Uvicorn to a single worker (here).

Based on my experiments with SQLite and Django - in particular how running multiple worker processes gave me an increase in how much traffic I could handle - I decided to try the same thing with Datasette itself.

Gunicorn remains one of the most well regarded options for deploying Python web applications. It acts as a process monitor, balancing requests between different workers and restarting anything that fails with an error.

I decided to experiment with this through the medium of a Datasette plugin. So I built datasette-gunicorn, a plugin that adds an extra command to Datasette that lets you start it like this:

datasette gunicorn my.db --workers 4

It takes most of the same arguments as Datasette's regular datasette serve command, plus that new -w/--workers option for setting the number of workers.

Initial benchmarks were very positive: 21 requests a second with a single worker, increasing to 75 requests/second with four! Not bad for an initial experiment. I also tested it serving a static page through Datasette and got up to over 500 requests a second with a warning that Locust needed to be moved to a separate machine for a full load test.

In writing the plugin I had to figure out how to build a new command that mostly copied parameters from the existing datasette serve Click command - I wrote a TIL about how I ended up doing that.

shot-scraper 1.0

Also this week: I released shot-scraper 1.0.

Despite the exciting version number this actually only has two small new features. Here's the full changelog:

New shot-scraper html URL command (documented here) for outputting the final HTML of a page, after JavaScript has been executed. #96

shot-scraper javascript has a new -r/--raw option for outputting the result of the JavaScript expression as a raw string rather than JSON encoded (shot-scraper javascript documentation). #95

Tutorial: Automating screenshots for the Datasette documentation using shot-scraper.

I bumped it to 1.0 because shot-scraper is mature enough now that I'm ready to commit to not breaking existing features (at least without shipping a 2.0, which I hope to avoid for as long as possible).

I'm always trying to get more brave when it comes to stamping a 1.0 release on my main projects.

(I really, really need to get Datasette 1.0 shipped soon.)

Releases this week

datasette-gunicorn: 0.1 - 2022-10-22
Plugin for running Datasette using Gunicorn
shot-scraper: 1.0 - (23 releases total) - 2022-10-15
A command-line utility for taking automated screenshots of websites
asgi-gzip: 0.2 - (2 releases total) - 2022-10-13
gzip middleware for ASGI applications, extracted from Starlette

TIL this week

Tags: django, djangocon, projects, sqlite, my-talks, gunicorn, datasette, weeknotes, shot-scraper, carlton-gibson

How I moderated the State of Django panel at DjangoCon US.

2018-10-22T19:48:53+00:00

On Wednesday last week I moderated the State of Django panel as the closing session for DjangoCon US 2018.

I think it went well, so I’m writing some notes on exactly how we did it. In my experience it’s worth doing this for things like public speaking: in six months time I might moderate another panel and I’ll be desperately trying to remember what went right last time.

Panels are hard. Bad panels are way too common, to the point that some good conferences actively avoid having panels at all.

In my opinion, a good panel has a number of important attributes:

The panel needs a coherent theme. It shouldn’t just be several independent speakers that happen to be sharing the same time slot.
Panels need to be balanced. Having just one or two of the speakers monopolize the conversation is bad for the audience and bad for the panelists themselves.
The moderator is there to facilitate the conversation, not to be the center of attention. I love public speaking so I feel the need to be particularly cautious here.
Panelists need to have diverse perspectives on the topics under discussion. A panel where everyone agrees with each other and makes the same points is a very boring panel indeed.

I originally pitched the panel to the DjangoCon US organizing committee as a “State of Django” conversation where core maintainers would talk about the current state of the project.

They countered with a much better idea: a panel that encompassed both the state of the Django framework and the community and ecosystem that it exists within. Since DjangoCon is primarily about bringing that community together this was a much better fit for the conference, and would make for a much more interesting and relevant discussion.

I worked together with the conference organizers to find our panelists. Nicholle James in particular was the driving force behind assembling the panelists and ensuring everything was in place for the panel to succeed.

We ended up with a panel representing a comprehensive cross-section of the organizations that make the Django community work:

Andrew Godwin, representing Django Core
Anna Makarudze, representing the DSF, Django Girls and Python Africa
Frank Wiles, President of the DSF
Jeff Triplett, President of DEFNA, member of the Board of Directors for the PSF
Josue Balandrano Coronel, representing DEFNA
Katherine Michel, representing DEFNA and DjangoCon US Website Chair
Kojo Idrissa, DEFNA North American Ambassador
Rachell Calhoun, representing Django Girls and PyLadies

As it was the closing session for the conference, I wanted the panel to both celebrate the progress of the community and project and to explore what we need to do next: what should we be working on to improve Django and it’s community in the future?

I had some initial thoughts on topics, but since the panel was scheduled for the last session of the conference I decided to spend the conference itself firming up the topics that would be discussed. This was a really good call: we got to create an agenda for the panel that was almost entirely informed by the other conference sessions combined with hot topics from the halfway track. We also asked conference attendees for their suggestions via an online form, and used those suggestions to further inform the topics that were discussed.

I made sure to have a 10-15 minute conversation one-on-one with each of the panelists during the conference. We then got together for an hour at lunch before the panel to sync up with the topics and themes we would be discussing.

Topic and themes

Our pre-panel conversations highlighted a powerful theme for the panel itself, which I ended up summarizing as “What can the Django project learn from the Django community?” This formed the framework for the other themes of the panel.

The themes themselves were:

Diversity and education - inspired by the work of Django Girls
Events and international focus, lead by DEFNA
Django governance: In particular James Bennett’s proposal to split up core
Big feature ideas for Django (guided by Andrew Godwin’s proposed Django async roadmap)
Money: who has it, and who needs it - understanding the role of the DSF.

One of the hardest parts for me was figuring out the order in which we would tackle these themes. I ended up settling on the above order about half an hour before the panel started.

Opening and closing

With eight panelists, ensuring that introductions didn’t drag on too long was particularly important. I asked each panelist to introduce themselves with a couple of sentences that highlighted the organizations they were affiliated with that were relevant to the panel. For our chosen group of panelists this was definitely the right framing.

I then asked each panelist to be prepared to close the panel with a call to action: something an audience member could actively do that would support the community going forward. This worked really well: it provided a great, actionable note to end both the panel and the conference.

Preparing and running the panel

We used our panel lunch together to check that no one would have calls to action that overlapped too much, and to provide a rough indication of who had things to say about each topic we planned to discuss.

This turned out to be essential: I’ve been on smaller panels where the panelists have been able to riff easily on each other’s points, but with eight panelists it turned out not everyone could even see each other, so as panel moderator it fell on me to direct questions to individuals and then prompt others for follow-ups. Thankfully the panel lunch combined with the one-to-one conversations gave me the information I needed for this.

I had written down a selection of questions for each of the themes. Having a selection turned out to be crucial: a few times the panelists talked about material that I had planned to cover in a later section, so I had to adapt as we went along. In the future I’ll spend more time on this: these written ideas were a crucial component in keeping the panel flowing in the right direction.

With everything in place the panel itself was a case of concentrating on what everyone was saying and using the selection of the next questions (plus careful ad-libbing) to guide the conversation along the preselected themes. I also tried to keep mental track of who had done the most speaking so I could ensure the conversation stayed as balanced as possible by inviting other panelists into the conversation.

The video of the panel should be out in around three weeks time, at which point you can evaluate for yourself if we managed to do a good job of it. I’m really happy with the feedback we got after the panel, and I plan to use a similar process for panels I’m involved with in the future.

Tags: andrew-godwin, django, djangocon, frank-wiles, panels, speaking, my-talks, jeff-triplett

Django #8936: Add view (read-only) permission to admin (closed)

2018-05-17T13:40:35+00:00

Django #8936: Add view (read-only) permission to admin (closed)

Opened 10 years ago. Closed 15 hours ago. I apparently filed this issue during the first DjangoCon back in September 2008, when Adrian and Jacob mentioned on-stage that they would like to see a read-only permission for the Django Admin. Thanks to Olivier Dalang from Fiji and Petr Dlouhý from Prague it’s going to be a feature shipping in Django 2.1. Open source is a beautiful thing.

Tags: django, django-admin, djangocon, open-source

The denormalized query engine design pattern

2017-08-16T22:49:22+00:00

I presented this talk at DjangoCon 2017 in Spokane, Washington. Below is the abstract, the slides and the YouTube video of the talk.

Abstract

Most web applications need to offer search functionality. Open source tools like Solr and Elasticsearch are a powerful option for building custom search engines… but it turns out they can be used for way more than just search.

By treating your search engine as a denormalization layer, you can use it to answer queries that would be too expensive to answer using your core relational database. Questions like “What are the top twenty tags used by my users from Spain?” or “What are the most common times of day for events to start?” or “Which articles contain addresses within 500 miles of Toronto?”.

With the denormalized query engine design pattern, modifications to relational data are published to a denormalized schema in Elasticsearch or Solr. Data queries can then be answered using either the relational database or the search engine, depending on the nature of the specific query. The search engine returns database IDs, which are inflated from the database before being displayed to a user - ensuring that users never see stale data even if the search engine is not 100% up to date with the latest changes. This opens up all kinds of new capabilities for slicing, dicing and exploring data.

In this talk, I’ll be illustrating this pattern by focusing on Elasticsearch - showing how it can be used with Django to bring new capabilities to your application. I’ll discuss the challenge of keeping data synchronized between a relational database and a search engine, and show examples of features that become much easier to build once you have this denormalization layer in place.

Video

Slides

Tags: definitions, denormalisation, design-patterns, django, djangocon, elasticsearch, my-talks, software-architecture

Videos from DjangoCon 2009

2010-01-03T11:02:11+00:00

Videos from DjangoCon 2009

The videos from September's DjangoCon are now available, including my "Cowboy development with Django" talk (also on the Internet Archive).

I talked about building Wildlife Near You on a /dev/fort trip to Alderney and the MP's expenses project at the Guardian.

Tags: conferences, django, djangocon, python, the-guardian, wildlifenearyou

Welcome to Django Dose

2009-09-21T18:21:01+00:00

Welcome to Django Dose

Launched at DjangoCon, a new Django community site designed to be a successor to TWiD, still with (shorter) podcasts but also featuring more news, articles and screencasts.

Tags: community, django, djangocon, djangodose, podcasts, screencasts, twid

EuroDjangoCon 2009

2009-02-12T16:59:57+00:00

EuroDjangoCon 2009

Tickets are now on sale for the conference, scheduled for 4th-6th of May (not March as I originally said) in Prague (followed by two days of development sprints).

Tags: django, djangocon, eurodjangocn, prague, python

EuroDjangoCon

2009-01-24T18:54:25+00:00

EuroDjangoCon

4th-6th of May 2009, in Prague. Talk submissions are open now, and registration starts on the 6th of February.

Tags: conferences, django, djangocon, eurodjangocon, events, prague, python

Introducing the Django Debug Toolbar

2008-09-21T18:32:24+00:00

Introducing the Django Debug Toolbar

Another project inspired by DjangoCon: a component based debugging toolbar for Django. I like the architecture so far.

Tags: debugging, django, djangocon, django-debug-toolbar, rob-hudson

DjangoCon and learning from Zope 2

2008-09-17T23:25:10+00:00

DjangoCon and learning from Zope 2

Mark Ramm presented probably the most thought-provoking talk at DjangoCon. He’s started writing it up as a series of posts.

Tags: django, djangocon, mark-ramm, python, turbogears, zope2

YouTube: djangocon tag

2008-09-16T02:43:00+00:00

YouTube: djangocon tag

Google have started posting videos of presentations at DjangoCon on YouTube.

Tags: django, djangocon, google, python, youtube

DjangoCon and PyCon UK

2008-09-15T15:20:20+00:00

September is a big month for conferences. DjangoCon was a weekend ago in Mountain View (forcing me to miss both d.Construct and BarCamp Brighton), PyCon UK was this weekend in Birmingham, I'm writing this from @media Ajax and BarCamp London 5 is coming up over another weekend at the end of this month. As always, I've been posting details of upcoming talks and notes and materials from previous ones on my talks page.

DjangoCon went really, really well. Huge thanks to conference chair Robert Lofthouse for pulling it all together in just two months and Leslie Hawthorne for making it all happen from Google's end. Google's facitilies were superb: the AV team were the best I've ever worked with and an army of Google volunteers made sure everything went smoothly. It's hard to see how it could have gone better; the principle complaint we got was that at only two days it was hard to justify the travel, something which future DjangoCons will definitely address.

Every session was recorded and the videos ~~should be going up on YouTube shortly~~ are now up on YouTube. For the impatient, you can subscribe to an Atom feed of a YouTube search for "DjangoCon". I recommend starting with Cal Henderson's keynote "Why I hate Django" which was both funny and insightful in equal parts. Malcolm's talk on ORM internals was another personal favourite.

PyCon UK was the second I've attended, but last year I only stayed for the first day. This time I stuck around and was enormously impressed by the grassroots feel of the conference and the enthusiastic atmosphere. I presented a tutorial on extending the Django admin and a lightning talk on Zeppelins, prepared two hours in advance after Jacob mentioned that the lightning talks were tending too much towards the technical side. It went down very well; I'm tempted to extend it to a half hour session for BarCamp London.

Unlike most conferences I attend, PyCon tickets included a sit-down dinner for all attendees complete with a "dramatic lecture" on the Lunar Society presented by Andrew Lound. This was a great fit for the conference, both for the Birmingham connection and the many analogies to the modern open source community - loose collaboration, patent concerns and what you might call an 18th century equivalent of the modern hacker ethic.

Next year the PyCon UK team will be hosting EuroPython, and I'm certain they'll do an excellent job of it. Meanwhile, Rob has already started making plans for a Euro DjangoCon in around six months time, probably taking place in Prague.

Tags: andrew-lound, barcamplondon5, cal-henderson, conferences, django, djangocon, djangocon08, google, leslie-hawthorne, pyconuk, pyconuk08, python, robert-lofthouse, speaking, my-talks, zeppelins

The web framework for ponies

2008-09-09T23:59:57+00:00

The web framework for ponies

At DjangoCon Cal Henderson suggested that Django should get a mascot with “magical powers”. Brian Veloso obliges.

Tags: brian-veloso, cal-henderson, django, djangocon, ponies

Django tickets with keyword "djangocon"

2008-09-08T03:02:11+00:00

Django tickets with keyword "djangocon"

Adrian and Jacob ran an “I want a pony” session during their closing keynote at DjangoCon—I’ve filed the feature requests as tickets tagged with the “djangocon” keyword.

Tags: adrian-holovaty, django, djangocon, djangocon08, iwantapony, jacob-kaplan-moss, python, tickets

Django 1.0 release party

2008-08-27T12:07:15+00:00

Django 1.0 release party

The big ass-party will be at the Tied House in Mountain View on Saturday the 6th from 7pm. RSVP on the linked announcement.

Tags: bigassparty, django, djangocon, mountainview, party, tiedhouse

DjangoCon & Django 1.0 updates

2008-07-30T10:25:24+00:00

DjangoCon & Django 1.0 updates

DjangoCon tickets will be released in two batches of 100. The first set will be available at 12 noon UTC on Thursday July 31st; the second set will be released at 6pm UTC on Friday August 1st.

Tags: django, djangocon, events, python, tickets

DjangoCon 2008

2008-07-20T20:49:07+00:00

DjangoCon 2008

The official DjangoCon site is up, along with a mostly complete schedule.

Tags: django, djangocon, python

Quoting Robert Lofthouse

2008-07-13T16:50:17+00:00

DjangoCon 2008. Venue: Gooleplex, San Francisco Bay Area. Dates: 6th and 7th Sept. Official post will be on djangoproject.com soon.

— Robert Lofthouse

Tags: django, djangocon, events, google, googleplex, python, robert-lofthouse, san-francisco

Quoting Me

2008-07-07T17:14:03+00:00

It looks like the first ever Django conference will take place in early September in the San Francisco bay area.

— Me, on Twitter

Tags: conferences, django, djangocon, events, san-francisco, twitter

Simon Willison's Weblog: djangocon

Themes from DjangoCon US 2024

Growing the Django Software Foundation (DSF)

Could we fund a Django LTS accessibility audit?

Django fellows continue to provide outstanding value

Django needs feature champions

htmx fits Django really well

Django Ninja has positive buzz

Valkey as a last-minute sponsor

Durham has a world-class collection of tubas

Announcing our DjangoCon US 2024 Talks!

Coping strategies for the serial project hoarder

Weeknotes: DjangoCon, SQLite in Django, datasette-gunicorn

Benchmarking SQLite in Django

datasette-gunicorn

shot-scraper 1.0

Releases this week

TIL this week

How I moderated the State of Django panel at DjangoCon US.

Topic and themes

Opening and closing

Preparing and running the panel

Django #8936: Add view (read-only) permission to admin (closed)

The denormalized query engine design pattern

Abstract

Video

Slides

Videos from DjangoCon 2009

Welcome to Django Dose

EuroDjangoCon 2009

EuroDjangoCon

Introducing the Django Debug Toolbar

DjangoCon and learning from Zope 2

YouTube: djangocon tag

DjangoCon and PyCon UK

The web framework for ponies

Django tickets with keyword "djangocon"

Django 1.0 release party

DjangoCon & Django 1.0 updates

DjangoCon 2008

Quoting Robert Lofthouse

Quoting Me