Simon Willison's Weblog: cookiecutter

TIL: Using uv to develop Python command-line applications

2024-10-24T05:56:21+00:00

TIL: Using uv to develop Python command-line applications

I've been increasingly using uv to try out new software (via uvx) and experiment with new ideas, but I hadn't quite figured out the right way to use it for developing my own projects.

It turns out I was missing a few things - in particular the fact that there's no need to use uv pip at all when working with a local development environment, you can get by entirely on uv run (and maybe uv sync --extra test to install test dependencies) with no direct invocations of uv pip at all.

I bounced a few questions off Charlie Marsh and filled in the missing gaps - this TIL shows my new uv-powered process for hacking on Python CLI apps built using Click and my simonw/click-app cookecutter template.

Tags: cli, packaging, pip, python, til, cookiecutter, uv, astral, charlie-marsh

simonw/docs cookiecutter template

2024-09-23T21:45:15+00:00

simonw/docs cookiecutter template

Over the last few years I’ve settled on the combination of Sphinx, the Furo theme and the myst-parser extension (enabling Markdown in place of reStructuredText) as my documentation toolkit of choice, maintained in GitHub and hosted using ReadTheDocs.

My LLM and shot-scraper projects are two examples of that stack in action.

Today I wanted to spin up a new documentation site so I finally took the time to construct a cookiecutter template for my preferred configuration. You can use it like this:

pipx install cookiecutter
cookiecutter gh:simonw/docs

Or with uv:

uv tool run cookiecutter gh:simonw/docs

Answer a few questions:

[1/3] project (): shot-scraper
[2/3] author (): Simon Willison
[3/3] docs_directory (docs):

And it creates a docs/ directory ready for you to start editing docs:

cd docs
pip install -r requirements.txt
make livehtml

Tags: documentation, projects, python, markdown, cookiecutter, sphinx-docs, read-the-docs, uv

Upgrading my cookiecutter templates to use python -m pytest

2024-08-17T05:12:47+00:00

Upgrading my cookiecutter templates to use python -m pytest

Every now and then I get caught out by weird test failures when I run pytest and it turns out I'm running the wrong installation of that tool, so my tests fail because that pytest is executing in a different virtual environment from the one needed by the tests.

The fix for this is easy: run python -m pytest instead, which guarantees that you will run pytest in the same environment as your currently active Python.

Yesterday I went through and updated every one of my cookiecutter templates (python-lib, click-app, datasette-plugin, sqlite-utils-plugin, llm-plugin) to use this pattern in their READMEs and generated repositories instead, to help spread that better recipe a little bit further.

Tags: projects, python, pytest, cookiecutter

Publish Python packages to PyPI with a python-lib cookiecutter template and GitHub Actions

2024-01-16T21:59:56+00:00

I use cookiecutter to start almost all of my Python projects. It helps me quickly generate a skeleton of a project with my preferred directory structure and configured tools.

I made some major upgrades to my python-lib cookiecutter template today. Here's what it can now do to help you get started with a new Python library:

Create a pyproject.toml file configured for use with setuptools. In my opinion this is the pattern with the current lowest learning curve - I wrote about that in detail in this TIL.
Add a skeleton README and an Apache 2.0 LICENSE file.
Create your_package/__init__.py for your code to go in.
Create tests/test_your_package.py with a skeleton test.
Include pytest as a test dependency.
Configure GitHub Actions with two workflows in .github/workflows - one for running the tests against Python 3.8 through 3.12, and one for publishing releases of your package to PyPI.

The changes I made today are that I switched from setup.py to pyproject.toml, and I made a big improvement to how the publishing workflow authenticates with PyPI.

Publishing to PyPI with Trusted Publishing

My previous version of this template required you to jump through quite a few hoops to get PyPI publishing to work. You needed to create a PyPI token that could publish a new package, then paste that token into a GitHub Actions secret, then publish the package, and then disable that token and create a new one dedicated to just updating this package in the future.

The new version is much simpler, thanks to PyPI's relatively new Trusted Publishers mechanism.

To publish a new package, you need to sign into PyPI and create a new "pending publisher". Effectively you tell PyPI "My GitHub repository myname/name-of-repo should be allowed to publish packages with the name name-of-package".

Here's that form for my brand new datasette-test library, the first library I published using this updated template:

Then create a release on GitHub, with a name that matches the version number from your pyproject.toml. Everything else should Just Work.

I wrote more about Trusted Publishing in this TIL.

Creating a package using a GitHub repository template

The most time consuming part of this project was getting my GitHub repository template to work properly.

There are two ways to use my cookiecutter template. You can use the cookiecutter command-line tool like this:

pipx install cookiecutter
cookiecutter gh:simonw/python-lib
# Answer a few questions here

But a more fun and convenient option is to use my GitHub repository template, simonw/python-lib-template-repository.

This lets you fill in a form on GitHub to create a new repository which will then execute the cookiecutter template for you and update itself with the result.

You can see an example of a repository created using this template at datasette/datasette-test.

Adding it all together

There are quite a lot of moving parts under the scenes here, but the end result is that anyone can now create a Python library with test coverage, GitHub CI and release automation by filling in a couple of forms and clicking some buttons.

For more details on how this all works, and how it's evolved over time:

A cookiecutter template for writing Datasette plugins from June 2020 describes my first experiments with cookiecutter
Dynamic content for GitHub repository templates using cookiecutter and GitHub Actions from August 2021 describes my earliest attempts at using GitHub repository templates for this
How to build, test and publish an open source Python library is a ten minute talk I gave at PyGotham in November 2021. It describes setup.py in detail, which is no longer my preferred approach.

Tags: github, projects, pypi, python, github-actions, cookiecutter

Cookiecutter Data Science

2021-11-18T15:21:59+00:00

Cookiecutter Data Science

Some really solid thinking in this documentation for the DrivenData cookiecutter template. They emphasize designing data science projects for repeatability, such that just the src/ and data/ folders can be used to recreate all of the other analysis from scratch. I like the suggestion to give each project a dedicated S3 bucket for keeping immutable copies of the original raw data that might be too large for GitHub.

Via Paige Bailey

Tags: data-science, cookiecutter

Dynamic content for GitHub repository templates using cookiecutter and GitHub Actions

2021-08-28T21:29:12+00:00

GitHub repository templates were introduced a couple of years ago to provide a mechanism for creating a brand new GitHub repository starting with an initial set of files.

They have one big limitation: the repositories that they create share the exact same contents as the template repository. They're basically a replacement for duplicating an existing folder and using that as the starting point for a new project.

I'm a big fan of the Python cookiecutter tool, which provides a way to dynamically create new folder structures from user-provided variables using Jinja templates to generate content.

This morning, inspired by this repo by Bruno Rocha, I finally figured out a neat pattern for combining cookiecutter with repository templates to compensate for that missing dynamic content ability.

The result: datasette-plugin-template-repository for creating new Datasette plugins with a single click, python-lib-template-repository for creating new Python libraries and click-app-template-repository for creating Click CLI tools.

Cookiecutter

I maintain three cookiecutter templates at the moment:

simonw/datasette-plugin, for creating new Datasette plugins. I've used that one for dozens of plugins myself.
simonw/click-app, which generates a skeleton for a new Click-based command-line tool. Many of my x-to-sqlite tools were built using this.
simonw/python-lib, for generating general-purpose Python libraries.

Having installed cookiecutter (pip install cookiecutter) each of these can be used like so:

% cookiecutter gh:simonw/datasette-plugin
plugin_name []: visualize counties
description []: Datasette plugin for visualizing counties
hyphenated [visualize-counties]: 
underscored [visualize_counties]: 
github_username []: simonw
author_name []: Simon Willison
include_static_directory []: y
include_templates_directory []:

Cookiecutter prompts for some variables defined in a cookiecutter.json file, then generates the project by evaluating the templates.

The challenge was: how can I run this automatically when a new repository is created from a GitHub repository template? And where can I get those variables from?

Bruno's trick: a self-rewriting repository

Bruno has a brilliant trick for getting this to run, exhibited by this workflow YAML. His workflow starts like this:

name: Rename the project from template

on: [push]

jobs:
  rename-project:
    if: ${{ github.repository != 'rochacbruno/python-project-template' }}
    runs-on: ubuntu-latest
    steps:
       # ...

This means that his workflow only runs on copies of the original repository - the workflow is disabled in the template repository itself by that if: condition.

Then at the end of the workflow he does this:

      - uses: stefanzweifel/git-auto-commit-action@v4
        with:
          commit_message: "Ready to clone and code"
          push_options: --force

This does a force push to replace the contents of the repository with whatever was generated by the rest of the workflow script!

This trick was exactly what I needed to get cookiecutter to work with repository templates.

Gathering variables using the GitHub GraphQL API

All three of my existing cookiecutter templates require the following variables:

A name to use for the generated folder
A one-line description to use in the README and in setup.py
The GitHub username of the owner of the package
The display name of the owner

I need values for all of these before I can run cookiecutter.

It turns out they are all available from the GitHub GraphQL API, which can be called from the initial workflow copied from the repository template!

Here's the GitHub Actions step that does that:

- uses: actions/github-script@v4
  id: fetch-repo-and-user-details
  with:
    script: |
      const query = `query($owner:String!, $name:String!) {
        repository(owner:$owner, name:$name) {
          name
          description
          owner {
            login
            ... on User {
              name
            }
            ... on Organization {
              name
            }
          }
        }
      }`;
      const variables = {
        owner: context.repo.owner,
        name: context.repo.repo
      }
      const result = await github.graphql(query, variables)
      console.log(result)
      return result

Here I'm using the actions/github-script action, which provides a pre-configured, authenticated instance of GitHub's octokit/rest.js JavaScript library. You can then provide custom JavaScript that will be executed by the action.

await github.graphql(query, variables) can then execute a GitHub GraphQL query. The query I'm using here gives me back the current repository's name and description and the login and display name of the owner of that repository.

GitHub repositories can be owned by either a user or an organization - the ... on User / ... on Organization syntax provides the same result here for both types of nested object.

The output of this GraphQL query looks something like this:

{
  "repository": {
    "name": "datasette-verify",
    "description": "Verify that files can be opened by Datasette",
    "owner": {
      "login": "simonw",
      "name": "Simon Willison"
    }
  }
}

I assigned an id of fetch-repo-and-user-details to that step of the workflow, so that the return value from the script could be accessed as JSON in the next step.

Passing those variables to cookiecutter

Cookiecutter defaults to asking for variables interactively, but it also supports passing in those variables as command-line parameters.

Here's part of my next workflow steps that executes cookiecutter using the variables collected by the GraphQL query:

- name: Rebuild contents using cookiecutter
  env:
    INFO: ${{ steps.fetch-repo-and-user-details.outputs.result }}
  run: |
    export REPO_NAME=$(echo $INFO | jq -r '.repository.name')
    # Run cookiecutter
    cookiecutter gh:simonw/python-lib --no-input \
      lib_name=$REPO_NAME \
      description="$(echo $INFO | jq -r .repository.description)" \
      github_username="$(echo $INFO | jq -r .repository.owner.login)" \
      author_name="$(echo $INFO | jq -r .repository.owner.name)"

The env: INFO: block exposes an environment variable called INFO to the step, populated with the output of the previous fetch-repo-and-user-details step - a string of JSON.

Then within the body of the step I use jq to extract out the details that I need - first the repository name:

export REPO_NAME=$(echo $INFO | jq -r '.repository.name')

Then I pass the other details directly to cookiecutter as arguments:

cookiecutter gh:simonw/python-lib --no-input \
  lib_name=$REPO_NAME \
  description="$(echo $INFO | jq -r .repository.description)" \
  github_username="$(echo $INFO | jq -r .repository.owner.login)" \
  author_name="$(echo $INFO | jq -r .repository.owner.name)"

jq -r ensures that the raw text value is returned by jq, as opposed to the JSON string value which would be wrapped in double quotes.

Cleaning up at the end

Running cookiecutter in this way creates a folder within the root of the repository that duplicates the repository name, something like this:

datasette-verify/datasette-verify

I actually want the contents of that folder to live in the root, so the next step I run is:

mv $REPO_NAME/* .
mv $REPO_NAME/.gitignore .
mv $REPO_NAME/.github .

Here's my completed workflow.

This almost worked - but when I tried to run it for the first time I got this error:

![remote rejected] (refusing to allow an integration to create or update .github/workflows/publish.yml)

It turns out the credentials provided to GitHub Actions are forbidden from making modifications to their own workflow files!

I can understand why that limitation is in place, but it's frustrating here. For the moment, my workaround is to do this just before pushing the final content back to the repository:

mv .github/workflows .github/rename-this-to-workflows

I leave it up to the user to rename that folder back again when they want to enable the workflows that have been generated for them.

Give these a go

I've set up three templates using this pattern now:

datasette-plugin-template-repository for creating new Datasette plugins - use this template
python-lib-template-repository for creating new Python libraries - use this template
click-app-template-repository for creating new Python Click CLI tools - use this template

Each of these works the same way: enter a repository name and description, click "Create repository from template" and watch as GitHub copies the new repository and then, a few seconds later, runs the workflow to execute the cookiecutter template to replace the contents with the final result.

You can see examples of repositories that I created using these templates here:

Tags: github-actions, cookiecutter

pypi-rename

2020-07-25T23:07:02+00:00

pypi-rename

I wanted to rename a PyPI package (renaming datasette-insert-api to datasette-insert as it’s about to grow some non-API features). PyPI recommend uploading a final release under the old name which points to (and depends on) the new name. I’ve built a cookiecutter template to codify that pattern.

Tags: projects, pypi, cookiecutter

Weeknotes: cookiecutter templates, better plugin documentation, sqlite-generate

2020-06-26T01:39:50+00:00

I spent this week spreading myself between a bunch of smaller projects, and finally getting familiar with cookiecutter. I wrote about my datasette-plugin cookiecutter template earlier in the week; here's what else I've been working on.

sqlite-generate

Datasette is supposed to work against any SQLite database you throw at it, no matter how weird the schema or how unwieldy the database shape or size.

I built a new tool called sqlite-generate this week to help me create databases of different shapes. It's a Python command-line tool which uses Faker to populate a new database with random data. You run it something like this:

sqlite-generate demo.db \
    --tables=20 \
    --rows=100,500 \
    --columns=5,20 \
    --fks=0,3 \
    --pks=0,2 \
    --fts

This command creates a database containing 20 tables, each with between 100 and 500 rows and 5-20 columns. Each table will also have between 0 and 3 foreign key columns to other tables, and will feature between 0 and 2 primary key columns. SQLite full-text search will be configured against all of the text columns in the table.

I always try to include a live demo with any of my projects, and sqlite-generate is no exception. This GitHub Action runs on every push to main and deploys a demo to https://sqlite-generate-demo.datasette.io/ showing the latest version of the code in action.

The demo runs my datasette-search-all plugin in order to more easily demonstrate full-text search across all of the text columns in the generated tables. Try searching for newspaper.

click-app cookiecutter template

I write quite a lot of Click powered command-line tools like this one, so inspired by datasette-plugin I created a new click-app cookiecutter template that bakes in my own preferences about how to set up a new Click project (complete with GitHub Actions). sqlite-generate is the first tool I've built using that template.

Improved Datasette plugin documentation

I've split Datasette's plugin documentation into five separate pages, and added a new page to the documentation about patterns for testing plugins.

The five pages are:

Plugins describing how to install and configure plugins
Writing plugins showing how to write one-off plugins, how to use the datasette-plugin cookiecutter template and how to package templates for release to PyPI
Plugin hooks documenting all of the available plugin hooks
Testing plugins describing my preferred patterns for writing tests for them (using pytest and HTTPX)
Internals for plugins describing the APIs Datasette makes available for use within plugin hook implementations

There's also a list of available plugins on the Datasette Ecosystem page of the documentation, though I plan to move those to a separate plugin directory in the future.

datasette-block-robots

The datasette-plugin template practically eliminates the friction involved in starting a new plugin.

sqlite-generate generates random names for people. I don't particularly want people who search for their own names stumbling across the live demo and being weirded out by their name featured there, so I decided to block it from search engine crawlers using robots.txt.

I wrote a tiny plugin to do this: datasette-block-robots, which uses the new register_routes() plugin hook to add a /robots.txt page.

It's also a neat example of the simplest possible plugin to use that feature - along with the simplest possible unit test for exercising such a page.

datasette-saved-queries

Another new plugin, this time with a bit more substance to it. datasette-saved-queries exercises the new canned_queries() hook I described last week. It uses the new startup() hook to create tables on startup (if they are missing), then lets users insert records into those tables to save their own queries. Queries saved in this way are then returned as canned queries for that particular database.

main, not master

main is a better name for the main GitHub branch than master, which has unpleasant connotations (it apparently derives from master/slave in BitKeeper). My datasette-plugin and click-app cookiecutter templates both include instructions for renaming master to main in their READMEs - it's as easy as running git branch -m master main before running your first push to GitHub.

I'm working towards making the switch for Datasette itself.

Tags: git, plugins, projects, robots-txt, sqlite, datasette, weeknotes, cookiecutter

click-app

2020-06-23T02:21:08+00:00

click-app

While working on sqlite-generate today I built a cookiecutter template for building the skeleton for Click command-line utilities. It’s based on datasette-plugin so it automatically sets up GitHub Actions for running tests and deploying packages to PyPI.

Tags: projects, python, cookiecutter

A cookiecutter template for writing Datasette plugins

2020-06-20T16:15:42+00:00

Datasette’s plugin system is one of the most interesting parts of the entire project. As I explained to Matt Asay in this interview, the great thing about plugins is that Datasette can gain new functionality overnight without me even having to review a pull request. I just need to get more people to write them!

datasette-plugin is my most recent effort to help make that as easy as possible. It’s a cookiecutter template that sets up the outline of a new plugin, combining various best patterns I’ve discovered over the past two years of writing my own plugins.

Once you’ve installed cookiecutter you can start building a new plugin by running:

cookiecutter gh:simonw/datasette-plugin

Cookiecutter will run a quick interactive session asking for a few details. It will then use those details to generate a new directory structure ready for you to start work on the plugin.

The datasette-plugin README describes the next steps. A couple of things are worth exploring in more detail.

Writing tests for plugins

I’m a big believer in automated testing: every single one of my plugins includes tests, and those test are run against every commit and must pass before new packages are shipped to PyPI.

In my experience the hardest part of writing tests is getting them started: setting up an initial test harness and ensuring that new tests can be easily written.

datasette-plugin adds pytest as a testing dependency and creates a tests/ folder with an initial, passing unit test in it.

The test confirms that the new plugin has been correctly installed, by running a request through a configured Datasette instance and hitting the /-/plugins.json introspection endpoint.

In doing so, it demonstrates how to run tests that interact with Datasette’s HTTP API. This is a very productive way to write tests.

The example test uses the HTTPX Python library. HTTPX offers a requests-style API but with a couple of crucial improvements. Firstly, it’s been built with asyncio support as a top-level concern. Secondly, it understands the ASGI protocol and can be run directly against an ASGI Python interface without needing to spin up an actual HTTP server. Since Datasette speaks ASGI this makes it the ideal tool for testing Datasette plugins.

Here’s that first test that gets created by the cookiecutter template:

from datasette.app import Datasette
import pytest
import httpx

@pytest.mark.asyncio
async def test_plugin_is_installed():
    app = Datasette([], memory=True).app()
    async with httpx.AsyncClient(app=app) as client:
        response = await client.get(
            "http://localhost/-/plugins.json"
        )
        assert 200 == response.status_code
        installed_plugins = {
            p["name"] for p in response.json()
        }
        assert "datasette-plugin-template-demo" in installed_plugins

My hope is that including a passing test that demonstrates how to execute test requests will make it much easier for plugin authors to start building out their own custom test suite.

Continuous integration with GitHub Actions

My favourite thing about GitHub Actions is that they’re enabled on every GitHub repository for free, without any extra configuration necessary.

The datasette-plugin template takes advantage of this. Not only does every new project get a passing test - it also gets a GitHub Action - in .github/workflows/test.yml - that executes the tests on every commit.

It even runs the test suite in parallel against Python 3.6, 3.7 and 3.8 - the versions currently supported by Datasette itself.

A second action in .github/workflows/publish.yml bakes in my opinions on the best way to manage plugin releases: it builds and ships a new package to PyPI every time a new tag (and corresponding GitHub release) is added to the repository.

For this to work you’ll need to create a PyPI API token and add it to your plugin’s GitHub repository as a PYPI_TOKEN secret. This is explained in the README.

Deploying a live demo of the template with GitHub Actions

Whenever possible, I like to ship my projects with live demos. The Datasette repository publishes a demo of the latest commit to https://latest.datasette.io/ on every commit. I try to do the same for my plugins, where it makes sense to do so.

What could a live demo of a cookiecutter template look like?

Ideally it would show a complete, generated project. I love GitHub’s code browsing interface, so a separate repository containing that generated project would be ideal.

So that’s what https://github.com/simonw/datasette-plugin-template-demo is: it’s a repository showing the most recent output of the latest version of the cookiecutter template that lives in https://github.com/simonw/datasette-plugin.

It’s powered by this GitHub Action, which runs on every push to the datasette-plugin repo, installs cookiecutter, uses cookiecutter against some fixed inputs to re-generate the project and then pushes the results up to datasette-plugin-template-demo as a new commit.

As a fun final touch, it uses the GitHub commit comments API to add a comment to the commit to datasette-plugin linking to the “browse” view on the resulting code in the datasette-plugin-template-demo repository. Here’s one of those commit comments.

Figuring out how to build this took quite a bit of work. Issue #4 has a blow-by-blow rundown of how I got it working.

I couldn’t resist tweeting about it:

Writing a GitHub Action for a repo that generates content for a second repo and pushes that content to the second repo and then posts a comment to the commit on the first repo that links to the newly committed code in the second repo
- Simon Willison (@simonw) June 19, 2020

Tags: github, plugins, projects, pypi, datasette, pytest, github-actions, cookiecutter