The "await me maybe" pattern for Python asyncio

2020-09-02T23:09:41+00:00

I've identified a pattern for handling potentially-asynchronous callback functions in Python which I'm calling the "await me maybe" pattern. It works by letting you return a value, a callable function that returns a value OR an awaitable function that returns that value.

Background

Datasette has been built on top of Python 3 asyncio from the very start - initially using Sanic, and as-of Datasette 0.29 using a custom mini-framework on top of ASGI 3, usually running under Uvicorn.

Datasette also has a plugin system, built on top of Pluggy.

Pluggy is a beautifully designed mechanism for plugins. It works based on decorated functions, which are called at various points by Datasette itself.

A simple plugin that injects a new JavaScript file into a page coud look like this:

from datasette import hookimpl

@hookimpl
def extra_js_urls():
    return [
        "https://code.jquery.com/jquery-3.5.1.min.js"
    ]

Datasette can then gather together all of the extra JavaScript URLs that should be injected into a page by running this code:

urls = []
for url in pm.hook.extra_js_urls(
    template=template.name,
    datasette=datasette,
):
    urls.extend(url)

What's up with the template= and datasette= parameters that are passed here?

Pluggy implements a form of dependency injection, where plugin hook functions can optionally list additional parameters that they would like to have access to.

The above simple example didn't need any extra information. But imagine a plugin that only wants to inject jQuery on the table.html template page:

@hookimpl
def extra_js_urls(template):
    if template == "table.html":
        return [
            "https://code.jquery.com/jquery-3.5.1.min.js"
        ]

Datasette actually provides several more optional argument for these plugin functions - see the plugin hooks documentation for full details.

What if we need to await something?

The datasette object that can be passed to plugin hooks is special: it provides an object that can be used for the following:

Executing SQL against databases connected to Datasette
Looking up Datasette metadata and configuration settings, including plugin configuration
Rendering templates using the template environment configured by Datasette
Performing checks against the Datasette permissions system

Here's the problem: many of those methods on Datasette are awaitable - await datasette.render_template(...) for example. But Pluggy is built around regular non-awaitable Python functions.

If my def extra_js_urls() plugin function needs to execute a SQL query to decide what JavaScript to include, it won't be able to - because you can't use await inside a regular Python function.

That's where the "await me maybe" pattern comes in.

The basic idea is that a function can return a value, OR a function-that-returns-a-value, OR an awaitable-function-that-returns-a-value.

If we want our extra_js_urls(datasette) hook to execute a SQL query in order to decide what URLs to return, it can look like this:

@hookimpl
def extra_js_urls(datasette):
    async def inner():
        db = datasette.get_database()
        results = await db.execute("select url from js_files")
        return [r[0] for r in results]

    return inner

Note that Python lets you define an async def inner() function inside the body of a regular function, which is what we're doing here.

The code that calls the plugin hook in Datasette can then look like this:

urls = []
for url in pm.hook.extra_js_urls(
    template=template.name,
    datasette=datasette,
):
    if callable(url):
        url = url()
    if asyncio.iscoroutine(url):
        url = await url
    urls.append(url)

I use this pattern in a bunch of different places in Datasette, so today I refactored that into a utility function:

import asyncio

async def await_me_maybe(value):
    if callable(value):
        value = value()
    if asyncio.iscoroutine(value):
        value = await value
    return value

This commit includes a bunch of examples where this function is called, for example this code which gathers extra body scripts to be included at the bottom of the page:

body_scripts = []
for extra_script in pm.hook.extra_body_script(
    template=template.name,
    database=context.get("database"),
    table=context.get("table"),
    columns=context.get("columns"),
    view_name=view_name,
    request=request,
    datasette=self,
):
    extra_script = await await_me_maybe(extra_script)
    body_scripts.append(Markup(extra_script))

Tags: async, patterns, plugins, python, datasette

Serverless Microservice Patterns for AWS

2019-06-12T00:13:38+00:00

Serverless Microservice Patterns for AWS

A handy collection of 19 architectural patterns for AWS Lambda collected by Jeremy Daly.

Via @PaulDJohnston

Tags: aws, lambda, patterns, software-architecture

Large Problems in Django, Mostly Solved: Search

2009-11-03T10:42:33+00:00

Large Problems in Django, Mostly Solved: Search

Eric Holscher shows how Haystack uses a number of common Django patterns (object registration, pluggable backends, QuerySet-style chaining and class-based views) to great effect in creating a powerful search application for Django. Makes me wonder if more of those patterns should be promoted to first class concepts within Django.

Tags: classbasedviews, django, eric-holscher, haystack, patterns, python, search

Collection: Search Patterns

2009-07-30T12:35:29+00:00

Collection: Search Patterns

Peter Morville’s enormous collection of screenshots of search engine interfaces.

Tags: design, patterns, peter-morville, search, ui, usability

Wikipatterns

2007-02-17T00:51:48+00:00

Wikipatterns

Great idea this: a wiki documenting patterns for successfully growing your own wiki.

Tags: atlassian, patterns, wiki