Simon Willison's Weblog: authentication

Weeknotes: django-sql-dashboard widgets

2021-03-21T05:50:25+00:00

A few small releases this week, for django-sql-dashboard, datasette-auth-passwords and datasette-publish-vercel.

django-sql-dashboard widgets and permissions

django-sql-dashboard, my subset-of-Datasette-for-Django-and-PostgreSQL continues to come together.

New this week: widgets and permissions.

To recap: this Django app borrows some ideas from Datasette: it encourages you to create a read-only PostgreSQL user and grant authenticated users the ability to run one or more raw SQL queries directly against your database.

You can execute more than one SQL query and combine them into a saved dashboard, which will then show multiple tables containing the results.

This week I added support for dashboard widgets. You can construct SQL queries to return specific column patterns which will then be rendered on the page in different ways.

There are four widgets at the moment: "big number", bar chart, HTML and Markdown.

Big number is the simplest: define a SQL query that returns two columns called label and big_number and the dashboard will display that result as a big number:

select 'Entries' as label, count(*) as big_number from blog_entry;

Bar chart is more sophisticated: return columns named bar_label and bar_quantity to display a bar chart of the results:

select
  to_char(date_trunc('month', created), 'YYYY-MM') as bar_label,
  count(*) as bar_quantity
from
  blog_entry
group by
  bar_label
order by
  count(*) desc

HTML and Markdown are simpler: they display the rendered HTML or Markdown, after filtering it through the Bleach library to strip any harmful elements or scripts.

select
  '## Ten most recent blogmarks (of ' 
  || count(*) || ' total)'
as markdown from blog_blogmark;

I'm running the dashboard application on this blog, and I've set up an example dashboard here that illustrates the different types of widget.

Defining custom widgets is easy: take the column names you would like to respond to, sort them alphabetically, join them with hyphens and create a custom widget in a template file with that name.

So if you wanted to build a widget that looks for label and geojson columns and renders that data on a Leaflet map, you would create a geojson-label.html template and drop it into your Django templates/django-sql-dashboard/widgets folder. See the custom widgets documentation for details.

Which reminds me: I decided a README wasn't quite enough space for documentation here, so I started a Read The Docs documentation site for the project.

Datasette and sqlite-utils both use Sphinx and reStructuredText for their documentation.

For django-sql-dashboard I've decided to try out Sphinx and Markdown instead, using MyST - a Markdown flavour and parser for Sphinx.

I picked this because I want to add inline help to django-sql-dashboard, and since it ships with Markdown as a dependency already (to power the Markdown widget) my hope is that using Markdown for the documentation will allow me to ship some of the user-facing docs as part of the application itself. But it's also a fun excuse to try out MyST, which so far is working exactly as advertised.

I've seen people in the past avoid Sphinx entirely because they preferred Markdown to reStructuredText, so MyST feels like an important addition to the Python documentation ecosystem.

HTTP Basic authentication

datasette-auth-passwords implements password-based authentication to Datasette. The plugin defaults to providing a username and password login form which sets a signed cookie identifying the current user.

Version 0.4 introduces optional support for HTTP Basic authentication instead - where the user's browser handles the authentication prompt.

Basic auth has some disadvantages - most notably that it doesn't support logout without the user entirely closing down their browser. But it's useful for a number of reasons:

It's easy to protect every resource on a website with it - including static assets. Adding "http_basic_auth": true to your plugin configuration adds this protection, covering all of Datasette's resources.
It's much easier to authenticate with from automated scripts. curl and roquests and httpx all have simple built-in support for passing basic authentication usernames and passwords, which makes it a useful target for scripting - without having to install an additional authentication plugin such as datasette-auth-tokens.

I'm continuing to flesh out authentication options for Datasette, and adding this to datasette-auth-passwords is one of those small improvements that should pay off long into the future.

A fix for datasette-publish-vercel

Datasette instances published to Vercel using the datasette-publish-vercel have previously been affected by an obscure Vercel bug: characters such as + in the query string were being lost due to Vercel unescaping encoded characters before the request got to the Python application server.

Vercel fixed this earlier this month, and the latest release of datasette-publish-vercel includes their fix by switching to the new @vercel/python builder. Thanks @styfle from Vercel for shepherding this fix through!

New photos on Niche Museums

My Niche Museums project has been in hiberation since the start of the pandemic. Now that vaccines are rolling out it feels like there might be an end to this thing, so I've started thinking about my museum hobby again.

I added some new photos to the site today - on the entries for Novelty Automation, DEVIL-ish Little Things, Evergreen Aviation & Space Museum and California State Capitol Dioramas.

Hopefully someday soon I'll get to visit and add an entirely new museum!

Releases this week

django-sql-dashboard: 0.4a1 - (10 releases total) - 2021-03-21
Django app for building dashboards using raw SQL queries
datasette-publish-vercel: 0.9.2 - (14 releases total) - 2021-03-20
Datasette plugin for publishing data using Vercel
datasette-auth-passwords: 0.4 - (9 releases total) - 2021-03-19
Datasette plugin for authentication using passwords

Tags: authentication, dashboard, django, postgresql, projects, zeit-now, weeknotes, django-sql-dashboard, sphinx-docs

datasette-auth-passwords

2020-07-13T23:39:06+00:00

datasette-auth-passwords

My latest plugin: datasette-auth-passwords provides a mechanism for signing into Datasette using a username and password (which is verified in order to set a ds_actor authentication cookie). So far it only supports passwords that are hard-coded into Datasette’s configuration via environment variables, but I plan to add database-backed user accounts in the future.

Tags: authentication, passwords, plugins, projects, datasette

Datasette 0.44: The annotated release notes

2020-06-12T03:11:06+00:00

I just released Datasette 0.44 to PyPI. With 128 commits since 0.43 this is the biggest release in a long time - and likely the last major release of new features before Datasette 1.0.

You can read the full release notes here, but I've decided to try something a little different for this release and write up some annotations here on my blog.

Writable canned queries
Datasette's Canned queries feature lets you define SQL queries in metadata.json which can then be executed by users visiting a specific URL. https://latest.datasette.io/fixtures/neighborhood_search for example.
Canned queries were previously restricted to SELECT, but Datasette 0.44 introduces the ability for canned queries to execute INSERT or UPDATE queries as well, using the new "write": true property (#800)

I originally intended this to be the main feature in the release.

Datasette 0.37 added the ability for plugins to write to the database. This marked a pretty huge philosophical shift for Datasette: from a read-only publishing system to a framework for building interactive SQL-driven applications. But you needed a plugin, such as my datasette-upload-csvs.

I realized back in March that canned queries could provide a simple, sensible way to start adding write functionality to Datasette core. A query could be configured like this:

{
  "databases": {
    "my-database": {
      "queries": {
        "add_twitter_handle": {
          "sql": "insert into twitter_handles (username) values (:username)",
          "write": true
        }
      }
    }
  }
}

And Datasette could then provide a form interface at /my-database/add_twitter_handle for executing that query. How hard could that be to implement?

The problem with "simple" features like this is that they open up a cascade of other features.

If you're going to have write queries, you probably need to restrict who can execute them - which means authentication and permissions. If forms are performing POSTs you need CSRF protection. If users are changing state you need a way to show them messages telling them what happened.

So let's talk about authentication and permissions.

Authentication
Prior to this release the Datasette ecosystem has treated authentication as exclusively the realm of plugins, most notably through datasette-auth-github.
0.44 introduces Authentication and permissions as core Datasette concepts (#699). This makes it easier for different plugins can share responsibility for authenticating requests - you might have one plugin that handles user accounts and another one that allows automated access via API keys, for example.

I demonstrated with datasette-auth-github and datasette-auth-existing-cookies that authentication could exist as completely separate layer from Datasette call (thanks to the magic of ASGI middleware). But I'd started to run into limitations of this approach.

Crucially, I wanted to be able to support more than one kind of authentication. Users might get authenticated with cookies (via a SSO mechanism such as GitHub's) but API clients need API keys. Now the different authentication plugins need to make sure they don't accidentally intefere with each other's logic.

Authentication in web applications always comes down to the same thing: inspecting aspects of the incoming HTTP request (headers, cookies, querystring variables etc) and deciding if they prove that the request is coming from a specific user or API integration.

I decided to use the word "actor" for this, since "user or API integration" is a bit of a mess. This means that authentication can work entirely via a new plugin hook, actor_from_request.

Here's a really simple implementation of that hook, copied directly from the documentation:

from datasette import hookimpl
import secrets

SECRET_KEY = "this-is-a-secret"

@hookimpl
def actor_from_request(datasette, request):
    authorization = request.headers.get("authorization") or ""
    expected = "Bearer {}".format(SECRET_KEY)

    if secrets.compare_digest(authorization, expected):
        return {"id": "bot"}

It returns an "actor dictionary" describing the authenticated actor. Datasette currently has no opinion at all on what shape this dictionary should take, though I expect conventions to emerge over time.

I have a new policy of never releasing a new plugin hook without also building a real-world plugin with it. For actor_from_request I've released datasette-auth-tokens, which lets you create secret API tokens in a configuration file and specify which actions they are allowed to perfom.

Permissions
Datasette also now has a built-in concept of Permissions. The permissions system answers the following question:
Is this actor allowed to perform this action, optionally against this particular resource?
You can use the new "allow" block syntax in metadata.json (or metadata.yaml) to set required permissions at the instance, database, table or canned query level. For example, to restrict access to the fixtures.db database to the "root" user:
{
    "databases": {
        "fixtures": {
            "allow": {
                "id" "root"
            }
        }
    }
}
See Defining permissions with "allow" blocks for more details.
Plugins can implement their own custom permission checks using the new permission_allowed(datasette, actor, action, resource) hook.

Authentication on its own isn't enough: you also need a way of deciding if an authenticated actor has permission to perform a specific action.

I was dreading adding permissions to Datasette. I have a long-running feud with Amazon IAM and the Google Cloud equivalent: I've been using AWS for over a decade and I still get completely lost any time I try to figure out the minimum set of permissions for something.

But... I want Datasette permissions to be flexible. My dream for Datasette is to nurture a growing ecosystem of plugins that can solve data collaboration and analysis problems way beyond what I've imagined myself.

Thanks to Datasette's plugin hooks, I think I've found a way to provide powerful plugins with minimum additional footprint to Datasette itself.

The key is the new permission_allowed() hook, which lets plugins receive an actor, action and optional resource and allows them to reply with "allow", "deny" or "I don't know, ask someone else".

Its partner is the datasette.permission_allowed(actor, action, resource) method, which plugins (and Datasette core) can call to check if the current actor is allowed to perform an action against a given resource (a specific database, table or canned query).

I invented a JSON/YAML based syntax for defining simple permission rules. If you want to provide access to a table to a specific user you can do so like this:

{
    "allow": {
        "id": "simonw"
    }
}

Or you can grant access to any actor with a role of "staff" like so:

{
    "allow": {
        "role": "staff"
    }
}

What are roles here? They're nothing at all to Datasette itself. Datasette authentication plugins can create actors of any shape, so if your plugin decides that "role" is a useful concept it can just bake it into the Datasette.

You can read more about how these "allow" blocks work in the documentation.

/-/permissions debug tool

Given my ongoing battle with opaque permission systems, I'm determined to try and make Datasette's take on permissions as transparent as possible. The new /-/permissions page - visible only to authenticated users with the debug-permissions permission - shows a rolling log of the last 200 permission checks carried out by that instance. My hope is that instance administrators and plugin authors can use this to figure out exactly what is going on.

datasette-permissions-sql

datasette-permissions-sql is my new proof-of-concept plugin that puts the permission hook to use.

It exercises a Datasette pattern I find really interesting: using SQL queries to configure custom behaviour. I first started eploring this in datasette-atom and datasette-ics.

This is best illustrated by an example metadata.yaml file. I prefer YAML over JSON for anything that includes a SQL query because YAML has support for multi-line strings:

databases:
  mydatabase:
    queries:
      promote_to_staff:
        sql: |-
          UPDATE users
          SET is is_staff=1
          WHERE id=:id
        write: true
plugins:
  datasette-permissions-sql:
  - action: view-query
    resource:
    - mydatabase
    - promote_to_staff
    sql: |-
      SELECT * FROM users
      WHERE is_staff = 1
      AND id = :actor_id

This block does two things. It configures a writable canned query called promote_to_staff. It then uses datasette-permissions-sql to define a permission rule that says that only authenticated actors who's id appears in the users table with is_staff=1 are allewod to execute that canned query

This is the beginnings of a full user management system in just a few lines of configuration. I'm really excited about exploring this concept further.

register_routes() plugin hooks
Plugins can now register new views and routes via the register_routes() plugin hook (#819). View functions can be defined that accept any of the current datasette object, the current request, or the ASGI scope, send and receive objects.

I thought the asgi_wrapper() hook might be enough to allow plugins to add their own custom routes and views, but the more I worked with it the more I wanted something a bit more high level.

Inspired by pytest, the view functions you can define using register_routes() benefit from a simple form of dependency injection. Any of the following counts as a valid view function - it will be called with the arguments it requests:

async def hello(request):
    return Response.html("Hello {}".format(
        html.escape(request.args.get("name"))
    ))

async def page(request, datasette):
    page = await datasette.get_databse().execute(
        "select body from pages where id = :id", {
            "id: request.url_vars["page_id"]
        }
    ).first()
    return Response.html(body)

async def asgi_hello_world(scope, receive, send):
    assert scope["type"] == "http"
    await send(
        {
            "type": "http.response.start",
            "status": 200,
            "headers": [
                [b"content-type", b"application/json"]
            ],
        }
    )
    await send({
        "type": "http.response.body",
        "body": b'{"hello": "world"}'
    })

Here's the code that makes this work - utility functions that pass in just the arguments that match a function's signature.

Tucked away at the end of the 0.44 release notes is this:

The road to Datasette 1.0
I've assembled a milestone for Datasette 1.0. The focus of the 1.0 release will be the following:
Signify confidence in the quality/stability of Datasette
Give plugin authors confidence that their plugins will work for the whole 1.x release cycle
Provide the same confidence to developers building against Datasette JSON APIs
If you have thoughts about what you would like to see for Datasette 1.0 you can join the conversation on issue #519.

It's time to start working towards the big 1.0!

Tags: authentication, permissions, plugins, projects, releasenotes, datasette, weeknotes, annotated-release-notes

NGINX: Authentication Based on Subrequest Result

2019-10-04T15:36:33+00:00

NGINX: Authentication Based on Subrequest Result

TIL about this neat feature of NGINX: you can use the auth_request directive to cause NGINX to make an HTTP subrequest to a separate authentication server for each incoming HTTP request. The authentication server can see the cookies on the incoming request and tell NGINX if it should fulfill the parent request (via a 2xx status code) or if it should be denied (by returning a 401 or 403). This means you can run NGINX as an authenticating proxy in front of any HTTP application and roll your own custom authentication code as a simple webhook-recieving endpoint.

Via Ishan Anand

Tags: authentication, nginx, webhooks

django-piston

2009-04-30T19:55:15+00:00

django-piston

Promising looking Django mini-framework for creating RESTful APIs, from the bitbucket team. Ticks all of Jacob’s boxes, even including built-in pluggable authentication support with HTTP Basic, Digest and OAuth out of the box.

Tags: apis, authentication, bitbucket, digest, django, jespernoehr, oauth, piston, python, rest, restful

Google's Usability Research on Federated Login

2008-09-22T20:56:33+00:00

Google's Usability Research on Federated Login

Fascinating—suggests an approach to federated auth based on the Amazon.com “Yes, I have a password” login flow. Feels convoluted to me but apparently it tests really well against a mainstream audience. The more research shared around this stuff the better.

Tags: amazon, authentication, federated, google, login, openid, usability

Quoting Nick Mathewson

2008-05-13T08:06:07+00:00

Something you had, Something you forgot, Something you were

— Nick Mathewson

Tags: authentication, nick-mathewson, security

Django snippets: Authenticate against Active Directory

2007-12-10T08:40:09+00:00

Django snippets: Authenticate against Active Directory

Uses a custom authentication backend with the Python ldap module. If Django hasn’t seen the user before a new Django user account is created with data from ldap.

Tags: activedirectory, authentication, django, ldap, python

OAuth Core 1.0

2007-12-05T03:39:10+00:00

OAuth Core 1.0

The final spec. Expect to see this crop up all over the place in the next few months.

Tags: apis, authentication, oauth

OAuth: Your valet key for the Web

2007-09-21T23:34:51+00:00

OAuth: Your valet key for the Web

OAuth is a really important new specification that aims to solve the “give this application permission to do X on my behalf” problem once and for all.

Tags: apis, authentication, oauth, openid, specification, web-services

Jottit

2007-09-16T21:43:16+00:00

Jottit

Aaron Swartz’s latest venture: a complete rethink of the Infogami concept. Well worth checking out for the extremely thoughtful way it introduces features, and the way account creation with a password remains optional until you want to add access control.

Tags: aaron-swartz, authentication, bitbots, infogami, jottit, usability, userflow, wiki

Wrong-headed impersonation

2007-03-05T14:38:58+00:00

Wrong-headed impersonation

Kim Cameron discusses user absent authentication, and emphasises the importance of delegation using delegation coupons.

Tags: authentication, delegation, delegationcoupons, identity, kimcameron

How is Google giving me access to this page?

2006-12-27T14:38:00+00:00

My answer to How is Google giving me access to this page? on Ask MetaFilter

Google have an open URL redirector, so you can craft a link that uses that:

Link via redirector

Tags: ask-metafilter, authentication, google, security, web