Simon Willison's Weblog: csrf

datasette PR #2689: Replace token-based CSRF with Sec-Fetch-Site header protection

2026-04-14T23:58:53+00:00

datasette PR #2689: Replace token-based CSRF with Sec-Fetch-Site header protection

Datasette has long protected against CSRF attacks using CSRF tokens, implemented using my asgi-csrf Python library. These are something of a pain to work with - you need to scatter forms in templates with <input type="hidden" name="csrftoken" value="{{ csrftoken() }}"> lines and then selectively disable CSRF protection for APIs that are intended to be called from outside the browser.

I've been following Filippo Valsorda's research here with interest, described in this detailed essay from August 2025 and shipped as part of Go 1.25 that same month.

I've now landed the same change in Datasette. Here's the PR description - Claude Code did much of the work (across 10 commits, closely guided by me and cross-reviewed by GPT-5.4) but I've decided to start writing these PR descriptions by hand, partly to make them more concise and also as an exercise in keeping myself honest.

New CSRF protection middleware inspired by Go 1.25 and this research by Filippo Valsorda. This replaces the old CSRF token based protection.

Removes all instances of <input type="hidden" name="csrftoken" value="{{ csrftoken() }}"> in the templates - they are no longer needed.

Removes the def skip_csrf(datasette, scope): plugin hook defined in datasette/hookspecs.py and its documentation and tests.

Updated CSRF protection documentation to describe the new approach.

Upgrade guide now describes the CSRF change.

Tags: csrf, security, datasette, ai-assisted-programming

A modern approach to preventing CSRF in Go

2025-10-15T05:03:46+00:00

A modern approach to preventing CSRF in Go

Alex Edwards writes about the new http.CrossOriginProtection middleware that was added to the Go standard library in version 1.25 in August and asks:

Have we finally reached the point where CSRF attacks can be prevented without relying on a token-based check (like double-submit cookies)?

It looks like the answer might be yes, which is extremely exciting. I've been tracking CSRF since I first learned about it 20 years ago in May 2005 and a cleaner solution than those janky hidden form fields would be very welcome.

The code for the new Go middleware lives in src/net/http/csrf.go. It works using the Sec-Fetch-Site HTTP header, which Can I Use shows as having 94.18% global availability - the holdouts are mainly IE11, iOS versions prior to iOS 17 (which came out in 2023 but can be installed on any phone released since 2017) and some other ancient browser versions.

If Sec-Fetch-Site is same-origin or none then the page submitting the form was either on the same origin or was navigated to directly by the user - in both cases safe from CSRF. If it's cross-site or same-site (tools.simonwillison.net and til.simonwillison.net are considered same-site but not same-origin) the submission is denied.

If that header isn't available the middleware falls back on comparing other headers: Origin - a value like https://simonwillison.net - with Host, a value like simonwillison.net. This should cover the tiny fraction of browsers that don't have the new header, though it's not clear to me if there are any weird edge-cases beyond that.

Note that this fallback comparison can't take the scheme into account since Host doesn't list that, so administrators are encouraged to use HSTS to protect against HTTP to HTTPS cross-origin requests.

On Lobste.rs I questioned if this would work for localhost, since that normally isn't served using HTTPS. Firefox security engineer Frederik Braun reassured me that *.localhost is treated as a Secure Context, so gets the Sec-Fetch-Site header despite not being served via HTTPS.

Update: Also relevant is Filippo Valsorda's article in CSRF which includes detailed research conducted as part of building the new Go middleware, plus this related Bluesky conversation about that research from six months ago.

Via lobste.rs

Tags: browsers, csrf, go, security, filippo-valsorda

Maintainers of Last Resort

2025-08-16T16:52:45+00:00

Maintainers of Last Resort

Filippo Valsorda founded Geomys last year as an "organization of professional open source maintainers", providing maintenance and support for critical packages in the Go language ecosystem backed by clients in retainer relationships.

This is an inspiring and optimistic shape for financially sustaining key open source projects, and it appears be working really well.

Most recently, Geomys have started acting as a "maintainer of last resort" for security-related Go projects in need of new maintainers. In this piece Filippo describes their work on the bluemonday HTML sanitization library - similar to Python’s bleach which was deprecated in 2023. He also talks at length about their work on CSRF for Go after gorilla/csrf lost active maintenance - I’m still working my way through his earlier post on Cross-Site Request Forgery trying to absorb the research shared their about the best modern approaches to this vulnerability.

Via lobste.rs

Tags: csrf, go, open-source, security, filippo-valsorda

GitHub OAuth for a static site using Cloudflare Workers

2024-11-29T18:13:18+00:00

GitHub OAuth for a static site using Cloudflare Workers

Here's a TIL covering a Thanksgiving AI-assisted programming project. I wanted to add OAuth against GitHub to some of the projects on my tools.simonwillison.net site in order to implement "Save to Gist".

That site is entirely statically hosted by GitHub Pages, but OAuth has a required server-side component: there's a client_secret involved that should never be included in client-side code.

Since I serve the site from behind Cloudflare I realized that a minimal Cloudflare Workers script may be enough to plug the gap. I got Claude on my phone to build me a prototype and then pasted that (still on my phone) into a new Cloudflare Worker and it worked!

... almost. On later closer inspection of the code it was missing error handling... and then someone pointed out it was vulnerable to a login CSRF attack thanks to failure to check the state= parameter. I worked with Claude to fix those too.

Useful reminder here that pasting code AI-generated code around on a mobile phone isn't necessarily the best environment to encourage a thorough code review!

Tags: csrf, github, oauth, projects, security, tools, ai, cloudflare, generative-ai, llms, ai-assisted-programming

Datasette 1.0a15

2024-08-16T05:06:51+00:00

Datasette 1.0a15

Mainly bug fixes, but a couple of minor new features:

Datasette now defaults to hiding SQLite "shadow" tables, as seen in extensions such as SQLite FTS and sqlite-vec. Virtual tables that it makes sense to display, such as FTS core tables, are no longer hidden. Thanks, Alex Garcia. (#2296)
The Datasette homepage is now duplicated at /-/, using the default index.html template. This ensures that the information on that page is still accessible even if the Datasette homepage has been customized using a custom index.html template, for example on sites like datasette.io. (#2393)

Datasette also now serves more user-friendly CSRF pages, an improvement which required me to ship asgi-csrf 0.10.

Tags: csrf, projects, releases, datasette

Exploring the SameSite cookie attribute for preventing CSRF

2021-08-03T21:09:02+00:00

In reading Yan Zhu's excellent write-up of the JSON CSRF vulnerability she found in OkCupid one thing puzzled me: I was under the impression that browsers these days default to treating cookies as SameSite=Lax, so I would expect attacks like the one Yan described not to work in modern browsers.

This lead me down a rabbit hole of exploring how SameSite actually works, including building an interactive SameSite cookie exploration tool along the way. Here's what I learned.

Background: Cross-Site Request Forgery

I've been tracking CSRF (Cross-Site Request Forgery) on this blog since 2005(!)

A quick review: let's say you have a page in your application that allows a user to delete their account, at https://www.example.com/delete-my-account. The user has to be signed in with a cookie in order to activate that feature.

If you created that page to respond to GET requests, I as an evil person could create a page at https://www.evil.com/force-you-to-delete-your-account that does this:

<img src="https://www.example.com/delete-my-account">

If I can get you to visit my page, I can force you to delete your account!

But you're smarter than that, and you know that GET requests should be idempotent. You implement your endpoint to require a POST request instead.

Turns out I can still force-delete accounts, if I can trick a user into visiting a page with the following evil HTML on it:

<form action="https://www.example.com/delete-my-account" method="POST">
<input type="submit" value="Delete my account">
</form>
<script>document.forms[0].submit()</script>

The form submits with JavaScript the instant they load the page!

CSRF is an extremely common and nasty vulnerability - especially since it's a hole by default: if you don't know what CSRF is, you likely have it in your application.

Traditionally the solution has been to use CSRF tokens - hidden form fields which "prove" that the user came from a form on your own site, and not a form hosted somewhere else. OWASP call this the Double Submit Cookie pattern.

Web frameworks like Django implement CSRF protection for you. I built asgi-csrf to help add CSRF token protection to ASGI applications.

Clearly it would be better if we didn't have to worry about CSRF at all.

As far as I can tell, work on specifying the SameSite cookie attribute started in June 2016. The idea was to add an additional attribute to cookies that specifies the policy for if they should be included in requests made to a domain from pages hosted on another domain.

Today, all modern browsers support SameSite. MDN has SameSite documentation, but a summary is:

SameSite=None - the cookie is sent in "all contexts" - more-or-less how things used to work before SameSite was invented. Update: One major edge-case here is that Safari apparently ignores None if the "Prevent cross-site tracking" privacy preference is turned on - and since that is on by default, this means that SameSite=None is effectively useless if you care about Safari or Mobile Safari users.
SameSite=Strict - the cookie is only sent for requests that originate on the same domain. Even arriving on the site from an off-site link will not see the cookie, unless you subsequently refresh the page or navigate within the site.
SameSite=Lax - cookie is sent if you navigate to the site through following a link from another domain but not if you submit a form. This is generally what you want to protect against CSRF attacks!

The attribute is specified by the server in a set-cookie header that looks like this:

set-cookie: lax-demo=3473; Path=/; SameSite=lax

Why not habitually use SameSite=Strict? Because then if someone follows a link to your site their first request will be treated as if they are not signed in at all. That's bad!

So explicitly setting a cookie with SameSite=Lax should be enough to protect your application from CSRF vulnerabilities... provided your users have a browser that supports it.

(Can I Use reports 93.95% global support for the attribute - not quite high enough for me to stop habitually using CSRF tokens, but we're getting there.)

What if the SameSite attribute is missing?

Here's where things get interesting. If a cookie is set without a SameSite attribute at all, how should the browser treat it?

Over the past year, all of the major browsers have been changing their default behaviour. The goal is for a missing SameSite attribute to be treated as if it was SameSite=Lax - providing CSRF protection by default.

I have found it infuriatingly difficult to track down if and when this change has been made:

Chrome/Chromium offer the best documentation - they claim to have ramped up the new default to 100% of users in August 2020. WebViews in Android still have the old default behaviour, which is scheduled to be fixed in Android 12 (not yet released).
Firefox have a blog entry from August 2020 which says "Starting with Firefox 79 (June 2020), we rolled it out to 50% of the Firefox Beta user base" - but I've not been able to find any subsequent updates. Update 26th August 2024: It turns out Firefox didn't ship this after all, going with their own Total Cookie Protection solution instead, which rolled out in April 2023.
I have no idea at all what's going on with Safari!

I started a Twitter thread to try and collect more information, so please reply there if you know what's going on in more detail.

The Chrome 2-minute twist

Assuming all of the above, the mystery remained: how did Yan's exploit fail to be prevented by browsers?

After some back-and-forth about this on Twitter Yan proposed that the answer may be this detail, tucked away on the Chrome Platform Status page for Feature: Cookies default to SameSite=Lax.

Note: Chrome will make an exception for cookies set without a SameSite attribute less than 2 minutes ago. Such cookies will also be sent with non-idempotent (e.g. POST) top-level cross-site requests despite normal SameSite=Lax cookies requiring top-level cross-site requests to have a safe (e.g. GET) HTTP method. Support for this intervention ("Lax + POST") will be removed in the future.

It looks like OkCupid were setting their authentication cookie without a SameSite attribute... which opened them up to a form-based CSRF attack but only for the 120 seconds following the cookie being set!

Building a tool to explore SameSite browser behaviour

I was finding this all very confusing, so I built a tool.

The code lives in simonw/samesite-lax-demo on GitHub, but the tool itself has two sides:

A server-side Python (Starlette) web application for setting cookies with different SameSite attributes. This is hosted on Vercel at https://samesite-lax-demo.vercel.app/
An HTML page on a different domain that links to that cookied site, provides a POST form targetting it, embeds an image from it and executes some fetch() requests against it. This is at https://simonw.github.io/samesite-lax-demo/

Hosting on two separate domains is critical for the tool to show what is going on. I chose Vercel and GitHub Pages because they are both trivial to set up to continuously deploy changes from a GitHub repository.

Using the tool in different browsers helps show exactly what is going on with regards to cross-domain cookies.

A few of the things I observed using the tool:

SameSite=Strict works as you would expect. It's particularly interesting to follow the regular <a href=...> link from the static site to the application and see how the strict cookie is NOT visible upon arrival - but becomes visible when you refresh that page.
I included a dynamically generated SVG in a <img src="/cookies.svg"> image tag, which shows the cookies (using SVG <text>) that are visible to the request. That image shows all four types of cookie when embedded on the Vercel domain, but when embedded on the GitHub pages domain it differs wildly:
- Firefox 89 shows both the SameSite=None and the missing SameSite cookies
- Chrome 92 shows just the SameSite=None cookie
- Safari 14.0 shows no cookies at all!
Chrome won't let you set a SameSite=None cookie without including the Secure attribute.
I also added some JavaScript that makes a cross-domain fetch(..., {credentials: "include"}) call against a /cookies.json endpoint. This didn't send any cookies at all until I added server-side headers access-control-allow-origin: https://simonw.github.io and access-control-allow-credentials: true. Having done that, I got the same results across the three browsers as for the <img test described above.

Safari ignoring SameSite=None looked like it was this bug: Cookies with SameSite=None or SameSite=invalid treated as Strict - it's marked as fixed but it's not clear to me if the fix has been released yet - I still saw that behaviour on my macOS 10.15.6 laptop or my iOS 14.7.1 iPhone.

Update: krinchan on Hacker News has an answer here:

The Safari "bug" is a new setting that's turned on by default: "Prevent cross-site tracking". It treats all cookies as SameSite=Lax, even cookies with SameSite=None.

Full Third-Party Cookie Blocking and More on the WebKit blog has more about this.

Most excitingly, I was able to replicate the Chrome two minute window bug using the tool! Each cookie has its value set to the timestamp when it was created, and I added code to display how many seconds ago the cookie was set. Here's an animation showing how Chrome on a form submission navigation can see the cookie that was set with SameSite missing at 114 seconds old, but that cookie is no longer visible once it passes 120 seconds.

Consider your subdomains

One last note about CSRF that you should consider: SameSite=Lax still allows form submissions from subdomains of your primary domain to carry their cookies.

This means that if you have a XSS vulnerability on one of your subdomains the security of your primary domain will be compromised.

Since it's common for subdomains to host other applications that may have their own security concerns, ditching CSRF tokens for Lax cookies may not be a wise step!

A Login CSRF attack is when a malicious forces a user to sign into an account controlled by the attacker. Why do this? Because if that user then saves sensitive information the attacker can see it.

Imagine I trick you into signing into an e-commerce account I control and saving your credit card details. I could then later sign in myself and buy things on your card!

Here's how that would work: Say the site's login form makes a POST to https://www.example.com/login with username and password as the form fields. If those credentials match, the site sets an authentication cookie.

I can set up my evil website with the following form:

<form action="https://www.example.com/login">
  <input type="hidden" name="username" value="my-username">
  <input type="hidden" name="password" value="my-password">
</form>
<script>document.forms[0].submit()</script>

I trick you into visiting my evil pge and you're now signed in to that site using an account that I control. I cross my fingers and hope you don't notice the "you are signed in as X" message in the UI.

An interesting thing about Login CSRF is that, since it involves setting a cookie but not sending a cookie, SameSite=Lax would seem to make no difference at all. You need to look to other mechanisms to protect against this attack.

But actually, you can use SameSite=Lax to prevent these. The trick is to only allow logins from users that are carrying at least one cookie which you have set in that way - since you know that those cookies could not have been sent if the user originated in a form on another site.

Another (potentially better) option: check the HTTP Origin header on the oncoming request.

Final recommendations

As an application developer, you should set all cookies with SameSite=Lax unless you have a very good reason not to. Most web frameworks do this by default now - Django shipped support for this in Django 2.1 in August 2018.

Do you still need CSRF tokens as well? I think so: I don't like the idea of users who fire up an older browser (maybe borrowing an obsolete computer) being vulnerable to this attack, and I worry about the subdomain issue described above.

And if you work for a browser vendor, please make it easier to find information on what the default behaviour is and when it was shipped!

Tags: chrome, cookies, csrf, security, samesite, starlette

OkCupid had a CSRF vulnerability

2021-08-02T22:12:36+00:00

OkCupid had a CSRF vulnerability

Good write-up of a (now fixed) CSRF vulnerability on OkCupid. Their site worked by POSTing JSON objects to an API. JSON POSTs are usually protected against CSRF because they can only be sent using fetch() or XMLHttpRequest, which are protected by the same-origin policy. Yan Zhu notes that you can use the enctype="text/plain" attribute on a form (introduced in HTML5) and a crafty hidden input element with name='{"foo":"' value='bar"}' to construct JSON in an off-site form, which enabled CSRF attacks.

Via How to boost your popularity on OkCupid using CSRF and a JSON type confusion on Hacker News

Tags: csrf, security

Datasette 0.58: The annotated release notes

2021-07-16T02:21:11+00:00

I released Datasette 0.58 last night, with new plugin hooks, Unix domain socket support, a major faceting performance fix and a few other improvements. Here are the annotated release notes.

Faceting performance improvement

Facets remains my favourite feature in Datasette: it turns out a simple group by / count against a column is one of the most productive ways I know of to start understanding new data.

Yesterday I stumbled across a tiny tweak (details in this issue) that gave me a 10x performance boost on facet queries! Short version: given the following example query:

select
  country_long,
  count(*)
from (
  select * from [global-power-plants]
  order by rowid
)
where
  country_long is not null
group by
  country_long
order by
  count(*) desc

Removing the unnecessary order by rowid from that inner query knocked the time down from 53ms to 7.2ms (and makes even more of a difference on larger tables).

I was surprised SQLite didn't perform that optimization automatically - so I started a thread on the SQLite forum and SQLite author D. Richard Hipp figured out a patch! It's not yet certain that it will land in a SQLite release but I'm excited to have found an issue interesting enough to be worth looking into. (UPDATE: it landed on trunk).

The get_metadata() plugin hook

New plugin hook: get_metadata(datasette, key, database, table), for returning custom metadata for an instance, database or table. Thanks, Brandon Roberts! (#1384)

Brandon Roberts contributed this hook as part of work he's been doing with Newsday nextLI - always exciting to see Datasette used by another news organization. Brandon has a live demo of the plugins he has been building: datasette-live-config, datasette-live-permissions, datasette-csv-importer and datasette-surveys. He also has a 6 minute demo video explaining the project so far.

The new hook allows plugins to customize the metadata displayed for different databases and tables within the Datasette interface.

There is one catch at the moment: the plugin doesn't yet allow for async calls (including await db.execute(sql)) because Datasette's own internals currently treat access to metadata as a sync rather than async feature.

There are workarounds for this. Brandon's datasette-live-config plugin opens an additional, synchronous connection to the DB which is completely fine for fast queries. Another option would be to keep metadata in an in-memory Python dictionary which is updated by SQL queries that run in an async background task.

In the longer run though I'd like to redesign Datasette's internals to support asynchronous metadata access - ideally before Datasette 1.0.

The skip_csrf() plugin hook

New plugin hook: skip_csrf(datasette, scope), for opting out of CSRF protection based on the incoming request. (#1377)

I wanted to write a plugin that supported an HTTP POST to a Datasette form that wasn't protected by Datasette's CSRF protection. This proved surprisingly difficult! I ended up shipping asgi-csrf 0.9 with a new mechanism for custom opting-out of CSRF protection based on the ASGI scope, then exposing that mechanism in a new plugin hook in Datasette.

CSRF is such a frustrating security issue to write code against, because in modern browsers the SameSite cookie attribute more-or-less solves the problem for you... but that attribute only has 90% global usage according to caniuse.com - not quite enough for me to forget about it entirely.

There also remains one obscure edge-case in which SameSite won't help you: the definition of "same site" includes other subdomains of your domain (provided it's not on the Public Suffix List). This means that for SameSite CSRF protection to work you need to be confident that no subdomains of your domain will suffer an XSS - and in my experience its common for subdomains to be pointed at third-party applications that may not have the same stringent XSS protection as your main code.

So I continue to care about CSRF protection in Datasette.

Unix domain socket support

New datasette --uds /tmp/datasette.sock option for binding Datasette to a Unix domain socket, see proxy documentation. (#1388)

I wrote about this in my weeknotes - this is a great way to run Datasette if you have it behind a proxy such as Apache or nginx and don't want to have the Datasette server listening on a high port.

"searchmode": "raw" in table metadata

"searchmode": "raw" table metadata option for defaulting a table to executing SQLite full-text search syntax without first escaping it, see Advanced SQLite search queries. (#1389)

SQLite's built in full-text search feature includes support for advanced operators: you can use operators like AND, OR and NEAR and you can add column specifiers like name:Simon to restrict searches to individual columns.

This is something of a two-edged sword: I've found innocent looking queries that raise errors due to unexpected interactions with the query language.

In issue 651 I switched to escaping all queries by default to prevent these errors from happening, with a ?_searchmode=raw query string option for opting back into the default functionality.

I've since had a few requests for a mechanism to enable this by default - hence the new "searchmode": "raw" option in table metadata.

Link plugin hooks now take a request

The menu_links(), table_actions() and database_actions() plugin hooks all gained a new optional request argument providing access to the current request. (#1371)

I have a plugin which needs to add links to different places depending on the subdomain that the Datasette instance is running on. Adding request to these plugin hooks proved to be the easiest way to achieve this.

This is a really nice thing about how Pluggy (the plugin library used by Datasette) works: adding new named parameters to hooks can be done without breaking backwards compatibility with existing plugins.

And the rest

Improved documentation for Running Datasette behind a proxy to recommend using ProxyPreservehost On with Apache. (#1387)
POST requests to endpoints that do not support that HTTP verb now return a 405 error.
db.path can now be provided as a pathlib.Path object, useful when writing unit tests for plugins. Thanks, Chris Amico. (#1365)

Tags: csrf, releasenotes, sqlite, datasette, annotated-release-notes, d-richard-hipp, samesite

Weeknotes: sqlite-utils updates, Datasette and asgi-csrf, open-sourcing VIAL

2021-06-28T17:23:21+00:00

Some work on sqlite-utils, asgi-csrf, a Datasette alpha and we open-sourced VIAL.

sqlite-utils

Last week's sqlite-utils 3.10 introduced a huge new feature: the ability to run joins directly against CSV and JSON files from the command-line.

I've since released sqlite-utils 3.11 and 3.12, much smaller releases.

3.11 added a new --schema option to the sqlite-utils memory command which lets you see the schema you'll be querying for the imported data:

$ curl 'https://api.github.com/users/dogsheep/repos' | \
  sqlite-utils memory - --schema
CREATE TABLE [stdin] (
   [id] INTEGER,
   [node_id] TEXT,
   [name] TEXT,
   [full_name] TEXT,
   [private] INTEGER,
   [owner] TEXT,
   [html_url] TEXT,
   [description] TEXT,
   ...
   [watchers] INTEGER,
   [default_branch] TEXT
);
CREATE VIEW t1 AS select * from [stdin];
CREATE VIEW t AS select * from [stdin];

3.12 focused on the Python library side of the package. It adds a new method, db.query(sql) which returns an iterator over Python dictionaries representing the results of a query.

This was a pretty obvious missing feature of the library: the rest of sqlite-utils deals with rows that are represented as dictionaries - you pass a list of Python dictionaries to db[table_name].insert_all(list_of_dicts) to create a table with the correct schema, for example. But if you wanted to execute SELECT queries you had to use db.execute() which would return a standard library cursor object which could then return tuples if you called .fetchall() on it.

It was only when I started to work on an interactive Jupyter notebook tutorial for sqlite-utils that I realized how weird it was not to have an equivalent method for reading data out of the database again.

Here's what the new method looks like:

db = Database(memory=True)
db["dogs"].insert_all([
    {"name": "Cleo"},
    {"name": "Pancakes"}
])
for row in db.query("select * from dogs"):
    print(row)
# Outputs:
# {'name': 'Cleo'}
# {'name': 'Pancakes'}

Full documentation here.

asgi-csrf and a Datasette alpha

I'm building a custom Datasette integration for a consulting client at the moment which needs to be able to accept POST form data as part of an API. Datasette has CSRF protection but for this particular project I need to opt-out of that protection for this one endpoint.

I ended up releasing asgi-csrf 0.9 with a new skip_if_scope= mechanism for dynamically disabling CSRF protection based on the incoming ASGI scope. I then shipped a Datasette 0.58a1 alpha release with a new skip_csrf(datasette, scope) plugin hook for plugins to take advantage of that mechanism.

Expect another alpha release shortly to preview the new get_metadata plugin hook contributed by Brandon Roberts. I've decided that alphas are the ideal way to explore new plugin hooks while they are still being developed as it lets projects pip install the alpha while making it clear that the interface may not yet be fully baked.

Open-sourcing VIAL

VIAL is the project I've been working on for VaccinateCA/VaccinateTheStates - see previous posts. It's a Django application which powers a crowd-sourced and scraper-driven effort to catalogue all of the places in the USA that you can get the Covid vaccine - 77,000 and counting right now.

We had always intended to open-source the code and now we have! github.com/CAVaccineInventory/vial is the newly-made-public repository.

I still need to produce a bunch of extra documentation about VIAL, likely including a video introduction to the project. But it's great to have it out there!

Releases this week

sqlite-utils: 3.12 - (80 releases total) - 2021-06-25
Python CLI utility and library for manipulating SQLite databases
datasette: 0.58a1 - (92 releases total) - 2021-06-24
An open source multi-tool for exploring and publishing data
asgi-csrf: 0.9 - (17 releases total) - 2021-06-23
ASGI middleware for protecting against CSRF attacks

TIL this week

Scraping Reddit via their JSON API

Tags: csrf, datasette, asgi, weeknotes, sqlite-utils, vaccinate-ca

Weeknotes: Rocky Beaches, Datasette 0.48, a commit history of my database

2020-08-21T00:52:16+00:00

This week I helped Natalie launch Rocky Beaches, shipped Datasette 0.48 and several releases of datasette-graphql, upgraded the CSRF protection for datasette-upload-csvs and figured out how to get a commit log of changes to my blog by backing up its database to a GitHub repository.

Rocky Beaches

Natalie released the first version of rockybeaches.com this week. It's a site that helps you find places to go tidepooling (known as rockpooling in the UK) and figure out the best times to go based on low tide times.

I helped out with the backend for the site, mainly as an excuse to further explore the idea of using Datasette to power full websites (previously explored with Niche Museums and my TILs).

The site uses a pattern I've been really enjoying: it's essentially a static dynamic site. Pages are dynamically rendered by Datasette using Jinja templates and a SQLite database, but the database itself is treated as a static asset: it's built at deploy time by this GitHub Actions workflow and deployed (currently to Vercel) as a binary asset along with the code.

The build script uses yaml-to-sqlite to load two YAML files - places.yml and stations.yml - and create the stations and places database tables.

It then runs two custom Python scripts to fetch relevant data for those places from iNaturalist and the NOAA Tides & Currents API.

The data all ends up in the Datasette instance that powers the site - you can browse it at www.rockybeaches.com/data or interact with it using GraphQL API at www.rockybeaches.com/graphql

The code is a little convoluted at the moment - I'm still iterating towards the best patterns for building websites like this using Datasette - but I'm very pleased with the productivity and performance that this approach produced.

Datasette 0.48

Highlights from Datasette 0.48 release notes:

Datasette documentation now lives at docs.datasette.io
The extra_template_vars, extra_css_urls, extra_js_urls and extra_body_script plugin hooks now all accept the same arguments. See extra_template_vars(template, database, table, columns, view_name, request, datasette) for details. (#939)
Those hooks now accept a new columns argument detailing the table columns that will be rendered on that page. (#938)

I released a new version of datasette-cluster-map that takes advantage of the new columns argument to only inject Leaflet maps JavaScript onto the page if the table being rendered includes latitude and longitude columns - previously the plugin would load extra code on pages that weren't going to render a map at all. That's now running on https://global-power-plants.datasettes.com/.

datasette-graphql

Using datasette-graphql for Rocky Beaches inspired me to add two new features:

A new graphql() Jinja custom template function that lets you execute custom GraphQL queries inside a Datasette template page - which turns out to be a pretty elegant way for the template to load exactly the data that it needs in order to render the page. Here's how Rocky Beaches uses that. Issue 50.
Some of the iNaturalist data that Rocky Beaches uses is stored as JSON data in text columns in SQLite - mainly because I was too lazy to model it out as tables. This was coming out of the GraphQL API as strings-containing-JSON, so I added a json_columns plugin configuration mechanism for turning those into Graphene GenericScalar fields - see issue 53 for details.

I also landed a big performance improvement. The plugin works by introspecting the database and generating a GraphQL schema that represents those tables, columns and views. For tables with a lot of tables this can get expensive, and the introspection was being run on every request.

I didn't want to require a server restart any time the schema changed, so I didn't want to cache the schema in-memory. Ideally it would be cached but the cache would become invalid any time the schema itself changed.

It turns out SQLite has a mechanism for this: the PRAGMA schema_version statement, which returns an integer version number that changes any time the underlying schema is changed (e.g. a table is added or modified).

I built a quick datasette-schema-versions plugin to try this feature out (in less than twenty minutes thanks to my datasette-plugin cookiecutter template) and prove to myself that it works. Then I built a caching mechanism for datasette-graphql that uses the current schema_version as the cache key. See issue 51 for details.

asgi-csrf and datasette-upload-csvs

datasette-upload-csvs is a Datasette plugin that adds a form for uploading CSV files and converting them to SQLite tables.

Datasette 0.44 added CSRF protection, which broke the plugin. I fixed that this week, but it took some extra work because file uploads use the multipart/form-data HTTP mechanism and my asgi-csrf library didn't support that.

I fixed that this week, but the code was quite complicated. Since asgi-csrf is a security library I decided to aim for 100% code coverage, the first time I've done that for one of my projects.

I got there with the help of codecov.io and pytest-cov. I wrote up what I learned about those tools in a TIL.

Backing up my blog database to a GitHub repository

I really like keeping content in a git repository (see Rocky Beaches and Niche Museums). Every content management system I've ever been has eventually desired revision control, and modeling that in a database and adding it to an existing project is always a huge pain.

I have 18 years of content on this blog. I want that backed up to git - and this week I realized I have the tools to do that already.

db-to-sqlite is my tool for taking any SQL Alchemy supported database (so far tested with MySQL and PostgreSQL) and exporting it into a SQLite database.

sqlite-diffable is a very early stage tool I built last year. The idea is to dump a SQLite database out to disk in a way that is designed to work well with git diffs. Each table is dumped out as newline-delimited JSON, one row per line.

So... how about converting my blog's PostgreSQL database to SQLite, then dumping it to disk with sqlite-diffable and committing the result to a git repository? And then running that in a GitHub Action?

Here's the workflow. It does exactly that, with a few extra steps: it only grabs a subset of my tables, and it redacts the password column from my auth_user table so that my hashed password isn't exposed in the backup.

I now have a commit log of changes to my blog's database!

I've set it to run nightly, but I can trigger it manually by clicking a button too.

TIL this week

Releases this week

datasette-graphql 0.14 - 2020-08-20
datasette-graphql 0.13 - 2020-08-19
datasette-schema-versions 0.1 - 2020-08-19
datasette-graphql 0.12.3 - 2020-08-19
github-to-sqlite 2.5 - 2020-08-18
datasette-publish-vercel 0.8 - 2020-08-17
datasette-cluster-map 0.12 - 2020-08-16
datasette 0.48 - 2020-08-16
datasette-graphql 0.12.2 - 2020-08-16
datasette-saved-queries 0.2.1 - 2020-08-15
datasette 0.47.3 - 2020-08-15
datasette-upload-csvs 0.5 - 2020-08-15
asgi-csrf 0.7 - 2020-08-15
asgi-csrf 0.7a0 - 2020-08-15
asgi-csrf 0.7a0 - 2020-08-15
datasette-cluster-map 0.11.1 - 2020-08-14
datasette-cluster-map 0.11 - 2020-08-14
datasette-graphql 0.12.1 - 2020-08-13

Tags: csrf, databases, git, github, natalie-downe, projects, graphql, datasette, inaturalist, weeknotes

Datasette 0.46

2020-08-09T16:57:58+00:00

Datasette 0.46

I just released Datasette 0.46 with a security fix for an issue involving CSRF tokens on canned query pages, plus a new debugging tool, improved file downloads and a bunch of other smaller improvements.

Via @simonw

Tags: csrf, projects, security, datasette

Weeknotes, I guess

2020-06-04T23:54:04+00:00

What a week. Hard to work up the enthusiasm to write about what I’ve been working on.

I’ve mainly been pushing towards shipping a Datasette release with writeable canned queries. This lead me down various other rabbit holes.

Authentication

Once you can write to a database, authentication and permissions become more than just a nice-to-have. I’ve used plugins for this in the past (datasette-auth-github and datasette-auth-existing-cookies), but to allow these plugins to work together with other features it makes sense to bring the concept of authentications and permission checks into Datasette core.

Issue #699 tracks my thinking on this. I've landed two new plugin hooks: actor_from_request, which lets plugins decide if the request is from an authenticated actor (a logged-in user or an authenticated API key of some sort) and permission_allowed which can answer if an actor is allowed to perform a specific action on a resource.

Flash messages

When you perform a write, I need to let you know if it worked or not. Django has messages, Flask calls them flash messages. I've now added these to Datasette, using signed cookies. The new /-/messages debug tool lets you try them out and see how they work.

CSRF protection

Still a work in progress (and a blocker on releasing the above new features). I shipped asgi-csrf 0.3 yesterday which is now ready for use in Datasette. The next step is to integrate it.

New milestone: Datasette 1.0

Writeable canned queries are the last major feature I want to add before Datasette 1.0. I've put some notes together on what this means: essentially I want 1.0 to signify that plugin builders and template authors can develop against Datasette with confidence that their stuff won't break until at least 2.0. I also started a Datasette 1.0 milestone.

Tags: csrf, datasette, weeknotes

Weeknotes: datasette-ics, datasette-upload-csvs, datasette-configure-fts, asgi-csrf

2020-03-04T02:27:47+00:00

I've been preparing for the NICAR 2020 Data Journalism conference this week which has lead me into a flurry of activity across a plethora of different projects and plugins.

datasette-ics

NICAR publish their schedule as a CSV file. I couldn't resist loading it into a Datasette on Glitch, which inspired me to put together a plugin I've been wanting for ages: datasette-ics, a register_output_renderer() plugin that can produce a subscribable iCalendar file from an arbitrary SQL query.

It's based on datasette-atom and works in a similar way: you construct a query that outputs a required set of columns (event_name and event_dtstart as a minimum), then add the .ics extension to get back an iCalendar file.

You can optionally also include event_dtend, event_duration, event_description, event_uid and most importantly event_tz, which can contain a timezone string. Figuring out how to handle timezones was the fiddliest part of the project.

If you're going to NICAR, subscribe to https://nicar-2020.glitch.me/data/calendar.ics in a calendar application to get the full 261 item schedule.

If you just want to see what the iCalendar feed looks like, add ?_plain=1 to preview it with a text/plain content type: https://nicar-2020.glitch.me/data/calendar.ics?_plain=1 - and here's the SQL query that powers it.

datasette-upload-csvs

My work on Datasette Cloud is inspiring all kinds of interesting work on plugins. I released datasette-upload-csvs a while ago, but now that Datasette has official write support I've been upgrading the plugin to hopefully achieve its full potential.

In particular, I've been improving its usability. CSV files can be big - and if you're uploading 100MB of CSV it's not particularly reassuring if your browser just sits for a few minutes spinning on the status bar.

So I added two progress bars to the plugins. The first is a client-side progress bar that shows you the progress of the initial file upload. I used the XMLHttpRequest pattern (and the drag-and-drop recipe) from Joseph Zimmerman's useful article How To Make A Drag-and-Drop File Uploader With Vanilla JavaScript - fetch() doesn't reliably report upload progres just yet.

I'm using Starlette and asyncio so uploading large files doesn't tie up server resources in the same way that it would if I was using processes and threads.

The second progress bar relates to server-side processing of the file: churning through 100,000 rows of CSV data and inserting them into SQLite can take a while, and I wanted users to be able to see what was going on.

Here's an animation screenshot of how the interface looks now:

Implementing this was trickier. In the end I took advantage of the new dedicaed write thread made available by datasette.execute_write_fn() - since that thread has exclusive access to write to the database, I create a SQLite table called _csv_progress_ and write a new record to it every 10 rows. I use the number of bytes in the CSV file as the total and track how far through that file Python's CSV parser has got using file.tell().

It seems to work really well. The full server-side code is here - the progress bar itself then polls Datasette's JSON API for the record in the _csv_progress_ table.

datasette-configure-fts

SQLite ships with a decent implementation of full-text search. Datasette knows how to tell if a table has been configured for full-text search and adds a search box to the table page, documented here.

datasette-configure-fts is a new plugin that provides an interface for configuring search against existing SQLite tables. Under the hood it uses the sqlite-utils full-text search methods to configure the table and set up triggers to keep the index updated as data in the table changes.

It's pretty simple, but it means that users of Datasette Cloud can upload a potentially enormous CSV file and then click to set specific columns as searchable. It's a fun example of the kind of things that can be built with Datasette`s new write capabilities.

asgi-csrf

CSRF is one of my favourite web application security vulnerabilties - I first wrote about it on this blog back in 2005!

I was surprised to see that the Starlette/ASGI ecosystem doesn't yet have much in the way of CSRF prevention. The best option I could find to use the WTForms library with Starlette.

I don't need a full forms library for my purposes (at least not yet) but I needed CSRF protection for datasete-configure-fts, so I've started working on a small ASGI middleware library called asgi-csrf.

It's modelled on a subset of Django's robust CSRF prevention. The README warns people NOT to trust it yet - there are still some OWASP recommendations that it needs to apply (issue here) and I'm not yet ready to declare it robust and secure. It's a start though, and feels like exactly the kind of problem that ASGI middleware is meant to address.

Tags: csrf, data-journalism, icalendar, plugins, projects, search, security, datasette, asgi, weeknotes, datasette-cloud

2020 Web Milestones

2020-01-24T04:43:16+00:00

2020 Web Milestones

A lot of stuff is happening in 2020! Mike Sherov rounds it up—highlights include the release of Chromium Edge (Microsoft’s Chrome-powered browser for Windows 7+), Web Components supported in every major browser, Deno 1.x, SameSite Cookies turned on by default (which should dramatically reduce CSRF exposure) and Python 2 and Flash EOLs.

Via @mikesherov

Tags: chrome, csrf, flash, internet-explorer, javascript, python, web, deno, samesite

Quoting Troy Hunt

2020-01-03T16:22:42+00:00

Come version 80, any cookie without a SameSite attribute will be treated as "Lax" by Chrome. This is really important to understand because put simply, it'll very likely break a bunch of stuff. [...] The fix is easy, all it needs is for everyone responsible for maintaining any system that uses cookies that might be passed from an external origin to understand what's going on. Can't be that hard, right? Hello? Oh...

— Troy Hunt

Tags: chrome, cookies, csrf, samesite, troy-hunt

OWASP Top 10 2007-2017: The Fall of CSRF

2018-08-06T22:02:39+00:00

OWASP Top 10 2007-2017: The Fall of CSRF

I was surprised to learn recently that CSRF didn’t make it into the 2017 OWASP Top 10 security vulnerabilities (after featuring almost every year since the list started). The credited reason is that web frameworks do a good enough job protecting against CSRF by default that it’s no longer a top-ten problem. Defaults really do matter.

Tags: csrf, owasp, security

What are key considerations when building behind the firewall web apps?

2013-09-15T15:24:00+00:00

My answer to What are key considerations when building behind the firewall web apps? on Quora

CSRF and XSS are still important: don't leave any security vulnerabilities which might allow an evil website out on the internet to run JavaScript that steals data from your behind-the-firewall web application.

Tags: csrf, enterprise, software-engineering, webapps, quora, saas

CSRF: Flash + 307 redirect = Game Over

2011-02-10T22:07:00+00:00

CSRF: Flash + 307 redirect = Game Over

Here’s the exploit that Django and Rails both just released fixes for. It’s actually a flaw in the Flash player. Flash isn’t meant to be able to make cross-domain HTTP requests with custom HTTP headers unless the crossdomain.xml file on the other domain allows them to, but it turns out a 307 redirect (like a 302, but allows POST data to be forwarded) confuses the Flash player in to not checking the crossdomain.xml on the host it is being redirect to.

Tags: crossdomainxml, csrf, django, flash, rails, security, recovered

Why do some people disable JavaScript in their browser?

2010-08-25T13:37:00+00:00

My answer to Why do some people disable JavaScript in their browser? on Quora

For security reasons.

Many (most?) web applications have security vulnerabilities due to JavaScript. The most common are XSS, where malicious JavaScript can be injected in to a page, stealing cookies and forcing users to perform actions that they did not intend, and CSRF, where unprotected forms can be abused to again perform unintended actions.

It's amazing how many developers are unaware of CSRF - and quite a lot still don't fully understand the consequences of XSS.

Disabling JavaScript in your browser makes the above attacks much harder to pull off. Once you understand how serious and widespread these problems are, it becomes very tempting to disable JavaScript for unknown sites and only enable it for sites you think have extremely skilled development teams.

That said, I don't personally disable JS in my browsers - but that's only because I haven't been bitten badly yet.

Tags: csrf, javascript, privacy, security, xss, quora

OpenCart CSRF Vulnerability

2010-05-25T00:00:00+00:00

OpenCart CSRF Vulnerability

Avoid OpenCart—it’s vulnerable to CSRF, but the maintainer has no intention of fixing it as “there is no way that I’m responsible for a client being stupid enough to click links in emails”.

Tags: csrf, security, recovered, opencart

Django 1.2 release notes

2010-05-17T21:11:00+00:00

Django 1.2 release notes

Released today, this is a terrific upgrade. Multiple database connections, model validation, improved CSRF protection, a messages framework, the new smart if template tag and lots, lots more. I’ve been using the 1.2 betas for a major new project over the past few months and it’s been smooth sailing all the way.

Via Django 1.2 released

Tags: csrf, django, multidb, open-source, python, releases, recovered

What's new in Django 1.2 alpha 1

2010-01-07T19:31:50+00:00

What's new in Django 1.2 alpha 1

Multiple database support, improved CSRF prevention, a messages framework (similar to the Rails “flash” feature), model validation, custom e-mail backends, template caching for much faster handling of the include and extends tags, read only fields in the admin, a better if tag and more. Very exciting release.

Via Django Weblog

Tags: alpha, csrf, django, django-admin, python, releases

Django ponies: Proposals for Django 1.2

2009-09-28T23:32:04+00:00

I've decided to step up my involvement in Django development in the run-up to Django 1.2, so I'm currently going through several years worth of accumulated pony requests figuring out which ones are worth advocating for. I'm also ensuring I have the code to back them up - my innocent AutoEscaping proposal a few years ago resulted in an enormous amount of work by Malcolm and I don't think he'd appreciate a repeat performance.

I'm not a big fan of branches when it comes to exploratory development - they're fine for doing the final implementation once an approach has been agreed, but I don't think they are a very effective way of discussing proposals. I'd much rather see working code in a separate application - that way I can try it out with an existing project without needing to switch to a new Django branch. Keeping code out of a branch also means people can start using it for real development work, making the API much easier to evaluate. Most of my proposals here have accompanying applications on GitHub.

I've recently got in to the habit of including an "examples" directory with each of my experimental applications. This is a full Django project (with settings.py, urls.py and manage.py files) which serves two purposes. Firstly, it allows developers to run the application's unit tests without needing to install it in to their own pre-configured project, simply by changing in to the examples directory and running ./manage.py test. Secondly, it gives me somewhere to put demonstration code that can be viewed in a browser using the runserver command - a further way of making the code easier to evaluate. django-safeform is a good example of this pattern.

Here's my current list of ponies, in rough order of priority.

Signing and signed cookies

Signing strings to ensure they have not yet been tampered with is a crucial technique in web application security. As with all cryptography, it's also surprisingly difficult to do correctly. A vulnerability in the signing implementation used to protect the Flickr API was revealed just today.

One of the many uses of signed strings is to implement signed cookies. Signed cookies are fantastically powerful - they allow you to send cookies safe in the knowledge that your user will not be able to alter them without you knowing. This dramatically reduces the need for sessions - most web apps use sessions for security rather than for storing large amounts of data, so moving that "logged in user ID" value to a signed cookie eliminates the need for session storage entirely, saving a round-trip to persistent storage on every request.

This has particularly useful implications for scaling - you can push your shared secret out to all of your front end web servers and scale horizontally, with no need for shared session storage just to handle simple authentication and "You are logged in as X" messages.

The latest version of my django-openid library uses signed cookies to store the OpenID you log in with, removing the need to configure Django's session storage. I've extracted that code in to django-signed, which I hope to evolve in to something suitable for inclusion in django.utils.

Please note that django-signed has not yet been vetted by cryptography specialists, something I plan to fix before proposing it for final inclusion in core.

django-signed on GitHub
Details of the Signing proposal on the Django wiki
Signing discussion on the django-developers mailing list

Improved CSRF support

This is mainly Luke Plant's pony, but I'm very keen to see it happen. Django has shipped with CSRF protection for more than three years now, but the approach (using middleware to rewrite form HTML) is relatively crude and, crucially, the protection isn't turned on by default. Hint: if you aren't 100% positive you are protected against CSRF, you should probably go and turn it on.

Luke's approach is an iterative improvement - a template tag (with a dependency on RequestContext) is used to output the hidden CSRF field, with middleware used to set the cookie and perform the extra validation. I experimented at length with an alternative solution based around extending Django's form framework to treat CSRF as just another aspect of validation - you can see the result in my django-safeform project. My approach avoids middleware and template tags in favour of a view decorator to set the cookie and a class decorator to add a CSRF check to the form itself.

While my approach works, the effort involved in upgrading existing code to it is substantial, compared to a much easier upgrade path for Luke's middleware + template tag approach. The biggest advantage of safeform is that it allows CSRF failure messages to be shown inline on the form, without losing the user's submission - the middleware check means showing errors as a full page without redisplaying the form. It looks like it should be possible to bring that aspect of safeform back to the middleware approach, and I plan to put together a patch for that over the next few days.

Luke's CSRF branch on bitbucket
My django-safeform on GitHub
Details of the CSRF proposal on the Django wiki
CSRF discussion on the django-developers mailing list

Better support for outputting HTML

This is a major pet peeve of mine. Django's form framework is excellent - one of the best features of the framework. There's just one thing that bugs me about it - it outputs full form widgets (for input, select and the like) so that it can include the previous value when redisplaying a form during validation, but it does so using XHTML syntax.

I have a strong preference for an HTML 4.01 strict doctype, and all those <self-closing-tags /> have been niggling away at me for literally years. Django bills itself as a framework for "perfectionists with deadlines", so I feel justified in getting wound up out of proportion over this one.

A year ago I started experimenting with a solution, and came up with django-html. It introduces two new Django template tags - {% doctype %} and {% field %}. The doctype tag serves two purposes - it outputs a particular doctype (saving you from having to remember the syntax) and it records that doctype in Django's template context object. The field tag is then used to output form fields, but crucially it gets to take the current doctype in to account.

The field tag can also be used to add extra HTML attributes to form widgets from within the template itself, solving another small frustration about the existing form library. The README describes the new tags in detail.

The way the tags work is currently a bit of a hack - if merged in to Django core they could be more cleanly implemented by refactoring the form library slightly. This refactoring is currently being discussed on the mailing list.

django-html on GitHub
Improved HTML discussion on the django-developers mailing list

Logging

This is the only proposal for which I don't yet have any code. I want to add official support for Python's standard logging framework to Django. It's possible to use this at the moment (I've done so on several projects) but it's not at all clear what the best way of doing so is, and Django doesn't use it internally at all. I posted a full argument in favour of logging to the mailing list, but my favourite argument is this one:

Built-in support for logging reflects a growing reality of modern Web development: more and more sites have interfaces with external web service APIs, meaning there are plenty of things that could go wrong that are outside the control of the developer. Failing gracefully and logging what happened is the best way to deal with 3rd party problems - much better than throwing a 500 and leaving no record of what went wrong.

I'm not actively pursuing this one yet, but I'm very interesting in hearing people's opinions on the best way to configure and use the Python logging module in production.

A replacement for get_absolute_url()

Django has a loose convention of encouraging people to add a get_absolute_url method to their models that returns that object's URL. It's a controversial feature - for one thing, it's a bit of a layering violation since URL logic is meant to live in the urls.py file. It's incredibly convenient though, and since it's good web citizenship for everything to have one and only one URL I think there's a pretty good argument for keeping it.

The problem is, the name sucks. I first took a look at this in the last few weeks before the release of Django 1.0 - what started as a quick proposal to come up with a better name before we were stuck with it quickly descended in to a quagmire as I realised quite how broken get_absolute_url() is. The short version: in some cases it means "get a relative URL starting with /", in other cases it means "get a full URL starting with http://" and the name doesn't accurately describe either.

A full write-up of my investigation is available on the Wiki. My proposed solution was to replace it with two complementary methods - get_url() and get_url_path() - with the user implementing one hence allowing the other one to be automatically derived. My django-urls project illustrates the concept via a model mixin class. A year on I still think it's quite a neat idea, though as far as I can tell no one has ever actually used it.

ReplacingGetAbsoluteUrl on the wiki
django-urls on GitHub
Recent get_absolute_url discussion on the django-developers mailing list

Comments on this post are open, but if you have anything to say about any of the individual proposals it would be much more useful if you posted it to the relevant mailing list thread.

Tags: cookies, cryptography, csrf, django, html, logging, luke-plant, markup, ponies, projects, python, security, signedcookies, signing, xhtml

Amazon Says Listing Problem Was an Error, Not a Hack

2009-04-14T08:32:40+00:00

Amazon Says Listing Problem Was an Error, Not a Hack

“A friend within the company told him that someone working on Amazon’s French site mistagged a number of keyword categories, including the ’Gay and Lesbian’ category, as pornographic, using what’s known internally as the Browse Nodes tool. Soon the mistake affected Amazon sites worldwide.”

Via Chris Shiflett

Tags: amazon, amazonfail, csrf, security

How to cause moral outrage from the entire Internet in ten lines of code

2009-04-13T19:48:53+00:00

How to cause moral outrage from the entire Internet in ten lines of code

Looks legit—the author claims to have sparked this weekend’s #amazonfail moral outrage (where Amazon where accused of removing Gay and Lesbian books from their best seller rankings) by exploiting a CSRF hole in Amazon’s “report as inappropriate” feature to trigger automatic takedowns. EDIT: His claim is disputed elsewhere (see comments)

Tags: amazon, amazonfail, csrf, prdisaster, security

17-year-old claims responsibility for Twitter worm

2009-04-12T19:22:19+00:00

17-year-old claims responsibility for Twitter worm

It was a text book XSS attack—the URL on the user profile wasn’t properly escaped, allowing an attacker to insert a script element linking out to externally hosted JavaScript which then used Ajax to steal any logged-in user’s anti-CSRF token and use it to self-replicate in to their profile.

Tags: csrf, security, twitter, worms, xss

Quoting Roy Fielding

2009-01-23T08:14:33+00:00

CSRF is not a security issue for the Web. A well-designed Web service should be capable of receiving requests directed by any host, by design, with appropriate authentication where needed. If browsers create a security issue because they allow scripts to automatically direct requests with stored security credentials onto third-party sites, without any user intervention/configuration, then the obvious fix is within the browser.

— Roy Fielding

Tags: browsers, credentials, csrf, royfielding, security

Quoting Jeremiah Grossman

2008-11-03T12:43:34+00:00

When visiting any Web page, the site owner is easily able to ascertain what websites you've visited (CSS color hacks) or places you're logged-in (JavaScript errors / IMG loading behavior). They can also automatically exploit your online bank, social network, and webmail accounts (XSS). Additionally, the browser could be instructed to hack devices on the intranet, including DSL routers and printers. And, if that's not enough, they could turn you into a felon by forcing requests to illegal content or hack other sites (CSRF).

— Jeremiah Grossman

Tags: csrf, jeremiah-grossman, security, xss

Web Security Horror Stories: The Director's Cut

2008-10-26T12:15:33+00:00

Web Security Horror Stories: The Director's Cut

Slides from the talk on web application security I gave this morning at <head>, the worldwide online conference. I just about managed to resist the temptation to present in my boxers. Topics include XSS, CSRF, Login CSRF and Clickjacking.

Tags: clickjacking, csrf, logincsrf, security, xss

Quoting Bill Zeller

2008-09-29T13:11:23+00:00

We've found CSRF vulnerabilities in sites that have a huge incentive to do security correctly. If you're in charge of a website and haven't specifically protected against CSRF, chances are you're vulnerable.

— Bill Zeller

Tags: bill-zeller, csrf, security

Simon Willison's Weblog: csrf

datasette PR #2689: Replace token-based CSRF with Sec-Fetch-Site header protection

A modern approach to preventing CSRF in Go

Maintainers of Last Resort

GitHub OAuth for a static site using Cloudflare Workers

Datasette 1.0a15

Exploring the SameSite cookie attribute for preventing CSRF

Background: Cross-Site Request Forgery

Enter the SameSite cookie attribute

What if the SameSite attribute is missing?

The Chrome 2-minute twist

Building a tool to explore SameSite browser behaviour

Consider your subdomains

Login CSRF and SameSite=Lax

Final recommendations

OkCupid had a CSRF vulnerability

Datasette 0.58: The annotated release notes

Faceting performance improvement

The get_metadata() plugin hook

The skip_csrf() plugin hook

Unix domain socket support

"searchmode": "raw" in table metadata

Link plugin hooks now take a request

And the rest

Weeknotes: sqlite-utils updates, Datasette and asgi-csrf, open-sourcing VIAL

sqlite-utils

asgi-csrf and a Datasette alpha

Open-sourcing VIAL

Releases this week

TIL this week

Weeknotes: Rocky Beaches, Datasette 0.48, a commit history of my database

Rocky Beaches

Datasette 0.48

datasette-graphql

asgi-csrf and datasette-upload-csvs

Backing up my blog database to a GitHub repository

TIL this week

Releases this week

Datasette 0.46

Weeknotes, I guess

Authentication

Flash messages

CSRF protection

New milestone: Datasette 1.0

Weeknotes: datasette-ics, datasette-upload-csvs, datasette-configure-fts, asgi-csrf

datasette-ics

datasette-upload-csvs

datasette-configure-fts

asgi-csrf

2020 Web Milestones

Quoting Troy Hunt

OWASP Top 10 2007-2017: The Fall of CSRF

What are key considerations when building behind the firewall web apps?

CSRF: Flash + 307 redirect = Game Over

Why do some people disable JavaScript in their browser?

OpenCart CSRF Vulnerability

Django 1.2 release notes

What's new in Django 1.2 alpha 1

Django ponies: Proposals for Django 1.2

Signing and signed cookies

Improved CSRF support

Better support for outputting HTML

Logging

A replacement for get_absolute_url()

Amazon Says Listing Problem Was an Error, Not a Hack

How to cause moral outrage from the entire Internet in ten lines of code

17-year-old claims responsibility for Twitter worm

Quoting Roy Fielding

Quoting Jeremiah Grossman

Web Security Horror Stories: The Director's Cut

Quoting Bill Zeller