Simon Willison's Weblog: regular-expressions

TRE Python binding — ReDoS robustness demo

2026-05-04T17:52:00+00:00

Research: TRE Python binding — ReDoS robustness demo

If it's good enough for antirez to add to Redis I figured Ville Laurikari's TRE regular expression engine was worth exploring in a little more detail.

I had Claude Code build an experimental Python binding (it used ctypes) and try some malicious regular expression attacks against the library. TRE handles those much better than Python's standard library implementation, thanks mainly to the lack of support for backtracking.

Tags: c, ctypes, python, regular-expressions, security

Redis Array Playground

2026-05-04T15:53:57+00:00

Tool: Redis Array Playground

Salvatore Sanfilippo submitted a PR adding a new data type - arrays - to Redis.

The new commands are ARCOUNT, ARDEL, ARDELRANGE, ARGET, ARGETRANGE, ARGREP, ARINFO, ARINSERT, ARLASTITEMS, ARLEN, ARMGET, ARMSET, ARNEXT, AROP, ARRING, ARSCAN, ARSEEK, ARSET.

The implementation is currently available in a branch, so I had Claude Code for web build this interactive playground for trying out the new commands in a WASM-compiled build of a subset of Redis running in the browser.

The most interesting new command is ARGREP which can run a server-side grep against a range of values in the array using the newly vendored TRE regex library.

Salvatore wrote more about the AI-assisted development process for the array type in Redis array type: short story of a long development.

Tags: c, redis, regular-expressions, salvatore-sanfilippo, ai, webassembly, generative-ai, llms, agentic-engineering

MicroQuickJS

2025-12-23T20:53:40+00:00

MicroQuickJS

New project from programming legend Fabrice Bellard, of ffmpeg and QEMU and QuickJS and so much more fame:

MicroQuickJS (aka. MQuickJS) is a Javascript engine targetted at embedded systems. It compiles and runs Javascript programs with as low as 10 kB of RAM. The whole engine requires about 100 kB of ROM (ARM Thumb-2 code) including the C library. The speed is comparable to QuickJS.

It supports a subset of full JavaScript, though it looks like a rich and full-featured subset to me.

One of my ongoing interests is sandboxing: mechanisms for executing untrusted code - from end users or generated by LLMs - in an environment that restricts memory usage and applies a strict time limit and restricts file or network access. Could MicroQuickJS be useful in that context?

I fired up Claude Code for web (on my iPhone) and kicked off an asynchronous research project to see explore that question:

My full prompt is here. It started like this:

Clone https://github.com/bellard/mquickjs to /tmp

Investigate this code as the basis for a safe sandboxing environment for running untrusted code such that it cannot exhaust memory or CPU or access files or the network

First try building python bindings for this using FFI - write a script that builds these by checking out the code to /tmp and building against that, to avoid copying the C code in this repo permanently. Write and execute tests with pytest to exercise it as a sandbox

Then build a "real" Python extension not using FFI and experiment with that

Then try compiling the C to WebAssembly and exercising it via both node.js and Deno, with a similar suite of tests [...]

I later added to the interactive session:

Does it have a regex engine that might allow a resource exhaustion attack from an expensive regex?

(The answer was no - the regex engine calls the interrupt handler even during pathological expression backtracking, meaning that any configured time limit should still hold.)

Here's the full transcript and the final report.

Some key observations:

MicroQuickJS is very well suited to the sandbox problem. It has robust near and time limits baked in, it doesn't expose any dangerous primitive like filesystem of network access and even has a regular expression engine that protects against exhaustion attacks (provided you configure a time limit).
Claude span up and tested a Python library that calls a MicroQuickJS shared library (involving a little bit of extra C), a compiled a Python binding and a library that uses the original MicroQuickJS CLI tool. All of those approaches work well.
Compiling to WebAssembly was a little harder. It got a version working in Node.js and Deno and Pyodide, but the Python libraries wasmer and wasmtime proved harder, apparently because "mquickjs uses setjmp/longjmp for error handling". It managed to get to a working wasmtime version with a gross hack.

I'm really excited about this. MicroQuickJS is tiny, full featured, looks robust and comes from excellent pedigree. I think this makes for a very solid new entrant in the quest for a robust sandbox.

Update: I had Claude Code build tools.simonwillison.net/microquickjs, an interactive web playground for trying out the WebAssembly build of MicroQuickJS, adapted from my previous QuickJS plaground. My QuickJS page loads 2.28 MB (675 KB transferred). The MicroQuickJS one loads 303 KB (120 KB transferred).

Here are the prompts I used for that.

Tags: c, javascript, nodejs, python, regular-expressions, sandboxing, ai, webassembly, deno, pyodide, generative-ai, llms, claude-code, fabrice-bellard

New dashboard: alt text for all my images

2025-04-28T01:22:27+00:00

New dashboard: alt text for all my images

I got curious today about how I'd been using alt text for images on my blog, and realized that since I have Django SQL Dashboard running on this site and PostgreSQL is capable of parsing HTML with regular expressions I could probably find out using a SQL query.

I pasted my PostgreSQL schema into Claude and gave it a pretty long prompt:

Give this PostgreSQL schema I want a query that returns all of my images and their alt text. Images are sometimes stored as HTML image tags and other times stored in markdown.

blog_quotation.quotation, blog_note.body both contain markdown. blog_blogmark.commentary has markdown if use_markdown is true or HTML otherwise. blog_entry.body is always HTML

Write me a SQL query to extract all of my images and their alt tags using regular expressions. In HTML documents it should look for either <img .* src="..." .* alt="..." or <img alt="..." .* src="..." (images may be self-closing XHTML style in some places). In Markdown they will always be ![alt text](url)

I want the resulting table to have three columns: URL, alt_text, src - the URL column needs to be constructed as e.g. /2025/Feb/2/slug for a record where created is on 2nd feb 2025 and the slug column contains slug

Use CTEs and unions where appropriate

It almost got it right on the first go, and with a couple of follow-up prompts I had the query I wanted. I also added the option to search my alt text / image URLs, which has already helped me hunt down and fix a few old images on expired domain names. Here's a copy of the finished 100 line SQL query.

Tags: accessibility, alt-text, postgresql, regular-expressions, sql, ai, django-sql-dashboard, generative-ai, llms, ai-assisted-programming, claude

tc39/proposal-regex-escaping

2025-02-18T21:53:56+00:00

tc39/proposal-regex-escaping

I just heard from Kris Kowal that this proposal for ECMAScript has been approved for ECMA TC-39:

Almost 20 years later, @simon’s RegExp.escape idea comes to fruition. This reached “Stage 4” at ECMA TC-39 just now, which formalizes that multiple browsers have shipped the feature and it’s in the next revision of the JavaScript specification.

I'll be honest, I had completely forgotten about my 2006 blog entry Escaping regular expression characters in JavaScript where I proposed that JavaScript should have an equivalent of the Python re.escape() function.

It turns out my post was referenced in this 15 year old thread on the esdiscuss mailing list, which evolved over time into a proposal which turned into implementations in Safari, Firefox and soon Chrome - here's the commit landing it in v8 on February 12th 2025.

One of the best things about having a long-running blog is that sometimes posts you forgot about over a decade ago turn out to have a life of their own.

Tags: blogging, ecmascript, javascript, regular-expressions, standards

datasette-enrichments-re2 0.1a0

2023-11-28T02:07:14+00:00

Release: datasette-enrichments-re2 0.1a0

Tags: regular-expressions, datasette

Why I invented "dash encoding", a new encoding scheme for URL paths

2022-03-05T21:50:38+00:00

Datasette now includes its own custom string encoding scheme, which I've called dash encoding. I really didn't want to have to invent something new here, but unfortunately I think this is the best solution to my very particular problem. Some notes on how dash encoding works and why I created it.

Update 18th March 2022: This turned out not to be the right idea for my project after all! I ended up settling on a Tilde encoding scheme instead.

Table names and rows in URLs

I've put a lot of thought into the design of Datasette's URLs.

Datasette exposes relational databases tables, as both web pages and a JSON API.

Consider a database in a SQLite file called legislators.db, containing a table called legislator_terms (example from this tutorial). The URL path to the web interface for that table will be:

/legislators/legislator_terms

And the JSON API will be here:

/legislators/legislator_terms.json

(Worth noting that Datasette supports other formats here too - CSV by default, and plugins can add more formats such as GeoJSON or Atom or iCal.)

Datasette also provides pages (and APIs) for individual rows, identified by their primary key:

For tables with compound primary keys, these pages can include the primary key values separated by commas:

/fixtures/compound_three_primary_keys/a,a,a

This is all pretty straightforward so far. But now we get to the challenge: what if a table's name or a row's primary key contains a forward slash or a period character?

This could break the URL scheme!

SQLite table names are allowed to contain almost any character, and Datasette is designed to work with any existing SQLite database - so I can't guarantee that a table with one of those characters won't need to be handled.

Consider a database with two tables - one called legislator_terms and another called legislator_terms/1 - given the URL /legislators/legislator_terms/1 it's no longer clear if it refers to the table with that name or the row with primary key 1 in the other table!

A similar problem exists for table names with as legislators.csv - which end in a format. Or primary key string values that end in .json.

Why URL encoding doesn't work here

Up until now, Datasette has solved this problem using URL percent encoding. This provides a standard mechanism for encoding "special" characters in URLs.

legislator_terms/1 encodes to legislator_terms%2F1

This should be enough to solve the problem. The URL to that weirdly named table can now be:

/legislators/legislator_terms%2F1

When routing the URL, the application can take this into account and identify that this it a table named legislator_terms/1, as opposed to a request for the row with ID 1 in the legislator_terms table.

There are two remaining problems.

Firstly, the "." character is ignored by URL encoding, so we still can't tell the difference between /db/table.json and a table called table.json. I worked around this issue in Datasette by supporting an optional alternative ?_format=json parameter, but it's messy and confusing.

Much more seriously, it turns out there are numerous common pieces of web infrastructure that "helpfully" decode escaped characters in URLs before passing them on to the underlying web application!

I first encountered this in the ASGI standard itself, which decoded characters in the path field before they were passed to the rest of the application.I submitted a PR adding raw_path to ASGI precisely to work around this problem for Datasette.

Over time though, the problem kept cropping up. Datasette aims to run on as many hosting platforms as possible. I've seen URL escaping applied at a higher level enough times now to be very suspicious of any load balancer or proxy or other web server mechanism that might end up executing between Datasette and the rest of the web.

Update: Flask core maintainer David Lord confirms on Twitter that this is a long-standing known problem:

This behavior in Apache/nginx/etc is why WSGI/ASGI can't specify "literal URL the user typed in", because anything in front of the app might modify slashes or anything else. So all the spec can provide is "decoded URL".

So, I need a way of encoding a table name that might include / and . characters in a way that will survive some other layer of the stack decoding URL encoded strings in the URL path before Datasette gets to see them!

Introducing dash encoding

That's where dash encoding comes in. I tried to design the fastest, simplest encoding mechanism I could that would solve this very specific problem.

Loose requirements:

Reversible - it's crucial to at any possible value survives a round-trip through the encoding
Avoid changing the string at all if possible. Otherwise I could use something like base64, but I wanted to keep the name in the URL as close to readable as possible
Survive interference by proxies and load balancer that might try to be helpful
Fast to apply the transformation
As simple as possible
Easy to implement, including in languages other than Python

Dash encoding consists of three simple steps:

Replace all single hyphen characters - with two hyphens --
Replace any forward slash / character with hyphen forward slash -/
Replace any period character . with hyphen period -.

To reverse the encoding, run those steps backwards.

Here the Python implementation of this encoding scheme:

def dash_encode(s: str) -> str:
     "Returns dash-encoded string - for example ``/foo/bar`` -> ``-/foo-/bar``"
     return s.replace("-", "--").replace(".", "-.").replace("/", "-/")

def dash_decode(s: str) -> str:
     "Decodes a dash-encoded string, so ``-/foo-/bar`` -> ``/foo/bar``"
     return s.replace("-/", "/").replace("-.", ".").replace("--", "-")

And the pytest tests for it:

@pytest.mark.parametrize(
     "original,expected",
     (
         ("abc", "abc"),
         ("/foo/bar", "-/foo-/bar"),
         ("/-/bar", "-/---/bar"),
         ("-/db-/table---.csv-.csv", "---/db---/table-------.csv---.csv"),
     ),
 )
 def test_dash_encoding(original, expected):
     actual = utils.dash_encode(original)
     assert actual == expected
     # And test round-trip
     assert original == utils.dash_decode(actual)

Here's the full commit.

This meets my requirements.

Capturing these with a regular expression

There was one remaining challenge. Datasette uses regular expressions - inspired by Django - to route requests to the correct page.

I wanted to use a regular expression to extract out dash encoded values, that could also distinguish them from / and - and . characters that were not encoded in that way.

Here's the pattern I came up with for strings matching this pattern:

([^\/\-\.]*|(\-/)|(\-\.)|(\-\-))*

Broken down:

[^\/\-\.]* means 0 or more characters that are NOT one of . or / or - - since we don't care about those characters at all
(\-/) means the explicit sequence -/
(\-\.) means the explicit sequence -.
(\-\-) means the explicit sequence --
Those four are wrapped in a group combined with the | or operator
The group is then wrapped in a (..)* - specifying that it can repeat as many times as you like

A better way to break down this regular expression is visually, using Debuggex:

Combining this into the full regular expression that matches a /database/table.format path is even messier, due to the need to add non-capturing group syntax (?:..) and named groups (?P<name>...) - it ends up looking like this:

^/(?P<database>[^/]+)/(?P<table>(?:[^\/\-\.]*|(?:\-/)*|(?:\-\.)*|(?:\-\-)*)*?)\.(?P<format>\w+)?$

Visualized with Debuggex:

Update: Thanks to suggestions from Matthew Somerville I simplified this further to:

^/(?P<database>[^/]+)/(?P<table>[^\/\-\.]*|\-/|\-\.|\-\-)*(?P<format>\.\w+)?$

Next steps: implementation

I'm currently working on integrating it into Datasette in this PR. The full history of my thinking around this problem can be found in issue 1439, with comments stretching back to August last year!

Tags: regular-expressions, urls, datasette

Search and replace with regular expressions in VS Code

2021-08-02T21:30:11+00:00

TIL: Search and replace with regular expressions in VS Code

Tags: regular-expressions

datasette-ripgrep: deploy a regular expression search engine for your source code

2020-11-28T06:51:06+00:00

This week I built datasette-ripgrep - a web application for running regular expression searches against source code, built on top of the amazing ripgrep command-line tool.

datasette-ripgrep demo

I've deployed a demo version of the application here:

ripgrep.datasette.io/-/ripgrep?pattern=pytest

The demo runs searches against the source code of every one of my GitHub repositories that start with datasette - 61 repos right now - so it should include all of my Datasette plugins plus the core Datasette repository itself.

Since it's running on top of ripgrep, it supports regular expressions. This is absurdly useful. Some examples:

Every usage of the .plugin_config( method: plugin_config\(
Everywhere I use async with httpx.AsyncClient (usually in tests): async with.*AsyncClient
All places where I use a Jinja | filter inside a variable: \{\{.*\|.*\}\}

I usually run ripgrep as rg on the command-line, or use it within Visual Studio Code (fun fact: the reason VS Code's "Find in Files" is so good is it's running ripgrep under the hood).

So why have it as a web application? Because this means I can link to it, bookmark it and use it on my phone.

Why build this?

There are plenty of great existing code search tools out there already: I've heard great things about livegrep, and a quick Google search shows a bunch of other options.

Aside from being a fun project, datasette-ripgrep has one key advantage: it gets to benefit from Datasette's publishing mechanism, which means it's really easy to deploy.

That ripgrep.datasette.io demo is deployed by checking out the source code to be searched into a all directory and then using the following command:

datasette publish cloudrun \
    --metadata metadata.json \
    --static all:all \
    --install=datasette-ripgrep \
    --service datasette-ripgrep \
    --apt-get-install ripgrep

all is a folder containing the source code to be searched. metadata.json contains this:

{
    "plugins": {
        "datasette-ripgrep": {
            "path": "/app/all",
            "time_limit": 3.0
        }
    }
}

That's all there is to it! The result is a deployed code search engine, running on Google Cloud Run.

(If you want to try this yourself you'll need to be using the just-released Datasette 0.52.)

The GitHub Action workflow that deploys the demo also uses my github-to-sqlite tool to fetch my repos and then shallow-clones the ones that begin with datasette.

If you have your own Google Cloud Run credentials, you can run your own copy of that workflow against your own repositories.

A different kind of Datasette plugin

Datasette is a tool for publishing SQLite databases, so most Datasette plugins integrate with SQLite in some way.

datasette-ripgrep is different: it makes no use of SQLite at all, but instead takes advantage of Datasette's URL routing, datasette publish deployments and permissions system.

The plugin implementation is currently 134 lines of code, excluding tests and templates.

While the plugin doesn't use SQLite, it does share a common philosophy with Datasette: the plugin bundles the source code that it is going to search as part of the deployed application, in a similar way to how Datasette usually bundles one or more SQLite database files.

As such, it's extremely inexpensive to run and can be deployed to serverless hosting. If you need to scale it, you can run more copies.

This does mean that the application needs to be re-deployed to pick up changes to the searchable code. I'll probably set my demo to do this on a daily basis.

Controlling processes from asyncio

The trickiest part of the implementation was figuring out how to use Python's asyncio.create_subprocess_exec() method to safely run the rg process in response to incoming requests.

I don't want expensive searches to tie up the server, so I implemented two limits here. The first is a time limit: by default, searches have a second to run after which the rg process will be terminated and only results recieved so far will be returned. This is achieved using the asyncio.wait_for() function.

I also implemented a limit on the number of matching lines that can be returned, defaulting to 2,000. Any more than that and the process is terminated early.

Both of these limits can be customized using plugin settings (documented in the README). You can see how they are implemented in the async def run_ripgrep(pattern, path, time_limit=1.0, max_lines=2000) function.

Highlighted linkable line numbers

The other fun implementation detail is the way the source code listings are displayed. I'm using CSS to display the line numbers in a way that makes them visible without them breaking copy-and-paste (inspired by this article by Sylvain Durand).

code:before {
    content: attr(data-line);
    display: inline-block;
    width: 3.5ch;
    -webkit-user-select: none;
    color: #666;
}

The HTML looks like this:

<pre><code id="L1" data-line="1">from setuptools import setup</code>
<code id="L2" data-line="2">import os</code>
<code id="L3" data-line="3">&nbsp;</code>
<code id="L4" data-line="4">VERSION = &#34;0.1&#34;</code>
...

I wanted to imitate GitHub's handling of line links, where adding #L23 to the URL both jumps to that line and causes the line to be highlighted. Here's a demo of that - I use the following JavaScript to update the contents of a <style id="highlightStyle"></style> element in the document head any time the URL fragment changes:

<script>
var highlightStyle = document.getElementById('highlightStyle');
function highlightLineFromFragment() {
    if (/^#L\d+$/.exec(location.hash)) {
        highlightStyle.innerText = `${location.hash} { background-color: yellow; }`;
    }
}
highlightLineFromFragment();
window.addEventListener("hashchange", highlightLineFromFragment);
</script>

It's the simplest way I could think of to achieve this effect.

Update 28th November 2020: Louis Lévêque on Twitter suggested using the CSS :target selector instead, which is indeed MUCH simpler - I deleted the above JavaScript and replaced it with this CSS:

:target {
    background-color: #FFFF99;
}

Next steps for this project

I'm pleased to have got datasette-ripgrep to a workable state, and I'm looking forward to using it to answer questions about the growing Datasette ecosystem. I don't know how much more time I'll invest in this - if it proves useful then I may well expand it.

I do think there's something really interesting about being able to spin up this kind of code search engine on demand using datasette publish. It feels like a very useful trick to have access to.

Better URLs for my TILs

My other project this week was an upgrade to til.simonwillison.net: I finally spent the time to design nicer URLs for the site.

Before:

til.simonwillison.net/til/til/javascript_manipulating-query-params.md

After:

til.simonwillison.net/javascript/manipulating-query-params

The implementation for this takes advantage of a feature I sneaked into Datasette 0.49: Path parameters for custom page templates. I can create a template file called pages/{topic}/{slug}.html and Datasette use that template to handle 404 errors that match that pattern.

Here's the new pages/{topic}/{slug}.html template for my TIL site. It uses the sql() template function from the datasette-template-sql plugin to retrieve and render the matching TIL, or raises a 404 if no TIL can be found.

I also needed to setup redirects from the old pages to the new ones. I wrote a TIL on edirects for Datasette explaining how I did that.

TIL this week

Redirects for Datasette

Releases this week

datasette-ripgrep 0.2 - 2020-11-27
datasette-ripgrep 0.1 - 2020-11-26
datasette-atom 0.8.1 - 2020-11-25
datasette-ripgrep 0.1a1 - 2020-11-25
datasette-ripgrep 0.1a0 - 2020-11-25
datasette-graphql 1.2.1 - 2020-11-24

Tags: async, css, projects, python, regular-expressions, datasette, weeknotes, cloudrun, ripgrep, baked-data

The unexpected Google wide domain check bypass

2020-03-09T23:27:41+00:00

The unexpected Google wide domain check bypass

Fantastic story of discovering a devious security vulnerability in a bunch of Google products stemming from a single exploitable regular expression in the Google closure JavaScript library.

Via Hacker News

Tags: regular-expressions, security

Weeknotes: ONA19, twitter-to-sqlite, datasette-rure

2019-09-13T21:35:52+00:00

I’ve decided to start writing weeknotes for the duration of my JSK fellowship. Here goes!

I started the fellowship last week, and this week I’ve been attending the Online News Association conference in New Orleans along with the other fellows.

Natalie and I have had a lot of fun exploring New Orleans, including several niche museums (documented in this Twitter thread, which I expect to continue adding to for years to come).

ONA is the largest digital news conference on the calendar, and as such it’s a great opportunity to meet all kinds of different journalists and get a start on figuring out how best to approach the fellowship.

twitter-to-sqlite

I’ve been working on this tool for a couple of weeks now, but this week it really started to prove its worth. twitter-to-sqlite is a utility I’m building to fetch data from Twitter and write it into a SQLite database. It has a comprehensive README, as do all of my recent projects: I’ve been doing this for long enough that I know that time spent on documentation will pay me back enormously in just a few months.

My initial goal for this tool was to use it for personal analytics, partly inspired by my Twitter followers project from last year. Then ONA sent out a spreadsheet with the Twitter names of most of the conference attendees and I realised there was an opportunity to use it for something more interesting.

I won’t share the attendee list here because it’s intended to be private to conference attendees, but the short version of what I did with it is this:

First, load the attendee list into a database using csvs-to-sqlite (after first converting the XLS file to CSV using Numbers):

$ csvs-to-sqlite attendees.csv ona.db

Next, fetch the full Twitter profiles for every one of those attendees. I added a --sql option to twitter-to-sqlite for this (issue #8) and ran the following:

$ twitter-to-sqlite users-lookup ona.db --sql="select Twitter from attendees"

Now I can run datasette ona.db and execute full-text searches against the complete Twitter profiles of those attendees - great for figuring out e.g. who is attending the conference from the Washington Post.

I took this a step further: can I identify people I follow on Twitter who are attending the conference? I can pull the list of IDs of people I follow like so:

$ twitter-to-sqlite friends-ids ona.db simonw

Now I can see which of my Twitter friends are attending the conference by loading it into Datasette and using this query:

select * from users
where screen_name in (select Twitter from attendees)
and id in (select followed_id from following where follower_id = 12497)

This is not a particularly efficient SQL query… and it doesn’t matter! The great thing about working against tiny SQLite databases that contain just a few thousand rows of data is that you can nest SQL queries like this with little concern for their performance - this one runs on my laptop in 18.213ms.

datasette-rure

SQLite has a REGEXP operator, which is documented thus:

The REGEXP operator is a special syntax for the regexp() user function. No regexp() user function is defined by default and so use of the REGEXP operator will normally result in an error message. If an application-defined SQL function named “regexp” is added at run-time, then the “X REGEXP Y” operator will be implemented as a call to “regexp(Y,X)”.

Python’s sqlite3 module lets you register custom functions, so this should be an easy fix. I’ve been holding off on implementing this in Datasette for a while though because Datasette allows user-provided queries, and Python’s regular expressions have a nasty quality: you can easily construct a regular expression and input that will hang the Python interpreter.

So I needed a safe regular expression library that I could trust not to hang my intepreter on the wrong input.

Google have such a library - re2 - but the various Python bindings for it aren’t cleanly installable via pip install.

I asked about this on Twitter and Alex Willmer pointed me to rure-python - a Python wrapper for the Rust regular expression library, which “guarantees linear time searching using finite automata” - exactly what I was looking for!

So I built a quick Datasette plugin: datasette-rure, which provides a regexp() function to enable the REGEXP operator, powered by the Rust regular expression engine.

I also added regexp_match(pattern, text, index) and regexp_matches(pattern, text) functions. Here are their examples from the README:

select regexp_match('.*( and .*)', title) as n from articles where n is not null
-- Returns the ' and X' component of any matching titles, e.g.
--     and Recognition
--     and Transitions Their Place
-- etc
select regexp_matches(
    'hello (?P<name>\w+) the (?P<species>\w+)',
    'hello bob the dog, hello maggie the cat, hello tarquin the otter'
)
-- Returns a JSON array:
-- [{"name": "bob", "species": "dog"},
--  {"name": "maggie", "species": "cat"},
--  {"name": "tarquin", "species": "otter"}]

Returning JSON from a SQLite custom function is a fun trick: it means you can then loop over the returned rows or even join them against other tables within SQL using the bundled json_each() function.

I deployed an interactive demo of Datasette running the plugin. I’ve been trying to do this for all of the plugins I release - datasette-jq is another recent example (the interactive demos are linked from the README).

Tags: projects, regular-expressions, sqlite, twitter, rust, datasette, weeknotes

Details of the Cloudflare outage on July 2, 2019

2019-07-12T17:36:25+00:00

Details of the Cloudflare outage on July 2, 2019

Best retrospective I’ve read in a long time. The outage was caused by a backtracking regex rule that was added to the Web Application Firewall project, which rolls out globally and skips most of Cloudflare’s regular graduar rollout process (delightfully animal themed, named DOG for the dogfooding PoP that their employees use, PIG for the Guinea Pig PoPs reserved for free customers, then Canary for the final step) so that they can deploy counter-measures to newly discovered vulnerabilities as quickly as possible—but the real value in the retro is that it provides an extremely deep insight into how Cloudflare organize, test and manage their changes. Really interesting stuff.

Via Hacker News

Tags: operations, regular-expressions, cloudflare, postmortem

r1chardj0n3s/parse: Parse strings using a specification based on the Python format() syntax.

2018-02-25T16:58:32+00:00

r1chardj0n3s/parse: Parse strings using a specification based on the Python format() syntax.

Really neat API design: parse() behaves almost exactly in the opposite way to Python’s built-in format(), so you can use format strings as an alternative to regular expressions for extracting specific data from a string.

Via requests-html/Pipfile

Tags: python, regular-expressions

A Regular Expression Matcher: Code by Rob Pike, Exegesis by Brian Kernighan

2017-12-05T18:36:12+00:00

A Regular Expression Matcher: Code by Rob Pike, Exegesis by Brian Kernighan

Delightfully clear and succinct 30-line C implementation of a regular expression matcher that supports $, ^, . and * operations.

Via Hacker News

Tags: c, regular-expressions, rob-pike

What are the best resources for learning regular expressions?

2012-02-21T17:47:00+00:00

My answer to What are the best resources for learning regular expressions? on Quora

The O'Reilly book on Regular Expressions is absolutely superb. It will help you build a much deeper understanding if how they actually work than any online tutorial I've seen.

Chapter 3 of "Text Processing in Python" is excellent too, and is available for free online:

http://gnosis.cx/TPiP/chap3.txt

Tags: programming, programming-languages, python, regular-expressions, quora

Escaping regular expression characters in JavaScript (updated)

2010-07-04T18:23:00+00:00

Escaping regular expression characters in JavaScript (updated)

The JavaScript regular expression meta-character escaping code I posted back in 2006 has some serious flaws—I’ve just posted an update to the original post.

Tags: escaping, javascript, regular-expressions, recovered

Introduction to Surlex

2010-04-11T19:23:35+00:00

Introduction to Surlex

A neat drop-in alternative for Django’s regular expression based URL parsing, providing simpler syntax for common path patterns.

Tags: codysoyland, django, python, regular-expressions, surlex, urls

RE2: a principled approach to regular expression matching

2010-03-12T09:28:21+00:00

RE2: a principled approach to regular expression matching

Google have open sourced RE2, the C++ regular expression library they developed for Google Code Search, Sawzall, Bigtable and other internal projects. Unlike PCRE it avoids the potential for exponential run time and unbounded stack usage and guarantees that searches complete in linear time, mainly by dropping support for back references.

Tags: c-plus-plus, google, open-source, pcre, re2, regular-expressions, russcox

Request Routing With URI Templates in Node.JS

2009-11-24T09:06:22+00:00

Request Routing With URI Templates in Node.JS

I quite like this approach (though the implementation is a bit “this” heavy for my taste). JavaScript has no equivalent to Python’s raw strings, so regular expression based routing ala Django ends up being a bit uglier in JavaScript. URI template syntax is more appealing.

Tags: django, javascript, nodejs, python, regular-expressions, uritemplates

Quoting Andrew Clover

2009-11-16T10:32:15+00:00

Every time you attempt to parse HTML with regular expressions, the unholy child weeps the blood of virgins, and Russian hackers pwn your webapp. Parsing HTML with regex summons tainted souls into the realm of the living. HTML and regex go together like love, marriage, and ritual infanticide.

— Andrew Clover

Tags: andrew-clover, funny, html, parsing, regular-expressions, stackoverflow, xhtml

Django security updates released

2009-10-10T00:24:59+00:00

Django security updates released

A potential denial of service vulnerability has been discovered in the regular expressions used by Django form library’s EmailField and URLField—a malicious input could trigger a pathological performance. Patches (and patched releases) for Django 1.1 and Django 1.0 have been published.

Tags: denial-of-service, django, python, regular-expressions, security

Introducing Yardbird

2009-05-22T23:13:39+00:00

Introducing Yardbird

I absolutely love it—an IRC bot built on top of Twisted that passes incoming messages off to Django code running in a separate thread. Requests and Response objects are used to represent incoming and outgoing messages, and Django’s regex-based URL routing is used to dispatch messages to different handling functions based on their content.

Tags: django, irc, regular-expressions, threads, twisted, yardbird

Python gems of my own

2008-11-03T11:59:48+00:00

Python gems of my own

Did you know you can pass 128 as a flag to Python’s re.compile() function to spit out a parse tree? I didn’t. re.compile(“pattern”, 128)

Via Eric Florenzano

Tags: eric-holscher, python, regular-expressions

Primality regex

2007-03-18T01:17:45+00:00

Primality regex

A regular expression that can identify prime numbers. Unsurprisingly, this one comes from the Perl community.

Tags: perl, primenumbers, regular-expressions

Wrapping Text With Regular Expressions

2006-06-29T11:12:51+00:00

Wrapping Text With Regular Expressions

Neat regexp trick.

Tags: regular-expressions

Escaping regular expression characters in JavaScript

2006-01-20T12:19:13+00:00

JavaScript's support for regular expressions is generally pretty good, but there is one notable omission: an escaping mechanism for literal strings. Say for example you need to create a regular expression that removes a specific string from the end of a string. If you know the string you want to remove when you write the script this is easy:

var newString = oldString.replace(/Remove from end$/, '');

But what if the string to be removed comes from a variable? You'll need to construct a regular expression from the variable, using the RegExp constructor function:

var re = new RegExp(stringToRemove + '$');
var newString = oldString.replace(re, '');

But what if the string you want to remove may contain regular expression metacharacters - characters like $ or . that affect the behaviour of the expression? Languages such as Python provide functions for escaping these characters (see re.escape); with JavaScript you have to write your own.

Here's mine:

RegExp.escape = function(text) {
  if (!arguments.callee.sRE) {
    var specials = [
      '/', '.', '*', '+', '?', '|',
      '(', ')', '[', ']', '{', '}', '\\'
    ];
    arguments.callee.sRE = new RegExp(
      '(\\' + specials.join('|\\') + ')', 'g'
    );
  }
  return text.replace(arguments.callee.sRE, '\\$1');
}

This deals with another common problem in JavaScript: compiling a regular expression once (rather than every time you use it) while keeping it local to a function. argmuments.callee inside a function always refers to the function itself, and since JavaScript functions are objects you can store properties on them. In this case, the first time the function is run it compiles a regular expression and stashes it in the sRE property. On subsequent calls the pre-compiled expression can be reused.

In the above snippet I've added my function as a property of the RegExp constructor. There's no pressing reason to do this other than a desire to keep generic functionality relating to regular expression handling the same place. If you rename the function it will still work as expected, since the use of arguments.callee eliminates any coupling between the function definition and the rest of the code.

Update 18th Feb 2025: 19 years after I published this RegExp.escape() has made it into the language!

Tags: escaping, javascript, regular-expressions

Lexical Analysis, Python-style

2005-04-28T15:02:50+00:00

Lexical Analysis, Python-style

Clever trick using named groups in regular expressions.

Via Daily Python-URL

Tags: regular-expressions

Unicode Regular Expressions

2004-03-03T03:06:47+00:00

Unicode Regular Expressions

[a \U00010450] Match “a” or U+10450 SHAVIAN LETTER PEEP

Tags: regular-expressions

Brief Guide to Regular Expressions

2003-12-04T01:48:15+00:00

Brief Guide to Regular Expressions

Regular expressions for everyone else

Via And suddenly itâ€™s a good day

Tags: regular-expressions

Capturing the power of re.split

2003-10-26T03:01:38+00:00

A couple of Python tips. The first is really a tip for Mozilla/Firebird: You can set up a Custom Keyword for instantly accessing Python module documentation using the string www.python.org/doc/current/lib/module-%s.html - I have this set up as pydoc, so I can type pydoc re to jump straight to the re module documentation. I only set it up half an hour ago and I've already used it about a dozen times.

The second tip is so powerful I've been kicking myself for not finding out about it sooner. It relates to the regular expression module's re.split() function. Just like string.split(), this lets you split up a string based on a certain token. With string.split() you the token you split on isn't included in the resulting array:

>>> 'pipe|separated|values'.split('|')
['pipe', 'separated', 'values']

This is also true of re.split:

>>> splitter = re.compile('<.>')
>>> splitter.split('hi<a>there<b>from<c>python')
['hi', 'there', 'from', 'python']

Here's the magic part though. If you put part or all of the regular expression in parenthesis the separating tokens get included in the resulting list:

>>> splitter = re.compile('(<.>)')
>>> splitter.split('hi<a>there<b>from<c>python')
['hi', '<a>', 'there', '<b>', 'from', '<c>', 'python']

Why is this a big deal? Because it suddenly makes writing simple parsers and tokenisers a whole heck of a lot easier. Using the above example, say you wanted to do something with each of the <?> style tags. You can just iterate through the resulting list identifying each tag using the regular expression you've already compiled and then altering just those list items, before joining the whole list back together again at the end.

Simple parsing and replacement of easily identified tags can already be achieved using the re.sub() method, which allows you to provide a callback function to process each matching token. The difference with using re.split() is that you can easily take in to account the order of the tokens, allowing you to build systems that can use special tags to define areas of documents without getting confused by nesting tag sets. As a simple example, you could build a basic event based XML parser using just a couple of expressions. In fact, I discovered this technique while examining the source code for the tinpy tiny python template module, which gives a clue to why I'm so interested in it.

Having discovered this feature in Python, I just had to see if it existed in other languages as well. Unsurprisingly it does; PHP's preg_split offers an optional PREG_SPLIT_DELIM_CAPTURE flag (added in PHP 4.0.5) and Javascript has similar behaviour to Python, including the splitting token if it is wrapped in parentheses.

I'm probably the last person to find out about this, but it's such a useful technique I felt I just had to share it with the world.

Tags: regular-expressions