Simon Willison's Weblog: parsing

GitHub Issues search now supports nested queries and boolean operators: Here’s how we (re)built it

2025-05-26T07:23:17+00:00

GitHub Issues search now supports nested queries and boolean operators: Here’s how we (re)built it

GitHub Issues got a significant search upgrade back in January. Deborah Digges provides some behind the scene details about how it works and how they rolled it out.

The signature new feature is complex boolean logic: you can now search for things like is:issue state:open author:rileybroughten (type:Bug OR type:Epic), up to five levels of nesting deep.

Queries are parsed into an AST using the Ruby parslet PEG grammar library. The AST is then compiled into a nested Elasticsearch bool JSON query.

GitHub Issues search deals with around 2,000 queries a second so robust testing is extremely important! The team rolled it out invisibly to 1% of live traffic, running the new implementation via a queue and competing the number of results returned to try and spot any degradations compared to the old production code.

Tags: elasticsearch, github, ops, parsing, ruby, scaling, search, github-issues

Lark parsing library JSON tutorial

2023-08-13T21:50:16+00:00

Lark parsing library JSON tutorial

A very convincing tutorial for a new-to-me parsing library for Python called Lark.

The tutorial covers building a full JSON parser from scratch, which ends up being just 19 lines of grammar definition code and 15 lines for the transformer to turn that tree into the final JSON.

It then gets into the details of optimization—the default Earley algorithm is quite slow, but swapping that out for a LALR parser (a one-line change) provides a 5x speedup for this particular example.

Via spandanb/learndb-py

Tags: compilers, json, parsing, python

Building Search DSLs with Django

2023-06-19T08:30:32+00:00

Building Search DSLs with Django

Neat tutorial by Dan Lamanna: how to build a GitHub-style search feature—supporting modifiers like “is:open author:danlamanna”—using PyParsing and the Django ORM.

Via Lobsters

Tags: django, dsl, parsing, python, search

Comby

2022-05-18T05:47:24+00:00

Comby

Describes itself as “Structural search and replace for any language”. Lets you execute search and replace patterns that look a little bit like simplified regular expressions, but with some deep OCaml-powered magic that makes them aware of comment, string and nested parenthesis rules for different languages. This means you can use it to construct scripts that automate common refactoring or code upgrade tasks.

Via Hacker News

Tags: parsing, refactoring, ocaml

pup

2020-02-14T16:25:27+00:00

pup

This is a great idea: a command-line tool for parsing HTML on stdin using CSS selectors. It’s like jq but for HTML. Supports a sensible collection of selectors and has a number of output options for the selected nodes, including plain text and JSON. It also works as a simple pretty-printer for HTML.

Tags: cli, html, parsing

Parsing CSV using ANTLR and Python 3

2018-04-06T14:33:58+00:00

Parsing CSV using ANTLR and Python 3

I’ve been trying to figure out how to use ANTLR grammars from Python—this is the first example I’ve found that really clicked for me.

Tags: antlr, csv, parsing, python

Parse shell one-liners with pyparsing

2017-10-22T13:35:41+00:00

Parse shell one-liners with pyparsing

Neat introduction to the pyparsing library, both for parsing tokens into labeled sections and constructing an AST from them.

Tags: parsing, python

Quoting Andrew Clover

2009-11-16T10:32:15+00:00

Every time you attempt to parse HTML with regular expressions, the unholy child weeps the blood of virgins, and Russian hackers pwn your webapp. Parsing HTML with regex summons tainted souls into the realm of the living. HTML and regex go together like love, marriage, and ritual infanticide.

— Andrew Clover

Tags: andrew-clover, funny, html, parsing, regular-expressions, stackoverflow, xhtml

HTML 5 Parsing

2009-07-11T23:36:57+00:00

HTML 5 Parsing

Firefox nightlies include a new parser that implements the HTML5 parsing algorithm (disabled by default), which uses C++ code automatically generated from Henri Sivonen’s Java parser first used in the HTML5 validator.

Tags: browsers, firefox, henri-sivonen, html5, john-resig, mozilla, parsing, validator

South's Design

2009-05-13T12:30:45+00:00

South's Design

Andrew Godwin explains why South resorts to parsing your models.py file in order to construct information about for creating automatic migrations.

Tags: andrew-godwin, django, models, orm, parsing, python, south

Simple Top-Down Parsing in Python

2008-07-19T23:37:59+00:00

Simple Top-Down Parsing in Python

Eye-opening tutorial on building a recursive descent parser for Python, in Python that uses top-down operator precedence.

Tags: compilers, effbot, fredrik-lundh, parsing, python, recursivedescent

python4ply tutorial

2008-03-11T05:49:22+00:00

python4ply tutorial

python4ply is a parser for Python written in Python using the PLY toolkit, which compiles to Python bytecode using the built-in compiler module. The tutorial shows how to use it to add support for Perl-style 1_000_000 readable numbers.

Tags: compilers, lexing, parsing, python, python4ply