Simon Willison's Weblog: dsl

yet-another-applied-llm-benchmark

2024-11-06T20:00:23+00:00

Nicholas Carlini introduced this personal LLM benchmark suite back in February as a collection of over 100 automated tests he runs against new LLM models to evaluate their performance against the kinds of tasks he uses them for.

There are two defining features of this benchmark that make it interesting. Most importantly, I've implemented a simple dataflow domain specific language to make it easy for me (or anyone else!) to add new tests that realistically evaluate model capabilities. This DSL allows for specifying both how the question should be asked and also how the answer should be evaluated. [...] And then, directly as a result of this, I've written nearly 100 tests for different situations I've actually encountered when working with LLMs as assistants

The DSL he's using is fascinating. Here's an example:

"Write a C program that draws an american flag to stdout." >> LLMRun() >> CRun() >> \
    VisionLLMRun("What flag is shown in this image?") >> \
    (SubstringEvaluator("United States") | SubstringEvaluator("USA")))

This triggers an LLM to execute the prompt asking for a C program that renders an American Flag, runs that through a C compiler and interpreter (executed in a Docker container), then passes the output of that to a vision model to guess the flag and checks that it returns a string containing "United States" or "USA".

The DSL itself is implemented entirely in Python, using the __rshift__ magic method for >> and __rrshift__ to enable strings to be piped into a custom object using "command to run" >> LLMRunNode.

Tags: dsl, python, ai, generative-ai, llms, evals, nicholas-carlini

Building Search DSLs with Django

2023-06-19T08:30:32+00:00

Building Search DSLs with Django

Neat tutorial by Dan Lamanna: how to build a GitHub-style search feature—supporting modifiers like “is:open author:danlamanna”—using PyParsing and the Django ORM.

Via Lobsters

Tags: django, dsl, parsing, python, search

Datasette table diagram using Mermaid

2022-02-14T19:43:15+00:00

Datasette table diagram using Mermaid

Mermaid is a DSL for generating diagrams from plain text, designed to be embedded in Markdown. GitHub just added support for Mermaid to their Markdown pipeline, which inspired me to try it out. Here’s an Observable Notebook I built which uses Mermaid to visualize the relationships between Datasette tables based on their foreign keys.

Via @simonw

Tags: dsl, github, visualization, datasette, observable, mermaid

Richard Jones: Something I'm working on...

2009-08-07T15:47:00+00:00

Richard Jones: Something I'm working on...

Python’s with statement appears to provide just enough syntactic sugar to create some really interesting DSL-style APIs—here’s a very promising example for laying out GUI applications.

Tags: dsl, gui, python, richard-jones, with

Metaprogramming JavaScript Presentation

2007-03-26T19:45:11+00:00

Metaprogramming JavaScript Presentation

Adam McCrea demonstrates some incredibly elegant DSL -style JavaScript based on chaining method calls together.

Tags: adam-mccrea, dsl, javascript, metaprogramming