Simon Willison's Weblog: cosmopolitan

sqlite-page-explorer

2025-02-06T20:39:15+00:00

Outstanding tool by Luke Rissacher for understanding the SQLite file format. Download the application (built using redbean and Cosmopolitan, so the same binary runs on Windows, Mac and Linux) and point it at a SQLite database to get a local web application with an interface for exploring how the file is structured.

Here's it running against the datasette.io/content database that runs the official Datasette website:

Via Hacker News

Tags: lua, sqlite, redbean, cosmopolitan

llamafile is the new best way to run an LLM on your own computer

2023-11-29T20:54:16+00:00

Mozilla’s innovation group and Justine Tunney just released llamafile, and I think it's now the single best way to get started running Large Language Models (think your own local copy of ChatGPT) on your own computer.

A llamafile is a single multi-GB file that contains both the model weights for an LLM and the code needed to run that model - in some cases a full local server with a web UI for interacting with it.

The executable is compiled using Cosmopolitan Libc, Justine's incredible project that supports compiling a single binary that works, unmodified, on multiple different operating systems and hardware architectures.

Here's how to get started with LLaVA 1.5, a large multimodal model (which means text and image inputs, like GPT-4 Vision) fine-tuned on top of Llama 2. I've tested this process on an M2 Mac, but it should work on other platforms as well (though be sure to read the Gotchas section of the README, and take a look at Justine's list of supported platforms in a comment on Hacker News).

Download the 4.29GB llava-v1.5-7b-q4.llamafile file from Justine's repository on Hugging Face.

curl -LO https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/llava-v1.5-7b-q4.llamafile
Make that binary executable, by running this in a terminal:

chmod 755 llava-v1.5-7b-q4.llamafile
Run your new executable, which will start a web server on port 8080:

./llava-v1.5-7b-q4.llamafile
Navigate to http://127.0.0.1:8080/ to start interacting with the model in your browser.

That's all there is to it. On my M2 Mac it runs at around 55 tokens a second, which is really fast. And it can analyze images - here's what I got when I uploaded a photograph and asked "Describe this plant":

How this works

There are a number of different components working together here to make this work.

The LLaVA 1.5 model by Haotian Liu, Chunyuan Li, Yuheng Li and Yong Jae Lee is described in this paper, with further details on llava-vl.github.io.
The models are executed using llama.cpp, and in the above demo also use the llama.cpp server example to provide the UI.
Cosmopolitan Libc is the magic that makes one binary work on multiple platforms. I wrote more about that in a TIL a few months ago, Catching up with the Cosmopolitan ecosystem.

Trying more models

The llamafile README currently links to binaries for Mistral-7B-Instruct, LLaVA 1.5 and WizardCoder-Python-13B.

You can also download a much smaller llamafile binary from their releases, which can then execute any model that has been compiled to GGUF format:

I grabbed llamafile-server-0.1 (4.45MB) like this:

curl -LO https://github.com/Mozilla-Ocho/llamafile/releases/download/0.1/llamafile-server-0.1
chmod 755 llamafile-server-0.1

Then ran it against a 13GB llama-2-13b.Q8_0.gguf file I had previously downloaded:

./llamafile-server-0.1 -m llama-2-13b.Q8_0.gguf

This gave me the same interface at http://127.0.0.1:8080/ (without the image upload) and let me talk with the model at 24 tokens per second.

One file is all you need

I think my favourite thing about llamafile is what it represents. This is a single binary file which you can download and then use, forever, on (almost) any computer.

You don't need a network connection, and you don't need to keep track of more than one file.

Stick that file on a USB stick and stash it in a drawer as insurance against a future apocalypse. You'll never be without a language model ever again.

Tags: mozilla, ai, generative-ai, cosmopolitan, llama, local-llms, llms, mistral, llamafile, justine-tunney, llama-cpp

djngo.com: Portable Django

2023-02-24T00:52:27+00:00

djngo.com: Portable Django

“A 20mb executable zip file with Python 3.6 and Django 2.2. Works on Windows, Linux, MacOSX with x86_64 and aarch64 (yes, Apple M1 and Raspberry Pi).” The latest wizardry from the ecosystem surrounding the Cosmopolitan project, which provides a should-be-impossible mechanism for running the same executable on a bunch of different platforms. This utility by Ariel Núñez bundles Python and Django and SQLite, such that a Django application can become a portable executable ready to run on multiple platforms. It’s currently limited to Python 3.6 and Django 2.2 since those are the versions that run under Cosmopolitan, but I expect we’ll see more recent versions of those dependencies in the future.

Via Hacker News

Tags: django, python, sqlite, redbean, cosmopolitan, raspberry-pi

Cosmopolitan: Compiling Python

2022-07-26T20:43:31+00:00

Cosmopolitan: Compiling Python

Cosmopolitan is Justine Tunney’s “build-once run-anywhere C library”—part of the αcτµαlly pδrταblε εxεcµταblε effort, which produces wildly clever binary executable files that work on multiple different platforms, and is the secret sauce behind redbean. I hadn’t realized this was happening but there’s an active project to get Python to work as this format, producing a new way of running Python applications as standalone executables, only these ones have the potential to run unmodified on Windows, Linux and macOS.

Tags: python, redbean, cosmopolitan, justine-tunney

Fullmoon

2022-02-18T18:41:19+00:00

Fullmoon

A “fast and minimalistic web framework” written in Lua, based on Redbean. The documentation for this is fantastic, and because it uses Redbean the development experience is to download the Redbean executable (which runs on every platform) and then drop your own Lua scripts into it using zip.

Via Hacker News

Tags: lua, redbean, cosmopolitan

redbean

2022-02-17T06:01:18+00:00

redbean

“redbean makes it possible to share web applications that run offline as a single-file αcτµαlly pδrταblε εxεcµταblε zip archive which contains your assets. All you need to do is download the redbean.com program below, change the filename to .zip, add your content in a zip editing tool, and then change the extension back to .com”.

redbean is implemented as a single C file with a dazzling array of clever tricks—most impressively, the single executable works on Linux, macOS, Windows and various BSDs!

It embeds Lua, and in June last year added SQLite too—so self-contained distributable web applications built with Redbean can now use Lua and SQLite for dynamic scripting. Performance sounds incredible: “redbean can serve 1 million+ gzip encoded responses per second on a cheap personal computer”.

Via lobste.rs

Tags: c, lua, sqlite, redbean, cosmopolitan

cosmopolitan libc

2021-02-27T06:02:11+00:00

cosmopolitan libc

“Cosmopolitan makes C a build-once run-anywhere language, similar to Java, except it doesn’t require interpreters or virtual machines be installed beforehand. [...] Instead, it reconfigures stock GCC to output a POSIX-approved polyglot format that runs natively on Linux + Mac + Windows + FreeBSD + OpenBSD + BIOS with the best possible performance and the tiniest footprint imaginable.” This is a spectacular piece of engineering.

Via Hacker News

Tags: c, cosmopolitan