Simon Willison's Weblog: transformers-js

Porting the Moebius 0.2B image inpainting model to run in the browser with Claude Code

2026-06-22T23:43:51+00:00

This morning on Hacker News I saw Moebius: 0.2B Lightweight Image Inpainting Framework with 10B-Level Performance, describing a small but effective inpainting model - a model where you can mark regions of an image to remove and the model imagines what should fill the space. The released model required PyTorch and NVIDIA CUDA, but since it described itself as 0.2B I decided to try and get it running using WebGPU in a browser. TL;DR: I got it working, and you can try the demo at simonw.github.io/moebius-web/. Read on for the details.

The finished tool

Here's a video demo of the finished tool:

You can open any image in it (non-square images get letterboxed), highlight areas to remove, click the "Run inpaint" button and wait for the model to do its magic.

A parallel agent side-project

My main project for today was landing a major feature in Datasette: a UI for creating and altering tables, as a follow-up to the insert and edit rows feature I released last week.

I was working on that in Codex Desktop (here's the PR) and often found myself spending 5-10 minutes spinning my fingers waiting for it to complete a mid-sized refactor or add the finishing touches to a change to the UI.

(An amusing thing about coding agents is that the harder a problem is the more time you have to get distracted while you wait for them to finish crunching!)

So I decided to spin up Claude Code in a terminal window and see how far I could get at porting Moebius to the web.

Some agentic research to kick off the project

My first step was to ask regular Claude about the feasibility of this project. In Claude.ai, which has the ability to clone repos from GitHub:

Clone https://github.com/hustvl/Moebius/ and tell me if they published the code and weights to run this model anywhere

(I hadn't spotted the link to the weights yet, that's tucked away in the "News" section.)

Then:

For Moebius what are the options for running it right now - Python and NVIDIA CUDA only or other options too?

And:

Muse on the feasibility of porting it to Transformers.js or similar and running it in a browser

I like telling models to "muse on X", it's the shortest way I've found of expressing that I want them to contemplate a problem for me without providing them with a concrete goal.

Here's that chat transcript. I copied out the last answer and saved it as research.md for Claude Code to read later.

Claude suggested using ONNX Runtime Web on the WebGPU backend - the layer below the Transformers.js library I had suggested.

That was enough to convince me it was worth setting Claude Code loose and seeing how far it could get.

I usually start projects like this by gathering as much information as the coding agent might need as possible. Since I didn't expect this project to actually work I did everything in my /tmp folder:

cd /tmp
mkdir Moebius
cd Moebius
# Grab the Moebius python code
git clone https://github.com/hustvl/Moebius
# And the model weights (Claude figured this out):
GIT_LFS_SKIP_SMUDGE=0 git clone \
  https://huggingface.co/hustvl/Moebius Moebius-weights
# Finally a couple of libraries we might use:
git clone https://github.com/huggingface/transformers.js
git clone https://github.com/microsoft/onnxruntime

Setting off Claude Code

I created a directory for the rest of the project and ran git init in that so Claude could start committing code notes:

mkdir /tmp/Moebius/moebius-web
cd /tmp/Moebius/moebius-web
git init
# Copy in that research.md from earlier
git add research.md
git commit -m "Initial research by Claude Opus 4.8"

I fired up a claude instance in the /tmp/Moebius folder, the level above all of the research materials I had prepared for it. I prompted:

Read ./moebius-web/research.md - your goal is to port this model to ONNX and WebGPU so we can run it directly in a browser, with a simple UI

As it started to work I dropped in this follow-up (typos included):

Bulid this in /tmp/Moebius/moebius-web and commit early and often, also maintain a notes.md file in there with notes about what you figure out along the way - also start by writing out a plan.md in there and update that plan as oy work too

I often ask agents to keep notes like this - the end result is often interesting, both for myself and for the next agent session that touches the same project. Here's what that notes.md file looked like at the end of the project.

I kicked it off and went back to my main project, checking in occasionally to see how Claude was doing. When it looked like it might have something that worked I prompted:

Tell me what URL I can visit in my own browser to try this

Then I tried it out in Chrome and pasted some errors (and screenshots of errors) back into Claude Code.

After a few rounds of this we had something that appeared to work! Time to put it on the internet so other people could use it.

How would we publish this to Hugging Face such that the model weights were on there and the HTML demo would show up in Hugging Face spaces?

Claude Code knows how to use the hf CLI tool, so I created a model repo on Hugging Face, then created a token that could write to that repo and dropped it into a /tmp/Moebius/token.txt file so Claude could use it.

It published the 1.24GB of converted ONNX weights to huggingface.co/simonw/Moebius-ONNX for me.

I'd seen other demos load weights into the browser from Hugging Face before, so I knew it was possible. I decided to host my own frontend code on GitHub Pages, so I said:

I want to publish the moebius-web folder to GitHub, minus the large files (so maybe minus the models/ folder), such that when I turn on GitHub Pages for that repo navigating to https://simonw.github.io/moebius-web/ serves the UI

Telling it the final URL was important in case it needed to fix the URLs in the demos that it was building so they would work when deployed to production.

After a few more rounds of iteration, in between working on my main project, we got to a working, deployed version!

Except... each time I reloaded the page it seemed to download ~1.3GB of model weights. Browser caching seemed pretty important for this!

anything clever we can do with serviceworkers or similar to help cache this stuff? It seems to reload every time, I am concerned that there might be something weird about the way HF redirects work that mean we don't benefit from browser caching

I knew that Transformers.js projects could handle this properly, so I grabbed a copy of the Whisper Web demo, dropped it into /tmp/Moebius/whisper-web and said:

look in /tmp/Moebius/whisper-web (with a subagent) and see how they do this

That project was entirely obfuscated, built JavaScript files so I figured using a subagent would avoid spending the rest of my top-level token context deciphering those files.

Claude figured out that it was using caches.open("transformers-cache") - the CacheStorage API - and added that to our project.

I've shared the full Claude Code transcript for this project (published using my claude-code-transcripts tool).

What did I learn from all of this?

This definitely counts as vibe coding: I didn't look at a single line of code from the project, restricting my input to testing, suggesting small feature improvements (like a progress bar for the large file downloads) and pointing the model in the direction of examples of how I wanted things to work.

Since I didn't write any code the amount I learned about the underlying technologies - WebGPU, ONNX, and the Moebius model itself - was very limited.

As is usually the case with this kind of project the most important things I learned concerned what was possible:

Claude Opus 4.8 is capable of converting a PyTorch model to ONNX, publishing the result to Hugging Face and then building out a web application and interface that can load and execute that model.
Chrome, Firefox and Safari are all now capable of running this kind of model - I tried it in all three.
The CacheStorage API works with ~1.3GB model files.
... which means we can have inpainting as a feature of a client-only web application! (If our users can tolerate the 1.3GB download.)

I felt like I should probably try and learn a little more about my project. I fired up Claude.ai and prompted:

Clone https://github.com/simonw/moebius-web/ and use it to teach me all about the model and ONNX and the process of converting a model to ONNX and WebGPU and basically everything I'd need to know in order to fully understand this repo

Here's the transcript and the understanding.md Markdown file it created, which I've now added to the GitHub repo. I found the explanation of ONNX particularly enlightening:

ONNX (Open Neural Network Exchange) is a portable, framework-neutral file format for neural networks. An .onnx file is essentially two things bundled together:

A computation graph — a directed graph of nodes, where each node is an operator (Conv, MatMul, Add, Einsum, Softmax, Gather, Resize, …) wired together by named tensors flowing between them. This is the "recipe" for the forward pass.

The weights — the learned parameter tensors (the convolution kernels, the embedding table, etc.), stored as initializers in that same graph.

Crucially, ONNX describes what to compute, abstractly, without saying how or on what hardware. The operator set is versioned by an opset number (this repo uses opset 18), which pins down exactly which operators exist and what their semantics are.

It turns out PyTorch has built in mechanisms for exporting to ONNX, as seen here in export_onnx.py:

torch.onnx.export(
    dec, (lat,), dec_path, opset_version=args.opset,
    input_names=["latent"], output_names=["image"],
    dynamic_axes={"latent": {0: "B"}, "image": {0: "B"}},
)

Claude also included a handy glossary and an only-slightly-broken ASCII-art diagram showing how the model pipeline fits together.

Tags: browsers, transformers-js, webgl, vibe-coding, coding-agents, claude-code, onnx

Load Llama-3.2 WebGPU in your browser from a local folder

2025-09-08T20:53:52+00:00

Load Llama-3.2 WebGPU in your browser from a local folder

Inspired by a comment on Hacker News I decided to see if it was possible to modify the transformers.js-examples/tree/main/llama-3.2-webgpu Llama 3.2 chat demo (online here, I wrote about it last November) to add an option to open a local model file directly from a folder on disk, rather than waiting for it to download over the network.

I posed the problem to OpenAI's GPT-5-enabled Codex CLI like this:

git clone https://github.com/huggingface/transformers.js-examples
cd transformers.js-examples/llama-3.2-webgpu
codex

Then this prompt:

Modify this application such that it offers the user a file browse button for selecting their own local copy of the model file instead of loading it over the network. Provide a "download model" option too.

Codex churned away for several minutes, even running commands like curl -sL https://raw.githubusercontent.com/huggingface/transformers.js/main/src/models.js | sed -n '1,200p' to inspect the source code of the underlying Transformers.js library.

After four prompts total (shown here) it built something which worked!

To try it out you'll need your own local copy of the Llama 3.2 ONNX model. You can get that (a ~1.2GB) download) like so:

git lfs install
git clone https://huggingface.co/onnx-community/Llama-3.2-1B-Instruct-q4f16

Then visit my llama-3.2-webgpu page in Chrome or Firefox Nightly (since WebGPU is required), click "Browse folder", select that folder you just cloned, agree to the "Upload" confirmation (confusing since nothing is uploaded from your browser, the model file is opened locally on your machine) and click "Load local model".

Here's an animated demo (recorded in real-time, I didn't speed this up):

I pushed a branch with those changes here. The next step would be to modify this to support other models in addition to the Llama 3.2 demo, but I'm pleased to have got to this proof of concept with so little work beyond throwing some prompts at Codex to see if it could figure it out.

According to the Codex /status command this used 169,818 input tokens, 17,112 output tokens and 1,176,320 cached input tokens. At current GPT-5 token pricing ($1.25/million input, $0.125/million cached input, $10/million output) that would cost 53.942 cents, but Codex CLI hooks into my existing $20/month ChatGPT Plus plan so this was bundled into that.

Via My Hacker News comment

Tags: javascript, ai, generative-ai, llama, local-llms, llms, ai-assisted-programming, transformers-js, webgpu, llm-pricing, vibe-coding, gpt-5, codex, gpt, onnx

Introducing EmbeddingGemma

2025-09-04T22:27:41+00:00

Introducing EmbeddingGemma

Brand new open weights (under the slightly janky Gemma license) 308M parameter embedding model from Google:

Based on the Gemma 3 architecture, EmbeddingGemma is trained on 100+ languages and is small enough to run on less than 200MB of RAM with quantization.

It's available via sentence-transformers, llama.cpp, MLX, Ollama, LMStudio and more.

As usual for these smaller models there's a Transformers.js demo (via) that runs directly in the browser (in Chrome variants) - Semantic Galaxy loads a ~400MB model and then lets you run embeddings against hundreds of text sentences, map them in a 2D space and run similarity searches to zoom to points within that space.

Tags: google, ai, embeddings, transformers-js, gemma, janky-licenses

DeepSeek Janus-Pro

2025-01-27T21:49:44+00:00

DeepSeek Janus-Pro

Another impressive model release from DeepSeek. Janus is their series of "unified multimodal understanding and generation models" - these are models that can both accept images as input and generate images for output.

Janus-Pro is the new 7B model, which DeepSeek describe as "an advanced version of Janus, improving both multimodal understanding and visual generation significantly". It's released under the not fully open source DeepSeek license.

Janus-Pro is accompanied by this paper, which includes this note about the training:

Our Janus is trained and evaluated using HAI-LLM, which is a lightweight and efficient distributed training framework built on top of PyTorch. The whole training process took about 7/14 days on a cluster of 16/32 nodes for 1.5B/7B model, each equipped with 8 Nvidia A100 (40GB) GPUs.

It includes a lot of high benchmark scores, but closes with some notes on the model's current limitations:

In terms of multimodal understanding, the input resolution is limited to 384 × 384, which affects its performance in fine-grained tasks such as OCR. For text-to-image generation, the low resolution, combined with reconstruction losses introduced by the vision tokenizer, results in images that, while rich in semantic content, still lack fine details. For example, small facial regions occupying limited image space may appear under-detailed. Increasing the image resolution could mitigate these issues.

The easiest way to try this one out is using the Hugging Face Spaces demo. I tried the following prompt for the image generation capability:

A photo of a raccoon holding a handwritten sign that says "I love trash"

And got back this image:

It's now also been ported to Transformers.js, which means you can run the 1B model directly in a WebGPU browser such as Chrome here at webml-community/janus-pro-webgpu (loads about 2.24 GB of model files).

Tags: ai, generative-ai, llms, transformers-js, vision-llms, deepseek, multi-modal-output, llm-release, ai-in-china

llama-3.2-webgpu

2024-09-30T16:27:22+00:00

llama-3.2-webgpu

Llama 3.2 1B is a really interesting models, given its 128,000 token input and its tiny size (barely more than a GB).

This page loads a 1.24GB q4f16 ONNX build of the Llama-3.2-1B-Instruct model and runs it with a React-powered chat interface directly in the browser, using Transformers.js and WebGPU. Source code for the demo is here.

It worked for me just now in Chrome; in Firefox and Safari I got a “WebGPU is not supported by this browser” error message.

Via @xenovacom

Tags: ai, webassembly, generative-ai, llama, llms, transformers-js, webgpu, onnx

Experimenting with local alt text generation in Firefox Nightly

2024-06-02T13:12:44+00:00

Experimenting with local alt text generation in Firefox Nightly

The PDF editor in Firefox (confession: I did not know Firefox ships with a PDF editor) is getting an experimental feature that can help suggest alt text for images for the human editor to then adapt and improve on.

This is a great application of AI, made all the more interesting here because Firefox will run a local model on-device for this, using a custom trained model they describe as "our 182M parameters model using a Distilled version of GPT-2 alongside a Vision Transformer (ViT) image encoder".

The model uses WebAssembly with ONNX running in Transfomers.js, and will be downloaded the first time the feature is put to use.

Via @mozhacks

Tags: accessibility, alt-text, firefox, javascript, mozilla, pdf, ai, webassembly, llms, transformers-js, onnx

The Tokenizer Playground

2024-03-19T02:18:59+00:00

The Tokenizer Playground

I built a tool like this a while ago, but this one is much better: it provides an interface for experimenting with tokenizers from a wide range of model architectures, including Llama, Claude, Mistral and Grok-1—all running in the browser using Transformers.js.

Via @xenovacom

Tags: ai, generative-ai, llms, transformers-js, tokenization

Adaptive Retrieval with Matryoshka Embeddings

2024-02-15T04:19:55+00:00

Adaptive Retrieval with Matryoshka Embeddings

Nomic Embed v1 only came out two weeks ago, but the same team just released Nomic Embed v1.5 trained using a new technique called Matryoshka Representation.

This means that unlike v1 the v1.5 embeddings are resizable - instead of a fixed 768 dimension embedding vector you can trade size for quality and drop that size all the way down to 64, while still maintaining strong semantically relevant results.

Joshua Lochner build this interactive demo on top of Transformers.js which illustrates quite how well this works: it lets you embed a query, embed a series of potentially matching text sentences and then adjust the number of dimensions and see what impact it has on the results.

Via @xenovacom

Tags: ai, llms, embeddings, nomic, transformers-js

Observable notebook: Detect objects in images

2023-10-01T15:46:14+00:00

Observable notebook: Detect objects in images

I built an Observable notebook that uses Transformers.js and the Xenova/detra-resnet-50 model to detect objects in images, entirely running within your browser. You can select an image using a file picker and it will show you that image with bounding boxes and labels drawn around items within it. I have a demo image showing some pelicans flying ahead, but it works with any image you give it - all without uploading that image to a server.

Via @simonw

Tags: javascript, machine-learning, transformers, ai, observable, transformers-js

Perplexity: interactive LLM visualization

2023-09-06T03:33:05+00:00

Perplexity: interactive LLM visualization

I linked to a video of Linus Lee's GPT visualization tool the other day. Today he's released a new version of it that people can actually play with: it runs entirely in a browser, powered by a 120MB version of the GPT-2 ONNX model loaded using the brilliant Transformers.js JavaScript library.

Via @thesephist

Tags: javascript, ai, webassembly, generative-ai, llms, transformers-js, onnx

Could you train a ChatGPT-beating model for $85,000 and run it in a browser?

2023-03-17T15:43:38+00:00

I think it's now possible to train a large language model with similar functionality to GPT-3 for $85,000. And I think we might soon be able to run the resulting model entirely in the browser, and give it capabilities that leapfrog it ahead of ChatGPT.

This is currently wild speculation on my part, but bear with me because I think this is worth exploring further.

Large language models with GPT-3-like capabilities cost millions of dollars to build, thanks to the cost of running the expensive GPU servers needed to train them. Whether you are renting or buying those machines, there are still enormous energy costs to cover.

Just one example of this: the BLOOM large language model was trained in France with the support of the French government. The cost was estimated as $2-5M, it took almost four months to train and boasts about its low carbon footprint because most of the power came from a nuclear reactor!

[ Fun fact: as of a few days ago you can now run the openly licensed BLOOM on your own laptop, using Nouamane Tazi's adaptive copy of the llama.cpp code that made that possible for LLaMA ]

Recent developments have made me suspect that these costs could be made dramatically lower. I think a capable language model can now be trained from scratch for around $85,000.

It's all about that LLaMA

The LLaMA plus Alpaca combination is the key here.

I wrote about these two projects previously:

Large language models are having their Stable Diffusion moment discusses the significance of LLaMA
Stanford Alpaca, and the acceleration of on-device large language model development describes Alpaca

To recap: LLaMA by Meta research provided a GPT-3 class model trained entirely on documented, available public training information, as opposed to OpenAI's continuing practice of not revealing the sources of their training data.

This makes the model training a whole lot more likely to be replicable by other teams.

The paper also describes some enormous efficiency improvements they made to the training process.

The LLaMA research was still extremely expensive though. From the paper:

... we estimate that we used 2048 A100-80GB for a period of approximately 5 months to develop our models

My friends at Replicate told me that a simple rule of thumb for A100 cloud costs is $1/hour.

2048 * 5 * 30 * 24 = $7,372,800

But... that $7M was the cost to both iterate on the model and to train all four sizes of LLaMA that they tried: 7B, 13B, 33B, and 65B.

Here's Table 15 from the paper, showing the cost of training each model.

This shows that the smallest model, LLaMA-7B, was trained on 82,432 hours of A100-80GB GPUs, costing 36MWh and generating 14 tons of CO2.

(That's about 28 people flying from London to New York.)

Going by the $1/hour rule of thumb, this means that provided you get everything right on your first run you can train a LLaMA-7B scale model for around $82,432.

Upgrading to Alpaca

You can run LLaMA 7B on your own laptop (or even on a phone), but you may find it hard to get good results out of. That's because it hasn't been instruction tuned, so it's not great at answering the kind of prompts that you might send to ChatGPT or GPT-3 or 4.

Alpaca is the project from Stanford that fixes that. They fine-tuned LLaMA on 52,000 instructions (of somewhat dubious origin) and claim to have gotten ChatGPT-like performance as a result... from that smallest 7B LLaMA model!

You can try out their demo (update: no you can't, "Our live demo is suspended until further notice") and see for yourself that it really does capture at least some of that ChatGPT magic.

The best bit? The Alpaca fine-tuning can be done for less than $100. The Replicate team have repeated the training process and published a tutorial about how they did it.

Other teams have also been able to replicate the Alpaca fine-tuning process, for example antimatter15/alpaca.cpp on GitHub.

We are still within our $85,000 budget! And Alpaca - or an Alpaca-like model using different fine tuning data - is the ChatGPT on your own device model that we've all been hoping for.

Could we run it in a browser?

Alpaca is effectively the same size as LLaMA 7B - around 3.9GB (after 4-bit quantization ala llama.cpp). And LLaMA 7B has already been shown running on a whole bunch of different personal devices: laptops, Raspberry Pis (very slowly) and even a Pixel 5 phone at a decent speed!

The next frontier: running it in the browser.

I saw two tech demos yesterday that made me think this may be possible in the near future.

The first is Transformers.js. This is a WebAssembly port of the Hugging Face Transformers library of models - previously only available for server-side Python.

It's worth spending some time with their demos, which include some smaller language models and some very impressive image analysis languages too.

The second is Web Stable Diffusion. This team managed to get the Stable Diffusion generative image model running entirely in the browser as well!

Web Stable Diffusion uses WebGPU, a still emerging standard that's currently only working in Chrome Canary. But it does work! It rendered me this image of two raccoons eating a pie in the forest in 38 seconds.

The Stable Diffusion model this loads into the browser is around 1.9GB.

LLaMA/Alpaca at 4bit quantization is 3.9GB.

The sizes of these two models are similar enough that I would not be at all surprised to see an Alpaca-like model running in the browser in the not-too-distant future. I wouldn't be surprised if someone is working on that right now.

Now give it extra abilities with ReAct

A model running in your browser that behaved like a less capable version of ChatGPT would be pretty impressive. But what if it could be MORE capable than ChatGPT?

The ReAct prompt pattern is a simple, proven way of expanding a language model's abilities by giving it access to extra tools.

Matt Webb explains the significance of the pattern in The surprising ease and effectiveness of AI in a loop.

I got it working with a few dozen lines of Python myself, which I described in A simple Python implementation of the ReAct pattern for LLMs.

Here's the short version: you tell the model that it must think out loud and now has access to tools. It can then work through a question like this:

Question: Population of Paris, squared?

Thought: I should look up the population of paris and then multiply it

Action: search_wikipedia: Paris

Then it stops. Your code harness for the model reads that last line, sees the action and goes and executes an API call against Wikipedia. It continues the dialog with the model like this:

Observation: <truncated content from the Wikipedia page, including the 2,248,780 population figure>

The model continues:

Thought: Paris population is 2,248,780 I should square that

Action: calculator: 2248780 ** 2

Control is handed back to the harness, which passes that to a calculator and returns:

Observation: 5057011488400

The model then provides the answer:

Answer: The population of Paris squared is 5,057,011,488,400

Adding new actions to this system is trivial: each one can be a few lines of code.

But as the ReAct paper demonstrates, adding these capabilities to even an under-powered model (such as LLaMA 7B) can dramatically improve its abilities, at least according to several common language model benchmarks.

This is essentially what Bing is! It's GPT-4 with the added ability to run searches against the Bing search index.

Obviously if you're going to give a language model the ability to execute API calls and evaluate code you need to do it in a safe environment! Like for example... a web browser, which runs code from untrusted sources as a matter of habit and has the most thoroughly tested sandbox mechanism of any piece of software we've ever created.

Adding it all together

There are a lot more groups out there that can afford to spend $85,000 training a model than there are that can spend $2M or more.

I think LLaMA and Alpaca are going to have a lot of competition soon, from an increasing pool of openly licensed models.

A fine-tuned LLaMA scale model is leaning in the direction of a ChatGPT competitor already. But... if you hook in some extra capabilities as seen in ReAct and Bing even that little model should be able to way outperform ChatGPT in terms of actual ability to solve problems and do interesting things.

And we might be able to run such a thing on our phones... or even in our web browsers... sooner than you think.

And it's only going to get cheaper

Tobias Lütke on Twitter:

H100s are shipping and you can half this again. Twice (or more) if fp8 works.
- tobi lutke (@tobi) March 17, 2023

The H100 is the new Tensor Core GPU from NVIDIA, which they claim can offer up to a 30x performance improvement over their current A100s.

Tags: ai, webassembly, generative-ai, chatgpt, llama, local-llms, llms, bloom, mlc, transformers-js, llm-tool-use, llama-cpp, tobias-lutke

Transformers.js

2023-03-16T23:41:55+00:00

Transformers.js

Hugging Face Transformers is a library of Transformer machine learning models plus a Python package for loading and running them. Transformers.js provides a JavaScript alternative interface which runs in your browser, thanks to a set of precompiled WebAssembly binaries for a selection of models. This interactive demo is incredible: in particular, try running the Image classification with google/vit-base-patch16-224 (91MB) model against any photo to get back labels representing that photo. Dropping one of these models onto a page is as easy as linking to a hosted CDN script and running a few lines of JavaScript.

Tags: javascript, machine-learning, transformers, ai, webassembly, generative-ai, llms, hugging-face, transformers-js