Simon Willison's Weblog: queues

How does Sidekiq really work?

2024-02-05T17:20:53+00:00

I really like this category of blog post: Dan Svetlov took the time to explore the Sidekiq message queue’s implementation and then wrote it up in depth.

Via Hacker News

Tags: queues, ruby, sidekiq

Transactionally Staged Job Drains in Postgres

2021-12-18T01:34:13+00:00

Transactionally Staged Job Drains in Postgres

Any time I see people argue that relational databases shouldn’t be used to implement job queues I think of this post by Brandur from 2017. If you write to a queue before committing a transaction you run the risk of a queue consumer trying to read from the database before the new row becomes visible. If you write to the queue after the transaction there’s a risk an error might result in your message never being written. So: write to a relational staging table as part of the transaction, then have a separate process read from that table and write to the queue.

Tags: postgresql, queues, scaling, brandur-leach

huey

2019-02-25T19:49:06+00:00

huey

Charles Leifer’s “little task queue for Python”. Similar to Celery, but it’s designed to work with Redis, SQLite or in the parent process using background greenlets. Worth checking out for the really neat design. The project is new to me, but it’s been under active development since 2011 and has a very healthy looking rate of releases.

Tags: python, queues, redis, sqlite, charles-leifer

Introducing Faktory

2017-10-25T03:09:39+00:00

Introducing Faktory

A brand new worker queue system from Mike Perham, the author of Sidekiq for Ruby. It’s written in Go on top of RocksDB and is explicitly designed to support clients and workers in multiple different languages.

Tags: go, queues, sidekiq

HotQueue

2010-12-22T11:51:00+00:00

HotQueue

A super-simple Python work queue using Redis. The API is neat, and makes clever use of generators for blocking consumption of queue items.

Tags: python, queues, redis, recovered, hotqueue

Introducing docent

2010-01-28T20:35:22+00:00

Introducing docent

Paul Mison’s clever little Flickr app for viewing galleries that have been added by your contacts. It runs in Python on App Engine and makes extensive use of the Task Queue API.

Tags: docent, flickr, google-app-engine, paul-mison, python, queues, taskqueue

BLPOP and BRPOP in Redis

2010-01-07T22:50:22+00:00

BLPOP and BRPOP in Redis

Added over Christmas—Redis now has blocking list pop operations. This means you can use Redis to drive a queue server without the need for polling—simply BLPOP against a key and, if it’s empty, your client will block until another client pushes an item on to the list. Multiple clients can block against the same key and only the first client will return when an item becomes available.

Tags: blocking, queues, redis

Announcing Alice and Wonderland

2009-07-17T09:12:45+00:00

Announcing Alice and Wonderland

Continuing the RabbitMQ “stuff to do with rabbits” naming convention, Alice is a RESTful interface to RabbitMQ which exposes information about vhosts/queues/users/exchanges/etc as JSON. Wonderland is a web UI for RabbitMQ implemented as a pure Ajax application which calls Alice.

Tags: ajax, alice, aliceinwonderland, javascript, json, message-queues, queues, rabbitmq, rest, wonderland

Rabbits and warrens

2009-04-07T09:13:39+00:00

Rabbits and warrens

Handy tutorial introduction to using RabbitMQ and AMQP with Python.

Tags: amqp, message-queues, python, queues, rabbitmq, tutorial

New Gearman Server & Library in C, MySQL UDFs

2009-01-13T16:41:57+00:00

New Gearman Server & Library in C, MySQL UDFs

Gearman, the job queue written for LiveJournal and now used by Digg and Yahoo!, has been rewritten in C. Looks like a good candidate for an easily configured lightweight message queue. Also includes hooks for writing MySQL functions that can interact with queues.

Tags: c, digg, eric-day, gearman, livejournal, message-queues, mysql, queues, scaling, yahoo

Flickr Engineers Do It Offline

2008-09-28T01:24:57+00:00

Flickr Engineers Do It Offline

Flickr wrote their own queuing mechanism (in PHP), and currently run ten queue servers on dedicated hardware for tasks like pushing new photos in to indexes, denormalisation and “backfills” which move data between clusters and run bulk scripts against large numbers of existing rows.

Tags: backfills, denormalisation, flickr, message-queues, queues

Queue everything and delight everyone

2008-07-04T22:38:01+00:00

Queue everything and delight everyone

Les Orchard explains why I’ve been getting interested in queues recently: “One of the problems it seems most modern web apps face is the tendency to want to do everything all at once, and all in the same code that responds directly to a user.”

Tags: les-orchard, queues

Standing in Line

2008-03-19T09:41:11+00:00

Standing in Line

Simon Wistow coins “CLAMP” for LAMP + Cache, and expresses the need for a dirt-simple, high performance open source queue system.

Tags: clamp, lamp, open-source, queues, simon-wistow