Build your own SQS or Kafka with Postgres

2024-07-31T17:34:54+00:00

Build your own SQS or Kafka with Postgres

Anthony Accomazzo works on Sequin, an open source "message stream" (similar to Kafka) written in Elixir and Go on top of PostgreSQL.

This detailed article describes how you can implement message queue patterns on PostgreSQL from scratch, including this neat example using a CTE, returning and for update skip locked to retrieve $1 messages from the messages table and simultaneously mark them with not_visible_until set to $2 in order to "lock" them for processing by a client:

with available_messages as (
  select seq
  from messages
  where not_visible_until is null
    or (not_visible_until <= now())
  order by inserted_at
  limit $1
  for update skip locked
)
update messages m
set
  not_visible_until = $2,
  deliver_count = deliver_count + 1,
  last_delivered_at = now(),
  updated_at = now()
from available_messages am
where m.seq = am.seq
returning m.seq, m.data;

Via lobste.rs

Tags: message-queues, postgresql, sql, sqs, kafka

Mass Video Conversion Using AWS

2007-04-03T23:44:37+00:00

Mass Video Conversion Using AWS

How to use S3, SQS, EC2, ffmpeg and some Python to bulk convert videos with Amazon Web Services.

Tags: amazon, aws, ec2, ffmpeg, python, s3, sqs

boto

2007-02-11T00:17:52+00:00

boto

Python library for accessing Amazon’s S3, SQS and EC2 Web Services, with excellent documentation.

Tags: amazon, aws, boto, ec2, python, s3, sqs

Simon Willison's Weblog: sqs

Build your own SQS or Kafka with Postgres

Mass Video Conversion Using AWS

boto