Simon Willison's Weblog: kubernetes

suitenumerique/docs

2025-03-17T18:51:50+00:00

New open source (MIT licensed) collaborative text editing web application, similar to Google Docs or Notion, notable because it's a joint effort funded by the French and German governments and "currently onboarding the Netherlands".

It's built using Django and React:

Docs is built on top of Django Rest Framework, Next.js, BlockNote.js, HocusPocus and Yjs.

Deployments currently require Kubernetes, PostgreSQL, memcached, an S3 bucket (or compatible) and an OIDC provider.

Tags: django, open-source, postgresql, s3, react, kubernetes

OpenAI's postmortem for API, ChatGPT & Sora Facing Issues

2024-12-13T05:29:10+00:00

OpenAI's postmortem for API, ChatGPT & Sora Facing Issues

OpenAI had an outage across basically everything for four hours on Wednesday. They've now published a detailed postmortem which includes some fascinating technical details about their "hundreds of Kubernetes clusters globally".

The culprit was a newly deployed telemetry system:

Telemetry services have a very wide footprint, so this new service’s configuration unintentionally caused every node in each cluster to execute resource-intensive Kubernetes API operations whose cost scaled with the size of the cluster. With thousands of nodes performing these operations simultaneously, the Kubernetes API servers became overwhelmed, taking down the Kubernetes control plane in most of our large clusters. [...]

The Kubernetes data plane can operate largely independently of the control plane, but DNS relies on the control plane – services don’t know how to contact one another without the Kubernetes control plane. [...]

DNS caching mitigated the impact temporarily by providing stale but functional DNS records. However, as cached records expired over the following 20 minutes, services began failing due to their reliance on real-time DNS resolution.

It's always DNS.

Via @therealadamg

Tags: dns, devops, kubernetes, postmortem, openai, chatgpt

Quoting Jake Teton-Landis

2024-09-25T18:08:19+00:00

We used this model [periodically transmitting configuration to different hosts] to distribute translations, feature flags, configuration, search indexes, etc at Airbnb. But instead of SQLite we used Sparkey, a KV file format developed by Spotify. In early years there was a Cron job on every box that pulled that service’s thingies; then once we switched to Kubernetes we used a daemonset & host tagging (taints?) to pull a variety of thingies to each host and then ensure the services that use the thingies only ran on the hosts that had the thingies.

— Jake Teton-Landis

Tags: sqlite, kubernetes, feature-flags, baked-data

Carving the Scheduler Out of Our Orchestrator

2023-02-02T21:46:35+00:00

Carving the Scheduler Out of Our Orchestrator

Thomas Ptacek describes Fly’s new custom-built alternative to Nomad and Kubernetes in detail, including why they eventually needed to build something custom to best serve their platform. In doing so he provides the best explanation I’ve ever seen of what an orchestration system actually does.

Via @tqbf

Tags: thomas-ptacek, kubernetes, fly

Two reasons Kubernetes is so complex

2022-01-27T18:25:38+00:00

Two reasons Kubernetes is so complex

I like how this article proposes that Kubernetes isn’t trying to be a tool for deploying containers—it’s more like an operating system for a cluster of machines, responsible for the same kind of goals as a regular operating system such as resource sharing and portability. And since everything is built as control loops which attempt to modify actual state to fit the declarative desired state, errors can occur asynchronously seconds or even minutes after the desired state has been updated.

Via Hacker News

Tags: kubernetes

Weeknotes: Learning Kubernetes, learning Web Components

2021-10-28T02:44:33+00:00

I've been mainly climbing the learning curve for Kubernetes and Web Components this week. I also released Datasette 0.59.1 with Python 3.10 compatibility and an updated Docker image.

Datasette 0.59.1

A few weeks ago I wrote about finding and reporting an asyncio bug in Python 3.10 that I discovered while trying to get Datasette to work on the latest release of Python.

Łukasz Langa offered a workaround which I submitted as a PR to the Janus library that Datasette depends on.

Andrew Svetlov landed and shipped that fix, which unblocked me from releasing Datasette 0.59.1 that works with Python 3.10.

The last step of the Datasette release process, after the package has been released to PyPI, is to build a new Docker image and publish it to Docker Hub. Here's the GitHub Actions workflow that does that.

It turns out this stopped working when I released Datasette 0.59! I was getting this cryptic error message half way through the image build process:

/usr/bin/perl: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory

I opened an issue for myself and started investigating.

The culprit was this section of the Datasette Dockerfile:

# software-properties-common provides add-apt-repository 
# which we need in order to install a more recent release 
# of libsqlite3-mod-spatialite from the sid distribution 
RUN apt-get update && \ 
    apt-get -y --no-install-recommends install software-properties-common && \ 
    add-apt-repository "deb http://httpredir.debian.org/debian sid main" && \ 
    apt-get update && \ 
    apt-get -t sid install -y --no-install-recommends libsqlite3-mod-spatialite && \ 
    apt-get remove -y software-properties-common && \

This was a hack I introduced seven months ago in order to upgrade the bundled SpatiaLite to version 5.0.

SpatiaLite 5.0 wasn't yet available in Debian stable back then, so I used the above convoluted hack to install it from Debian unstable ("Sid") instead.

When the latest stable version of Debian, Debian Bullseye, came out on October 9th my hack stopped working! I guess that's what I get for messing around with unstable software.

Thankfully, Bullseye now bundles SpatiaLite 5, so the hack I was using is no longer necessary. I upgraded the Datasette base image from python:3.9.2-slim-buster to 3.9.7-slim-bullseye, installed SpatialLite the non-hacky way and fixed the issue.

Doing so also dropped the size of the compressed Datasette image from 94.37MB to 78.94MB, which is nice.

Learning Kubernetes

Datasette has been designed to run in containers from the very start. I have dozens of instances running on Google Cloud Run, and I've done a bunch of work with Docker as well, including trying out mechanisms to programatically launch new Datasette containers via the Docker API.

I've dragged my heels on really getting into Kubernetes due to the infamously tough learning curve, but I think it's time to dig in, figure out how to use it and work out what new abilities it can provide me.

I've spun up small a Kubernetes cluster on Digital Ocean, mainly because I trust their UI to help me not spend hundreds of dollars by mistake. Getting the initial cluster running was very pleasant.

Now I'm figuring out how to do things with it.

DigitalOcean's Operations-ready DigitalOcean Kubernetes (DOKS) for Developers course (which started as a webinar) starts OK and then gets quite complicated quite fast.

I got Paul Bouwer's hello-kubernetes demo app working - it introduced me to Helm, but that operates at a higher level than I'm comfortable with - learning my way around kubectl and Kubernetes YAML is enough of a mental load already without adding an extra abstraction on top.

I'm reading Kubernetes: Up and Running which is promising so far.

My current goal is to figure out how to run a Datasette instance in a Kubernetes container with an attached persistent volume, so it can handle SQLite writes as well as reads. It looks like StatefulSets will be key to getting that to work. (Update: apparently not! Graham Dumpleton and Frank Wiles assure me that I can do this with just a regular Deployment.)

I'll be sure to write this up as a TIL once I get it working.

Learning Web Components

Datasette's visualization plugins - in particular datasette-vega - are long overdue for some upgrades.

I've been trying to find a good pattern for writing plugins that avoids too much (ideally any) build tool complexity, and that takes advantage of modern JavaScript - in particular JavaScript modules, which Datasette has supported since Datasette 0.54.

As such, I'm deeply intrigued by Web Components - which had a big moment this week when it was revealed that Adobe had used them extensively for Photoshop on the web.

One of my goals for Datasette visualization plugins is for them to be usable on other external pages - since Datasette can expose JSON data over CORS, being able to drop a visualization into an HTML page would be really neat (especially for newsroom purposes).

Imagine being able to import a JavaScript module and add something like this to get a map of all of the power plants in Portugal:

<datasette-cluster-map
  data="https://global-power-plants.datasettes.com/global-power-plants/global-power-plants.json?country_long=Portugal">
</datasette-cluster-map>

I'm hoping to be able to build components using regular, unadorned modern JavaScript, without the complexity of a build step.

As such, I've been exploring Skypack (TIL) and Snowpack which help bridge the gap between build-tooling-dependent npm packages and the modern world of native browser ES modules.

I was also impressed this week by Tonic, a framework for building components without a build step that weighs in at just 350 lines of code and makes extremely clever use of tagged template literals and async generators.

This morning I saw this clever example of a Single File Web Component by Kristofer Joseph - I ended up creating my own annotated version of his code which I shared in this TIL.

Next step: I need to write some web components of my own!

Releases this week

datasette: 0.59.1 - (99 releases total) - 2021-10-24
An open source multi-tool for exploring and publishing data
datasette-hello-world: 0.1 - 2021-10-21
The hello world of Datasette plugins

TIL this week

Tags: docker, datasette, kubernetes, web-components, weeknotes

Smooth sailing with Kubernetes

2021-07-01T23:30:57+00:00

Smooth sailing with Kubernetes

Scott McCloud (of Understanding Comics) authored this comic introduction to Kubernetes, and it’s a really good explanation of the core concepts. I’d love to have something like this for Datasette—I still feel like I’m a long way from being able to explain the project with anything like this amount of clarity.

Via @kaslinfields

Tags: comics, kubernetes

Experiences with running PostgreSQL on Kubernetes

2018-08-13T14:30:52+00:00

Experiences with running PostgreSQL on Kubernetes

Fascinating interview that makes a solid argument for the idea that running stateful data stores like PostgreSQL or Cassandra is made harder, not easier when you add an orchestration tool like Kubernetes into the mix.

Via @mipsytipsy

Tags: postgresql, kubernetes