Simon Willison's Weblog: rate-limiting

datasette-ip-rate-limit 0.1a0

2026-05-14T04:10:23+00:00

Release: datasette-ip-rate-limit 0.1a0

The datasette.io site was being hammered by poorly-behaved crawlers, so I had Codex (GPT-5.5 xhigh) build a configurable rate limiting plugin to block IPs that were hammering specific areas of the site too quickly.

Here's the production configuration I'm using on that site for the new plugin:

  datasette-ip-rate-limit:
    header: Fly-Client-IP
    max_keys: 10000
    exempt_paths:
    - "/static/*"
    - "/-/turnstile*"
    rules:
    - name: demo-databases
      paths:
      - "/global-power-plants/*"
      - "/legislators/*"
      window_seconds: 60
      max_requests: 60
      block_seconds: 20

Tags: rate-limiting, datasette, codex

TIL: Rate limiting by IP using Cloudflare's rate limiting rules

2025-07-03T21:16:51+00:00

TIL: Rate limiting by IP using Cloudflare's rate limiting rules

My blog started timing out on some requests a few days ago, and it turned out there were misbehaving crawlers that were spidering my /search/ page even though it's restricted by robots.txt.

I run this site behind Cloudflare and it turns out Cloudflare's WAF (Web Application Firewall) has a rate limiting tool that I could use to restrict requests to /search/* by a specific IP to a maximum of 5 every 10 seconds.

Tags: rate-limiting, security, cloudflare, til

Reservoir Sampling

2025-05-08T21:00:22+00:00

Reservoir Sampling

Yet another outstanding interactive essay by Sam Rose (previously), this time explaining how reservoir sampling can be used to select a "fair" random sample when you don't know how many options there are and don't want to accumulate them before making a selection.

Reservoir sampling is one of my favourite algorithms, and I've been wanting to write about it for years now. It allows you to solve a problem that at first seems impossible, in a way that is both elegant and efficient.

I appreciate that Sam starts the article with "No math notation, I promise." Lots of delightful widgets to interact with here, all of which help build an intuitive understanding of the underlying algorithm.

Sam shows how this algorithm can be applied to the real-world problem of sampling log files when incoming logs threaten to overwhelm a log aggregator.

The dog illustration is commissioned art and the MIT-licensed code is available on GitHub.

Via Hacker News

Tags: algorithms, logging, rate-limiting, explorables, sam-rose

DeepSeek API Docs: Rate Limit

2025-01-18T18:24:38+00:00

DeepSeek API Docs: Rate Limit

This is surprising: DeepSeek offer the only hosted LLM API I've seen that doesn't implement rate limits:

DeepSeek API does NOT constrain user's rate limit. We will try out best to serve every request.

However, please note that when our servers are under high traffic pressure, your requests may take some time to receive a response from the server.

Want to run a prompt against 10,000 items? With DeepSeek you can theoretically fire up 100s of parallel requests and crunch through that data in almost no time at all.

As more companies start building systems that rely on LLM prompts for large scale data extraction and manipulation I expect high rate limits will become a key competitive differentiator between the different platforms.

Tags: rate-limiting, ai, generative-ai, llms, deepseek, ai-in-china

aiolimiter

2024-02-20T01:15:27+00:00

aiolimiter

I found myself wanting an asyncio rate limiter for Python today—so I could send POSTs to an API endpoint no more than once every 10 seconds. This library worked out really well—it has a very neat design and lets you set up rate limits for things like “no more than 50 items every 10 seconds”, implemented using the leaky bucket algorithm.

Tags: async, python, rate-limiting

Quoting Push notification two-factor auth considered harmful

2022-09-17T14:45:21+00:00

However, six digits is a very small space to search through when you are a computer. The biggest problem is going to be getting lucky, it's quite literally a one-in-a-million shot. Turns out you can brute force a TOTP code in about 2 hours if you are careful and the remote service doesn't have throttling or rate limiting of authentication attempts.

— Push notification two-factor auth considered harmful

Tags: rate-limiting, security

Scaling a High-traffic Rate Limiting Stack With Redis Cluster

2018-04-26T18:34:25+00:00

Scaling a High-traffic Rate Limiting Stack With Redis Cluster

Brandur Leach describes the simple, elegant and performant design of Redis Cluster, and talks about how Stripe used it to scaled their rate-limiting from one to ten nodes.

Tags: rate-limiting, redis, scaling, brandur-leach, stripe

How could GitHub improve the password security of its users?

2013-11-20T17:50:00+00:00

My answer to How could GitHub improve the password security of its users? on Quora

By doing exactly what they're doing already: adding more sophisticated rate limiting, and preventing users from using common weak passwords.

Their account security practices are already best-in-industry: they support two-factor authentication and their "Security History" interface at https://github.com/settings/secu... is the best I've seen on any website.

The way they store passwords (correctly, using bcrypt) had nothing to do with this particular security incident.

Tags: github, open-source, passwords, rate-limiting, security, quora

Does Twitter use a 3rd party software for rate limiting their APIs? If yes, who's the 3rd party?

2010-11-21T14:27:00+00:00

My answer to Does Twitter use a 3rd party software for rate limiting their APIs? If yes, who's the 3rd party? on Quora

I wrote up a technique for doing simple rate limiting using memcached a while ago, which I later found out was somewhat similar to how the Twitter API does it.

http://simonwillison.net/2009/Ja...

Tags: apis, rate-limiting, twitter, quora

Rate limiting with memcached

2009-01-07T22:27:08+00:00

On Monday, several high profile "celebrity" Twitter accounts started spouting nonsense, the victims of stolen passwords. Wired has the full story - someone ran a dictionary attack against a Twitter staff member, discovered their password and used Twitter's admin tools to reset the passwords on the accounts they wanted to steal.

The Twitter incident got me thinking about rate limiting again. I've been wanting a good general solution to this problem for quite a while, for API projects as well as security. Django Snippets has an answer, but it works by storing access information in the database and requires you to run a periodic purge command to clean up the old records.

I'm strongly averse to writing to the database for every hit. For most web applications reads scale easily, but writes don't. I also want to avoid filling my database with administrative gunk (I dislike database backed sessions for the same reason). But rate limiting relies on storing state, so there has to be some kind of persistence.

Using memcached counters

I think I've found a solution, thanks to memcached and in particular the incr command. incr lets you atomically increment an already existing counter, simply by specifying its key. add can be used to create that counter - it will fail silently if the provided key already exists.

Let's say we want to limit a user to 10 hits every minute. A naive implementation would be to create a memcached counter for hits from that user's IP address in a specific minute. The counter key might look like this:

ratelimit_72.26.203.98_2009-01-07-21:45

Increment that counter for every hit, and if it exceeds 10 block the request.

What if the user makes ten requests all in the last second of the minute, then another ten a second later? The rate limiter will let them off. For many cases this is probably acceptable, but we can improve things with a slightly more complex strategy. Let's say we want to allow up to 30 requests every five minutes. Instead of maintaining one counter, we can maintain five - one for each of the past five minutes (older counters than that are allowed to expire). After a few minutes we might end up with counters that look like this:

ratelimit_72.26.203.98_2009-01-07-21:45 = 13
ratelimit_72.26.203.98_2009-01-07-21:46 = 7
ratelimit_72.26.203.98_2009-01-07-21:47 = 11

Now, on every request we work out the keys for the past five minutes and use get_multi to retrieve them. If the sum of those counters exceeds the maximum allowed for that time period, we block the request.

Are there any obvious flaws to this approach? I'm pretty happy with it - it cleans up after itself (old counters quietly expire from the cache), it shouldn't use much resources (just five active cache keys per unique IP address at any one time) and if the cache is lost the only snag is that a few clients might go slightly over their rate limit. I don't think it's possible for an attacker to force the counters to expire early.

An implementation for Django

I've put together an example implementation of this algorithm using Django, hosted on GitHub. The readme.txt file shows how it works - basic usage is via a simple decorator:

from ratelimitcache import ratelimit

@ratelimit(minutes = 3, requests = 20)
def myview(request):
    # ...
    return HttpResponse('...')

Python decorators are typically functions, but ratelimit is actually a class. This means it can be customised by subclassing it, and the class provides a number of methods designed to be over-ridden. I've provided an example of this in the module itself - ratelimit_post, a decorator which only limits on POST requests and can optionally couple the rate limiting to an individual POST field. Here's the complete implementation:

class ratelimit_post(ratelimit):
    "Rate limit POSTs - can be used to protect a login form"
    key_field = None # If provided, this POST var will affect the rate limit
    
    def should_ratelimit(self, request):
        return request.method == 'POST'
    
    def key_extra(self, request):
        # IP address and key_field (if it is set)
        extra = super(ratelimit_post, self).key_extra(request)
        if self.key_field:
            value = sha.new(request.POST.get(self.key_field, '')).hexdigest()
            extra += '-' + value
        return extra

And here's how you would use it to limit the number of times a specific IP address can attempt to log in as a particular user:

@ratelimit_post(minutes = 3, requests = 10, key_field = 'username')
def login(request):
    # ...
    return HttpResponse('...')

The should_ratelimit() method is called before any other rate limiting logic. The default implementation returns True, but here we only want to apply rate limits to POST requests. The key_extra() method is used to compose the keys used for the counter - by default this just includes the request's IP address, but in ratelimit_post we can optionally include the value of a POST field (for example the username). We could include things like the request path here to apply different rate limit counters to different URLs.

Finally, the readme.txt includes ratelimit_with_logging, an example that over-rides the disallowed() view returned when a rate limiting condition fails and writes an audit note to a database (less overhead than writing for every request).

I've been a fan of customisation via subclassing ever since I got to know the new Django admin system, and I've been using it in a bunch of projects. It's a great way to create reusable pieces of code.

Tags: counters, django, github, memcached, projects, python, ratelimitcache, rate-limiting, security, twitter

Decorator to limit request rates to individual views

2008-09-24T13:13:29+00:00

Decorator to limit request rates to individual views

Neat piece of code for public facing web APIs written in Django. Update: some smart criticisms in the comments.

Tags: apis, decorators, django, python, rate-limiting