Simon Willison's Weblog: cgi

Serving 200 million requests per day with a cgi-bin

2025-07-05T23:28:31+00:00

Serving 200 million requests per day with a cgi-bin

Jake Gold tests how well 90s-era CGI works today, using a Go + SQLite CGI program running on a 16-thread AMD 3700X.

Using CGI on modest hardware, it’s possible to serve 2400+ requests per second or 200M+ requests per day.

I got my start in web development with CGI back in the late 1990s - I was a huge fan of NewsPro, which was effectively a weblog system before anyone knew what a weblog was.

CGI works by starting, executing and terminating a process for every incoming request. The nascent web community quickly learned that this was a bad idea, and invented technologies like PHP and FastCGI to help avoid that extra overhead and keep code resident in-memory instead.

This lesson ended up baked into my brain, and I spent the next twenty years convinced that you should never execute a full process as part of serving a web page.

Of course, computers in those two decades got a lot faster. I finally overcame that twenty-year core belief in 2020, when I built datasette-ripgrep, a Datasette plugin that shells out to the lightning fast ripgrep CLI tool (written in Rust) to execute searches. It worked great!

As was pointed out on Hacker News, part of CGI's problem back then was that we were writing web scripts in languages like Perl, Python and Java which had not been designed for lightning fast startup speeds. Using Go and Rust today helps make CGI-style requests a whole lot more effective.

Jake notes that CGI-style request handling is actually a great way to take advantage of multiple CPU cores:

These days, we have servers with 384 CPU threads. Even a small VM can have 16 CPUs. The CPUs and memory are much faster as well.

Most importantly, CGI programs, because they run as separate processes, are excellent at taking advantage of many CPUs!

Maybe we should start coding web applications like it's 1998, albeit with Go and Rust!

To clarify, I don't think most people should do this. I just think it's interesting that it's not as bad an idea as it was ~25 years ago.

Via Hacker News

Tags: cgi, go, performance, sqlite

websocketd

2019-01-26T02:38:53+00:00

websocketd

Delightfully clever piece of design: “It’s like CGI, twenty years later, for WebSockets”. Simply run “websocketd --port=8080 my-program” and it will start up a WebSocket server on port 8080 and fire up a new process running your script every time it sees a new WebSocket connection. Standard in and standard out are automatically hooked up to the socket connection. Since it spawns a new process per connection this won’t work well with thousands of connections but for smaller scale projects it’s an excellent addition to the toolbok—and since it’s written in Go there are pre-compiled binaries available for almost everything.

Via Hacker News

Tags: cgi, go, websockets

Are there any alternatives to CGI for web servers?

2013-08-10T12:14:00+00:00

My answer to Are there any alternatives to CGI for web servers? on Quora

Yes. CGI stopped being relevant around the turn if the century! Many languages can be embedded in web servers now (mod_php, mod_python, mod_perl for Apache etc), there's the FastCGI protocol which allows web servers to communicate with external processes without needing to start a brand new process for every request, and it's also common these days to run an HTTP server written in the same language as your application and proxy requests to it - Unicorn, gunicorn, Jetty are all examples of this.

Tags: cgi, web-development, quora

sfical.py

2008-06-27T08:09:42+00:00

sfical.py

Neat idea: write a CGI script that turns a proprietary API (in this case the SalesForce events API) in to standard ical format, then run it on your Mac’s local Apache server and subscribe to it from iCal.

Tags: apache, cgi, icalendar, mac, macos, salesforce, simon-fell

.php? .cgi? .who-cares?

2007-02-09T01:01:09+00:00

.php? .cgi? .who-cares?

J-P Stacey argues that “URLs need to be hackable by the developer as well as by the user”. There’s certainly room for improvement in keeping complex URL structures maintainable from a server-side developer’s perspective.

Tags: cgi, urls

SubWiki

2006-12-16T19:19:51+00:00

SubWiki

A wiki that uses Subversion for its data repository, implemented as a Python CGI.

Via Google Code gets wikis and file downloads

Tags: cgi, python, subversion, wiki

Apache Module mod_actions

2004-08-20T22:44:50+00:00

Apache Module mod_actions

Channel all requests for a specific file type through a CGI script.

Tags: cgi

Pwyky (A Python Wiki)

2004-01-26T07:08:36+00:00

Pwyky (A Python Wiki)

A neat little wiki in a single CGI file

Via the disobey nonsense network

Tags: cgi

cgi_buffer

2003-11-22T18:36:28+00:00

cgi_buffer is voodoo magic for Perl, Python and PHP scripts that automatically handles a bunch of bandwidth saving HTTP tricks such as Content-Length headers (which enable persistent connections), ETags for caching and GZip content compression. Pretty neat.

Tags: cgi

The difference between POST and GET

2003-10-25T03:49:19+00:00

How important is the ability to tell the difference between data sent by POST and data sent by GET (i.e in the query string) when developing web applications? Some web frameworks (such as PHP) provide separate mechanisms for accessing POST and GET data. Others (such as Python's cgi module) provide a single interface to form information that doesn't distinguish between the two. I already have a strong opinion on this but I'm going to leave it open for discussion here for a bit before weighing in.

Tags: cgi, http, php, python

The Python Web SIG

2003-10-18T22:42:09+00:00

Python now has a Web SIG. SIGs are Special Interest Groups, each with a target to develop and improve a certain aspect of the Python language, standard library or community. The Web SIG has two purposes: create a plan for improving Python's web client abilities (including things like the ability to parse CSS) and work on improving Python's server side capabilities.

I'm hoping that the SIG can work to bring together some of the work being done with web frameworks within the Python community. As demonstrated by this wiki page, Python users are spoilt for choice when it comes to selecting a framework for developing web applications. This is a dual edged sword - on the one hand diversity is a good thing, but on the other hand actually chosing a framework for a project has become a lengthy and arduous task. A few web frameworks (such as mod_python) are partially standardised around the interface defined by Python's CGI module, but this module has some severe shortcomings (the greatest of which is probably an inability to distinguish between GET and POST data). I would like to see the SIG define a more capable interface that covers more common web abilities, then encourage existing frameworks to provide a compatability layer for that interface. That way, developers could code to the standard interface safe in the knowledge that selecting and moving between frameworks would require very little in the way of changes to developed code.

The SIG is open to all, so I encourage anyone with an interest in Python's web programming capabilities to sign up and get involved.

Tags: cgi, http, python

Ludicrously simple templates with Python

2003-07-28T23:37:05+00:00

A long, long time ago I wrote my first ever PHP templating system. It was pretty simple; it consisted of a function that took two arguments: the name of a template file, and an associative array of replacements to make on that file.

I've finally got around to playing with Python CGIs for web development recently, and decided I needed a similar system. Thanks to Python's powerful string formatting operator, it ended up as a one-liner:

def template(file, vars):
    return open(templatedir.template, 'r').read() % vars

Presuming you've set templatedir at the top of the script, the above function lets you load a template and make some simple replacements on it with a single function call. For example:

<h3>%(title)s</h3>

%(body)s

<p class="footer">Posted: %(date)s</p>

With the above saved in the template directory as "entry.tpl", the template function above can be used thus:


print template('entry.tpl', {
    'title':'A blog entry', 
    'body':'Entry goes here...',
    'date':'3rd July 2003'})

The work is all done by the % vars bit at the end of the line. Since vars is a dictionary, Python substitutes the named items in the dictionary for their corresponding %(varname)s tokens in the string loaded from the template file. More information on string formatting operations can be found in the manual.

As templating systems go, it's far from the most useful or complete solution. It does however show that a little Python can go quite a long way.

Tags: cgi, php, python

More Python advocacy

2002-07-05T13:31:47+00:00

More Python advocacy: PYTHON: Yes, You SHOULD Be Using it! The article contains some background information on Python and why it is worth knowing, but the bulk of the article consists of a getting started guide for Python on Linux, complete instructions on using the interactive prompt, code samples and a small CGI script. It is worth noting that the CGI script example should not be deployed anywhere accessible to the public as it could allow crackers to execute code of their chosing on your web server.

Tags: cgi, python