Simon Willison's Weblog: apache

mod_wasm: run WebAssembly with Apache

2022-10-04T00:53:47+00:00

mod_wasm: run WebAssembly with Apache

Brand new Apache module from a team at VMWare: mod_wasm builds on top of wasmtime to let you write WebAssembly programs that are exposed to the world by Apache, using a mechanism that looks similar to old CGI scripts (headers passed in environment variables, request body sent to standard input). Includes a demo Docker image that runs using Python-compiled-to-WebAssembly.

Via Hacker News

Tags: apache, webassembly

Weeknotes: Apache proxies in Docker containers, refactoring Datasette

2021-11-22T05:43:44+00:00

Updates to six major projects this week, plus finally some concrete progress towards Datasette 1.0.

Fixing Datasette's proxy bugs

Now that Datasette has had its fourth birthday I've decided to really push towards hitting the 1.0 milestone. The key property of that release will be a stable JSON API, stable plugin hooks and a stable, documented context for custom templates. There's quite a lot of mostly unexciting work needed to get there.

As I work through the issues in that milestone I'm encountering some that I filed more than two years ago!

Two of those made it into the Datasette 0.59.3 bug fix release earlier this week.

The majority of the work in that release though related to Datasette's base_url feature, designed to help people who run Datasette behind a proxy.

base_url lets you run Datasette like this:

datasette --setting base_url=/prefix/ fixtures.db

When you do this, Datasette will change its URLs to start with that prefix - so the hompage will live at /prefix/, the database index page at /prefix/fixtures/, tables at /prefix/fixtures/facetable etc.

The reason you would want this is if you are running a larger website, and you intend to proxy traffic to /prefix/ to a separate Datasette instance.

The Datasette documentation includes suggested nginx and Apache configurations for doing exactly that.

This feature has been a magnet for bugs over the years! People keep finding new parts of the Datasette interface that fail to link to the correct pages when run in this mode.

The principle cause of these bugs is that I don't use Datasette in this way myself, so I wasn't testing it nearly as thoroughly as it needed.

So the first step in finally solving these issues once and for all was to get my own instance of Datasette up and running behind an Apache proxy.

Since I like to deploy live demos to Cloud Run, I decided to try and run Apache and Datasette in the same container. This took a lot of figuring out. You can follow my progress on this in these two issue threads:

The short version: I got it working! My Docker implementation now lives in the demos/apache-proxy directory and the live demo itself is deployed to datasette-apache-proxy-demo.fly.dev/prefix/.

(I ended up deploying it to Fly after running into a bug when deployed to Cloud Run that I couldn't replicate on my own laptop.)

My final implementation uses a Debian base container with Supervisord to manage the two processes.

With a working live environment, I was finally able to track down the root cause of the bugs. My notes on #1519: base_url is omitted in JSON and CSV views document how I found and solved them, and updated the associated test to hopefully avoid them ever coming back in the future.

The big Datasette table refactor

The single most complicated part of the Datasette codebase is the code behind the table view - the page that lets you browse, facet, search, filter and paginate through the contents of a table (this page here).

It's got very thorough tests, but the actual implementation is mostly a 600 line class method.

It was already difficult to work with, but the changes I want to make for Datasette 1.0 have proven too much for it. I need to refactor.

Apart from making that view easier to change and maintain, a major goal I have is for it to support a much more flexible JSON syntax. I want the JSON version to default to just returning minimal information about the table, then allow ?_extra=x parameters to opt into additional information - like facets, suggested facets, full counts, SQL schema information and so on.

This means I want to break up that 600 line method into a bunch of separate methods, each of which can be opted-in-to by the calling code.

The HTML interface should then build on top of the JSON, requesting the extras that it knows it will need and passing the resulting data through to the template. This helps solve the challenge of having a stable template context that I can document in advance of Datasette 1.0

I've been putting this off for over a year now, because it's a lot of work. But no longer! This week I finally started to get stuck in.

I don't know if I'll stick with it, but my initial attempt at this is a little unconventional. Inspired by how pytest fixtures work I'm experimenting with a form of dependency injection, in a new (very alpha) library I've released called asyncinject.

The key idea behind asyncinject is to provide a way for class methods to indicate their dependencies as named parameters, in the same way as pytest fixtures do.

When you call a method, the code can spot which dependencies have not yet been resolved and execute them before executing the method.

Crucially, since they are all async def methods they can be executed in parallel. I'm cautiously excited about this - Datasette has a bunch of opportunities for parallel queries - fetching a single page of table rows, calculating a count(*) for the entire table, executing requested facets and calculating suggested facets are all queries that could potentially run in parallel rather than in serial.

What about the GIL, you might ask? Datasette's database queries are handled by the sqlite3 module, and that module releases the GIL once it gets into SQLite C code. So theoretically I should be able to use more than one core for this all.

The asyncinject README has more details, including code examples. This may turn out to be a terrible idea! But it's really fun to explore, and I'll be able to tell for sure if this is a useful, maintainable and performant approach once I have Datasette's table view running on top of it.

git-history and sqlite-utils

I made some big improvements to my git-history tool, which automates the process of turning a JSON (or other) file that has been version-tracked in a GitHub repository (see Git scraping) into a SQLite database that can be used to explore changes to it over time.

The biggest was a major change to the database schema. Previously, the tool used full Git SHA hashes as foreign keys in the largest table.

The problem here is that a SHA hash string is 40 characters long, and if they are being used as a foreign key that's a LOT of extra weight added to the largest table.

sqlite-utils has a table.lookup() method which is designed to make creating "lookup" tables - where a string is stored in a unique column but an integer ID can be used for things like foreign keys - as easy as possible.

That method was previously quite limited, but in sqlite-utils 3.18 and 3.19 - both released this week - I expanded it to cover the more advanced needs of my git-history tool.

The great thing about building stuff on top of your own libraries is that you can discover new features that you need along the way - and then ship them promptly without them blocking your progress!

Some other highlights

s3-credentials 0.6 adds a --dry-run option that you can use to show what the tool would do without making any actual changes to your AWS account. I found myself wanting this while continuing to work on the ability to specify a folder prefix within S3 that the bucket credentials should be limited to.
datasette-publish-vercel 0.12 applies some pull requests from Romain Clement that I had left unreviewed for far too long, and adds the ability to customize the vercel.json file used for the deployment - useful for things like setting up additional custom redirects.
datasette-graphql 2.0 updates that plugin to Graphene 3.0, a major update to that library. I had to break backwards compatiblity in very minor ways, hence the 2.0 version number.

csvs-to-sqlite 1.3 is the first relase of that tool in just over a year. William Rowell contributed a new feature that allows you to populate "fixed" database columns on your imported records, see PR #81 for details.

TIL this week

Releases this week

datasette-publish-vercel: 0.12 - (18 releases total) - 2021-11-22
Datasette plugin for publishing data using Vercel
git-history: 0.4 - (6 releases total) - 2021-11-21
Tools for analyzing Git history using SQLite
sqlite-utils: 3.19 - (90 releases total) - 2021-11-21
Python CLI utility and library for manipulating SQLite databases
datasette: 0.59.3 - (101 releases total) - 2021-11-20
An open source multi-tool for exploring and publishing data
datasette-redirect-to-https: 0.1 - 2021-11-20
Datasette plugin that redirects all non-https requests to https
s3-credentials: 0.6 - (6 releases total) - 2021-11-18
A tool for creating credentials for accessing S3 buckets
csvs-to-sqlite: 1.3 - (13 releases total) - 2021-11-18
Convert CSV files into a SQLite database
datasette-graphql: 2.0 - (32 releases total) - 2021-11-17
Datasette plugin providing an automatic GraphQL API for your SQLite databases
asyncinject: 0.2a0 - (2 releases total) - 2021-11-17
Run async workflows using pytest-fixtures-style dependency injection

Tags: apache, proxies, refactoring, supervisord, docker, datasette, weeknotes, git-scraping, sqlite-utils

How can I determine which web server a particular website is using (Apache, IIS, Nginx, etc)?

2011-01-05T13:37:00+00:00

My answer to How can I determine which web server a particular website is using (Apache, IIS, Nginx, etc)? on Quora

If you're on Linux or OS X, use curl with the -I option (to make a HEAD request and see the HTTP headers):

$ curl -I www.op3intl.com
HTTP/1.1 200 OK
Date: Thu, 06 Jan 2011 03:31:28 GMT
Server: Microsoft-IIS/6.0

...

Tags: apache, iis, nginx, servers, web-development, webservers, quora

What are the advantages of running Apache behind nginx as opposed to just Apache by itself?

2010-10-12T14:09:00+00:00

My answer to What are the advantages of running Apache behind nginx as opposed to just Apache by itself? on Quora

I do this for all of my Django stuff - I have Django running on modwsgi on a stripped down Apache (almost no configuration except for the modwsgi stuff), then I put an nginx on port 80 which serves the static files directly and proxies dynamic requests back to Apache.

There are a few advantages to this. Firstly, I use non-threaded Apache so each Apache worker process tends to be pretty big (having a full Python interpreter + Django application running in it). It's a horrible waste of resources for one of those fat processes to sit around serving up a static file instead of serving a dynamic request. Secondly, nginx deals with slow connections for me - if someone on a modem (or 3G phone) hits my site I don't tie up an Apache process for several seconds to respond to their request. Instead, nginx sucks the response in from Apache in a few ms and then spools it out to the slow client at whatever rate they can handle. Since nginx is evented it can handle hundreds or even thousands of slow clients at once using very little memory.

Finally, if I'm running an nginx proxy at the "front" of my stack it's trivial for me to start load balancing to more than one Apache servers if I need to.

Tags: apache, nginx, webservers, quora

Who are major competitors to Solr?

2010-09-02T18:01:00+00:00

My answer to Who are major competitors to Solr? on Quora

ElasticSearch is a really interesting one - it's the same underlying search library (Lucene) and the same integration model (an HTTP interface) but takes quite a different approach. It hasn't been around for a long time but it looks very impressive: http://www.elasticsearch.com/

Other than that, popular open source search engines include Sphinx and Xapian. I'm a big fan of talking to a search engine via HTTP, so I've been keeping an eye on the http://www.flax.co.uk/ project which does that for Xapian.

Tags: apache, lucene, search, search-engines, solr, sphinx-search, xapian, quora

apache.org incident report for 04/09/2010

2010-04-14T09:08:58+00:00

apache.org incident report for 04/09/2010

An issue was posted to the Apache JIRA containing an XSS attack (disguised using TinyURL), which stole the user’s session cookie. Several admin users clicked the link, so JIRA admin credentials were compromised. The attackers then changed the JIRA attachment upload path setting to point to an executable directory, and uploaded JSPs that gave them backdoor access to the file system. They modified JIRA to collect entered passwords, then sent password reset e-mails to team members and captured the new passwords that they set through the online form. One of those passwords happened to be the same as the user’s shell account with sudo access, leading to a full root compromise of the machine.

Tags: apache, incident, jira, passwords, security, tinyurl, urls, xss

jacobian's django-deployment-workshop

2010-02-19T14:28:35+00:00

jacobian's django-deployment-workshop

Notes and resources from Jacob’s 3 hour Django deployment workshop at PyCon, including example configuration files for Apache2 + mod_wsgi, nginx, PostgreSQL and pgpool.

Tags: apache, deployment, django, jacob-kaplan-moss, modwsgi, nginx, pgpool, postgresql, pycon, python, sysadmin

Traffic Server

2009-11-01T12:15:27+00:00

Traffic Server

Mark Nottingham explains the release of Traffic Server, a new Apache Incubator open source project donated by Yahoo! using code originally developed at Inktomi around a decade ago. Traffic Server is a HTTP proxy/cache, similar to Squid and Varnish (though Traffic Server acts as both a forward and reverse proxy, whereas Varnish only handles reverse).

Tags: apache, cache, http, inktomi, mark-nottingham, open-source, proxies, squid, trafficserver, varnish, yahoo

Justniffer

2009-09-25T22:12:51+00:00

Justniffer

Packet sniffing tool that can output sniffed HTTP traffic formatted the same way as an Apache access_log file.

Tags: apache, justniffer, logging, networking, packetsniffing

apache.org incident report for 8/28/2009

2009-09-03T08:56:12+00:00

apache.org incident report for 8/28/2009

Various apache.org sites were down for a while last week—here the Apache Infrastructure Team provide a detailed description of what happened (a security breach on a minor server, which provided non-priveleged SSH access to mirror servers via an SSH key used for backups) and how they are responding. Useful for neophyte sysadmins like myself.

Tags: apache, security, sysadmin

"MongoDB is fantastic for logging"

2009-08-26T19:09:26+00:00

"MongoDB is fantastic for logging"

Sounds tempting... high performance inserts, JSON structured records and capped collections if you only want to keep the past X entries. If you care about older historic data but still want to preserve space you could run periodic jobs to roll up log entries in to summarised records. It shouldn’t be too hard to write a command-line script that hooks in to Apache’s logging directive and writes records to MongoDB.

Tags: apache, json, logging, mongodb

Fabric, Django, Git, Apache, mod_wsgi, virtualenv and pip deployment

2009-07-28T11:56:09+00:00

Fabric, Django, Git, Apache, mod_wsgi, virtualenv and pip deployment

I’m slowly working my way through this stack at the moment—next stop, fabric.

Tags: apache, deployment, django, fabric, gareth-rushgrove, git, modwsgi, pip, python, virtualenv, wsgi

moddims

2009-07-21T18:18:15+00:00

moddims

Apache 2 module which exposes ImageMagick as a URL-driven service, allowing you to request an image from a whitelisted host server and resize, thumbnail or alter the quality of it.

Via Dave Artz

Tags: apache, imagemagick, images, moddims, resizing, thumbnails

Yahoo! proposal to open source "Traffic Server" via the ASF

2009-07-07T12:37:02+00:00

Yahoo! proposal to open source "Traffic Server" via the ASF

Traffic Server is a “fast, scalable and extensible HTTP/1.1 compliant caching proxy server” (presumably equivalent to things like Squid and Varnish) originally acquired from Inktomi and developed internally at Yahoo! for the past three years, which has been benchmarked handling 35,000 req/s on a single box. No source code yet but it looks like the release will arrive pretty soon.

Tags: apache, asf, caching, open-source, proxies, squid, trafficserver, varnish, yahoo

Installing Django, Solr, Varnish and Supervisord with Buildout

2009-06-07T13:54:44+00:00

Installing Django, Solr, Varnish and Supervisord with Buildout

Useful, detailed instructions... but I still think this stuff is Way Too Difficult at the moment. I’m a big fan of the idea of sites that are assembled from multiple smaller web services talking HTTP to each other, but ensuring all the moving parts stay running is massively more painful than just running Apache and MySQL.

Tags: apache, bertrand-mathieu, buildout, django, mysql, operations, python, rest, solr, supervisord, sysadmin, varnish

Phusion Passenger for nginx

2009-04-20T04:53:55+00:00

Phusion Passenger for nginx

Passenger (aka mod_rails / mod_rack) enables easy deployment of Rails and Ruby apps under Apache... and the latest version adds support for nginx as well. It works as an HTTP proxy and process manager, spawning worker processes and forwarding HTTP requests to them via a request queue. It can also handle Python WSGI applications—anyone tried it out for that yet?

Tags: apache, deployment, nginx, passenger, python, rails, ruby

Scaling Django web apps on Apache

2009-04-10T09:23:21+00:00

Scaling Django web apps on Apache

Cool to see this kind of article cropping up on IBM developerWorks, but it’s a shame they don’t mention mod_wsgi.

Tags: apache, django, ibm, modwsgi, python

Future roadmap for mod_wsgi

2009-03-19T17:27:01+00:00

Future roadmap for mod_wsgi

mod_wsgi 3.0 isn’t too far off, and will include Python 3.0 support, WSGI application preloading and internal web server redirection (similar to nginx X-Accel-Redirect). Version 4.0 plans a major architectural change that will allow multiple versions of Python to be run from the same Apache.

Tags: apache, graham-dumpleton, modwsgi, nginx, python, wsgi

Load spikes and excessive memory usage in mod_python

2009-03-16T17:26:59+00:00

Load spikes and excessive memory usage in mod_python

“The final answer? Stop using mod_python, use mod_wsgi and run it with daemon mode instead. You will save yourself a lot of headaches by doing so.”

Tags: apache, graham-dumpleton, modpython, modwsgi, python, wsgi

The Django and Ubuntu Intrepid Almanac

2009-02-14T15:42:58+00:00

The Django and Ubuntu Intrepid Almanac

Will Larson’s impressively comprehensive guide to configuring and securing an Ubuntu VPS from scratch to run Django, using PostgreSQL and Apache/mod_wsgi behind nginx.

Tags: apache, django, modwsgi, nginx, postgresql, sysadmin, ubuntu, vps, will-larson

Apache Qpid

2009-02-05T23:01:41+00:00

Apache Qpid

A new open source AMQP message queue with implementations in C++ and Java, developed by engineers from Red Hat, IONA and JP Morgan Chase. Anyone tried this yet? Looks pretty good on paper.

Tags: amqp, apache, c-plus-plus, iona, java, jpmorganchase, message-queues, open-source, qpid, red-hat

asql - Apache SQL querying

2008-10-14T11:22:45+00:00

asql - Apache SQL querying

Command line tool for loading web server log files in common log format in to a SQLite database, with a built-in interactive shell.

Tags: apache, asql, logs, sqlite

sfical.py

2008-06-27T08:09:42+00:00

sfical.py

Neat idea: write a CGI script that turns a proprietary API (in this case the SalesForce events API) in to standard ical format, then run it on your Mac’s local Apache server and subscribe to it from iCal.

Tags: apache, cgi, icalendar, mac, macos, salesforce, simon-fell

mod_rpaf for Apache

2008-06-24T17:02:55+00:00

mod_rpaf for Apache

A more secure alternative to Django’s equivalent middleware: sets the REMOTE_ADDR of incoming requests from whitelisted load balancers to the X-Forwarded-For header, without any risk that if the load balancers are missing attackers could abuse it to spoof their IP addresses.

Tags: apache, django, http, load-balancing, middleware, modrpaf, rpaf, security, xforwardedfor

Apache proxy auto-re-loader

2008-02-18T09:44:02+00:00

Apache proxy auto-re-loader

Neat trick: set your 502 (Bad Gateway) error document to include a meta refresh tag, automating the refresh needed should a server you are proxying to be temporarily unavailable.

Tags: apache, metarefresh, ned-batchelder, proxies

themaneater.com Launch

2008-02-10T08:10:42+00:00

themaneater.com Launch

The Maneater’s online edition is where Adrian cut his web development teeth, so it’s great to see them up and running on Django. Important to note that KeepAlive can completely murder Apache/Django performance.

Tags: adrian-holovaty, apache, django, keepalive, themaneater

pysolr

2008-01-09T20:50:23+00:00

pysolr

Python wrapper for Solr, the search web service wrapper for Lucene. One thing I’m not clear on: do you need to configure Solr with the fields you’ll be indexing in advance, or can Solr create new fields on the fly to match the data you send it?

Tags: apache, lucene, pysolr, python, search, solr

Damien Katz: New Gig

2008-01-02T20:35:24+00:00

Damien Katz: New Gig

IBM have employed Damien Katz to work full time on CouchDB. The work will be under the Apache license with the ASF owning the copyright.

Tags: apache, asf, couchdb, damien-katz, ibm

Hacky holidays on OS X

2007-12-29T11:49:15+00:00

Hacky holidays on OS X

Jeremy Keith documents how to get PHP 5 and Apache 2 virtual hosts running on Leopard.

Tags: apache, apache2, jeremy-keith, leopard, macos, php, php5, virtualhosts

Configuring Apache httpd

2007-10-12T12:52:49+00:00

Configuring Apache httpd

Ben Laurie shows how to build up an Apache configuration file from first principles.

Tags: apache, ben-laurie, httpd