Simon Willison's Weblog: logging

Reservoir Sampling

2025-05-08T21:00:22+00:00

Yet another outstanding interactive essay by Sam Rose (previously), this time explaining how reservoir sampling can be used to select a "fair" random sample when you don't know how many options there are and don't want to accumulate them before making a selection.

Reservoir sampling is one of my favourite algorithms, and I've been wanting to write about it for years now. It allows you to solve a problem that at first seems impossible, in a way that is both elegant and efficient.

I appreciate that Sam starts the article with "No math notation, I promise." Lots of delightful widgets to interact with here, all of which help build an intuitive understanding of the underlying algorithm.

Sam shows how this algorithm can be applied to the real-world problem of sampling log files when incoming logs threaten to overwhelm a log aggregator.

The dog illustration is commissioned art and the MIT-licensed code is available on GitHub.

Via Hacker News

Tags: algorithms, logging, rate-limiting, explorables, sam-rose

Queryable Logging with Blacklite

2023-08-21T18:13:35+00:00

Queryable Logging with Blacklite

Will Sargent describes how he built Blacklite, a Java library for diagnostic logging that writes log events (as zstd compressed JSON objects) to a SQLite database and maintains 5,000 entries in a “live” database while entries beyond that range are cycled out to an archive.db file, which is cycled to archive.timestamp.db when it reaches 500,000 items.

Lots of interesting notes here on using SQLite for high performance logging.

“SQLite databases are also better log files in general. Queries are faster than parsing through flat files, with all the power of SQL. A vacuumed SQLite database is only barely larger than flat file logs. They are as easy to store and transport as flat file logs, but work much better when merging out of order or interleaved data between two logs.”

Via asgi-log-to-sqlite/issues/1

Tags: java, logging, sqlite, zstd

Replaying logs to exercise the new API

2021-03-03T17:00:00+00:00

Originally posted to my internal blog at VaccinateCA

22 days ago n1mmy pushed a change to help.vaccinate which logged full details of inoming Netlify function API traffic to an Airtable database.

What an asset that is! The Airtable table over here currently contains over 9,000 logged API calls, including the full JSON POST body, when the call was receieved and which authenticated user made the call.

This morning I exported that data as CSV from Airtable, and wrote a Python script to replay those requests against my new imitation implementation of the API.

Here's what that script looks like running against my localhost development server:

You can track the work in this issue - the replay script helped me get to a place where every single report from the past 22 days can be safely ingested by the new API, with the exception of a tiny number of reports against locations which have since been deleted (which isn't supposed to happen - we try to soft-delete rather than full-delete things - but apparently a few deletes had slipped through).

API logging

Since the Airtable API logs have so clearly proved their value, Jesse proposed using the same trick for the Django app. I implemented that today: the full incoming request body and outgoing response are now recorded in an ApiLog model in Django. You can see those in the Django admin here: https://vaccinateca-preview.herokuapp.com/admin/api/apilog/

Here's the ORM model and the view decorator that logs requests.

Unit tests for the new API

I added tests for the submitReport API. The tests are driven by example JSON fixtures - so far I've created two of those, but I hope that having them in this format will make it really easy to add more as we find edge-cases in the API and expand it with new features.

Those API test fixtures live in vaccinate/api/test-data/submitReport. Here's the test code that executes them.

Documentation for the new API

I wrote API documentation! You can find that here.

Now that the API is documented I intend to update the documentation in lock-step with changes made to the API itself - using the pattern where every commit includes the change, the tests for the change AND the documentation for the change in a single unit.

Dual-writing to Django

The combination of the replay script and the unit tests has left me feeling pretty confident that the replacement API is ready to start accepting traffic.

The plan is to run the new system in parallel with Airtable for a few days to thoroughly test it and make sure it covers everything we need. Our Netlify functions offer a great place to do this, so this afternoon I submitted a pull request to help.vaccinate to silently dual-write incoming API requests to the Django API, catching and logging any exceptions without intefering with the rest of the application flow.

Testing this locally helped me identify some bugs in the way the Django app verified JWT tokens that originated with the help.vaccinate application.

Everything else

Here are my other commits from today.

Tags: apis, logging, vaccinate-ca, vaccinate-ca-blog

Logging to SQLite using ASGI middleware

2019-12-16T22:30:46+00:00

I had some fun playing around with ASGI middleware and logging during our flight back to England for the holidays.

asgi-log-to-sqlite

I decided to experiment with SQLite as a logging mechanism. I wouldn’t use this on a high traffic site, but most of my Datasette related projects are small enough that logging HTTP traffic directly to a SQLite database feels like it should work reasonable well.

Once your logs are in a SQLite database, you can use Datasette to analyze them. I think this could be a lot of fun.

asgi-log-to-sqlite is my first exploration of this idea. It’s a piece of ASGI middleware which wraps an ASGI application and then logs relevant information from the request and response to an attached SQLite database.

You use it like this:

from asgi_log_to_sqlite import AsgiLogToSqlite
from my_asgi_app import app

app = AsgiLogToSqlite(app, "/tmp/log.db")

Here’s a demo Datasette instance showing logs from my testing: asgi-log-demo-j7hipcg4aq-uc.a.run.app

As always with Datasette, the data is at its most interesting once you apply some facets.

Intercepting requests to and from the wrapped ASGI app

There are a couple of interesting parts of the implementation. The first is how the information is gathered from the request and response.

This is a classic pattern for ASGI middleware. The ASGI protocol has three key components; a scope dictionary describing the incoming request, and two async functions called receive and send which are used to retrieve and send data to the connected client (usually a browser).

Most middleware works by wrapping those functions with custom replacements. That’s what I’m doing here:

class AsgiLogToSqlite:
    def __init__(self, app, file):
        self.app = app
        self.db = sqlite_utils.Database(file)
    # ...
    async def __call__(self, scope, receive, send):
        response_headers = []
        body_size = 0
        http_status = None

        async def wrapped_send(message):
            nonlocal body_size, response_headers, http_status
            if message["type"] == "http.response.start":
                response_headers = message["headers"]
                http_status = message["status"]

            if message["type"] == "http.response.body":
                body_size += len(message["body"])

            await send(message)

        start = time.time()
        await self.app(scope, receive, wrapped_send)
        end = time.time()

My wrapped_send() function replaces the original send() function with one that pulls out some of the data I want to log from the messages that are being sent to the client.

I record a start time, then await the original ASGI application, then record an end time when it finishes.

Logging to SQLite using sqlite-utils

I’m using my sqlite-utils library to implement the logging. My first version looked like this:

db["requests"].insert({
    "path": scope.get("path"),
    "response_headers": str(response_headers),
    "body_size": body_size,
    "http_status": http_status,
    "scope": str(scope),
}, alter=True)

sqlite-utils automatically creates a table with the correct schema the first time you try to insert a record into it. This makes it ideal for rapid prototyping. In this case I captured stringified versions of various data structures so I could look at them in my browser with Datasette.

The alter=True argument here means that if I attempt to insert a new shape of record into an existing tables any missing columns will be added automatically as well. Again, handy for prototyping.

Based on the above, I evolved the code into recording the values I wanted to see in my logs - the full URL path, the User-Agent, the HTTP referrer, the IP and so on.

This resulted in a LOT of duplicative data. Values like the path, user-agent and HTTP referrer are the same across many different requests.

Regular plain text logs can solve this with gzip compression, but you can’t gzip a SQLite database and still expect it to work.

Since we are logging to a relational database, we can solve for duplicate values using normalization. We can extract out those lengthy strings into separate lookup tables - that way we can store mostly integer foreign key references in the requests table itself.

After a few iterations, my database code ended up looking like this:

with db.conn:  # Use a transaction
    db["requests"].insert(
        {
            "start": start,
            "method": scope["method"],
            "path": lookup(db, "paths", path),
            "query_string": lookup(db, "query_strings", query_string),
            "user_agent": lookup(db, "user_agents", user_agent),
            "referer": lookup(db, "referers", referer),
            "accept_language": lookup(db, "accept_languages", accept_language),
            "http_status": http_status,
            "content_type": lookup(db, "content_types", content_type),
            "client_ip": scope.get("client", (None, None))[0],
            "duration": end - start,
            "body_size": body_size,
        },
        alter=True,
        foreign_keys=self.lookup_columns,
    )


def lookup(db, table, value):
    return db[table].lookup({
        "name": value
    }) if value else None

The table.lookup() method in sqlite-utils is designed for exactly this use-case. If you pass it a value (or multiple values) it will ensure the underlying table has those columns with a unique index on them, then get-or-insert your data and return you the primary key.

Automatically creating tables is fine for an initial prototype, but it starts getting a little messy once you have foreign keys relationships that you need to be able to rely on. I moved to explicit table creation in an ensure_tables() method that’s called once when the middleware class is used to wrap the underlying ASGI app:

    lookup_columns = (
        "path",
        "user_agent",
        "referer",
        "accept_language",
        "content_type",
        "query_string",
    )

    def ensure_tables(self):
        for column in self.lookup_columns:
            table = "{}s".format(column)
            if not self.db[table].exists:
                self.db[table].create({
                    "id": int,
                    "name": str
                }, pk="id")
        if not self.db["requests"].exists:
            self.db["requests"].create({
                "start": float,
                "method": str,
                "path": int,
                "query_string": int,
                "user_agent": int,
                "referer": int,
                "accept_language": int,
                "http_status": int,
                "content_type": int,
                "client_ip": str,
                "duration": float,
                "body_size": int,
            }, foreign_keys=self.lookup_columns)

I’m increasingly using this pattern in my sqlite-utils projects. It’s not a full-grown migrations system but it’s a pretty low-effort way of creating tables correctly provided they don’t yet exist.

Here’s the full implementation of the middleware.

Configuring the middleware for use with Datasette

Publishing standalone ASGI middleware for this kind of thing is neat because it can be used with any ASGI application, not just with Datasette.

To make it as usable as possible with Datasette I want it made available as a plugin.

I’ve tried two different patterns for this in the past.

My first ASGI middleware was asgi-cors. I published that as two separate packages to PyPI: asgi-cors is the middleware itself, and datasette-cors is a very thin plugin wrapper around it that hooks into Datasette’s plugin configuration mechanism.

For datasette-auth-github I decided not to publish two packages. Instead I published a single plugin package and then described how to use it as standalone ASGI middleware in its documentation.

This lazier approach is confusing: it’s not at all clear that a package called datasette-auth-github can be used independently of Datasette. But I did get to avoid having to publish two packages.

datasette-configure-asgi

Since I want to do a lot more experiments with ASGI plugins in the future, I decided to try solving the ASGI configuration issue once and for all. I built a new experimental plugin, datasette-configure-asgi which can be used to configure ANY ASGI middleware that conforms to an expected protocol.

Here’s what that looks like at the configuration level, using a metadata.json settings file (which I should really rename since it’s more about configuration than metadata these days):

{
  "plugins": {
    "datasette-configure-asgi": [
      {
        "class": "asgi_log_to_sqlite.AsgiLogToSqlite",
        "args": {
          "file": "/tmp/log.db"
        }
      }
    ]
  }
}

The implementation of this plugin is very simple: here’s the entire thing:

from datasette import hookimpl
import importlib


@hookimpl
def asgi_wrapper(datasette):
    def wrap_with_classes(app):
        configs = datasette.plugin_config("datasette-configure-asgi") or []
        for config in configs:
            module_path, class_name = config["class"].rsplit(".", 1)
            mod = importlib.import_module(module_path)
            klass = getattr(mod, class_name)
            args = config.get("args") or {}
            app = klass(app, **args)
        return app

    return wrap_with_classes

It hooks into the asgi_wrapper plugin hook, reads its configuration from the datasette object (using plugin_config()), then loops through the list of configured plugins and dynamically loads each implementation using importlib. Then it wraps the ASGI app with each of them in turn.

Open questions

This is where I’ve got to with my experiments so far. Should you use this stuff in production? Almost certainly not! I wrote it on a plane just now. It definitely needs a bit more thought.

A couple of obvious open questions:

Python async functions shouldn’t make blocking calls, since doing so will block the entire event loop for everyone else.

Interacting with SQLite is a blocking call. Datasette works around this by running SQL queries in a thread pool; my logging plugin doesn’t bother with that.

Maybe it should? My hunch is that inserting into SQLite in this way is so fast it won’t actually cause any noticeable overhead. It would be nice to test that assumption thoroughly though.

Log rotation. This is an important detail for any well designed logging system, and I’ve punted on it entirely. Figuring out an elegant way to handle this with underlying SQLite databases files would be an interesting design challenge - relevant issue.

Would my SQLite logging middleware work with Django 3.0? I don’t see why not - the documentation covers how to wrap entire Django applications with ASGI middleware. I should try that out!

This week’s Niche Museums

These are technically my weeknotes, but logging experiments aside it’s been a quiet week for me.

I finally added paragraph breaks to Niche Museums (using datasette-render-markdown, implementation here) As a result my descriptions have been getting a whole lot longer. Added this week:

The Tonga Room in San Francisco
London Silver Vaults in London
Rosie the Riveter National Historical Park in Richmond, CA
LA Bureau of Street Lighting Museum in Los Angeles
Aye-Aye Island in Madagascar
Monarch Bear Grove in San Francisco
Alverstone Mead Red Squirrel Hide on the Isle of Wight

Tags: django, logging, projects, prototyping, sqlite, datasette, asgi, weeknotes, sqlite-utils

Logs vs. metrics: a false dichotomy

2019-08-03T16:46:55+00:00

Logs vs. metrics: a false dichotomy

Nick Stenning discusses the differences between logs and metrics: most notably that metrics can be derived from logs but logs cannot be reconstituted starting with time-series metrics.

Via Charity Majors

Tags: logging, observability

Targeted diagnostic logging in production

2019-07-24T05:44:39+00:00

Targeted diagnostic logging in production

Will Sargent defines diagnostic logging as “debug logging statements with an audience”, and proposes controlling this style if logging via a feature flat system to allow detailed logging to be turned on in production against a selected subset if users in order to help debug difficult problems. Lots of great background material in the topic of observability here too.

Via Charity Majors

Tags: logging, observability

Running gunicorn behind nginx on Heroku for buffering and logging

2017-10-02T01:57:20+00:00

Heroku's default setup for Django uses the gunicorn application server. Each Heroku dyno can only run a limited number of gunicorn workers, which means a limited number of requests can be served in parallel (around 4 per dyno is a good rule of thumb).

Where things get nasty is when you have devices on slow connections - like mobile phones. Heroku's router buffers headers but it does not buffer response bodies, so a slow device could hold up a gunicorn worker for several seconds. Too many slow devices at once and the site will become unavailable to other users.

This issue is explained and discussed here: Don't use Gunicorn to host your Django sites on Heroku

That article recommends using waitress as an alternative to gunicorn, but in the comments at the bottom of the article people suggest using a Heroku nginx-buildpack as an alternative.

Here is a slightly out-of-date tutorial on getting this all set up: https://koed00.github.io/Heroku_setups/

I used the following commands to set up the buildpacks:

heroku stack:set cedar-14
heroku buildpacks:clear
heroku buildpacks:add https://github.com/beanieboi/nginx-buildpack.git
heroku buildpacks:add https://github.com/heroku/heroku-buildpack-python.git

Unfortunately the nginx buildpack is not yet compatible with the new heroku-16 stack, so until the nginx buildpack has been updated it's necessary to run the application on the older cedar-14 stack. See this discussion for details: ryandotsmith/nginx-buildpack#68

Adding nginx in this way also gives us the opportunity to fix another limitation of Heroku: its default logging configuration. By default, log lines produced by Heroku (visible using heroku logs --tail or with a logging addon such as Papertrail) look like this:

    Oct 01 18:01:06 simonwillisonblog heroku/router: at=info
        method=GET path="/2017/Oct/1/ship/" host=simonwillison.net
        request_id=bb22f67e-6924-4e81-b6ad-74d1f465cda7
        fwd="2001:8003:74c5:8b00:79e4:80ed:fa85:7b37,108.162.249.198"
        dyno=web.1 connect=0ms service=338ms status=200 bytes=4523 protocol=http

Notably missing here is both the user-agent string and the referrer header sent by the browser! If you're a fan of tailing log files these omissions are pretty disappointing.

The nginx buildback I'm using loads a default configuration file at config/nginx.conf.erb. By including my own copy of this file I can override the original and define my own custom log format.

Having applied this change, the new log lines look like this:

    2017-10-02T01:44:38.762845+00:00 app[web.1]:
        measure#nginx.service=0.133 request="GET / HTTP/1.1" status_code=200
        request_id=8b6402de-d072-42c4-9854-0f71697b30e5 remote_addr="10.16.227.159"
        forwarded_for="199.188.193.220" forwarded_proto="http" via="1.1 vegur"
        body_bytes_sent=12666 referer="-" user_agent="Mozilla/5.0 (Macintosh;
        Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko)
        Chrome/61.0.3163.100 Safari/537.36"

This blog entry started life as a commit message.

Tags: django, logging, nginx, user-agents, heroku, gunicorn

System Administration: What service/product do you recommend for central logging of events and errors from multiple servers? Why?

2012-02-15T18:26:00+00:00

My answer to System Administration: What service/product do you recommend for central logging of events and errors from multiple servers? Why? on Quora

We rolled our own solution to this using MongoDB, due to its super-fast writes and ability to store, index and search JSON. We were also attracted by its capped collections, which make it easy to e.g. only log the last 100,000 items.

It hasn't given us any problems, but we also haven't spent the time to build a good UI for it do we aren't getting as much value out of it as we could. That's the disadvantage of rolling your own: you have to build the whole thing.

Tags: cloud, logging, php, sysadmin, quora

clarity

2009-11-04T22:36:04+00:00

clarity

A web interface for tailing and grepping the log files in /var/log, written in Ruby and EventMachine.

Tags: clarity, eventmachine, logging, ruby

Python Logging 101

2009-09-29T18:40:33+00:00

Python Logging 101

A really useful introduction to Python’s logging module by that module’s author, Vinay Sajip.

Tags: logging, python, vinaysajip

Django ponies: Proposals for Django 1.2

2009-09-28T23:32:04+00:00

I've decided to step up my involvement in Django development in the run-up to Django 1.2, so I'm currently going through several years worth of accumulated pony requests figuring out which ones are worth advocating for. I'm also ensuring I have the code to back them up - my innocent AutoEscaping proposal a few years ago resulted in an enormous amount of work by Malcolm and I don't think he'd appreciate a repeat performance.

I'm not a big fan of branches when it comes to exploratory development - they're fine for doing the final implementation once an approach has been agreed, but I don't think they are a very effective way of discussing proposals. I'd much rather see working code in a separate application - that way I can try it out with an existing project without needing to switch to a new Django branch. Keeping code out of a branch also means people can start using it for real development work, making the API much easier to evaluate. Most of my proposals here have accompanying applications on GitHub.

I've recently got in to the habit of including an "examples" directory with each of my experimental applications. This is a full Django project (with settings.py, urls.py and manage.py files) which serves two purposes. Firstly, it allows developers to run the application's unit tests without needing to install it in to their own pre-configured project, simply by changing in to the examples directory and running ./manage.py test. Secondly, it gives me somewhere to put demonstration code that can be viewed in a browser using the runserver command - a further way of making the code easier to evaluate. django-safeform is a good example of this pattern.

Here's my current list of ponies, in rough order of priority.

Signing and signed cookies

Signing strings to ensure they have not yet been tampered with is a crucial technique in web application security. As with all cryptography, it's also surprisingly difficult to do correctly. A vulnerability in the signing implementation used to protect the Flickr API was revealed just today.

One of the many uses of signed strings is to implement signed cookies. Signed cookies are fantastically powerful - they allow you to send cookies safe in the knowledge that your user will not be able to alter them without you knowing. This dramatically reduces the need for sessions - most web apps use sessions for security rather than for storing large amounts of data, so moving that "logged in user ID" value to a signed cookie eliminates the need for session storage entirely, saving a round-trip to persistent storage on every request.

This has particularly useful implications for scaling - you can push your shared secret out to all of your front end web servers and scale horizontally, with no need for shared session storage just to handle simple authentication and "You are logged in as X" messages.

The latest version of my django-openid library uses signed cookies to store the OpenID you log in with, removing the need to configure Django's session storage. I've extracted that code in to django-signed, which I hope to evolve in to something suitable for inclusion in django.utils.

Please note that django-signed has not yet been vetted by cryptography specialists, something I plan to fix before proposing it for final inclusion in core.

django-signed on GitHub
Details of the Signing proposal on the Django wiki
Signing discussion on the django-developers mailing list

Improved CSRF support

This is mainly Luke Plant's pony, but I'm very keen to see it happen. Django has shipped with CSRF protection for more than three years now, but the approach (using middleware to rewrite form HTML) is relatively crude and, crucially, the protection isn't turned on by default. Hint: if you aren't 100% positive you are protected against CSRF, you should probably go and turn it on.

Luke's approach is an iterative improvement - a template tag (with a dependency on RequestContext) is used to output the hidden CSRF field, with middleware used to set the cookie and perform the extra validation. I experimented at length with an alternative solution based around extending Django's form framework to treat CSRF as just another aspect of validation - you can see the result in my django-safeform project. My approach avoids middleware and template tags in favour of a view decorator to set the cookie and a class decorator to add a CSRF check to the form itself.

While my approach works, the effort involved in upgrading existing code to it is substantial, compared to a much easier upgrade path for Luke's middleware + template tag approach. The biggest advantage of safeform is that it allows CSRF failure messages to be shown inline on the form, without losing the user's submission - the middleware check means showing errors as a full page without redisplaying the form. It looks like it should be possible to bring that aspect of safeform back to the middleware approach, and I plan to put together a patch for that over the next few days.

Luke's CSRF branch on bitbucket
My django-safeform on GitHub
Details of the CSRF proposal on the Django wiki
CSRF discussion on the django-developers mailing list

Better support for outputting HTML

This is a major pet peeve of mine. Django's form framework is excellent - one of the best features of the framework. There's just one thing that bugs me about it - it outputs full form widgets (for input, select and the like) so that it can include the previous value when redisplaying a form during validation, but it does so using XHTML syntax.

I have a strong preference for an HTML 4.01 strict doctype, and all those <self-closing-tags /> have been niggling away at me for literally years. Django bills itself as a framework for "perfectionists with deadlines", so I feel justified in getting wound up out of proportion over this one.

A year ago I started experimenting with a solution, and came up with django-html. It introduces two new Django template tags - {% doctype %} and {% field %}. The doctype tag serves two purposes - it outputs a particular doctype (saving you from having to remember the syntax) and it records that doctype in Django's template context object. The field tag is then used to output form fields, but crucially it gets to take the current doctype in to account.

The field tag can also be used to add extra HTML attributes to form widgets from within the template itself, solving another small frustration about the existing form library. The README describes the new tags in detail.

The way the tags work is currently a bit of a hack - if merged in to Django core they could be more cleanly implemented by refactoring the form library slightly. This refactoring is currently being discussed on the mailing list.

django-html on GitHub
Improved HTML discussion on the django-developers mailing list

Logging

This is the only proposal for which I don't yet have any code. I want to add official support for Python's standard logging framework to Django. It's possible to use this at the moment (I've done so on several projects) but it's not at all clear what the best way of doing so is, and Django doesn't use it internally at all. I posted a full argument in favour of logging to the mailing list, but my favourite argument is this one:

Built-in support for logging reflects a growing reality of modern Web development: more and more sites have interfaces with external web service APIs, meaning there are plenty of things that could go wrong that are outside the control of the developer. Failing gracefully and logging what happened is the best way to deal with 3rd party problems - much better than throwing a 500 and leaving no record of what went wrong.

I'm not actively pursuing this one yet, but I'm very interesting in hearing people's opinions on the best way to configure and use the Python logging module in production.

A replacement for get_absolute_url()

Django has a loose convention of encouraging people to add a get_absolute_url method to their models that returns that object's URL. It's a controversial feature - for one thing, it's a bit of a layering violation since URL logic is meant to live in the urls.py file. It's incredibly convenient though, and since it's good web citizenship for everything to have one and only one URL I think there's a pretty good argument for keeping it.

The problem is, the name sucks. I first took a look at this in the last few weeks before the release of Django 1.0 - what started as a quick proposal to come up with a better name before we were stuck with it quickly descended in to a quagmire as I realised quite how broken get_absolute_url() is. The short version: in some cases it means "get a relative URL starting with /", in other cases it means "get a full URL starting with http://" and the name doesn't accurately describe either.

A full write-up of my investigation is available on the Wiki. My proposed solution was to replace it with two complementary methods - get_url() and get_url_path() - with the user implementing one hence allowing the other one to be automatically derived. My django-urls project illustrates the concept via a model mixin class. A year on I still think it's quite a neat idea, though as far as I can tell no one has ever actually used it.

ReplacingGetAbsoluteUrl on the wiki
django-urls on GitHub
Recent get_absolute_url discussion on the django-developers mailing list

Comments on this post are open, but if you have anything to say about any of the individual proposals it would be much more useful if you posted it to the relevant mailing list thread.

Tags: cookies, cryptography, csrf, django, html, logging, luke-plant, markup, ponies, projects, python, security, signedcookies, signing, xhtml

Justniffer

2009-09-25T22:12:51+00:00

Justniffer

Packet sniffing tool that can output sniffed HTTP traffic formatted the same way as an Apache access_log file.

Tags: apache, justniffer, logging, networking, packetsniffing

"MongoDB is fantastic for logging"

2009-08-26T19:09:26+00:00

"MongoDB is fantastic for logging"

Sounds tempting... high performance inserts, JSON structured records and capped collections if you only want to keep the past X entries. If you care about older historic data but still want to preserve space you could run periodic jobs to roll up log entries in to summarised records. It shouldn’t be too hard to write a command-line script that hooks in to Apache’s logging directive and writes records to MongoDB.

Tags: apache, json, logging, mongodb

Python logging from multiple processes

2009-08-13T23:55:02+00:00

Python logging from multiple processes

Use Python’s socket log handler to send all log messages to a single server—the python-loggingserver project implements such a server as a Twisted application with a handy web interface for viewing the aggregated logs.

Tags: logging, python, twisted

django-db-log

2008-05-13T08:07:49+00:00

django-db-log

Middleware that logs Django exceptions to the database, using a clever scheme based on an MD5 of the traceback text to group duplicate errors in to batches.

Tags: david-cramer, django, djangodblog, exceptions, logging, middleware

daemon.py

2008-01-08T21:58:54+00:00

daemon.py

Neat little Python module for daemonizing a process; handles logging and pid files out of the box.

Via Daemon Best Practices

Tags: daemon, daemonizing, logging, pid, python

LoggerFS

2007-10-29T10:40:41+00:00

LoggerFS

Clever use of FUSE: a virtual filesystem which looks out for lines appended to a log file (matched with a regular expression) and stores them in a database instead.

Tags: fuse, loggerfs, logging

jQuery Logging

2007-10-19T12:52:50+00:00

jQuery Logging

Brilliant four line jQuery plugin that lets you insert Firebug console.log() calls directly in to chains.

Via Jabbering Giraffe

Tags: debugging, dominic-mitchell, firebug, javascript, jquery, logging

django-logging

2007-04-24T06:50:43+00:00

django-logging

Looks neat—includes the ability to use Python’s standard logging module to log messages to a footer appended to your site’s HTML output.

Tags: django, logging, python