Simon Willison's Weblog: brad-fitzpatrick

Weeknotes: Covid-19, First Python Notebook, more Dogsheep, Tailscale

2020-04-01T20:29:59+00:00

My covid-19.datasettes.com project publishes information on COVID-19 cases around the world. The project started out using data from Johns Hopkins CSSE, but last week the New York Times started publishing high quality USA county- and state-level daily numbers to their own repository. Here's the change that added the NY Times data.

It's very easy to use this data to accidentally build misleading things. I've been updating the README with links about this - my current favourite is Why It’s So Freaking Hard To Make A Good COVID-19 Model by Maggie Koerth, Laura Bronner and Jasmine Mithani at FiveThirtyEight.

First Python Notebook

Ben Welsh from the LA Times teaches a course called First Python Notebook at journalism conferences such as NICAR. He ran a free online version the course last weekend, and I offered to help out as a TA.

Most of the help I provided came before the course: Ben asked attendees to confirm that they had working installations of Python 3 and pipenv, and if they didn't volunteers such as myself would step in to help. I had Zoom and email conversations with at least ten people to help them get their environments into shape.

This XKCD neatly summarizes the problem:

One of the most common problems I had to debug was PATH issues: people had installed the software, but due to various environmental differences python3 and pipenv weren't available on the PATH. Talking people through the obscurities of creating a ~/.bashrc file and using it to define a PATH over-ride really helps emphasize how arcane this kind of knowledge is.

I enjoyed this comment:

"Welcome to intro to Tennis. In the first two weeks, we'll discuss how to rig a net and resurface a court." - Claus Wilke

Ben's course itself is hands down the best introduction to Python from a Data Journalism perspective I have ever seen. Within an hour of starting the students are using Pandas in a Jupyter notebook to find interesting discrepancies in California campaign finance data.

If you want to check it out yourself, the entire four hour workshop is now on YouTube and closely follows the material on firstpythonnotebook.org.

Coronavirus Diary

We are clearly living through a notable and very painful period of history right now. On the 19th of March (just under two weeks ago, but time is moving both really fast and incredibly slowly right now) I started a personal diary - something I've never done before. It lives in an Apple Note and I'm adding around a dozen paragraphs to it every day. I think it's helping. I'm sure it will be interesting to look back on in a few years time.

Dogsheep

Much of my development work this past week has gone into my Dogsheep suite of tools for personal analytics.

I upgraded the entire family of tools for compatibility with sqlite-utils 2.x.
pocket-to-sqlite got a major upgrade: it now fetches items using Pocket's API pagination (previously it just tried to pull in 5,000 items in one go) and has the ability to only fetch new items. As a result I'm now running it from cron in my personal Dogsheep instance, so "Save to Pocket" is now my preferred Dogsheep-compatible way of bookmarking content.
twitter-to-sqlite got a couple of important new features in release 0.20. I fixed a nasty bug in the --since flag where retweets from other accounts could cause new tweets from an account to be ignored. I also added a new count_history table which automatically tracks changes to a Twitter user's friends, follower and listed counts over time (#40).

I'm also now using Dogsheep for some journalism! I'm working with the Big Local News team at Stanford to help track and archive tweets by a number of different US politicians and health departments relating to the ongoing pandemic. This collaboration resulted in the above improvements to twitter-to-sqlite.

Tailscale

My personal Dogsheep is currently protected by client certificates, so only my personal laptop and iPhone (with the right certificates installed) can connect to the web server it is running on.

I spent a bit of time this week playing with Tailscale, and I'm really impressed by it.

Tailscale is a commercial company built on top of WireGuard, the new approach to VPN tunnels which just got merged into the Linux 5.6 kernel. Tailscale first caught my attention in January when they hired Brad Fitzpatrick.

WireGuard lets you form a private network by having individual hosts exchange public/private keys with each other. Tailscale provides software which manages those keys for you, making it trivial to set up a private network between different nodes.

How trivial? It took me less than ten minutes to get a three-node private network running between my iPhone, laptop and a Linux server. I installed the iPhone app, the Ubuntu package, the OS X app, signed them all into my Google account and I was done.

Each of those devices now has an additional IP address in the 100.x range which they can use to talk to each other. Tailscale guarantees that the IP address will stay constant for each of them.

Since the network is public/private key encrypted between the nodes, Tailscale can't see any of my traffic - they're purely acting as a key management mechanism. And it's free: Tailscale charge for networks with multiple users, but a personal network like this is free of charge.

I'm not running my own personal Dogsheep on it yet, but I'm tempted to switch over. I'd love other people to start running their own personal Dogsheep instances but I'm paranoid about encouraging this when securing them is so important. Tailscale looks like it might be a great solution for making secure personal infrastructure more easily and widely available.

Tags: brad-fitzpatrick, data-journalism, projects, python, teaching, datasette, dogsheep, weeknotes, tailscale, covid19, ben-welsh

Quoting Brad Fitzpatrick

2020-01-30T17:17:29+00:00

I used to tolerate and expect complexity. Working on Go the past 10 years has changed my perspective, though. I now value simplicity above almost all else and tolerate complexity only when it's well isolated, well documented, well tested, and necessary to make things simpler overall at other layers for most people.

— Brad Fitzpatrick

Tags: brad-fitzpatrick, complexity, go

Quoting Brad Fitzpatrick

2009-12-25T23:31:15+00:00

I think it's really important to know the whole stack even if you don't operate within the whole stack.

— Brad Fitzpatrick

Tags: brad-fitzpatrick, programming

pubsubhubbub

2009-04-20T18:49:45+00:00

pubsubhubbub

From Brad Fitzpatrick, a simple but clever way of using web hooks (HTTP callbacks) to inform subscribers that an Atom feed has updated in almost real-time—solving the constant polling problem and making it easier for small sites to offer publish-subscribe APIs. Any Atom feed can delegate subscriber updates to a “hub” server. An example hub server implementation is provided running on App Engine.

Tags: atom, brad-fitzpatrick, google-app-engine, pubsub, pubsubhubbub, python, realtime, webhooks

Using Memcache with Google App Engine

2008-05-28T23:11:06+00:00

Using Memcache with Google App Engine

Brad Fitzpatrick’s 20% time project.

Tags: brad-fitzpatrick, google, google-app-engine, memcache

A proposal: email to URL mapping

2008-02-08T11:39:41+00:00

A proposal: email to URL mapping

Brad’s just too damn smart. A simple solution to mapping an e-mail address to an OpenID that takes advantage of existing technology (YADIS) and doesn’t adversely affect e-mail privacy.

Tags: brad-fitzpatrick, email, openid, urls, yadis

Naming twins in Python and Perl

2008-01-07T11:03:22+00:00

Naming twins in Python and Perl

Simple anagram problem solved in Perl and Python, with a bunch more solutions in the comments. The C# solution provides an interesting example of LINQ in action.

Tags: anagrams, brad-fitzpatrick, csharp, linq, perl, programming, python

Blogger: OpenID commenting

2007-11-30T19:10:26+00:00

Blogger: OpenID commenting

I may be wrong, but I think this is the first Google property to support OpenID in any way.

Via Brad Fitzpatrick

Tags: blogger, brad-fitzpatrick, google, openid

Google Announces the OpenSocial API

2007-10-31T16:34:58+00:00

Google Announces the OpenSocial API

I doubt the similarity between this and Brad Fitzpatrick’s social graph paper are a coincidence—what IS impressive is that he only joined Google a couple of months ago.

Tags: apis, brad-fitzpatrick, google, opensocial, social-graph

HTTPOnly cookie support in Firefox

2007-09-06T06:27:33+00:00

HTTPOnly cookie support in Firefox

Five years after the bug was filed, HTTPOnly cookie support has gone in to the Mozilla 1.8 branch. This is a defence in depth feature that has been in IE for years—it lets you set cookies that aren’t available to JavaScript, and hence can’t be hijacked in the event of an XSS flaw.

Tags: brad-fitzpatrick, firefox, httponly, internet-explorer, javascript, mozilla, security

Thoughts on the Social Graph

2007-08-17T23:47:58+00:00

Thoughts on the Social Graph

I think social network portability will happen within the next year.

Tags: brad-fitzpatrick, david-recordon, openid, social-graph, social-network-portability