Simon Willison's Weblog: san-francisco

Quoting Aaron Straup Cope

2021-04-01T22:40:41+00:00

If you measure things by foot traffic we [the SFO Museum] are one of the busiest museums in the world. If that is the case we are also one of the busiest museums in the world that no one knows about. Nothing in modern life really prepares you for the idea that a museum should be part of an airport. San Francisco, as I've mentioned, is funny that way.

— Aaron Straup Cope

Tags: aaron-straup-cope, museums, san-francisco

Monarch Bear Grove on Niche Museums

2019-12-16T21:19:05+00:00

Monarch Bear Grove on Niche Museums

Monarch Bear Grove is my favourite hidden corner of Golden Gate Park in San Francisco. It has stone circles formed from pieces of a Spanish monastery that was exported to the USA by press baron William Randolph Hearst. And there are druids. You should read the whole thing. (I added paragraph breaks for this using datasette-render-markdown—Niche Museums is basically a full-blown blog now.)

Via @simonw

Tags: museums, projects, san-francisco

Quoting In the Basement of the King

2019-03-28T21:11:49+00:00

For the Fairmont, the Tonga Room is an inherited embarrassment, as though it were a local lord whose ancestors captured a repellent goblin and chained him up in the cellar, but the goblin is inexplicably adored by the townsfolk and the children, who sneak the goblin food and treats, and cry when the goblin’s master moves to strike it.

— In the Basement of the King

Tags: san-francisco, tikibar

Generating a commit log for San Francisco's official list of trees

2019-03-13T14:49:48+00:00

San Francisco has a neat open data portal (as do an increasingly large number of cities these days). For a few years my favourite file on there has been Street Tree List, a list of all 190,000 trees in the city maintained by the Department of Public Works.

I’ve been using that file for Datasette demos for a while now, but last week I noticed something intriguing: the file had been recently updated. On closer inspection it turns out it’s updated on a regular basis! I had assumed it was a static snapshot of trees at a certain point in time, but I was wrong: Street_Tree_List.csv is a living document.

Back in September 2017 I built a scraping project relating to hurricane Irma. The idea was to take data sources like FEMA’s list of open shelters and track them over time, by scraping them into a git repository and committing after every fetch.

I’ve been meaning to spend more time with this idea, and building a commit log for San Francisco’s trees looked like an ideal opportunity to do so.

sf-tree-history

Here’s the result: sf-tree-history, a git repository dedicated to recording the history of changes made to the official list of San Francisco’s trees. The repo contains three things: the latest copy of Street_Tree_List.csv, a README, and a Circle CI configuration that grabs a new copy of the file every night and, if it has changed, commits it to git and pushes the result to GitHub.

The most interesting part of the repo is the commit history itself. I’ve only been running the script for just over a week, but I already have some useful illustrative commits:

7ab432cdcb8d7914cfea4a5b59803f38cade532b from March 6th records three new trees added to the file: two Monterey Pines and a Blackwood Acacia.
d6b258959af9546909b2eee836f0156ed88cd45d from March 12th shows four changes made to existing records. Of particular interest: TreeID 235981 (a Cherry Plum) had its address updated from 412 Webster St to 410 Webster St and its latitude and longitude tweaked a little bit as well.
ca66d9a5fdd632549301d249c487004a5b68abf2 lists 2151 rows changed, 1280 rows added! I found an old copy of Street_Tree_List.csv on my laptop from April 2018, so for fun I loaded it into the repository and used git commit amend to back-date the commit to almost a year ago. I generated a commit message between that file and the version from 9 days ago which came in at around 10,000 lines of text. Git handled that just fine, but GitHub’s web view sadly truncates it.

csv-diff

One of the things I learned from my hurricane Irma project was the importance of human-readable commit messages that summarize the detected changes. I initially wrote some code to generate those by hand, but then realized that this could be extracted into a reusable tool.

The result is csv-diff, a tiny Python CLI tool which can generate a human (or machine) readable version of the differences between two CSV files.

Using it looks like this:

$ csv-diff one.csv two.csv --key=id
1 row added, 1 row removed, 1 row changed

1 row added

  {"id": "3", "name": "Bailey", "age": "1"}

1 row removed

  {"id": "2", "name": "Pancakes", "age": "2"}

1 row changed

  Row 1
    age: "4" => "5"

The csv-diff README has further details on the tool.

Circle CI

My favourite thing about the sf-tree-history project is that it costs me nothing to run - either in hosting costs or (hopefully) in terms of ongoing maintenance.

The git repository is hosted for free on GitHub. Because it’s a public project, Circle CI will run tasks against it for free.

My .circleci/config.yml does the rest. It uses Circle’s cron syntax to schedule a task that runs every night. The task then runs this script (embedded in the YAML configuration):

cp Street_Tree_List.csv Street_Tree_List-old.csv
curl -o Street_Tree_List.csv "https://data.sfgov.org/api/views/tkzw-k3nq/rows.csv?accessType=DOWNLOAD"
git add Street_Tree_List.csv
git config --global user.email "treebot@example.com"
git config --global user.name "Treebot"
sudo pip install csv-diff
csv-diff Street_Tree_List-old.csv Street_Tree_List.csv --key=TreeID > message.txt
git commit -F message.txt && \
  git push -q https://${GITHUB_PERSONAL_TOKEN}@github.com/simonw/sf-tree-history.git master \
  || true

This script does all of the work.

First it backs up the existing Street_Tree_list.csv as Street_Tree_List-old.csv, in order to be able to run a comparison later.
It downloads the latest copy of Street_Tree_List.csv from the San Francisco data portal
It adds the file to the git index and sets itself an identity for use in the commit
It installs my csv-diff utility from PyPI
It uses csv-diff to create a diff of the two files, and writes that diff to a new file called message.txt
Finally, it attempts to create a new commit using message.txt as the commit message, then pushes the result to GitHub

The last line is the most complex. Circle CI will mark a build as failed if any of the commands in the run block return a non-0 exit code. git commit returns a non-0 exit code if you attempt to run it but none of the files have changed.

git commit ... && git push ... || true ensures that if git commit succeeds the git push command will be run, BUT if it fails the || true will still return a 0 exit code for the overall line - so Circle CI will not mark the build as failed.

There’s one last trick here: I’m using git push -q https://${GITHUB_PERSONAL_TOKEN}@github.com/simonw/sf-tree-history.git master to push my changes to GitHub. This takes advantage of Circle CI environment variables, which are the recommended way to configure secrets such that they cannot be viewed by anyone browsing your Circle CI builds. I created a personal GitHub auth token for this project, which I’m using to allow Circle CI to push commits to GitHub on my behalf.

Next steps

I’m really excited about this pattern of using GitHub in combination with Circle CI to track changes to any file that is being posted on the internet. I’m opening up the code (and my csv-diff utility) in the hope that other people will use them to set up their own tracking projects. Who knows, maybe there’s a file out there that’s even more exciting than San Francisco’s official list of trees!

Tags: csv, data-journalism, git, projects, san-francisco, git-scraping

Late night dining near Great American Music Hall

2017-10-27T20:12:00+00:00

My answer to Late night dining near Great American Music Hall on Ask MetaFilter

Tommy’s Joynt is a couple of blocks away and is a San Francisco institution - great comfort food, inexpensive, crammed with personality and open late.

Tags: ask-metafilter, san-francisco, greatamericanmusichall, dinner

Quoting Justin Krause

2017-10-27T19:22:59+00:00

In the Bay Area, we have a collection of fiefdoms. Villages are parading as cities, addressing problems myopically. For example, Brisbane (a city of 5,000 people between San Francisco and SFO) is currently blocking a large housing development for local reasons. It’s NIMBY-ism on a broad scale – a regional tragedy of the commons.

— Justin Krause

Tags: san-francisco

San Francisco Recommendations: comics, fashion, food, and more

2017-01-27T21:29:00+00:00

My answer to San Francisco Recommendations: comics, fashion, food, and more on Ask MetaFilter

I'm not a huge comics person, but that said I strongly recommend a trip to Isotope Comics in Hayes Valley on a weekday. Go on a weekday when the owner is running the store. He is incredibly charismatic and positively vibrates with enthusiasm about comics. He also has an incredible collection of toilet seats lining the upper wall of the store, each one illustrated by a famous comic book artist. Make sure to ask him about them.

Like I said, I'm not a huge comics fan, but Isotope really feels like something special to me. Friends of mine who are big time into comics definitely appreciate it.

Tags: ask-metafilter, san-francisco, travel, resolved

What are good event venues to have a 100 person weekend hackathon in San Francisco?

2014-02-24T18:54:00+00:00

My answer to What are good event venues to have a 100 person weekend hackathon in San Francisco? on Quora

As a general rule, tech companies in San Francisco are very supportive of hackathons and keen to host events. Eventbrite have hosted this kind of event in their office in the past, and I'm certain there are a bunch of other companies with decent spaces that would be worth approaching. Does the hackathon have a specific topic? I suggest approaching companies related to that topic as a starting point.

Tags: events, san-francisco, quora, hackathons

For a 2-day developer conference in SF, which two days of the week are best?

2013-10-30T15:02:00+00:00

My answer to For a 2-day developer conference in SF, which two days of the week are best? on Quora

It depends on the topic and the audience.

If you're expecting people from out of town, you should go with Mon-Tue or Thu-Fri to allow people the chance to tack on a weekend to enjoy the city.

If it's a conference for VCs, remember that they traditionally have their partner meetings on Monday.

If the audience are likely to be parents, don't schedule it over a weekend.

Look out for other conflicting events too, as that's a bigger concern than if it should be on particular days of the week.

Tags: conferences, san-francisco, quora

Is it worth it for an aspiring web developer in NYC to attend the San Francisco Startup Jobs Fair in November? What I mean is what are my chances that a company would be interested in me, allowing that my code skills are up to snuff, if I don't live ...

2013-10-13T10:51:00+00:00

My answer to Is it worth it for an aspiring web developer in NYC to attend the San Francisco Startup Jobs Fair in November? What I mean is what are my chances that a company would be interested in me, allowing that my code skills are up to snuff, if I don't live out in SF yet? on Quora

Not being in SF already is no problem at all - demand for talented developers is crazy high, so provided you are a great developer the fact you would need to relocate won't be a barrier.

If you do travel to SF for the job fair, make sure you get as much out of the trip as possible - try to line up some in-person interviews with companies that look interesting to you in the days before and after the event itself.

Tags: rails, san-francisco, web-development, quora, nyc

Events (leisure): What are the best places in San Francisco for organizing a small meetup?

2013-08-08T17:20:00+00:00

My answer to Events (leisure): What are the best places in San Francisco for organizing a small meetup? on Quora

If it's a technical event, there are dozens of tech companies with excellent facilities who might be convinced to host the event for free.

Tags: events, san-francisco, quora, meetup, sanfranciscobayarea

What are the best tech community event sites in San Francisco?

2012-01-03T12:09:00+00:00

My answer to What are the best tech community event sites in San Francisco? on Quora

We have a large and growing community-maintained list of mostly-tech events in San Francisco here:

http://lanyrd.com/places/san-fra... (60 upcoming events at time of writing)

That page has RSS and iCal feeds that you can subscribe to. You can also sign in to the site with Twitter to get recommendations of events that might interest you based on what your twitter contacts re attending in the future.

Tags: events, san-francisco, quora

What Social Media events or conferences are taking place in November 2011 in San Francisco or New York?

2011-08-26T16:31:00+00:00

My answer to What Social Media events or conferences are taking place in November 2011 in San Francisco or New York? on Quora

There don't seem to be many "social media" events happening in November - September and October are peak season.

GigaOm have a new event called RoadMap (in SF) - they haven't published much information about it, but it might fit your criteria: http://event.gigaom.com/gigaomro...

#140conf BrandsConf is on in New York in November - again, not much information posted about it yet: http://2011.brandsconf.com/

ad:tech New York is in November: http://www.ad-tech.com/ny/

We have lists of upcoming conferences in both NY and SF - you might find one of these useful: http://lanyrd.com/places/new-york/ and http://lanyrd.com/places/san-fra...

Tags: conferences, events, san-francisco, social-media, quora, nyc

What tech conferences are upcoming in San Francisco?

2011-06-27T16:28:00+00:00

My answer to What tech conferences are upcoming in San Francisco? on Quora

We have a pretty good list here: http://lanyrd.com/places/san-fra...

Tags: conferences, san-francisco, technology, quora

Are any major tech conferences held in San Francisco in August or early September?

2011-06-19T18:16:00+00:00

My answer to Are any major tech conferences held in San Francisco in August or early September? on Quora

Dreamforce (the Salesforce/Cloud computing conference) is pretty huge - 30th August to 2nd September: http://www.salesforce.com/dreamf...

We have a full list of (mostly tech) events here: http://lanyrd.com/places/san-fra...

Tags: conferences, san-francisco, technology, quora, sanfranciscobayarea

Which Bay Area startups are going to SXSW 2011?

2011-02-19T12:45:00+00:00

My answer to Which Bay Area startups are going to SXSW 2011? on Quora

http://lanyrd.com/ will be there - both myself and Nat.

Tags: san-francisco, startups, sxsw, quora, sanfranciscobayarea

Where can I find a calendar of upcoming tech events in the San Francisco / Bay area?

2010-12-29T11:29:00+00:00

My answer to Where can I find a calendar of upcoming tech events in the San Francisco / Bay area? on Quora

Take a look at http://lanyrd.com/places/san-fra...

Tags: calendars, networking, san-francisco, startups, quora, sanfranciscobayarea

Where is a good list of social media events in the San Francisco Bay Area for 2011?

2010-12-29T09:25:00+00:00

My answer to Where is a good list of social media events in the San Francisco Bay Area for 2011? on Quora

Take a look at http://lanyrd.com/places/san-fra...

Tags: events, san-francisco, social-media, quora, sanfranciscobayarea

What UX/UI conferences in the SF Bay area are worth attending?

2010-11-20T17:18:00+00:00

My answer to What UX/UI conferences in the SF Bay area are worth attending? on Quora

BayCHI is excellent from what I've heard: http://www.baychi.org/

We have a list of upcoming UX events in California here, with an iCal and RSS feed that you can subscribe to: http://lanyrd.com/topics/user-ex...

Tags: conferences, san-francisco, ui, quora, ux, sanfranciscobayarea

What are the best websites for finding out about events in San Francisco in advance, i.e., before they take place?

2010-10-14T18:45:00+00:00

My answer to What are the best websites for finding out about events in San Francisco in advance, i.e., before they take place? on Quora

PlanCast is your best bet for general events, but if you're interested in conferences we have a growing list on http://lanyrd.com/places/san-fra...

Tags: events, san-francisco, quora, sanfranciscobayarea, meetups

Mr. Penumbra's Twenty-Four-Hour Book Store

2009-06-12T18:07:37+00:00

Mr. Penumbra's Twenty-Four-Hour Book Store

Enormously entertaining short story about data visualisation and creepy San Francisco bookshops by Robin Sloan.

Tags: bookshops, robin-sloan, san-francisco, shortstory, visualisation

Freebase Hack Day

2008-10-24T00:06:06+00:00

Freebase Hack Day

I’m finding Freebase increasingly interesting at the moment, and their public hack day on the 8th November in San Francisco looks like it could be a lot of fun. They’ll be previewing Acre, a new server-side JavaScript application platform targeted at building Freebase powered applications. Hit “view source” at the bottom of the hack day site to see what an Acre app looks like.

Tags: acre, events, freebase, hackday, javascript, san-francisco

Quoting Robert Lofthouse

2008-07-13T16:50:17+00:00

DjangoCon 2008. Venue: Gooleplex, San Francisco Bay Area. Dates: 6th and 7th Sept. Official post will be on djangoproject.com soon.

— Robert Lofthouse

Tags: django, djangocon, events, google, googleplex, python, robert-lofthouse, san-francisco

Quoting Me

2008-07-07T17:14:03+00:00

It looks like the first ever Django conference will take place in early September in the San Francisco bay area.

— Me, on Twitter

Tags: conferences, django, djangocon, events, san-francisco, twitter

Introducing EveryBlock

2008-01-23T21:56:55+00:00

Introducing EveryBlock

EveryBlock launched! Adrian Holovaty, Wilson Miner, Paul Smith and Daniel X. O’Neil’s startup which answers the question, “What’s happening in my neighborhood?” Cities covered by the launch are Chicago, New York and San Francisco.

Tags: adrian-holovaty, chicago, daniel-x-oneil, django, everyblock, new-york, paul-smith, san-francisco, wilson-miner

Flickr: The Commons

2008-01-16T21:38:57+00:00

Flickr: The Commons

Exciting pilot collaboration with the Library of Congress to release images with “no known copyright restrictions”. The header photo (of a bench) is one of my favourite spots in the world, in Mission Dolores Park, San Francisco.

Tags: flickr, libraryofcongress, missiondolores, photos, publicdomain, san-francisco, the-commons

Minako Organic Japanese Restaurant

2007-04-19T08:38:13+00:00

Minako Organic Japanese Restaurant

On 18th and Mission in San Francisco. We ate there this evening—the meal took three hours and was worth every minute.

Tags: san-francisco, sushi, yelp