Simon Willison's Weblog: flickr

What's the best way to learn how to scale web applications?

2012-02-23T14:42:00+00:00

My answer to What's the best way to learn how to scale web applications? on Quora

Read "Building Scalable Websites" by Cal Henderson. It's a few years old now but still very relevant - it basically covers everything he learnt the hard way scaling Flickr. It's a really fun read, too.

You might also find it useful to browse through the conference resources (slides,videos and notes) from talks on scaling that our community has collected on Lanyrd: http://lanyrd.com/topics/scaling... and if you try searching for "cal" you'll find some of the talks he's given on the subject.

Tags: flickr, programming, scaling, quora

Has anyone implemented a message queue with mysql and many workers?

2012-01-03T11:11:00+00:00

My answer to Has anyone implemented a message queue with mysql and many workers? on Quora

Flickr built their own message queue on top of MySQL: http://code.flickr.com/blog/2008...

Tags: flickr, mysql, quora

Tip: Flickr standard photo response as slideshow

2011-01-25T03:51:00+00:00

Tip: Flickr standard photo response as slideshow

Neat trick—you can construct a URL to Flickr’s slideshow widget that includes the results of any API method, including the all-powerful flickr.photos.search. It’s a shame you can’t embed the resulting slideshow in an iframe.

Tags: apis, flickr, kellan-elliott-mccrea, widgets, recovered

Quoting Jason Scott

2010-12-26T15:54:00+00:00

I am, frankly, a mixture of disappointed and sad that after Yahoo! shut down Geocities, Briefcase, Content Match, Mash, RSS Advertising, Yahoo! Live, Yahoo! 360, Yahoo! Pets, Yahoo Publisher, Yahoo! Podcasts, Yahoo! Music Store, Yahoo Photos, Yahoo! Design, Yahoo Auctions, Farechase, Yahoo Kickstart, MyWeb, WebJay, Yahoo! Directory France, Yahoo! Directory Spain, Yahoo! Directory Germany, Yahoo! Directory Italy, the enterprise business division, Inktomi, SpotM, Maven Networks, Direct Media Exchange, The All Seeing Eye, Yahoo! Tech, Paid Inclusion, Brickhouse, PayDirect, SearchMonkey, and Yahoo! Go!… there are still people out there going “Well, Yahoo certainly will never shut down Flickr, because _______________” where ______ is the sound of donkeys.

— Jason Scott

Tags: flickr, jason-scott, yahoo, recovered

Porting Flickr to YUI 3: Lessons in Performance (at YUIConf 2010)

2010-11-10T18:33:00+00:00

Porting Flickr to YUI 3: Lessons in Performance (at YUIConf 2010)

Some very interesting tips here. The new Flickr photo pages suffered from what I’ve been calling “Flash of Un-Behavioured Content”, where slow loading JavaScript results in poor behaviour from some UI controls. They started using “Action Queueing”, where a small JS stub ensures a loading indicator is shown for clicks on features that have not yet fully loaded. Also, it turns out some corporate firewalls (Sonicwall in particular) dislike URLs over 1600 characters, and filter out any URL with xxx in it.

Tags: flickr, javascript, urls, yui, recovered

What company had the first API?

2010-10-11T15:27:00+00:00

My answer to What company had the first API? on Quora

They weren't the first to have an API, but Flickr were the first consumer web site that really pushed the concept in my opinion. They originally promoted it as "you can always get your data back if you want to", but they then greatly benefited from the ecosystem that grew up around it.

Tags: apis, flickr, programming, technology, quora, computers

What is the largest production deployment of Server Side JavaScript?

2010-08-27T12:24:00+00:00

My answer to What is the largest production deployment of Server Side JavaScript? on Quora

I believe Flickr used to use Rhino for scripting the image processing (resizing, thumbnailing, sharpening) that was applied to every single uploaded photo. No idea if that's still the case though.

Tags: flickr, javascript, webapps, web-development, webservers, quora

Quoting Kellan Elliott-McCrea

2010-05-18T18:21:00+00:00

With Flickr you can get out, via the API, every single piece of information you put into the system. [...] Asking people to accept anything else is sharecropping. It’s a bad deal. Flickr helped pioneer “Web 2.0″, and personal data ownership is a key piece of that vision. Just because the wider public hasn’t caught on yet to all the nuances around data access, data privacy, data ownership, and data fidelity, doesn’t mean you shouldn’t be embarrassed to be failing to deliver a quality product.

— Kellan Elliott-McCrea

Tags: data, flickr, kellan-elliott-mccrea, sharecropping, web20, recovered

WildlifeNearYou talk at £5 app, and being Wired (not Tired)

2010-04-11T20:42:11+00:00

Two quick updates about WildlifeNearYou. First up, I gave a talk about the site at £5 app, my favourite Brighton evening event which celebrates side projects and the joy of Making Stuff. I talked about the site's genesis on a fort, crowdsourcing photo ratings, how we use Freebase and DBpedia and how integrating with Flickr's machine tags gave us a powerful location API for free. Here's the video of the talk, courtesy of Ian Oszvald:

£5 App #22 WildLifeNearYou by Simon Willison and Natalie Downe from IanProCastsCoUk on Vimeo.

Secondly, I'm excited to note that WildlifeNearYou spin-off OwlsNearYou.com is featured in UK Wired magazine's Wired / Tired / Expired column... and we're Wired!

Tags: api, crowdsourcing, fivepoundapp, flickr, freebase, natalie-downe, owlsnearyou, my-talks, wildlifenearyou, wired

5 Questions for Simon Willison

2010-02-10T14:31:57+00:00

5 Questions for Simon Willison

I got interviewed about WildlifeNearYou for the Flickr code blog, in particular the way the site uses machine tags.

Tags: flickr, interviews, wildlifenearyou

glitch zen

2010-02-10T11:36:14+00:00

glitch zen

Glitch is the upcoming online game from Tiny Speck, many of whom are ex Flickr and indeed ex Game Never Ending before that. Glitch Zen is the first fan site.

Tags: flickr, glitch, glitchzen, gne, tinyspeck

WildlifeNearYou can now tag your Flickr photos for you

2010-02-04T17:01:09+00:00

WildlifeNearYou can now tag your Flickr photos for you

I’m really excited about this feature: if you opt-in, WildlifeNearYou will now write name and latin name tags to your Flickr photos after you’ve marked the species in the photo. This is even more interesting when you combine it with our suggest-a-species feature (the photo won’t get tagged until you’ve approved the suggestion). We also set the location on photos which don’t yet have one, but the real fun is the machine tags we’ve added, which allow developers to use the Flickr API to find photos by their WildlifeNearYou metadata (trip, species and place IDs). As a neat extra touch, the identifiers we use in the machine tags are the same as the ones used by our custom wlny.eu URL shortener, so it’s trivial to turn a machine tag in to the URL for that page on the main site.

Tags: flickr, machinetags, metadata, tagging, wildlifenearyou

Introducing docent

2010-01-28T20:35:22+00:00

Introducing docent

Paul Mison’s clever little Flickr app for viewing galleries that have been added by your contacts. It runs in Python on App Engine and makes extensive use of the Task Queue API.

Tags: docent, flickr, google-app-engine, paul-mison, python, queues, taskqueue

WildlifeNearYou: It began on a fort...

2010-01-12T22:53:20+00:00

Back in October 2008, myself and 11 others set out on the first /dev/fort expedition. The idea was simple: gather a dozen geeks, rent a fort, take food and laptops and see what we could build in a week.

The fort was Fort Clonque on Alderney in the Channel Islands, managed by the Landmark Trust. We spent an incredibly entertaining week there exploring Nazi bunkers, cooking, eating and coding up a storm. It ended up taking slightly longer than a week to finish, but 14 months later the result of our combined efforts can finally be revealed: WildlifeNearYou.com!

WildlifeNearYou is a site for people who like to see animals. Have you ever wanted to know where your nearest Llama is? Search for "llamas near brighton" and you'll see that there's one 18 miles away at Ashdown Forest Llama Farm. Or you can see all the places we know about in France, or all the trips I've been on, or everywhere you can see a Red Panda.

The data comes from user contributions: you can use WildlifeNearYou to track your trips to wildlife places and list the animals that you see there. We can only tell you about animals that someone else has already spotted.

Once you've added some trips, you can import your Flickr photos and match them up with trips and species. We'll be adding a feature in the future that will push machine tags and other metadata back to Flickr for you, if you so choose.

You can read more about WildlifeNearYou on the site's about page and FAQ. Please don't hesitate to send us feedback!

What took so long?

So why did it take so long to finally launch it? A whole bunch of reasons. Week long marathon hacking sessions are an amazing way to generate a ton of interesting ideas and build a whole bunch of functionality, but it's very hard to get a single cohesive whole at the end of it. Tying up the loose ends is a pretty big job and is severely hampered by the fort residents returning to their real lives, where hacking for 5 hours straight on a cool easter egg suddenly doesn't seem quite so appealing. We also got stuck in a cycle of "just one more thing". On the fort we didn't have internet access, so internet-dependent features like Freebase integration, Google Maps, Flickr imports and OpenID had to be left until later ("they'll only take a few hours" no longer works once you're off /dev/fort time).

The biggest problem though was perfectionism. The longer a side-project drags on for, the more important it feels to make it "just perfect" before releasing it to the world. Finally, on New Year's Day, Nat and I decided we had had enough. Our resolution was to "ship the thing within a week, no matter what state it's in". We're a few days late, but it's finally live.

WildlifeNearYou is by far the most fun website I've ever worked on. To all twelve of my intrepid fort companions: congratulations - we made a thing!

Tags: devfort, django, flickr, projects, python, wildlifenearyou

russell davies: datadecs

2010-01-07T21:58:20+00:00

russell davies: datadecs

Personalised christmas decorations made from data from Twitter, Doppler, last.fm and Flickr. The Twitter snowman came from a 3D printer—the size of the head varies depending on your number of followers. Best of all though is the Flickr decoration which represents the apertures you’ve used over the past year.

Tags: 3dprinters, datadecs, dopplr, flickr, lastfm, russell-davies, twitter, visualisation

Language Detection: A Witch's Brew?

2009-12-05T17:30:33+00:00

Language Detection: A Witch's Brew?

The Flickr team make the case for using the Accept-Language header over IP detection to pick a site’s language, with a simple UI for switching languages in case you get it wrong. They’ve been using this for two and a half years without any significant problems.

Tags: flickr, http, i18n, l10n, languagdetection, language

"That's maybe a bit too dorky, even for us."

2009-09-28T22:39:27+00:00

"That's maybe a bit too dorky, even for us."

Astonishingly exciting: Flickr now have machine tag support for OpenStreetMap—tag a photo with osm:way=WAY_ID and Flickr will figure out what OSM feature you are talking about and link to it with a human readable description.

Tags: flickr, machinetags, openstreetmap, photography

Red Dust

2009-09-23T14:20:46+00:00

Red Dust

Tom Coates used Flickr’s new Galleries feature (which lets you build a curated collection of up to 18 photos from other Flickr users and add your commentary) to construct a stunning compilation of photos of the Sydney dust storms.

Tags: duststorms, flickr, photography, sydney, tom-coates

Building Rome in a Day

2009-07-29T15:41:03+00:00

Building Rome in a Day

“The ﬁrst system capable of city-scale reconstruction from unstructured photo collections”—computer vision techniques used to construct 3D models of cities using 10s of thousands of photos from Flickr. Reminiscent of Microsoft PhotoSynth.

Via Digital Urban

Tags: 3d, computer-vision, flickr, photos, photosynth, research, rome

The Twitpocalypse is Near: Will Your Twitter Client Survive?

2009-06-09T10:52:38+00:00

The Twitpocalypse is Near: Will Your Twitter Client Survive?

Twitter tweet IDs will shortly tick over past the maximum signed 32 bit integer, potentially breaking applications. I learnt this lesson when the same thing happened to Flickr photo IDs: never store numeric IDs from external systems as integers, always use strings.

Tags: apis, flickr, ids, maxint, twitter

Flickr Shapefiles Public Dataset 1.0

2009-05-22T18:12:10+00:00

Flickr Shapefiles Public Dataset 1.0

Another awesome Geo dataset from the Yahoo! stable—this time it’s Flickr releasing shapefiles (geometrical shapes) for hundreds of thousands of places around the world, under the CC0 license which makes them essentially public domain. The shapes themselves have been crowdsourced from geocoded photos uploaded to Flickr, where users can “correct” the textual location assigned to each photo. Combine this with the GeoPlanet WOE data and you get a huge, free dataset describing the human geography of the world.

Tags: creative-commons, crowdsourcing, flickr, geoplanet, geospatial, maps, shapefiles, yahoo

Quoting Kellan Elliott-McCrea

2009-04-12T16:00:41+00:00

We’re using the same trick on flic.kr to avoid having to maintain a look up database, though we’re using base 58.

— Kellan Elliott-McCrea

Tags: base58, flickr, kellan-elliott-mccrea, revcanonical, urls

Building Fast Client-side Searches

2009-03-19T15:35:03+00:00

Building Fast Client-side Searches

Flickr now lazily loads your entire contact list in to memory for auto-completion. Extensive benchmarking found that a control character delimited string was the fastest option for shipping thousands of contacts around as quickly as possible.

Tags: ajax, autocomplete, flickr, javascript, json

Panda Tuesday; The History of the Panda, New APIs, Explore and You

2009-03-04T11:49:21+00:00

Panda Tuesday; The History of the Panda, New APIs, Explore and You

Flickr’s Rainbow Vomiting Panda of Awesomeness now has a family of associated APIs.

Tags: apis, flickr, pandas

Found in space

2009-02-18T22:52:34+00:00

Found in space

The Astrometry bot on Flickr (which detects which part of the night sky is contained within your photo and adds notes to some of the more interesting stars) is the most delightful use of the Flickr API I’ve ever seen. This interview provides some background, including a link to a paper on the “scale and rotation invariant hashing algorithm” that is used to build the index.

Tags: astrometry, astronomy, flickr

Web Hooks and the Programmable World of Tomorrow

2009-02-16T21:00:23+00:00

Web Hooks and the Programmable World of Tomorrow

Tour de force presentation on Web Hooks by Jeff Lindsay. Tons of really good ideas—provided your application isn’t Flickr sized, there’s a good chance you could implement web hooks pretty cheaply and unleash a huge flurry of creativity from your users. GitHub makes a great case study here.

Tags: apis, flickr, github, jeff-lindsay, webhooks

husk.org. a flickr machine tag browser

2008-12-15T23:24:42+00:00

husk.org. a flickr machine tag browser

Flickr recently added API methods for exploring the machine tags used by the community. Paul Mison has built a neat OS X Finder style interface for exploring them, using JSONP and jQuery.

Via Flickr Developer Blog

Tags: flickr, javascript, jquery, json, jsonp, machinetags, paul-mison

Dopplr: New city pages, with public tips and Creative-Commons-licenced, Flickr-powered goodness

2008-12-01T00:43:03+00:00

Dopplr: New city pages, with public tips and Creative-Commons-licenced, Flickr-powered goodness

Explains why I’ve been unable to convince any of the Dopplr crew to come out and do fun things for the past month.

Tags: creative-commons, dopplr, flickr

On UI Quality (The Little Things): Client-side Image Resizing

2008-11-12T23:00:15+00:00

On UI Quality (The Little Things): Client-side Image Resizing

Two neat tips for cleanly scaling down images in IE 6 and 7 from Flickr’s Scott Schiller.

Tags: flickr, imagescaling, internet-explorer, scott-schiller

The Flickr Panda

2008-10-04T09:42:55+00:00

The Flickr Panda

Reminds me of the prime number shitting bear.

Tags: flickr, funny, panda, primenumbershittingbear