Simon Willison's Weblog: lucene

Reducing search indexing latency to one second

2020-06-26T17:06:08+00:00

Reducing search indexing latency to one second

Really detailed dive into the nuts and bolts of Twitter’s latest iteration of search indexing technology, including a great explanation of skip lists.

Tags: data-structures, lucene, scaling, search, twitter

Who are major competitors to Solr?

2010-09-02T18:01:00+00:00

My answer to Who are major competitors to Solr? on Quora

ElasticSearch is a really interesting one - it's the same underlying search library (Lucene) and the same integration model (an HTTP interface) but takes quite a different approach. It hasn't been around for a long time but it looks very impressive: http://www.elasticsearch.com/

Other than that, popular open source search engines include Sphinx and Xapian. I'm a big fan of talking to a search engine via HTTP, so I've been keeping an eye on the http://www.flax.co.uk/ project which does that for Xapian.

Tags: apache, lucene, search, search-engines, solr, sphinx-search, xapian, quora

How do Solr, Lucene, Sphinx and Searchify compare?

2010-08-26T14:14:00+00:00

My answer to How do Solr, Lucene, Sphinx and Searchify compare? on Quora

Lucene is a Java library for creating and searching through a full text index. If you want to make use of it, you'll need to write your own Java code that integrates with it.

Solr is a web service that is built on top of the Lucene library. You can talk to it over HTTP from any programming language - so you can take advantage of the power of Lucene without having to write any Java code at all. Solr also adds a number of features that Lucene leaves out such as sharding and replication.

Tags: databases, lucene, search, search-engines, solr, sphinx-search, web-development, quora

Which major companies are using Solr for search?

2010-08-25T11:43:00+00:00

My answer to Which major companies are using Solr for search? on Quora

The Guardian newspaper uses Solr for its Open Platform Content API. http://www.guardian.co.uk/open-p...

Tags: lucene, open-source, search, solr, quora

[UPDATE] Spatial Search in Apache Lucene and Solr

2010-07-20T18:28:00+00:00

[UPDATE] Spatial Search in Apache Lucene and Solr

Spacial search is finally coming (back) to Solr—trunk now supports sorting and boosting by distance.

Tags: geospatial, lucene, search, solr, recovered, spatialsearch

Elastic Search

2010-02-11T18:33:14+00:00

Elastic Search

Solr has competition! Like Solr, Elastic Search provides a RESTful JSON HTTP interface to Lucene. The focus here is on distribution, auto-sharding and high availability. It’s even easier to get started with than Solr, partly due to the focus on providing a schema-less document store, but it’s currently missing out on a bunch of useful Solr features (a web interface and faceting are the two that stand out). The high availability features look particularly interesting. UPDATE: I was incorrect, basic faceted queries are already supported.

Via ElasticSearch blog

Tags: elasticsearch, http, java, json, lucene, rest, scaling, search, sharding, solr

Digg Search: Now With 99.987% Less Suck

2009-04-10T22:17:57+00:00

Digg Search: Now With 99.987% Less Suck

Really nice implementation of faceted search, still using Lucene and Solr under the hood.

Tags: digg, facets, full-text-search, lucene, search, solr

Guardian + Lucene = Similar Articles + Categorisation

2009-03-11T12:53:39+00:00

Guardian + Lucene = Similar Articles + Categorisation

Alf Eaton loaded 13,000 Guardian articles tagged Science in to Solr and Lucene and is using Solr’s MoreLikeThisHandler to find related articles and automatically apply Guardian tags to Nature News articles.

Tags: alf-eaton, full-text-search, guardian, lucene, naturenews, openplatform, search, solr, tagging

Whoosh

2009-02-12T12:49:59+00:00

Whoosh

A brand new, pure-python full text indexing engine (think Lucene). Claims to offer performance in the same league as wrappers to C or Java libraries. If this works as well as it claims it will be an excellent tool for adding search to projects that wish to avoid a dependency on an external engine.

Tags: full-text-search, lucene, open-source, python, search, whoosh

solango

2009-02-04T12:22:46+00:00

solango

Another attempt at a Django/Solr integration library, based on code written for “a top 20 newspaper site” (I’d love to know which one). This is well documented, uses a registration model clearly inspired by the Django admin which keeps search related metadata out of your regular models and includes management commands for re-indexing and generating Solr schema.xml files.

Tags: django, lucene, python, search, solr

pysolr

2008-01-09T20:50:23+00:00

pysolr

Python wrapper for Solr, the search web service wrapper for Lucene. One thing I’m not clear on: do you need to configure Solr with the fields you’ll be indexing in advance, or can Solr create new fields on the fly to match the data you send it?

Tags: apache, lucene, pysolr, python, search, solr

Apache Solr 1.1

2007-01-13T01:16:21+00:00

Apache Solr 1.1

Solr is the search Web Service built on top of Lucene. The latest release introduces JSON, Python and Ruby response formats in addition to XML.

Tags: json, lucene, python, ruby, search, solr, webservice, xml