Thursday, 2 December 2010

Where's the open source search engine?

We have a wildly popular open source crowd sourced encyclopedia in Wikipedia. Everyone uses search engines, so is there a crowd sourced one?

You've probably never heard of the Open Directory Project at, but it's the largest (in practice the only) open source web site directory, claiming upwards of 4 million entries.

The reason you never heard of it is the same reason you stopped using the original part of Yahoo, the manually maintained site directory, ten years ago. People discovered that automatically compiled indexes were a lot more comprehensive and useful. Google has billions of pages, not just millions.

If anyone remembers Altavista, the first popular automated index, it started to knock out Yahoo by being more comprehensive. Then Google came along with the clever page rank trick, ordering listings based on the number of other pages that linked to them, which dealt with the main advantage of manual directories, putting the interesting stuff at the front. So while ODP was a worthy competitor to the Yahoo directory, now they're just footnotes to search engine history.