Wikia's & open source reduce cost setting up search engine
Google Inc. and other search engines face far more than just a new rival in Wikia Inc.; they face the prospect of hundreds, even thousands of new competitors.
The entire search-engine project that Wikia is working on will enter the open-source domain, drastically reducing the cost for just about anyone to make a search engine, said Gil Penchina, the company's CEO. Instead of paying millions of dollars to index the Web and create the software to build a search page, new search companies will find these items free online thanks to the open-source and free software communities.
"In search, it still costs about $5 million to $10 million to build a site," said Penchina during an interview in Taipei. "We want to make it possible for anyone to build a search site for $500. We don't view Google as the competition; we view cost as the competition."
The project, which was started by Wikipedia co-founder Jimmy Wales, consists of four components: the indexing of the Web, developing a search engine application, developing an algorithm, and having people filter sites and rank results.
Indexing the Web
One of the most expensive components of a search engine is the effort needed to index the Web. Companies have to buy servers and software to crawl the Web looking at every page to create a comprehensive list of what's on the Web.
"Your average search start-up will spend over $1 million buying servers and collecting data. That's bad for a couple of reasons. One is that everyone spends millions of dollars doing what is essentially the same work, which is like writing an encyclopedia all over again. Well, what if all of that data was available over the GNU Free Documentation License, which is the free-content license? So our goal is to make a crawl of the Web publicly available," Penchina said.
The cost of indexing the Web is one of the main hurdles to starting a search engine, and for-profit companies have raised the bar year after year by indexing the Web more and more often. It used to be cataloged once a week, or once a day. Now it's once an hour or even more often. The high cost of running these crawls has become a competitive weapon.
Wikia believes its crawl of the Web will cost nearly nothing because it's asking Internet users to help by downloading Web crawling software from Grub, which will use their computers during idle time to crawl the Web, and send results back to Wikia for the index.
So far, a thousand people have downloaded the application, and Penchina is hoping for 100,000 or more. The goal is to post the entire index online, as well as regular updates, so anyone can use them.
Asking the Internet community for help this way is reminiscent of the Search for Extraterrestrial Intelligence (SETI) project, which asks users to run a free application that downloads and analyzes radio telescope data and sends the results back to a computer operated by the SETI@home group.
The search-engine application
Another essential search-engine element is an application that provides a place to type in searches, a button to say "go," and the ability to view results. Right now, Wikia is looking at using Lucene, an open-source application. Wikia plans to either invest more into the Lucene project to make sure the software works well, or build its own software to serve the same purpose.
"That's sort of an ongoing fight right now, either building into Lucene or building our own application," Penchina said. "We sort of have two projects going right now. We're sort of doing both."
Another key piece of search technology is an algorithm to determine search rankings, ensuring that users find what they're looking for. Such algorithms are the secret recipes of search-engine companies, never to be revealed for fear of abuse by hackers or other people. But at Wikia, the idea is to create an algorithm and post it on the Internet for free so anyone can see how results are determined by the search engine. People who don't like the search results can offer Wikia tips or create their own algorithms, download all of the free search-engine material made available by Wikia, and start their own search companies, Penchina said.
Finding people to help
The collaboration part of the whole project is having users sort through and filter Web pages, as well as tweak search results by their own work. It adds a human touch to the search-engine process, and Wikia's founders said they hope it will lead to better Internet searches.
Money may come from the Wikia search engine when it's finally done, but that's not the point of the project.
"We believe you can do good and make money. In search, we haven't figured out how yet," said Penchina. But if you build something that millions of people use and find value in, somebody's going to pay you for something. I mean, maybe we'll get consulting fees, maybe we'll sell advertising, maybe we'll sell some premium software or something. I don't know. I don't care. And I'm not even really thinking about it because I've got to get a million people to use it first."