An unlikely trio of companies have joined forces to improve and help standardise the way information is crawled or searched on the Web.
Google, Yahoo and Microsoft today announced support for Sitemaps 0.90 (www.sitemaps.org), a free, unified format for publishers and web masters to submit their content.
Google said Sitemap provides an easy way for webmasters to make their sites more search engine friendly. It does this by conveniently allowing webmasters to list all of their URLs along with optional metadata, such as the last time the page changed, to improve how search engines crawl and index their websites.
A Sitemap is an XML file that can be made available on a website and acts as a marker for search engines to crawl certain pages. The goal is get the information searched on the web information indexed more comprehensively and efficiently.
"A lot of publishers and web masters have been asking for a more standardised way to interact with search engines, Tim Mayer, senior director of global web search at Yahoo, told internetnews.com.
"Now, the content publisher can get their best pages in the index, and it also lets them raise their hands and let the search engine know when there are new pages.
Mayer said standardisation will also help with bandwidth efficiency because the search companies won't look an extra time at content that hasn't changed.
Users stand to benefit because the system is meant to get newer results out quicker. Danny Sullivan, editor-in-chief of Search Engine Watch, lauded the agreement.
"This is a great development for the whole community and addresses a real need of web masters in a very convenient fashion," said Sullivan. "I believe it will lead to greater collaboration in the industry for common standards, including those based around robots.txt, a file that gives web crawlers direction when they visit a website."
In a briefing with internetnews.com Wednesday afternoon, only Google and Yahoo were involved, but later in the day, a release was issued that included Microsoft, a last minute signatory of the deal.
"I am sure this will be the first of many industry initiatives you will see us working and collaborating on," said Ken Moss, general manager of Microsoft's Windows Live Search, in a statement.
The initiative was originally driven by Yahoo and Google and builds on an earlier version of Sitemaps released by Google in June, 2005. Mayer said the companies have "an open invitation" to other search providers, web publishers and content management providers to join the standardisation effort.
The three companies said they will continue to collaborate on the Sitemaps protocol and publish enhancements on a jointly maintained website which provides all of the details about the Sitemaps protocol.
No responsibility can be taken for the content of external Internet sites.