Searching the World Live Web

Live Web search got a lot bigger yesterday, when Google launched its new blogsearch engine. There's no direct link on the Google index page yet. You'll find it in the roster of services behind the "more" link. There are 29 of those, and Blog Search is the newest.

But the news is still big. It legitimizes the Live Web -- and blogging in particular -- in a big way.

Far as I know, the blog search category was born when David Sifry put a hack he called Technorati on Penguin Computing Linux box that lived in his basement while he and I worked on "Building With Blogs" -- a feature for the February 2003 issue of Linux Journal: http://www.linuxjournal.com/article/6497. Dave needed to research blogs, and created a tool for it. As of today, Technorati's traffic is #751 on Alexa, pushing 80 million pageviews per day. (Disclosure: I'm on Technorati's Advisory Board.)

Other Live Web search pioneers include Bloglines, Blogpulse, Feedster, IceRocket and PubSub. The results they yield are radically different than what you get with Wide Web search. Also from each other. Mostly the results are newer. They're also more likely to come from individuals and live news services than from companies with static sites.

Let's say you want to search for Katrina and Interdictor. The latter is the weblog of Michael Barnett, who helped keep DirectNIC's data center up and running through Hurricane Katrina and the crisis that followed. Far more than a simple blog, Interdictor has also served as a message board, a tech support line and a zero-bullshit news service.

You'll get results on Google's and Yahoo's, main pages; but they won't be especially current (today is September 14, and the top result on Google is from September 3). Nor can you plumb them through the dimension of time, staring at now.

Do the same search on Blogpulse, and you get results listed backward in time with the latest at the top. You can also watch trend results for the same search. You can refine results by incrementally adding search terms. And you can track conversations from one URL's "seed".

Search for the same thing on Feedster and you'll get results listed either by relevance or date.

Do the same search on Icerocket and you'll get results grouped by date, starting with today. You can refine your search to today, past week, past month or by date range. You can also follow trends here too, and look back on your sesrch history.

Do the same search on Technorati and you'll get results from two hours to two days old, with the most recent at the top. (The company tries to index everything within minutes.) You can also find 5,680 posts tagged "katrina" and 5 tagged "interdictor".

Do the same search with Google's Blog Search, and you'll get 2,355 results. While the overall look is similar to its Wide Web results, you get a choice of sort by relevance or date. You can also go back 100 pages through the first 1000 results, and subscribe to feeds of the search as well. (And, as you'd expect, it's much faster than all the others.)

You can't search through PubSub, but you can subscribe to keywords and combinations of keywords. These searches are syndicated, so you can receive them in your own aggregator. In fact, most of the live web engines provide feeds for searches of keywords, URLs or combinations of both.

Of the Wide Web engines, only A9 also competes in Live Web search, using IceRocket.

All of them run on Linux, by the way. (No news there, but worth reporting, of course.)

So, what's the difference between the Wide Web and the Live Web? Glad you asked.

The simple difference is that the Live Web is syndicated. That means every time something is posted, or updated, a notification goes out, informing the world about it. The most familiar syndication method is RSS, which commonly stands for Really Simple Syndication. There are a number of different syndication formats (Google's Blogger uses Atom), but as a class we tend to call them all RSS. Those familiar little orange XML buttons are the common symbol for Live Web search feeds.

Wide Web search engines send out spiders to crawl through every site on the Web. On Google, that's about 8.2 billion pages.

Live Web search engines only crawl syndicated pages, and only when they're notified by a fresh feed from those pages. So, while Technorati searches through 17.1 million sources, it only indexes pages that send out fresh syndicated feeds.

Here's another way of looking at it: Wide Web indexing is proactive, and archives everything, while Live Web indexing is reactive, and archives only what's fresh.

Of course, a Live Web engine can archive much more than that, over a long period of time; but what matters most is usually what's freshest -- or both relevant AND fresh.

Another difference is in the rate of change in technologies, standards and practices. This results in highly varied search experiences that are bound to change over time. In just the last few months, "tagging" posts (or photos on Flickr, or bookmarks on Del.icio.us) with categorical keywords has proven a handy way to discover and peruse ad hoc groupings. Technorati has been providing tag search (along with tagging methods) for several months now, and others are bound to follow. Meanwhile, Wide Web search has remained a very consistent experience ever since Google taught users to trust PageRank: http://www.google.com/intl/en/technology/.

In the few hours that have passed since Google's Blog Search has appeared, there have been many posts in the Blogosphere predicting the death of the incumbent Live Web engines. A few minutes ago I spoke to Jason Goldman, who runs both Blogger and Blog Search at Google. Rather than predicting the death of competitors in the Live Web space, he said he expected it to become energized, and to grow. He also appreciated David Sifry's blog post, welcoming Google to the space: http://www.sifry.com/alerts/archives/000340.html.

Somewhere in there, a friend sent me a message reminding me that Apple also publicly welcomed IBM to personal computing in 1982, when the IBM PC was introduced. The implication was that IBM flattened Apple. In a way, that happened. But it's worth also noting that Apple is still very much around, healthy, and a leader in its industry (and some others too). Jason also reminded me that there were many predictions of death for competitors when Google bought Blogger. Instead, the blog creation tool business only got bigger.

Every industry needs its mainstays and its pioneers. The Live Web has both now. And it will be better for everybody if they all do what they do best.