When Chris Anderson, editor in chief of Wired, first penned an article in 2004 called The Long Tail, he was thinking about e-commerce. The article -- and later a book by the same name -- showed that web-based sellers like Amazon and Netflix had much larger sales potential than mass-market vendors like Barnes & Noble and Blockbuster.
"People are going deep into the catalog, down the long, long list of available titles ... And the more they find, the more they like," Anderson wrote about the trend in book, video and music sales. The Long Tail, he discovered, was roughly 90 percent of the content out there; the "hits" account for just 10 percent. For example, Amazon stocks some 2.3 million books while Barnes & Noble has 130,000. (Those are 2004 figures, of course.)
He wasn't necessarily thinking about search engines, but of course the web, like Amazon, is filled with a long, long list of content. Applying Anderson's model, Google is the Amazon of web search, spidering the whole web and making the "whole catalog" available. Microsoft and Yahoo, meanwhile, focused on the most popular topics and did a poor job at delivering the rest of the web.
Google Owns the Long Tail
At least that is the insight of Yusuf Mehdi, senior vice president of the online audience group for Microsoft Bing. Microsoft now realizes, Mehdi said in a speech at the Search Engine Strategies show in New York, that search is about the long tail -- something that Google got many years ago.
"Think about the explosion in the long tail. You have to crawl, index and make sense of that," Mehdi said. "On any given month, one-third of queries that show up on Bing -- it's the first time we've ever seen that query. A huge chunk of those, we'll never see again. They're like gone with the wind. The challenge of being able to be up to speed to understand that new flow of data and to be able to index the right thing so you can respond in subsecond time is a very, very hard problem."
It's exactly that mastery of the long tail that has enabled Google to create and dominate the online advertising market.
Seeking User Intent
But Microsoft is betting that there is still value in the "head" of popular searches as Twitter, Facebook and other social-networking tools bring massive new content focused on celebrities and current events.
"The problem gets harder when it comes to new forms of unstructured data," Mehdi continued. "How do I get to things that are closed off, such as Twitter feeds or YouTube videos or things that you can't crawl easily? What happens when you can't understand the URL?"
Mehdi announced a number of enhancements to Bing, including the ability to search Twitter from Bing and integration with the popular service FourSquare, which lets users see where friends are. In the Bing integration, users can visualize those locations on a map. Bing also now features live video feeds integrated into maps so, Mehdi said, you can see how long the line at Starbucks is.
Ultimately, Microsoft is trying to solve the big problem in search: Understanding what users are really asking. "It's more of a dialogue with the consumer," Mehdi said. "We are about understanding user intent, and in mapping the intent into tasks and into actions."