I received a press release today with the results of a uTest comparison of Google, Bing, Yahoo! and Google's new sandbox search engine, Caffeine. uTest used "crowdsourcing" for their comparison. The 1,100 testers were competing with each other for cash prizes.
They ranked the search engines based on result accuracy, speed, real-time relevance and usability. Caffeine wasn't included in the ranking. I was told that it was down during part of the testing, so not enough testers had a change to offer feedback. When the three search engines were ranked, Google came out on top in every single category.
uTest also found over 600 bugs. Bing had the most reported bugs, 321, while Yahoo! had the fewest, 70. Google had 130 reported bugs, while users only found 50 bugs in Caffeine. Caffeine's number is likely low because only 22% of participants were able to test it.
It's not overly surprising. Google has a relatively stable and familiar user interface, while Bing is brand new. Yahoo! is trying to trade their search engine for Bing, so they had very few bugs but poorer results. Bing also surprised 30% of users favorably, which also makes sense. Most of the best parts of Bing are only found by digging.