Bing It On Challenge Rigged? Microsoft’s Claims Are “A Little Fishy,” Says Yale Law Prof.

Microsoft seemed reminiscent about the 70s when it launched it's own search engine version of the famous Pepsi Challenge last year, which it called the "Bing It On Challenge." Much like the soft drink ad, Microsoft's marketing videos had a spokesperson ask random people on the street to do a side-by-side, blind search-test between Bing and Google.

Surprise, surprise: Bing won the challenge two-to-one.

But those ads also asked you, the consumer, to visit Bingiton.com for an open-to-the-public version of the Bing It On challenge, which its said 5 million people had already tried. Big mistake.

That's where Ian Ayres, professor at Yale Law decided to perform his own test. Ayres was suspicious of the Bing claim that it won two-to-one immediately after seeing the first few ads. He thought that with the two competitors' search results being mostly identical, most average users wouldn't be able to tell a difference, and therefore Microsoft couldn't have gotten the 2:1 ratio.

Ayres, first of all, found out that the "nearly 2:1" claim was based on far fewer than 5 million users. It was actually an independent research company, Answers Research, which was hired by Microsoft to conduct a blind test using "a representative online sample of nearly 1,000 people, ages 18 and older from across the U.S."

Microsoft didn't keep track of the results of the 5 million-visited Bing It On website, because Microsoft said it was "non-scientific" and intended as a fun way for consumers to experiment with both search engines.

That's when Ayres, according to his post on the Freakonomics blog, decided to set up his own Bing It On Challenge, with the help of Microsoft's own promotional website. He and four of his law students set up a similar experiment to the one that Answers Research performed. Here are the results:

We found that, to the contrary of Microsoft's claim, 53 percent of subjects preferred Google and 41 percent Bing (6 percent of results were "ties").  This is not even close to the advertised claim that people prefer Bing "nearly two-to-one."  It is misleading to have advertisements that say people prefer Bing 2:1 and also say join the millions of people who've taken the Bing-It-On challenge, if, as in our study, the millions of people haven't preferred Bing at a nearly a 2:1 rate.  Microsoft might have realized this and has more recently altered its advertising to back off their original claim to just say that people "prefer" Bing.

Ayres' study was set up statistically randomly and published in a research paper on Yale's website.

The research team randomly assigned participants to search for a search term taken from one of three options: Bing's suggested search terms (the BingItOn website suggests five terms right under the search box for people who can't think of anything to search for besides, say, "Bing lies."), all-around popular search terms, and self-chosen search terms only. They found that when the Bing-suggested search terms were used, it resulted in a tie. When subjects used either the generally popular searches or their own, over 50 percent of participants preferred Google to Bing, compared to the under 40 percent who chose Bing.

Ayres accuses Microsoft using gentle language, saying, "These secondary tests indicate that Microsoft selected suggested search words that it knew were more likely to produce Bing-preferring results." He calls several of Microsoft's claims "a little fishy."

A little more firmly, Ayres, the Law Professor, says that he thinks Google has a "colorable deceptive advertising claim against Microsoft" that could be worth a lot of money.

What will Microsoft's response be - Bing it on?