uTest, a startup that allows companies to outsource their QA testing to ‘the cloud’ has just concluded its latest quarterly bug battle, during which it put some of the world’s largest social networks to the test. Hundreds of participants (many of which have been involved in product testing for over a year) did their best to uncover flaws across Facebook, MySpace, and LinkedIn, with $3,000 up for grabs for the testers who identified the most crucial bugs.
Below are the final results for the number of bugs found. It’s worth noting that this data is prone to bias and may well overstate the number of ’showstopping bugs’ (testers probably had a strong incentive to rate their bugs as ’showstopper’ so as to have a better chance at the prize). But it also meshes fairly well with anecdotal experience.
- Facebook: 416 uTesters found 243 bugs (.58 bugs/tester), including 14.40% that were deemed
showstoppers by the tester - LinkedIn: 399 uTesters found 250 bugs (.63 bugs/tester), including 9.78% that were deemed
showstoppers by the tester. - MySpace: 304 uTesters found 225 bugs (.74 bugs/tester), including 10.80% that were deemed
showstoppers by the tester
Aside from logging each bug that they found, testers were also asked to rate each social network in a handful of categories and to detail some of the most apparent issues. Overall, the contest deemed Facebook to have the most complete feature set, but the social network was criticized for not having a high enough default security level (the options to keep things private are there, but are not always set by default).
LinkedIn took the title of “Best in Overall Quality”, but testers had issues syncing data with Outlook, which they considered “problematic for the ’professional social network’”
MySpace didn’t take the top prize in any of the bug testing categories, but was considered to have “a loyal following”, as it drew consistently high marks from a subset of users. Common complaints included “Overlapping images, backgrounds, titles, etc.” and slow page load times.
Testers’ choice for best overall quality:Â
1.  LinkedIn (45%)Â
2.  Facebook (37%)  Â
3.  MySpace (17%)Â
Â
Testers’ choice for best usability: Â
1.  Facebook (39%)Â
2.  LinkedIn (38%)Â
3.  MySpace (17%)Â
Â
Testers’ choice for best feature set:Â
1.  Facebook (46%)Â
2.  LinkedIn (36%)Â
3.  MySpace (18%)
These results mean nothing! Here is why:
1. This was essentially nothing more than a contest to determine who can find the most and worst bugs. Biases are guaranteed and overstatement of the severity of a bug is most definitely assured when money is on the table.
2. This is truly unfair testing. How can someone compare Facebook (and its huge applications platform) with LinkedIn and their small (compare to Facebook) feature set. Upon starting to read the article, I already knew what the results were.
Seriously, do we need a company like that to tell us that software is buggy. To be honest, I am very surprised that only around 250 bugs were found per network. I am actually surprise that LinkedIn has that many considering their feature set. I would have thought that Facebook and MySpace have many more when considering all the different browsers, clients, applications, media, and content that exists on these sites.
The other thing that is unclear is what were the limitations on the testing? For example, were testers of facebook allowed to report bugs of 3rd party applications?
The whole thing just does not sound right.
Just my 2 cents…
Get a life….
I agree with the idea that comparing the three web sites doesn’t seem to have common ground. But I see the advantage of making testing open for the public.
If this model is picked up by big players, then the job of QA/software testers becomes in danger.
I disagree. The results mean quite a bit.
Even large sites with gazillions of dollars and masses of people STILL have bugs.
uTest was able to identify key problems quickly and on the cheap.
Krgrds,
E. David Zotter
umm…. friendster!
Um…yeah, completely worthless. Basically it’s a matter of “OH LET’S GET THE MOST POPULAR THINGS ON THE INTERWEB TOGETHER AND COMPARE THEM AND SEE WHAT PEOPLE MAKE IT OF IT AND THAT’LL SURELY GET US FAME!”
I’m not sure how anyone can claim this to be worthless without knowing which bugs were uncovered? It looks like there was decent QA for the contest (accuracy, severity, etc.) and the feedback was sent to each site. This could be an awesome tool for the social networks themselves – an opportunity to improve.
I like the concept of uTest and if they make the web more usable, I’m all for it. Yes, they happened to be reporting bugs on 3 of the largest social sites, 2 of which compete heavily with one pulling away strongly – but so what? How does that make this any less valuable? Great PR move for uTest.
You notice that MySpace is 3rd in all 3 of the polls. I never liked MySpace from the begining.
what a joke, they get people in cheap to QA web products. stupid folks man wonder who have time to QA for $1 – $5 HOPE
ha ha
In uTest the tester chooses the severity of the bug posted. The higher the severity the more you get paid. I don’t think they even have the ability to lower the severity if they don’t agree.
Actually, while the tester initially specifies the severity of each bug, it’s the customer who sets the final severity, and makes the approve/reject decision for any bug. This is what establishes the price for each approved bug.
Good to know, I’ve seen approved bugs where I thought the severity was silly so I thought that maybe it was not changeable!
Good to know! I’ve seen approved bugs where IMHO they did not match what they should be and they were approved, therefore I figured it was not possible.
while it’s frustrating to hear that ANY users may have had a difficult experience using these products, this study is fundamentally flawed. why? because LinkedIn operates in an entirely different business space and relies on a totally decoupled set of activities than MySpace or Facebook. Since the site is about showing the best PROFESSIONAL NETWORKING face to the outside world and not about bringing users original content, the success criteria is a bit skewed in LinkedIn’s favor. It’s a long discussion, but essentially, the inclusion of LinkedIn totally undermines any credibility of the test results. It’s as if they included Craigslist and Amazon in the study!
I like http://UserFix.com concept for reporting bugs and requesting features to any social network or site.
I really enjoyed the results…
It just goes to show that even the big sites with millions of dollars of VC can still have a huge amount of unrealized bugs in their production software.
Sure sure, the various social websites all have different features and functionality. However, these are all in the top 5, so I’d expect a higher level of software design.
From a marketing perspective, it is a smart demonstration of uTests’s service for easily and quickly identifying bugs on the cheap.
Clearly they caught things that small armies of people within Facebook/LinkedIn/Myspace missed.
Sometimes the best way involves brute force!
Krgrds,
E. David Zotter
Pss.. Havent yet started using a myspace account