The graph above is hilarious. It represents the way in which people rate videos on YouTube. As you can see, there are some 1-stars and a huge amount of 5-stars, and then basically no 2, 3, or 4 stars. As such, YouTube has a blog post today admitting that maybe its star system isn’t the best way to vote on videos.
Of course, anyone who has used YouTube for an extended period of time will already know this. And really, the same seems to be true of basically all 1 to 5 star crowd rating systems. It’s easy to know if a video (or anything) is good or bad, but how on Earth do you determine if it’s 2 star, 3 star, or 4 star-worthy? Everyone likely has their own opinions about what would constitute those ratings, and naturally, they’re all completely subjective.
Apple’s App Store faces a similar problem. And it also shares another issue that YouTube faces, most people are probably only going to vote if they absolutely love or hate an app (or a video). Hence, the 1 and 5 star usage.
YouTube asks in its post, “Would a thumbs up/thumbs down be more effective, or does favoriting do the trick of declaring your love for a video?” Yes, the two vote option (thumbs up/thumbs down) or the one vote option (favoriting) are both better methods because they’re more defined.
Zagat’s subscriber-based numerical rating system works pretty well in major markets like NYC. A NYC 25 is usually much better than a NYC 20.
Zagat voters, however, don’t vote on a 30 point scale – I think it’s only a 0-3 point scale in the submission forms. (They then average all votes and multiply by 10.)
I’ve always been a bit against the rating system, but it’s also set the standard for a lot of other sites out there which now have the exact same kind of thing. The only site that comes to mind in which the five-star system actually works somewhat well is Amazon. Other than that it’s pretty worthless.
I would go for the two-vote option. I’m not going to favorite videos that don’t pertain to me just because I think they’re funny. Once I absorb someone’s content, unless it’s viral or it is mentioning me in it, I usually don’t need to return. So why add that video to my favorites if it really isn’t?
Yeah, I feel like people put time aside for thinking about ratings in proportion to how much time they spend with what they’re rating. For example, you buy something on Amazon for $20–that’s a huge chunk of change, and you’re going to want to talk about the product. If you don’t like it, you might also feel obligated to prevent other people from making the same mistake of purchasing that product. On the other hand, people go through tens of Youtube videos at a time–unless you’re trying to flag the video as a MUST see or a MUST NOT see (after all, all that’s at stake is ~30 seconds on average for the next guy), it’s not going to be worth it for you to do a middle-of-the-road critique and actually decide between 2, 3, and 4 stars. I think that’s what explains the ambiguity of the middle ratings.
finally MG! there’s a fundamental difference between ratings and rankings. why has it taken so long to grasp such a simple concept?
ratings are extremely subjective (without defined definitions) and very limiting. On the other hand, rankings actually differentiate between different options telling you which one is better than the other. for one, i know http://www.flickchart.com asks this very question.
i hate to promote on TC boards, but it’s hard not to when that’s EXACTLY what we’ve created with Rank ‘em (www.gorankem.com). it’s a crowdsourced ranking platform that allows music fans to rank their favorite songs from all their favorite artists.
When you combine the collective intelligence opinion for each artist, it creates a pretty powerful resource for all of us to use whether we are the most casual music observer or the most diehard fanatic.
we’re in private alpha at the moment, but feel free to request an invitation.
Urbanspoon is easier to use than Zagat. They use Thumbs up/down and it’s free.
Agreed. What’s more, if you think the 5 star system is unsuited to YouTube videos (which don’t change over time), they’re even more unsuited to restaurant/bar reviews which are a lot more volatile.
If you’re interested, I discussed this very topic a few months ago: http://www.theb...s-it-all-wrong/
They should show the distribution on each video (i.e. 1 stars: 5, 2 stars: 10, 3 stars: 4, 4 stars: 10, 5 stars: 30) except do it with a LIKE, NEUTRAL, DISLIKE (3 star system) since most videos will get either a good rating or a bad rating or no rating at all.
Won’t lie to you guys, the rating systems that are being used are pretty silly. Starring in my opinion is dumb. I think the options should be based on language people actually use to signify their level of interest in something. Simple as that. Thumbs up and down is too narrow. They had it ok with the range of the starring. But starring doesn’t matter to anything else except for hotels and restaurant. And those have actually taken up the context of starring.
Everything else trying to attach itself to starring mechanisms is an unwanted distraction.
I completely agree. Rating systems should use the natural language people use when they tell a friend about a video they just watched. That was “awesome” “stupid” “interesting” “hilarious” whatever. YouTube could just capture these in a separate text box when you’re posting a comment and then show these below the titles when you’re scrolling down a list of videos. “Cat falls asleep on fan” might have “Top Impressions: Hilarious (32) Cute (19) Funny (8).” “New Musician lights up the stage in LA” might have “Top Impressions: Awesome (41) Sick (19) Talented (7)”.
This seems like a much better way to let people know what to expect when they watch a video than just saying it has 3.5 stars.
Like Buzzfeed does with their site (OMG, WTF, Cute, etc.)
For sites like youtube, where people just watch content for fun a two star/ one start system suits more. Either I like the content or I dont.
But for topics that I am very critical/passionate about I would want to rate it on a more granular scale. If I am a food enthusiast or a fashion guru and if I am asked to rate an item/design I would prefer a wider scale than just a yes or a no.
But for youtube I just dont care. Either the video is awesome or just not worth my time,
It would seem to be obvious that the percentage of people that do not bother to vote is itself a useful metric that can be figured into the total score. You would just have to ascertain what a no-vote implies in aggregate. Thumbs up, thumbs down, and no-vote should be a sufficient basis for a simple rating.
I’m a fan of the two-vote option, or even a three-vote option… dislike, neutral, and like. I don’t think I’ve ever given anything a 2 or a 4 rating.
5 star rating doesn’t work for a site like youtube because the stars are good for the person who made the video rather than the person who is viewing the video.
On a site like Netflix you wouldn’t 5 star a bad movie because you don’t want it recommending crappy movies to you. If the star system put more weight on the person rating it rather than the person who posted it, it may be more valuable.
Netflix does very well with a 1-5 system, and I see movies across the spectrum, with the top of the bell curve around 2.5-3, just as you’d expect. It’s not an inherent problem with a 1-5 rating system.
Netflix would do much better with just a Lat.fm type “Love/Skip” feature. Or Thumbs Up/Down.
@Charbax: Please do not change the Netflix staring system.
I think Netflix rating works because it uses clearer terms…
Hated it.
Didn’t like it.
Liked it.
Really liked it.
Loved it.
While YouTube is…
Poor.
Nothing Special.
Worth Watching.
Pretty Cool.
Awesome!
Netflix guides you towards the middle with “Liked it”… Youtube guides you towards the end with “Awesome!”
And who is to say that awesome is better than pretty cool?
“That was pretty cool dude!”
“That was awesome dude!”
Both phrases could said using the same tone and inflection and mean the same thing. There’s not enough difference.
But take the phrases…
“I really liked that movie!”
“I loved that movie!”
That’s a difference I can easily comprehend, and therefore, my ratings on Netflix are easier to make, which is why the rating systems works so well on Netflix, and not so much on YouTube.
I propose that YouTube change their “descriptors” to what Netflix uses and see if people rate things differently.
Good point. Never thought about it that way but it’s true.
You and @Balpreet up above have points that work well together. The rating system is really dependent on the content, and clarity of the rating system. I’d also add in the use of the rating in relation to the user.
Site’s like YouTube are generally:
* short form content therefore “hit/miss” – harder to get more granular ratings
Where as a site like Netflix has:
* long form content with nuances – easier to get granular ratings
* clear rating descriptors
* clear understanding that my more specific ratings == better recommendations for me
In my experience, the Netflix star rating system also seems to work reasonably well.
The only 5 star system that comes to mind that works is the Netflix. Having it average out user ratings works well for me and is (usually) pretty acurate.
5=Loved, this is one the best i’ve ever seen
4=Yeah, this is a good movie, I’d watch it again
3=So-so, wouldn’t watch it again
2=I only finished the movie cuz I thought it might get better
1=Couldn’t even finish
In the App Store it just doesn’t work because half the amount of 1/5 voters are idiots. When they uninstall something they just downloaded in some promotion or just because it was free or cheap and later notice that they don’t need it, they give it 1/5 when the rating dialog pops-up. So the ratings system totally fails there. Maybe a more in depth review would be the answer with some basic questions which calculate a rating. In this way people who don’t care enough to give a decent rating just don’t rate at all because they are too lazy.
Of course this doesn’t apply to Youtube.
since the yahoo post reminded me of how yahoo music did it, with their 5 star system that worked, i think youtube just erred going the 5 route instead of a 2 or 3 star system. i actually think their up or down votes are pretty irrelevant but since they are moving on to feature video’s they could have a system where they feature the most voted video of the day on their front page. i think it might be too easy to game that incentive though.
I give this article 2 stars
5 Stars for me.
Just rate it by how long the user watched the video – if it sucks, they’ll stop watching
This is a very good point. I think that five start systems are supposed to work on the basic idea that the one and five votes will average out to a value somewhere in the middle. In this way the two, three, and four star ratings are averages based on the ratio of one star votes to five star votes.
However, in general it seems that most people just vote the same. They go along with the crowd so you don’t get that averaging out.
One place where you got a consistent difference in the ratings on YouTube was the last election. The ratings for the Obama videos were uniformly between 4 and 5. The ratings for the McCain videos were very close to 2.5. They all liked the Obama videos. Only half of the viewers liked the McCain videos.
The 5 star system should be scrapped in favour of something like the Zune rating system. You either love it, hate it, or you don’t rate it.
They should use the Zune system, you either love it, hate it, or don’t rate it.
Yup, that’s the only way!
If you think something is so-so, you simply are just not going to rate it.
This is a very good point. I think that five start systems are supposed to work on the basic idea that the one and five votes will average out to a value somewhere in the middle. In this way the two, three, and four star ratings are averages based on the ratio of one star votes to five star votes.
However, in general it seems that most people just vote the same. They go along with the crowd so you don’t get that averaging out.
How about simply checking how long a video is watched before quitting?
Google obviously does that already.
The “Love It” “Hate It” rating is much better.
Yup, like Last.fm has Love/Skip. That would be very good for watching playlists on Youtube. A shortcut on the keyboard like the Space bar or Enter button should make you rate “Love” and the ctrl button could be used to skip the video which also basically tells Google “Don’t show me this type of stuff again”. After a while Google will be able to generate a personal TV for you with one awesome video after the other and it will replace TV.
One place where you got a consistent difference in the ratings on YouTube was the last election. The ratings for the Obama videos were uniformly between 4 and 5. The ratings for the McCain videos were very close to 2.5. They all liked the Obama videos. Only half of the viewers liked the McCain videos.
This is a very good point. I think that five start systems are supposed to work on the basic idea that the one and five votes will average out to a value somewhere in the middle. In this way the two, three, and four star ratings are averages based on the ratio of one star votes to five star votes.
However, in general it seems that most people just vote the same. They go along with the crowd so you don’t get that averaging out.
Even — | – | ± | + | ++ is much better.
Don’t know, why anyone should rate a bad video with a star (improve it)! This is ridiculous!
two points:
the high number of 5-star ratings on the site is actually a symptom of the viral nature of the site. although some people surf youtube looking for videos, the majority of the views come from people following links to the latest must-watch video. naturally, these videos are highly rated.
secondly, there is little incentive in rating a video in the first place. if users received a benefit for doing so, they would likely put more thought behind the value given. in that case, there would more likely be more 2-, 3-, and 4-star ratings.
two points:
the high number of 5-star ratings on the site is actually a symptom of the viral nature of the site. although some people surf youtube looking for videos, the majority of the views come from people following links to the latest must-watch video. naturally, these videos are highly rated.
secondly, there is little incentive in rating a video in the first place. if users received a benefit for doing so, they would likely put more thought behind the value given. in that case, there would more likely be more 2-, 3-, and 4-star ratings.
Thumbs are definitely the way to go… here is a great example of the Best TechCrunch Videos.
Yesterday, I finished writing a fantastic, modular AJAX / Scriptaculous / Prototype class for my site. Now I’m likely scrapping it all and going to thumbs up / down system. Thanks TechCrunch…..
A rating system requires someone to be actively logged in. I don’t know what the login numbers on YouTube are but I’d guess that they’d have to be under 50% if not much lower. I’d also guess that people mostly login to rate when they are on an extreme, really loving or hating a video. It seems unreasonable to assume that someone would login to rate something a 3.
Evaluating the interest in a video should be measured by whether or not people are actually watching it. This is obviously something that happens naturally.
In my business, people like to know how their videos are performing. This can all be done passively by tracking where people are skipping, re-watching, dropping off or sharing their video with others. You can then aggregate this data to look at how effective a video is overall. You can get a sense for the type of video tracking that is possible here. http://wistia.c...roduct/tracking
YouTube has an opportunity to highlight videos by releasing metrics such as “Most Engaging” or “most shared”. This kind of usage-based rating would be completely passive, more effective, and could become the basis of better recommendations.
Actually, I think the App Store has the most varied ratings of any star system I’ve seen. Granted, I find that most ratings range only from two to four stars, but they do range.
The reason the App Store doesn’t have nearly all 5 star reviews like YouTube is that the *only* time Apple asks you to rate an app is when you delete it. Needless to say, it’s hard to imagine a more extreme selection bias for selecting folks who are going to give low ratings. (The only thing worse I could imagine is popping up a dialog that says “App XYZ has just crashed. Would you like to rate it?”).
For what it’s worth, I suspect that Apple has put in this selection bias on purpose, as it’s in their interest to make their star ratings more spread out. They have way more apps than they can handle, and they need to do everything they can to identify bad ones.
Ah! The infamous J-Curve.
Perhaps the folks at YouTube should read the book I’m working on for O’Reilly: Building Web Reputation Systems. The draft chapters are all up on the web.
Here’s a relevant excerpt: http://building...as_effects.html
Um, hello? Google knew this a long time ago. Look at Gmail—they considered doing a five star system but decided to do a one or nothing star system.
However, I am not sure about moving away from the star system … how would you translate existing indivudual votes?
Well the 1 points are very important. I’m never surprised by the one point videos.
here’s an idea.
amount of time watched (downloaded) vs length of video.
if it sucks you just stop watching, if it’s amazing you watch the whole thing and click on the next link.
Yelp, Netflix, and Amazon both have 5-star rating systems that work just fine. Other people have highlighted some of the reasons YouTube is currently different: no benefit to rating, people more likely to be virally directed to things they like, easier for people to work the ratings system, and less investment in viewing.
I think YouTube should use the 5-star input and the 5-star output, but de-link the two somewhat. YouTube has a lot of data on user behavior, so they know things like how often people watch a video all the way through, how often they watch it again, how often they come back, and how a video gets shared. They could roll a lot of that data up into a video star ranking, and also include people’s manual votes. And if they wanted more useful votes, they could do some of the things that others do to encourage that.
I think the rating system across the world – in print media, in reviews, whatever – needs an overhaul.
Startups, step up to this challenge!
BTW – We have a local search site on the way that was from the start aimed at solving this crap. Look for it to drop in the next month or so.
Also, thanks to Google Chrome Frame, we decided today to block all non-Chrome Frame enabled IE versions from use, why you ask? Cause it their fucking browser that sucks not our sick ass app, that’s why!
As someone who works for a large media company currently using the thumbs up/down model, I have to say I find it to be just as bad as a model.
Here are my observed complaints:
Dislike or thumbs down is a horrible display for content that you are attempting to promote. It auto imparts a negative connotation to content where as stars are not nearly as negative. Sure if you site is filled with negatives you have some content issues, but that is for another group to tackle. My role is to create a system of surfacing content.
Interaction and result state. We have reviewed so many versions of this model and have yet to find a solution that we are happy with. For ease of use, we have had to separate overall ratings and the rating element itself. Not nearly as clean as the 5 star method where there is one element that shows current ratings and allows you to supply your own.
While I clearly understand the youtube issue, I think the single star/tmb up might be the best scenario with a result state that shows the total star count. It doesn’t look negative but it is simple to use with it’s binary state. You either liked it or not, but at least you don’t count nor display the nots.
Again, this is for promoted, original content. UGC might have differing parameters…
Screw rating systems… seriously. What the hell does it matter, aside from personal preferences, if you click stars, thumbs, arrows, or even smiley faces for that matter…
Why don’t you just use real data that reflects popularity like media consumption patterns. After all, you’re powered by google. For example, drop the stupid 5 stars, that’s a complete joke and arbitrary as all hell. Instead, go with playback patterns, drop off times, full plays vs skipping to the next track… Basically, inject some steroids into your play count, start aggregating the analytics on the back end, and provide useful data that actually stands for something. Or, perhaps i’m talking outta my ass. Maybe it would be better to up the stagnant star system. Actually, hell yeah, how about 1 – 100 stars? I’m down…
Fantastic job! IEEE SocialCom about a month ago had this poster session published; which has a few more datapoints: http://dme.uma....ocialcomm09.pdf
Perhaps they could keep the presentation of the rating as-is (star system), but instead of asking users to manually give a star rating, calculate it using a weighted scoring function that considers:
- % of viewers who watch the whole video
- % of viewers who watch it more than once in the same session
- % of viewers who favorited, shared, flagged, or added the video to their playlist
- % of viewers who left a comment
- % of viewers who click “like this” or something similar
They should move the numeric ratings to a feeling based rating i.e. love, like, hate, etc.
Why would you visualize this data using a line chart. You’re clearly after not after showing a trend in this data. Using a bar chart would be much more effective.
Someone should read “Show Me the Numbers: Designing Tables and Graphs to Enlighten” by Stephen Few.
even a bi-directional rating system doesn’t work. it still allows for griefing.
the best way? explicit “like” with an implicit “i don’t like.” you only need one button — the “like” or the “favorite” or the “thumbs up” — whatever you wanna call it.
your rating/ranking basically becomes the number of thumbs up divided by the number of views. a view without a thumbs up being the implicit “don’t like.”
m3mnoch.
http://www.ted.com has the best rating system IMHO.
Pressing any option is basically a vote for the video (a 5 star). But, you can find “Jaw Dropping” or “Inspiring” (or any 5 other labels), which tend to actually be jaw dropping or inspiring or whatever.
Rather then a thumbs up/down, I vote for a plus/minus system where good videos get a higher positive vote. Sporcles.com uses this for comments, and when a comment gets to -5 you have to expand it to read it.
Netflix system is horrible in my opinion. 3 should be more of a so-so. I don’t like saying I have to like or dislike a movie. There are many that I feel indifferent to.
YouTube might take a look at the voting system that Reddit uses. In my opinion the simplicity of voting and how it ranks articles should be used as an example for YouTube how to handle things. They already have the “thumb-up-thumb-down” system for user comments, they should replace the 5-star system with something similar.
I think just a thumbs up vote would be better. To stop groups of people trying to sabotage someone’s video.