|
|
My EPinions Rating Philosophy...And Why The Site Would Be Better If More People Adopted It!Sep 21 '07 Write an essay on this topic.
Popular Products in Books
The Bottom Line EPinions is a wonderful tool for writers, but in order to survive as a consumer tool, its contributors need to exercise more discrimination . . .
I have been writing on ePinions now for about six years and I love it here. I write my reviews, which people who know me know I enjoy quite a bit. A few months ago, there was the "10 for $10" promotion and I was bummed when I stopped making a bonus after my first hundred articles! I'm a critic and I've been called a philosopher for some of my views and when I'm not writing reviews or working on my small business, I'm deeply involved with writing my novels and short stories. A few months ago, one of the Advisors or CLs started a dialogue with me outside the site about what I could do to make my body of work here more prominent, suggestions which included participating in the message boards, writing general articles and not being a jerk. Well, I tried writing on the message boards, but I seem to have different philosophies than many of the people in the ePinions culture. That's fine. Every website has a culture and many sites like this one have a clique and at some point, one needs to decide if they want to be on the inside or are happy working independent of that. Those who choose the independent route must simply accept that they will likely not advance or be recognized by the mainstream. Well, as a longtime jerk my only option was to start to write some of the general writing articles and I've spent this month adding to my inventory a significant number (1 in three) of general articles, mostly lists of "Best of"s. I've enjoyed that quite a bit and because many readers are not part of the message boards, I thought I would take some time to fully lay out my ePinions Philosophy for those who are considering trusting me or are thinking about the longevity of the site. The Backstory I grew up in Rochester, New York where I lived on the outskirts of the more affluent suburb of Brighton. I lived in the fortunate borderland between the city and the suburb that allowed me to go to the better (Brighton) schools and have all the safety and security of living in the 'burbs, without the pesky problem of being so close to other schoolmates that I actually was troubled by such things as friends. In Rochester, the local newspaper, "The Democrat And Chronicle," had (up until recently) a resident movie critic Whose Name I Shall Not Speak. Said critic was so liberal with his rating system that one would think he was rating on a scale of 6 - 10. If he suspected a movie would be crap, it seems he just would not go to see it. Now, on a ten point scale, you have a lot of room to illustrate a discriminating nature and while virtually anyone who cares will read the article and argument for why you rated a work what you did, a number on a ten point scale immediately draws the eye. The sheer number of 10s this reviewer gave out for works that were not perfect or classic was just disgusting. My EPinions Philosophy My general disdain for the Reviewer Whose Name I Shall Not Write led me early on to develop my own system of rating things. I strive for consistency and I love a ten-point scale. Ten point scales make sense to me because they allow a sophisticated range that clearly illustrates the difference between works. I rate my works on a ten-point scale in this way: Movies - 3 points plot, 3 points character, 3 points acting, 1 point special effects, Music - 3 points lyrics, 3 points voice, 3 points music, 1 point album composition/arrangement, Books - 3 points plot, 4 points character, 3 points style With this degree of consistency, those who read my works are likely to have a very straightforward gauge of where things fall. This also allows two important things: 1. In movies or music, works that are catchy and use gimmicks to overwhelm do not blow away more substantive works (say what you will, but "Jurassic Park" is no "The Godfather!") and 2. No one gets points simply for showing up. One of my fundamental problems with ePinions - with the star system - is that there is no option for zero stars. On a ten point scale, ratings become much clearer and the difference between something like Heather Nova's album "South" (a perfect album in my pantheon) and Dar Williams' album "My Better Self" (which has one song that just gums up the near perfection of the album) becomes clearer. On a more popularly recognizable example, a ten point scale can illustrate the difference between "The Empire Strikes Back" (perfect!) and "Alien" (the shot of Sigourney Weaver's crack is SO dated!). Now, I am not arguing for the idea of converting ePinions to a ten-point scale; it is far too late for that. But, in order to understand my EPinions Philosophy, it's a nice place to start the discussion. On a ten point scale, it's easy to argue that 10s are reserved for perfection, zeroes are reserved for "these are so bad everyone involved should be shot." With a ten-point scale, such levels of discrimination are easy to illustrate and form a much more universal standard than the somewhat more vague guidelines for the five-star system ePinions has. Five stars for "Excellent" is arguably not the same as "Perfect;" "Excellent" opens up a bit more territory than "Perfect." In converting from a ten-point scale to a five point one, the reasonable way to rate is to simply divide by 2: 9 - 10 = 5 stars, 7 - 9 = 4 stars, 5 - 7 = 3 stars, 3 - 5 = 2 stars, Below 3 = 1 star (you can see where the zero star option would be handy in balancing this!). When discussed in these more concrete terms, my philosophy becomes rather reasonable and clear: Contributors to ePinions must make a strong effort to clearly discriminate when using the star system if ePinions is going to survive and grow as a useful tool for users. As a natural result of utilizing a more discriminating sense of rating products, the body of a reviewer's works ought to end up resembling something close to a bell curve. The Arguments Epinions suffers a serious problem (if not now in reality, than the serious potential for the future) with having a freakish split between the ePinions culture of contributors and its usefulness as a tool for readers. Whenever I have pitched the idea that individuals need to take more responsibility for how they rate things, the most common responses I get are: 1. "I don't pay attention to the star rating," 2. "The review body is much more important to me," and 3. "I'm just writing to please myself, I don't care what anyone else thinks." The star rating might not mean anything to some contributors, but it is the essential tool by which users evaluate a product on the site. For example, if I am looking for a new garbage disposal and a list of five garbage disposals comes up, the only information I am immediately privy to is: the item name, how many people have rated it, and the star-rating. If I am looking for a new garbage disposal and I want to know which ones to consider, the star rating becomes critically important. Why would I waste my time on looking at a garbage disposal that is rated 1-star with fifteen reviews? No, there's a clear consensus that this is not a quality product. In fact, sad to say for reviewers who took the time to rate those products, such a consensus of a negative for the product is grounds for a user to not even open the reviews. The star rating clearly illustrates that this is not a quality product. Similarly, if there are fifteen reviews for another garbage disposal and the net star-rating is five-stars, this will pique my attention, especially when the remaining three have all averaged out to three stars. In this example, it becomes quite clear that the star rating is the first line a USER has with understanding the quality of a product. (In reality, of course, the five garbage disposals that come up will have two that aren't rated, one that is rated by one person and two that are rated by twenty people each and average out to just under five stars.) Users look at the star rating right away. The idea that the review body is important is a good argument, but it's an argument that needs to be looked at through the lens of a practical user. The practical user to ePinions is a person who is looking for something, who wants an educated opinion that comes from experience. The strength of ePinions is that the contributors are not burdened with corporate sponsorships, advertising revenues that bind them to less critical reviews or professional ego. EPinions mortgages the usual trappings of so-called professionalism for cold, hard realism. We offer the world real-world, practical reviews on everything and as a result, our perspective is not corrupted by any corporate agenda . . . . . . so long as our reviews are read. It is my firm belief that the average ePinions user, faced with two items they are considering, will read the top two reviews and one dissent of each. So, going back to my garbage disposal example, fifteen people have rated Disposal #1 an average of five stars, looking at the list, Disposal #3, which has fifteen ratings averaged out to three stars is also in my budget. Based on budget and rating, I'm going to open those two reviews for further information. For Disposal #1, the top two reviews are five-star ratings and because they are right there at the top, they are the ones I read. They are absolutely glowing reviews, well written (Very Helpful and Most Helpful ratings). Because it seems so great and the "pros" and "cons" of twelve of the other reviews seem to echo what I've already read in the prior two articles, there's little reason to open them. However, at the bottom of the "Very Helpful" reviews, there is a two star rating. Because this is such an extreme difference from what I've just read, I feel compelled to open it because I need to understand if the reviewer was simply biased or if there is some extreme problem that the other reviewers simply did not catch. If I find that last review to have real concerns, it increases my desire to check out some reviews of Disposal #3. My point here, of course, is that the body of a review is truly where the information is, but as users, people are less likely to get to the body unless there is a clear reason from the star rating and the position of the article in the list. Sure, you might get one of those diligent consumers who reads all ten reviews, but it's not the average (time constrained) reviewer. The least effective argument against being more discriminating is the idea that we write on ePinions only to please ourselves. It is this exact mentality that will make ePinions a poor tool for consumers because it simply reinforces the clique culture of the contributors. I write to please myself, for sure. I share that writing with the rest of the world because I have something to say. I think people ought to give a damn about what I have to say because I have a keen mind, sharp (if unconventional) insights and an ability to express myself that makes my meanings clear and (hopefully) reasonable (as well as entertaining from time to time). As a tool for consumers, ePinions is not served by those who uniformly rate things high or those who only rate the things they like. Paraphrased from "Black Adder," it's people like that who will laugh at anything, they'd laugh at a Shakespeare comedy. This is the toughest nut to sell to the establishment ePinions reviewers, the ones who are high in the ranks because they buy into the cliques, have a web of trust of people who will simply reinforce their standing and are more concerned with ePinions as an outlet for personal expression as opposed to a tool for curious consumers. This is not to say that the majority of people in the Top 1000 do not have writing skills, indeed, some of my favorite contributors on ePinions who do not buy into my philosophy are truly great writers. The problem is, many of them are not great reviewers. Allow me to explain: at the top of my article, I cited a few examples of movies and music because they are inoffensive, but offer a good idea of a way to evaluate a system of evaluation. I shall return to that example to make this part of the argument. "The Empire Strikes Back" is almost inarguably a great film. If I go to a reviewer and see that they have rated "The Empire Strikes Back" five stars, I have a sense that that person has a real appreciation for cinematic greatness. Indeed, a four-star review where the person argues that the dialogue is a little stilted, I can accept as well. Even if I did not think "The Empire Strikes Back" was a perfect film, if I see a reviewer has rated it five stars, this clues me into the idea that this reviewer is falling along with public opinion and while it might not be my personal tastes, they seem to get it. I look at the person who rates "The Empire Strikes Back" with one star and their review just says "this movie is dumb" and I just say "this person is a poor reviewer." The problem comes in with the less-than perfect movies. I look at various reviewers who have reviewed "The Empire Strikes Back" with five stars and I want to know which one is the better reviewer or the reviewer I truly believe I can trust more. Reviewer #1 has rated "The Empire Strikes Back," "The Godfather," "Just Friends," "Scary Movie 4" and "The Lake House" all with five stars. My thought is going to be either "This person likes everything" or "This person has no taste" or "This person doesn't care about the star system." If it's the latter, this is not a useful reviewer for consumers to the site looking to make an educated decision about whether or not to purchase something. If it is either of the former two (the reviews all describe those films in glowing terms on par with "The Empire Strikes Back") this is someone I would avoid. Reviewer #2 has rated the same five films, but the ratings are all over the board, with the first two at five stars, the next two at one star and the last one with three stars; my reaction is to say "This is a person who sees lots of movies and has the ability to discriminate between them!" That is a useful reviewer to someone as a consumer. The other permutation, of course, are the ones who I look up and they have rated "The Empire Strikes Back," "Star Trek II," "Alien," "Enemy Mine," and "Solaris," and I evaluate them in a similar fashion with the caveat "These are people who are primarily interested in science fiction films." The useful reviewer of the two in such a circumstance will also be the one who illustrates a clear sense of discrimination. A Good Place To Reiterate The Point Not everything in life is perfect or even excellent. It's not all sunshine and lollipops. And for those people for whom it is all sunshine and lollipops, the ones who make good reviewers are the ones who are able to say "Yeah, well, this day is too hot" or "This lemon-flavored lollipop is more satisfying than the unflavored lollipop over there." To make the example even more painfully clear; all movies are not great, all science fiction movies are not great. There are continuums. The useful reviewers are the ones who either attempt to explore the entire continuum and rate components against each other (i.e. I rate all movies, so "The Empire Strikes Back" is in direct competition with "Just Friends.") or they pick a niche and rate components against each other ("Scary Movie 4" is in direct competition against other lamebrained humor films like "Date Movie" and "The 40-Year Old Virgin;" leave whatever prejudices against the lamebrained humor genre at the door). The useful reviewers will either make that clear explicitly (i.e. in their "About Me" section) or implicitly through a scan of their body of works (i.e. I see this person is only rating science fiction films and the reviews are pretty much all over the board). The reviewers who are not useful to consumers (those who bother to investigate the complete body of a reviewer's work) are going to be the ones who apparently love everything they review. In a de facto way, this is the problem with the conceit of those who insist on only writing about what they enjoy. I get the impetus to do that, it's certainly more fun. However, if one is only reviewing items they know they will be rating with a four or five star rating, it cheats the consumers of a clear understanding of their abilities and discriminatory capacities. This person liked "The Godfather" and "The Empire Strikes Back" equally, well fine, but is it because they are particularly clever or because they're pretty much mainstream? It is our ability to dissent and express ourselves on issues that are not our favorite thing in the world that help define the quality of our reviewing skills. Cheating our readers of an opportunity to see how the rest of the world stacks up is a detriment to ourselves as reviewers and to the site. After all, the real weakness of the person who fills up the site with five-star reviews without anything else in their body of work is that their body of work may be huge without meaning. If one is only going to rate things with five stars, the least they can do is have incredibly well written reviews, but very few of them (i.e. you know these few items I've reviewed are truly great because of out of a database of 12,000,000 items, I've only bothered to rate these 100 items - implicitly, everything else is crap). Using My EPinions Philosophy The first reason for contributors to not fall in line with my ePinions philosophy is clear and simple: it's not profitable. Sure, it might be useful to have the majority of your reviews end up as average to clearly delineate between the quality products and the average ones, but it is not a great way to get one's review read. EPinions is biased in its system of promotion towards the extremes. If you want a chance at getting your review up on the front page of the site (which is a guaranteed way to increase traffic - nothing proved this to me like my "Star Trek: Voyager" reviews!) it needs to be rated 4 or 5 stars or 1 or 2 stars. Three star ratings are not given any sort of natural promotion technique. Until ePinions adds a "The Middle" to the front columns that feature "The Good" and "The Bad," contributors will have less reason to rate products as a function of reality. But outside that, my overall philosophy is one that can only help to strengthen ePinions as a community where contributors are useful to users. Almost everyone on ePinions with any significant body of work would no doubt have at least one review that they would re-rate years later. Rescaling reviews is time consuming and difficult to justify outside the technology reviews (I'm sorry, Tech people, but while every new cutting edge technology might honestly be 5-stars, five years later, it's not. Technologies reviews are the only ones that leap right to mind as the place where simply going back and adjusting prior star reviews along a more useful continuum makes absolute sense, even if it is time consuming), but for those of us reviewing in other mediums (especially media), evaluating your body of work makes real sense. The law of averages ought to apply to a truly discriminating reviewer. To throw myself in front of my own train, out of the 1332 reviews I've written (not counting the list/writer's corner articles), my reviews fall thus: 5 star - 119 4 star - 358 3 star - 516 2 star - 235 1 star - 104 It's not quite a bell curve, BUT, it does illustrate that I have experienced a significant number of things and I have an ability to tell the difference between them. For sure, I seek out things I hope I will enjoy so the scale is naturally leaning toward the positive. But it is not overwhelming leaning towards the positive. There is a clear sense when looking at my scale that the 5 star reviews mean something. Anyone looking at my body of reviews is likely to say, "Wow, out of 1332 reviews, there are only 119 five star reviews, this reviewer is really critical!" As far as reviewers go, critical thinking is essential. Using my EPinions Philosophy, ePinions reviewers should consider the consumer aspect of the site. Ideally, if the Powers That Be were to support such an initiative within the Community, there would be a simple tool on every reviewer's "About Me" page that categorized their body of work. The reason ePinions has worked so well for those who are part of the clique or establishment is that most consumers do not have the time or willpower to read the entire body of work of the biggest (most voluminous) reviewers. This conceit benefits those reviewers, but not the site. (The Counterargument To All My Conceits) There are - I know, because they've popped up on the message boards whenever I've made similar arguments - people who will angrily disagree with my statements and my Philosophy. They will be people in authority who have access to the polling data that ePinions generates when it has users (not site members) fill out their survey when they leave the site. There will be people who disagree with this Philosophy because it might inhibit their writing style or because there seems to be no hard data to support some of my conceits (most notably my assumed way many people use the site and decide with reviews to read). The importance of the star system for reviewing works on ePinions is proved by . . . shudder . . . the success of "Consumer Reports." The magazine "Consumer Reports," like it or not, has been an industry standard for virtually every industry because it provides a clear, concise, visual evaluation from hundreds of consumer reviewers - uncluttered by writing style or personal opinion - that clearly defines which products are worth users time and attention. How does one use "Consumer Reports?" They look at a chart where products are evaluated based on (essentially) a star system (a graphic indicator) and price. "Consumer Reports" survives and has thrived in the same marketplace and user base as ePinions for decades by providing users with Name, Rating, Price. Users likely enjoy ePinions for the thoroughness of the reviews, but when it comes to which reviews people are likely to read, that same user philosophy as the "Consumer Reports" model likely applies. In Conclusion In conclusion, as a practical measure, I would love for people who have a similar philosophy to this - the idea that ratings need to mean something - to contact me. I have a very small web of trust because I don't have the time or energy to go through another person's entire body of work to see if I truly do trust them. If you feel this philosophy is similar to the one your reviews embody and you would like the ePinions site to embody, drop me a line. Please include the total number of reviews you have written along with your star-count breakdown. I'm not here advocating a coup to the Establishment, indeed, it ought to be clear that I care deeply about the future of ePinions and its viability as a site. Furthermore, I do not view those at the top as adversaries in any way; I've received no indication before writing this that some sort of reviewer evaluation tool like a graphic comparing ratings to a bell curve is out of the realm of possibility for the site. But if an alternative cadre of reviewers concerned with the site's long-term viability and the integrity of comparative reviews is to form to challenge the Establishment, I believe my Philosophy is the way for us to go. It's time to encourage our reviewers to set real standards for ourselves and challenge us to use the star rating more responsibly by rating along a real, useful spectrum.  |
| Read all comments (9)|Write your own comment |
