Changing Epinions Into a Meritocracy
Feb 06 '02
The Bottom Line Changes to the ratings system at epinions are needed to force a shift from a marketing-driven to a quality-driven writing community.
I've only been writing on epinions for about five months, and although I've written all or part of fifty reviews, I've only published fifteen. Yet I love to write, and in concept I think epinions is a great site. So why am I not more involved? The answer lies in one word - quality.
When I started with epinions, I was attracted to the idea that the system was a meritocracy, rewarding (financially or through personal recognition) quality writing that really helps the reader. It turns out that is not the case. Quality writing is not king at epinions, despite the many essays by long-time members insisting that is true. No, marketing is king. Selling your "product" (ie. your review) is king. Politics (in both the best and worst senses of that word) is king.
I came to this conclusion by analysis, not accident. I couldn't understand why I saw so many highly rated reviews that were poorly written and not useful for any reader. I couldn't understand why some authors whose work was mediocre got many more hits and higher ratings than others - less well known - with very thoughtful and intelligent reviews. This question bothered me, so I started reading the essays in this subject area (General Comments on Epinions) and testing the statements made in those essays against my own results from a survey look at the reviews on the site. My conclusion is that we have a writing community that is structurally biased in favour of marketing (ie. it rewards marketing) and against writing quality and helpful reviews. And, the reason for that structural bias is a rating system in which ratings are capable of being used to "buy" the support of fellow community members.
(I should say, by the way, that I have been lucky enough to have almost all of my reviews rated highly, despite spending only a little time on the marketing side of epinions Thus, I don't have a personal axe to grind here. Rather, I am commenting on what I see as a structural flaw in the system, which clearly disadvantages many potentially valuable members, and in my case motivates me to be less involved in the site than I otherwise would be.)
The evidence, it turns out, is obvious once you focus on it. Average ratings on the site are much higher than you would anticipate if they were fair reflections of the quality of the work. It is clear that anything less than a "very helpful" is a rebuke (mild or severe, depending on whether the rating is "helpful" or lower). "Somewhat helpful" actually means "not up to an acceptable standard", and "helpful" means "read this review only after you've read all of the competent ones". "Not helpful" really means "are you kidding me". The reason for this? The ratings you give will affect the ratings you get. If you want other members to rate your reviews, and rate them highly, you must, without fail, rate lots of their reviews, and rate them highly. It is not a guarantee of a good rating in return, but it is a prerequisite. There appear to be few exceptions to this.
Further, those who have already established large webs of trust virtually always get highly rated reviews, no matter how good or bad their reviews actually are. Sure, they are often better reviewers than most, whether because of experience, or practice, or talent, so the reviews themselves are often better. But that is not always the case, and I have found many poor reviews by popular writers that have received high ratings. The reason? Again, reciprocity. These are the people with power on the site You don't want to mess with them, because whether or not they will personally "get back" at you through negative or revenge ratings (I think generally they don't do that), their massive webs of trust will either downrate you (not a "friend", so to speak) or, even worse, stop looking at your reviews altogether. In either case, the success of your own reviews is not based on their quality, but on the nature and extent of your own ratings of others.
The same impact is seen in the use by some (not all, of course) of vindictive or sales-oriented trusting, which are two sides of the same coin. How often have you seen someone add you to their web of trust out of the blue, then wait a couple of weeks to see how you rate their reviews, only to drop you if your ratings or their volume do not come up to their expectations? In fact, it is expected that you will rate fellow members that trust you, and some members with big webs of trust spend hours every day doing just that. This has nothing to do with writing quality reviews, yet the status and ratings of epinions members are heavily influenced by how actively they build and maintain their webs of trust. At the beginning, I thought the web of trust was about how many people thought your work was good. That was wrong (and naive). In most cases, it is about "you scratch my back, I'll scratch yours".
I also looked at the members (not many, thankfully) who rate dozens or even hundreds of reviews every day. Give that a thought for a second. If the average review is, say, 500 words, how long does it take each day to read, think about, and rate 100 of them? By my count (and I'm a pretty fast reader), that would be at least two hours, maybe three or more. So, do you think those high volume rating machines actually read and think about all of those reviews? The answer is, no, they do not. They are marketing, and getting a whole lot of high ratings out there will result in many other members rating their reviews and adding them to their webs of trust.
Finally, I looked at some epinions satellite sites, like epinionsaddicts and the others of that category. I wanted to know what is going on at those sites, and I found out. About 10% is discussions about epinions, and sharing of epinions experiences, much like any other topic-specific chat room. The other 90% is, either overtly or subtly, "look at me, I have some reviews you should go and read (and rate highly)". (I kept expecting to find one that allowed you to buy banner ads for your reviews.)
If I go back and read this, it sounds like it might be a blunt condemnation of the members of epinions. That is not intended. It is a blunt condemnation all right - but of the rules of the game, not the players.
Look, imagine a company where employee performance reviews are done by vote amongst all of the employees. Any employee can rate you, regardless of how well they know you, and there are no formal consequences arising out of the honesty (or lack thereof) of your rating decisions. What will happen? I've seen this in a number of companies, and lots of studies have been done of the group psychology that applies. In a small company (8-10 people), ratings are very honest and the process can be strongly positive in building team dynamics and helping people to improve. Once you get any larger than that, ratings become less and less about actual performance, and more about internal politics and self-interest. If you have a company of 1000 people, all able to rate anyone, success goes to the person who spends a couple of hours every day going around the office ("hi, Jack, how're the kids?") and getting their face and personality in front of others. People make overt or unconscious deals about how they will rate each other - either agreeing to trade high ratings, or just rating the people they know and like highly and expecting that they will get the same in return. In general, the actual quality of their work is only a factor if they are really bad, and everyone finds that out.
That is exactly what is happening at epinions, with the result that the "best" reviewers are those that market themselves most effectively, and many good reviewers are not seen (and many probably quietly give up when they fail to get the recognition their work actually deserves).
Now, this diatribe is not primarily a negative one. The point is not to attack, and I am not a big fan of those who whine about things without being constructive. I believe this problem can be solved, but it requires some changes to the rules of the game at epinions.
My proposal is that epinions adopt two changes:
1. The current rating should be replaced with a 1-10 rating system, and a structure established so that each member rating others would have to maintain a balance in their ratings. They would not be able to rate on a high average or low average. The system would not just discourage it, but prevent it. There are two ways of doing this, described below.
2. Every member would have a numerical member rating, composed of the average of all of the (numerical) ratings of their reviews for the past, say, six months. That way, any person seeking to find out about a subject will naturally look first at those whose epinions have been the most highly rated. It will be easy to tell. Unlike now, where everyone even remotely competent has an overall "Very Helpful" rating, so that it is impossible to distinguish between them (unless they are top reviewers, which is more about volume than quality), reviewers would actually be ranked on quality.
A revised, quality-driven system only works if political or marketing-based ratings are discouraged by the design of the system, and the distribution of ratings fairly reflects the distribution of quality of ratings on the site. I propose two possible ways of achieving this:
A. Automated Bell Curve. Under this system, each member could give any rating (on the 1 is lowest, 10 is highest, system) to any review. However, after they rate, say, ten reviews their ratings would be put on a bell curve and recalculated to fit it exactly. This is not a complicated algorithm, and basically results in each member's ratings being "normalized" so that they are comparable to the ratings of all other members. The system would thus recognize that some people rate using a higher or lower scale than others, and would recalculate the high ones downward and the low ones upward. This is technically sound, and statistically correct, but aesthetically and socially displeasing. Not only would the constant recalculations be annoying (and processor hogs), but they take away some of the person's freedom to rate as they see fit. I therefore don't recommend this approach.
There is a variation on this, often used in academic institutions for marking systems. In this variation, a member's ratings would be compared automatically to a bell curve, and as long as they remain within a predetermined deviation from the bell curve, they would be left alone. If a member attempted a rating that would take their overall ratings outside of the range of acceptable deviation, they would be given a choice. They could change the rating, change their rating on something else in a manner that gets them back onside, or let the system do an automatic recalc of all existing ratings. This adds more freedom, but it is still a processor hog and is still intrusive.
B. Currency System. This is the system I prefer. When a person signs up as a member, they are issued 100 ratings points, which act as a type of currency. They can allocate up to 10 points to any review they rate. However, each time they rate they only get 6 points replenished. If their average of rating given out is 6, they will always be fine, and can rate as many reviews as they wish. If they tend to the high side, though, they will run out of ratings points and be limited in their ability to rate others. Mathematically, after rating 100 reviews their average can be no more than 6.94, and after rating 1000 reviews it can be no more than 6.09. If you game this system, you see that reviews will be rated on a bell curve that peaks at just over 6, with a broader distribution than is now seen, and a clearer differentiation between good and excellent reviews. Further, members will be motivated to review newbies, since they are more likely to get lower ratings at the beginning.
I prefer this approach, because while it accomplishes the result of forcing ratings to be more realistic, it allows us to rate every review exactly as we wish. The only difference from now (aside from the numerical vs. categorical system) is that we can only spend what we have in our wallets. We don't have unlimited gold stars to give out for whatever reason we can think of.
Does this mean there will be no more marketing? Probably not. There will still be people who try to get together and rate each other highly in an organized manner, but there will be less of them. Right now, there is no downside to doing that, so everyone has to do it if they want to become "successful" at epinions. In this proposed system, for every gold star you give out, you have to tell someone their work isn't good enough. Most of the members of epinions are good people, and they will simply not be willing to play games with the system if they have to whack people to accomplish that. The only ones who will are the real bad guys, and I believe that the members generally will spot those people and limit their effectiveness.
And certainly the most common rating problem today - spreading an unlimited supply of high ratings around as a marketing tool - will no longer be possible.
Look, I think epinions has a lot of potential, but if it is just a community based on marketing your product and politicking your way to recognition, it's not really very interesting. I get enough of that every day in the office and the boardroom. I don't need more of it when I am trying to enjoy myself. But if the community were to change the rating system so that ratings have to be fair (on average), and a person's combined rating for all of their reviews is a key part of their on-line identity, I believe epinions can start to be driven by quality writing and thus achieve its potential.
I would welcome comments or responding essays on this subject. I think the subject is worth discussing, and if through this essay I can generate some of that discussion (whether you agree or disagree), I will have accomplished something useful for epinions and its members.
 |
|
|
|
|
|
|
Epinions.com ID:
gandalf2001
|
|
Location: Canada
Reviews written: 16
Trusted by: 6 members
About Me:
Epinions toughest ratings.
|
|
|
© 1999-2009 Shopping.com, Inc.