Twinkle Twinkle Rating StarsFeb 18 '01 Write an essay on this topic.
Popular Products in Books
The Bottom Line Do you know what those star ratings really mean? I sure didn't.
While I originally came to Epinions to write reviews (and earn money) I quickly became reliant upon the Epinions of others in order to make purchasing decisions. Whenever I'm about to purchase something that is available for review here I spend some time reading both positive and negative reviews on that subject. Of course, many people (especially non-members) are simply interested in the quick-and-dirty approach to product reviews. Hence, the "Overall Rating" system represented by 1 to 5 stars at the top of a product's page. Assumption v. Reality I always assumed the Overall Rating was simply a composite of the total star ratings given by those who who have reviewed the product or service. This is not the case! In fact a recent email from Baron Jackson of Epinions Community Care stated: "The overall star rating is calculated based on the following criteria: Average product rating, with extra weight given to high quality reviews. The number of reviews about the product, combined with the Recency of reviews about the product allows desired products and their information easily found in the categories." Say what? First off, I'm assuming there's a word or two missing from that last sentence, but I didn't post this to critique his writing style. Instead, I'm wondering how useful this makes the Overall ratings to a consumer such as myself. Let's look at each set of criteria seperately: Average product rating Okay, this is a no brainer. Epinions advertises itself as the place to get "unbiased advice" and "personalized recommendations", so obviously the Overall Rating should be based on the average of the ratings given by the reviewers. [W]ith extra weight given to high quality reviews I understand the need/desire to weight better written reviews higher. This would help eliminate (or at least lesson) people "stacking the deck" for or against a product by entering a large number of garbage reviews with extreme ratings. However, it does call the claim of "unbiased advice" into question. The number of reviews about the product combined with the Recency of reviews about the product I'm not really sure how the number of reviews a product receives invalidates the star rating given in those reviews. I mean what, exactly, does "Overall Rating" mean anyway? I can see how a review of a service can be affected by time, but I'm not sure how this effects product reviews. If a car (or a computer, or a toaster) is a piece of garbage, it's not going to get any better over time. What this means Well, I'm not really sure what it means, but two things are clear: 1)Epinions has some secret formula (distant cousin to the IS formula perhaps ;) for determining a product's Overall rating. 2)The Overall Rating is, therefore, essentially meaningless. I often read movie and book reviewers whose views I disagree with if they have proven themselves to be consistent. They become a known quantity and I can still base my purchasing decision based off of their opinions and ratings. However, the Overall ratings are, I now know, based on an unknown system that therefore lacks consistency and is thus useless to me as a consumer. An Example - Page Creators Page Creators is a web-hosting service that I have never seen a positive review of anywhere. Here at Epinions it has two reviews: one by me and one by estrabd. Both of us gave overall ratings of One Star. Additionally, there are three secondary categories that can be rated. I gave all three 1 star each, while estrabd gave 1 star in two categories and 2 stars in the third. My review was written 01/8/01 and estrabd's was written 11/29/00. One would think that the Overall Rating would therefore be 1 star, maybe 1.5. However, the listed overall rating is 2 stars! How can this be?!? I did a little experiment with this review and altered my star ratings temporarily. By increasing my overall from 1 star to 5 stars I was able to increase the Overall Rating of Page Creators to 3 stars. Altering my secondary ratings did not seem to change the overall rating in any way. A quick glance around Epinions showed that most items with fewer than 5 reviews (the easiest to compare) do not have Overall Ratings that seem to be a reflection of the reviews posted. No 1 Star Ratings? I can't help but wonder where that second star comes from for Page Creators. So, I did a little nosing around and it appears that there is not a single product or service on Epinions with an Overall rating of only 1 star! I used the new handy little "View the lowest rated..." link in each category and I came up with 3 or 4 items rated 1.5 stars overall, but none that had a 1 star rating. Hmmm.... Something doesn't sit right here. Final Words The only thing I have been unable to take into account in this article is if a person rates an item without writing a review as I know of no way to discover who has done so, but I don't think adding that information in would change my basic conclusions. If someone knows of a way to learn how many people have rated an item (and what those ratings are), please let me know and I will update this article as appropriate.  |
| Read all comments (7)|Write your own comment |
