|
|
Why All The Bickering?May 12 '02 (Updated May 14 '02) Write an essay on this topic.
Popular Products in Books
The Bottom Line Maybe Epinions isn't tweaked enough yet - I have some suggestions from what I've seen and heard so far...
This great quote was shared recently by 'sleeper54' "We all want progress, but if you're on the wrong road, progress means doing an about-turn and walking back to the right road; in that case, the man who turns back soonest is the most progressive." --C. S. Lewis (Don’t read this if you have a weak stomach!) I’ve been a member at Epinions for over two years, but I’ve been away from the site for most of those two years. A lot’s changed (I still have to go back and edit all my previous reviews – doh!), but I still have something to share. I’ve been seeing a lot of bickering amongst members, and noticed many things that could be improved to help smooth things over. My changes would work perfectly (I believe this, but also realize some bugs and details would need to be worked out), to create a site that would make everybody happy! My general plan (of an Epinions “Utopia” ;} ) wouldn’t change anything that has already been accomplished by members (besides things like titles – which could possibly still be used at a later date), at Epinions – but would significantly change how things are accomplished in the future. A lot of this content is actually from a ‘comment’ I had made on SMITHSWOODSIDE’s review in “General comments about Epinions.com”, “What has Noah got to do with Epinions?” (May 6, 02) – and I was given feedback suggesting I publish a review/essay on it. Here’s what I’m throwing out for feedback: Comments: Anything below an H rating should have a compulsory comment from the rater. If the rater doesn’t give a comment for OT, NH, SH the writer can petition Epinions to have the rating removed (or maybe there’s a connection in the program that allows the writer to remove it when there’s no comment). In General: Epinions is a perfect creation, a solid forum. However, the current system is too top heavy, and the overall system is too pyramidal/patriarchal I’ve seen and heard of some high quality reviews being pushed down by entrenched members. I don’t fault members for this because the system is designed (intentionally or not) that way. But it seems to me there has to be a better way for every review to have a chance at being the top review (even if it’s from a brand new member who’s only written the one review). After all, if it’s the best, it’s the best – and should be treated as such for the potential buyer’s benefit. On titles: I’ve read Top Reviewer’s reviews and sometimes found them to be too long, too bland, and often containing more bloated speculation on a product than solid feedback for a potential buyer. It’s my impression that ‘titles’ are currently serving to feed egos more than help get quality advice to buyers. I’ve found that feedback from ‘title holders’ is often the worst feedback I get! This may be because their rushed (having to review many reviews), over taxed mentally from reading, possibly self-protective or just don’t care very much. It’s also been the case that some Advisors aren’t even giving feedback (brought up in the ‘comments’ section of the review I mentioned above). More often than not, from what I’ve experienced, ‘title holders’ give the same feedback everyone else does – and this begs the question “why do we have them?”. Is it to guarantee feedback? I have a better idea – keep reading. Conceptual Site Design Changes (with general technical overview): This IS a democracy!: Irreverence is needed at the bottom to not only bring top reviews (from the one opinion written by the new member, to the hundreds written by prolific members) to the potential buyers attention, but to keep members honest. For the current system, or any future system, there should be an appeal process for anything rated below SH – for any other reason than the reviewer NEEDS the feedback. This should not be retroactive, obviously. Grandfather clause is IN EFFECT! No One’s Gonna Like Me: MEMBER ratings should be LIMITED. I’d recommend no more than 50 ratings, but the actual number used is obviously debatable. I say this because I’ve noticed too many members that have cliques formed and get an automatic bombardment of VH’s from other members. This ultimately (and immediately) pushes other reviews and reviewers (especially those unestablished reviewers) to the bottom. This should not be retroactive, obviously. Grandfather clause is IN EFFECT! Potential: Every review should be treated as if it has potential. Top reviews (ultimately) should stay available (in public, for potential buyers) from non-member visits, not member ratings. This isn’t much different from the current system. The thing is, they gain public status differently. Published reviews from ALL members should have a mandatory number of ratings from members (maybe somewhere between 30 and 50) before they become public, but members should be limited in rating other members reviews (for example, one member is only able to rate every twentieth review of another member). The goal is to prevent monopolies, unless the monopoly is deserved. In conjunction, every member should be required (or their membership is revoked) to rate a list of rotating members, within say ten days of receiving some notification. Of course Epinions uses discretion with members that have to be away for a prolonged time etc. Overhaul: I recommend an overhaul in the member rating schematic. A seven tiered system instead of five, using more objective terminology instead of subjective. The goal here is to make ratings as objective as possible. A scale based on “Helpfulness” is too relative to the rater’s concept (and what’s trying to be accomplished is putting reviews that appeal to many on top). The great people here (especially those with the new titles) are struggling (in my opinion) to make some type of objective determination using very subjective criteria. The current ‘five’ tiered system is fine for potential buying or non-member visits, but for member ratings the system should allow for more (objective) distinction between reviews. What follows is a completely experimental suggestion of a rating change in general; 1. proper grammar (1 “awful’ to 3 ‘great’) 2. gives description (1 ‘none’, 2 ‘minimal’, 3 ‘adequate’, 4 ‘solid’, to 5 ‘just what the buyer needs’) 3. tells of experience with product (1 ‘none’ to 3 ‘everything I’d want to know’) 4. length (minus 1 ‘no continuity, too long, uninformative’ to plus one ‘perfect length for this product’) (etc., as in three more criteria needed) These criteria are still (obviously) somewhat subjective, but with the other changes above this also gives writers immediate feedback – and they could appeal an “overall” rating to a randomly selected seven member (not a member that rated the review) ‘committee’ to vote on the appeal. Some of these things would be very easy if maintained on the Epinions website, and email notification or delays wouldn’t be a problem because the member could be prevented from account actions until they’ve voted an issue, for example. It’s Not Complicated: My run down is incomplete and long, but I think you get the general idea. Some different rules would actually make the system here a lot simpler and better. The goal is to get the best review to the buyer, right? No matter if the reviewer has written 1 or 100 reviews. So who decides if it’s best? The Epinions self-rated system works because members decide which reviews are best. BUT, we all also have a vested interest in OUR review being the best, so a few minor limitations and adaptations to the system would effectively allow ANYONE to earn Eroyalties with a good review (and get top review status), and the best review would earn the most because of non-member visits still. In addition, with this system entrenched reviewers won’t have their reviews vaulted to the top automatically. They’ll still have to work for it, and most will stay where they are because they HAVE BEEN and WILL work for it! See what I mean? What do you think? - for Epinions – I waive any or all rights of claim to your implementation (if it suits your fancy) of any suggestion I’ve made above :} I'VE DECIDED TO ADD SOME OF THE COMMENTS ABOUT THIS ESSAY TO THIS ESSAY, TO MAKE IT EASIER FOR READERS TO SEE SOME OF THE MAJOR POINTS OF DEBATE BROUGHT UP AND MY RESPONSE TO THEM. WHAT FOLLOWS IS MY RESPONSE TO TWO COMMENTS, ONE FROM 'SLEEPER54' AND ONE FROM 'SCMRAK': (THIS IS NOT INCLUSIVE OF ALL COMMENTS, ORIGINAL COMMENTS MAY BE VIEWED IN THE COMMENTS SECTION) To 'sleeper54', May 13th: I'll try just using my words and quoting as little of the review as I can, just to set the context of my thoughts as needed... I’ll do the same thing, using you comment Mandatory comments, and deleting SH NH – Looks like we’re in agreement, we can start forming a coalition to get this one passed! If we have to, we can shift this to just OT NH in our petition… :} you said... "Epinions is a perfect creation..." Conceptually Epinions is a perfect creation (in my opinion, yikes!), I stand behind that. "...too pyramidal/patriarchal..." Agree with 'pyramidal'...but perhaps 'matriarchal' may more often apply ?? Matriarchal, patriarchal – that fact that this can happen, in a forum designed to allow (supposedly) anyone to voice an opinion and have a fair chance that their review will make it to the top, means that the system is flawed. On the over-worked title holders, I agree completely with what you’ve said in your comment. You mention, once again, deletion of SH NH, and agreement with an appeal system. I too agree completely, with a possible tweak of just OT and NH, or maybe all three (the way you’ve described, allowing appropriate time for appeal before deletion). On ‘Digestion Time’, we agree on a minimum number of rates before a review can be seen in public. This however must be in conjunction with (the) other changes. For example, if a review is overall SH but received the minimum number of ratings, while another review is VH but didn’t get the minimum number… see what I’m getting at? Even if the SH will soon be deleted, a public SH or H in theory aren’t as helpful as the VH that’s not public. "...members should be limited in rating other members reviews..."...I see your intent, but it would be tough to tell me that I can't rate 'greencheeseboy' as often as I want. The prevention of members rating other members reviews is essential to a fair system. This strikes at the heart of all the bickering – which will always exist (right now, however, the fighting is caused by the flaw in the system). Using a java applet interface for rating would make this possible, and the system would keep track of this aspect. All it means is that you can’t go to your favorite reviewer and “vault” them to the top (which is what’s happening with the VH bombarding), people will have to get to the top through random ratings from random members instead of their cliques. This will ultimately limit bias and monopoly forming while bringing the (most close to truly) best review to the potential buyer. To repeat, this is essential. "The goal is to prevent monopolies, unless the monopoly is deserved."...Nothing but big problems with that, IMHO. I disagree, most systems we have in our culture prevent monopolies, why not here? Checks and balances… and a couple rules enforced as law. Too easy! (j/k) Really, it’s very possible – see above. "...every member should be required (or their membership is revoked) to rate a list of rotating members..."...Will never work, IMHO. You can not tell a volunteer rater who and what and when they will rate... Ahhhh, the fear of change… ever present. I think this is what you’re indicating, that members won’t like the change. Well, they have several options – most immediately quit or stay. Making an honest assessment, I don’t think a fairer system (if presented in the right way) will cause many people to stomp off in a huff. They’ll probably welcome it, if they don’t lose anything they’ve already gained. And actually, volunteer organizations change once you start paying people. Huge success comes from volunteer organizations that have rules, even when possible dismissal or forfeiture are penalties. They’re so numerous, but I could mention a few - the Red Cross, the Army, the Congress, technically every job is volunteer - following rules isn’t, but since when has that made it impossible to implement fairness? IMHO, I disagree it will never work! Your "tiered rating system" comments... Each category would need to use the same 'scale' i.e. 1 to 5 or whatever. And you're really going to ask people to make 5,6,7 decisions on each review rate ?? And they called my suggestion 'too hard' !! ...:-) If the rating system were displayed across the bottom of a review, as it is now, with little boxes to check for each category (and considering the automatic feedback, less of a need for commentary) I think it would probably take less time to accomplish – and with the other changes be far more fair! Again, IMHO, I disagree that it’d be too hard. you talked about appeal of an 'overall' rating... I like the idea of a process for an appeal of an overall rating. But not sure how useful it would be. Or how often it might be used. If I had a review with an overall rating of H, would I appeal that because I think it should be a VH ?? Sounds odd... I initially specify that appeals would only be for SH and below, and given the current furor over ratings I’m certain it would get plenty of use. Besides, just because the process exists doesn’t mean it has to be used all the time. Writers can easily edit the review to make it better! After a few denied appeals from a random seven member vote, I’m sure they’ll start editing more than appealing. Thanks for your valuable feedback, some of these clarifications have helped me understand my own thinking better. And always remember, nothing is impossible. If members don’t like a more objective and fair ‘behind the scenes’ system for presenting top opinions, maybe they shouldn’t take part – but IMHO, I really believe that the current members care about Epinions more than you think. They’re probably not going to stop reviewing and being a part of the community. They’ll still be earning Eroyalties, still be writing, still be reviewing, and still be valued members. We just need some fairness – and limit the ability to create massive monopolies. Do away with titles and renovate the ratings system – truly how hard could that be (conceptually, not technically). I think members will adapt easily (the conceptual part) to a new less convoluted system. Steve :-} sleeper54 "We all want progress, but if you're on the wrong road, progress means doing an about-turn and walking back to the right road; in that case, the man who turns back soonest is the most progressive." --C. S. Lewis To 'scmrak', May 14th: ‘scmrak’ U say “First, limiting to an arbitrary number of ratings wouldn't address the (theoretical) main purpose of Epinions: that of providing a juried forum for exchange of information about products and services.” Epinions is a vehicle. What you describe above is the ‘vehicle’ means through which Epinions accomplishes its main purpose – that being to provide a potential buyer with a top quality review. It can be said that without the buyer Epinions wouldn’t exist, but it can’t be said that without Epinions the buyer wouldn’t exist. You have the concepts flipped. The ‘forum’ is then second to the buyer and ultimately the main purpose of Epinions - getting a (best) review to a potential buyer. That said, if a new member writes one review (a theoretical “best” review) there’s no way in the current system of VH bombarding (unless it happens to be the only review) to get that “best” review to the potential buyer. To spell it out more clearly – first comes the buyer. Second comes Epinions attempting to get that buyer a review of a particular product. Lastly (what you speak of) is Epinions system (or ‘forum’) for getting the ‘best’ review to the buyer. The current system is radically flawed in the last respect. Well, okay, not radically… it’s flawed though. As a side note; don’t juries have 12 members? U say “Who's to say that the first N members to reach a review are those best-qualified to judge its quality and veracity?” Who’s to say that just because someone has written 400 reviews that their opinion is the best? Who’s to say that because someone has a WOT of 800 members that their opinion is the best? The current system gives no incentive for new members to write opinions. Is a politician of 20 years the best man for the job, just because he’s been there 20 years? How does the new politician (with new and better ideas) get voted into office when she has to fight the entrenched incumbant of 20 years who has a million dollar bankroll? The best system to label reviews as best and worst is one that relies on a minimum and maximum number of member ratings, but from ‘random ratings from random members’. U say “Heck, there are reviews around here that I've hit with thirty or forty VH ratings already accumulated that I found to be complete plagiarism! Besides, the overall rating of a review is a function of the aggregate size of the rating members' WOTs, not of a raw count.” The argument you’re using is actually benefiting this idea more than your criticism. The added aspect of plagiarism is a good point – maybe that can be the new job for Advisors (to root out plagiarism). The overall ‘weight’ of the review being based on WOT as opposed to raw count is contradictory – both are raw counts. The WOT may be some other aspect of the same raw count, even so, it’s the WOT that’s leading to the VH bombarding. In other words, the system is doubled flawed in that not only are good reviews from new members not as ‘good’ because their WOT isn’t established yet – it’s also not as good because they aren’t being bombarded with VH by the members that are a part of their WOT. In addition, in respect to your point, members with mutual WOT are being inflated in this same way. All the things Epinions and its members tried to prevent have not worked, and in considerable numbers there’s WOT and rating swapping. U say, “Second, the concept of a mandatory explanation of a sub-HELPFUL rating is somewhat troubling. Not that it doesn't address a problem, mind you, but it's usually unnecessary for twenty-seven people to leave similar comments on a review: "You need to put in the number of tracks/number of pages/horsepower/maximum output/RAM/format..." In fact, I'd think it would frighten off newbies and encourage ratings inflation to avoid having to leave a comment. Advisors have already been charged by the Category Leads with leaving such comments; in the few categories where I regularly read, there are almost always comments on reviews explaining ratings -- and have been for as long as I've been here.” So why can’t raters put unique feedback instead of a regurgitation of what everyone else is saying, especially those being asked and paid (albeit minimally) to do so? Again, it supports the idea that not so many ratings are necessary. Mandatory comments are necessary for feedback to the writer though, because if you tell someone ‘your idea stinks’ or ‘your review needs work’ with an SH, NH or OT you need to tell the writer why you think this. U say, “As to the statement: (from me) I’ve read Top Reviewer’s reviews and sometimes found them to be too long, too bland, and often containing more bloated speculation on a product than solid feedback for a potential buyer. Yes. Some of them are, partially because the positions have been blind popularity contests for the past year or so.” We agree. U ask (albeit rhetorically) “Addition of the human element is supposed to take care of that, but don't hold your breath. Rhetorically speaking, have you left H or SH ratings, with, of course, an explanatory comment on them?” Rhetorically or not, yes I have, yes I do – and I never regurgitate what others have written. U say, “There's already an appeal process installed: any member who wishes to have a rating changed is free to email asking for a reconsideration. I receive three to five of these requests per day, and almost always get back to the writer immediately -- and almost everyone I know here does the same. Anyone who asks me why I rated something the way I did will get an answer (if it's not already there in a comment); it's up to him or her to decide whether my suggestions are worth the effort or not.” You missed the idea on this one. Accidentally you read it to mean (probably due to the way I wrote it, no doubt) individual ratings, but the idea here is for an appeal process through Epinions on an “overall” rating using a randomly generated seven-member vote. Although I’ve never had someone ask me to change a rating, I will do the same in terms what you’ve said above. U say, “By the by, what the heck does "irreverence at the bottom" have to do with this?” This is a multiple layered concept, but it entails everything we’ve been speaking of. Imagine what I mean, and you’ll probably be right on track with the idea I’m expressing. U say, “A few other things: (from me) Top reviews (ultimately) should stay available (in public, for potential buyers) from non-member visits, not member ratings I don't follow this; will you clarify?” Of course I’ll clarify. What I’m saying is there should be two ‘levels’ for each review. The first level is the initially published review. This level entails the process of a writer getting the maximum 50 ratings from members, with the minimum 30 before it becomes public. During the first (this idea is slightly different from previously expressed, as it’s more detailed) 5-day’s after posting a review (in the private or members area) the writer receives the mandatory ratings from members and edits the review or petitions raters for ratings changes. During the next 5-day period the member is able, if they choose, to petition an Epinions 7-member (randomly generated) ‘board’ on an “overall” rating. The ‘board’ should not have any member that previously rated the review. ADDED: Modification for further clarification added to the paragraph above: The review is then moved from the private area to a public area where potential buyers may view it. OT, NH or SH reviews could possibly be prevented from being published in the public area. U say, (from me) “BUT, we all also have a vested interest in OUR review being the best. If you're saying "we all think OUR opinion is the best," you're right. Of course, all but one of us is wrong in that case, don't you agree? If you're saying "we all earn more if OUR opinion is rated highest," though, you're wrong. If substandard, misleading, incomprehensible, ingrammatical, incomplete reviews become the order of the day, the site has no chance to survive -- in which case, none of us will earn anything from it.” I’m not sure what you’re driving at here. Technically we do earn more if our review is rated the highest. On your idea about poor reviews being the ‘order of the day’, how does that concept relate to this idea I’ve expressed? If anything, this system (that I’ve suggested) would completely prevent what you’re speaking of. Currently if a product has no review and a writer creates one, something that is completely horrible (as you’ve described above) it will be the only review seen. With what I’ve proposed that would be unlikely, if not impossible. BTW, I believe the word you’ve used (“ingrammatical”) should actually be “ungrammatical”. Sorry, any grammatical error that deals with the actual terminology regarding grammar and the like really bugs me, sorry. U say, “On the other hand, your schema for an overhaul of the rating system shows merit. I don't see where it makes the rating process less subjective than it already is -- but it might force people to attempt a quantitative approach. One potential problem is in weighting: is proper grammar more important than being informative? Or does indecipherable grammar and spelling detract so much that you cannot extract the information? In either case, the objectivity of such a system becomes muffled. You might be surprised, by the way, how many people already have a mental system similar to this.” Thanks for the compliment? Many members think the rating system should be overhauled… and I’m assuming you have the same perspective or would at least support a good idea for change. I’ll add you to the list of ‘possibles’ for the “Epinions Coalition for Change”… :} You raise excellent points regarding the ‘weighting’ of category ratings. As far as your comment, “I don't see where it makes the rating process less subjective than it already is -- but it might force people to attempt a quantitative approach.”, you of course answer your own question. That is indeed one way in which it might help to make the system more objective, Kudos! As far as your comment “You might be surprised, by the way, how many people already have a mental system similar to this.”, this of course defeats the purpose of an objective system, is contradictory to the Epinions concept (in some ways) and is (actually) the exact point I’m making. Thanks for bringing it up! :} Steve :}  |
| Read all comments (20)|Write your own comment |
|
Ads by Google
|
