What Good Are Ratings?

[BigRich]

I'm with Chuck on this one too. I remember when Matt and I were starting out website and contemplated how we would review whiskey. For us, it was the false sense of objectivity between a "87" and an "88" that bothered us the most. We use a simple scale of:

Probably Pass

Average for the style

Stands Out

Must Try

Must Buy

Even the "Must Buy" is very rare. We only use it if the drink is exceptional and a good value. For instance Gold Bowmore may taste like heaven but most people can't afford that so we wouldn't give that a must buy.

[dmarkle]

Even the "Must Buy" is very rare. We only use it if the drink is exceptional and a good value. For instance Gold Bowmore may taste like heaven but most people can't afford that so we wouldn't give that a must buy.

Yeah, "Must Buy" is a funny rating. For me, the new OF BIB, while not very complex, is a must buy at $15, but the Pappy 23 isn't a must buy at $200. Maybe that should be split out into "I must buy" and "You must buy for me".

[squire]

Well, for good or ill the numerical rating system is in place and I doubt that will change. As for informed opinions this board provides very good reference points.

[imbibehour]

I prefer a 10 point scale but that's just me.

Everyone has their own rating system internally, so it's kind of hard to bring everyone together.

Not sure how it will all play out.

[PAspirit1]

I like a 100 point system for most things. With a ten point system, for example, you end up saying something like "these are both eights but one is a stronger eight. It's almost a nine but not quite. I wish I could express that."

[kickert]

I like a 100 point system for most things. With a ten point system, for example, you end up saying something like "these are both eights but one is a stronger eight. It's almost a nine but not quite. I wish I could express that."

I use a 10 point scale that is really almost a 100 proof scale. I like to use whole and half numbers, but then when I find an "8 that is a bit better than another 8" I will break out the 8.1 (or 7.9). I never rate under a 5, but that is because I think all bourbon is at least tolerable. I could go down below a 5, but it would have to taste like fingernail polish remover to make it all the way to very bottom of my scale. In reality most of my ratings fall between 7 and 9.5. A bad bottom of the shelfer might get a 6.

[squire]

Essentially the same scale I use Ben.

[cowdery]

Since this has come up again I realized I never explained how the magazines and other major rating outfits do it. This is a 10-point scale but since they allow tenths, just move the decimal point and it's a 100-point scale.

0 -5 FAULTY - There is something technically wrong with the product.

6 - 7 POOR - The product has little character or complexity and lacks balance.

7 - 7.5 AVERAGE - The product is okay but nothing special.

7.6 - 7.9 GOOD - The balance is good and there are elements of complexity.

8 - 9 VERY GOOD - The product is well balanced and complex.

9+ EXCEPTIONAL - The product is very complex, deep and rich, with lots of character.

Note that 0-7 is products that probably shouldn't be sold let alone entered into contests. 7 is 'acceptable.' Only above 8 does it get competitive. That's the space where most competitions play. Nothing below 7, only a few there, everything else 8 to 10. But there are no tens because nothing is perfect.

Usually when you are asked to rate, say, customer service in a survey, 5 is considered 'acceptable.' Anything below a 5 is considered some degree of less-than acceptable, anything above a 5 is considered some degree of exceptional. Below average, better than average.

Why doesn't it work that way with beverages? The rationale is that, indeed, anything less than 5 is unacceptable, and while 5 to 8 might be acceptable, only 8 to 10 is award-worthy. The purpose of the competition is to determine which of the award-worthy products is best.

I'm neither attacking this system nor defending it, just explaining it.

[macdeffe]

Is one scale better than others. I am not sure.

Personally I have chosen to use a scale that goes like this :

5 Made me cry

4 Superb Whisky

3 Good Whisky

2 Average (Forgetable?)

1 Flawed

0 Made me cry!

This is similar to a lot of other scales mentioned here, the wording is just different.

The reason I use this scale is that I don't find my consistency and skills good enough to grade on a 100p scale. So I have chosen to use a scale that suits me. If others like to use a 100p scale, I don't have any problems with that

Whatever scale others are using its very easy to spot when they consider a whisk(e)y to be good, and I always need a to read some reviews to calibrate my tastes toward the reviewer

Steffen

[Special Reserve]

Is one scale better than others. I am not sure.

Personally I have chosen to use a scale that goes like this :

5 Made me cry

4 Superb Whisky

3 Good Whisky

2 Average (Forgetable?)

1 Flawed

0 Made me cry!

This is similar to a lot of other scales mentioned here, the wording is just different.

The reason I use this scale is that I don't find my consistency and skills good enough to grade on a 100p scale. So I have chosen to use a scale that suits me. If others like to use a 100p scale, I don't have any problems with that

Whatever scale others are using its very easy to spot when they consider a whisk(e)y to be good, and I always need a to read some reviews to calibrate my tastes toward the reviewer

Steffen

Steffen,

I love your scale. Particularly the made me cry at both ends.

Will

[cowdery]

My point is that most people, including most of the people who have posted their own scales here, put 'average' in the middle and flow out from there. In the 10-point scale used by competitions, 'average' is about a 7. That is the disconnect.

[kickert]

My point is that most people, including most of the people who have posted their own scales here, put 'average' in the middle and flow out from there. In the 10-point scale used by competitions, 'average' is about a 7. That is the disconnect.

That all depends on what you are averaging. If you are averaging possible products then 5 would be in the middle. 10 would be a honey barrel of 15 yo SW uncut and unfiltered, 1 (or 0) would be a bottle of foreshots straight off the still of a sugar water run and 5 would be a bourbon, but a really awful one.

However, if you average the scores of all available product, then 7 is probably a pretty accurate assessment. After all, just because there can be worse products, doesn't mean they are actually on the market. Because those are not being sold, the entire average creeps up.

[Special Reserve]

IIRC, the ratings that I've seen on a one hundred scale range from the 70's to about 99. With the lowest ratings not for mid-shelf products but for lower shelved products.

The ratings done by the professionals (or at least those being paid for their ratings) the effective scale is 70 to 100.

[flintlock]

The ratings done by the professionals (or at least those being paid for their ratings) the effective scale is 70 to 100.

[p_elliott]

I bet this is to be able to give the maximum number of advertiser's products as high a rating as possible. Rating one whiskey a 93 and one a 94 isn't likely to anger either one. Rating one a 4 and one a 5 is going to get you a phone call. "Yeah, money's tight this year...advertising dollars might have to be cut..."

But then, I'm cynical...

[p_elliott]

Ryan

In addition to your post let me relay a story to you: Chuck Cowdery was asked to do a whiskey judging competition. It got to the Rye Whiskey division and Chuck was out voted and High West I think Rendezvous won. Chuck had a melt down how can a rectifier win best distiller? Good Question Chuck! They argued back and forth and the regulations of the contest were checked. High west won the competition but the rules may be rewritten for next year. Your not going to piss in Chucks face and tell him it's raining, he's not going to buy it.

Paul

[JohnHansell]

I don't think John Hansel would go for this, I think John tells it like it is.

Chuck isn't opposed to stepping on some toes either read some of his post here. He has flat out pissed off some rectifiers.

[OscarV]

And we have lost a lot more advertising because of our ratings than we ever got because of our ratings.

[flintlock]

I should have prefaced my comments by saying that I meant that, across all sorts of rated products, a 100 point system seems to simply allow more room for products to be in the top 10. Whether it's wine, cigars, wristwatches, or ice cream...an article with a bunch of 4's and 5's wouldn't give the same sense of getting "vital statistics". I think numerical ratings are pointless, personally, but people expect them.

I certainly didn't mean to imply that John or Chuck ever allowed advertisers to sway their views on things, and I hope it didn't sound like that. It was really more of a comment on the post-Wine Spectator/Cigar Afficionado need to rate everything under the sun, 1 - 100.

[squire]

I would rate an average whisky between 5-6 but haven't found any that I would describe as awful.

[T Comp]

For those that don't subscribe to Mr. Cowdery's Bourbon Country Reader you should because his recent review of Angel's Envy and WhistlePig (Vol. 13, No.3) is a terrific demonstration why ratings are not necessary. The review is more informing than any number or grade could ever be and the absence of one intrigues a potential buyer even more.

[Lucas Jackson]

My point is that most people, including most of the people who have posted their own scales here, put 'average' in the middle and flow out from there. In the 10-point scale used by competitions, 'average' is about a 7. That is the disconnect.

Someone mentioned a report-card style rating F,D,C,B,A. C at 70 being average and leaving 0-69 for failing. I would agree that 'average' is about 7, not in the 'middle'. It seems illogical, but seems to be accepted as a norm. Perhaps once it reaches 'bad' it doesn't matter to most if it makes it all the way to 'awful'

I think, as many have observed, knowing the rater's 'tastes' is most important. Diluting the (my) need for a tangible 'X', 'Y' rating.

Great topic, got my wheels turning.

[kickert]

Someone mentioned a report-card style rating F,D,C,B,A. C at 70 being average and leaving 0-69 for failing. I would agree that 'average' is about 7, not in the 'middle'. It seems illogical, but seems to be accepted as a norm. Perhaps once it reaches 'bad' it doesn't matter to most if it makes it all the way to 'awful'

I think, as many have observed, knowing the rater's 'tastes' is most important. Diluting the (my) need for a tangible 'X', 'Y' rating.

Great topic, got my wheels turning.

I agree with you Lucus. An average or 7 (of 70) makes sense and I like the grading analogy. If in class you miss every other question you get a 50 and you fail. We don't think "average" mastery is the middle of knowing it all and knowing nothing. Average is a judgement of what is out there based on what is possible.

[cigarnv]

Over the years I have paid little attention to the "experts" who review spirits, wines, beers, cigars, etc. All of them have a bias, IMO, as we all do regardless of if we choose to admit it or not. In many cases we don't even realize we have a bias which can be driven by a single exposure... good or bad to a product, person or organization.

On the flip side I find that blind tastings among knowledgeable consumers provides one the best and most honest evaluations of a product. A tasting group of 12 reasonably knowledgeable individuals each tasting 2oz. of "blinded" spirit over several days with no results shown until all are in will IMO produce about as honest a rating as one can hope to get.

In the end does it really matter who makes a product or where it comes from...... isn't the objective to find something that just tastes damn good....??

Just an opinion, nothing more....

[squire]

Yep, pretty much, it's what's in the bottle that counts.