What Good Are Ratings?

[cowdery]

I'm on record, in my book and elsewhere, as not thinking very much of rating systems. Their basic flaw is that they give a sheen of objectivity to something that is inherently subjective.

I had a long conversation about this with Jim Murray once and his conclusion was, "we owe it to people to give them some kind of guidance."

That point I concede. Therefore, I'm comfortable with a four or five point scale along these lines:

0 = bad

1 = barely acceptable

2 = good

3 = very good

4 = great

My problem with the 100 point systems is exactly what comes up here from time to time. "How can X get a 96 when Y gets a 95?" Consequently, one of the rules of my proposed system is no fractions, no 3 1/2 stars. The idea is to rate in a broad way without ranking.

Subjectivity is still an issue as it always will be. The best bet for someone who wants to use anyone's ratings for guidance is to get to know the tastes of different critics. Either find one who generally agrees with you or just evaluate all of them based on what they seem to like and dislike.

Anyway, I don't want to write a long essay here. I just mean to open the discussion.

[BBQ+Bourbon]

Great subject, Chuck.

I wonder what happens when the authors who use the rating system encounter a subject that recalibrates their point system. It's useful in that it gives that persons relative rankings of the respective subjects at the time of writing, but that's only useful if that persons' tastes jive with the readers'.

I decide what to buy based on what I read on this board. If there are a good number of 'yea' responses to a new bottle, I will buy one and try it. A few dissenters won't discourage me unless their tastes and mine have jived in the past, AND the bottle is expensive enough to make me think twice.

[cowdery]

Similarly, what if the ratings are from a tasting panel and the panel members change?

[Virus_Of_Life]

I'd completely agree that 0-4 is a much more reasonable way of rating Whiskey. As you've said before, nothing ever gets a 42.

[Joshua]

I like 0-4 as well. When rating my bourbon in my database, I frequently end up doing "Well, I would say 97, but this got 98 and I like the 97 better, but the other 97 isn't as good, 99 is too high, wait this is dumb." Then I put down a number in the general ballpark. 0-100 makes things way too tough. 0-4 seems really easy though.

The bickering about if WLW 07 or 08 is better, or if the 09 higher proof makes it best gets silly. It boils down to preference at some point and people end up splitting hairs.

[dmarkle]

How about "by occasion" instead? Here are some samples...

0 = I'm getting divorced. She's taking everything. (Early Times)

1 = I'm degreasing engine parts or mixing with Coke (Jim Beam White Label)

2 = I'm home from work and it's time to take the edge off (newer HH Old Fitz BIB)

3 = I'm celebrating the end of a fine week, perhaps with another whiskey lover in attendance (BTAC)

4 = I'm proving that America makes the best whiskey in the world, and here's why, so shove it, Scotch snobs! (Pappy 20. JRPS., etc)

[fishnbowljoe]

I like my rating system. On my spreadsheet of bourbons I have, and/or have had, in the last column I enter my feelings as to whether or not I would buy a whiskey again. Looks something like this.

BUY AGAIN

NO

PROBABLY NOT

MAYBE

PROBABLY

YES

DEFINITELY

I guess it's a variation of a four point rating system. Just add a couple more points. :grin: At the very least, it gives me a reference point to work with in case I revisit, or run across a bottle of whiskey I haven't had or seen in a while. Cheers! Joe

[dmarkle]

I like my rating system....

[fishnbowljoe]

You know, today at the store when I was perusing the shelves, I kept thinking of your tagline, and I bought two Weller 107's.

[DeanSheen]

I like your system Chuck but it makes too much sense to be implemented.

[mrviognier]

While tasting descriptors and qualitative assessment of any beverage is absolutely a subjective affair, we - as Americans - are fairly obsessed with ratings. Many people won't see a movie that doesn't get two thumbs up, won't eat dinner at an establishment that doesn't earn at least two stars, and won't buy a wine that doesn't garner at least a 90-point rating from Parker or Spectator.

Distilled spirits, I'm happy to say, seems to be less dominated by ratings and the consumer demand for them. I think this is because spirits are so brand-driven. That said, if you're aim is to provide a critical review of Bourbon (or anything else), you need to provide the reader with some means of assessing your qualitative reaction to that which you're reviewing. This need isn't so much to provide the reader with an 100% accurate/objective assessment, but to allow the reader a means to calibrate their palate to yours, the reviewer.

Take, as an example, the single-most influential critic out there: wine critic Robert Parker. Having known (and read) him for 20+ years, I've come to know that what 'rocks his world' are wines that are highly-extracted, with relatively high in pH, alcohol and that have a touch of residual sugar. So, when I see a "95-100" point rating from Bob, I know the wine reviewed is going to exhibit this style. If that's the style of wine I'm looking for, chances are I'll like this wine. Conversely, if I like leaner, higher-acid wines, it's a good bet to stay away from this Parker darling.

[nor02lei]

Rating by points seem ok to me as it the most concise way of telling how much you like a whiskey. I can agree though that 100 points is far too much, but on the other hand 4 seem a little rough to me.

A few years ago I had a prescription of whisky magazine and they had a 10-point scale with quarter as the lowest step. That meant 40 steps theoretically but only about 20 in practice. I thought that was a reasonably way of rating.

I do personally go almost exclusively from what I read here on the forum.

Leif

[rocky480]

BUY AGAIN

NO

PROBABLY NOT

MAYBE

PROBABLY

YES

DEFINITELY

For my personal ratings, I like this system. I use this for cigars since I used to try a wide variety and applied it to bourbon.

In general, I'll look for reviews or ratings of more expensive bottles before making the leap to purchase simply because I'm trying to figure out if it's worth spending the money. From that standpoint, I think the ratings can be helpful, but they can't be taken as absolute given the inherent subjectivity of taste.

Mike

[callmeox]

I recall having the same discussion here on SB a while back and my binary rating system in use back then has not changed.

0 - Don't buy it again

1 - Buy it again

[Special Reserve]

I like my rating system. On my spreadsheet of bourbons I have, and/or have had, in the last column I enter my feelings as to whether or not I would buy a whiskey again. Looks something like this.

BUY AGAIN

NO

PROBABLY NOT

MAYBE

PROBABLY

YES

DEFINITELY

I guess it's a variation of a four point rating system. Just add a couple more points. :grin: At the very least, it gives me a reference point to work with in case I revisit, or run across a bottle of whiskey I haven't had or seen in a while. Cheers! Joe

I'm less indecisive with the grading. The purchasing categories are more like:

No

OK

Sure

Absolutely

The standard are price and availability adjusted.

Good thread Chuck, made even better by the quality of replies.

[pepcycle]

I think we should have the Serve It To Scale

The low end of the scale is: Bring to a party of strangers and willing to leave it.

The high end of the scale: Put on the table at the Gazebo and lay claim to it.

Sort of an audience worthiness scale.

The second lowest point on the scale would be: Serve to posers who claim they only like one whisky and can pick it out blind.

What other levels might fit?

Ex wife?

Inlaws?

[mrviognier]

How 'bout:

- Wouldn't be stranded on an island without a barrel of it;

- Wouldn't object to being served it;

- Wouldn't drink it unless it was the only Bourbon on the bar; and,

- Wouldn't clean my garage floor with it.

[Trace Tippler]

I am pleased to have read Chuck Cowdery's book early on in my 'bourbon career" and find much to agree with in his assessment of the uselessness of these 100-pt rating systems. I took his basic system and expanded it. At the top of my tasting notes notebook I made up is a way for me to remind myself of the overall experience of a particular bottle without assigning numbers. Mine goes like this:

-Undrinkable

-Bad

-Okay

-Good

-Very Good

-Outstanding

-Sublime

The first may mean I really don't like something, there is a cork problem, a bad batch or whatever. Bad means I don't care for it as it's made but someone else might. The "ok, good, very good and outstanding" should make sense by themselves while the last one, "sublime" is something I'm looking forward to when I have more experience. Those sublime moments, when words fail completely and it's too incredible to think about but rather just enjoy it while it lasts. Based on what I've tasted so far, I know this is possible with bourbon.

Cordially,

Trace T.

[PaulO]

I was thinking Chuck's 0-4 system made a lot of sense to me. It seemed very close to about the number of categories I would place bourbons or beer brands into. Then it came to me; 0, 1, 2, 3, 4 correspond perfectly to F, D, C, B, A. Then we can get into plus and minus if you like.

[StraightBoston]

In my bourbon inventory I rate both taste and value on a three-point scale (+/o/-) corresponding to better-than-average, average, worse-than-average -- where "average" corresponds roughly to Jim Beam Black.

[Res/st-or]

The 0-100 scale must make sense to the retail industry. if an 86 turns to be not such a good bourbon, at least the number is high enough to confuse consumers to buy it anyway. Nobody would want to be associated with a 0 or 1 on Chuck's scale.

I have been confused by this as well, Chuck brought this to my attention, and I've thought about it since I first read this thread. I am glad to be involved with this site. I learn something new every day.

I think Chuck's scale makes sense. It is simple and precise. Retailers love to confuse, keeping us in the gray area. Gray = green to them. Or so I believe.

[cowdery]

It's Lake Wobegon marketing, where all of the bourbons are above average.

[ILLfarmboy]

In my bourbon inventory I rate both taste and value on a three-point scale (+/o/-) corresponding to better-than-average, average, worse-than-average -- where "average" corresponds roughly to Jim Beam Black.

I like this idea. Both taste and value are important. A rating system that includes both but keeps them seperate allows the consumer to decide for himself. Case in point, I don't care if somethging gets a 4 out of 4, or a 97 out od 100, if it costs a $80+, I usually won't buy it, or if I do, its not something I'm gonna stock up on.

[George]

I like the 0-4 scale very much.

[sailor22]

This need isn't so much to provide the reader with an 100% accurate/objective assessment, but to allow the reader a means to calibrate their palate to yours, the reviewer.

This has always been my understanding of ratings. I can usually shush out the relationship between the reviewer and my palate with the 100 point system, even if the Lake Woebegone effect is in play. Granted, it takes some time to understand the reviewers preferences.

The 0-4 scale makes perfect sense and should a huge help in making generalized buying decisions. I might modify it just a bit for my own use;

0 = bad

1 = barely acceptable

2 = good

3 = very good

4 = special and worthy of relaxed contemplation