What Good Are Ratings?

[p_elliott]

I'm on record, in my book and elsewhere, as not thinking very much of rating systems. Their basic flaw is that they give a sheen of objectivity to something that is inherently subjective.

I had a long conversation about this with Jim Murray once and his conclusion was, "we owe it to people to give them some kind of guidance."

That point I concede. Therefore, I'm comfortable with a four or five point scale along these lines:

0 = bad

1 = barely acceptable

2 = good

3 = very good

4 = great

My problem with the 100 point systems is exactly what comes up here from time to time. "How can X get a 96 when Y gets a 95?" Consequently, one of the rules of my proposed system is no fractions, no 3 1/2 stars. The idea is to rate in a broad way without ranking.

Subjectivity is still an issue as it always will be. The best bet for someone who wants to use anyone's ratings for guidance is to get to know the tastes of different critics. Either find one who generally agrees with you or just evaluate all of them based on what they seem to like and dislike.

Anyway, I don't want to write a long essay here. I just mean to open the discussion.

Although I agree with your point in general it doesn't solve anything. Everything is going to get rated in blogs and magazines as being 3 or better with very few 3's. We're right back where we started. To be honest I have yet to see whiskey writer/reviewer come out and say "This product sucks don't buy it" No offense meant to Chuck or John.

[BourbonJoe]

To be honest I have yet to see whiskey writer/reviewer come out and say "This product sucks don't buy it" No offense meant to Chuck or John.

Never bite the hand that feeds you.

Joe :usflag:

[callmeox]

While they may not say those words exactly due to the lack of professionalism required to dip that low, if you peruse the whiskey ratings on MA, you will find low scores that lead you to the same conclusion.

[CorvallisCracker]

To be honest I have yet to see whiskey writer/reviewer come out and say "This product sucks don't buy it" No offense meant to Chuck or John.

When, in 2007, I encountered the lineup of Hirsch Canadian Rye whiskies at a local liquor store (8, 10 and 12yo expressions), I checked John's review, which stated, "They’re not good enough to drink neat or on the rocks and too expensive to drink as a mixer", giving them scores of 75, 72 and 70, which was enough to persuade me not to buy any.

For my own purposes I use a 0-5:

0. Sucks

1. Doesn't suck

2. Good

3. Very Good

4. Excellent

5. Memorable (as in, I'll remember it for the rest of my life.

Very few things have ever gotten this last one. In bourbon, it's only been

VSOF S-W produced

WT 12 yo (this version and the CGF)

Hirsch 16 BW

OFBB 2007 release

PVW 20

In red wine, only

1982 Ch. Pichon-Lalande (six bottles consumed from 1992 to 2002)

1986 Ch. Lafite-Rothschild (magnum opened 2006)

1986 Penfolds Grange Hermitage (opened 2006)

1987 Modavi Reserve Cabernet at release (lost complexity as it aged)

and in the course of the last 30 years, I've sampled thousands of red wines.

Price factors in on whether I recommend it. A "doesn't suck" wine costing $3 is Recommended. An "excellent" one costing $150 is not, because you can get wines that good for a lot less.

[callmeox]

John's ratings are a more granular version of the 0-4 scale so I don't believe that his system is really too far from Chuck's.

Going somewhat by memory here. Forgive any detail issues.

60-69 (0) "avoid" with two expressions listed

70-79 (1) "average" around 60 listings

80-89 (2) too many to count on my phone

90-94 (3) many here as well

95-100 (4) less than 50 expressions

As one wouldlikely assume, the vast majority of bottlings fall in the 2-3 range with the far ends of the scale populated by the best and the worst out there. Is is a bell curve type distribution? I'm too lazy to key them all in to find out, but it looks like it would be close.

One can dispute the second lowest range being "average" as that's where the majority should appear, but his intent is not to force a distribution across the entire range, but to grade each expression as a single unit.

[cowdery]

Hansell and I have both panned things. John was very hard on Old Crow Reserve and has been hard on the most recent WRMC releases.

Since I don't necessarily try to review everything I sometimes pan with my silence. I have also been charged a few times of "damning with faint praise."

The problem isn't really with the reviewers. As many have noted, the right way to use critics is to learn their tastes and calibrate them to your own, regardless of the rating system they use. My objection is to the sheep who walk into a store and buy whatever has the highest number. I think the 100-point type systems enable that behavior.

[OscarV]

In an earlier post it was mentioned that they rate with "value" in mind.

I think a 1 to 5 scale is best with absolutley no consideration of the price.

My palate knows nothing of my financial condition and I only want it to tell me what it taste like.

[bourbon-n00b]

Since I don't necessarily try to review everything I sometimes pan with my silence.

[BourbonJoe]

I use ratings, any ratings, to determine if I'm going to try something. If I like it, I'll load up, regardless of price. If Hansell or Cowdery, or anybody else rates something low (very unusual), even if i'm not tuned to their taste, I will not try it (unless somebody puts it out at the gazebo, for instance). Life is too short for mediocre bourbon (at least in my case).

Joe :usflag:

[jburlowski]

... To be honest I have yet to see whiskey writer/reviewer come out and say "This product sucks don't buy it" ...quote]

F. Paul Pacult can be brutally (and entertainingly) honest is his reviews (Spirit Journal). One of the main reasons I enjoy reading them.

A few examples chosen at random:

"... What am I, a crash test dummy? This is nothing more than clumsy production by somebody who doesn't have a clue about flavor subtlety or nuance. Avoid like herpes."

"... In the mouth the entry at first is salty, then it turns unpleasantly bitter and astringent on the tongue; the midpalate is no better as the baked, industrial taste goes sour and extremely astringent. The finish is tanky and sufury. Terrible to the point of being undrinkable."

He uses a five star system: Not Recommended (one or two stars); Recommended (three stars); Highly Recommended (four stars); and Highest Recommendation (five stars).

[ILLfarmboy]

Salty with a sulfury finish. Sounds like an Islay malt he didn't like. I can't imagine any bourbon being that bad.:grin:

[cowdery]

Could you elaborate on this a bit, in terms of reader usability? How would we best be able to determine a "didnt review because I didn't like it well enough" from a "didn't review because I haven't got around to it, dont have a sample handy, etc"?

Not really. Sometimes I didn't get around to it because I didn't like it, or I just didn't have anything to say about it. If there's something widely heralded and I don't like it I won't stay quiet, but sometimes something just doesn't interest me enough to get a sample let alone write about it.

I also don't actually use the scale I'm proposing, but if I did use a rating system it would be something like that. When I do review something you actually have to read the review to see what I think. No shortcuts.

Some people try to write a few words about everything that comes along. Good for them. I can't drink that much. That's not a judgment. Reviewing everything just isn't my platform.

I also don't go out of my way to beat up on something you probably won't buy anyway.

I've stated many times that I tend not to care about something if I don't know who made it, which knocks out a lot of non-distiller products.

[cowdery]

This website for Ridgemont Reserve 1792, minimal as it is, illustrates some of what we were talking about in this thread. Look in the lower left hand corner. Accordiing to the Beverage Testing Institute, 1792 is a 93, ranking above the brands 1792 considers its competitors: Knob Creek, Woodford Reserve (both 90) and Gentleman Jack (82).

I only just discovered this "under construction" website, but the copyright date is 2009, so Sazerac obviously isn't in any hurry to add to it.

I was prompted to look at it by a press release. Apparently Ken Pierce is now the point man there, though he is titled as Chief Chemist and Brand Ambassador, not Master Distiller.

[doubleblank]

As Chuck stated in his first post, finding "critics" with preferences that correlate with yours is the best way to use their written reviews and/or scores. Robert Parker often gets hammered by wine industry people/other critics for his percieved love of full bodied balls-to-the-wall wines. I was a long time subscriber to his publication and found his tastes and mine were fairly well aligned and used his notes with much success to find great wines (often at low prices too). The wine universe is so large, some guidance can be helpful in finding wines that we would never ever taste on our own.

But the bourbon world is so much smaller that I (and I imagine 99% of the people on this board) don't need a critic's opinion as to whether to buy something or not. I know I'll get around to tasting almost everything out there at a Gazebo, bar, fellow SB'ers, distillery tour, etc and form my own opinion from there.

By the way, did you know 50% of all bourbons are below average!

Randy

[OscarV]

By the way, did you know 50% of all bourbons are below average!

[doubleblank]

I was just using the definition of "average".....50% are above and 50% are below, by definition.

I recall a south Texas Democrat ranting about how half of the kids in his district scored below average and we needed to do something about that! "More taxes and more school spending" was his solution. That might increase the kids' knowledge, but 50% are still below average!

Randy

[callmeox]

I think that 49 percent are below average and 1% are just average.

Don't get me started about the rest.

[OscarV]

I recall a south Texas Democrat ranting about how half of the kids in his district scored below average and we needed to do something about that! "More taxes and more school spending" was his solution. That might increase the kids' knowledge, but 50% are still below average!

Way way off topic here, but you failed to mention Texas Republicans wanting to rewrite history text books to fit their current fundamentalist Christian idealolgy.

Which is OK except for the fact these books get sold across the entire USA because Texas is such a huge market the book manufacturers sell them elsewhere uncorrected.

True, Democrats answer to everything is to pile more money on it.

Republicans answers are to deny the facts.

Democrats hearts are in the right place but not their brain.

Republicans have neither a heart or a brain.

OK, now back on topic, if you were to take your favorite bourbons and factor them into a percentile, what percent would that be of all bourbons?

Mine would be very very small, like around a few percentage.

[imbibehour]

THe 100 point scale I find just a bit too broad. For me ratings in general are kind of a starting point, but since not everyone has the same palate I don't get wrapped up in them.

I think actually the simpler scale that you offered if it were put into use gives less room for people to have opinions since there's not much room to have a difference between say... 4 and 5.

In the end... they are all just numbers really and that's all I look at them as.

[jburlowski]

I was just using the definition of "average".....50% are above and 50% are below, by definition.

Randy

I beleive you are refering to median, not average.

[marco246]

I've come up with a five star rating system which I use to refresh my memory when it comes time to add to my stock of bourbons. No pluses or minuses except for at the three-star level. There is only ONE three-star rated bourbon, and for me that is the benchmark. Everything else is above or below. For me the benchmark bourbon is Maker's Mark. Here's an example of how I rate stuff:

* Old Fitzgerald's 1849

** Bulleit Frontier

***- Blanton's Single Barrel

*** Maker's Mark

***+ Knob Creek

**** Four Roses Single Barrel, Lim Ed 2009, OESQ

***** Pappy Van Winkel's Family Reserve, 20 yo

What this means in practice is that I'll not seek out a bourbon rated below Maker's Mark unless for some special reason. Natually this rating system is completely subjective, but it is useful for my purposes. Price is not a factor in the ratings.

[cowdery]

One of the main reasons I hate ratings is because they are a crutch for lazy people looking for a short cut. They are looking for a short cut because they have no base of knowledge and no interest in obtaining one. People who buy-by-the-numbers are people who do everything by the short cut route and who are looking for social validation more than they are a great bottle of whiskey.

As a writer, I hate the idea that people don't even read the reviews, they just look at the numbers. Ideally, a critic will give you enough information to make up your own mind but you as the reader have to do some work too. You have to think. You don't need to be a writer to pull a two-digit number out of the air and you don't need to think to turn those numbers into a shopping list.

The people who buy-by-the-numbers wouldn't recognize "brine and spice, apple pip, and traces of aniseed"* if it bit them.

I don't blame or condemn the writers, publications, and entities like BTI that give ratings. They are forced to do it because they depend on advertising revenue or (in the case of BTI) fees to stay in business. The producers would scream bloody murder if any of those entities switched to the type of "good-better-best" ranking system we've been discussing here.

The sad reality is that their 93 will sell more bottles of 1792 than positive reviews by me and every other writer combined. And selling more bottles is what the producers are in business to do.

For the people who give ratings the pointlessness of it doesn't harm their credibility as long as they're running an honest game and so far as I know everyone is. That it's a silly and meaningless game is beside the point as long as people buy-by-the-numbers. The only harm it does is put great bottles of whiskey into the cabinets of people who manifestly do not deserve them.

As long as there are people with money to spend who believe the ratings mean something there will be ratings.

* From Dominic Roskrow's Guest Review of Caol Ila, 25 Year Old on "What Does John Know." He gave it an 88.

[doubleblank]

An average, in general, approximates an arithmetic mean IIRC.

Randy

[ILLfarmboy]

One of the main reasons I hate ratings is because they are a crutch for lazy people looking for a short cut. They are looking for a short cut because they have no base of knowledge and no interest in obtaining one. People who buy-by-the-numbers are people who do everything by the short cut route and who are looking for social validation more than they are a great bottle of whiskey.

There are plenty of people like that, mostly stuck up snobs with no real interest in whiskey, or who think of the whiskey in their glass as a fashion accessory. But there are also plenty of us enthusiasts who have a limited funds. And some of us don't live in KY or a 20 min. drive from Binny's. If something new and interesting comes out, I'd like to know what experienced palates think about it before I spend $$ and more money on shipping. ..........And I'd like to know how good a value it is.

[mrviognier]

I actually like using a 100-point system better than other options. In utilizing stars, 5-points, etc. there's really not much room to assign a rating which allows for much subtlety within a peer group. The difference between a "***" and a "****" is probably wider than, say, a "86" and a "89".