PDA

View Full Version : What Good Are Ratings?



cowdery
08-07-2010, 13:56
I'm on record, in my book (http://cowdery.home.netcom.com/bourst.html) and elsewhere, as not thinking very much of rating systems. Their basic flaw is that they give a sheen of objectivity to something that is inherently subjective.

I had a long conversation about this with Jim Murray once and his conclusion was, "we owe it to people to give them some kind of guidance."

That point I concede. Therefore, I'm comfortable with a four or five point scale along these lines:

0 = bad
1 = barely acceptable
2 = good
3 = very good
4 = great

My problem with the 100 point systems is exactly what comes up here from time to time. "How can X get a 96 when Y gets a 95?" Consequently, one of the rules of my proposed system is no fractions, no 3 1/2 stars. The idea is to rate in a broad way without ranking.

Subjectivity is still an issue as it always will be. The best bet for someone who wants to use anyone's ratings for guidance is to get to know the tastes of different critics. Either find one who generally agrees with you or just evaluate all of them based on what they seem to like and dislike.

Anyway, I don't want to write a long essay here. I just mean to open the discussion.

BBQ+Bourbon
08-07-2010, 14:22
Great subject, Chuck.

I wonder what happens when the authors who use the rating system encounter a subject that recalibrates their point system. It's useful in that it gives that persons relative rankings of the respective subjects at the time of writing, but that's only useful if that persons' tastes jive with the readers'.

I decide what to buy based on what I read on this board. If there are a good number of 'yea' responses to a new bottle, I will buy one and try it. A few dissenters won't discourage me unless their tastes and mine have jived in the past, AND the bottle is expensive enough to make me think twice.

cowdery
08-07-2010, 14:29
Similarly, what if the ratings are from a tasting panel and the panel members change?

Virus_Of_Life
08-07-2010, 14:33
I'd completely agree that 0-4 is a much more reasonable way of rating Whiskey. As you've said before, nothing ever gets a 42.

Joshua
08-07-2010, 17:28
I like 0-4 as well. When rating my bourbon in my database, I frequently end up doing "Well, I would say 97, but this got 98 and I like the 97 better, but the other 97 isn't as good, 99 is too high, wait this is dumb." Then I put down a number in the general ballpark. 0-100 makes things way too tough. 0-4 seems really easy though.

The bickering about if WLW 07 or 08 is better, or if the 09 higher proof makes it best gets silly. It boils down to preference at some point and people end up splitting hairs.

dmarkle
08-07-2010, 18:47
How about "by occasion" instead? Here are some samples...

0 = I'm getting divorced. She's taking everything. (Early Times)
1 = I'm degreasing engine parts or mixing with Coke (Jim Beam White Label)
2 = I'm home from work and it's time to take the edge off (newer HH Old Fitz BIB)
3 = I'm celebrating the end of a fine week, perhaps with another whiskey lover in attendance (BTAC)
4 = I'm proving that America makes the best whiskey in the world, and here's why, so shove it, Scotch snobs! (Pappy 20. JRPS., etc)

fishnbowljoe
08-07-2010, 19:03
I like my rating system. On my spreadsheet of bourbons I have, and/or have had, in the last column I enter my feelings as to whether or not I would buy a whiskey again. Looks something like this.


BUY AGAIN

NO
PROBABLY NOT
MAYBE
PROBABLY
YES
DEFINITELY


I guess it's a variation of a four point rating system. Just add a couple more points. :grin: At the very least, it gives me a reference point to work with in case I revisit, or run across a bottle of whiskey I haven't had or seen in a while. Cheers! Joe

dmarkle
08-07-2010, 19:08
I like my rating system....

You know, today at the store when I was perusing the shelves, I kept thinking of your tagline, and I bought two Weller 107's.

fishnbowljoe
08-07-2010, 19:37
You know, today at the store when I was perusing the shelves, I kept thinking of your tagline, and I bought two Weller 107's.

Cool. :grin: At least I'm good for something. :skep: Joe

DeanSheen
08-07-2010, 20:31
I like your system Chuck but it makes too much sense to be implemented.

mrviognier
08-07-2010, 20:33
While tasting descriptors and qualitative assessment of any beverage is absolutely a subjective affair, we - as Americans - are fairly obsessed with ratings. Many people won't see a movie that doesn't get two thumbs up, won't eat dinner at an establishment that doesn't earn at least two stars, and won't buy a wine that doesn't garner at least a 90-point rating from Parker or Spectator.

Distilled spirits, I'm happy to say, seems to be less dominated by ratings and the consumer demand for them. I think this is because spirits are so brand-driven. That said, if you're aim is to provide a critical review of Bourbon (or anything else), you need to provide the reader with some means of assessing your qualitative reaction to that which you're reviewing. This need isn't so much to provide the reader with an 100% accurate/objective assessment, but to allow the reader a means to calibrate their palate to yours, the reviewer.

Take, as an example, the single-most influential critic out there: wine critic Robert Parker. Having known (and read) him for 20+ years, I've come to know that what 'rocks his world' are wines that are highly-extracted, with relatively high in pH, alcohol and that have a touch of residual sugar. So, when I see a "95-100" point rating from Bob, I know the wine reviewed is going to exhibit this style. If that's the style of wine I'm looking for, chances are I'll like this wine. Conversely, if I like leaner, higher-acid wines, it's a good bet to stay away from this Parker darling.

nor02lei
08-08-2010, 01:41
Rating by points seem ok to me as it the most concise way of telling how much you like a whiskey. I can agree though that 100 points is far too much, but on the other hand 4 seem a little rough to me.
A few years ago I had a prescription of whisky magazine and they had a 10-point scale with quarter as the lowest step. That meant 40 steps theoretically but only about 20 in practice. I thought that was a reasonably way of rating.
I do personally go almost exclusively from what I read here on the forum.

Leif

rocky480
08-08-2010, 09:29
BUY AGAIN

NO
PROBABLY NOT
MAYBE
PROBABLY
YES
DEFINITELY


For my personal ratings, I like this system. I use this for cigars since I used to try a wide variety and applied it to bourbon.

In general, I'll look for reviews or ratings of more expensive bottles before making the leap to purchase simply because I'm trying to figure out if it's worth spending the money. From that standpoint, I think the ratings can be helpful, but they can't be taken as absolute given the inherent subjectivity of taste.

Mike

callmeox
08-08-2010, 09:52
I recall having the same discussion here on SB a while back and my binary rating system in use back then has not changed.

0 - Don't buy it again
1 - Buy it again

Special Reserve
08-08-2010, 09:52
I like my rating system. On my spreadsheet of bourbons I have, and/or have had, in the last column I enter my feelings as to whether or not I would buy a whiskey again. Looks something like this.


BUY AGAIN

NO
PROBABLY NOT
MAYBE
PROBABLY
YES
DEFINITELY


I guess it's a variation of a four point rating system. Just add a couple more points. :grin: At the very least, it gives me a reference point to work with in case I revisit, or run across a bottle of whiskey I haven't had or seen in a while. Cheers! Joe

I'm less indecisive with the grading. The purchasing categories are more like:

No
OK
Sure
Absolutely

The standard are price and availability adjusted.

Good thread Chuck, made even better by the quality of replies.

pepcycle
08-08-2010, 09:54
I think we should have the Serve It To Scale

The low end of the scale is: Bring to a party of strangers and willing to leave it.

The high end of the scale: Put on the table at the Gazebo and lay claim to it.

Sort of an audience worthiness scale.

The second lowest point on the scale would be: Serve to posers who claim they only like one whisky and can pick it out blind.

What other levels might fit?

Ex wife?
Inlaws?

mrviognier
08-08-2010, 11:40
How 'bout:

- Wouldn't be stranded on an island without a barrel of it;
- Wouldn't object to being served it;
- Wouldn't drink it unless it was the only Bourbon on the bar; and,
- Wouldn't clean my garage floor with it.

Trace Tippler
08-08-2010, 14:44
I am pleased to have read Chuck Cowdery's book early on in my 'bourbon career" and find much to agree with in his assessment of the uselessness of these 100-pt rating systems. I took his basic system and expanded it. At the top of my tasting notes notebook I made up is a way for me to remind myself of the overall experience of a particular bottle without assigning numbers. Mine goes like this:

-Undrinkable
-Bad
-Okay
-Good
-Very Good
-Outstanding
-Sublime

The first may mean I really don't like something, there is a cork problem, a bad batch or whatever. Bad means I don't care for it as it's made but someone else might. The "ok, good, very good and outstanding" should make sense by themselves while the last one, "sublime" is something I'm looking forward to when I have more experience. Those sublime moments, when words fail completely and it's too incredible to think about but rather just enjoy it while it lasts. Based on what I've tasted so far, I know this is possible with bourbon.

Cordially,
Trace T.

PaulO
08-08-2010, 16:08
I was thinking Chuck's 0-4 system made a lot of sense to me. It seemed very close to about the number of categories I would place bourbons or beer brands into. Then it came to me; 0, 1, 2, 3, 4 correspond perfectly to F, D, C, B, A. Then we can get into plus and minus if you like.

StraightBoston
08-08-2010, 20:06
In my bourbon inventory I rate both taste and value on a three-point scale (+/o/-) corresponding to better-than-average, average, worse-than-average -- where "average" corresponds roughly to Jim Beam Black.

Res/st-or
08-09-2010, 16:37
The 0-100 scale must make sense to the retail industry. if an 86 turns to be not such a good bourbon, at least the number is high enough to confuse consumers to buy it anyway. Nobody would want to be associated with a 0 or 1 on Chuck's scale.

I have been confused by this as well, Chuck brought this to my attention, and I've thought about it since I first read this thread. I am glad to be involved with this site. I learn something new every day.

I think Chuck's scale makes sense. It is simple and precise. Retailers love to confuse, keeping us in the gray area. Gray = green to them. Or so I believe.

cowdery
08-09-2010, 21:34
It's Lake Wobegon marketing, where all of the bourbons are above average.

ILLfarmboy
08-09-2010, 21:54
In my bourbon inventory I rate both taste and value on a three-point scale (+/o/-) corresponding to better-than-average, average, worse-than-average -- where "average" corresponds roughly to Jim Beam Black.

I like this idea. Both taste and value are important. A rating system that includes both but keeps them seperate allows the consumer to decide for himself. Case in point, I don't care if somethging gets a 4 out of 4, or a 97 out od 100, if it costs a $80+, I usually won't buy it, or if I do, its not something I'm gonna stock up on.

George
08-10-2010, 04:30
I like the 0-4 scale very much.

sailor22
08-10-2010, 07:10
This need isn't so much to provide the reader with an 100% accurate/objective assessment, but to allow the reader a means to calibrate their palate to yours, the reviewer.

This has always been my understanding of ratings. I can usually shush out the relationship between the reviewer and my palate with the 100 point system, even if the Lake Woebegone effect is in play. Granted, it takes some time to understand the reviewers preferences.

The 0-4 scale makes perfect sense and should a huge help in making generalized buying decisions. I might modify it just a bit for my own use;

0 = bad
1 = barely acceptable
2 = good
3 = very good
4 = special and worthy of relaxed contemplation

p_elliott
08-10-2010, 08:35
I'm on record, in my book (http://cowdery.home.netcom.com/bourst.html) and elsewhere, as not thinking very much of rating systems. Their basic flaw is that they give a sheen of objectivity to something that is inherently subjective.

I had a long conversation about this with Jim Murray once and his conclusion was, "we owe it to people to give them some kind of guidance."

That point I concede. Therefore, I'm comfortable with a four or five point scale along these lines:

0 = bad
1 = barely acceptable
2 = good
3 = very good
4 = great

My problem with the 100 point systems is exactly what comes up here from time to time. "How can X get a 96 when Y gets a 95?" Consequently, one of the rules of my proposed system is no fractions, no 3 1/2 stars. The idea is to rate in a broad way without ranking.

Subjectivity is still an issue as it always will be. The best bet for someone who wants to use anyone's ratings for guidance is to get to know the tastes of different critics. Either find one who generally agrees with you or just evaluate all of them based on what they seem to like and dislike.

Anyway, I don't want to write a long essay here. I just mean to open the discussion.

Although I agree with your point in general it doesn't solve anything. Everything is going to get rated in blogs and magazines as being 3 or better with very few 3's. We're right back where we started. To be honest I have yet to see whiskey writer/reviewer come out and say "This product sucks don't buy it" No offense meant to Chuck or John.

BourbonJoe
08-10-2010, 10:07
To be honest I have yet to see whiskey writer/reviewer come out and say "This product sucks don't buy it" No offense meant to Chuck or John.

Never bite the hand that feeds you.
Joe :usflag:

callmeox
08-10-2010, 10:38
While they may not say those words exactly due to the lack of professionalism required to dip that low, if you peruse the whiskey ratings on MA, you will find low scores that lead you to the same conclusion.

CorvallisCracker
08-10-2010, 10:59
To be honest I have yet to see whiskey writer/reviewer come out and say "This product sucks don't buy it" No offense meant to Chuck or John.

When, in 2007, I encountered the lineup of Hirsch Canadian Rye whiskies at a local liquor store (8, 10 and 12yo expressions), I checked John's review, which stated, "They’re not good enough to drink neat or on the rocks and too expensive to drink as a mixer", giving them scores of 75, 72 and 70, which was enough to persuade me not to buy any.

For my own purposes I use a 0-5:

0. Sucks
1. Doesn't suck
2. Good
3. Very Good
4. Excellent
5. Memorable (as in, I'll remember it for the rest of my life.

Very few things have ever gotten this last one. In bourbon, it's only been

VSOF S-W produced
WT 12 yo (this version (http://www.straightbourbon.com/forums/showthread.php?t=11074) and the CGF)
Hirsch 16 BW
OFBB 2007 release
PVW 20

In red wine, only

1982 Ch. Pichon-Lalande (six bottles consumed from 1992 to 2002)
1986 Ch. Lafite-Rothschild (magnum opened 2006)
1986 Penfolds Grange Hermitage (opened 2006)
1987 Modavi Reserve Cabernet at release (lost complexity as it aged)

and in the course of the last 30 years, I've sampled thousands of red wines.

Price factors in on whether I recommend it. A "doesn't suck" wine costing $3 is Recommended. An "excellent" one costing $150 is not, because you can get wines that good for a lot less.

callmeox
08-10-2010, 11:22
John's ratings are a more granular version of the 0-4 scale so I don't believe that his system is really too far from Chuck's.

Going somewhat by memory here. Forgive any detail issues.

60-69 (0) "avoid" with two expressions listed
70-79 (1) "average" around 60 listings
80-89 (2) too many to count on my phone
90-94 (3) many here as well
95-100 (4) less than 50 expressions

As one wouldlikely assume, the vast majority of bottlings fall in the 2-3 range with the far ends of the scale populated by the best and the worst out there. Is is a bell curve type distribution? I'm too lazy to key them all in to find out, but it looks like it would be close.

One can dispute the second lowest range being "average" as that's where the majority should appear, but his intent is not to force a distribution across the entire range, but to grade each expression as a single unit.

cowdery
08-10-2010, 11:50
Hansell and I have both panned things. John was very hard on Old Crow Reserve and has been hard on the most recent WRMC releases.

Since I don't necessarily try to review everything I sometimes pan with my silence. I have also been charged a few times of "damning with faint praise."

The problem isn't really with the reviewers. As many have noted, the right way to use critics is to learn their tastes and calibrate them to your own, regardless of the rating system they use. My objection is to the sheep who walk into a store and buy whatever has the highest number. I think the 100-point type systems enable that behavior.

OscarV
08-10-2010, 12:54
In an earlier post it was mentioned that they rate with "value" in mind.
I think a 1 to 5 scale is best with absolutley no consideration of the price.
My palate knows nothing of my financial condition and I only want it to tell me what it taste like.

bourbon-n00b
08-10-2010, 13:09
Since I don't necessarily try to review everything I sometimes pan with my silence.

Could you elaborate on this a bit, in terms of reader usability? How would we best be able to determine a "didnt review because I didn't like it well enough" from a "didn't review because I haven't got around to it, dont have a sample handy, etc"?

BourbonJoe
08-10-2010, 13:32
I use ratings, any ratings, to determine if I'm going to try something. If I like it, I'll load up, regardless of price. If Hansell or Cowdery, or anybody else rates something low (very unusual), even if i'm not tuned to their taste, I will not try it (unless somebody puts it out at the gazebo, for instance). Life is too short for mediocre bourbon (at least in my case).
Joe :usflag:

jburlowski
08-10-2010, 13:52
[quote=p_elliott;214435]... To be honest I have yet to see whiskey writer/reviewer come out and say "This product sucks don't buy it" ...quote]

F. Paul Pacult can be brutally (and entertainingly) honest is his reviews (Spirit Journal). One of the main reasons I enjoy reading them.

A few examples chosen at random:
"... What am I, a crash test dummy? This is nothing more than clumsy production by somebody who doesn't have a clue about flavor subtlety or nuance. Avoid like herpes."

"... In the mouth the entry at first is salty, then it turns unpleasantly bitter and astringent on the tongue; the midpalate is no better as the baked, industrial taste goes sour and extremely astringent. The finish is tanky and sufury. Terrible to the point of being undrinkable."

He uses a five star system: Not Recommended (one or two stars); Recommended (three stars); Highly Recommended (four stars); and Highest Recommendation (five stars).

ILLfarmboy
08-10-2010, 15:12
Salty with a sulfury finish. Sounds like an Islay malt he didn't like. I can't imagine any bourbon being that bad.:grin:

cowdery
08-10-2010, 23:58
Could you elaborate on this a bit, in terms of reader usability? How would we best be able to determine a "didnt review because I didn't like it well enough" from a "didn't review because I haven't got around to it, dont have a sample handy, etc"?

Not really. Sometimes I didn't get around to it because I didn't like it, or I just didn't have anything to say about it. If there's something widely heralded and I don't like it I won't stay quiet, but sometimes something just doesn't interest me enough to get a sample let alone write about it.

I also don't actually use the scale I'm proposing, but if I did use a rating system it would be something like that. When I do review something you actually have to read the review to see what I think. No shortcuts.

Some people try to write a few words about everything that comes along. Good for them. I can't drink that much. That's not a judgment. Reviewing everything just isn't my platform.

I also don't go out of my way to beat up on something you probably won't buy anyway.

I've stated many times that I tend not to care about something if I don't know who made it, which knocks out a lot of non-distiller products.

cowdery
08-17-2010, 09:33
This website (http://www.1792bourbon.com/Index.htm) for Ridgemont Reserve 1792, minimal as it is, illustrates some of what we were talking about in this thread. Look in the lower left hand corner. Accordiing to the Beverage Testing Institute, 1792 is a 93, ranking above the brands 1792 considers its competitors: Knob Creek, Woodford Reserve (both 90) and Gentleman Jack (82).

I only just discovered this "under construction" website, but the copyright date is 2009, so Sazerac obviously isn't in any hurry to add to it.

I was prompted to look at it by a press release. Apparently Ken Pierce is now the point man there, though he is titled as Chief Chemist and Brand Ambassador, not Master Distiller.

doubleblank
08-17-2010, 10:28
As Chuck stated in his first post, finding "critics" with preferences that correlate with yours is the best way to use their written reviews and/or scores. Robert Parker often gets hammered by wine industry people/other critics for his percieved love of full bodied balls-to-the-wall wines. I was a long time subscriber to his publication and found his tastes and mine were fairly well aligned and used his notes with much success to find great wines (often at low prices too). The wine universe is so large, some guidance can be helpful in finding wines that we would never ever taste on our own.

But the bourbon world is so much smaller that I (and I imagine 99% of the people on this board) don't need a critic's opinion as to whether to buy something or not. I know I'll get around to tasting almost everything out there at a Gazebo, bar, fellow SB'ers, distillery tour, etc and form my own opinion from there.

By the way, did you know 50% of all bourbons are below average!

Randy

OscarV
08-17-2010, 13:02
By the way, did you know 50% of all bourbons are below average!



I assume that this is your personel opinion.
My question is, only 50%?
I could go along with that if you have another percentage below as undrinkable, like maybe in the 20% range.

I love a good bourbon and I drink as much as possible, probably to much but most of it I could easily pass on.
I find my range getting more and more narrow as time goes by but that upper 6% to 10% has me sticking to boubon and all the other spirits are just folly.

doubleblank
08-17-2010, 13:25
I was just using the definition of "average".....50% are above and 50% are below, by definition.

I recall a south Texas Democrat ranting about how half of the kids in his district scored below average and we needed to do something about that! "More taxes and more school spending" was his solution. That might increase the kids' knowledge, but 50% are still below average!

Randy

callmeox
08-17-2010, 13:52
I think that 49 percent are below average and 1% are just average.

Don't get me started about the rest.

OscarV
08-17-2010, 14:18
I recall a south Texas Democrat ranting about how half of the kids in his district scored below average and we needed to do something about that! "More taxes and more school spending" was his solution. That might increase the kids' knowledge, but 50% are still below average!

Way way off topic here, but you failed to mention Texas Republicans wanting to rewrite history text books to fit their current fundamentalist Christian idealolgy.
Which is OK except for the fact these books get sold across the entire USA because Texas is such a huge market the book manufacturers sell them elsewhere uncorrected.
True, Democrats answer to everything is to pile more money on it.
Republicans answers are to deny the facts.
Democrats hearts are in the right place but not their brain.
Republicans have neither a heart or a brain.

OK, now back on topic, if you were to take your favorite bourbons and factor them into a percentile, what percent would that be of all bourbons?
Mine would be very very small, like around a few percentage.

imbibehour
08-17-2010, 14:44
THe 100 point scale I find just a bit too broad. For me ratings in general are kind of a starting point, but since not everyone has the same palate I don't get wrapped up in them.

I think actually the simpler scale that you offered if it were put into use gives less room for people to have opinions since there's not much room to have a difference between say... 4 and 5.

In the end... they are all just numbers really and that's all I look at them as.

jburlowski
08-17-2010, 17:09
I was just using the definition of "average".....50% are above and 50% are below, by definition.

Randy

I beleive you are refering to median (http://www.purplemath.com/modules/meanmode.htm), not average.

marco246
08-17-2010, 18:29
I've come up with a five star rating system which I use to refresh my memory when it comes time to add to my stock of bourbons. No pluses or minuses except for at the three-star level. There is only ONE three-star rated bourbon, and for me that is the benchmark. Everything else is above or below. For me the benchmark bourbon is Maker's Mark. Here's an example of how I rate stuff:

* Old Fitzgerald's 1849
** Bulleit Frontier
***- Blanton's Single Barrel
*** Maker's Mark
***+ Knob Creek
**** Four Roses Single Barrel, Lim Ed 2009, OESQ
***** Pappy Van Winkel's Family Reserve, 20 yo

What this means in practice is that I'll not seek out a bourbon rated below Maker's Mark unless for some special reason. Natually this rating system is completely subjective, but it is useful for my purposes. Price is not a factor in the ratings.

cowdery
08-17-2010, 21:15
One of the main reasons I hate ratings is because they are a crutch for lazy people looking for a short cut. They are looking for a short cut because they have no base of knowledge and no interest in obtaining one. People who buy-by-the-numbers are people who do everything by the short cut route and who are looking for social validation more than they are a great bottle of whiskey.

As a writer, I hate the idea that people don't even read the reviews, they just look at the numbers. Ideally, a critic will give you enough information to make up your own mind but you as the reader have to do some work too. You have to think. You don't need to be a writer to pull a two-digit number out of the air and you don't need to think to turn those numbers into a shopping list.

The people who buy-by-the-numbers wouldn't recognize "brine and spice, apple pip, and traces of aniseed"* if it bit them.

I don't blame or condemn the writers, publications, and entities like BTI that give ratings. They are forced to do it because they depend on advertising revenue or (in the case of BTI) fees to stay in business. The producers would scream bloody murder if any of those entities switched to the type of "good-better-best" ranking system we've been discussing here.

The sad reality is that their 93 will sell more bottles of 1792 than positive reviews by me and every other writer combined. And selling more bottles is what the producers are in business to do.

For the people who give ratings the pointlessness of it doesn't harm their credibility as long as they're running an honest game and so far as I know everyone is. That it's a silly and meaningless game is beside the point as long as people buy-by-the-numbers. The only harm it does is put great bottles of whiskey into the cabinets of people who manifestly do not deserve them.

As long as there are people with money to spend who believe the ratings mean something there will be ratings.


* From Dominic Roskrow's Guest Review of Caol Ila, 25 Year Old on "What Does John Know." He gave it an 88.

ILLfarmboy
08-18-2010, 22:16
One of the main reasons I hate ratings is because they are a crutch for lazy people looking for a short cut. They are looking for a short cut because they have no base of knowledge and no interest in obtaining one. People who buy-by-the-numbers are people who do everything by the short cut route and who are looking for social validation more than they are a great bottle of whiskey.


There are plenty of people like that, mostly stuck up snobs with no real interest in whiskey, or who think of the whiskey in their glass as a fashion accessory. But there are also plenty of us enthusiasts who have a limited funds. And some of us don't live in KY or a 20 min. drive from Binny's. If something new and interesting comes out, I'd like to know what experienced palates think about it before I spend $$ and more money on shipping. ..........And I'd like to know how good a value it is.

mrviognier
08-20-2010, 05:35
I actually like using a 100-point system better than other options. In utilizing stars, 5-points, etc. there's really not much room to assign a rating which allows for much subtlety within a peer group. The difference between a "***" and a "****" is probably wider than, say, a "86" and a "89".

BigRich
08-20-2010, 06:39
I'm with Chuck on this one too. I remember when Matt and I were starting out website and contemplated how we would review whiskey. For us, it was the false sense of objectivity between a "87" and an "88" that bothered us the most. We use a simple scale of:

Probably Pass
Average for the style
Stands Out
Must Try
Must Buy

Even the "Must Buy" is very rare. We only use it if the drink is exceptional and a good value. For instance Gold Bowmore may taste like heaven but most people can't afford that so we wouldn't give that a must buy.

dmarkle
08-20-2010, 06:45
Even the "Must Buy" is very rare. We only use it if the drink is exceptional and a good value. For instance Gold Bowmore may taste like heaven but most people can't afford that so we wouldn't give that a must buy.

Yeah, "Must Buy" is a funny rating. For me, the new OF BIB, while not very complex, is a must buy at $15, but the Pappy 23 isn't a must buy at $200. Maybe that should be split out into "I must buy" and "You must buy for me". :)

squire
12-18-2010, 12:00
Well, for good or ill the numerical rating system is in place and I doubt that will change. As for informed opinions this board provides very good reference points.

imbibehour
12-18-2010, 14:12
I prefer a 10 point scale but that's just me.

Everyone has their own rating system internally, so it's kind of hard to bring everyone together.

Not sure how it will all play out.

PAspirit1
12-18-2010, 17:30
I like a 100 point system for most things. With a ten point system, for example, you end up saying something like "these are both eights but one is a stronger eight. It's almost a nine but not quite. I wish I could express that."

kickert
12-18-2010, 17:54
I like a 100 point system for most things. With a ten point system, for example, you end up saying something like "these are both eights but one is a stronger eight. It's almost a nine but not quite. I wish I could express that."

I use a 10 point scale that is really almost a 100 proof scale. I like to use whole and half numbers, but then when I find an "8 that is a bit better than another 8" I will break out the 8.1 (or 7.9). I never rate under a 5, but that is because I think all bourbon is at least tolerable. I could go down below a 5, but it would have to taste like fingernail polish remover to make it all the way to very bottom of my scale. In reality most of my ratings fall between 7 and 9.5. A bad bottom of the shelfer might get a 6.

squire
12-18-2010, 21:24
Essentially the same scale I use Ben.

cowdery
12-18-2010, 23:51
Since this has come up again I realized I never explained how the magazines and other major rating outfits do it. This is a 10-point scale but since they allow tenths, just move the decimal point and it's a 100-point scale.

0 -5 FAULTY - There is something technically wrong with the product.

6 - 7 POOR - The product has little character or complexity and lacks balance.

7 - 7.5 AVERAGE - The product is okay but nothing special.

7.6 - 7.9 GOOD - The balance is good and there are elements of complexity.

8 - 9 VERY GOOD - The product is well balanced and complex.

9+ EXCEPTIONAL - The product is very complex, deep and rich, with lots of character.

Note that 0-7 is products that probably shouldn't be sold let alone entered into contests. 7 is 'acceptable.' Only above 8 does it get competitive. That's the space where most competitions play. Nothing below 7, only a few there, everything else 8 to 10. But there are no tens because nothing is perfect.

Usually when you are asked to rate, say, customer service in a survey, 5 is considered 'acceptable.' Anything below a 5 is considered some degree of less-than acceptable, anything above a 5 is considered some degree of exceptional. Below average, better than average.

Why doesn't it work that way with beverages? The rationale is that, indeed, anything less than 5 is unacceptable, and while 5 to 8 might be acceptable, only 8 to 10 is award-worthy. The purpose of the competition is to determine which of the award-worthy products is best.

I'm neither attacking this system nor defending it, just explaining it.

macdeffe
12-19-2010, 03:18
Is one scale better than others. I am not sure.

Personally I have chosen to use a scale that goes like this :

5 Made me cry
4 Superb Whisky
3 Good Whisky
2 Average (Forgetable?)
1 Flawed
0 Made me cry!

This is similar to a lot of other scales mentioned here, the wording is just different.

The reason I use this scale is that I don't find my consistency and skills good enough to grade on a 100p scale. So I have chosen to use a scale that suits me. If others like to use a 100p scale, I don't have any problems with that

Whatever scale others are using its very easy to spot when they consider a whisk(e)y to be good, and I always need a to read some reviews to calibrate my tastes toward the reviewer

Steffen

Special Reserve
12-19-2010, 04:16
Is one scale better than others. I am not sure.

Personally I have chosen to use a scale that goes like this :

5 Made me cry
4 Superb Whisky
3 Good Whisky
2 Average (Forgetable?)
1 Flawed
0 Made me cry!

This is similar to a lot of other scales mentioned here, the wording is just different.

The reason I use this scale is that I don't find my consistency and skills good enough to grade on a 100p scale. So I have chosen to use a scale that suits me. If others like to use a 100p scale, I don't have any problems with that

Whatever scale others are using its very easy to spot when they consider a whisk(e)y to be good, and I always need a to read some reviews to calibrate my tastes toward the reviewer

Steffen


Steffen,

I love your scale. Particularly the made me cry at both ends.

Will

cowdery
12-22-2010, 15:36
My point is that most people, including most of the people who have posted their own scales here, put 'average' in the middle and flow out from there. In the 10-point scale used by competitions, 'average' is about a 7. That is the disconnect.

kickert
12-23-2010, 07:50
My point is that most people, including most of the people who have posted their own scales here, put 'average' in the middle and flow out from there. In the 10-point scale used by competitions, 'average' is about a 7. That is the disconnect.

That all depends on what you are averaging. If you are averaging possible products then 5 would be in the middle. 10 would be a honey barrel of 15 yo SW uncut and unfiltered, 1 (or 0) would be a bottle of foreshots straight off the still of a sugar water run and 5 would be a bourbon, but a really awful one.

However, if you average the scores of all available product, then 7 is probably a pretty accurate assessment. After all, just because there can be worse products, doesn't mean they are actually on the market. Because those are not being sold, the entire average creeps up.

Special Reserve
12-23-2010, 08:01
IIRC, the ratings that I've seen on a one hundred scale range from the 70's to about 99. With the lowest ratings not for mid-shelf products but for lower shelved products.

The ratings done by the professionals (or at least those being paid for their ratings) the effective scale is 70 to 100.

flintlock
12-23-2010, 09:24
The ratings done by the professionals (or at least those being paid for their ratings) the effective scale is 70 to 100.

I bet this is to be able to give the maximum number of advertiser's products as high a rating as possible. Rating one whiskey a 93 and one a 94 isn't likely to anger either one. Rating one a 4 and one a 5 is going to get you a phone call. "Yeah, money's tight this year...advertising dollars might have to be cut..."

But then, I'm cynical... :rolleyes:

p_elliott
12-23-2010, 09:33
I bet this is to be able to give the maximum number of advertiser's products as high a rating as possible. Rating one whiskey a 93 and one a 94 isn't likely to anger either one. Rating one a 4 and one a 5 is going to get you a phone call. "Yeah, money's tight this year...advertising dollars might have to be cut..."

But then, I'm cynical... :rolleyes:

I don't think John Hansel would go for this, I think John tells it like it is.
Chuck isn't opposed to stepping on some toes either read some of his post here. He has flat out pissed off some rectifiers.

p_elliott
12-23-2010, 09:54
Ryan

In addition to your post let me relay a story to you: Chuck Cowdery was asked to do a whiskey judging competition. It got to the Rye Whiskey division and Chuck was out voted and High West I think Rendezvous won. Chuck had a melt down how can a rectifier win best distiller? Good Question Chuck! They argued back and forth and the regulations of the contest were checked. High west won the competition but the rules may be rewritten for next year. Your not going to piss in Chucks face and tell him it's raining, he's not going to buy it.

Paul

JohnHansell
12-23-2010, 13:48
I don't think John Hansel would go for this, I think John tells it like it is.
Chuck isn't opposed to stepping on some toes either read some of his post here. He has flat out pissed off some rectifiers.

Thank you! I was about to chime in, but you beat me to it. I really don't care whether someone's an advertiser or not. I'm still going to be honest about how I feel about a whiskey. And we have lost a lot more advertising because of our ratings than we ever got because of our ratings.

OscarV
12-23-2010, 13:56
And we have lost a lot more advertising because of our ratings than we ever got because of our ratings.



That's interesting.
Today's companies are to super-sensative for their own good.
For example I have disagreed with your ratings and reviews a lot but I ain't cancelling my subscription to Malt Advocate.
I want to hear all views.
BTW, the recent issue, Winter 2010 is packed with lots of good stuff.

flintlock
12-23-2010, 15:25
I should have prefaced my comments by saying that I meant that, across all sorts of rated products, a 100 point system seems to simply allow more room for products to be in the top 10. Whether it's wine, cigars, wristwatches, or ice cream...an article with a bunch of 4's and 5's wouldn't give the same sense of getting "vital statistics". I think numerical ratings are pointless, personally, but people expect them.

I certainly didn't mean to imply that John or Chuck ever allowed advertisers to sway their views on things, and I hope it didn't sound like that. It was really more of a comment on the post-Wine Spectator/Cigar Afficionado need to rate everything under the sun, 1 - 100.

squire
12-23-2010, 15:42
I would rate an average whisky between 5-6 but haven't found any that I would describe as awful.

T Comp
01-22-2011, 18:39
For those that don't subscribe to Mr. Cowdery's Bourbon Country Reader you should because his recent review of Angel's Envy and WhistlePig (Vol. 13, No.3) is a terrific demonstration why ratings are not necessary. The review is more informing than any number or grade could ever be and the absence of one intrigues a potential buyer even more.

Lucas Jackson
01-22-2011, 20:56
My point is that most people, including most of the people who have posted their own scales here, put 'average' in the middle and flow out from there. In the 10-point scale used by competitions, 'average' is about a 7. That is the disconnect.

Someone mentioned a report-card style rating F,D,C,B,A. C at 70 being average and leaving 0-69 for failing. I would agree that 'average' is about 7, not in the 'middle'. It seems illogical, but seems to be accepted as a norm. Perhaps once it reaches 'bad' it doesn't matter to most if it makes it all the way to 'awful'

I think, as many have observed, knowing the rater's 'tastes' is most important. Diluting the (my) need for a tangible 'X', 'Y' rating.

Great topic, got my wheels turning.

kickert
01-22-2011, 21:29
Someone mentioned a report-card style rating F,D,C,B,A. C at 70 being average and leaving 0-69 for failing. I would agree that 'average' is about 7, not in the 'middle'. It seems illogical, but seems to be accepted as a norm. Perhaps once it reaches 'bad' it doesn't matter to most if it makes it all the way to 'awful'

I think, as many have observed, knowing the rater's 'tastes' is most important. Diluting the (my) need for a tangible 'X', 'Y' rating.

Great topic, got my wheels turning.

I agree with you Lucus. An average or 7 (of 70) makes sense and I like the grading analogy. If in class you miss every other question you get a 50 and you fail. We don't think "average" mastery is the middle of knowing it all and knowing nothing. Average is a judgement of what is out there based on what is possible.

cigarnv
01-23-2011, 08:12
Over the years I have paid little attention to the "experts" who review spirits, wines, beers, cigars, etc. All of them have a bias, IMO, as we all do regardless of if we choose to admit it or not. In many cases we don't even realize we have a bias which can be driven by a single exposure... good or bad to a product, person or organization.

On the flip side I find that blind tastings among knowledgeable consumers provides one the best and most honest evaluations of a product. A tasting group of 12 reasonably knowledgeable individuals each tasting 2oz. of "blinded" spirit over several days with no results shown until all are in will IMO produce about as honest a rating as one can hope to get.

In the end does it really matter who makes a product or where it comes from...... isn't the objective to find something that just tastes damn good....??

Just an opinion, nothing more....

squire
01-25-2011, 20:53
Yep, pretty much, it's what's in the bottle that counts.

sailor22
01-26-2011, 06:20
On the flip side I find that blind tastings among knowledgeable consumers provides one the best and most honest evaluations of a product. A tasting group of 12 reasonably knowledgeable individuals each tasting 2oz. of "blinded" spirit over several days with no results shown until all are in will IMO produce about as honest a rating as one can hope to get.

I understand your point but keep in mind a rating generated from many people is in the end a popularity contest. Valuable only in so far as you have some knowledge of the other tasters palates and preferences. Use it as a rough guide for what to try next perhaps but trust your own taste first.

cigarnv
01-26-2011, 07:23
I understand your point but keep in mind a rating generated from many people is in the end a popularity contest. Valuable only in so far as you have some knowledge of the other tasters palates and preferences. Use it as a rough guide for what to try next perhaps but trust your own taste first.

Steve... agree with you fully... if it is "blindly" popular I will usually give it a try....

politely
02-01-2011, 23:10
I like ratings and they do affect my purchasing decisions, but I don't really focus on the exact ratings. I care more whether it's highly rated by the specific reviewer within their own framework, and whether other reviewers have similar opinions. I do take note when a reviewer marks an expression as special or extraordinary. But I think the reviews provide, at best, a subjective view and I take the reviews with some salt. I suppose reviewers could be a special breed with amazing consistency, but I find that my own opinion of a particular bottle changes from time to time, depending on the day, my mood, taste, and changes in the bottle (ie, time and volume). My guess is that if a reviewer checked back, their rating of the last pour would likely be different from the first pour. Actually, I'm not sure that the rating at the opening of the bottle is necessarily the most accurate or best. People generally don't finish a bottle the day they open it, and so, in my view, the changes over time are important, but aren't really given any consideration.

Buffalo Bill
10-17-2011, 17:45
In my bourbon inventory I rate both taste and value on a three-point scale (+/o/-) corresponding to better-than-average, average, worse-than-average -- where "average" corresponds roughly to Jim Beam Black.

2 months ago I hit a string of rare JB Black nips that were way better than average. Went back and bought a box, average.

Go figure...

BB

Flyfish
11-15-2011, 11:58
In my bourbon inventory I rate both taste and value on a three-point scale (+/o/-) corresponding to better-than-average, average, worse-than-average -- where "average" corresponds roughly to Jim Beam Black.

You are very close to the two-point system I use to rate bourbon: 0 = Don't like it; 1 = Like it. Off the top of my head, can't remember any 0s. :grin:

Tom Troland
11-15-2011, 15:56
I think whiskey reviews are best regarded as entertainment, stuff made up more to amuse than to enlighten, just like comic books. The best reviewers, like Paul Pacult, are clever writers. They amuse us. For example, Pacult once described a distilled spirit (whiskey, I think) as “buxom”. So what is a buxom spirit? Is it stacked with flavor? Despite my ignorance of his meaning, I was amused by his word. And I gather that buxom is good, at least in his eyes. We all have our preferences.

Most whiskey reviews contain very little useful information. Just like comic books. Whiskey reviewers often invent ten or more descriptors for a given sample. John Hansell is fond of this approach. For example, his recent review of Elijah Craig 20 Year Old identifies aromas of “nutty toffee, pecan pie, apricot, berried jam, and nougat, peppered with cinnamon, mint, cocoa, and tobacco”. Not to mention ”polished leather and dried spice” in the finish. Now, with all due respect to John, how could anyone possibly imagine all of those aromas combined? John might as well be describing his latest acid trip. The description is fascinating, but it is unlikely to resemble anyone else’s experience with the same substance.

The pointlessness of whiskey reviews (except as entertainment) is well illustrated by the reviews in Whisky Magazine. In an honest attempt to be more informative, Whisky Magazine offers two independent reviews of each whiskey. But if you read the two reviews, the texts have almost nothing in common. Imagine cutting out each whiskey review in a given issue of the magazine. Now try to match up the pairs of reviews that describe the same whiskeys. I am quite sure you would do little better than random chance.

Beyond the words of whiskey reviews there are the numbers. Especially the 100 point scale. Numbers normally imply quantitative measurement, at least in the science field where I work. But reviewers’ numbers are all made up. You might as well rate women’s beauty on a 100 point scale. Or guys’ good looks. Or acid trips. And Jim Murray believes his palate is so sensitive that even the 100-point scale is too coarse. So he awards half points. As if he is measuring body temperature with a thermometer under his tongue. Or somewhere else.

Of course, whiskey reviews are quite harmless. They offer good entertainment for many (like me), bragging rights for distillers, and an opportunity to collect and drink by the numbers for those who are so inclined. But we should at least acknowledge the obvious. This stuff is all made up. Just like comic books.

cowdery
11-15-2011, 16:55
Tom,

You're being unfairly hard on comic books.

Jono
11-15-2011, 17:10
Well, in a similar vein, most of us would agree on a broad rating of 7s, 8s, 9s and 10s for looks...be it men or women...and though the exact rating may differ I bet a large % would be within 10 pts. So, there is probably value in identifying the best from the average and bad. Once you find a reviewer with a similar palate the reviews carry more weight. I cannot tell you if a 87 pt whiskey or wine etc. is worse than a 91 pt but I assume both are good.
A 74 pt product should be noticeably worse than a 85+. General agreement is probably valid with some outliers expected.
I only pick up 2-4 flavor notes in most whiskey or wine so the pantry list of notes usually just adds color. I am not saying they don't exist, but most are passing experiences that come and go quickly in the overall taste. I think the mind can create an expectation and the note will be found if searched for in the mix.

camduncan
11-15-2011, 18:39
Tom,

You're being unfairly hard on comic books.

Too funny Chuck, too funny! :slappin:

Tom Troland
11-16-2011, 07:43
Chuck,

You caught my bias, I never did like comic books although I always enjoyed Mad Magazine.

Despite my comments about whiskey reviews, I do read them avidly. And I respect the whiskey knowledge of people like John Hansell, Jim Murray, Paul Pacult and yourself (even though you are not primarily in the business of reviewing whiskeys). If a new whiskey release comes out, and it is expensive, I'm more likely to try it if respected reviewers rate it highly. But I don't give much weight to the exact numbers on their scales nor to their words of description unless they are funny.