How Valid and Reliable Are Wine Judging Events? Part 1

After making wine for a few years, many of us amateur winemakers entered our creations into wine judging events. If you are like me, the first time you dipped your winemaking toes tentatively into the seemingly daunting waters of such competitions you were simply delighted by not being frost bitten by the judges’ evaluations. “A third place? Wow! That’s great!!! Participation Ribbon? Fine and Dandy!! A Breathing Award for my wine? Ok, I’ll accept that too!” Such affirmations undoubtedly boosted your confidParticipationence and motivated you to further enhance your skills in the winemakers’ equivalence of the backstroke or butterfly. Perhaps you even began free styling in the ever-warming waters of all-that-is-winemaking.

Yet it is also likely that the thrill you experienced over these early successes were occasionally replaced by dismay gold-seal-red-ribbon-award-3669340and discouragement when some of your winemaking entries failed to earn any awards at all. Maybe your entries even received an affirmative thumbs-down by the panel of judges who sniffed, tasted, and eyeballed them. Of course, negative feedback can be very instructive in perfecting skills in any area of pursuit. But what if you begin to suspect that the feedback is considerably less than perfect? Perhaps your doubts about the judging were raised when you later entered your panned wines into other competitions and received very favorable evaluations. Or perhaps you tasted a few highly rated wines at these events and noticed imperfections that the judges apparently didn’t notice. Such experiences could cause you to question the validity of wine judging events as a whole.

In many instances, this questioning of judges’ results is purely a case of “sour grapes” by those of us who have difficulty graciously accepting constructive criticism. Be this as it may, following extensive reading on this topic, I have come to the conclusion that some of the problems related to wine judging rest firmly on the shoulders of the judges themselves and the false beliefs many of them hold about their ability to accurately assess a wine. For example, world-famous wine critic Robert Parker once told an interviewer that he tastes 10,000 wines a year—which comes to more than 27 wines every single day—and remembers the rating of every wine he has tasted over the past thirty-two years within a few points. Yet, in a public blind tasting of fifteen top wines from Bordeaux 2005—which he called “the greatest vintage of my lifetime”—Parker could not correctly identify any of the wines, much less rate them similarly to his previous evaluations.

Robert Parker, wine critic

Robert Parker, wine critic

In far too many instances, I have read interviews with wine experts like Parker who falsely claim that their sensory and perceptual skills are machine-like in accuracy. “Objectivity” is the word they often bandy about when referring to “proper” wine judging. In researching this article, I have repeatedly read statements such as the following from recognized wine-judging experts when they provide advice on how to properly evaluate a wine: “Being able to evaluate your wine objectively is vitally important to successful winemakers.” (see

The problem with this statement and the belief underlying it is that when it comes to evaluating a wine using the nose and the mouth without the aid of pH meters, acid testers, sulfite meters, etc., all judgments are, by definition, subjective in nature, not objective. The dictionary definition of “objective” is “not influenced by personal feelings or opinions in considering and representing facts,” whereas, the definition of “subjective” is “based on or influenced by personal feelings, tastes, or opinions.” From the research findings that we are about to review, it will become clear that wine judges’ evaluations are very much influenced by a host of factors that are often unique to the individual judges. Unfortunately, our understanding of the wine-judging process continues to be muddied by some wine experts persisting in their claims that what they do when evaluating a wine is somehow divorced from their personal feelings, tastes, or opinions.

The argument I would like to make here is that wine judging is a subjective experience, and therefore, susceptible to human error. Yet how error ridden is wine judging? In an attempt to answer this question, my next article will discuss the wine-judging studies discreetly conducted over the past ten years by California winemaker and retired statistics professor Robert Hodgson. His findings may well surprise you.

How Valid and Reliable Are Wine Judging Events? Part 2

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s