When it comes to natural supplements, nothing beats fish oil. These little capsules of omega-3s, which are made with everything from mackerel, sardines, and anchovies to krill and vat-grown algae, are more popular than glucosamine and probiotics combined.

The millions of Americans who take fish oil supplements have become all too familiar with the intensely fishy burps they cause. But it’s a small price to pay to reduce the risk of heart disease, which remains the nation’s leading cause of death.

The problem is fish oil doesn’t do anything. Or it doesn’t seem to do what we thought it did. What began as a series of groundbreaking nutritional discoveries in the late 1990s has turned into a perplexing drip of null findings.

That’s why the cardiologists, nutritionists, and doctors were paying keen attention the American Heart Association’s Scientific Sessions, held last week in Chicago, where the results of two major fish oil studies, following more than 34,000 subjects, were presented. Harvard’s JoAnn Manson delivered the results of the first study: another null finding. Fish oil supplements, her study found, do little to prevent heart disease. Fifteen minutes later, standing at the same podium, another Harvard scientist, Deepak Bhatt, claimed the opposite. In his study, a purified fish oil intervention reduced the risk of coronary events by an astonishing 25 percent.

These developments in the fish oil story are intriguing, confusing, exciting, and depressing all at once. They show, above all, that the progress of science is not always as linear as we think. As research methods have improved, statistical power has grown, and scientists have tested ever more precise hypotheses, the truth about fish oil remains, like fish itself, slippery.

Exquisitely carnivorous Greenlanders

The fish oil and heart health story begins in 1970, when two Danish scientists traveled to the northwest coast of Greenland to study an indigenous population that had been described as “probably the most exquisitely carnivorous people on earth.” Despite their extremely meaty diet, this community appeared curiously free of diabetes and between 1963 and 1967 experienced a mere three cases of heart disease. The scientists studied 130 local Inuit and made an intriguing discovery: They had blood lipids lower than their Danish counterparts.

It wasn’t genetics. Inuit people who lived in Denmark had blood lipid levels similar to typical Danes. After nearly a decade of further study, the scientists concluded that the difference came down to diet. Specifically, it was the high consumption of fish and marine mammals, whose omega-3 fatty acids were present in the Greenlanders’ blood.

A few years later, the New England Journal of Medicine published a study that examined seafood consumption and coronary heart disease in 852 middle-aged Dutch men over a 20-year span. It found a striking “inverse dose–response relation.”

“Mortality from coronary heart disease,” the authors wrote, “was more than 50 percent lower among those who consumed at least 30 g of fish per day than among those who did not eat fish.”

The fish oil gold rush begins

By 1990, seafood had captured the attention of cardiac researchers. That year, 51 studies were published exploring the link between fish oil and heart disease, up from four a decade earlier. But the research was mainly “observational.” It compared health outcomes of people who ate a lot of fish to those who did not, and by doing so, it uncovered intriguing correlations between eating fish and cardiovascular health. But these studies did not, and could not, prove causation.

Was it the oil in fish that was causing people to have less heart disease? Perhaps. But there were other equally plausible causes. For example, it could be that affluent people tend to eat more fish, and that affluent people, like rugged Greenlanders, also tend to spend more time outside exercising, and that it’s actually exercise that causes people to have less heart disease, which would make fish consumption a marker for less heart disease but not the cause. Or it could be something even simpler: When people eat fish, they have less room in their diet for Big Macs, fried chicken, and candy bars.   

What was needed was a large-scale randomized controlled trial (RCT), which is considered the “gold standard” of nutritional research because it can demonstrate cause. That began in 1993, when 11,324 heart attack survivors from across Italy were given fish oil, vitamin E, or nothing. The vitamin E had no effect. The fish oil, by comparison, was astonishing, causing a 10 percent reduction in heart disease.

Another sensational RCT followed almost a decade later. In this one, a fish oil supplement led to a 19 percent reduction in heart disease in 18,645 Japanese subjects. Fish oil, amazingly, could even confer a health benefit to a population at the upper end of seafood consumption.

In 2008, scientists published 114 studies on fish oil and heart disease. As fish oil’s biological importance became further elucidated, particularly its role in reducing inflammation, it became a wellness panacea, offering apparent benefits for everything from blood pressure and triglycerides to pain, vision, and mental health.

An avalanche of null findings

But the good times wouldn’t last. In 2010, the first major null RCT, published in the New England Journal of Medicine, found that fish oil “did not significantly reduce the rate of major cardiovascular events among patients who had had a myocardial infarction and who were receiving state-of-the-art antihypertensive, antithrombotic, and lipid-modifying therapy.”

By 2012, nearly 20 million American adults were using a fish oil supplement of some kind. But among scientists, the relationship was souring, and fast. A big RCT, also published in the NEJM, found that a daily supplement of omega-3 fatty acids “did not reduce the rate of cardiovascular events in patients at high risk for cardiovascular events.”

It got worse. The Journal of the American Medical Association published a meta-analysis of fish oil, which examined 20 studies comprising 68,680 subjects. It soberly concluded, “Omega-3 PUFA supplementation was not associated with a lower risk of all-cause mortality, cardiac death, sudden death, myocardial infarction, or stroke.” A year later, fish oil research would reach its zenith, with 169 papers published, a high point never again to be reached.

This year was shaping up to be fish oil’s worst to date. In July, another major meta-analysis, this one billing itself as “the most extensive systematic assessment of effects of omega-3 fats on cardiovascular health to date,” reported that fish oil supplements have “little to no effect on mortality or cardiovascular health.” Its illusory benefits, furthermore, “spring from trials with higher risk of bias.” Scientifically, fish oil looked dead.

And if that wasn’t enough, the final nail in the coffin came last Saturday in Chicago, in the form of JoAnn Mason’s VITAL study, an RCT that followed 25,871 participants over five years. The results indicated that “supplementation with n-3 [omega-3] fatty acids did not result in a lower incidence of major cardiovascular events.”  

But wait, is fish oil back?

Except fish oil wasn’t dead. Because within minutes of the VITAL results, Deepak Bhatt announced the results of his REDUCE-IT RCT, results that were so exciting, a roomful of cardiologists gave it a standing ovation. In this study, which followed 8,179 people over four years, fish oil reduced the risk of cardiovascular events by 25 percent, and this in subjects who were already being treated with heart disease drugs like statins. All of a sudden, fish oil was alive and flopping in the bottom of the boat.

All of which raises the question: What the heck is going on? Why would fish oil look healthy, then stop looking healthy, then suddenly look healthy again?

One of those answers is statins. This popular class of cholesterol-lowering drugs didn’t exist when the first fish oil studies were done. It may be that whatever positive effect fish oil once had was made redundant by statins. Statins, in other words, may have eaten fish oil’s lunch. That, of course, doesn’t explain the recent REDUCE-IT results. But that study featured a very large dose.

D’oh, it’s dose

In fact, dose could explain a lot of what’s going on. In the first-ever fish oil RCT, subjects received 1 gram of fish oil. The Japanese study that followed in 2007 featured 1.8 grams. And the recent REDUCE-IT trial featured such a large dose — 2 grams twice daily for a total of 4 grams — it blurs the line between nutritional supplement and pharmaceutical intervention.

Many of the null studies, by comparison, used small — and in some cases, tiny — doses. For example, in that null study published in 2010, subjects consumed about 0.375 grams of fish oil. In the VITAL study, subjects consumed 840 milligrams of omega-3s. Statins wouldn’t work if people didn’t get a sufficiently large dose. So why would a low dose of fish oil work, especially in subjects who are already being treated with statins?

Then again, fish oil’s mixed track record may come down to type. The Japanese study and the recent REDUCE-IT study — the two most dramatic results in the history of fish oil research — did not use standard fish oil. They used EPA, which is one of two major fatty acids found in fish oil. The other is called DHA. Of the two, DHA tends to get more attention because there’s a lot of it in the human brain. But EPA has its own biological importance. It is converted into a family of molecules called “prostaglandins” and “specialized proresolving mediators,” which regulate inflammation, thin the blood, and are believed to lower heart attack risk.

So it might be EPA. But it might be the dose. It might be both EPA and dose. And statins may also play a role.

Then again, the harshest fish oil critics might be right — fish oil might not do anything, and its apparent benefits may be due to nothing more than poor study design or chance, although that seems like a long shot at this point. However, while heart health has been a focus of fish oil research, it is not the only area being studied. There is evidence that fish oil lengthens gestation in pregnant women and improves anxiety. EPA may also have an antidepressant effect.  

Finally, it’s possible there is some variable at play that no one has paid adequate attention to — like oxidation. Despite its reputation, fish oil is not inherently fishy. It only smells that way once omega-3 fatty acids begin reacting with oxygen. (Another word for this kind of oxidation is “rancidity.”) And some lipid scientists believe that taking fish oil that has become oxidized does not deliver the same benefits.  

Should people stop taking fish oil? Should they start? What about people who never eat fish? And is a high dose of EPA the answer to our problems?  After half a century of study, we may not have the answer we want — but we do have better questions. What we need, believe it or not, is more research. As it happens, another major fish oil study is underway. It should be reporting about a year from now.

Richard Bazinet is an associate professor and the Canada research chair in brain lipid metabolism in the department of nutritional sciences at the University of Toronto.

Mark Schatzker is the author of The Dorito Effect and the writer in residence at the Modern Diet and Physiology Research Center at Yale University.

Few consumer products offer as staggering a range of choice as wine. You can buy a bottle of Dark Horse Big Red Blend for $8. Or for around $500, you can get a 2012 bottle of Sloan Proprietary Red. Yet for each bottle, the same question applies: Is it any good?

For decades, Americans turned to professional critics like Robert Parker to help them make that determination. But the internet changed all that.

The rise of the wine-rating crowd

In 2004, Eric LeVine — then a group program manager at Microsoft — launched CellarTracker, a site where amateur wine enthusiasts can rate wines. Today, CellarTracker is the web’s most popular “community” or “crowdsourced” wine review website, containing 6.3 million reviews from 113,000-plus users for more than 2.2 million different wines.

Many professional critics, not surprisingly, have scoffed at the idea that mere amateurs understand, let alone have the ability to rate, wine. In a 2012 column for the website Wine Spectator, critic Matt Kramer described the wisdom of the crowd as a “pernicious delusion.” “One hundred people who don’t know much about, say, Auxey-Duresses,” he wrote, “adds up to 100 muddied, baffled and often duplicative conclusions.” Critic Steve Body concurred in a 2014 post titled “Crowd-Sourced Ratings and Why They Suck” on his website ThePourFool: “The readers and users of these sites are almost always slaves to their personal preferences and current trends.”

That’s the standard knock against amateur critics. Compared with the paid professionals — who very often evaluate wines blind — they are untrained, are subject to bias, and lack expertise.

But is that true? In 2016 (when this piece was first published), we decided to test the hypothesis by comparing community wine reviews for California wines from 2014 or earlier with those of professionals by running a simple correlation — a common data analysis tool.

For the professional scores, we purchased memberships to three sites: Wine Advocate (Robert Parker’s site), International Wine Cellar, and Jancis Robinson. We limited ourselves to these three because they had the most wines in common with CellarTracker. (Shortly after we obtained our data set, International Wine Cellar merged with Vinous and now goes by that name.)  

Altogether, we obtained scores for 10,679 wines on Wine Advocate, 12,182 on International Wine Cellar, and 1,167 for Jancis Robinson and compared them with 51,689 mean scores for wines on CellarTracker. We then matched the wines and their scores between our select group of experts and the multitude of enthusiasts and analyzed them with a software program called Prism.

Before we say anything more about the results, it’s worth considering what this sort of comparison can and cannot accomplish. In science, this kind of study is called “observational” — you may be able to tease out a relationship, but you won’t know why it’s there. For example, if it turns out the experts and enthusiasts differ significantly in their estimation of wine quality, this does not necessarily mean the experts possess a more refined or objective understanding of wine. It could just as easily be interpreted to mean that wine experts can’t relate to the kind of wine ordinary people find enjoyable.

But that’s not what we found. Nor did we find perfect agreement. We discovered something altogether more interesting.

So let’s start with the big question: How do the wine enthusiasts compare with the experts? Very well. Have a look:

There are 9,119 dots on this diagram — one dot for each wine rated by both CellarTracker and Wine Advocate. Each dot, furthermore, represents two scores, one by Wine Advocate and the other the mean score on CellarTracker. As you can see, it doesn’t look as though there’s anywhere close to 9,119 dots in this diagram. What really stands out is the big blob in the top right. This is a visual representation of CellarTracker and Wine Advocate agreeing. Most of the CellarTracker scores are very close to, and in many cases identical to, the scores on Wine Advocate.

Amateur and professional wine scores correlate very tightly

How similar? We ran a statistical tool called a Spearman correlation and got a figure of 0.576. A perfect correlation is 1. An utter non-correlation is 0. A score of 0.576 may not sound impressive at first, but it can actually get worse than 0 — a negative correlation, which is what you would see if you compared, say, shortness with the likelihood of playing professional basketball.

The CellarTracker scores correlated with International Wine Cellar nearly as well, with a value of 0.555.

And the weakest correlation was with Jancis Robinson, at 0.424.

But to get an even better understanding of just how well CellarTracker relates to the experts, the best thing to do is leave CellarTracker out of it. It’s when you compare the experts with one another that things start to get much more interesting.

Among experts, the strongest Spearman correlation is between Wine Advocate and International Wine Cellar, which, at 0.568, is still lower than the 0.576 between Wine Advocate and CellarTracker —and it only gets worse from there.

Jancis Robinson and Wine Advocate comes in at 0.208, and Jancis Robinson and International Wine Cellar isn’t much better at 0.222. That’s why the blobs are more diffuse when experts are compared with each other than when they’re compared with CellarTracker. There is less agreement.

What does it all mean?

Amateurs appear more expert than the experts

It looks very much like the enthusiasts actually do a better job of agreeing with the experts than the experts do with each other. That might sound odd, but out of thousands of wines we analyzed, only a handful contradicted this pattern. Simply put, if you want to know what the experts think, the best place to look appears to be, of all places, CellarTracker. 

Why do the scores correlate so well? The data doesn’t tell us that.  

It’s possible all those enthusiasts on CellarTracker already knew what Robert Parker and other experts said about each wine and just parroted their scores. That said, when you consider that a 2007 Seghesio Family Vineyards Zinfandel Sonoma County was rated 1,406 times on CellarTracker, for an average score of 90.5, does it really seem likely that 1) they had all read the expert scores, and b) they were consciously or unconsciously swayed as a result?

If so, how likely is that to play out over thousands and thousands of bottles?

It’s worth pointing out that our focus on California also comes with limitations. We did this because California presumably represents a more homogeneous group of wines than would a mix of regions. But we don’t know if the relationship we observed applies to other wine-producing countries or regions, like Spain, Chile, Burgundy, and Oregon. (That would require a larger study.) 

And it’s not like the experts and the enthusiasts always agree.

If you return to the first figure, comparing CellarTracker and Wine Advocate, you’ll see numerous dots that aren’t part of the main cluster. For example, on the bottom row, there is a single dot just to the left of 90, at 88. If you look over to the y-axis, you can see that it lands on 52. The wine in question is the 1999 Testarossa Chardonnay Sleepy Hollow, and it appears to be a case of serious disagreement — a difference in score of 36 points. Wine Advocate gives the wine a score that, according to its own scoring system, is between “above average” and “very good,” while CellarTracker says it’s terrible.

Upon further inspection of the data, however, we noticed that only a single CellarTracker user scored 1999 Testarossa Chardonnay Sleepy Hollow, representing an extremely small sample size. In fact, correlations between amateur and expert only become stronger as the quantity of community reviewers increases. The larger the number of enthusiasts, the more the scores match those of the experts. Cases of strong disagreement, however interesting they may appear, are extremely infrequent.

The better the wine, the more experts agree with the amateurs

There is also a tendency for scores to converge as wines improve in quality. This is evident in the arrow shape of the clusters in figures comparing CellarTracker with Wine Advocate and CellarTracker with International Wine Cellar. (Notice how the cluster points up and to the right.) Average scores, furthermore, are high. On Wine Advocate, the average score was 89, on International Wine Cellar it was 91, and it was 17 out of 20 for Jancis Robinson. On CellarTracker, it was 89. This tells us that experts and enthusiasts alike don’t seem to be spending a great deal of time scoring mediocre wines.  

As with New York City restaurants, there are many very good wines but few that achieve true greatness. This is something both expert and enthusiast agree on, as indicated by the pointiness of that arrow shape.

So what did we learn about the CellarTracker users? They drink good wine. And when you consider that according CellarTracker’s founder, Eric Levine, the average user has rated 49 wines and there are 2,311 users who have rated more than 500 wines, this group sounds less and less like a rowdy horde of merlot swillers. It sounds like they take their wine seriously. When it comes to online comments sections or restaurant reviews on Yelp, the internet has a reputation for being overly representative of the ill-informed and overly opinionated. This doesn’t seem to be the case with CellarTracker.

Ultimately, we think our analysis is very supportive of community wine reviews. If non-professional wine enthusiasts truly were lacking in knowledge or expertise, we would expect to see little or no correlation with professionals. We saw just the opposite.

But the news isn’t all bad for the professionals. Their divergence in opinion could come down to differences in personal taste. Maybe the reason Jancis Robinson correlates relatively poorly with International Wine Cellar is because she has her own distinctive palate. Perhaps you’re better off finding a critic whose taste matches your own, since these nuances get lost in averages.

In the end, however, CellarTracker offers two compelling advantages. The first is price. Joining CellarTracker is free, whereas Wine Advocate costs $99 for one year and Jancis Robinson costs around $110. The second is breadth. For California alone, CellarTracker covers more wines than all the critics we examined combined.

So if you think it’s worth spending a hundred bucks to access the wine scores published by Wine Advocate, go for it. But you might just be better off putting those dollars toward actual wine. For about $60, you can get a bottle of 2011 Littorai Pinot Noir Savoy Vineyard, which got a 90 from International Wine Cellar and a 92 from Wine Advocate. CellarTracker gave it a 91.

Mark Schatzker is the author of The Dorito Effect: The Surprising New Truth About Food and Flavor. Richard Bazinet is a professor of nutritional sciences at the University of Toronto and the Canada research chair in brain lipid metabolism.

s