Portugal decriminalized drugs in 2001. Barely anything changed.

It turns out one of the most common claims about Portugal’s widely acclaimed drug policy could be very misleading.

In 2001, Portugal decriminalized the use and possession of illicit drugs while it kept criminal penalties for production, sales, and trafficking. This meant using any drug — from marijuana to cocaine to heroin — couldn’t be punished with prison time as it had been in the past, but growing and selling such drugs could be. Since then, the radical move has been widely praised by drug reformers, who point out that — despite warnings from critics — drug use didn’t spike in the country amid decriminalization.

The positive assessment of Portugal’s drug policy was most popularly propagated by a 2009 paper by Glenn Greenwald for the libertarian Cato Institute, which found drug use didn’t increase among adolescents and teens. The Washington Post’s Christopher Ingraham reiterated the point in a recent article about Portugal’s few deaths to drugs.

But while it’s true that decriminalization didn’t cause a spike in drug use (or deaths) in Portugal, that could be because decriminalization just didn’t change much, if anything, in the country’s legal system. In a 2014 paper, UC Berkeley’s Hannah Laqueur found that even before Portugal passed its decriminalization law, it was already loosely enforcing its anti-drug laws. On any given year, a handful to a few dozen people could be in prison for drug possession. So the law was really only codifying an existing practice.

“When you actually look at the practice on the ground, the change was even smaller,” Laqueur said. “Even though drug possession was criminal before the passage, if you look at the actual number of people who were in prison for drug possession, it was tiny — 10, 20, maybe 30 people in any given year in the entire country. That’s less than 1 percent of the prison population. So in many ways, the law in Portugal was just implementing what was de facto criminal justice practice.”

It’s possible that people could have changed their behaviors in reaction to Portugal merely passing its decriminalization law. The Cato paper reported, for instance, that some people felt safer getting anti-drug treatment because they no longer feared getting arrested.

But as Laqueur’s paper points out, the research shows that people don’t react just to laws on the books — they react to laws on the books that are enforced with enough certainty and swiftness that doing something illegal would almost definitely result in punishment. And if people in Portugal were already being by and large left alone for their drug use, it’s unlikely they really changed their behaviors simply because the government altered some words in its legal code.

Laqueur’s research shows one of the tricky aspects of analyzing the effects of policy: it’s not always as simple as looking at what happened before and after a law passed.

This is a problem with, for example, evaluating whether the death penalty affects crime rates. In 2013, Maryland lawmakers abolished capital punishment. But no one had been executed in the state since 2005. Yet if there had been a crime spike in 2014, an analyst could try to link the increase to the abolition of the death penalty. But that would be a mistake, since the death penalty wasn’t really active in the state for eight years before it was abolished — meaning it very likely wasn’t acting as a deterrent to crime anyway.

There’s a similar issue with evaluating the effects of marijuana legalization. One of the big criticisms of legalization is that it will lead to skyrocketing pot use, so everyone is looking to Colorado’s drug surveys to see the effect of legalization before and after the state began selling pot in 2014. The problem is that Colorado pot users could already legally buy their pot from vendors in a medical system that was so relaxed that experts like Mark Kleiman, of New York University’s Marron Institute, have dubbed it “de facto legalization.” So if actual legalization doesn’t cause pot use to rise, it could just be that everyone who was using marijuana in the state could already get it from a medical dispensary.

This doesn’t mean that analyzing the effects of certain laws is impossible; it just means it’s more difficult than looking at the circumstances before and after the law was passed. Without the extra caution, people could end up praising a decriminalization law in Portugal that really didn’t have much effect on its legal system.

(h/t: Keith Humphreys, drug policy expert at Stanford University.)