Facebook tried to manipulate users’ emotions. But we have no idea if it succeeded

The backlash to a study by Facebook data scientist Adam Kramer, written with Jamie Guillory of UC San Francisco and Jeffrey Hancock of Cornell, which varied the emotional content of posts included on nearly 700,000 Facebook users’ walls and then tracked the emotional content of postings from those users to see if positive and negative emotions had a “contagion” effect, has been swift and strident. But the backlash has thus far missed a crucial point — the study’s methodology is garbage.

“Facebook intentionally made thousands upon thousands of people sad,” Slate’s Katy Waldman wrote. University of Maryland’s James Grimmelmann raised the possibility that the study might actually be illegal: federal law requires informed consent for federally funded studies (and Guillory and Hancock’s institutions receive lots of federal funding, though the study itself didn’t), a standard he thinks the study (which treated users’ acquiescence to the Facebook privacy policy as informed consent) failed to meet.

The most serious critique of all comes from the University of Michigan’s Christian Sandvig, who notes that with a sample that large, some participants almost certainly are severely clinically depressed, and creating a more negative Facebook experience could have grave consequences for them:

But I think the most alarmed reactions are a bit overblown. Waldman and Sandvig are not wrong that Facebook intentionally causing the sadness of hundreds of thousands of people would be really immoral; it would be! Making people sad is wrong. But we don’t even know that that’s what happened. We don’t know because the study’s methodology contains a really big flaw that makes interpreting its findings the way Kramer et al want them to be interpreted dubious.

Bloomberg View’s Leonid Bershidsky notes that the study relied on the Linguistic Inquiry and Word Count (LIWC) text analysis software, which relies on a dictionary of words classified as “sad” or “angry” or “happy,” and, by counting up their usages, arrives at an estimation of emotions expressed in a given text. LIWC’s own creators are clear on the limitations of this approach, in particular the way it rewards repetition. “Once you say something, you generally don’t need to say it again in the same paragraph or essay,” the LIWC website explains. “The nature of discourse, then, is we usually say something and then move on to the next topic. Saying the same thing over and over again is generally bad form.”

But it’s worse than that. LIWC also can’t read sarcasm or negations, the psychologist John Grohol notes. The former is a huge problem on social media — can you imagine trying to read your News feed as though it’s intended 100 percent sincerely? — and the latter is concerning whenever you have to deal with short snippets of text. Imagine that one of your friends posts this super-dark status update:

I’m not happy, glad, pleased, or loved.

That’s four happy words out of seven total in the sentence - 57.14 percent happy. And sure enough, according to the free LIWC analyzer the software’s creators make available, the sentence gets a 57.14 score for “positive emotions” and a 0 for “negative emotions.” Here’s what the results look like (ignore the right two columns, which give the average scores for previous analyzed texts):

This isn’t LIWC’s fault, per se. These kinds of things can cancel out over the course of long texts like books or even journal articles, and LIWC can be very useful in those contexts. But in short things like status updates, this is a real problem.

If you were in the experimental group of the study, chosen to receive more positive items in your News feed, then you’d be a lot likelier to see “I’m not happy, glad, pleased, or loved” than those in the group chosen to receive fewer positive items. So it’s hard to say, just using LIWC, that users in those respective groups really did receive the emotional content they were meant to receive. And it’s hard to say, using LIWC, their emotional reaction. Suppose I’m in the positive-updates group, see “I’m not happy, glad, pleased, or loved,” and other positive-scored but in fact negative content, and then post “I’m not happy, glad, pleased, or loved either.” That’d be scored as a person seeing positive updates and posting positive things in response.

I don’t know how much that sort of analysis error occurred in the study, and maybe in aggregate LIWC got it right. But Facebook’s researchers don’t really know how a big a problem that was either. In response to the controversy over the paper, Kramer posted a note on (where else?) his Facebook wall saying, “In hindsight, the research benefits of the paper may not have justified all of this anxiety.” Seeing how limited those benefits really are, he’s probably right about that.