“That’s just not a good move,” my father snickered. “I mean, maybe if you’re Ryan Gosling. But that is not a good look for Charlie Rose.”

It was only a matter of time, I figured, before one of the recent sexual abuse allegations would come up during a recent visit home. My father chose to focus on the Charlie Rose “trick” of surprising women who were working at his home by emerging from the shower, semi-naked.

My father’s tactic represents a common one for people who want to criticize Rose and the other sexual predators filling our newsfeeds right now: He took a shot at Rose’s physical attractiveness, or lack thereof.

I can’t lie; it’s been vengefully satisfying to see powerful men like Harvey Weinstein, Louis C.K., Matt Lauer, Kevin Spacey, and Rose fall from grace over the past several months. Hearing victims speak out about these men’s aggression, manipulations, and perversion of power — and hearing others who wield comparable power openly criticize them on national stages (what’s up, John Oliver!) — gives me hope that things are changing.

People are finally holding some of these men, as well as the deeply embedded patriarchy that supports them, accountable. What’s not as heartening or progressive is the discourse surrounding that accountability. In October, Samantha Bee came out swinging in a video addressed to Harvey Weinstein, insisting, “Your dick is ugly”  — and it was hard not to hear a broader judgment about the producer’s appearance. Many others went further. Howard Stern called Weinstein a “big fat guy,” adding, “There is no girl on the planet that wants to see Harvey Weinstein naked and is going to get aroused.” About Rose, Seth Meyers said in on his show: “Usually when someone that old is walking around naked, a couple of male nurses lead him right back to his room.”

Why should we care about someone like Weinstein (or Rose) being body-shamed? Because body-shaming him body-shames everyone else who looks like him but did nothing wrong. It also distracts from the problem with what he did, which has nothing to do with his looks. What made these acts abusive is the lack of consent, not the appearance of the predators.

And that perpetuates rape culture. As long as we keep acting as if sexual abuse is wrong because the abuser is physically unattractive or sexually deviant, abusers deemed attractive and “normal” will be more likely to get away with it.

The attractiveness trap in discussions of sexual assault

Given his crudity, and his own obsession with other people’s looks, Donald Trump makes for a tempting target for ridicule. But as Lindsey Averill wrote for CNN in response to a New Yorker cover depicting Donald Trump as a plus-sized beauty queen, “beneath this joke at Trump’s expense are sexist and fat-shaming ideas.” Research has shown that body-shaming is harmful to the targets’ mental and physical health. Yet we can’t seem to get past this idea that if someone doesn’t fit societal beauty ideals, sexual activity with them is inherently repulsive.

One unfortunate effect of this is that we consider conventionally unattractive people less credible victims of sexual assault or harassment. We wonder who would be attracted enough to someone to target them, as if these crimes were purely about sexual allure, as opposed to toxic abuse of power. Hence Donald Trump’s claim that he couldn’t have sexually assaulted journalist Natasha Stoynoff because, “look at her … I don’t think so.”

Body-shaming can also become a sort of code for calling out violations we can’t quite put our fingers on. We use words like “gross” and “disgusting” and “creepy” when what we really mean is “nonconsensual.”

The problem is, in part, that many people still have trouble understanding what “nonconsensual” even means. Eighteen percent of college students in a 2005 Washington Post poll said that if someone hasn’t said “no,” they’ve consented to sex. 32 percent of college men in a survey published in the journal Violence and Gender said they’d “force a woman to have sexual intercourse” if they knew they could get away with it, compared to 13.6 percent who said they’d “rape a woman.”

Those figures should prompt lots of serious discussion — none of which would have to do with supposedly unappealing bodies.

The Charlie Rose scandal, for example, could have been an opportunity to talk about how you can sexually harass someone without saying a word, because nudity without consent is harassment.

Relatedly, many criticisms of sexual predators are overly focused on supposedly repulsive sexual acts. Both the Louis C.K. and Weinstein accusations have given commenters the chance to pronounce that men masturbating in front of women is inherently “gross.” Never mind that plenty of couples enjoy mutual masturbation consensually.

This kind of commentary perpetuates sex negativity, spreading the idea that the only acceptable way to have sex involves a conventionally attractive cis heterosexual married couple in the missionary position with the lights off. Disparaging the act of masturbation in front of a partner, or displaying a penis, disparages those who enjoy these acts just as mocking the idea of older people being sexual shames the elderly. And that’s not progress at all.

“The notion of sex positivity doesn’t demonize any sexual desire except nonconsensual,” says Good Vibrations staff sexologist Carol Queen. “Why it would be any more problematic for someone to masturbate in front of a person than any other nonconsensual thing is ridiculous.”

Other malign effects of our misguided focus on standard notions of attractiveness

This kind of discourse feeds into other myths. Among other things, the focus on the conventional desirability of predators and their victims downplays assaults by women. After all, women are stereotyped to be more physically attractive and universally desired by men, leading people to wonder, when women in power take advantage of men, why wouldn’t he want it?

We’ve collectively bought into a fallacious binary that says women are the “fairer sex” — fundamentally more beautiful, gentler, less sexually aggressive, and less threatening — while men are ever and always poised on the cusp of violence and sexual depravity. So, sexual harassment at the hands of a woman is deemed not only more forgivable but almost laughable. Again: A sexual abuser’s looks and gender are irrelevant to the crime.

So. During my future family dinners, I’ll be using the recent allegations as a jumping-off point to talk about consent. I’ll point out that many couples enjoy mutual masturbation, but that masturbating in front of someone requires consent, just like any sexual act. I’ll explain that what makes Charlie Rose’s shower trick an unforgivable violation is not his age or looks, but that he’s using his power to deprive people of their consent before his nudity even entered the picture. When you aren’t given the chance to say “no,” you can’t say “yes.”

And that’s wrong — even if Charlie Rose looked like Ryan Gosling.

Suzannah Weiss is a freelance writer and editor currently serving as a contributing editor for Teen Vogue and a regular contributor to Glamour, Vice, Bustle, and other publications. She is working on a book about sex and feminism. This piece was adapted from an essay first published by the Establishment. Twitter: @suzannahweiss.

First Person is Vox’s home for compelling, provocative narrative essays. Do you have a story to share? Read our submission guidelines, and pitch us at firstperson@vox.com.