ACM – Mitigating Negative Experiences Online

Introduction

Price of Social Media

We’ve come to accept this harm as the “price of admission” for social media. This is unacceptable. Parents deserve to know the true risks, and teens deserve effective tools to manage them.

Social media companies possess the technology and expertise to drastically reduce this harm while preserving connection. I led teams at Facebook that built machine-learning systems to evaluate billions of content pieces daily and created safety tools for teens. We published our results and proved it’s possible when safety is a true priority.

Recently, however, company leaders have shirked responsibility, investing in marketing that downplays the harm and creating flawed features that offer only an illusion of safety.

The root issue is an over-reliance on “objective” content moderation. While necessary, this approach is insufficient. In the real world, if a student reports harassment, the teacher doesn’t first determine if a specific “bad word” was used. The user’s experience is the ground truth of harm. We must build systems that recognize this.

The good news is that the engineering mindset needed already exists. We’ve successfully tackled spam and security. Online safety requires the same approach: capturing the right data and creating effective feedback loops. The key difference is that safety engineering must treat user feedback as an essential component of the system.

Harm needs to be understood from the perspective of the person experiencing it.

To reduce harm, we must first see it through the eyes of the teen. Testing with teen avatars quickly reveals concrete examples:

• Unwanted Sexual Advances: Comments or DMs asking for sex, sexual GIFs, or unsolicited intimate pictures.

• Bullying: Coded comments, contextual put-downs, and variations on slurs that are profoundly distressing to the target.

• Graphic Violence: Videos of severe accidents (e.g., broken bones) without visible gore, circumventing moderation policies.

• Eating Disorder Content: Images like a plate with a single piece of broccoli, or manipulated body shapes, recommended by the thousands.

• Self-Harm Content: Black-and-white images with poems about worthlessness, delivered in volume.

• Sexual Content Without Nudity: Graphic descriptions of demeaning sex acts using cartoons or enthusiastic storytelling.

Different Kinds of Harm, and Strategies to Understand and Reduce It

Harm on social media generally falls into three categories:

Relying solely on content enforcement creates a dual risk: over-enforcement (removing acceptable content) and under-enforcement (allowing harmful content). Most intense harm does not objectively violate policy. Sextortion often uses compliments; severe bullying can look positive to an outsider. Harm is contextual and about volume.

Therefore, most teen harm cannot be solved by more moderation alone. The opportunity for innovation lies in understanding the combination of contact/content, user experience, severity, and context.

Effective Reporting

Effective reporting is a well-implemented feedback system that is:

• Easy and rewarding to use.

• Able to capture what happened, where (context), and how bad it was (intensity).

• Designed to provide immediate support, independent of content moderation.

• Language Matters: Teens avoid the word “Report.” Changing the prompt to “I don’t want to see this” doubled or tripled usage. Options must match their descriptions of harm (e.g., “Someone is spreading rumors about me” instead of “Bullying”).

• Steps Can Be Helpful: Contrary to dogma, teens will complete multiple steps if each one has a clear purpose and benefit.
  By applying these insights, reporting completion rates soared from 10% to 82%, and 60% of teens felt better after using the tools.

Developing Safe Products

A safety framework for teens has four elements:

1. Effective reporting to understand what happened, where, and how bad it was.

2. Using that information to help the victim and protect others.

3. Giving feedback to the person who initiated the harm.

4. Measuring and monitoring the program’s effectiveness.

Lessons Learned from Developing Tools for Teens

The key lessons from improving teen safety tools are:

• Language Matters: Avoid “Report.” Use user-centric language.

• Context Matters: An unwanted DM is different from a public comment.

• Intensity Matters: Capturing severity is essential for providing appropriate, personalized support.

Creating Effective Safety Tools

An effective safety tool must have four features:

1. Prevention: It stops harm from occurring.

2. Resiliency: It is resistant to manipulation or workarounds.

3. Protection for Others: It captures data to prevent others from being harmed.

4. Ease of Use: It is on by default or requires one click.

Example: Unwanted Sexual Advances in Direct Messaging

A rapid, effective solution for unwanted DMs could include:

1. A prominent “help” button in early message threads.

2. Simple options to report: “It’s gross,” “They’re fake,” “They’re harassing me.”

3. Immediate, rewarding feedback: “We’ve blocked them,” plus resources like helplines.

4. Tracking how often a user initiates unwanted contact.

5. Sending a respectful nudge after multiple flags (e.g., “We’ve gotten feedback your messages were unwelcome. Please be respectful.”).

6. Escalating to feature limits if the behavior continues.

The current safety model—focusing on objectively harmful content—is necessary but addresses only a fraction of the problem. We must pivot to a framework centered on the teen’s experience as the ground truth.

Safety tools must be as resilient and easy-to-use as security tools, gathering data that protects both the individual and the community. We must “red team” these tools to test their strength.

Broadening our scope unlocks immense innovation: using LLMs to understand user issues, training AI on behavioral data, and integrating human experience directly into system design.

We must start by asking: What rate of unwanted advances is acceptable? What would I build for my own children? The answers to these questions must be the foundation for everything we do.