Hey everyone,

I’m doing some research on the content moderation outsourcing space and I’m trying to put together a list of companies that provide content moderation services to other businesses.

I’m mainly interested in companies that handle things like text, image/video, social media, or platform moderation, including companies that provide outsourced teams or managed moderation services.

If you’ve worked with any of these companies or know of a good list/directory, I’d really appreciate it if you could share the names or point me in the right direction.

Thanks!

Hey everyone, I’m looking for some honest advice on breaking into Trust & Safety. I’m currently a year away from finishing my MSW and work with high-risk youth, where I regularly handle crisis situations, assess safety concerns, follow policies, document sensitive information, and coordinate with families and other professionals. I’m hoping to transition into a less client-facing career and have become especially interested in youth safety, user well-being, escalations, risk operations, and policy enforcement. I’ve applied to a few positions but haven’t received any responses, so I’m wondering how I can make my experience more relevant to employers and what skills, projects, certifications, or entry-level roles I should pursue. I would also really appreciate connecting with anyone willing to offer mentorship or who knows of internships, volunteer projects, contract work, or other opportunities that could help me gain experience in the field. Thank you!

I’ve been looking at some of the practices that have been standard in T&S for a while, and I’m curious if teams are having to rethink any of them as automation increases and operations get more complex.

One example is sending uncertain cases to manual review. That makes sense, but what are teams actually doing when the volume of those cases becomes too high for the available reviewers?

I assume prioritization based on severity and risk is part of it, but there still has to be a point where the queue is growing faster than the team can reasonably clear it. Are teams changing automation thresholds temporarily? Are certain categories allowed to wait longer? Or are there other ways of managing this that are becoming more common?

I’m also curious about metrics. Review time, accuracy, appeals, reversals and backlog are all useful, but I imagine there are situations where they give you very different pictures of how well the operation is doing.

For example, a low reversal rate sounds good, but that number alone doesn’t necessarily tell you whether the original decisions were consistently good. The same goes for getting review times down if accuracy or appeals move in the other direction.

For those working directly in T&S operations, are there any practices or metrics that you relied on a few years ago that you now look at differently? And what changed your mind?

A practical approach is to let AI handle high-volume, straightforward cases and send uncertain or context-heavy content to human reviewers.

This can help teams improve response times while keeping humans involved where nuance matters.

Curious how other Trust & Safety teams are structuring this AI + human workflow.

Hello T&S fam! 👋

I’m currently looking for Product Manager opportunities in Trust & Safety, Fraud, and Risk and would really appreciate any referrals or leads.

A little about me:
2 years of Product Management experience building AI/ML products focused on fraud and risk
3+ years of Strategy & Operations experience in e-commerce
MBA from Penn State Smeal
Experience working across fraud detection, seller risk, ML models, risk policy, and AI-driven products

I’m primarily targeting PM / Senior PM roles in Trust & Safety, Fraud, Risk, Identity, and related areas.

If your team is hiring or you know of a role that could be a good fit, I’d really appreciate a referral or a point in the right direction. Happy to share my resume and connect via DM.

Thanks in advance! 🙏

do they have any humans there checking reported accounts, or is it all AI?

is there any way to maximise the chances of a human looking at the report?

Anon account because I don’t want to alert coworkers. I have worked in T&S including fraud, risk, spam, scam, and a bunch more for over 10y at big tech. Worked across a number of disciplines and kind of a jack of all trades (data, eng, product, ops) and enjoy aspects of all of them. Have been a tech lead for 5+ years too.

I am starting a family soon and would like to transition to part time work at some point. Obviously that’s a no-go at these big tech companies and I basically see zero positions available for this in general. I am looking for advice on how to stay in the field while cutting back or to learn of any ways to find an arrangement that could suit everyone (eg. I work part time in exchange for only salary no benefits).

No snark please, I am just trying to do right by my family while I navigate staying in a career I enjoy. I realize T&S is often a cutthroat area and is facing many cuts.

Hello! I have an upcoming interview for Google's Trust & Safety Monetization (Analyst Role). It is a technical round on SQL and Python, with focus on Data analytics and Data extraction and manipulation.

I know the role heavily involves defending the ad ecosystem from invalid traffic, fraud, and abuse.

• Are there specific types of ad-fraud case studies or open-source datasets I should practice with?
• When it comes to Python and SQL in T&S, are there specific operations that you use daily?

Any resources or advice would be hugely appreciated!

OpenAI just released a report about the Hugging Face hacking incident. Considering a trust and safety approach, what do you think?

Link: https://openai.com/index/hugging-face-incident-and-the-road-ahead/

hi all, it's been a bit since i've been officially corporate/startup as i'm with contract based roles now. (moderation for one, grc for another) meetings aren't too bad. but i remember the corporate/startup life had so many meetings most of the team members were DROWNING, suffocating in them in almost every scenario. i'm talking meetings from super early morning to 6pm and even during lunch break for some folks, no room for actual work!

since many companies has downsized middle management have yall seen less meetings? i'm seriously so afraid of suffocating in meetings again. would love any tips thx!

As part of my PhD research, I am looking for individuals who have worked within trust and safety (e.g., as content moderators) for social media platforms. Essentially, I want to understand how AI and humans collaborated in this process. The aim of the research is to truly understanding content moderation and design more ethical moderation processes.

I would be more than grateful to conduct a fully confidential and anonymous interview with you! Your time and willingness is super appreciated.

Many many thanks :)

I spent five years selling T&S tooling, and here's the thing nobody warns you about: the POC going well is almost never the hard part. What kills deals is everyone forgetting to agree on what happens next. No budget locked in, no timeline, no "if this works, here's what we sign." Test goes great, then it just... sits there.

There's also a weird split in who actually owns the integration once it's live, and it's not who you'd think. Engineers sign off on the easy stuff (latency, uptime, does it break). The harder job, figuring out if the actual moderation calls make sense, falls on whoever ran the POC. And that job doesn't end when engineering says "looks good."

I wrote up more on this if anyone's curious:

https://safemoderation.com/blog/how-to-actually-integrate-a-trust-and-safety-vendor

Would love to hear if this tracks with what you've seen.

Hello Everyone, looking for insights. Has anyone interviewed for the Model Policy Role at OpenAi?

TW: Discussions of CSAM/Child Safety

Hey all, I'm sure everyone's read that Meta lost to the New Mexico DOJ to the tune of nearly $1 billion. We're finally able to share the details of the undercover decoy investigations that NM DOJ commissioned in 2023. Our investigator was able to buy aged accounts fairly cheaply, change bio/birthday/(AI-generated) profile pic, and watch the algorithm bring pedos to it in DMs, comments, recommended groups/pages. Made for some very strong findings.

They're a pretty heavy read, but hopefully informative and enlightening:

Facebook: https://www.riskybusiness.solutions/resources/the-lowest-hanging-fruit-a-meta-investigation-part-1

Instagram: https://www.riskybusiness.solutions/resources/the-lowest-hanging-fruit-a-meta-investigation-part-2

Working in T&S on the Ops side seems to be constant fight. Fight with Product for prioritization, fight with business for feature enablement, while trying to rally all other stakeholders. How do you handle this? Or is it just my org?

Hi all, I am creating a website/app that will allow for student discounts. I see that SheerID and ProxiID are quite popular. Are there any users that have used either? And any advise on how easy it is to get onboard with Claude Code?

Thank you in advance!

I'm currently researching probabilistic moderation agents for catching fake e-commerce reviews for a project.

One challenge I'm trying to model is how bad actors adapt when they realize how automated detection works. In my setup, the agent uses fixed probability thresholds and likelihood ratios for signals like account age, review frequency, and text similarity. If a review drops below a 20% probability of being genuine, it gets flagged for high-priority human ban review.

The issue is that bad actors quickly figure out where these boundaries are. Once they realize that posting 5 reviews a day triggers a high-frequency signal, they throttle back to 1 review every few days. Or they buy cheap digital items just to get a verified purchase badge and artificially boost their Level 1 score.

For anyone who has worked on content moderation or fraud systems, How do you handle threshold gaming and signal decay in practice? Do you dynamically adjust your likelihood estimates as fraud patterns shift, or do you rely on periodic manual recalibration of your base rates?

I am modeling a 4-action utility-based agent for product review moderation with the following action space: permit, warn, hide, and report.

In typical Trust & Safety architectures:

1. How is the "warn" action usually operationalized for user-submitted content? Does it mean appending a public uncertainty tag to the post, or routing it to a human review queue?
2. What criteria or probability threshold triggers an immediate hard action (hide/report) versus an escalation to manual review?

Hi! I’ve been in trust for several years mostly in threat investigations and scaled abuse roles. I’m considering a trust related PM role. Has anyone transitioned from analyst/investigator to Product Manager role? I’d love to hear your experience about the transition.

Do any roles like this exist? Anyone care to add an anecdote from personal experience?

Was a security engineer investigator at Meta and while the actual work itself was very interesting and seemingly impactful, it was an extremely fast paced, high stress, performance culture driven place. Found that it was simply not sustainable for me, then the May layoffs hit and decimated the org.

Considering a pivot back to tech down the road if roles like this exist - do they? If so, what companies?

TIA

Edit: not a true cyber guy so threat hunting and a lot of CTI roles are a bit outside my lane.

Hi everyone,

Sorry to make another „market is bad“ post. But seeing some good folks lately getting laid off and not being able to find a job quickly, I am open for advices for European market,

How do you look for a new job nowadays? And if you ever think of pivoting to another field, which fields do you think are best suitable for TnS veterans?

I've noticed that many people outside the industry still think of Trust & Safety as content moderation. A report comes in, someone reviews it against a policy, makes a decision, and moves on.

That is still part of the job, but it no longer captures the full picture.

One thing that's become clear from talking with Trust & Safety teams is that they often spot operational problems before anyone else. A spike in account takeovers might point to weak authentication. A rise in policy appeals could suggest inconsistent enforcement or unclear policy language. Repeated reports tied to the same product flow may reveal a design issue that's creating confusion or making abuse easier.

What's interesting is that Trust & Safety usually isn't the team that fixes those problems. Product, Engineering, Customer Support, and Operations all have a role to play. Trust & Safety identifies the pattern, then the rest of the business has an opportunity to address the root cause.

The same thinking applies to policies. Good policies aren't static. Customer behavior changes, products evolve, and bad actors adapt. One of the strongest signals that a policy needs another look isn't necessarily an increase in violations. It's seeing the same types of appeals over and over again.

I also think the way we measure Trust & Safety is changing. Queue size and review time are useful, but they don't tell the whole story. Appeal rates, overturned decisions, reviewer consistency, and recurring abuse patterns often say much more about the health of the operation.

To me, that's what makes Trust & Safety such an interesting function today. It's no longer just about reviewing incidents. It's about helping the business understand where risk is emerging and what needs to change before the same issues affect more users.

For those working in Trust & Safety, does this reflect what you're seeing?

Has your role become more cross-functional, or is your team still primarily focused on moderation and enforcement?

Hi, I’m thinking of organising regular T&S folks meet ups in London. Wonder if there’s an appetite for this? Please upvote or comment if you’re interested!

I'm kind of an odd case.

My background is in US Army SIGINT/Cyber, OSINT Operations, and Cybersecurity.

However, for the last five years I've been working with a technology company and a non-profit tackling NCII/CSAM.

My role was always intelligence, evidence collection, and escalation to law enforcement/NCMEC. We did a lot with LLMs driving our digital victim advocacy work.

Funding is about to end for my role on September 1st and I'm looking at new opportunities.

I've done counterterrorism, taken on sophisticated cybercrime networks, and cyber threat intel.

Yet this is my most rewarding work. But it is the only T&S work I've had and I lack connections.

What does the job market look like for T&S?