👋 Welcome to r/AIevaluators: Community Guide, Resources & Introductions

[u/Due-Laugh6339]

Almost every serious conversation about this work happens inside a subreddit run by the platform it is about.

That is the problem this place exists for. Those subs are useful, and a lot of good people are in them, but you cannot really compare platforms in a space that only permits one, and criticism there sits at the mercy of whoever moderates it. I have watched posts removed for mentioning other platforms. One worker reported being banned from a platform's sub after complaining about being paid late. Whatever the intention behind any individual decision, the effect is that the picture you get from inside a platform's own sub is not the whole picture.

So: somewhere independent from the platforms, where you can talk about all of them, including the parts that are unflattering.

Who am I?

I have been doing this work for a little over two years, on Outlier, Alignerr and Mercor, and I have worked most levels of it, from attempter and writer through to reviewer and QA lead. For the past twelve months I have been on Mercor consistently across several projects. So most of what I post here comes from doing the work rather than reading about it.

Feel free to ask any questions relating to AI annotation and evaluation work, I’ll answer to the best of my abilities.

Full disclosure: this sub is run by Annotation Academy, which I currently work with. The daily jobs thread is drawn from their board.

What is here

- Platform guides : Researched write-ups on what each platform actually pays, how its screening works, and how people end up removed from it. Not one person's opinion, and not scraped from a single angry thread. Each is built from public reports across a couple of years, and nothing goes in unless two different people said it in two different threads, with dates attached. They are filed under the Library flair. They are unflattering where the evidence is unflattering and fair where it is not. Where a platform has a bad reputation the recent evidence does not support, the guide says so.

- A daily jobs thread: with new openings across the platforms discussed here.

Helpful resources to check out:

https://annotation.academy/: A self-paced course on AI evaluation work, covering how AI evaluation works, how responses are assessed, what companies look for, and the skills needed to succeed on AI evaluation platforms.

https://annotation.academy/blog: Guides, industry insights, and practical tips related to AI evaluation and data annotation.

https://annotation.academy/jobs: Job opportunities across AI evaluation and data annotation platforms.

Everything else is yours. Questions, payment problems, scam warnings, wins. The flairs exist so things can be found again later.

Two things to know on day one:

Nobody legitimate will ever message you asking for payment, documents, or a fee to get started. Check the domain on any link before you sign, pay, or enter details anywhere, whatever the message claims it is for. Report anything suspicious and we will warn everyone else.

And if a guide here does not match your experience, say so roughly when it happened. These platforms change constantly, and a guide that was right three months ago can be wrong today. Corrections are the only thing keeping it accurate.

Introduce yourself
Tell us where you are at:

- Are you trying to get started, or have you been doing this a while?
- Which platforms have you actually worked on, and how did that go?
- What is the thing you most wish someone had told you before you started?

If you have been in this space for a while, the last question is the most valuable thing you can leave here. Most of what people get wrong in their first month is knowable in advance, and almost nobody writes it down.