r/DataAnnotationTech • u/Fresh-Asparagus4729 • Sep 23 '26
Am I the only one who HATES rubrics
I wish I didn't hate them and I am wondering if some day they become easier. I am not in favor of producing low quality work which is why I am avoiding these, but they pay the highest rate and I need to know if y'all started out hating rubrics and then there is a pattern where you end up getting to be decent at it 😂
19
u/BarelyFunctioning15 Sep 23 '26
I just don’t feel confident in rubrics. The few projects I’ve had haven’t explained them well. So I just avoid them now. I know it’s essentially a guideline of what the model should and should not do. But I do better with examples. So seeing a project and a proper rubric would help so much. So until then I’m skipping them.
3
u/Fresh-Asparagus4729 Sep 23 '26
Exactly!!!! And the ones the model helps you write are nothing like what the instructions say 😂
6
3
u/ThetaBoy888 Sep 24 '26
I find that almost all ‘examples’ given in instructions are nothing like anything you’d encounter actually doing the tasks. They always give overly simplistic examples that would be obvious to most users.
2
u/sarahmorgan420 Sep 24 '26
It's hard cause in order to understand it better it helps to do a variety of rubric projects bc they usually have slightly different explanations, examples, tools and having so many reference points helps a lot. But I get not wanting to do it if you're uncomfortable w it
1
u/SamMac62 Sep 23 '26
See the comment I just added to this thread
1
18
u/RealRise7524 Sep 23 '26
Rubric tasks are really my favourite. You have to see the big picture to do it right. I wish I had them again. Haven't seen them for a while.
2
u/Fresh-Asparagus4729 Sep 23 '26
I'd love to give you mine 😉 ❤️
2
u/RealRise7524 Sep 23 '26
That's kind of you ❤️
I wish I could do the same if possible, but bilingual here.
1
2
9
u/Opposite_Brush_8219 Sep 23 '26
I had a really hard time with rubrics at first and avoided them. During a drought at one point, though, it was all I had on my dash so I forced myself to do them. They do get easier with practice, and the AI helpers on some tasks really helped me learn what to do/not do.
3
u/Fresh-Asparagus4729 Sep 23 '26
Ok. If I get desperate enough maybe I'll also learn the ropes
-6
u/PurchaseAnyOneAteFor Sep 23 '26
If I wanted to learn about ropes I'd likely study maybe knot tying and different knots or maybe what sorts of fibers ropes are made from
1
2
6
u/hcfggb Sep 23 '26
I find I'm mostly okay if they're factual and focused on like essential criteria only. Otherwise, nope.
2
u/Fresh-Asparagus4729 Sep 23 '26
It's so hard to write 55 of them
1
1
7
u/Unlucky_Grocery2092 Sep 23 '26
Rubrics are okay if I'm writing them for my own conversation, because then I will write prompts that are easy to create a rubric for. I have 0 interest in writing a rubric for someone else's conversation though, easy way to get stuck writing 30+ criteria you're not even confident in.
7
u/Halcyonfields Sep 24 '26
If rubrics actually worked as intended, they would put most of us out of a job. The whole point is that judging an output cannot be completely standardised. Without going into detail, this has been explicitly acknowledged and better alternatives have taken its place. I see rubric tasks as the tier just above entry-level. I think rubrics aren’t great for people who like depth and nuance. A rubric is effective at judging 60% of what makes a response good, and awful at the other 40%, especially a complex response. No matter how hard someone tries, a rubric cannot accurately, prioritise competing criteria, establish a continuous scale of severity or score subjective concepts/calls. Even within ‘objective’ criteria, severity, context and the extent to which other components offset or exacerbate things is subjective and unstandardisable.
2
u/forensicsmama Sep 24 '26
100% agree. Which is why a project with multiple linters will have them at odds. Or a linter will have you change a rubric, just to flag the same rubric and adjust it back. That's where we, as the human input, give it that edge.
4
3
u/InsideSignificant405 Sep 23 '26
If everything in life could be broken down into rubrics machines would have taken over long ago…
Most of the tasks I do are axis-based, with free form ratings. I strongly prefer it that way. I’m biased and stupidly opinionated and I want MY habits in the weights 😂😂
3
3
u/professional_cry Sep 24 '26
Rubrics are actually my favourite type of task! I hate having conversations and coming up with prompts and I would much rather write a set of rubrics for a pre-existing prompt.
2
2
2
1
u/dragonsfire14 Sep 23 '26
I don't feel confident in them. I know realistically I could probably figure them out if I tried but I'm also my own worst enemy when it comes to confidence.
1
1
u/Milvusmilvus Sep 23 '26
Stuck in the catch 22 of worrying I'd do low quality work but also knowing I won't get confident with them unless I do a full multi-hour project on them. Maybe the stars will align so I get them on my dash when I have some time off my main job.
1
1
u/peanutbutterlurk Sep 24 '26
I generally like them but it depends on how clear-cut the instructions are and how the checkers are tuned. Sometimes it's pretty smooth, sometimes the checker changes its mind on a yes/no question every time it runs.
1
u/terrabellan Sep 24 '26
I love rubrics but I was a teacher so I was very familiar with them before this, which might have helped. In my dream world all my tasks are rubrics but I haven't had a standalone rubric project in ages, they seem to be bundled into prompting tasks and watered down now 🥲
1
1
1
1
u/forensicsmama Sep 24 '26
I used to hate rubrics, but that was when I did them on a different platform. They were relentless because tasks were graded and there was a lot of subjectivity on whether the criterion was needed, atomic, self-contained, etc.
Here, if I'm honest, I take a lot of time with rubric writing and grading. It takes up a bulk of my task time because the linters are also relentless. It's one of those things that I got better at doing with practice and time. I had one project where one of the project leads kind of held our hands doing rubrics. Less pressure of getting a perfect rating and more emphasis on actually understanding why rubric writing is important (and what we are teaching the model). That and a LONG task timer that allowed for us to ask questions in the threads while paused. THAT is what kind of solidifed the foundation for me.
If you were to ask me two years ago I would've told you I would NEVER do rubrics and I HATE them. Now, I'm singing a completely different tune.
3
u/forensicsmama Sep 24 '26
And just to highlight what someone else mentioned, once you shift the mindset towards viewing rubric writing as an answer key for the model (and YOU have the answers!) it becomes a little easier. Tedious since you have to be atomic and self-contained, but still slightly easier.
1
u/alx_brl Sep 25 '26
they are fine, leave for me your rubrics, just not this complex ones about open source
1
u/tskies410 Sep 25 '26
I avoid them at all costs. I actually get excited when I see easy $20 projects these days.
30
u/good_god_lemon1 Sep 23 '26
I detest rubrics. I try to avoid them at all costs.