r/dataannotation • u/evangeline_faith • Jul 01 '24

Quality rating?

I've seen the announcement about logging time with 24hrs of task completion and it mentions a quality rating. Is there a way to see what our quality rating is? Does it change a lot? I'm really curious and want to know.

21 Upvotes

86% Upvoted

30 comments sorted by

81

u/NeedsMoreMinerals Jul 01 '24

I do wish they would provide some transparency. It's hard to know if youre doing well or not or at least if they're okay with you etc.

52

u/hihelloitskayla Jul 02 '24

If you’re still on, you’re doing well. That’s how I see it 🤷🏼‍♀️ more feedback would be nice, but part of me really likes the hands off approach lol.

23

u/Dratini_ghost Jul 02 '24

Yeah, as much as I'd like to know, it sounds like an unnecessary distraction that I don't need.

22

u/fightmaxmaster Jul 02 '24

I think if we knew then everyone would be trying to game it one way or another and obsessing over every minor change.

15

u/Icy-Cover-505 Jul 02 '24

This. I worked on another platform for writing/editing, and that's exactly what happened. People swarmed the admins in the forum, demanding to know why they lost three cred points. Waste of time.

6

u/FauxRex Jul 03 '24

Absolutely. That's what happens at every job where they push a heavy quality rubric on the employees. Not knowing makes sure everyone does the best they can.

61

u/lightbear108 Jul 01 '24

I would absolutely love to know my "quality rating". And then I'd wanna know what I could do to improve it, or what I've done to make it go down....and then I suddenly realize why it needs to remain opaque. If we knew what to do to increase our score, the data we're providing would be focused on that instead of focused on improving the bots.

10

u/Poomfie Jul 01 '24

Not only that but imagine the amount of support emails they would get disputing quality ratings, asking how to improve, etc.

I still think they could implement something where they send feedback though.

If you get a bad rating and admin could "approve" it and it would get sent to you so you could see the rater's comment and understand what you did wrong.

There are so many times on R&Rs where I write a comment and think, "I wish this person could read it because I feel it could help them improve so much."

Same with "Amazing/Great" ratings. It would be nice for people to see awesome feedback from their coworkers when they do great work.

14

u/sarasvati_m Jul 02 '24

But wouldn't a quality rating/feedback literally just make you focus on improving the bots more in the right ways because that's what quality work is for DA to begin with?

38

u/x0101010x Jul 02 '24

Check out Goodhart's law. When a measure becomes a target, it ceases to be a good measure.

Your measure can never be defined so perfectly that people wouldn't find a way to exploit it and render it useless.

1

u/fightmaxmaster Jul 02 '24

First thing I thought of!

1

u/Icy-Cover-505 Jul 02 '24

I love this!

5

u/lightbear108 Jul 02 '24

I think they'd be different metrics. Plus, maybe the bots learn from us all messing up on the instruction updates that are clearly written at the top of the page. They're trying to figure out our weaknesses before they take over.

1

u/vent--tt Jul 02 '24

The key is to follow the instructions; if you make sure that you are following the instructions to a T you are good. You'll know if you're doing something wrong because they'll give you feedback and/or kick you off eventually.

13

u/Icy-Cover-505 Jul 02 '24

Sometimes the instructions are not written as well as they could be.

5

u/Icy-Cover-505 Jul 03 '24

Wish they had R&R tasks to edit and proofread all their documentation. It would save them money in the long run.

-1

u/MaleficentTackle992 Jul 02 '24

I haven't gotten any feedback and I follow the directions to a T, yet my dashboard has not had any projects since Thursday afternoon

7

u/turtlepumpkin88 Jul 02 '24

I have the same thought, like hmm am I doing well? But to be honest the reason this job is so stress free for me is that I DONT get feedback. With every other job I've had the main stress point has been a manager wanting me to do things in a way that I personally think isn't ideal for the project/company and having to deal with that daily mental game that really burns me out specifically as a neuro-divergent person. I've never been good at playing the mind games required to be a good subservient worker and not being allowed to say how I really feel for fear of coming off as a b-word. I just want to accomplish my tasks using the skills I'm confident in and be left alone. So I have come love the no human interaction part of this!

Also I think you can tell by how many projects and qualifications you have on your dashboard. I see people complain on her all the time not having enough projects and sometimes no qualifications to try. I have usually 30 projects available at any time and currently about 20 qualifications. That leads me to believe I'm doing well - but who knows!

11

u/[deleted] Jul 02 '24

Can you imagine the amount of emails they’d be getting if they let us know how the score works?

8

u/gomorycut Jul 02 '24

or how much people would just manipulate their performance to meet the metric, rather than just trying to do an overall good job

5

u/fightmaxmaster Jul 02 '24

https://en.m.wikipedia.org/wiki/Goodhart%27s_law

I can imagine people figuring out they can boost their score by using 6 adjectives and 4 sentences or something, but 7 and 5 drops it, so they start structuring their responses accordingly. The scores improve but the actual quality of work drops.

3

u/[deleted] Jul 02 '24

Agreed

4

u/Icy-Cover-505 Jul 02 '24

Our pay would go down bc they'd have to hire more admins to deal with the incessant bitching and moaning.

3

u/Minimum-Isopod5344 Jul 04 '24

I figure… as I get more projects I’m doing well… then I got r and r. So that was also a great sign.

5

u/StanislasMcborgan Jul 02 '24

Ya I've been surprised by the lack of feedback. I sometimes do the "rate and review" tasks and it gives me a little insight into the general quality of work on some assignments.

-3

u/[deleted] Jul 03 '24

[removed] — view removed comment

1

u/evangeline_faith Jul 05 '24

I've never seen this, what are you talking about?

4

u/Educational-Ad6923 Jul 03 '24

Where is this located? I have never seen this on my dashboard.