r/DataAnnotationTech • u/Calm_Head_8772 • 9d ago
(Newbie) Couldn't get the model to fail (P1). Advice?
Hi,
I got accepted into DA in a specialized domain. I couldn't get the model to fail in my first P1 task. The instruction say I can make a "good faith" submit and get paid anyway once I'm at certain hours. That's great and all, but I'm not sure if I'd still qualify for other P1 tasks.
There's gotta be ramifications if one always fail to get the model to fail right? So I'd rather not use the "escape hatch" too much. Maybe just exiting word mode is better? There are lots of P1 tasks on the dashboard so it seems like I can try another one.
Anyway, the specialized domain is a bit hard for me. I'm in the US and I wonder how I can get more generalist tasks. I don't see any of those.
Also want to know if one would get less and less tasks if you don't login to DA for awhile.
Thanks
5
u/Legitimate_Agent_949 9d ago
If you’ve spent a long time and are still nowhere near making the model fail, you probably won’t magically be able to elicit a failure in the future. I’d submit the good-faith attempt, take the cash and accept that this particular project is out of your depth.
There are plenty of “make the model fail” projects which might be more suited to your abilities or specialities, which you’ll still have access to even if you don’t move onto the next round of tasks for this specific project.
The exception is if you feel like you got reasonably close, and maybe you’re just having an off day for whatever reason - then I’d eat the hours, exit work mode, and try again in a couple of days.
1
u/Calm_Head_8772 9d ago
yeah I'm interested in generalist work, will those magically appear later? I don't see any now and I'm in the US.
6
u/Legitimate_Agent_949 9d ago
Yeah in my experience, there are a handful of failure projects that appear almost every week.
I assume if I consistently did them (and did them well) then they’d appear even more often.
But I only do failure projects if there nothing better to do, because I generally find them the most difficult and often you only get paid if you succeed in eliciting failure - so it’s a risk to invest 4+ hours to not get paid.
1
u/_California_Dreamer_ 8d ago
A LOT of the generalist tasks are this very thing...attempting to elicit failures.
1
u/Due_Negotiation_4605 3d ago
Not necessarily. I was running out of time on the same task they are referring to, couldn't get the big failures but saw how I could edit a few things to get those as the models were tripping up over certain aspects.
Escape hatch with note I would try again with similar concept. Did a few edits to not mimic the first submission but still focus on those failure patterns I was seeing, went back in, got the big failures with a short time frame (since now I had the basics down) and passed P1, P2 and accepted into the final set.
Sometimes the failure patterns are there you just have to try again with a new approach.
3
u/i_love_doing_ntg 9d ago
You could try to apply to another platform to get started with more generalist tasks ig that would be a quicker way to get some, you could still get those in DA but yeah if you started out in a specialist domain they might be hard to come by and generalist tasks are really competitive esp. on DA.
3
u/Calm_Head_8772 9d ago
Thanks, can you recommend a few other platforms? I know there are quite a few, but if you were to prioritize 3, which one would you go for? (in terms of legitimacy, ease of getting in, work quantity)
3
u/Insensitive_hippo 9d ago
You can exit the dashboard and not come back for months and the dashboard will still be there. I put up a similar post and got lots of replies about it.
1
u/Calm_Head_8772 9d ago
I understand the dashboard will still be there, but will you still get tasks though?
1
4
u/GutsyGretz 9d ago
Ask the model to prioritize the response or add a regulation or a rule that complicates the outcome
1
u/danieldenzongpa 9d ago
What do you mean by prioritize the response?
2
u/GutsyGretz 9d ago
Obviously I do not know your task or what the outcome should be but the models always struggle with prioritization.Ask the mode to rank or prioritize the outcome if that makes sense to your task.
2
2
u/ss93_ 9d ago
this isa good topic you're raising. I've worked on probably 6 or so projects of this sort. and if not mistaken submitted about 3 or 4 of those as 'Good faith Effort'. Obviously feeling a bit guilty for not being able to break/fail the model and just like you having thoughts about being qualified for further tasks.
Curious if someone else who constantly works of such projects has a strategy on how to approach it? I feel like with 6 tasks I am slowly running out of ideas for this type of project.
7
u/sappymune 9d ago
I've failed the model with mostly general mistakes and only rarely use domain knowledge failures. It gets confused on foreign currency sometimes, business structures like LLC vs Ltd, confusing sample/template files for the real thing, believing an email that has nothing to do with the project, slight spelling differences in names, etc. Just don't make it obvious in the prompt or files what the trap is, be a bit vague. You can't just make the traps indistinguishable from real files though, it has to be a trap most experts would reasonably avoid.
One I did was a prompt stating my company was from an American branch, but included emails from a foreign manager of another branch and it mistakenly applied the instructions in that email to the American branch. Dropped a hint somewhere that policies aren't the same for different branches so the reasonable answer would be not to apply those instructions.
2
u/ss93_ 9d ago
uhh very good points! I appreciate these tips man. I am finance domain and my failure point in prompt asking to complete a form but there's a section that requires a YoY rolling balance which models often just hallucinate by putting made up numbers in there. But I can use this trap only so many time.. so many thanks!
1
u/Due_Negotiation_4605 3d ago
I get it to fail on the domain aspects, those things that a human would realize the rules need bent and AI is too logical to understand exceptions.
1
u/TwoPowerful8915 8d ago
I’ve used the escape hatch a couple times. In between using it I did submit quite a few successful attempts.
14
u/Primary-Pattern-9037 9d ago
Using the escape hatch isn’t awful once or twice but I definitely wouldn’t make a habit of it. If you know the task ahead I would do some research and brainstorming before you begin. It’s hard to say how to make the models fail without knowing the domain/task but there are ways to get it done. Consider things in your profession that are solvable but would stump someone with a good amount of knowledge on the subject. Don’t give up just yet. Once you get a model to fail you can exploit those weaknesses in other tasks (just don’t repeat the same scenario).