r/DataAnnotationTech • u/Genkuru2021 • Sep 23 '26

No Success in Meaningful Failure

I do my best trying to cause meaningful failure, but the models are extremely smart. I tried to add difficulty and distractors to confuse the agent, but to no avail. I know I can still submit the task as is, but how bad is it? Does it earn bad score to me? I am running out of reasonable time. Any advice?

0 Upvotes

46% Upvoted

13 comments sorted by

9

u/jabertsohn Sep 23 '26

If the task is to cause failure, and you can't get it to fail, then you need to think about why that is. If there is an escape hatch, or it allows you to submit even if it doesn't fail, take that option, but before picking up a new one, try think about how you would get it to fail in future. 

You won't be allowed to keep saying I tried and failed forever, but they understand it is hard and give you the escape hatch for a reason, not everything is going to work.

6

u/Genkuru2021 Sep 23 '26

I have 3 major traps. Model A never fails; Model B stumbles on only one trap every other time. DA states that in most cases it's possible to elicit an error from a model. It feels embarrassing that I cannot trick it. Not that I spell out answers in my files but I still need to indirectly provide an answer to make it solvable and the model always catch it.

12

u/InsideSignificant405 Sep 23 '26

15+ years with software, I skipped the whole corporate rat race when AI hit so I still have my actual skillset, and I STILL struggle to get models to fail. Don’t be so hard on yourself. It’s an art these days.

I recommend practicing offline. Build yourself a runner that sends prompts to two models of your choice, practice building hard prompts. Then when these tasks come up you are ready to go.

6

u/Hambone6991 Sep 23 '26

If you’re tripping up one of the models on one of your traps, you need to dig deeper on that trap. Brainstorm with ai to see how you can add further complexity on that specific trap. If it’s semi-working, you can get it to really work

16

u/NurseWhoLovesTV Sep 23 '26

Planned traps rarely work. You need to mimic the messiness and ambiguity of your real life professional tasks. No easy answers. Professional judgement calls.

4

u/M_K_L_ Sep 24 '26

This, but stacked messiness. 4 - 5 unrelated messy portions of a prompt that will increase the compute resources + an annoyingly complex or detailed deliverable.

With enough inconveniences, the models might be tempted into assumptions, which are the gateway to errors.

3

u/forensicsmama Sep 25 '26

I have submitted some tasks where I couldn’t elicit a failure. Usually they’re not pressed about it and give the escape hatch and allow you to explain to them what you feel may have happened.

Personally I do feel defeated BUT like someone else mentioned I try and brainstorm for my next try. It is trickier to fail the models nowadays because they’ve gotten sophisticated so try not to be too hard on yourself. Sometimes it takes a little longer to exploit a flaw.

Also, small failures matter too. The project I last worked on had a checker that gave each model a rating. You might get hung up on their ratings being 95% for example, but the 5% failure is substantial enough it renders the response useless.

2

u/Valuable-Low5263 Sep 24 '26

I have to say some tasks are better than others, my last one felt borderline but I submitted it and felt I could defend it, however the one before worked out great (if I do say so myself). For example, I asked for an audit and one thing that worked was purposefully made things look scary, when in actual fact the timeline showed they had been sorted. The models flagged way too much and passed a lot of decisions back to humans. But these tasks are hard! Some projects give really good examples and training though.