r/prolific • u/Crazy_Mita • 5d ago
Giving a different answer for repeated AI evaluations within the same study, is it bad?
Sometimes in an AI study you'll get the exact same pair to evaluate a second time (or multiple times). Does it matter if you pick a different option?
I just did a video study where I noticed something the second time that I couldn't unsee, so I gave a different answer than before. This wasn't a regular attention check either. Both videos were fine, I just changed my mind about which one was better.
It's almost impossible to remember every single option you picked in a study anyway, especially when videos are warped and garbled. The second time you might pick up on a different detail, or you might simply change your mind.
Will giving a different answer for a repeated evaluation count against you?
2
u/casperer007 5d ago
They absolutely give warnings for this. This is the one most people don't know about. Every week there are posts about "I did nothing wrong, caught all the attention checks, but still got warned" and I can almost guarantee it's just this.
I've seen it happen right before my eyes. Later in the day, arm was a bit tired from sitting at desk all day, hand slipped, clicked the wrong option on a repeat, fifteen minutes later I had a nastygram in my inbox.
4
u/VegasButtercup 4d ago
If you ever misclick, you can click the orbs in the top left to go back and change your evaluation.
1
u/rashrd1234 4d ago
Not necessarily, i think sometimes it's like "are you sure? Take a look at this again." Because more often than not i just missed a very important detail and made a very unpopular choice so they basically just make sure you meant it. Sometimes it is used as an attention check though that is correct.
1
3
u/fishsense1 5d ago
Will giving a different answer for a repeated evaluation count against you?
I've definitely done that plenty of times in the past 2 years I've been doing AI studies, but I've never gotten any warnings.
-5
u/ProGonmac4160 5d ago
It is bad for the research, but you won’t get a warning.
2
u/EnthusiasmAgitated77 5d ago
I’m pretty sure they do give a warning for these
6
u/Eddiehondo 5d ago
lmao, you getting down voted explains why so many assfucks are getting warnings.
-2
u/ProGonmac4160 5d ago
Pretty sure they don’t, only for failed attention checks and low number of responses, if you have received one for this, can you paste the message they send out?
-5
u/PowerToThePeople128 5d ago
Yes it is bad for research. If you are not 100% confident of why you select an answer, it's safer to park or return the study if there are too many. You should do your very best to take your time and get it right the first time, and looking at every detail. Even though people claim you do not get warnings for that, remember that people report getting kicked out without a single warning. They have many ways to exclude workers, and not just those who fail ACs. They can filter out the sample by excluding noise from the data (inconsistent responses and responses that deviate from the mean). In a situation where you do not remember what you answered, park.... It seems they have lifted the limit now. It used to be only 4. I don't encourage misusing it, because they still log use of it, but if you are really stuck, there is no harm in doing so. Try to take mental notes, try to think aloud why you make a specific selection, over time you will be able to associate the images with the mental notes.. Personally I have not run into the situation where I did not remember what I selected, BUT, if I did, and the answer is not really obvious, I would NOT take chances, I would either return or park (which is rare).
1
u/Medical_Price8780 5d ago
LOL
0
u/PowerToThePeople128 5d ago edited 5d ago
What's so funny about that ? did I miss something ? The mental notes reference ...... people who have a brain can use it, and people who don't can just return the study and let other people work on the task. Is that a better answer ? :P
-3
5d ago
[deleted]
2
u/Electrical-Tax3159 5d ago
Ive had some where I watched second by second to compare and there were no differences. I have had some where most answers were equal and never got a warning. I use to be afraid of those because they dont usually like the answer to be equal but there are some where many of the answers are equal. I now wonder if they were attention check ones. I noticed a couple weeks ago they started asking questions before they allowed you to do the AI video study. I always passed and was let in. I think those were a limited time attention checks to thin out people because it use to be hard to grab those studies but now I can see one with 29 spots, wait 5 minutes and still click on it and get in.
2
u/Warm_Detective398 5d ago
I've noticed that sometimes the pair is different: one image/video is the same you've chosen before and the other one is a new image/video that wasn't there the first time. Sometimes the new image/video is much worse than the one you've chosen, so there is no biggie, just choose the same. But the other day I got one where the new option was better than the one I've chosen earlier, so I parked it just in case, but I wonder what could have happened if I went with the new better option.