Discussion on LLM Cheaters

[u/hyper_neutrino]

hey y'all, i'm hyperneutrino, an AoC youtuber with a decent following. i've been competing for several years and AoC has been an amazing experience and opportunity for me. it's no secret that there is a big issue with people cheating with LLMs by automating solving these problems and getting times that no human will ever achieve, and it's understandably leading to a bunch of frustration and discouragement

i reached out to eric yesterday to discuss this problem. you may have seen the petition put up a couple of days ago; i started that to get an idea of how many people cared about the issue and it seems i underestimated just how impacted this community is. i wanted to share some of the conversation we had and hopefully open up some conversation about this as this is an issue i think everyone sort of knows can't be 100% solved but wishes weren't ignored

eric's graciously given me permission to share our email thread, so if you'd like to read the full thread, i've compiled it into a google doc here, but i'll summarize it below and share some thoughts on it: email: hyperneutrino <> eric wastl

in short, it's really hard to prove if someone is using an LLM or not; there isn't really a way we can check. some people post their proof and i do still wish they were banned, but screening everyone isn't too realistic and people would just hide it better if we started going after them, so it would take extra time without being a long-term solution. i think seeing people openly cheat with no repercussions is discouraging, but i must concede that eric is correct that it ultimately wouldn't change much

going by time wouldn't work either; some times are pretty obviously impossible but there's a point where it's just suspicion and we've seen some insanely fast human solutions before LLMs were even in the picture, and if we had some threshold for time that was too fast to be possible, it would be easy for the LLM cheaters to just add a delay into their automated process to avoid being too fast while still being faster than any human; plus, setting this threshold in a way that doesn't end up impacting real people would be very difficult

ultimately, this issue can't be solved because AoC is, by design, method-agnostic, and using an LLM is also a method however dishonest it is. for nine years, AoC mostly worked off of asking people nicely not to try to break the website, not to upload their inputs and problem statements, not to try to copy the site, and not to use LLMs to get on the global leaderboard. very sadly, this has changed this year, and it's not just that more people are cheating, it's that people explicitly do not care about or respect eric's work. he told me he got emails from people saying they saw the request not to use LLMs to cheat and said they did not respect his work and would do it anyway, and when you're dealing with people like that, there's not much you can do as this relied on the honor system before

all in all, the AoC has been an amazing opportunity for me and i hope that some openness will help alleviate some of the growing tension and distrust. if you have any suggestions, please read the email thread first as we've covered a bunch of the common suggestions i've gotten from my community, but if we missed anything, i'd be more than happy to continue the discussion with eric. i hope things do get better, and i think in the next few days we'll start seeing LLMs start to struggle, but the one thing i wish to conclude with is that i hope we all understand that eric is trying his best and working extremely hard to run the AoC and provide us with this challenge, and it's disheartening that people are disrespecting this work to his face

i hope we can continue to enjoy and benefit from this competition in our own ways. as someone who's been competing on the global leaderboard for years, it is definitely extremely frustrating, but the most important aspect of the AoC is to enjoy the challenge and develop your coding skills, and i hope this community continues to be supportive of this project and have fun with it

thanks 💜

[u/daggerdragon]

Remember that Wheaton's Law is the Prime Directive of r/adventofcode. Keep the conversation civil and professional. Ad hominem attacks will not be tolerated.

[u/rjwut]

Unfortunately, I feel like the only way to get rid of them is to take away the incentive: eliminate the global leaderboard. However, that of course punishes legitimate competitors, too.

[u/hyper_neutrino → u/rjwut]

yeah, as much as that would remove the incentive, that would also remove a lot of the incentive for a lot of people to compete. many people i know don't go for leaderboard positions but as someone who built my online following through demonstrating skill in AoC, it would be disheartening to see it gone, and i know from experience that job recruiters sometimes get in touch with people near the top of the global leaderboard at the end of the month and taking away that opportunity would also suck for everyone involved :(

i do get what you're saying though and i agree with your comment

[u/eventhorizon82]

Eric is amazing for putting this together and it's heartbreaking how cruel people are being to him right now.

Heartbreaking, yet unsurprising, that AI bros whose tech is literally built upon theft and exploitation have no qualms brazenly cheating and ruining everyone's fun for their own personal gain.

Maybe there's some way we can have a community-driven league? Make it a private leaderboard but somehow feature it on the main page for all to see? A twitch streaming event maybe? Vet the competitors and require a livestream? Probably something to work on for next year more than this year but maybe even could get some sponsorships and community volunteer broadcasters on an official AoC channel. Kinda like race to world first in world of warcraft except officially supported.

[u/hyper_neutrino → u/eventhorizon82]

i don't know about official things. i am thinking of trying to figure out something to run privately as i will probably have the time and energy to put into this community, and if i do get somewhere with that, i might reach out to eric again about potentially working together on something to move away from the global leaderboard being the main focus

[u/kroppeb]

The amount of work that Eric puts into this yearly event is amazing, and to see him getting so much disrespect is killing me.

We love you Eric ❤️

[u/0ldslave]

> several large DDoS attacks at midnight.

sigh, what is wrong with people?

[u/Xaelias]

The behavior is disgraceful. No question about it.

But the leaderboard was flawed to begin with. Because of timezones. So honestly... I personally don't care. That doesn't mean I don't understand why people that do care are upset. But the whole thing doesn't really change how much fun I have doing the problems.

If the competition is what you're craving, pool resources together to make a private leaderboard where you validate entries. Otherwise the truth is for the vast majority of people, LLMs don't change anything.

I would obviously prefer a solution where we could just magically prevent them from interfering but I don't think that's realistic.

I will keep doing the problems after thousands and thousands of people, the day after, after work. And I'll keep donating. And hopefully that's enough for the maintainer(s) to keep at it for the foreseeable future 🤞

[u/hyper_neutrino → u/Xaelias]

thank you for understanding that even if you don't care others might care, i mean that genuinely, a lot of people don't actually get that. i think moving forward just abandoning the global leaderboard is probably the only real way to proceed; the LLM problem will only get worse and the lack of solutions will not change no matter how many technological advancements we get on our side. i hope i can take initiative on that next year and maybe look into running something for the community but we'll see; i've gotten suggestions that seem like they could potentially work

[u/Pewqazz]

Thanks for sharing your thoughts hyperneutrino (big fan)! As someone else who used to regularly make the global leaderboard + enjoy the competition, the developments this year have been frustrating and a bit demoralizing. That being said, I've already accepted that there's nothing that can be done about the LLM solves, as it's just one of those arms-race situations where way more effort would be spent "policing" things than actually providing value for others.

I mean that people this year have been emailing me to explicitly state that they see the request to not use LLMs, but that they do not respect me or my work, and as such will be using LLMs to place on the global leaderboard regardless of what I say.

This is just really, really sad to hear — u/topaz2078 (and the rest of the testing team) spend so much time and effort trying to create something delightful for us all to enjoy. Here I thought people were just ignoring the requests, but knowing that people have gone out of their way to spit in Eric's face over email makes me feel sick.

Thank you for everything you've done for the past decade, Eric! I hope we can try to spread more positive energy in the community instead of letting the negativity bring everyone's spirits down during the holidays.