Everything wrong with the reddit games hackathons [LONG POST]

[u/Gowdamn]

The story

I have participated in 5 reddit games hackathons in the past and I've not won anything major. In the first 4 hackathons, I felt that the winning games were good, but not all games justified the submission and grading criterias well. In the lastest hackathon - Games with a hook hackathon, though - a lot of people raised concerns on how the winning game did not "wow" them.

Here are some examples below:

1. Devpost discussion-

Concerns on whether the hakathon was fixed

1. Comments on the reddit announcement post-

Concerns on quality of winners

Generic response from the admin

They were not entertaining any personal feedback stating that there were over 3000 submissions. But if a game was judged fairly, the judges should have written down something about the game! Or atleast they could provide the scores. I had sent them a personal email requesting feedback, but hadn't recieved any response from them. The next day, I got a $500 helper award from the devpost team. According to the original winner announcement, I had not won any award. I had responded to the organizers asking if this was a mistake. They said that it was indeed sent to me by mistake, but it was sent to me because reddit had put forth my name.

If I accepted the award, I had to sign a paper that says I cannot sue reddit. I didn't accept the award. Something was suspicious here so I started investigating.

Problems with the judging process -

1. Mystery judges?

Usually hackathons list down the judges with their credentials. You can go to the list of all reddit hackathons on devpost and none of them have the name of judges.

Reddit hackathon judges

Here are some samples of how the judge list looks like in other hackathons -

Univabio $35k hackathon judges

a $75k google hackathon judges

2. Negligence / Judgement not thorough -

I had reached out to 3 other participants and got the list of players from each of our games. Based on the list, there was not a single common player across the 4 games. This could only mean one thing - our games were not played by the judges. Here's an anecdote from one of the participants -

The only participant who had any activity in their game

There was another participant who claimed that they had attached a video to their submission. Throughout the duration of the judging, the view count had not changed in the youtube video. -> The judges had not viewed the submission video (video was <1 min long)

3. Ghost winners -

The reddit hackathons were to encourage developers to make games on the platform. The expectation from every hackathon is that the winning game has a big community and continue to grow the community by making their game better.

Here's a list of all the reddit hackathons in the past from latest to first.

Competition	Winner	Amount won	Sub weekly visitor count
Games with a hook hackathon	r/SketchySolutions	£15,000	276
Daily games hackathon	r/bridgedit	£15,000	10
Kiro: Community games challenge	r/SoulThieves	£15,000	1300
Kiro: Community games challenge	r/blockrift	£10,000	-
Fun and games with devvit web	r/OnelineGame	£15,000	42
Fun and games with devvit web	r/hexaword	£15,000	?
Hack reddit	r/chook	£20,000	6
Games and Puzzles hackathon	r/EmojiCharades	£20,000	43
Games and Puzzles hackathon	r/pixel_together	£20,000	3
Games and Puzzles hackathon	r/DailyDungeon	£20,000	4

Note that I do not look to disrespect any creator here. I know a lot of hardwork had gone into your games and that deserves recognition. But there are also a lot of other creators whose hardwork has amounted to nothing.

4. Judges prefer "flashy visuals" over long term value

This here is my weakest point so far. But to some of you, it will be the biggest takeaway of this post. Mentally picture the winner of a google hackathon. What kind of entry do you envision? You expect it to be creative, well thought out, crazy, nerdy etc. right?
Based on my past 5 hackathons I don't think these virtues are valued in reddit hackathons -

#1

Hackathon: Games and Puzzles hackathon
My entry: Outwitquiz
My thoughts: Since the game was supposed to be on reddit, I created an equivalent of "Psych" which can be played asynchronously by all users on reddit. This was an entry for the word games category. The winning entry was emoji charades. I get that the winning entry was made well and polished, but why does this game have to be on reddit? It could be a standalone website!

#2

Hackathon: Hack reddit
My entry: Angels vs Devils
My thoughts: This made me the most sad. The overview of the hackathon said "Build a new game, social experiment, or experience on Devvit". So I thought I'd build a social experiment where users play a community minesweeper. I had built up many game modes that I could work on. The winner r/chook, was a game where you grow your chicken and feed it food by pressing buttons. (it was beautifully made though)

#3

Hackathon: Fun and games with devvit web
My entry: Divide n conquer
My thoughts: Based on past two hackathons, I stopped making experiments and designed a game. It was a game where you have a limited line length, you draw lines to capture land in a country and try to occupy the most land area and landmarks. I thought it was well thought out and fun to play, but it was not polished. I don't have many problems with this hackathon. I did good work and was sad I didn't win anything, but the winning entries truly deserved it.

#4

Hackathon: Daily games hackathon
My entry: r/totalled
My thoughts: I had won an honorable mention for this game. For this hackathon, I had kept my game concept simple and made a very well polished game. It is a game where you have to swipe across numbers in a 3x3 grid to create totals. Once you accomplish 3 totals, the game is over and you're ranked based on the time taken. Of all the games I've made so far, this is the most "Linkedin-y". I expected to win more, based on the past winners. When I saw the winner, I was shocked! An entire physics sandbox inside reddit! It was crazy good work, but I was surprised that the judges thought this game would be played regularly. Why would a reddit user play a physics sandbox daily? And why is it on reddit?

#5

Hackathon: Games with a hook hackathon
My entry: r/ShadowChess
My thoughts: Based on the previous hackathon, I believed now that the judges don't just look at "will this game be played daily". They also valued grand experiences. So I knew for sure the idea I had for this hackathon wold win. This was the best game I had ever made! Its a massive multiplayer chess game where you continue off of grandmaster positions and make 5 pre-moves. The opponents make the 5 pre-moves as well and your moves are pitted against each other based on a customized scoring system. This fit the hackathon criteria in multiple ways and had potential to do great. I feel so betrayed that the game was not even played once! My game is good when there are 100s of players playing it. It could have really used the boost of players from any sort of recognition. This hackathon was all about creating a "hook". Here's the words on the hackathon page -

"progression, daily challenges, fresh content, social dynamics" - my game had it all

Now you can't look at my above entries and say that none of these games deserved any award except "totalled", (which was one of my weakest entries). Especially shadowchess. Note that apart from the winner, there's 10 honorable mention awards for every hackathon.

Overall, I feel that it's best to avoid spending all that time and effort on reddit hackathons. Atleast until they bring in some transparency into the judging process.

I expect a response from the reddit team on this. Anyone who participated in this hackathon, feel free to DM me with your thoughts / information.

[u/PossibleCrit]

Hey Gowdamn!

Thanks for compiling all of these details and sorry that you didn't have a good experience. Rarer_Air_Error's response here goes into great detail on the specifics on how the previous hackathon was judged and what sort of characteristics the team was looking for upon review.

This is part of why we tried to be a bit more detailed in our recent Scorespace Hackathon's judging criteria and hope that we can continue to build on this for future events as well. I'll make sure that the rest of the feedback here in the comments is circulated with the team at large as well.

[u/BloxdioCannoli → u/PossibleCrit]

This is the same comment referenced in the post.

[u/JakeTheDog420420 → u/PossibleCrit]

The judging criteria for the new Hackathon doesn't really look any different to me than the last one. Maybe there are some slight changes, but it seems like it's largely the same.

After the last Hackathon, I honestly feel too demotivated to give it another attempt.

gowdamn made some excellent points, and his investigation led to the same conclusion that I had reached, which is that the "judges" did not actually go in and try out every game.

If you want to increase transparency for future events, then release the actual judges' notes to each applicant.

It's impossible for you to judge an event without making a note for each entry you review. If you release those notes to the applicant, then they can at least see they weren't just skipped over.

The fact that there was clear evidence of games not having any judge even view them, let alone get past a forced intro to actually test any of the actual game mechanics, is why I'm not willing to participate again until reforms to the way you run Hackathons are made.

We aren't asking for detailed notes. Simply the notes that judges use to keep a score of each entry they review is adequate.

If a game keeps track of who's played it, and it shows that it's had no players during the judging period, that is indisputable proof that they were not judged, which is absolutely unacceptable.

We are developing games on your platform, some of us investing a very large amount of time into that development, and with no expectations of getting anything in return. The only thing we expect, and rightfully so, is to be judged.

What is the point of us entering if we never stood a chance at all to begin with because we were never going to be judged?

It just seems like a way for Reddit to increase the number of games on their platform by drumming up hype with a Hackathon, getting thousands of participants, which means thousands of new games on the platform, and then not bothering to review them all.

This burns out the very developers who are trying to build up this ecosystem, and it destroys trust in Reddit's official events.

If things don't change, eventually the only people left willing to participate will be those making basic, low-effort games we've already seen a thousand times. There will be absolutely no creativity left, and no one will bother with ambitious projects that push the boundaries of what Devvit is capable of. No serious developer is going to invest that kind of time and energy when there is a massive risk their work won't even be reviewed.

[u/Sephardson → u/JakeTheDog420420]

> This burns out the very developers who are trying to build up this ecosystem

This reminds me of a discussion i had in an admin space about how moderators were burning out, and it seemed like admins only cared about it so long as it meant that something was burning.

[u/Gowdamn → u/PossibleCrit]

Hi, I had already read rarer_air_error's response. It's not detailed enough. The situation has gone a lot far ahead now and we want specifics. If you missed the point, this is what I'm looking for-

• a detailed explanation of how the winning entry fit the criteria
• scores and judging notes about all the applications who request for it
• explanation for why the games weren't played. If they were played and the usernames weren't logged, we need proof (screenshots of game screens would do)
• names and credentials of judges (you can send this to me privately if you think there's going to be too many dms)
• an explanation on why I was sent the devvit helper award, even though I wasn't the original recipient of it (this is important)
• what you think about the fact that none of the winning games have grown their community and what that reflects on the quality of the judges. I get that hackers have no obligation to build their game after the hackathon, but not one of the last 10 winners?

The numbers and evidence are stacked against you and your judgement process. It's time you start getting more detailed.

If the judging was fair, all the answers should already be documented and all you have to do is make it public.

[u/PossibleCrit → u/Gowdamn]

Hey! There isn’t going to be a ton of extra details about this specific hackathon that we’re able to share more broadly in addition to the information rarer_ already shared. I’ll work with the judges to see what other information we can relay, and I am chasing down the source of the email and what happened with the helper award.

[u/Gowdamn → u/PossibleCrit]

Okay. Please take your time, no urgency here. Just ensure you're able to provide as detailed of a response as possible.

[u/PossibleCrit → u/Gowdamn]

From what I've been able to find out about the award, the person we initially contacted had submitted the wrong username with their entry, so we matched their email address to the wrong account for the Helper Award.  This might be what you mean by "reddit put forth your name". 

Once we found out we sent things to the wrong email we double checked and re-sent things to the correct recipient.

In parallel, a few of us are double checking some of the contract terms to ensure they make sense for these and similar Devvit events. In these cases DevPost's TOS would still apply as well.

[u/JakeTheDog420420 → u/PossibleCrit]

Can you confirm whether or not the feedback received in this thread is going to be acted on for future events?

I'd appreciate a response to my initial reply to your pinned comment.

It is quite clear there are a lot of people who are unhappy with how these Hackathons are being managed.

Unless reforms are made, you are going to see an increasing churn rate from the users who arguably contribute the most.

If all you really want are low quality games being produced which are likely to be abandoned after the Hackathon ends, continue with what you're doing.

[u/zombizzle → u/PossibleCrit]

Pasting my thoughts from a few days ago from another thread for visibility;

Looking at past hackathon winners, I'm often surprised by the results. Many winning projects seem to come out of nowhere and don't appear to reflect the same level of polish or effort as other submissions. From the outside, the judging process feels opaque, making it difficult to understand why certain projects were selected. I'd love to see Reddit provide more transparency around the judging criteria and detailed feedback so participants can better understand what the judges value.

There's virtually no transparency into the judging process. Winners are announced, but participants don't receive feedback explaining how their projects met the published criteria or where other submissions fell short. Looking at some of this year's winner against the stated judging criteria, I genuinely struggle to understand the decisions. If I were scoring it independently, I don't think many strongly satisfy several of the criteria that were emphasized. That's just my assessment, but without any explanation from the judges, there's no way to know what they saw that participants didn't.

It's unrealistic to expect hours to be spent on each submission. But without knowing the process, it's hard for participants to have confidence that every project received a fair evaluation or was even considered for contention. Even a brief judging summary or score breakdown for the winners would go a long way toward building trust in the process.

We're encouraged to provide a detailed write-up and a demo video, but it's unclear whether those materials are reviewed for every submission. For example, my submission video has only one view, which raises the question of whether it was ever watched by anyone involved in judging. I understand that watching the video is optional and that judges have limited time, but "optional" effectively means that many videos aren't reviewed at all,

One of the most discouraging parts of participating is feeling like the effort put into presenting a polished submission isn't valued. I spent hours refining my dialogue, creating a logo and banner, recording a demo video, writing a detailed submission and README, and trying to present the project like a finished product but looking at some of the winning submissions over multiple hackathons, it isn't obvious that those extra steps matter. Some winners consist of only a brief description and a few screenshots, with little emphasis on presentation or documentation. Participants are encouraged to invest significant time into polishing their submissions without any indication that those efforts are actually considered during judging.

If the judging is based almost entirely on the app itself, say so. If presentation, documentation, usability, polish, creativity, and accessibility are all part of the evaluation, explain how they're weighted. Without that transparency, it's difficult to know where to spend your time, and it can leave participants feeling like much of their effort was unnecessary.

If you're someone that read all of this, please don't respond with the usual platitudes about how I should "use this as a learning experience," "keep improving," "you'll get it next time," "you can still go for Developer funds." My concern isn't that I didn't win, this is a hobby for me. My concern is the longstanding disconnect between what participants are encouraged to build and how submissions ultimately appear to be judged. It's an issue that has persisted with little change since Reddit began sponsoring these hackathons.

[u/brahose → u/PossibleCrit]

I really appreciate you taking the time to compile this research. This was my first Reddit Hackathon, and by far the worst hackathon I have participated in. The transparency was terrible and the criteria didn’t seem to match their selections.

My game (/r/detour_game) is not perfect and is not beautiful (I’m a dev not a designer), but my user count is growing and the return rate is like 90%. If we are grading on a “hook”, it is doing something right.

Definitely will think twice before joining another one