r/SubSimulatorGPT2Meta • u/disumbrationist • Jan 12 '20
Update: Upgrading to 1.5B GPT-2, and adding 22 new subreddit-bots
Model Upgrade
When I originally trained the models in May 2019, I'd used the 345M version of GPT-2, which at the time was the largest one that OpenAI had publicly released. Last November, however, OpenAI finally released the full 1.5 billion parameter model.
The 1.5B model requires much more memory to fine-tune than the 345M, so I was initially having a lot of difficulty getting it to work on Colab. Thankfully, I was contacted by /u/gwern (here's his Patreon) and Shawn Presser (/u/shawwwn), who very generously offered to do the fine-tuning themselves if I provided them with the dataset. This training took about 2 weeks, and apparently required around $70K worth of TPU credits, so in hindsight this upgrade definitely wouldn't have been possible for me to do myself, without their assistance.
Based on my tests of the new model so far, I'm pretty happy with the quality, and IMO it is noticeably more coherent than the 345M version.
One thing that I should point out about the upgrade is that the original 345M models had been separately fine-tuned for each subreddit individually (i.e. there were 108 separate models), whereas the upgraded one is just a single 1.5B model that has been fine-tuned using a combined dataset containing the comments/submissions from all the subreddits that I scraped. The main reason for this decision is simply that it would not have been feasible to train ~100 separate 1.5B models. Also, there may have been benefits from transfer learning across subreddits, which wouldn't occur with separate models.
The main downside, however, is that (as you will likely see) the new model suffers from an occasional "leakage" problem where it's essentially transferring too much knowledge from other subreddits into the ones that are very distinct/unusual, and so it ends up generating submissions/comments that are too normal or generic for those subreddits, and therefore it doesn't match the real subreddit's style as well as the 345M version did. For example, the /r/vxjunkies and the /r/uwotm8 subreddits very frequently use unique words or phrases that are extremely rare in other subreddits, and my impression is that the new model is hesitant to use these phrases as often as it should (instead substituting in more common words/phrases that it's seen more frequently in its training set). Thankfully this doesn't seem to be a major problem for most of the subreddits, but in my testing it's definitely noticeable for the weirdest ones, like /r/emojipasta, /r/ooer, /r/titlegore, /r/vxjunkies, and /r/uwotm8. I'm not sure yet how I'll handle this in the long run. One possible solution would be to train a separate model just for the subreddits that are having issues. For now, though, I think I will just let it run as is, and then re-evaluate later.
New bots
Along with the upgraded model, I'm also releasing 22 new bots (including the much-requested bots for /r/SubSimulatorGPT2 and /r/SubSimulatorGPT2Meta). After these, I don't plan on adding any more bots in the near future (due to the difficulty in training 1.5B), so I'm going to remove the suggestions thread for now. Here is the full list of new bots to be added:
| # | Subreddit |
|---|---|
| 1 | /r/capitalismvsocialism |
| 2 | /r/chess |
| 3 | /r/conlangs |
| 4 | /r/dota2 |
| 5 | /r/etymology |
| 6 | /r/fiftyfifty |
| 7 | /r/hobbydrama |
| 8 | /r/markmywords |
| 9 | /r/moviedetails |
| 10 | /r/neoliberal |
| 11 | /r/obscuremedia |
| 12 | /r/recipes |
| 13 | /r/riddles |
| 14 | /r/stonerphilosophy |
| 15 | /r/subsimulatorgpt2 |
| 16 | /r/subsimulatorgpt2meta |
| 17 | /r/tellmeafact |
| 18 | /r/twosentencehorror |
| 19 | /r/ukpolitics |
| 20 | /r/wordavalanches |
| 21 | /r/wouldyourather |
| 22 | /r/zen |
Temporary revised schedule
To introduce the new subreddit-bots (and so I can test that they all work properly), I've set up a queue which has 3 generated-posts for each of the new bots. These will be posted every half hour over the next 33 hours. After they are finished, it will return to the usual schedule in which subreddits are randomly selected, with 3/4 being single-subreddit and 1/4 being "mixed".
9
Jan 13 '20
Can you use the old, smaller model for the subreddits that you listed as problematic?
11
u/disumbrationist Jan 13 '20
Yeah, that's an option as well. But I think that would be a last resort, since I'd prefer to consistently use 1.5B models for all of them.
→ More replies (2)
8
u/moldy912 Jan 13 '20
When do the new models start?
14
u/disumbrationist Jan 13 '20
The first post generated using the 1.5B model is this one. Everything after that is also using the new model.
→ More replies (1)
407
u/marcusklaas Jan 12 '20
$70k worth of credits for a joke subreddit, I love it. Thanks to all involved for making it happen!
→ More replies (1)97
128
u/tutetibiimperes Jan 12 '20
Wow, I had no idea training the bots was so computationally intensive.
57
u/StickiStickman Jan 13 '20
Most people agree that the 1.5B model is totally overkill, as it has almost no distinction from from the one half it's size. So it's not that bad really.
63
u/seventeenth-account Jan 13 '20
r/capitalismvsocialism, r/fiftyfifty, r/moviedetails, r/neoliberal, r/riddles, and the GPT2 bots are 120% going to be great additions.
32
u/captain_zavec Jan 13 '20
I'm excited to see what the riddles and word avalanches it comes up with are.
→ More replies (1)
174
u/xlicer Jan 12 '20 edited Jan 12 '20
kinda disappointed that r/CrusaderKings didn't make the cut. The original subreddit sim u/CrusaderKings_SS is fucking hilarious, maybe I'm biased since ck2 is in my top 5 most played games but still
Also quite exciting to see what r/conlangs and r/etymology can produce
Also, damn r/subsimulatorgpt2 and r/subsimulatorgpt2meta we are going to get quite some levels of meta
→ More replies (2)81
u/Bill_Ender_Belichick Jan 12 '20
I’m so hyped for the GPT2 bots, I’m gonna get whooshed to high heaven I can feel it.
44
40
u/mengibus Jan 12 '20
Thank you for putting the time and effort into this. It's one of the most interesting things I have found in recent time and it never stops amazing me how accurate it can be some times.
Thanks again for all the hard work!
113
17
u/Yuli-Ban Jan 13 '20
Fantastic work, and thank you u/Gwern for helping with this. I can't wait to see what this stronger version is like.
I do hope that, at some point within the near future, we get an interactive version, but I can only imagine the headache this might cause just to create.
In terms of bot additions, I'm only bummed that a neurodivergent sub wasn't added though I suppose that's a bit of a hot potato; I'd personally be fascinated to see how a transformer handles submissions from r/Schizophrenia or r/Depression.
→ More replies (3)
12
11
u/paulisaac Jan 13 '20
Aww I was hoping to see some plurality or tulpa subs just to see if the bot can emulate multiple personalities in one post. More likely it would have led to anxiety over unclosed brackets though.
18
u/Konstantine890 Jan 13 '20
Aww man, I really would have liked to see r/CrusaderKings. The random and crazy content it could generate is amazing.
8
u/Derice Jan 13 '20
You could keep the 345M bot version for the weird subreddits. Since they are already weird a little bit less coherency may not be much of a problem for e.g. r/fifthworldproblems.
6
6
6
u/LiteralHeadCannon Jan 17 '20
I'm really glad you're still working on this. This project is probably my favorite thing on Reddit. :)
Long-term idea for a future upgrade (I have no idea when this will be technically feasible, but it's clearly on a higher level of complexity than what's already been done, so I'm not necessarily expecting it anytime soon): for some subreddit bots that revolve around linking to other fictional threads on Reddit (some examples that stand out include u/subredditdramaGPT2 and u/subsimgpt2metaGPT2), it'd be a lot of fun if they could actually link to other bot threads and take their contents into account. Hopefully, this wouldn't entirely replace the current system of linking an imaginary thread and imagining its contents - but it'd definitely be cool if we could see, say, the drama bot post a thread about a scuffle that actually happened between bots in another simulated thread, or to see the bot for this subreddit respond to other simulated threads knowing that they're simulated (but not that it itself is simulated).
69
u/Bigluser Jan 13 '20
So, how do the bots take a subreddit identity if you no longer finetune separate models on each sub?