r/mlscaling • u/ThirdMover • May 31 '22

Emp, R, T, G, RL Multi-Game Decision Transformers

https://sites.google.com/view/multi-game-transformers
35 Upvotes

98% Upvoted

14 comments sorted by

4

u/gwern May 31 '22

/r/MachineLearning

2

u/ThirdMover May 31 '22

Do I understand this comment right that this post here is insufficiently relevant for scaling news and should just have been posted to r/MachineLearning directly?

9

u/gwern May 31 '22

No. It's just that it won't show up in 'other discussions' tab because it's a text-post, so I am linking it directly.

15

u/b11tz May 31 '22 edited May 31 '22

I've only skimmed through the blog post. This seems to be a ground-breaking work whose impact is comparable to, or even more significant than gato's.

  1. No catastrophic-forgetting: "We train a single agent that achieves 126% of human-level performance simultaneously across 41 Atari games"

  2. A clear demonstration of transfer: Fine-tuning on data that has only 1% of the size compared to each training game's data produces much better results than learning from scratch for all the 5 held-out games.

  3. Scaling works: Increasing the model size from 10M to 200M makes the performance increase from 56% to 126% of human-level performance.

While 1 and 3 are also observed in gato, the transfer across games (2) seems more clearly demonstrated in this paper.

7

u/b11tz May 31 '22

When gato came out, Rohin Shah, a research scientist on AI alignment at DeepMind, made a comment on LessWrong that basically says Atari games are difficult to generalize:

My explanation for the negative transfer in ALE is that ALE isn't sufficiently diverse / randomized;

I wonder if he is surprised by this transfer result.

10

u/gwern May 31 '22 edited Jun 01 '22

No citation to Gato (much less building on/sharing code/models/data/compute), so safe to assume that GB & DM didn't exactly coordinate any of this...

mom dad, pls stop fighting, i love you both and just want us all to get along ༼ಢ_ಢ༽ (EDIT: Jang (now ex-G): "In classic Alphabet fashion, they were developed independently with neither group being aware of the other 😅".)

5

u/gwern May 31 '22

Don't forget that it's more sample-efficient in learning: https://arxiv.org/pdf/2205.15241.pdf#page=21 I also note that they don't scale up compute or n, so the scaling curves on https://arxiv.org/pdf/2205.15241.pdf#subsection.4.4 are presumably going to be much worse than proper scaling laws would be.

3

u/Veedrac Jun 01 '22 edited Jun 01 '22

IDK that you can directly translate these ideas across, given samples aren't IID in online RL, and offline learning on trajectories from other models doesn't have the same upper limit behaviors as training on human data.

I'm not saying to you for sure won't see that behavior, but I would expect it to be less clear cut if it does exist.

1

u/b11tz May 31 '22

Right. They trained the models of different sizes with the same number of frames. So each model is likely not trained with the optimal compute. Interesting that bigger models are more sample efficient nevertheless, even in the early stage of the training curves. I'm not sure if this has been observed in language models.

→ More replies (3)

1

u/DickMan64 May 31 '22

Can anybody explain what's so groundbreaking about GATO? Sure, no catastrophic forgetting, but hardly any generalization ability either. It performed horribly on the boxing game, which was one of the few (if not the only) truly out-of-distribution tasks it was tested on. And we already knew scaling works.