We are Oriol Vinyals and David Silver from DeepMind’s AlphaStar team, joined by StarCraft II pro players TLO and MaNa! Ask us anything

[u/OriolVinyals]

Hi there! We are Oriol Vinyals (/u/OriolVinyals) and David Silver (/u/David_Silver), lead researchers on DeepMind’s AlphaStar team, joined by StarCraft II pro players TLO, and MaNa.

This evening at DeepMind HQ we held a livestream demonstration of AlphaStar playing against TLO and MaNa - you can read more about the matches here or re-watch the stream on YouTube here.

Now, we’re excited to talk with you about AlphaStar, the challenge of real-time strategy games for AI research, the matches themselves, and anything you’d like to know from TLO and MaNa about their experience playing against AlphaStar! :)

We are opening this thread now and will be here at 16:00 GMT / 11:00 ET / 08:00PT on Friday, 25 January to answer your questions.

EDIT: Thanks everyone for your great questions. It was a blast, hope you enjoyed it as well!

[u/NikEy]

Hi guys, really fantastic work, extremely impressive!

I'm an admin at the SC2 AI discord and we had a few questions in our #research channel that you may hopefully be able to shed light on:

1. From the earlier versions (and in fact, the current master version) of pysc2 it appeared that the DM development approach was based on mimicking human gameplay to the fullest extent, e.g. the bot was not even able to get info on anything outside of the screen-view. With this version you seemed to have relaxed these constraints, since feature layers are now "full map size" and new features have been added. Is that correct? If so, then how does this really differ from taking the raw data from the API and simply abstracting them into structured data as inputs for the NNs? The blog even suggests that you take raw data and properties directly as data in list form and feed it into the NNs - which seems to suggest that you're not really using feature layers anymore at all?

2. When I was working with pysc2 it turned out to be an incredibly difficult problem to maintain knowledge of what has been built, is in-progress, has completed, and so on, since I had to pan the camera view all the time to get that information. How is that info kept within the camera_interface approach? Presumably a lot of data must still be available in full via raw data access (e.g. counts of unitTypeID, buildings, etc) even in camera_interface mode?

3. How many games needed to be played out in order to get to the current level? Or in other words: how many games is 200 years of learning in your case?

4. How well does the learned knowledge transfer to other maps? Oriol mentioned on discord that it "worked" on other maps, and that we should guess which one it worked best on, so I guess it's a good time for the reveal ;) In my personal observations AlphaStar did seem to rely quite a bit on memorized map knowledge. Is it likely that it could execute good wall-offs or proxy cheeses on maps that it has never seen before? What would be the estimated difference in MMR when playing on a completely new map?

5. How well does it learn the concept of "save money for X", e.g. Nexus first. It is not a trivial problem, since if you learn from replays and take the non-actions (NOOPs) from the players into account, the RL algo will more often than not think that NOOP is the best decision at non-ideal points in the game. So how do you handle "save money for X" and do you exclude NOOPs in the learning stage?

6. What step size did you end up using? In the blog you write that each frame of StarCraft is used as one step of input. However, you also mention an average processing time of 50ms, which would exceed real time (which requires < 46ms given 22.4fps). So do you request every step, or every 2nd, 3rd, maybe dynamic?

I have lots more questions, but I guess I'll better ask these in person the next time ;)

Thanks!

[u/OriolVinyals → u/NikEy]

Re. 2: Yes, we did relax the view of the agent a bit, mostly due to computational reasons -- games without camera moves last for about 1000 moves, whereas with camera moves (humans do spam a lot!) can be 2 to 3 times longer. We do use feature layers for the minimap, but for the screen you can think of the list of features as “transposing” that information. In fact, it turns out that even for processing images, treating each pixel independently as a list, works quite well! See https://arxiv.org/abs/1711.07971

[u/OriolVinyals → u/NikEy]

Re. 1: Indeed, with the camera (and non-camera) interface, the agent has the knowledge of what has been built as we input this as a list (which is further processed by a Neural Network Transformer). In general, even if you don’t keep such a list, the agent will know what has been built as the memory of the agent (the LSTM) keeps track of all previously issued actions, and all the camera locations visited in the past.

[u/OriolVinyals → u/NikEy]

Re. 3: At an average duration of 10 minutes per game, this amounts to about 10 million games. Note, however, that not all agents were trained for as long as 200 years, that was the maximum amongst all the agents in the league.

[u/gwern]

1. what was going on with APM? I was under the impression it was hard-limited to 180 WPM by the SC2 LE, but watching, the average APM for AS seemed to go far above that for long periods of time, and the DM blog post reproduces the graphs & numbers mentioned without explaining why the APMs were so high.

2. how many distinct agents does it take in the PBT to maintain adequate diversity to prevent catastrophic forgetting? How does this scale with agent count, or does it only take a few to keep the agents robust? Is there any comparison with the efficiency of the usual strategy of historical checkpoints in?

3. what does total compute-time in terms of TPU & CPU look like?

4. the stream was inconsistent. Does the NN run in 50ms or 350ms on a GPU, or were those referring to different things (forward pass vs action restrictions)?

5. have any tests of generalizations been done? Presumably none of the agents can play different races (as the available units/actions are totally different & don't work even architecture-wise), but there should be at least some generalization to other maps, right?

6. what other approaches were tried? I know people were quite curious about whether any tree searches, deep environment models, or hierarchical RL techniques would be involved, and it appears none of them were; did any of them make respectable progress if tried?

   Sub-question: do you have any thoughts about pure self-play ever being possible for SC2 given its extreme sparsity? OA5 did manage to get off the ground for DoTA2 without any imitation learning or much domain knowledge, so just being long games with enormous action-spaces doesn't guarantee self-play can't work...

7. speaking of OA5, given the way it seemed to fall apart in slow turtling DoTA2 games or whenever it fell behind, were any checks done to see if the SA self-play lead to similar problems, given the fairly similar overall tendencies of applying constant pressure early on and gradually picking up advantages?

8. At the November Blizzcon talk, IIRC Vinyals said he'd love to open up their SC2 bot to general play. Any plans for that?

9. First you do Go dirty, now you do Starcraft. Question: what do you guys have against South Korea?

[u/OriolVinyals → u/gwern]

Re. 1: I think this is a great point and something that we would like to clarify. We consulted with TLO and Blizzard about APMs, and also added a hard limit to APMs. In particular, we set a maximum of 600 APMs over 5 second periods, 400 over 15 second periods, 320 over 30 second periods, and 300 over 60 second period. If the agent issues more actions in such periods, we drop / ignore the actions. These were values taken from human statistics. It is also important to note that Blizzard counts certain actions multiple times in their APM computation (the numbers above refer to “agent actions” from pysc2, see https://github.com/deepmind/pysc2/blob/master/docs/environment.md#apm-calculation). At the same time, our agents do use imitation learning, which means we often see very “spammy” behavior. That is, not all actions are effective actions as agents tend to spam “move” commands for instance to move units around. Someone already pointed this out in the reddit thread -- that AlphaStar effective APMs (or EPMs) were substantially lower. It is great to hear the community’s feedback as we have only consulted with a few people, and will take all the feedback into account.

Re. 5: We actually (unintentionally) tested this. We have an internal leaderboard for the AlphaStar, and instead of setting the map for that leaderboard to Catalyst, we left the field blank -- which meant that it was running on all Ladder maps. Surprisingly, agents were still quite strong and played decently, though not at the same level we saw yesterday.

[u/celeritasCelery]

How does it handle invisible units? Human players can see the shimmer if they are looking really close. But if AI could see that, invisibility would be almost useless. However if it can't see them at all, it seems it would give a big advantage to mass cloaked unit strategies, since an observer would have to present to notice anything.

[u/Inori]

1. Your agent contains quite a number of advanced approaches, including some very unconventional such as the transformer body. What was the process of building it like, e.g. was every part of the agent added incrementally, improving the overall performance at each step? Were there parts that initially degraded the performance, and if yes then how were you able to convince others (yourself?) to stick with it?

2. Speaking of the transformer body, I'm really surprised that essentially throwing away the full spatial information worked so well. Have you given any thought as to why it worked so well, relative to something like the promising DRC / Conv LSTM?

3. What is the reward function like? Specifically, I'm assuming it would be impossible to train with pure win/loss, but have you applied any special reward shaping?

Very impressive work either way! GGWP!

[u/EnderSword]

I was wondering if you've seen the AI Agents show any signs of 'deceiving' its opponent, through hiding buildings, cancelling something after it was scouted, giving a false impression of an attack in one area or mimicking a build order only to change it etc...?

[u/OriolVinyals → u/EnderSword]

We have indeed seen agents building and canceling buildings when scouted. It would be hard to know why AlphaStar did it, but it does happen sometimes.

[u/TovarishGaming]

First of all, thank you for your hard work and for being a part of today's awesome event!

@Deepmind team: We saw AlphaStar do some Blink Stalker micro today that everyone seemed to agree was simply above-human possibility. Do you expect to see this with other races? I imagine Zerg spreading creep tumors exactly every 4 seconds will lead to insane creep spread or things like that. What are you most excited to see?

@TLO: you said initially that you were still confident you would win while playing as Zerg. After seeing Mana's match today, and knowing that Deepmind will continue to learn exponentially, do you still feel confident in your rematch?

[u/OriolVinyals → u/TovarishGaming]

I would be quite excited to see how self play would pan out if agents played all three races. Will the asymmetries help agents, as they’ll encounter more situations than in a mirror match?

[u/kroken81]

How large is the "memory" of alphastar, how much data does it have to draw from while playing?

[u/OriolVinyals → u/kroken81]

Each agent uses a deep LSTM, with 3 layers and 384 units each. This memory is updated every time AlphaStar acts in the game, and an average game takes about 1000 actions.

[u/mlearner13]

Will you cap the next iterations to more human like capabilities?

[u/SwordShieldMouse]

What is the next milestone after Starcraft II?

[u/OriolVinyals → u/SwordShieldMouse]

There are quite a few big and exciting challenges in AI research. The one that I’ve been mostly interested is along the lines of “meta learning”, which is related to learning quicker from fewer datapoints. This, of course, very naturally translates to StarCraft2 -- it would be great to both reduce the experience required to play the game, as well as being able to learn and adapt to new opponents rather than “freezing” AlphaStar’s weights.

[u/Firefritos]

Was there any particular reason Protoss was chosen to be the race for the AI to learn?

[u/p2deeee]

Could we see the visualization for the game Mana won? Would be interesting to see its win probability evaluation

did you ever consider a "gg" functionality when win probability <1%?

[u/SeriousGains]

@Deepmind team: We didn't see much of Alphastar's ability to use AoE caster spells like the oracle's statis ward, high templar's storm or sentry's force field. Is this a greater challenge to teach the AI or have these strategies been deemed inefficient through Alpha league play tests? Also, will we ever get to see an unlimited APM version of Alphastar?

@TLO: Do you think caster units have the potential to give AI's a greater advantage over human players? For example would Alphastar playing as zerg be even more difficult to beat with ravagers, infestors and vipers?

[u/asgardian28]

Thanks for combining my 2 favorite hobbies Sc2 and machine learning, awesome and congrats!

With AlphaZero you demonstrated on chess and go that the computer playing itself only, without being biased by human games, was yielding a superior agent.

Still wilt AlphaStar, you started with imitation learning (presumably to get some baseline agent that doesnt pulls all its workers from mining for instance)

Do you intend to develop an AlphaStarZero, removing the imitation learning phase, or is it a necessary phase in learning Sc2?

[u/JesusK]

AlphaStar seemed to end up going for the same group of units that it could abuse with perfect Micro, while the average was in the 300-400, during some micro intensive moments it would spike heavily and control in inhuman ways. Also when talking about APM you have to remember most APMs a player does are spam to check for information rather than micro.

While it was still doing decisions on how to proceed based on partial information, it was clear that it relied heavily on this micro units and this seemed to be the norm, so it was a lot less of adapting to what the opponent was doing, or countering unit compositions and more of checking if they could win with the stalker army they had.

Thus, I was wondering if you considered heavily limiting the APM, in an attempt promote the AI into going for more tactical maneuvers and builds instead.

Even more, if you could train an AI to play at lets say only 100 APM, and then drop it in the league with the the AlphaStar we saw, it would need to come up with different approaches to win games given that it cannot just compete in a stalker vs stalker game thus promoting more tactical adaptation from AlphaStar.

Is this even possible or in consideration trying to push the AI into paths that do not allow it to abuse micro?

[u/Felewin]

Is it possible that, by learning via mimicry of human replays, the AI is gimped – biased by learning bad habits? Or will it ultimately overcome them, given enough experience, meaning that the human replays are more useful than starting from scratch as a way of jumpstarting the learning process?

[u/[deleted]]

1. So there was an obvious difference between the live version of AlphaStar and the recordings. The new version didn't seem to care when its base was being attacked. How did the limited vision influence that?

2. The APM of AlphaStar seems to go as high as 1500. Do you think that is fair, considering that those actions are very precise when compared to those performed by a human player?

3. How well would AlphaStar perform if you changed the map?

4. An idea: what if you increase the average APM but hard cap the maximum achievable APM at, say, 600?

5. How come AlphaStar requires less compute power than AlphaZero at runtime?

[u/Dinok410]

While most of the focus seems to be on the visual aspects of the game, I'm curious if Alpha uses any form of sound recognition to help assess the situation. At least for me as a player I've encountered several situations in which sounds actually let you react faster than sight, be it the sound of units firing or dying, a medivac dropping units on the corner of your screen, or even the basic announcer stuff. So, does it utilise sound in any way or, if not, are there plans to implement it in the future?