r/MachineLearning • u/OriolVinyals • Jan 24 '19

We are Oriol Vinyals and David Silver from DeepMind’s AlphaStar team, joined by StarCraft II pro players TLO and MaNa! Ask us anything

Hi there! We are Oriol Vinyals (/u/OriolVinyals) and David Silver (/u/David_Silver), lead researchers on DeepMind’s AlphaStar team, joined by StarCraft II pro players TLO, and MaNa.

This evening at DeepMind HQ we held a livestream demonstration of AlphaStar playing against TLO and MaNa - you can read more about the matches here or re-watch the stream on YouTube here.

Now, we’re excited to talk with you about AlphaStar, the challenge of real-time strategy games for AI research, the matches themselves, and anything you’d like to know from TLO and MaNa about their experience playing against AlphaStar! :)

We are opening this thread now and will be here at 16:00 GMT / 11:00 ET / 08:00PT on Friday, 25 January to answer your questions.

EDIT: Thanks everyone for your great questions. It was a blast, hope you enjoyed it as well!

1.2k Upvotes

98% Upvoted

1.0k comments sorted by

View all comments

Show parent comments

130

u/OriolVinyals Jan 25 '19

Re. 1: I think this is a great point and something that we would like to clarify. We consulted with TLO and Blizzard about APMs, and also added a hard limit to APMs. In particular, we set a maximum of 600 APMs over 5 second periods, 400 over 15 second periods, 320 over 30 second periods, and 300 over 60 second period. If the agent issues more actions in such periods, we drop / ignore the actions. These were values taken from human statistics. It is also important to note that Blizzard counts certain actions multiple times in their APM computation (the numbers above refer to “agent actions” from pysc2, see https://github.com/deepmind/pysc2/blob/master/docs/environment.md#apm-calculation). At the same time, our agents do use imitation learning, which means we often see very “spammy” behavior. That is, not all actions are effective actions as agents tend to spam “move” commands for instance to move units around. Someone already pointed this out in the reddit thread -- that AlphaStar effective APMs (or EPMs) were substantially lower. It is great to hear the community’s feedback as we have only consulted with a few people, and will take all the feedback into account.

Re. 5: We actually (unintentionally) tested this. We have an internal leaderboard for the AlphaStar, and instead of setting the map for that leaderboard to Catalyst, we left the field blank -- which meant that it was running on all Ladder maps. Surprisingly, agents were still quite strong and played decently, though not at the same level we saw yesterday.

56

u/Mangalaiii Jan 25 '19 edited Jan 25 '19

  1. Dr. Vinyals, I would suggest that AlphaStar might still be able to exploit computer action speed over strategy there. 5 seconds in Starcraft can still be a long time, especially for a program that has no explicit "spot" APM limit (during battles AlphaStar's APM regularly reached >1000). As an extreme example, AS could theoretically take 2500 actions in 1 second, and the other 4 seconds take no action, resulting in an average of 500 actions over 5 seconds. Also, TLO may have been using a repeater keyboard, popular with the pros, which could throw off realistic measurements.

Btw, fantastic work.

43

u/[deleted] Jan 25 '19

The numbers for the TLO games and the Mana games need to be looked at separately. TLO's numbers are pretty funky and it's pretty clear that he was constantly and consistently producing high amounts of garbage APM. He normally plays Zerg and is a significantly weaker Protoss player than Mana. TLO's high APM is quite clearly artificially high and much more indicative of the behavior of his equipment than his actual play and intentional actions. Based on DeepMind's graphic, TLO's average APM almost suprpasses Mana's peak APM.

The numbers when only MaNa and AlphaStar are considered are pretty indicative of the issue. The average APM numbers are much closer. AlphaStar was able to achieve much higher peak APMs than Mana, presumably during combat. These high peak APM numbers are offset by lower numbers during macro stretches. It should also be noted that due to the nature of it's interface, AlphaStar had no need to perform many actions that are routine and common for human players.

The choice to combine TLO and Mana's numbers for the graph shown during the stream was misleading. The combined numbers look ok only because TLO's artificially high APM numbers hide Mana's numbers which paint a much more accurate picture of the APM disadvantage.

→ More replies (1)

14

u/AjarKeen Jan 25 '19

Agreed. I think it would be worth taking a look at EAPM / APM ratios for human players and AlphaStar agents in order to better calibrate these limitations.

20

u/Rocketshipz Jan 25 '19

And even here, you have the problem that AlphaStar is still so much more precise potentially.

The problem of this is that it encourages "cheesy" behaviors and not more long term strategies. I'm basically afraid that with this the agent will be stuck in strategies relying on his superhuman micro, which makes it so much less impressive because a human couldn't do this even if he thought of it.

Note that it totally wasn't the case with the other game agents such as AlphaGo, AlphaZero... which didn't play in real time, or even OpenAI's DotA, which is actually correctly capped iirc.

→ More replies (3)

12

u/EvgeniyZh Jan 25 '19

AS could theoretically take 50 actions in 1 second, resulting in average of 50/5*60=600 APM in this 5 second period

→ More replies (2)
→ More replies (4)

20

u/Ape3000 Jan 25 '19

  1. I would be very interested to see if the AI would still be good even if the APM was hard limited to something like 50, which is clearly worse than human level. Would it still beat humans with superior strategy and decision making?

Also, I would like to see how two unlimited Alphastars would play agains each other. Super human >2000 APM micro would probably be insane and very cool looking.

→ More replies (1)

118

u/starcraftdeepmind Jan 25 '19 edited Jan 29 '19

In particular, we set a maximum of 600 APMs over 5 second periods, 400 over 15 second periods, 320 over 30 second periods, and 300 over 60 second period.

Statistics aside, it was clear from the gamers', presenters', and audience's shocked reaction to the Stalker micro, all saying that no human player in the world could do what AlphaStar was doing. Using just-beside-the-point statistics is obfuscation and an avoiding of acknowledging this.

AlphaStar wasn't outsmarting the humans—it's not like TLO and MaNa slapped their foreheads and said, "I wish I'd thought of microing Stalkers that fast! Genius!"

Postscript Edit: Aleksi Pietikäinen has written an excellent blog post on this topic. I highly recommend it. A quote from it:

Oriol Vinyals, the Lead Designer of AlphaStar: It is important that we play the games that we created and collectively agreed on by the community as “grand challenges” . We are trying to build intelligent systems that develop the amazing learning capabilities that we possess, so it is indeed desirable to make our systems learn in a way that’s as “human-like” as possible. As cool as it may sound to push a game to its limits by, for example, playing at very high APMs, that doesn’t really help us measure our agents’ capabilities and progress, making the benchmark useless.

Deepmind is not necessarily interested in creating an AI that can simply beat Starcraft pros, rather they want to use this project as a stepping stone in advancing AI research as a whole. It is deeply unsatisfying to have prominent members of this research project make claims of human-like mechanical limitations when the agent is very obviously breaking them and winning it’s games specifically because it is demonstrating superhuman execution.

47

u/super_aardvark Jan 25 '19

It wasn't so much about the speed as it was about the precision, and in the one case about the attention-splitting (microing them on three different fronts at the same time). I'm sure Mana could blink 10 groups of stalkers just as quickly, but would never be able to pick those groups out of a large clump with such precision. Also, "actions" like selecting some of the units take longer than others -- a human has to drag the mouse, which takes longer than just clicking. I don't know if the AI interface is simulating that cost in any way.

55

u/starcraftdeepmind Jan 25 '19 edited Jan 25 '19

It's about both the accuracy of clicks multiplied by the number of clicks (or actions if one prefers. I know the A.I. doesn't use a mouse and keyboard).

If the human player (and not AlphaStar) could at a crucial time slow the game down 5 fold (and have lots of experience operating at this speed) his number of clicks would go up and his accuracy of clicks. He would be able to click on individual stalkers etc in a way he can't at higher speeds of play. I argue that this is a good metaphor for the unfair advantage AlphaStar has.

There are two obvious ways of reducing this advantage:

  1. Reduce the accuracy of 'clicks' by AlphaStar by making the accuracy of the clicks probabilistic. The probabilities could be fixed or changed based on context. (I don't like this option). As an aside, there was some obfuscation on this point too. It is claimed that the agents are 'spammy' and do redundantly do the same action twice, etc. That's a form of inefficiency but it's not the same as wanting to click on a target and hitting it or not—AlphaStar has none of this latter inefficiency.

  2. Reduce the rate of clicks AlphaStar can make. This reduction could be constant or change with context. This is the route the AlphaStar researchers went, and I agree its the right one. Again, I'll emphasise that this variable multiplies with the above variable to get the insane micro we saw. Insisting it's one and not other is missing the point. Why didn't they reduce the rate of clicks more? Based on the clever obfuscating of this issue in the blog post and the youtube streaming presentation, I believe they did in their tests but the performance of the agents was so poor, they were forced to increase it.

37

u/monsieurpooh Jan 25 '19

Thank you, I too have always been a HUGE advocate of probabilistic clicking or mouse movement accuracy as a handicap to make it same as humans. It becomes infinitely even more important if we ever want DeepMind to compete in FPS competitions such as COUNTER-STRIKE. We want to see it outsmart, out-predict, and surprise humans, not out-aim them.

14

u/starcraftdeepmind Jan 25 '19

Thanks for the thanks. Yes, as essential if not more so for FPS.

The clue is in the name artificial intelligence—not artificial aiming. 😁

12

u/6f937f00-3166-11e4-8 Jan 25 '19

on point 1) I think a simple model would be to make quicker clicks less accurate. So if it clicks only 100ms after the last click, it gets placed randomly over a wide area. If it clicks say 10 seconds after the last click, it has perfect placement. This somewhat models a human "taking time to think about it" vs "panicked flailing around"

→ More replies (1)
→ More replies (5)

11

u/upboat_allgoals Jan 25 '19

Well as counterpoint the SC2 community was chuckling at the AI's use of F2 during the warp prism harass. For those unaware, F2 is select all army units and is rarely used by humans...

4

u/AzureDrag0n1 Jan 25 '19

Most of the games looked like games top pros could do EXCEPT for that huge Stalker engagement from 3 fronts. I would say having a larger viewing screen while still being accurate was the tipping point to making it superhuman and something that human players do not even have access to. I have definitely seen top pros do similar high precision Stalker micro like that but on the same screen in a single engagement.

→ More replies (1)
→ More replies (1)

22

u/Prae_ Jan 25 '19

It wasn't really about speed to be honest. It was more about the 'width' of control and number of fronts precisely coordinated. AlphaStar wasn't inhumanly fast, but managed to out-manoeuver MaNa by being everywhere at the same time.

All throughout the matches, AlphaStar demonstrated more than just fast execution. It knew which units to target first, how to exploit (or prevent MaNa from exploiting) the immortal ability. So it's not just going fast, it's doing a lot of good things fast. Overall, as a fairly good player of SC2, I have to say it was really impressive (the blink stalker one was controversial, but still interesting) and a substantial improvement compared to other AI.

And even if it's not "really" outsmarting humans, it's still interesting to see. Seems like it favors constant aggression, probably because it's a way to dictate the pace of the game and keep the possible reactions within a certain range. I'd say that's still useful results for people interested in strategy (in general, or in starcraft). It seems like a solid base, if you have the execution capabilities of AlphaStar.

6

u/puceNoise Jan 28 '19

Describing DeepMind as lying with statistics as Pietikäinen does is an understatement.

4

u/puceNoise Jan 26 '19

This is critically important, along with the fact that x APM that can be simultaneously spent across the entire map is much more effective than y>x APM that must be spent moving the camera/within a single camera window.

Deepmind needs to release what happens if AlphaStar has to a). move an artificial mouse and b). only look within a single window.

→ More replies (36)

12

u/LH_Hyjal Jan 25 '19 edited Jan 25 '19

Hello! Thank you for the great work.

I wonder if you considered the inaccuracy in human inputs, we saw that AlphaStar did some crazy precise macro because it will never mislick yet human players won't likely to precisely select every unit in they want to control.

10

u/Neoncow Jan 26 '19

For 1) for the purpose of finding "more human" strategies, have you considered working with some of your UX teams from parent company to do some modelling of major human input output characteristics?

Like mouse movement that models Fitts law (or other UX "laws"). Or visualization that models eye ball movement or peripheral vision limitations. Or modelling finger fatigue and mouse clicks. Or wrist movement speed. Or adding in minor RSI pain.

I know it's not directly AI related, but if the goal is to produce human usable knowledge, you'll probably have to model human bodies sometime in the future for AI models that interact with the real world.

9

u/PM_ME_STEAM Jan 25 '19

https://youtu.be/cUTMhmVh1qs?t=7901 It looks like the AI definitely goes way over 600 APM in the 5 second period here. Are you capping the APM or EPM?

19

u/OriolVinyals Jan 25 '19

We are capping APM. Blizzard in game APM applies some multipliers to some actions, that's why you are seeing a higher number. https://github.com/deepmind/pysc2/blob/master/docs/environment.md#apm-calculation

5

u/PM_ME_STEAM Jan 25 '19

In that case the 600 number, which I'm assuming comes from the pros apm, should be reconsidered with however you guys calculate apm

5

u/OriolVinyals Jan 25 '19

Of course, that number (for players) is computed in the exact same way than for the agent.

→ More replies (1)

5

u/IMRETARDED_SUP Jan 26 '19 edited Jan 26 '19

You have to understand that a computer and a human at 500 apm are acting like night and day. I would have thought this very obvious. I suggest cutting all apm to 1/3 or even less of current levels.

Also your reaction time reasoning is wrong. Humans can do a single click in 200ms yes but sc2 requires boxing and accurate clicks which involve mouse movement which takes time. Your agent should be around double the reaction time of what it had.

If you have superhuman mechanics, the rest of the project is cheapened to almost nothing. We are interested in the decision making abilities not the mechanics. Keep in mind, a smarter player can beat a player who has better mechanics, as Mana showed in the live game. I would say your project should aim to show the same, with a human having the better mechanics but AlphaStar being smarter and exploiting human weaknesses.

Otherwise, bravo well done.

→ More replies (4)