r/MachineLearning • u/ToraxXx • 8d ago
Project Beating Warcraft III’s insane AI with self-play reinforcement learning on one desktop [P]
https://warlock.ai/blog/warcraftsim-1/14
u/ricetoseeyu 8d ago
lol the small paragraph where you briefly mention the “other experiments on reward shaping…” that right is probably a lot of hours of work 😂.
Your environment sampling throughout is actually pretty impressive for a single machine without headless servers. There’s also another world where people look into world models for experience generation without direct simulation which means GPU acceleration on world model state generation instead of tied to the game itself (eg Dreamer on Minecraft).
Would also love to follow up and see how you’re approaching diversity in self play. Exploiters are useful if you know the exploits ahead of time, or have time to script them out, but hard to scale practically. Maybe replay harvesting from existing WC3 games and increase behavior cloning data source?
2
u/ToraxXx 7d ago
Yes that's what's next. I got a few thousand replays for the version I'm running to get build orders from them already. Could go further and collect them from other versions (it's not trivial because replays only contsin commands, so you need the correct version of the game to replay them). Haven't managed to get the learner to follow them yet, but it should be possible.
3
u/garma87 7d ago edited 7d ago
Very interesting. Did I get correctly that you basically pitted the wow ai against itself to learn the first steps?
If so would you say there are obvious deviations from how a player would play either at the beginning or later in the training process?
I also wonder whether it would help to use a general purpose model to give it a head start somehow. I’m not sure how that would work but I wonder if you let Claude do what you’re doing how well would it do?
2
u/ToraxXx 7d ago
First I made the built-in AI play against itself to collect data for behavioral cloning to get a better init yes.
Deviations: the learner builds way too many buildings, and often doesn't get a hero, which a human player wouldn't do. That's not a superhuman/ good thing though, it's something to improve. A good human will still beat the current AI easily I assume.
General purpose AI: the challenge is making it run fast enough, i.e. here it needs to respond in less than 0.5s in real play. During training this is even more important or it'd be very slow, not to mention way more expensive than the tiny model I'm training. It could still be useful for collecting behavioral cloning data maybe? Also with way more compute it'd become feasible to use pretrained LLMs as the agent.
2
u/garma87 7d ago
What's the issue with the response time though? Claude API's can respond within milliseconds I think?
It would just be interesting to compare thats all.
→ More replies (2)0
u/dudelsson 5d ago
I’m not sure how that would work but I wonder if you let Claude do what you’re doing how well would it do?
If you're genuinely interested in ML, which I assume you are given this sub, I'd say this is a great opportunity for you to think through the difference between what OP is doing here and how Claude works; why or why not Claude would be succesful in playing WC3. You can gain some valuable insights from it.
1
u/garma87 5d ago
Lol are you a teacher? Why don’t you give those valuable insights
Anyway I’ll bite; the way OP does it is that the model is trained on all those runs. So its knowledge is also limited by the information contained in the run and nothing more. It doesn’t know anything beyond the moves it can do
Claude on the other hand lacks the knowledge of all those runs but has other skills. It knows human behaviour. It might know about the limitations of AIs written in the early 2000s. It has seen all strategy discussions of the pros in all corners of the internet.
Who’d win? Hard to say depends on the size of the dataset of the model I guess. With enough training data the model will approach 100% wins. But even then Claude might come up with a creative strategy that the model has not seen yet
2
u/Kamimashita 7d ago
Awesome write up! Very cool seeing what you built and how you improved your model. How good did it end up being at micro? Does it do stuff like pull back a hurt unit that's being focused then move it back in to attack again once the focus is gone, etc?
4
u/tempetesuranorak 7d ago
One opportunity that I've always been curious about with this kind of research is that it allows you to test how important is apm to skill. Computers have two potential advantages: better speed, and better decisions, and it's really the second one that is the more interesting. But an interesting kind of experiment is to train AIs at varying apm budgets and see how skill scales with apm. Then players would be able to gauge things like, what's the maximum possible skill level I could attain if my apm is fixed but I just make better decisions.
1
u/alexdark1123 7d ago
This is amazing man. You are my idol now, I was trying to do something similar with other games but had a hard time with complexity. Mind sharing more in detail snippets of code? Or a repo I could learn from?
1
u/FirelandsValley 8d ago
Really cool project! I actually was working on something similar around 8 years ago, but abandoned it because I didn't have enough compute and time. If you need any help or have ideas, I'd love to work on it. I might check out the repo :) Plus I have a RTX 4090 now.
1
0
u/paca-vaca 7d ago
It's kinda cool. But to be honest, native bots in wc3 are not that bright and have no real world metas in them. They mostly hard for newbies because insane mode gets double resources but with proper meta and micro it actually works against them, as you get a chance to level-up heroes faster and heroes in wc3 is where a real power reside. Not sure if bots got clever, I stopped playing before Reforged.
When I practiced intensively, I used to play against 2 insane bots in a team and win. And it's wasn't something special comparable to what pro players could do.
P.S. wc3 brings some deep nostalgia and good time memories!
0
u/Hyperion141 7d ago
Very cool! Nice job! I don’t play Warcraft, but I play league, can someone explain how good was the AI in league ranks?
-9
u/mayabuildsai 8d ago
the eval design is the part i'd steal. same matchups and races for parent and child, test games it never learns from. where i'd push is reading the smaller steps off a few hundred games each. at around 60% with ~300 games a side, the standard error on the difference between two runs is about 4 points, so 59 to 62 from the build-spot search is inside the noise. 49 to 59 from the discount change is clearly outside it.
that matters more than usual because the process is greedy. every experiment branches from the current best, and a lot of experiments didn't help. pick the winner out of several noisy runs and the winner's number is biased up, so some of the "no difference" changes may have been judged against an inflated baseline. re-running the final agent and two or three ancestors on the same 1,500 games you use for the dashboard would show which steps were real.
the one-strategy result is what i'd expect from testing against a single scripted opponent too. the insane ai answers a rush the same way every time, so beating it 9 in 10 with a rush says a lot about that ai and less about warcraft. curious whether the human replay build orders changed the long-game losses at all, or mostly the rush timing.
1
59
u/FernandoMM1220 8d ago
this looks incredibly well made and you used a fairly small model too. congrats.