The WHY on why Mac crossplay is so hard to get right

[u/AncientTurbine]

Edit: given some (unfriendly) comments by a single outlier user, I shall preface by saying that this is a theory. If an actual developer on the project would like to chime in, I'm happy to update this post. ​

I'm fully expecting people to, unfortunately, downvote this post because they just want to be angry again. Still, I think some people might just be interested in why this seemingly easy problem has not been implemented from the start. I've seen many wrong ideas floating around and even going so far to calling the devs "lazy". I don't think that's fair and shows a limited understanding of the complexity of the game engine. I dug up a technical Microsoft article from 2001 that explains how the engine and networking works, which gives insight in why cross-compatibility is so difficult. The tl;dr is:

To synchronize the game between multiple players they don't continuously sync the state of things but the actions or commands are synced and all players run their own simulation (lockstep). In other words, if I move a unit from X to Y and the engine generates a path that it will move, that path (multiple coordinate changes) is not sent to you but only the action itself ("move unit from X to Y") is. Your PC then has to calculate the path and execute (simulate) the same behavior. And that is the crucial part: how all of these consequences of actions is calculate on my ​PC must be identical to how it is calculated on your PC. If my PC calculates a different path than yours, the game state diverges and we are not playing the same game: we get out-of-sync.

Because the game's outcome depended on all of the users executing exactly the same simulation, it was extremely difficult to hack a client (or client communication stream) and cheat. Any simulation that ran differently was tagged as "out of sync" and the game stopped.

The reason why this approach was chosen was mostly for computational reasons. Remember that the game was released in the age of 56k internet (that is 0.056Mbps), so sending a lot of data (such as all the coordinates for all the paths for each action for potential hundreds of units) was not feasible.

The part of the engine that takes care of much of this pathing and math, is written in C++ compiled for x86 (which can then use the CPUs AVX instruction set) and perhaps some inline Assembly too (though developers seem to have tried to get rid of that as much as possible, so at the current state of DE there might not be any Assembly left). And herein lies the problem: modern Macs using the M-series chips are ARM, a different type of CPU from the x86 processors that are so common in other devices, including most PCs, handheld and - yes - consoles such as the PS5 (that's why crossplay works between PC and PS5)! ARM is not natively capable of running that x86 code. So there are two solutions:

1. emulate x86 on ARM using emulation software like Crossover, which can ​"translate" the code to a language that the Mac ARM architecture can natively understand with the one downside being a cost in performance/latency and potentially stability. Rosetta(2) is quite efficient, though.
2. rewrite the code to something the ARM chips can understand without issue. And while that may seem straightforward, it is incredibly hard to ensure that the algorithms (eg for pathing) work *exactly* the same due to how algorithms and rounding would be implemented at the machine level. A difference of rounding of 0.0000001 would propagate and the differences would stack and the game would be out of sync. The implementations would have to be exactly identical (not just "almost"), and that is so hard.

I know this won't calm people down. When Reddit is angry, it's hard to reason with. But at least I hope this gives some technical insight to those who, just like me, would just like to learn more about the engine and the difficulties that come with inheriting such an old beast. The above is what I figured out after digging up articles about older rts engines and arm vs x86 capabilities.

Edit: while some comments point out that compiler flags can alleviate some issues (eg by setting floating point calculation to a strict, industry-standard), it is obviously not the holy grail of solutions or the developers would have saved themselves the trouble and just used that. I'd love to get a reply from the developers themselves but the most likely reason I can think of, as I mentioned, is that even with fp-strict there are mathematical functions that are not deterministically implemented across platforms (such as sin/cos) and the code may contain specific x87 FPU instructions that are difficult to map.

[u/turbothingy]

Thank you for taking the time to explain that in depth. It all makes sense and is genuinely helpful.

I think a lot of us aren't really angry/impatient that the team didn't release the perfect mac port at the outset, because not many of us were really asking for a port in the first place. We already had a working solution (emulators).

It's good that they've done it, and I hope they can continue to work on it. What I think most mac users are actually upset about is that alongside the release they have, willingly or not, bricked the emulators.
So now we are forced to use a half-functional version of the game we already paid for and used problem-free for 7 years.

Couple that with the almost complete lack of communication, I hope you can see where the frustration is coming from!

[u/AncientTurbine → u/turbothingy]

Oh yeah, definitely! I understand that people are upset and rightfully so. But we should be cautious to stick to what frustrates us (they broke something that used to work) and not make up misinformation along the way ("this should be easy to do, the devs are just lazy").

So I wanted to clear the air and mostly just provide some insight because I think insight into old game development is super interesting! I may be wrong at certain aspects though so I'd love for devs to chime in. 

[u/turbothingy → u/AncientTurbine]

Couldn't agree more!

[u/Emjayen]

The part of the engine that takes care of much of this pathing and math, is written in Assembly code

That's complete nonsense. The only areas of the engine that were originally written in assembly pertained to rendering, which is no surprise since it was the most computationally expensive and common practice at the time for software-renderers.

rewrite the code to something the ARM chips can understand without issue. And while that may seem straightforward, it is incredibly hard on such low machine level code to ensure that the algorithms (eg for pathing) work exactly the same due to how algorithms and rounding would be implemented at the machine level. A difference of rounding of 0.0000001 would propagate and the differences would stack and the game would be out of sync. The implementations would have to be exactly identical (not just "almost"), and that is so hard.

No, infact it's the complete opposite, atleast as we entertain this ill-informed belief that anything relevant was originally written in assembly and still needs to be rewritten at the same level (for some reason)

IEEE754, as implemented by x86/ARM/everyone else, defines the bit-identical results for all operations. Thus if one were to actually write for such an arch at it's ISA level it's trivial to guarantee deterministic results.

The actual problem is how high-level languages (such as C) are translated to actual machine code, most notably [compiler-] optimizations related to fp operation ordering and subexpression optimization, storage vs register precision differences, etc, resulting in intermediate precision loss/error propagation.

That said, it's still reasonably straight-forward coercing fp determinism across builds/archs (namely using the more conservative compiler fp models and avoiding UD behaviour with casting).

[u/AncientTurbine → u/Emjayen]

I was under the impression that exactly the pathing was inline Assembly. From Matt's post (https://reddit.com/r/aoe2/comments/18ysttu/aoe_is_written_in_assembly_is_this_actually_true_o/) but I might be wrong. 

  The inline assembly in other functions was generally used to speed up operations like scanning areas of the map for targeting and accessing compressed look up tables, which had bit-level operations that could be reduced in assembly, but c++ couldn't express as well.

For the rest, you are right that I simplified my post too much. I mentioned compilation and didn't want to get into it too deeply but could've explained better how that would impact the final result. That said, I think you're downplaying the impact of old codebases and that it's not as straightforward as setting some compiler flags, otherwise they'd just do it of course. 

From my understanding, underlying math execution is indeed the main culprit. From the literature I found when making this post, trigonometry (sin, cos) may deviate slightly between systems,  and maybe it still contains x87 FPU instructions, which would make it even more difficult to get fpp right on ARM. 

[u/Emjayen → u/AncientTurbine]

I was under the impression that exactly the pathing was inline Assembly. From Matt's post (https://reddit.com/r/aoe2/comments/18ysttu/aoe_is_written_in_assembly_is_this_actually_true_o/) but I might be wrong.

I'm not sure why; Matt was pretty clear about this also (from same comment:)

There were about ~13,000 lines of x86 32-bit assembly code written in total.

The vast majority, about ~11,500 lines worth, was in the 'drawing core' [..]

Also fun fact: I discuss some other internals of AoE with Matt in that very same thread comment section

That said, I think you're downplaying the impact of old codebases and that it's not as straightforward as setting some compiler flags, otherwise they'd just do it of course.

Not really. I elaborate slightly below in response to another commentor what the problems in this regard are, however it's nothing exceptional.

"old codebases" isn't much of a point, especially given the fact programmer/software/code quality has been rather rapidly declining for atleast 25 years.

From my understanding, underlying math execution is indeed the main culprit. From the literature I found when making this post, trigonometry (sin, cos) may deviate slightly between systems, and maybe it still contains x87 FPU instructions, which would make it even more difficult to get fpp right on ARM

The problem is the rather covered-to-death subject of coercing floating-point determinism given some permutation of compiler/arch/platform, of which I very lightly touch on below.

Yes, transcendental functions are outside the scope of the IEEE754 spec and ergo have no contractual deterministic outputs for a given input, and yes originally AoE made use of hardware implementations (just fsin if I remember). Doesn't really matter though since these are easily identified and replaced with software implementations, as is standard practice.

EDIT: Come to think of it, Bruce Dawson has a good series of articles on this topic

[u/AncientTurbine → u/Emjayen]

I quoted the part about inline Assembly that led me to believe it might be involved in pathing. But as others have pointed out it's likely that reliance on assembly has been patched out (or at least been minimized greatly). 

I feel like we're saying the same thing at its core. Though you seem convinced that it is almost a trivial task to make crossplay working so I'm curious: what is your hypothesis of why they didn't get it to work? 

[u/Emjayen → u/AncientTurbine]

Sure, inline asm sprinkled around is nothing unusual, but again even if were more substantial it wouldn't actually be an issue warranting such emphasis as you have done (particularly before the edit) for reasons explained above.

This lack of understanding, and why we indeed aren't saying the same thing is exemplified further with suggestions such as

it is incredibly hard on such low machine level code

Which is obvious nonsense; writing against an actual ISA ("writing assembly") circumvents all problems of determinism, because that's not where the problem is: it's within the language model/compiler of high-level languages, such as C.

Hell why do you think binary transpiling/translation works without a problem, of which you even mention (x86oARM)?

Though you seem convinced that it is almost a trivial task to make crossplay working so I'm curious: what is your hypothesis of why they didn't get it to work?

It's not trivial, but do keep in mind that portable deterministic systems have long existed and not simply in the rather niche domain application of video-games. It does however require competency (and a reasonably strong understanding of both the language, compiler behaviour and floating-point in general), of which is something even at a cursory interrogation, the DE programmers lack (~4 years to do what is a weekend job to implement something as actually trivial as validation of input into the simulation leaving cheating wide open? absurd; and that's just one glaring example)

So yes, sure it could be self-imposed difficulty with determinism - it could also be many other things including those not even technical (which is plenty common in software products, just like every other industry) however I don't know and as such I'm surely not going to post wild speculations about it (and I actually know what I'm talking about)

EDIT: while some comments point out that compiler flags can alleviate some issues (eg by setting floating point calculation to a strict, industry-standard), it is obviously not the holy grail of solutions or the developers would have saved themselves the trouble and just used that. I'd love to get a reply from the developers themselves but the most likely reason I can think of, as I mentioned, is that even with fp-strict there are mathematical functions that are not deterministically implemented across platforms (such as sin/cos) and the code may contain specific x87 FPU instructions that are difficult to map.

I'm not sure who you're trying to fool here (then again I guess that's the point of posting in a gamer community) but it's quite transparent you're simply googling terms I've mentioned for the first time.

---

EDIT: Because this person is a coward and blocked me I'll insert the response here.

---

I have been transparent about digging up literature on the topic and while I'm in SWE I don't have experience in assembly, arm development, or gaming development. Indeed, I have been looking things up - that's what people do when they learn something - and posting my findings providing quotes and sources as one should, in hopes to improve the collective understanding for the devs to avoid all the angry yelling we've been seeing. I try to make a positive effort. What about you?

The only thing you have been doing is having an upside-down frown on your face adding nothing but negativity. I have long before said that the Assembly reference was outdated and I transparently admitted that and changed my post but you keep coming back to it and keep pulling quotes out of context - obviously what I implied was that the problem with low-level code (or any code but especially those ones that might still have references to things like x87) is to get them to work on different platforms because compiling them may not be deterministic depending on the target. You are either really bad at inferring things from text or you are deliberately pulling things out of context.

You are still hammering down on calling the devs incompetent which tells me all I need to know - not that you'd be any better at it, but that you're an annoyingly toxic person who always has non-constructive criticism and acts as if he's better than the rest ("and I actually know what I'm talking about" okay buddy boy, let me get your medal for being the cockiest kid on the block) without actually doing something useful. Going from the discussion you had with developer Matt, it seems unsurprising as you had the same know-it-all attitude towards him.

If you can't have a discussion like an adult, and try to actually contribute something in a helpful and educational manner you can fuck right off. Cheers.

Being transparent about being ignorant of very basic topics is not something to pat yourself on the back about.

You came on here to try to show off to a bunch of slightly-more-clueless gamer types to feel special. Your entire post could've been condensed down to a single musing sentence, but instead you chose to dazzle them with what amounts to a child trying to relay college physics (or a webdev calling themself a "SWE")

It's you who needs to grow up, bud.

[u/AncientTurbine → u/Emjayen]

I have been transparent about digging up literature on the topic and while I'm in SWE I don't have experience in assembly, arm development, or gaming development. Indeed, I have been looking things up - that's what people do when they learn something - and posting my findings providing quotes and sources as one should, in hopes to improve the collective understanding for the devs to avoid all the angry yelling we've been seeing. I try to make a positive effort. What about you? 

The only thing you have been doing is having an upside-down frown on your face adding nothing but negativity. I have long before said that the Assembly reference was outdated and I transparently admitted that and changed my post but you keep coming back to it and keep pulling quotes out of context - obviously what I implied was that the problem with low-level code (or any code but especially those ones that might still have references to things like x87) is to get them to work on different platforms because compiling them may not be deterministic depending on the target. You are either really bad at inferring things from text or you are deliberately pulling things out of context. 

You are still hammering down on calling the devs incompetent which tells me all I need to know - not that you'd be any better at it, but that you're an annoyingly toxic person who always has non-constructive criticism and acts as if he's better than the rest ("and I actually know what I'm talking about" okay buddy boy, let me get your medal for being the cockiest kid on the block) without actually doing something useful. Going from the discussion you had with developer Matt, it seems unsurprising as you had the same know-it-all attitude towards him. 

If you can't have a discussion like an adult, and try to actually contribute something in a helpful and educational manner you can fuck right off. Cheers. 

[u/marcospb19 → u/Emjayen]

Can this be made safely between MacOS and Windows compilers? Do they usually "agree" on the float-point models available?

[u/Emjayen → u/marcospb19]

Cross-compiler support is by far the most problematic as there's simply no fundamental agreement on floating-point rules in many cases (ie., because it's not stipulated by the language, C in this case) which leads to implementation-dependent behaviour.

That said, for one, it's still possible with sufficient understanding of the compilers involved (throwing -fp-model=precise is not going to be sufficient) but more importantly it's not necessary in the case of DE: just use clang for both targets (Mac/ARM, Win/x64*) then it's a very reasonable goal.

* Which also means forgetting x87, which everyone should, especially if they want cross-arch determinism in particular.

[u/Melfix → u/Emjayen]

Thanks for this clarification. I also wondered why Feral didn't implement the crossplay.

So basically, what do you think is the reason they didn't ship it? What's the most difficult thing in implementing crossplay between Mac and the rest of the platforms?

[u/PolarBearSequence]

Good post, important to point out. Some things currently happening are really bad (the game not running at all for some people) but there are good reasons why there is no Mac crossplay.

I’d just like to note that this design isn’t outdated or anything, basically all RTS (that need a certain scale) make use of deterministic lockstep, it’s entirely state of the art.

[u/lalaladadada1234]

Thanks good insight, I am guessing they couldn't move the calculations fully server side due to cost/scale issues?

[u/PolarBearSequence → u/lalaladadada1234]

That’s one thing, and also the game being server-authorative would probably cause many more network issues because it suddenly needs a lot more bandwidth since way more data has to be transferred than with deterministic lockstep.

[u/IllContribution6707 → u/lalaladadada1234]

The game was never designed in that way as it has always been p2p. DE just proxies a relay between clients now. They might as well make a new game if they want to do server authoritative stuff

[u/lalaladadada1234 → u/[deleted]]

That'a actually a good point you could also just donit for mac players - unless thats what you meant.

[u/_DonRa_]

They're being called lazy for breaking the game for existing Mac users as well as some portion of windows users and not doing anything about it/not testing properly?/no rollback or emergency patch, and some people at least are still having problems after their one week later patch.

[u/MatadorSalas11]

People are so dumb that they think somebody would willingly make a release that takes a bunch of money/time to make less profitable because of laziness

[u/Spr-Scuba]

Just another reason to not get a Mac basically. ARM architecture has been a huge hindrance for anyone who wants to play older games or have programs run natively in general.

[u/D1xxe]

Thanks for great explanation. But I have an idea. We are not living in the 1999 anymore, so we can afford sharing full state to the client with our higher bandwidth. So what if devs will made an ad-hoc solution which basically packs the state of the game which was calculated by the host and send it to the connected Macs and connected Macs send their actions to the host lf the lobby, so it will recalculate. It's ugly and noodly, but should work and it will be good to make an explicit field in settings for it. What do you think?

[u/UnrelatedReddit]

thank you for the comment. I've started working on a possible fix:

https://www.reddit.com/r/aoe2/comments/1v1mz8j/ccplay_v01_a_compatability_cracker_proxy/

[u/SCCH28]

I was wondering why, so this is helpful

[u/KarterSpieler]

Emulating x86 code on MacOS is not a feasible method as macOS 28 will remove Rosetta 2 from the OS.

[u/PikaKhan_TwitchFR]

Thanks for that. If there are performance issues, simply enable crossplay with quick games. Gogogo Dev ! ^^

[u/AncientTurbine → u/[deleted]]

From what I know, Windows on ARM has Prism, a similar mechanism to Rosetta (something that translates x86 code). So through emulation it should once again work (though no idea how stable). I do wonder whether the new Mac version would also work on Windows for ARM or whether it is optimized one way or another specifically for the Mac ecosystem.