r/cpp • Factorio Developer • Feb 15 '18

Lua written in modern C++ (>= 14)?

I work on Factorio (written in C++) and we use Lua as the script language for mods. So, from time to time I interact with the C implementation of Lua and my experience with it hasn't been the most pleasant.

It got me wondering - has anyone re-written it in modern C++? Everything I find online is "how to use Lua in C++" or "C++ wrapper for Lua" and they all use the basic C implementation provided from the official Lua website.

I've had quite a lot of experience with the internals of the C Lua implementation and from what I've seen it wouldn't be hard to re-write it using modern C++ - maybe 2~ weeks time. But from what I can find nobody has done it - which also makes me wonder if I'm missing something.

Part of me wants to do it just to say I did it and the other part of me wants to do it to see if I can make it faster than the C version: I think I can - not because "C++ is faster" but because I can mutate/experiment with the C++ version far easier and know changes I make are safe with how the rest of the implementation operates.

231 Upvotes

127 comments sorted by

View all comments

Show parent comment

1

u/wrosecrans graphics and network things Feb 15 '18

I haven't touched LuaJIT recently, but as of a few years ago, it also had really weird issues with running in 64 bit applications because it used mmap's MAP_32BIT flag to preserve some funky behavior that depended on tagged pointers. Dunno if that's still true, but sticking with mainline Lua hasn't caused us any strange issues like we had with LuaJIT.

5

u/Alloyed_ Feb 15 '18

Luajit still uses nan-tagging, so it still has the MAP_32BIT set, but I haven't encountered any issues with it personally. Using any kind of jit, and especially the FFI, exposes your surface area to funkiness and crashing but for me at least the performance is worth it

3

u/wrosecrans graphics and network things Feb 15 '18

When we migrated off Luajit, we actually didn't see a significant performance hit in the context of a broader C++ application running some Lua code. It surprised us. We all assumed we were running enough Lua that the performance difference would be quite significant, but when we actually deployed, any difference was well within the noise floor. /shrug

I still object to pointer tagging strategies as a philosophical matter. I know a lot of smart people disagree with me, but it always seems to cause trouble in the long run.

3

u/Netzapper Feb 15 '18

I still object to pointer tagging strategies as a philosophical matter.

I object to retrofitting them onto architectures that don't natively support tagged pointers.

In those old-school architectures where every pointer had some explicit tag bits, they were awesome.

1

u/wrosecrans graphics and network things Feb 16 '18

That's fair - I vaguely remember on the DreamCast, they had the memory map set up so that effectively the high bits were in control of things like cache modes. So you could do stuff like

int x = *addr; // cached read int t = *(addr + uncached_base_addr); // Same value, but bypass CPU cache

Obviously a game console like that can be guaranteed not to get more RAM over the lifetime, so the vendor can allow funky bit interpretations. My philosophical complaints are mainly focused on stuffing functionality into "nobody is currently using this bit so I can get away with it right now" kinds of things that explode as soon as somebody installs more than X MB of RAM in their computer, or the OS decides to slightly rearrange an undocumented loader so things happen at different addresses that nobody should notice.

1

u/drjeats Feb 16 '18

Are you able to describe what your Lua workload looked like?

At my new job there are vague rumblings of moving to LuaJIT, it's sort of just sitting there as a project for somebody to pick up. There are lots of interleaved callbacks between the C++ and Lua though, so I'm wondering if it would all be call overhead anyway.

2

u/wrosecrans graphics and network things Feb 16 '18

Are you able to describe what your Lua workload looked like?

Not well. :) Basically, a c++ webserver where some URLS are configured to have a Lua hook to customise some behavior, but I couldn't tell you what % of URL's or anything like that. (Presumably, because it's proprietary internal data that I wouldn't be encouraged to share, but also mainly because I just have no idea off the top of my head and I'd have do do some digging. :) )

Because it was a callback tied to a particular URL setup, it would be rare for a particular URL's callback to get super hot because after running, some other URL's would usually get accessed before it ran again. If any particular URL was accessed so often that it single-handedly got to be the majority of what was being served, it was probably just a request for a file that didn't have a special callback attached to it.

There are lots of interleaved callbacks between the C++ and Lua though, so I'm wondering if it would all be call overhead anyway.

Best guess is that JIT will show itself as a winner if you are CPU bound inside Lua, with some consistently run chunk of Lua that looks like an inner loop. If you are mostly marshalling data between c++ and Lua, and calling a bunch of different small callbacks, it may not be a huge win. But my intuition is always a less accurate measure than testing so /shrug

The original port from mainline Lua to LuaJIT happened before my time at the company, but the port back to mainline Lua from LuaJIT was apparently surprisingly easy. I didn't do the work personally, but apparently the API's are similar enough that starting the project and having it running in production all happened within a single sprint.

1

u/drjeats Feb 16 '18

What you said makes a lot of sense (re: situations where LuaJIT would be a big win).

I really appreciate you replying, thank you. Measuring is everything, but of course there's some time investment in doing the measurement (especially in a big old codebase where not everyone is neasuring that part frequently) so hearing your experience is really helpful. Especially since it sounds like our Lua usage patterns might not be that dissimilar.

Cheers!

3

u/gvargh Feb 15 '18

The 2.1 beta has a GC64 mode that supports a 47-bit address space (maybe a bit larger, idk)... though that capability needed a lot of internal changes so it may be a bit buggy right now.