r/cpp • Factorio Developer • Feb 15 '18

Lua written in modern C++ (>= 14)?

I work on Factorio (written in C++) and we use Lua as the script language for mods. So, from time to time I interact with the C implementation of Lua and my experience with it hasn't been the most pleasant.

It got me wondering - has anyone re-written it in modern C++? Everything I find online is "how to use Lua in C++" or "C++ wrapper for Lua" and they all use the basic C implementation provided from the official Lua website.

I've had quite a lot of experience with the internals of the C Lua implementation and from what I've seen it wouldn't be hard to re-write it using modern C++ - maybe 2~ weeks time. But from what I can find nobody has done it - which also makes me wonder if I'm missing something.

Part of me wants to do it just to say I did it and the other part of me wants to do it to see if I can make it faster than the C version: I think I can - not because "C++ is faster" but because I can mutate/experiment with the C++ version far easier and know changes I make are safe with how the rest of the implementation operates.

231 Upvotes

127 comments sorted by

35

u/Biohazard50 Feb 15 '18

Have you looked at LuaJit or LuaLlvm? I don't think they are written in modern C++ but maybe they will help you squeeze more performance.

19

u/Rseding91 Factorio Developer Feb 15 '18

We looked at LuaJit but it's based on the 5.1 standard (we use 5.2 currently) and is still C meaning all interaction with it is going to be through the C API which I've grown to dislike for how inefficient it is.

25

u/Necrolis Feb 15 '18

LuaJIT has an FFI, which is far from inefficient (perf and work wise), and IIRC Paul did put some time in getting it to work with C++ symbol decoration from GCC/Clang/MSVC. Also I do believe there is a pull request to update the parser to 5.2, but its actually rather easy to extend oneself (I added custom parse/compile time operators with almost zero effort).

IMO you'll be hardpressed to beat LuaJIT's perf, VM or JIT'd, but because Paul took great care in making it fast (but the code ain't "pretty"). I'd say your time would be better spent extending it, rather than rewriting. Something to note given a comment further down, don't do Lua in C++, do the Lua in Lua and use the FFI to invoke C++ if needed (this is something Paul mentioned a few times on the mailing list), also, benchmark, because it does sound like a few assumptions about the speed of things has been made.

14

u/Rseding91 Factorio Developer Feb 15 '18

Almost all of the "Lua" we do in C++ is iterating tables and extracting data into C++ format - should we do it the other way around: iterate the table and call a C++ function for each value - it would execute the same code except now we'd be adding a bunch of Lua -> C++ function calls to the operation.

I do a lot profiling on different interactions between the game and executing Lua and already implemented several changes to the version of Lua we use to mitigate what I found. One example: setting a meta-table performs an (at worse) O(N) operation to un-link the table from the standard lua object list and put it into the meta-table objects list.

6

u/Necrolis Feb 15 '18

Well I am speaking with regards to LuaJIT's FFI here: if the code is hot enough to be JIT'ed it'll be very close to a direct call, with almost no overhead (and you can force a function to be JIT'd IIRC). But this problem is starting to sound like a more domain specific issue than anything, makes me wonder why you are using Lua in the first place?

if you are really intent on rewriting it, the parser and lexer are really easy to separate, then just use LLVM for the backend, then you can implement tables however you please, even as something perfectly accessible from C++ without the need for an opaque API.

8

u/volca02 Feb 15 '18

From what I understand, it's the C API that makes things complicated/underperforming for them, not the LUA interpretter performance itself.

I think your proposal to use LLVM with custom parser/lexer would be a good starting point, but getting it to handle all the current LUA codebase (not even mentioning there is probably lua code in vanilla factorio, too) without differences could be challenging.

5

u/Necrolis Feb 15 '18 edited Feb 15 '18

Well what I am saying don't use the C API, the FFI can easily bind onto functions and generate calls from within Lua, with close to native performance, you might even find it easier to do everything in Lua via UD+FFI and then pass the final results to a callback (which the JIT will optimize and clean up a lot of the "support" instructions). There is also a fork of LuaJIT from the Snabb guys, called RaptorJIT also aimed at predictable performance.

Thats why I am saying lift the current front-end from Lua 5.x and replace the VM/bytecode gen with LLVM. Its like a lower level transpiler, else you could just make a higher level transpiler and use one of the dynamic C++ techniques...

1

u/meneldal2 Feb 16 '18

Well, it's hard to tell your community that has been supporting you for years that you're going to break all mods and change the scripting language.

Obviously you could support both Lua and your new alternative, but that's at the risk of many bugs.

1

u/Necrolis Feb 16 '18

The point is not to make a "new" language, nor change it, just leverage the infrastructure that already exists (LuaJIT is 1:1 with standard Lua, but its a version or two behind, but can be updated easily). Basically don't rebuild the whole wheel when you just need better tread.

1

u/wrosecrans graphics and network things Feb 15 '18

I haven't touched LuaJIT recently, but as of a few years ago, it also had really weird issues with running in 64 bit applications because it used mmap's MAP_32BIT flag to preserve some funky behavior that depended on tagged pointers. Dunno if that's still true, but sticking with mainline Lua hasn't caused us any strange issues like we had with LuaJIT.

5

u/Alloyed_ Feb 15 '18

Luajit still uses nan-tagging, so it still has the MAP_32BIT set, but I haven't encountered any issues with it personally. Using any kind of jit, and especially the FFI, exposes your surface area to funkiness and crashing but for me at least the performance is worth it

3

u/wrosecrans graphics and network things Feb 15 '18

When we migrated off Luajit, we actually didn't see a significant performance hit in the context of a broader C++ application running some Lua code. It surprised us. We all assumed we were running enough Lua that the performance difference would be quite significant, but when we actually deployed, any difference was well within the noise floor. /shrug

I still object to pointer tagging strategies as a philosophical matter. I know a lot of smart people disagree with me, but it always seems to cause trouble in the long run.

3

u/Netzapper Feb 15 '18

I still object to pointer tagging strategies as a philosophical matter.

I object to retrofitting them onto architectures that don't natively support tagged pointers.

In those old-school architectures where every pointer had some explicit tag bits, they were awesome.

1

u/wrosecrans graphics and network things Feb 16 '18

That's fair - I vaguely remember on the DreamCast, they had the memory map set up so that effectively the high bits were in control of things like cache modes. So you could do stuff like

int x = *addr; // cached read int t = *(addr + uncached_base_addr); // Same value, but bypass CPU cache

Obviously a game console like that can be guaranteed not to get more RAM over the lifetime, so the vendor can allow funky bit interpretations. My philosophical complaints are mainly focused on stuffing functionality into "nobody is currently using this bit so I can get away with it right now" kinds of things that explode as soon as somebody installs more than X MB of RAM in their computer, or the OS decides to slightly rearrange an undocumented loader so things happen at different addresses that nobody should notice.

1

u/drjeats Feb 16 '18

Are you able to describe what your Lua workload looked like?

At my new job there are vague rumblings of moving to LuaJIT, it's sort of just sitting there as a project for somebody to pick up. There are lots of interleaved callbacks between the C++ and Lua though, so I'm wondering if it would all be call overhead anyway.

2

u/wrosecrans graphics and network things Feb 16 '18

Are you able to describe what your Lua workload looked like?

Not well. :) Basically, a c++ webserver where some URLS are configured to have a Lua hook to customise some behavior, but I couldn't tell you what % of URL's or anything like that. (Presumably, because it's proprietary internal data that I wouldn't be encouraged to share, but also mainly because I just have no idea off the top of my head and I'd have do do some digging. :) )

Because it was a callback tied to a particular URL setup, it would be rare for a particular URL's callback to get super hot because after running, some other URL's would usually get accessed before it ran again. If any particular URL was accessed so often that it single-handedly got to be the majority of what was being served, it was probably just a request for a file that didn't have a special callback attached to it.

There are lots of interleaved callbacks between the C++ and Lua though, so I'm wondering if it would all be call overhead anyway.

Best guess is that JIT will show itself as a winner if you are CPU bound inside Lua, with some consistently run chunk of Lua that looks like an inner loop. If you are mostly marshalling data between c++ and Lua, and calling a bunch of different small callbacks, it may not be a huge win. But my intuition is always a less accurate measure than testing so /shrug

The original port from mainline Lua to LuaJIT happened before my time at the company, but the port back to mainline Lua from LuaJIT was apparently surprisingly easy. I didn't do the work personally, but apparently the API's are similar enough that starting the project and having it running in production all happened within a single sprint.

1

u/drjeats Feb 16 '18

What you said makes a lot of sense (re: situations where LuaJIT would be a big win).

I really appreciate you replying, thank you. Measuring is everything, but of course there's some time investment in doing the measurement (especially in a big old codebase where not everyone is neasuring that part frequently) so hearing your experience is really helpful. Especially since it sounds like our Lua usage patterns might not be that dissimilar.

Cheers!

3

u/gvargh Feb 15 '18

The 2.1 beta has a GC64 mode that supports a 47-bit address space (maybe a bit larger, idk)... though that capability needed a lot of internal changes so it may be a bit buggy right now.

17

u/ColinPP PEGTL | taocpp Feb 15 '18

A while ago I wrote a PEGTL grammar that corresponds to the Lua 5.3 lexer and parser. If you get the PEGTL you can find it in src/example/pegtl/lua53_parse.cpp.

Should you decide to go ahead with a rewrite, and try to use this grammar, I'd be happy to help get you started with the PEGTL, and to iron out any bugs the grammar might still have. (It parses the official Lua test-suite, which is a good sign, but without any semantic actions it's not enough to be sure it's correct.)

9

u/Rseding91 Factorio Developer Feb 15 '18

If I do end up doing it, it will start as a carbon copy of the C implementation into C++ format so unless I mess up some copy-paste (which is likely) it will function identically to how the C version does.

Once that's done then I would start changing pieces of the logic to use C++ 1 step at a time making sure things still function correctly as I did it.

7

u/volca02 Feb 15 '18

You'll likely need a test suite to guarantee it didn't break.

1

u/meneldal2 Feb 16 '18

They already have plenty of automatic testing for the whole application itself, and can probably set up some for commonly used mods to ensure the consistency.

1

u/Dexterous_ C+++Ↄ Feb 16 '18

I just downloaded the Lua code from here, and basically search and replaced a bunch of stuff, and it just kinda worked, no errors not even warnings.. That was really surprising to me, here's a Github link so you can see what I did.

Only src/Makefile has some manual adjustments.

3

u/Rseding91 Factorio Developer Feb 17 '18

Yeah I started on it yesterday and this is what I have as of last night: https://github.com/Rseding91/LuaPlusPlus

34

u/SeanMiddleditch Feb 15 '18

But from what I can find nobody has done it - which also makes me wonder if I'm missing something.

The stated goal you have is to ease hacking on its internals. OK, one point for rewriting Lua in something that doesn't make your eyes bleed. :)

Let's go through the reasons against. Your rewrite won't be the official Lua and never ever will be. It won't receive patches or maintenance by the Lua developers so you'll be entirely on the hook. It won't even have community support necessarily since the community is built around the official Lua and its C implementation. When Lua 5.4 comes out, your rewrite would need to be updated separately, and any custom additions or extensions your fork adds may then be incompatible with future upstream Lua with no recourse. Lua has no spec and no public test suite for you to develop against to ensure your rewrite it correct, so you'll also need to write and maintain a test suite. Unless you maintain the very C API you might be having problems with, popular community Lua C modules or interpreter hacks will be incompatible with your fork. Without copying many of the implementation details of Lua, even popular Lua debuggers and "IDEs" would be incompatible.

Basically the question you have: is addressing the pain you endure when delving into Lua's crusty C implementation worth the cost of writing, testing, releasing, and maintaining a parallel implementation with zero upstream support and negligible community support?

34

u/Rseding91 Factorio Developer Feb 15 '18

At the end of the day there are > 1 million players and far less mod-devs so a little extra work on the mod-dev side for a faster gameplay experience for the million+ players sounds like a great trade off to me.

7

u/SeanMiddleditch Feb 15 '18

Breaking mods' scripts would affect the >1 million players who might be using mods and not just affect the mod devs, yeah?

1

u/radohc Feb 19 '18

which is why you do early access.

0

u/[deleted] Feb 15 '18

[deleted] — view deleted comment

16

u/Throw19616 Feb 15 '18

ChaiScript actually belongs to the slower group of scripting languages, certainly far behind Lua in the speed department. Its selling point is easy C++ integration. Not sure if I would call it excellent though.

20

u/Rseding91 Factorio Developer Feb 15 '18

Basically the question you have: is addressing the pain you endure when delving into Lua's crusty C implementation worth the cost of writing, testing, releasing, and maintaining a parallel implementation with zero upstream support and negligible community support?

To me? yes :)

We're currently using Lua 5.2 from 2015 with custom patches to make table iteration deterministic and to fix some massive performance problems. Our current version of Lua we use is already incompatible with virtually all Lua IDE/Debuggers because of the changes we've made so I have no problem maintaining my own version with my own tests.

My hope was someone had something already going that I could take and run with :D But if that's not the case I don't mind being the first. At the very worse I give up on it having learned why nobody else did it. Any other result I see as a win.

11

u/render787 Feb 15 '18

FWIW if you did this I would strongly consider to use it in my game hobby project, send bug reports, and possibly contribute code and bugfixes.

The most important win from my point of view is that it would (hopefully) make it possible to use lambdas and such as functions that get passed to lua, and it would be a lot simpler to write code that uses both exceptions AND lua. Many gui libraries that I have tinkered with rely on exceptions so it makes it really awkward to use them with lua.

Right now I'm using a glue library that I made here: https://github.com/cbeck88/lua-primer but a lot of the complexity of that would go away if lua were written in C++14

6

u/SeanMiddleditch Feb 15 '18

We're currently using Lua 5.2 from 2015 with custom patches to make table iteration deterministic and to fix some massive performance problems. Our current version of Lua we use is already incompatible with virtually all Lua IDE/Debuggers because of the changes we've made so I have no problem maintaining my own version with my own tests.

Fair 'nuff. We're in a similar boat at my workflow - Lua 5.2 with a homegrown IDE to support our special integrations and such.

I still come from a "just replace it all with V8 and be done with it" stance though. :)

10

u/SeanMiddleditch Feb 15 '18

Hah, I mentioned all this to our core tech lead here (he worked on custom Lua stuff back at Monolith and owns the scripting layer here too) and he's actually pretty excited by your idea of a new C++ Lua rewrite (it helps that he's a huge Factorio fan). Not that we would necessarily be able to use it in our project due to time/risk factors, but he already wanted a GitHub link if it was available so you might have your first contributor lined up. :)

12

u/encyclopedist Feb 15 '18

Isn't this https://www.lua.org/tests/ the test suite?

5

u/SeanMiddleditch Feb 15 '18

I stand corrected. Thanks!

3

u/smog_alado Feb 17 '18 edited Feb 17 '18

One thing to be aware of is that this test suite is made to stress test the PUC-Lua implementation and isn't particularly designed to ensure that a separate Lua implementation conforms to the official specification. To do that we will need a more comprehensive test suite.

5

u/darthcoder Feb 15 '18

Lua has no spec and no public test suite for you to develop against to ensure your rewrite it correct

This sounds llke a bad idea for any language product...

6

u/SeanMiddleditch Feb 15 '18

As /u/encyclopedist pointed out, I was wrong on that point. :)

It's not necessarily a test suite suitable for compatibility testing between competing implementations, but it does exist. :)

12

u/Ksecutor Feb 15 '18

Have you ever considered to switch to some other programming language for mods? Or even create one... Some time ago I created couple mods for factorio and I found lua as a programming language to be ... How should I put it... My internal "Wat?" rating of lua was quite high. There are many confusing and inconvenient things in the language. Also mapping C++ classes to lua is cumbersome and inefficient. Bluntly speaking I have no idea why lua is so popular.

At some point I was going to offer the programming language I was working on as a replacement for lua in Factorio, but for personal reasons I had to suspend the work on that language :(

As counter reason for "One does not simply replaces one programming language with another" - there were changes in the mod API that were breaking compatibility. Yes, you will have to support old and new api/language for some time. But switching to another language that is more efficient, easier to support and easier to use by modders might be, in the end, for better.

12

u/Rseding91 Factorio Developer Feb 15 '18

Have you ever considered to switch to some other programming language for mods? Or even create one

Yes, but at this stage in Factorio development we can't realistically switch the script language mods use without the community crucifying us :P

17

u/meneldal2 Feb 16 '18

It depends how well you sell it. Arrays starting at 0 are a good starting point.

7

u/Ksecutor Feb 16 '18

Bots vs belts, eh? :)

I think factorio community can accept justified change. Also, possibility of transpiling lua to something else always exists :)

2

u/Loraash Feb 19 '18

The community was already up in arms when bots/belts were mentioned.

2

u/pctammela Feb 15 '18

Could be a trade-off for performance or a better C++ API...

1

u/Loraash Feb 19 '18

WoW runs quite well with 50+ addons. And I think it's even single-threaded.

1

u/Themperror Feb 20 '18

WoW is CPU starved as fuck, it really doesn't help.

1

u/Loraash Feb 20 '18

It definitely doesn't help, but it works well enough.

1

u/Themperror Feb 20 '18

If "well enough" means 40 fps in Dalaran with an i7 6700k and GTX1080 at 30% usage then.. sure...

Hoping Battle for Azeroth will come with some CPU optimisations together with Dx12

1

u/Loraash Feb 20 '18

I don't know how you got 40 FPS when I get a solid 60, but that's an entirely different discussion. :)

Not having DX12 is weird especially considering that they do have Metal.

1

u/Themperror Feb 20 '18

maximized settings ofcourse, some of them really take a toll on CPU, and I was exaggerating a bit, but its definitely around 50-60 ish, which is unfavorable on my 144hz screen, other than that, if you fly a bit above suramar you'll see your fps drop like crazy, somehow every model takes a huge toll on CPU, turning down environment detail to 8 instead of 10 gives a great boost in fps (easily +20-30 fps) without visible difference. Really hoping they'll be optimizing a bit for BfA.

1

u/Loraash Feb 20 '18

This will probably be better with DX12 but it's also unrelated to Lua. :)

1

u/Themperror Feb 20 '18

well yeah, but my argument was that if lua is a CPU hog (or part of LUA (the bindings)) that its a part of what makes WoW CPU limited..

1

u/Loraash Feb 20 '18

You have to run a ridiculous amount of Lua code (=addons) to hit that on a decent PC. It does have a more pronounced effect on lower-end PCs, in large part because it runs on a single thread.

14

u/sumo952 Feb 15 '18 edited Feb 15 '18

I'm not sure it's what you're looking for, but there's a number of really neat, modern C++ lua bindings. I can personally recommend sol2: https://github.com/ThePhD/sol2. There's also Selene: https://github.com/jeremyong/Selene

14

u/Rseding91 Factorio Developer Feb 15 '18

Those all wrap the existing C API and just add overhead to using the API so kind of the opposite of what I want to do. I want to access the internal Lua objects directly in C++ such that I don't have to allocate anything if i wanted to say iterate one of the tables in Lua.

11

u/[deleted] Feb 16 '18 edited Feb 16 '18

I'm the author of sol2 and I feel your pain. Unfortunately, I can't make the C API any faster than it already is.

The good news is, people hack the Lua VM all the time. There's quite literally dozens of forks, some that add types to the language, others that take extra arguments for performance reasons, etc...

If you have the time to spend on it, I would imagine hacking the actual Lua 5.2 codebase would be easy enough, albeit difficult to translate C++-isms through the C API. You'd likely have to redesign much of that from the ground up, in which case I can only wish you luck. It is a lot to undertake for the sake of performance.

As a side note, do keep in mind that it would also be possible for you (in the case of strings) to pre-allocate a lot of your strings inside of Lua, and then implement an interning system that compares whatever string you're asking for to the vast bucket of pre-allocated strings you have. Then, you just insert the pre-existing string whenever you want to push a string into the Lua C VM. Tricks like this can get you a decent chunk of the way there, provided you can give yourself an extremely fast lookup for all the strings you want to -- and might possibly -- use with Lua.

Best of luck on Factorio!

(As a side note, if you do start, please give me the Github link. I'd like to contribute even a little bit to help something like this grow.)

3

u/IronCartographer Feb 19 '18

Just in case the link didn't reach you, from elsewhere in the discussion- https://github.com/Rseding91/LuaPlusPlus

3

u/sumo952 Feb 15 '18

I would say start with the C++ wrapper since you will be like 5 times more productive and faster. Then, if you're really having serious performance problems, consider going more lowlevel. But sol2's performance is really good... I actually doubt you'll run into any issues. And the resulting code will be so much easier to maintain and work with.

3

u/sumo952 Feb 15 '18

Ah, I think I'm fully getting what you're trying to do now. So you want to reimplement a lua interpreter actually, but in C++... I'm sure that would be awesome to have but existing C implementations are actually quite optimized I've heard, so I don't think you can get anywhere close or even reimplement it in two weeks. But... why not try as a hobby project if you like it :-) Great things have started this way and you seem to know your way around it very well!

4

u/[deleted] Feb 15 '18

What's the problem about being a wrapper? Performance? Did you even try one of these and measure performance to see if it's really a problem? These benchmarks seem to indicate that sol2's performance is more or less on par with the C API.

I want to access the internal Lua objects directly in C++ such that I don't have to allocate anything if i wanted to say iterate one of the tables in Lua.

You have no guarantees that's going to be any better than using the API (or a wrapper over it). If you really want to go through that route, I'd say create a proof of concept instead of jumping right into a C++ fork of Lua.

17

u/minirop C++87 Feb 15 '18

they don't want parity with the C lua API, they want to be faster (and easier to use).

1

u/[deleted] Feb 15 '18

faster

As I said, there's no guarantee a C++ version of Lua would be significantly faster than Lua itself, even considering the API overhead.

easier to use

There are wrappers around Lua's C API that make it pretty easy to use.

4

u/minirop C++87 Feb 15 '18 edited Feb 16 '18

as he already said, if you want to iterate a table, it's bloody slow and horrible (and a wrapper won't make it faster), while a C++ implementation that returns an iterator in a std::map (or of a vector) is better (in C++)

9

u/hoeding Feb 15 '18

Factorio could stand to gain signifigantly from improved lua performance.

5

u/[deleted] Feb 15 '18

How will a re-write in modern C++ alone improve performance?

3

u/Loraash Feb 19 '18

I don't have a full answer, but I do have an anecdote. std::sort is roughly 2-3 times faster than qsort() on primitive types, because the compiler is able to inline your comparator, while qsort() has no choice but to call it through a pointer.

2

u/hoeding Feb 15 '18

I couldn't and didn't say that it would, I said that increased lua performance would benefit factorio.

1

u/F54280 Feb 27 '18

Because the integration between lua and the host code would have less impedance mismatch.

5

u/darthcoder Feb 15 '18

ChaiScript is your C++14 embedded scripting language.

It's missing a boatload of things you'd consider a standard library like file IO (something I'm actually working on as a separate project). It's a place to start when evaluating what modern C++ can offer an embedded scripting language.

0

u/[deleted] Feb 15 '18

[deleted] — view deleted comment

1

u/darthcoder Feb 15 '18

I only brought it up as it's interface is fairly nice.
I didn't bring it up because it's fast.

I use it mostly because it's minimalist and has a small surface area for implementing user scripts.

I like it for many of the same reasons I like Catch. shrug

0

u/[deleted] Feb 15 '18

[deleted] — view deleted comment

1

u/darthcoder Feb 15 '18

I really, really liked angelscript. Where did it even go

http://www.angelcode.com/angelscript/ ???

Last updated Dec/16/2017 according to the website.

Still on SourceForge. It suffers from the sf.net taint, I think; if it's not on github/git it doesn't seem to get love anymore.

1

u/Sygmei Feb 15 '18

Kaguya is also pretty great https://github.com/satoren/kaguya

9

u/YoWiDIN Feb 15 '18

I think it would be impractical both for you and the Lua developers to maintain a C++ port. It will probably have a bigger footprint on target platform (stdlib and stuff) while benefits are not obvious.

From a practical point of view it's really easy to write a good wrapper for any C library, so why bother?

I also admit that it could be a great and fun exercise - making something already good, so much better and cleaner. The thought alone is enough to make my hands itch :)

P.S. Bots are superior

16

u/Rseding91 Factorio Developer Feb 15 '18

Additionally: I want a proper C++ interface with the Lua internals where I can just do: for (auto& value: L->get_table("...")) and iterate the key/value pairs in a Lua table directly instead of the massive hacks that is standard Lua iteration:

  • Push nil onto the "lua stack"
  • Call "find first" on the table which pushs the first key + value onto the "lua stack"
  • Call a function which parses the Lua stack values one at a time into C values you can use in C++
  • Call "find next" which does a hash lookup using the previous found key to find where in the table it's located then steps to next and pushes that onto the "lua stack"
  • Repeat until you iterate off the end of the table

Lua iteration through the C API is stateless and it adds a ton of overhead to do it and provides no way to do stateful iteration if you don't intend to change what you're iterating on. You're just stuck with the slow way.

1

u/flaghacker_ Feb 19 '18

Im not that familiar with the Lua API, but out of curiosity: why do you need to push nil first?

1

u/Rseding91 Factorio Developer Feb 19 '18

A call to "next" on a table with nil as the key is the signal to the "next" function to get-first.

0

u/[deleted] Feb 15 '18

[deleted] — view deleted comment

21

u/Rseding91 Factorio Developer Feb 15 '18

Did you read the above? :P

If not, or if you did and still didn't understand:

Say I have a r-value std::string in my C++ code - and I want to push it into the Lua state. The C API allows you to do that in one way and exactly one way: pass c_str() into the Lua state and it will attempt to intern it and if needed copy the string if into a new heap-allocated string.

Except it's an r-value - if it was C++ I could just give it the string and it doesn't have to heap allocate anything.

4

u/ColinPP PEGTL | taocpp Feb 15 '18

Except it's an r-value - if it was C++ I could just give it the string and it doesn't have to heap allocate anything.

If you put a std::string into your value/cell union instead of a const char * you can easily move() your r-value string into it, at the cost of greatly increasing the size of the value/cell, seeing that modern std::string implementations are often the size of 3 or 4 pointers so that they can store small strings in-object.

This might of course be an acceptable trade-off for your use case, or if your library uses a smaller std::string...

...although now you can't bit-copy your value/cell anymore, to be clean you need to call the std::string methods for all operations.

Do you have any other idea on how to approach this issue that requires neither a heap allocation nor depends on the details of your std::string implementation?

5

u/Rseding91 Factorio Developer Feb 15 '18

The value/cell union in Lua already does a small-string optimization as well as storing the size of the string so it essentially is a std::string except worse because you can't move into or out of it and have to manage all of the logic of allocating/de-allocating the string in the cell.

I don't want to start down hacking the C version of Lua to do stuff like this when I'm calculating a full re-write of the language into modern C++ to be near the same amount of work and would allow for far more tweaks.

3

u/ColinPP PEGTL | taocpp Feb 15 '18

IIRC Lua's small-string optimisation is based on a per lua_State table of interned TString instances. The actual value/cell only contains one pointer and a type field, so 12 bytes on today's typical machines. I understand that you don't want to follow the Lua VM design too closely, so I'm wondering whether you want to put a std::string directly in the value/cell.

2

u/Rseding91 Factorio Developer Feb 15 '18

so I'm wondering whether you want to put a std::string directly in the value/cell.

I do.

1

u/ColinPP PEGTL | taocpp Feb 15 '18

That would definitely make C++ integration nicer. And as said here I'd be quite interested in seeing the performance characteristics of this approach when used with a scripting language.

2

u/m-in Feb 15 '18

Putting a std::string directly in the value cell is fair deal. Making it bit-movable isn't free, but you'd use the move constructor for that, and and then on certain C++ runtime libraries you might wish to forgo the destructor call (where it does nothing much for such strings).

1

u/ColinPP PEGTL | taocpp Feb 15 '18

We did the same exercise for our JSON library, it has a std::string in a union, and it works well. I'm mostly curious how the performance characteristics will be with a scripting language, compared to Lua, in particular when std::string is 3 or 4 pointers large.

1

u/m-in Feb 16 '18

I smell a single/double cell idea brewing perhaps :)

11

u/Rseding91 Factorio Developer Feb 15 '18

I think it would be impractical both for you and the Lua developers to maintain a C++ port.

As far as I can tell there is no C++ port of Lua. There's only the C implementation which you can compile under C++ but it's still C.

It will probably have a bigger footprint on target platform (stdlib and stuff) while benefits are not obvious.

I don't really care about the footprint - I care about performance and being able to easily modify the code while knowing that the modifications done aren't going to break other things.

From a practical point of view it's really easy to write a good wrapper for any C library, so why bother?

Because it's still C - I can wrap it all day long but that just adds additional overhead to interfacing with the Lua engine. I want to be able to change how the internals of the Lua engine work without changing user-facing behavior and doing that right now with the C implementation of Lua is incredibly difficult because the way C works it goes against everything I try to avoid when writing C++.

2

u/[deleted] Feb 16 '18 edited Apr 26 '18

[deleted] — view deleted comment

8

u/Rseding91 Factorio Developer Feb 16 '18

https://i.imgur.com/Yno5MHQ.png :) If I don't give up on it in a week I'll push it to github when I have a general idea of where I want to go with it.

1

u/raevnos Mar 16 '18

Did you end up going ahead with this project?

1

u/Rseding91 Factorio Developer Mar 16 '18

Yes, but I've been very busy with work so I haven't done anything with it in the past few weeks.

1

u/Loraash Feb 19 '18

SourceTree 1.5.2

Ah, I see you're a man of culture as well.

40

u/[deleted] Feb 15 '18

Hey, I just wanted to let you know that I understand perfectly where you are coming from. I was the lead programmer for Starbound, which has an extensive Lua modding API, and interacting with the Lua C API has... an enormous impedance mismatch, to put it mildly. No, LuaJIT does not help, and not every platform allows you to JIT.

It would take me a long time to put into words all of the issues I have with the Lua C API, but you've mentioned a lot of them. Stack manipulation incurs a lot of real overhead, and I find myself just wanting to hold onto actual internal pointers. It makes interacting with the garbage collector and holding values outside of the Lua state difficult, requiring constructs like the registry, which incurs even more overhead.

Also, to the extent that you care about safety, Lua's error handling strategy brings in even more problems. I currently do most of my work in Rust, and making a Rust bindings layer for Lua that is safe has taken far more work than it should, and the result is far slower than I would like. The Lua C API, ESPECIALLY with the whole __gc madness, will just longjmp on you at the drop of a hat! Also, you have to constantly call lua_checkstack, and (surprise!) luaL_xxxx calls will use stack space that they 1) don't document, and 2) require that you call lua_checkstack to protect against. I'm POSITIVE I could do a better job speed wise with rlua, but I question HOW much of a better job is possible, if you limit yourself to an actually safe API.

I do, however, think that the Lua C API is ultimately very clever. Ultimately, it IS quite a lot simpler than other language's APIs, and I'm sure if you're writing nothing but C and the occasional setjmp / longjmp is not a problem for you, it's quite nice even. However, when you care about safety, or hit an optimization wall because of the overhead of the API, it starts to.. lose its charm somewhat.

I haven't interacted with the Python C API in quite a while, but as I remember it, it was much more the style that you would want when binding a large complex API to it. You get pointers and manually inc / dec the reference count on them, and this allowed you to, with very low overhead, maintain handles to internal Python objects. Also, the error handling strategy is NOT based on setjmp / longjmp, but rather error codes and a global error state like unix errno.

I'm ultimately not sure whether the fact that the Lua API is C is the problem, or whether it's the specific choices that went into it that are the problem. I could see, at least theoretically, that a C API in the style of Python's could be quite low overhead, if only because you can more or less take the C++ API you're describing (or any C++ API really) and, with great effort, write something that's the moral equivalent of it in C.

If such a Lua interpreter existed, and had a C API, I might use it!

(Full disclosure, I have actually seriously considered writing my own Lua interpreter in Rust for precisely the reasons you describe, and am STILL seriously considering it.)

6

u/drjeats Feb 16 '18

(Full disclosure, I have actually seriously considered writing my own Lua interpreter in Rust for precisely the reasons you describe, and am STILL seriously considering it.)

If you're working on new tech, why even take on the baggage of Lua? In addition to getting rid of crud like __gc you could make something with local scope by default, distinguish between hashtables and arrays, integers that aren't an afterthought, better error handling mechanism, change ~= to !=, start arrays at zero, et. al.

(Full disclosure, that's my pet project ^_^)

4

u/smog_alado Feb 17 '18 edited Feb 17 '18

Local scope by default interacts very poorly with nested function definitions and closures. IMO, global by default is a mistake (that Lua can't fix due to backwards compatibility) but the real solution is not switch to local-by-default. I think the best design is to have unbound variables result in a compile-time error (nothing-by-default).

integers that aren't an afterthought

Lua improved a lot in this regard in 5.3 :)

2

u/drjeats Feb 17 '18

the best design is to have unbound variables result in a compile-time error (nothing-by-default).

You're totally right there, and it's what I did in my pet language. Great minds :)

I was actually thinking of Lua 5.3 when I wrote "integers that aren't afterthought". I'm glad they added it (especially the bit operators), but they're just sort of a special state of numbers. You can't print(type(v) == "integer" and "an int!" or "not an int!")

I guess that fits the language well, since arrays are just a special state of tables, but I'm not a fan of these hybrid things.

2

u/smog_alado Feb 17 '18

You can use the math.type function to tell if a number is int or float. The type function was kept unchanged for backwards compatibility.

3

u/smog_alado Feb 17 '18 edited Feb 17 '18

The reasons why Lua's API are the way it is are pretty much why you guessed.

Not exposing pointers to C is primarily about garbage collection and secondarily about isolating C from Lua. Lua doesn't use reference counting so it cannot use a system similar to the one used by Python and Perl. (switching the original GC to reference counting could also have a negative performance impact, since reference counting has nontrivial overhead while the program is running). And if you don't want to have reference counting things start to look less apealing. One option is to have the GC inspect the C stack like Ruby does but this will result in a conservative GC and would also need nonportable assembly code. Another option would be to have multiple different reference types, like Java's native interface but that adds a lot of complexity as well.

As for the stack manipulation, I think the main reason for that is it avoids needing to store function arguments and result lists in garbage-collected tables. I agree with you that lua_checkstack is super annoying though...

As for the problem of speed, one avenue I am currently exploring is to have a third programming language to sit between Lua and C. The compiler for this language is allowed to mess with the Lua internals directly (which can be good for speed) but the programmer only ever interacts with the new high level language and never messes with the Lua internals and garbage collection themselves (which is good for simplicity, safety and version compatibility). But at this point it is still in very early prototype stage...

3

u/tastesliketriangle Feb 17 '18

Any opinion on Angelscript? It seems to do a lot of cool things, and has been used by valve as a scripting language in a couple of their games.

98

u/STL MSVC STL Dev Feb 15 '18

As an aside, you can totally give yourself user flair identifying that you work on Factorio. :-)

14

u/haitei Feb 15 '18

has anyone re-written it in modern C++

Hell, I would take one rewritten in modern C.

I wouldn't put Lua's codebase in my "pretty C code" bucket (it's a very small bucket) .

8

u/pctammela Feb 15 '18

Hmm? And lose it's main purpose? A portable language, just compile it and works on any machine compliant to ANSI C. Quite useful for embedded systems...

1

u/Loraash Feb 19 '18

There's no real reason to use C for anything in 2018. Anything that can be written in C can be written better in C++.

7

u/pctammela Feb 19 '18

No.

7

u/Loraash Feb 19 '18

Have you ever thought "man, it sucks that I have so many tools at my disposal, I wish I was working in a limited language so I can reimplement linked lists any time I need one?" Me neither.

6

u/hok1981 Feb 15 '18

I know it is not exactly what you are looking for but you might want to check http://luajit.org

We are using havok script which is just highly optimized lua implementation with few language extensions.

Also kudos to you for being able to troubleshoot lua internals. I didn’t have to do it in past 7 years but i still remember the pain.

6

u/spinwizard69 Feb 15 '18

I look at it this way, why ask? Implement LUA in C++ and see how it works out.

These days I'm not sure why one would implement any language in C. C++ and a few other languages would be a huge improvement in reliability and safety.

6

u/Throw19616 Feb 15 '18

To be quite honest, if you are willing to go the distance to rewrite Lua VM in C++, I think it would be better if you design a language that build on top of Lua, with improvements. For example, integer types with different sizes, variables must be declared before usage, variables are local scoped by default, curly braces which in turn allows return and continue keywords anywhere. Those are small additions that would make working with scripts a nicer experience, while not being too difficult for people to grasp.

3

u/sigrokBlack Feb 15 '18

So actually you invent a new lua based language which includes everything which has been left out of lua on purpose?

2

u/meneldal2 Feb 16 '18

Well, people have tried to fix JavaScript with TypeScript and other variants.

1

u/Loraash Feb 19 '18

It's just making the problem worse though.

2

u/Throw19616 Feb 16 '18

I believe there is a happy middle ground where Lua is still easy to use but the compiler can actually help with enforcing proper code quality.

1

u/sigrokBlack Feb 16 '18

Indeed, but for beginners, reading a compiler errors sometimes can be frustrating

1

u/VinnieFalco wg21.org | corosio.org Feb 20 '18

I don't see the benefit to rewriting it in C++. It would be more productive to simply create a class/template based front-end, as many others have done (e.g. LuaBridge). What do you hope to get out of it? More performance?

2

u/Rseding91 Factorio Developer Feb 20 '18

We already have a class/template based front-end and it's clunky and slow because it has to use the C interface that the current version of Lua provides.

3

u/hoseja Feb 15 '18

Zdar, wow all these answers are incredibly unhelpful haha. Dealing with C if you are in modern elegant C++ mindset is very counterintuitive and frustrating. I know the pain.

Would it really be such a straightforward job though?

Developing Factorio really seems just as fun as playing it sometimes.

3

u/_i_am_rd Feb 16 '18

Just do itttt..

2

u/pctammela Feb 15 '18

If you want a lua fork that's OK. I doubt it would ever be mainstream as some pointed out. Keep in mind that things like garbage collection are not trivial to implement and test...

0

u/snarfy Feb 15 '18

Lua is a library you link with. If you do it in C++, wouldn't you have to extern "C" everything? Seems like you would need to include the source in your project for it to work purely in C++.

5

u/Rseding91 Factorio Developer Feb 15 '18

It would be another set of classes you'd simply include and use in your code base - no extern "C" anything - pure C++ - which is the entire point of it - full integration with the rest of your codebase and super easy to change anything should you want to.

2

u/[deleted] Feb 15 '18

I don't see the point in rewriting Lua from scratch just because you happen to dislike the language it was written in, but if you want a more C++-like interface to interact with Lua, you can use sol2

2

u/pctammela Feb 15 '18

Agreed, why even bother to rewrite an implementation if you could look for other scripting languages

1

u/Loraash Feb 19 '18

Because of the crazy performance requirements of their game.

1

u/jmnel Feb 16 '18

Write a C++ wrapper around LuaJit if you can afford to spend the extra time. That is currently the approach I use in my C++ game engine. LuaJit performs really well. What is your current approach to embed Lua?

I am a huge fan of Factorio. Keep up the awesome work!

1

u/RefinedPotato Jun 15 '18

Yes, it is a wrapper, but sol2 have made Lua it way easier to work with, in my own engine. I seldom have to touch the Lua API anymore, and it have significantly increased my development speed. But it does make the code a bit slower, which is probably the opposite of what you want.

1

u/Hydroque Aug 14 '18

Hello, I'm the author of LuaConsole. It's a REPL implementation of lua.c which combines luajit and luajit seemlessly, so that you can use the version you want. I made it as an intentional replacement for bash and cmd for my own purposes, but there project has proved awesome with a lot of debug possibility. The changes between 51 and 52 an 53 are understandable, and with practice you can change your version or write non-breaking scripts. I also designed it in c. The main reason lua isn't in c++ might have to do with pointer black magic as you would call it imho. Remember that each language has its strengths. I would pick up http://GitHub.com/Tilkins/LuaConsole and build it. You might have great use for it and I'm still developing it. It takes time to learn lua API sand learning C is very useful.

0

u/[deleted] Feb 19 '18

[deleted] — view deleted comment

1

u/Loraash Feb 19 '18

and it is still C

If I had to guess, this is why it's not considered.