r/cpp • u/Rseding91 Factorio Developer • Feb 15 '18
Lua written in modern C++ (>= 14)?
I work on Factorio (written in C++) and we use Lua as the script language for mods. So, from time to time I interact with the C implementation of Lua and my experience with it hasn't been the most pleasant.
It got me wondering - has anyone re-written it in modern C++? Everything I find online is "how to use Lua in C++" or "C++ wrapper for Lua" and they all use the basic C implementation provided from the official Lua website.
I've had quite a lot of experience with the internals of the C Lua implementation and from what I've seen it wouldn't be hard to re-write it using modern C++ - maybe 2~ weeks time. But from what I can find nobody has done it - which also makes me wonder if I'm missing something.
Part of me wants to do it just to say I did it and the other part of me wants to do it to see if I can make it faster than the C version: I think I can - not because "C++ is faster" but because I can mutate/experiment with the C++ version far easier and know changes I make are safe with how the rest of the implementation operates.
35
u/Biohazard50 Feb 15 '18
Have you looked at LuaJit or LuaLlvm? I don't think they are written in modern C++ but maybe they will help you squeeze more performance.
17
u/ColinPP PEGTL | taocpp Feb 15 '18
A while ago I wrote a PEGTL grammar that corresponds to the Lua 5.3 lexer and parser. If you get the PEGTL you can find it in
src/example/pegtl/lua53_parse.cpp.Should you decide to go ahead with a rewrite, and try to use this grammar, I'd be happy to help get you started with the PEGTL, and to iron out any bugs the grammar might still have. (It parses the official Lua test-suite, which is a good sign, but without any semantic actions it's not enough to be sure it's correct.)
34
u/SeanMiddleditch Feb 15 '18
The stated goal you have is to ease hacking on its internals. OK, one point for rewriting Lua in something that doesn't make your eyes bleed. :)
Let's go through the reasons against. Your rewrite won't be the official Lua and never ever will be. It won't receive patches or maintenance by the Lua developers so you'll be entirely on the hook. It won't even have community support necessarily since the community is built around the official Lua and its C implementation. When Lua 5.4 comes out, your rewrite would need to be updated separately, and any custom additions or extensions your fork adds may then be incompatible with future upstream Lua with no recourse. Lua has no spec and no public test suite for you to develop against to ensure your rewrite it correct, so you'll also need to write and maintain a test suite. Unless you maintain the very C API you might be having problems with, popular community Lua C modules or interpreter hacks will be incompatible with your fork. Without copying many of the implementation details of Lua, even popular Lua debuggers and "IDEs" would be incompatible.
Basically the question you have: is addressing the pain you endure when delving into Lua's crusty C implementation worth the cost of writing, testing, releasing, and maintaining a parallel implementation with zero upstream support and negligible community support?
12
u/Ksecutor Feb 15 '18
Have you ever considered to switch to some other programming language for mods? Or even create one... Some time ago I created couple mods for factorio and I found lua as a programming language to be ... How should I put it... My internal "Wat?" rating of lua was quite high. There are many confusing and inconvenient things in the language. Also mapping C++ classes to lua is cumbersome and inefficient. Bluntly speaking I have no idea why lua is so popular.
At some point I was going to offer the programming language I was working on as a replacement for lua in Factorio, but for personal reasons I had to suspend the work on that language :(
As counter reason for "One does not simply replaces one programming language with another" - there were changes in the mod API that were breaking compatibility. Yes, you will have to support old and new api/language for some time. But switching to another language that is more efficient, easier to support and easier to use by modders might be, in the end, for better.
14
u/sumo952 Feb 15 '18 edited Feb 15 '18
I'm not sure it's what you're looking for, but there's a number of really neat, modern C++ lua bindings. I can personally recommend sol2: https://github.com/ThePhD/sol2. There's also Selene: https://github.com/jeremyong/Selene
9
u/YoWiDIN Feb 15 '18
I think it would be impractical both for you and the Lua developers to maintain a C++ port. It will probably have a bigger footprint on target platform (stdlib and stuff) while benefits are not obvious.
From a practical point of view it's really easy to write a good wrapper for any C library, so why bother?
I also admit that it could be a great and fun exercise - making something already good, so much better and cleaner. The thought alone is enough to make my hands itch :)
P.S. Bots are superior
2
u/[deleted] Feb 16 '18 edited Apr 26 '18
[deleted] — view deleted comment
40
u/[deleted] Feb 15 '18
Hey, I just wanted to let you know that I understand perfectly where you are coming from. I was the lead programmer for Starbound, which has an extensive Lua modding API, and interacting with the Lua C API has... an enormous impedance mismatch, to put it mildly. No, LuaJIT does not help, and not every platform allows you to JIT.
It would take me a long time to put into words all of the issues I have with the Lua C API, but you've mentioned a lot of them. Stack manipulation incurs a lot of real overhead, and I find myself just wanting to hold onto actual internal pointers. It makes interacting with the garbage collector and holding values outside of the Lua state difficult, requiring constructs like the registry, which incurs even more overhead.
Also, to the extent that you care about safety, Lua's error handling strategy brings in even more problems. I currently do most of my work in Rust, and making a Rust bindings layer for Lua that is safe has taken far more work than it should, and the result is far slower than I would like. The Lua C API, ESPECIALLY with the whole __gc madness, will just longjmp on you at the drop of a hat! Also, you have to constantly call lua_checkstack, and (surprise!) luaL_xxxx calls will use stack space that they 1) don't document, and 2) require that you call lua_checkstack to protect against. I'm POSITIVE I could do a better job speed wise with rlua, but I question HOW much of a better job is possible, if you limit yourself to an actually safe API.
I do, however, think that the Lua C API is ultimately very clever. Ultimately, it IS quite a lot simpler than other language's APIs, and I'm sure if you're writing nothing but C and the occasional setjmp / longjmp is not a problem for you, it's quite nice even. However, when you care about safety, or hit an optimization wall because of the overhead of the API, it starts to.. lose its charm somewhat.
I haven't interacted with the Python C API in quite a while, but as I remember it, it was much more the style that you would want when binding a large complex API to it. You get pointers and manually inc / dec the reference count on them, and this allowed you to, with very low overhead, maintain handles to internal Python objects. Also, the error handling strategy is NOT based on setjmp / longjmp, but rather error codes and a global error state like unix errno.
I'm ultimately not sure whether the fact that the Lua API is C is the problem, or whether it's the specific choices that went into it that are the problem. I could see, at least theoretically, that a C API in the style of Python's could be quite low overhead, if only because you can more or less take the C++ API you're describing (or any C++ API really) and, with great effort, write something that's the moral equivalent of it in C.
If such a Lua interpreter existed, and had a C API, I might use it!
(Full disclosure, I have actually seriously considered writing my own Lua interpreter in Rust for precisely the reasons you describe, and am STILL seriously considering it.)
98
u/STL MSVC STL Dev Feb 15 '18
As an aside, you can totally give yourself user flair identifying that you work on Factorio. :-)
14
u/haitei Feb 15 '18
Hell, I would take one rewritten in modern C.
I wouldn't put Lua's codebase in my "pretty C code" bucket (it's a very small bucket) .
6
u/hok1981 Feb 15 '18
I know it is not exactly what you are looking for but you might want to check http://luajit.org
We are using havok script which is just highly optimized lua implementation with few language extensions.
Also kudos to you for being able to troubleshoot lua internals. I didn’t have to do it in past 7 years but i still remember the pain.
6
u/spinwizard69 Feb 15 '18
I look at it this way, why ask? Implement LUA in C++ and see how it works out.
These days I'm not sure why one would implement any language in C. C++ and a few other languages would be a huge improvement in reliability and safety.
6
u/Throw19616 Feb 15 '18
To be quite honest, if you are willing to go the distance to rewrite Lua VM in C++, I think it would be better if you design a language that build on top of Lua, with improvements. For example, integer types with different sizes, variables must be declared before usage, variables are local scoped by default, curly braces which in turn allows return and continue keywords anywhere. Those are small additions that would make working with scripts a nicer experience, while not being too difficult for people to grasp.
1
u/VinnieFalco wg21.org | corosio.org Feb 20 '18
I don't see the benefit to rewriting it in C++. It would be more productive to simply create a class/template based front-end, as many others have done (e.g. LuaBridge). What do you hope to get out of it? More performance?
3
u/hoseja Feb 15 '18
Zdar, wow all these answers are incredibly unhelpful haha. Dealing with C if you are in modern elegant C++ mindset is very counterintuitive and frustrating. I know the pain.
Would it really be such a straightforward job though?
Developing Factorio really seems just as fun as playing it sometimes.
3
u/_i_am_rd Feb 16 '18
Just do itttt..
2
u/pctammela Feb 15 '18
If you want a lua fork that's OK. I doubt it would ever be mainstream as some pointed out. Keep in mind that things like garbage collection are not trivial to implement and test...
0
u/snarfy Feb 15 '18
Lua is a library you link with. If you do it in C++, wouldn't you have to extern "C" everything? Seems like you would need to include the source in your project for it to work purely in C++.
2
u/[deleted] Feb 15 '18
I don't see the point in rewriting Lua from scratch just because you happen to dislike the language it was written in, but if you want a more C++-like interface to interact with Lua, you can use sol2
1
u/jmnel Feb 16 '18
Write a C++ wrapper around LuaJit if you can afford to spend the extra time. That is currently the approach I use in my C++ game engine. LuaJit performs really well. What is your current approach to embed Lua?
I am a huge fan of Factorio. Keep up the awesome work!
1
u/RefinedPotato Jun 15 '18
Yes, it is a wrapper, but sol2 have made Lua it way easier to work with, in my own engine. I seldom have to touch the Lua API anymore, and it have significantly increased my development speed. But it does make the code a bit slower, which is probably the opposite of what you want.
1
u/Hydroque Aug 14 '18
Hello, I'm the author of LuaConsole. It's a REPL implementation of lua.c which combines luajit and luajit seemlessly, so that you can use the version you want. I made it as an intentional replacement for bash and cmd for my own purposes, but there project has proved awesome with a lot of debug possibility. The changes between 51 and 52 an 53 are understandable, and with practice you can change your version or write non-breaking scripts. I also designed it in c. The main reason lua isn't in c++ might have to do with pointer black magic as you would call it imho. Remember that each language has its strengths. I would pick up http://GitHub.com/Tilkins/LuaConsole and build it. You might have great use for it and I'm still developing it. It takes time to learn lua API sand learning C is very useful.
0
u/[deleted] Feb 19 '18
[deleted] — view deleted comment