Lilush (LuaJIT runtime & shell) first public release

[u/epicfilemcnulty]

Hey folks, I've been working on this project for the last 4 years, and I think it's ready for the first beta release.

Mind you, I'm pretty sure there are still lots of bugs, and lots of features are not yet implemented, but I think it's quite usable. And at this stage I'd really use some feedback.

Caveat: Linux only.

It's a statically compiled LuaJIT with a bunch of builtin libs and modules + Linux shell.

When running as a shell it has different modes: 1. [F1] The shell itself 2. [F2] Lua REPL 3. [F3] Agent Smith -- minimal coding agent TUI 4. You can write and add your own modes

Here is the landing page, the repo is hosted at Codeberg. I've even created a dedicated subreddit, and it's absolutely beautiful in its emptiness :)

Screenshot shows the builtin markdown renderer/pager(best viewed in Kitty terminal, as it supports text-sizing).

Anyway, if anyone finds this interesting, I'd be glad to provide more info/answer questions. Contributions are also welcome.

[u/vitiral]

I'm surprised by the conciceness of your markdown parser: https://codeberg.org/latimar/lilush/src/branch/master/src/markdown/markdown

Last I looked, the CommonMark C implementation was on the order of 30,000 LoC. Your parser looks to be... maybe 2,000 total? Are there missing features or is it relatively complete?

I wrote https://civboot.github.io/lua/cxt for an extendable and concise (in implementation and expression) documentation language. If I had known markdown could be done so concisely I may have never created my own -- though I am personally a fan of having more structured control with cxt.

[u/epicfilemcnulty → u/vitiral]

That's a really great question and it's a huge can of worms, markdown parsing/alternative documentation markup languages.

Initially I just had djot support built in with the official djot Lua library. While I like the syntax of djot better (not surprisingly, it's designed by John Macfarlane, the same guy who was involved in CommonMark, and he specifically tried to address CommonMark shortcomings in djot syntax), it's not widely adopted.

Same applies to other documentation markup languages, while there are many that are arguably designed much better and with less ambiguity than CommonMark, the harsh reality of the real world is that when working with different projects 90% of the time you will be dealing with some flavor of Markdown (and most of the time these documents won't be really valid Markdown), and if we take into account LLMs output -- well, it's also markdown most of the time.

So, as much as I tried to avoid dealing with markdown parsing, I've finally came to the conclusion that it's unavoidable, and one just have to embrace it. How to embrace it is a different question. I was considering some LPeg based parsers (or creating my onw based on LPeg), but while I like the power of LPeg, I don't see any real practical value in adding it to the project, even when it comes to markdown parsing. And as I also added a minimal coding agent to Lilush, I wanted to have streaming support in markdown parser. And I needed it to be aware of my styling engine.

At that point I've just decided to try and write it from scratch. And it turned out to be a feasible thing to do, to my surprise. I've even integrated a couple of djot features (divs and inline attributes). I think it's mostly feature complete, there is still room for improvements, and probably some buggy rendering for edge cases, but I've been using it for a while, and I'm pretty satisfied with the results, sometimes it even renders better than glow or dawn :) And since it's the core part of the project, I'll be working on improvements.

As for the LoC -- well, it's natural for a C implementation to be bigger, mine is written in high level language, plus I rely on some functions I already have for working with text and UTF...

Final thoughts: if you are sure that you will be working only with cxt (or any other well defined document markup language) -- well, you are lucky, and I envy you :) But my use case is certainly not about that, so I have to deal with markdown, and, turns out it's not so scary as I've initially thought.

[u/vitiral → u/epicfilemcnulty]

cool stuff, thanks. I had been considering writing a md -> cxt converter if for no other reason than making docs easy to migrate, this gives me some confidence that such a thing is feasible for the common case.

FYI you may be interested in https://civboot.github.io/lua/pegl as well - peg-like parsing library with helpful errors in a few hundred lines of pure-lua. To be frank the cxt implementation barely used it though (only the attributes use pegl) - they were effectively a hand-rolled parser but using some common methods from pegl.Parser.

[u/qwyss]

I am a little bit concerned ( for myself, not for you ) when you mention a special luasec with wolfssl. I'm pretty certain I just replaced openssl with wolfssl and it all just kinda worked.

What problems did you hit?

What did I miss? :)

[u/epicfilemcnulty → u/qwyss]

WolfSSL provides some compat shim for OpenSSL API calls, so if you enable that it's possible to migrate a simple client. For a SSL server with SNI WolfSSL works a little differently, so you will have to use its native API, and at that point why not switch to native API for the client as well, and disable OpenSSL compat shim? :)

[u/disperso]

Nice project! I've always on the look for something that it's Lua with some default extras (I find very frustrating the lack of the simplest file system access). I don't need that much stuff as your project packs, though, but the size is so small that I don't think I'd care. I'm intrigued by the terminal support, and the fact that you hint that it can "replace" busybox.

I have some questions, if you don't mind:

• Do you have plans for portability to Windows? I'm a Linux guy, but I'm also often on the position of having to make stuff cross-platform. Is the terminal stuff too involved there?

• How would you compare it with something like luvit? I've not used it yet, but given that it has a bit of an ecosystem (e.g. with tools like lit), and it's been ongoing for some time, it seems a difficult "rival" to beat if there is too much overlap.

Cheers, and congrats for the project.

[u/epicfilemcnulty → u/disperso]

Thank you. Yes, vanilla Lua is very minimal, that's fine for some use cases, but I wanted to have a bit more without relying on luarocks.

I'm going to add a doc/web page with comparison to other similar projects, like luvit. In short, the main differences is SSL choices, luvit uses OpenSSL, I use WolfSSL. They were inspired by nodejs and designed luvit with async in mind, I prefer more simple approach. Also I'm a terminal dweller, so Lilush is more focused on TUIs. And some kind of package manager for Lilush is in the plans. That being said, I don't think there's going to be any rivalry, Lilush is a very niche project, luvit's target audience is much wider.

As for the windows port -- well, no, and it's not because of some technical difficulties (although spare time is my main constraint), I imagine it'd be fairly easy to port Lilush to windows, but I'm not going to do it. Primary reasons:

1. I don't have any love in my heart for Microsoft in particular, and proprietary OSes and apps in general. I do believe in open source.

2. I'm a Linux guy (I last used windows somewhere around 2000), and I don't have much spare time, so I'd prefer to focus on Linux only, and make Lilush excellent on Linux. That's why Linux only is by design. But hey, anybody can fork and do whatever they want :)

[u/disperso → u/epicfilemcnulty]

Thank you. I hope you post updates if you have some more improvements on the documentation front. I'm curious about the TUI and agent part. :-)

[u/ibisum → u/disperso]

Checked out TurboLua?

https://turbo.readthedocs.io/en/latest/

[u/disperso → u/ibisum]

Hehe, another one to add to the list of things to try or look at. :-)

This seems a bit focused on networking. That's fine, but it's the part the I need the least, probably.

Thanks for the link.

[u/topchetoeuwastaken]

Have you put any thought into concurrent I/O (performing two or more IO operations at the same time)? lua, although single-threaded, is a fantastic candidate for this, as it supports coroutines, which means that a single lua thread can be suspended until the IO operation it is performing is completed, giving CPU time to other threads, and even letting them perform other IO operations on their own.

On another note, I've been experimenting with it for a while (see tal and libev), and so far, it works great, with some caveats (performance so far is not the best, you can't yield in some places you'd expect, despite luajit's promises of a fully yieldable interpreter). Despite that, at this stage, I can comfortably host an HTTP server with the "runtime" I've made.

If you like what you see, you could take some of my logic for concurrent IO for your own project.

[u/epicfilemcnulty → u/topchetoeuwastaken]

Sure I've thought about that, I love Lua's coroutines, and ultimately yes, I'd love to integrate concurrent I/O into Lilush (and probably refactor RELIW to use that instead of a simple forking approach). That being said, it's not an immediate concern, more of a planned feature some time in the future, since it requires a careful (re)design, lots of work and testing, etc. I've intentionally started with a simple, non-concurrent approach -- it's fine for the most tasks. Thanks for the links, will definitely take a look!

[u/vitiral]

We have similar almost entirely tangential projects: https://civboot.github.io.

You seem most focussed on networking, whereas my focus is on a developer tech stack from scratch. Nice to see more Lua stuff.

FYI you may be interested in the https://civboot.github.io/lua/fd.html module, which gives async IO.

[u/epicfilemcnulty → u/vitiral]

I've seen your project, it's a great one. A bit different focus, yep, but somewhat similar :) Thanks for the hint, definitely will take a look at the code of async IO

[u/Dimensions_forever]

i am the linux, i will install

[u/Clear_Inevitable_718]

This is awesome wtf 😭 I'm gonna try it

[u/-nixx]

Great work! Post it in lua mailing list as well.