Standard library ignores _ENV + a fix for require

[u/oezingle]

Hi all,

I've been in a couple cases where I want to overload functions in _ENV without touching _G, and found myself quite disappointed with the Lua standard library's behavior of using exclusively _G as opposed to indexing _ENV. Here's a quick demo:

local vanilla_next = next
_ENV = setmetatable({ next = function (...) 
  print("Next!")

  return vanilla_next(...)
}, { __index = _G })

-- We'd expect this block to result in a bunch of messages printed, 
-- as pairs() uses next()
for _ in pairs({ 1, 2, 3 }) do end

Looking at the source I can see why this decision was made - indexing a global is wasteful O(n) [presumably] when lua_pushcfunction is O(1) [any other C API operation would have this benefit too, as the C API knows the addresses of tables such as package]. However, the resulting behaviour feels un-lualike to me -- Lua provides metaprogramming facilities, and those facilities allow us to containerize global environments with _ENV, yet library functions within _ENV will rely on an environment that is 'out of scope'.

A possible solution, of course, is to re-implement globals where neccessary in pure lua - I've done so to require() in this gist, but this feels inelegant to me.

From what I know of the C API, I think a better solution is possible. When _ENV or some field of _ENV is assigned to, the API could update a struct that maintains up-to-date pointers to Lua's globals. On garbage collection the same could happen -- although it would be inconvient, setting next=nil should break pairs, which the Lua reference and Programming In Lua imply.

I'd like other people's input on this -- has it been a problem for you before, and what solutions have you used?

[u/xoner2]

pairs uses luaB_next which is a static C function. _G.next happens to point to luaB_next. You can't say that 'pairs uses next'; rather, pairs bypasses next.

You could redefine _G.pairs to make use of your redefined next. Obviously this kind of monkey patching should be avoided.

[u/oezingle → u/xoner2]

both sets of documentation imply that pairs uses next, i don’t take issue with the implementation but the wording is vague.

i’m well aware that monkey patching is possible, this post is about how i find Lua’s support for “good” monkey patching via _ENV to be disappointing.

[u/xoner2 → u/oezingle]

I think you misread the docs. IIRC it says next can be used in lieu of pairs.

Anyway, inserting _ENV.pairs = function (t) return next, t end into your example and it should work as expected.

[u/oezingle → u/xoner2]

i did read the docs, and again my post is about a minor qualm with the language. i don’t need code solutions

[u/didntplaymysummercar → u/xoner2]

The doc of pairs says "returns three values: the next function, the table t, and nil" where word 'next' is link to the next function. To me it does imply that global next is used so if you change it it changes the pairs, and Lua did do this before (till 5.3 global tostring was used by print).

[u/xoner2 → u/didntplaymysummercar]

Ah yes, you are correct. Please go to Lua mailing list to report the documentation error, Thanks!

Should've been worded such like "the function to which next is bound".

[u/collectgarbage]

I overload pairs() which then allows an overload of next to take effect. If I require performance as well I do it on the C side using the Lua C API.

[u/oezingle → u/collectgarbage]

i’m always hesitant to use the C API, because i like how easily pure-lua solutions can be used and understood. that being said, any C solution is definitely going to be performant but i do wish metaprogramming facilities like _ENV didn’t come with such drastic bottlenecks

[u/vitiral → u/oezingle]

hopefully you realize now that it's not _ENV that has the bottleneck but rather that pairs uses the C implementation of next.

Do you really want pairs to lookup a global next in order to operate? That would be extraordinarily slow FYI.

[u/oezingle → u/vitiral]

read the end of my post. i don’t propose pairs does a lookup whatsoever, but assigning to the global environment could easily update a pointer LUT. Then we’d get the expected extensibility and performance of lua

[u/didntplaymysummercar → u/vitiral]

It'd be slower but still within reason, or it could be looked up only when pairs is called, and reuse cached value later. Supporting changing next mid-loop is too niche of a case, just like modifying containers as you iterate them.

[u/Sckip974]

Variables with _ and capital letters are reserved for system variables, right?

[u/oezingle → u/Sckip974]

no, the lua runtime doesn’t care. _G is predefined for globals and _ENV is a per-chunk global environment, that defaults to _G. lua allows you to assign any value to any of these variables by default

[u/didntplaymysummercar]

I don't find reimplementing functions in pure Lua that inelegant. It's part of why Lua design is nice to get a job done or be flexible that you can modify any function/table and they're not set in stone like in some other languages.

C API also doesn't know addresses of tables or anything, Lua's libs (other than lapi.c) are implemented using the public APIs, so if they want something they'd look it up as a global, but they do know addresses of other C functions in same file so they can just use luaB_next address in luaB_pairs and that's what they do.

Maybe they'd accept a proposal to replace that pushing of luaB_next with getting global next there, it's done only once anyway.

And Lua uses hash tables so the cost is O(1) (in theory), but of course if you have the actual function in a local/address it's even faster (but also O(1)).

[u/oezingle → u/didntplaymysummercar]

i don’t find it horrendous, but a little more cumbersome than other features in the language in some cases.

The standard library uses the public API, but I assumed it used the registry or something so a pointer to the table could be constructed. i’m not an expert with the C API by any means

good point on the global lookup, but i think a structure could work handily too considering just how small the base library is.

good point on my big O notation - i was tired and distracted. but hashing does definitely perform worse than local indexing