Looking for some help - add fenv and other 5.1 features to ravi-distro

[Dibyendu Majumdar]

Hi,

I was wondering if anyone would be able to help me add some Lua 5.1
compatibility features to Lua 5.3 in my Ravi-Distro package.

I include Lua 5.3.5 in the distro with these enhancements:

* The LuaJIT bit library is included
* All available 5.1 and 5.2 compatibility options are enabled,
including LUA_COMPAT_FLOATSTRING .

These compatibility options enable Lua 5.3 to run LuaJIT's dynasm tool
for instance.

However I would like to improve backwards compatibility with Lua 5.1
even further; in particular would like to add support for fenv(). But
I have never used Lua 5.1, and despite good intentions, haven't been
able to work on this aspect. I would very much appreciate any help.

Thanks and regards
Dibyendu

[Soni L.]

it's rather impractical: there are huge semantic differences between how
they work.

e.g.

local _ENV = ...
function foo()
_ENV = ...
end
function bar()
print(foo)
end

This is impossible in Lua 5.1.

Post by Dibyendu Majumdar
Hi,
I was wondering if anyone would be able to help me add some Lua 5.1
compatibility features to Lua 5.3 in my Ravi-Distro package.
* The LuaJIT bit library is included
* All available 5.1 and 5.2 compatibility options are enabled,
including LUA_COMPAT_FLOATSTRING .
These compatibility options enable Lua 5.3 to run LuaJIT's dynasm tool
for instance.
However I would like to improve backwards compatibility with Lua 5.1
even further; in particular would like to add support for fenv(). But
I have never used Lua 5.1, and despite good intentions, haven't been
able to work on this aspect. I would very much appreciate any help.
Thanks and regards
Dibyendu

[Sean Conner]

Post by Dibyendu Majumdar
Hi,
I was wondering if anyone would be able to help me add some Lua 5.1
compatibility features to Lua 5.3 in my Ravi-Distro package.
* The LuaJIT bit library is included
* All available 5.1 and 5.2 compatibility options are enabled,
including LUA_COMPAT_FLOATSTRING .
These compatibility options enable Lua 5.3 to run LuaJIT's dynasm tool
for instance.
However I would like to improve backwards compatibility with Lua 5.1
even further; in particular would like to add support for fenv(). But
I have never used Lua 5.1, and despite good intentions, haven't been
able to work on this aspect. I would very much appreciate any help.

It may be possible. setfenv() is used to set an environment table to
either a function or userdata. If it's a function, this table become the
global environment for the function; for a userdata, it's similar to
setuservalue(). Given that Lua 5.3, section 3.3.2 states:

As such, chunks can define local variables, receive arguments, and
return values. Moreover, such anonymous function is compiled as in
the scope of an external local variable called _ENV (see §2.2). The
resulting function always has _ENV as its only upvalue, even if it
does not use that variable.

So any code loaded via load() (or similar) will hvae _ENV as the first
upvalue, you can use setupvalue() to switch out the _ENV value currently
set. Arbitrary functions might be a bit harder to locate the proper _ENV to
swap (if not outright impossible in some cases).

setfenv() at least, the Lua function, not the C function, can also modify
a function in the call stack. Per the description:

Sets the environment to be used by the given function. f can be a
Lua function or a number that specifies the function at that stack
level: Level 1 is the function calling setfenv. setfenv returns the
given function.

As a special case, when f is 0 setfenv changes the environment of
the running thread. In this case, setfenv returns no values.

-spc (As always, the devil is in the details ... )

[Muh Muhten]

Post by Sean Conner
(if not outright impossible in some cases).

In some cases, it is outright impossible, in the sense that deciding
which upvalue was _ENV is not a function of the code.

Consider the examples I give for
<https://gist.github.com/f469d25d375e4e742b74c7a938c58ac8#limitations>:
the only way to tell the difference is to actually check the upvalues,
and see that one of them is the same as the global table. Of course,
this is not guaranteed either if _ENV is redefined.

What I believe you *could* do is, on finding an upvalue which is the
global table, debug.upvaluejoin it to the first upvalue of a chunk (or
more directly, a throwaway C closure) constructed with the new
environment as its only upvalue. I have not tried it. It probably
breaks expected semantics in all sort of subtle ways.

It's probably for the best to accept that fenv semantics are not
implementable in terms of _ENV. This is kind of expectable, since part
of the rationale for the change was (paraphrasing) that the setfenv
mechanism was too powerful and more suited to the debug library.

Incidentally, the _ENV mechanism is essentially syntactic sugar and
afaict can be mechanically translated to just introducing a local
named _ENV at the beginning of a chunk and rewriting all would-be
global accesses to index _ENV. There's one tiny difference (it's now a
local, not an upvalue, of the main function) but that seems like a
corner case that will only come up if you use the debug library or
otherwise muck with internals. As such, it might be easier to retarget
the 5.3 compiler to the 5.1 VM(!) if you *need* this functionality.
Certainly 5.2.

[Tim Hill]

Post by Sean Conner
So any code loaded via load() (or similar) will hvae _ENV as the first
upvalue, you can use setupvalue() to switch out the _ENV value currently
set. Arbitrary functions might be a bit harder to locate the proper _ENV to
swap (if not outright impossible in some cases).

Sadly that’s not true. Any chunk that is COMPILED will honor this, but code that is obtained via string.dump() followed by load() often will not. Such a dumped function may or may not have _ENV as one of its upvalues, and even if it does, it may not be the first.

—Tim

[Sean Conner]

Post by Sean Conner
So any code loaded via load() (or similar) will hvae _ENV as the first
upvalue, you can use setupvalue() to switch out the _ENV value currently
set. Arbitrary functions might be a bit harder to locate the proper _ENV to
swap (if not outright impossible in some cases).

Sadly that’s not true. Any chunk that is COMPILED will honor this, but
code that is obtained via string.dump() followed by load() often will not.
Such a dumped function may or may not have _ENV as one of its upvalues,
and even if it does, it may not be the first.

I think there's a bug in the description for string.dump(). It states,
"[w]hen (re)loaded, those upvalues receive fresh instances containing nil."
But load() states "When you load a main chunk, the resulting function will
always have exactly one upvalue, the _ENV variable (see §2.2). However, when
you load a binary chunk created from a function (see string.dump), the
resulting function can have an arbitrary number of upvalues."

Running this:

local function upvals(name,f)
local function upv(i)
local n,v = debug.getupvalue(f,i)
if v then
print("",i,n,v)
return upv(i+1)
end
end

print(name)
upv(1)
end

local a = 0
local function foo(x)
a = a + 1
print(x)
print(os.getenv(x))
return a
end

upvals("foo",foo)

local dumped = string.dump(foo)
local reloaded = load(dumped)

upvals("reloaded",reloaded)

I get:

foo
1 a 0
2 _ENV table: 0x89b5568
reloaded
1 a table: 0x89b5568

So the first upvalue will have _ENV (in this case, the global _ENV), but
not the rest (also, the first upvalue for reloaded() should be a number, not
a table).

-spc (Which presents yet another problem with simulating setfenv() in Lua
5.2 or higher ... hmmm ... perhaps not a good idea after all)

[Muh Muhten]

Post by Sean Conner
I think there's a bug in the description for string.dump(). It states,
"[w]hen (re)loaded, those upvalues receive fresh instances containing nil."
But load() states "When you load a main chunk, the resulting function will
always have exactly one upvalue, the _ENV variable (see §2.2). However, when
you load a binary chunk created from a function (see string.dump), the
resulting function can have an arbitrary number of upvalues."

Sure. When (re)loaded, the dumped function has fresh upvalues that
aren't the same as those or any other function, which are initially
nil. Then load() (and lua_load()) set the first upvalue.

... hm, perhaps load setting the first upvalue of functions that
aren't main chunks (linenumber != 0) just doesn't make much sense in
the first place? Or if the incompatibility there seems gratuitous
(maybe it can go into 5.4?), indeed, mention the first upvalue in the
description of string.dump.

Initially I expected lua_load to not set the first upvalue, but that
doesn't seem to be the case.