Discussion:
[ANN] Lua 5.4.0 (work2) now available
Luiz Henrique de Figueiredo
2018-06-18 20:33:32 UTC
Permalink
Lua 5.4.0 (work2) is now available for testing at
http://www.lua.org/work/lua-5.4.0-work2.tar.gz

The checksums are
MD5 3cdf2a4eb84dde6b6aaf5d2d1de07be9 -
SHA1 e8484e61c5c338e3ec2f75dbe0f6703d079fecf9 -

This is a work version. All details may change in the final version.

An updated reference manual is included and also available at
http://www.lua.org/work/doc

The complete diffs from work1 to work2 are available at
http://www.lua.org/work/diffs-lua-5.4.0-work1-work2.html
http://www.lua.org/work/diffu-lua-5.4.0-work1-work2.html

Previous work versions are available at
http://www.lua.org/work/

The focus of Lua 5.4.0 is performance. We're especially interested in
feedback about its performance but all feedback is welcome. Thanks.
--lhf
Dibyendu Majumdar
2018-06-18 21:16:28 UTC
Permalink
On 18 June 2018 at 21:33, Luiz Henrique de Figueiredo
Post by Luiz Henrique de Figueiredo
Lua 5.4.0 (work2) is now available for testing at
http://www.lua.org/work/lua-5.4.0-work2.tar.gz
Congratulations! Is there a test pack yet?

Thanks and Regards
Dibyendu
Italo Maia
2018-06-18 21:29:48 UTC
Permalink
That looks really nice!
Post by Dibyendu Majumdar
On 18 June 2018 at 21:33, Luiz Henrique de Figueiredo
Post by Luiz Henrique de Figueiredo
Lua 5.4.0 (work2) is now available for testing at
http://www.lua.org/work/lua-5.4.0-work2.tar.gz
Congratulations! Is there a test pack yet?
Thanks and Regards
Dibyendu
--
"A arrogância é a arma dos fracos."

===========================
Me. Italo Moreira Campelo Maia
Co-fundador do Grupo de Usuários Python do Ceará
Secretário ForHacker (fb.com/ForHackerSpace)
Desenvolvedor Full-Stack, Escritor, Empresário, Visionário
-----------------------------------------------------
Meu Livro <http://bit.ly/flask-amazon>, Site <http://www.italomaia.com/>,
Blog <http://eusouolobomau.blogspot.com/>
===========================
Roberto Ierusalimschy
2018-06-19 12:11:20 UTC
Permalink
Post by Dibyendu Majumdar
On 18 June 2018 at 21:33, Luiz Henrique de Figueiredo
Post by Luiz Henrique de Figueiredo
Lua 5.4.0 (work2) is now available for testing at
http://www.lua.org/work/lua-5.4.0-work2.tar.gz
Congratulations! Is there a test pack yet?
It should be available soon (today or tomorrow).

-- Roberto
Luiz Henrique de Figueiredo
2018-06-19 12:52:03 UTC
Permalink
A test suite is available at
http://www.lua.org/work/lua-5.4.0-w2-tests.tar.gz
Dibyendu Majumdar
2018-06-19 20:02:31 UTC
Permalink
On 19 June 2018 at 13:52, Luiz Henrique de Figueiredo
Post by Luiz Henrique de Figueiredo
A test suite is available at
http://www.lua.org/work/lua-5.4.0-w2-tests.tar.gz
Thank you. I see various occurrences of undef in the tests - is that
intentional?

Regards
Dibyendu
Roberto Ierusalimschy
2018-06-19 21:06:48 UTC
Permalink
Post by Dibyendu Majumdar
Thank you. I see various occurrences of undef in the tests - is that
intentional?
Yes. As I said, I've found useful to use undef (which is equal to nil)
as a way to document that I am removing an element from a table.

-- Roberto
Dibyendu Majumdar
2018-06-19 21:54:00 UTC
Permalink
Post by Roberto Ierusalimschy
Post by Dibyendu Majumdar
Thank you. I see various occurrences of undef in the tests - is that
intentional?
Yes. As I said, I've found useful to use undef (which is equal to nil)
as a way to document that I am removing an element from a table.
Okay.
t***@justdreams.de
2018-06-18 22:59:11 UTC
Permalink
Post by Luiz Henrique de Figueiredo
The focus of Lua 5.4.0 is performance. We're especially interested in
feedback about its performance but all feedback is welcome. Thanks.
--lhf
I have silly little piece of code I use to compare language performance.
Includes I/O and arrays(table in Lua obviously). When called with 20
iterations Lua 5.4 runs 5.8s vs 7.8s in Lua 5.3. That's quite a
significant performance jump! That is looking awesome!
t***@justdreams.de
2018-06-18 23:02:34 UTC
Permalink
Post by t***@justdreams.de
Post by Luiz Henrique de Figueiredo
The focus of Lua 5.4.0 is performance. We're especially interested in
feedback about its performance but all feedback is welcome. Thanks.
--lhf
I have silly little piece of code I use to compare language performance.
Includes I/O and arrays(table in Lua obviously). When called with 20
iterations Lua 5.4 runs 5.8s vs 7.8s in Lua 5.3. That's quite a
significant performance jump! That is looking awesome!
Here is the code, in case someone is wondering:

#!/usr/bin/env lua

-- based on code by Erik Wrenholt

local BAILOUT = 16
local MAX_ITERATIONS = 1000
local RESOLUTION = 40
local RESFLOAT = 40 / 1.0
local RESLOW = 1-RESOLUTION
local RESHGH = RESOLUTION-2
local write, clock, format = io.write, os.clock, string.format


local iterate = function( x, y )
local cr = y-0.5
local zi = 0.0
local zr = 0.0

for i = 0, MAX_ITERATIONS do
local tmp = zr * zi
local zr2 = zr * zr
local zi2 = zi * zi
zr = zr2 - zi2 + cr
zi = tmp + tmp + x
--print(zr,zi)
if (zi2+zr2 > BAILOUT) then
return ' '
end
end
return '*'
end

local mandelbrot = function( n )
while n>0 do
for y = RESLOW, RESHGH do
local output = { }
for x = RESLOW, RESHGH do
output[ x+RESOLUTION ] = iterate( x/RESFLOAT, y/RESFLOAT )
end
print( table.concat( output, '' ) );
end
n = n-1
end
end

local n = 1
if arg[1] then
n = tonumber( arg[ 1 ] )
end

local t = clock()
mandelbrot( n )
print( format( "Lua Time Elapsed %.2f", clock() - t ) )
Luiz Henrique de Figueiredo
2018-06-19 01:09:18 UTC
Permalink
Post by t***@justdreams.de
I have silly little piece of code I use to compare language performance.
Includes I/O and arrays(table in Lua obviously). When called with 20
iterations Lua 5.4 runs 5.8s vs 7.8s in Lua 5.3. That's quite a
significant performance jump! That is looking awesome!
Thanks for the feedback.

If you're curious, check out the new VM instructions with luac -l.
BTW, we also welcome feedback on luac. We may have missed something
when listing the new VM instructions.
Dibyendu Majumdar
2018-06-19 22:07:12 UTC
Permalink
On 19 June 2018 at 02:09, Luiz Henrique de Figueiredo
Post by Luiz Henrique de Figueiredo
If you're curious, check out the new VM instructions with luac -l.
While the improved performance of 5.4 is very welcome to everyone - I
do feel sad that there was a certain beauty in the minimalism of the
5.3 instruction set which is now lost. As always happens, performance
related changes introduce complexity (and in some ways ugliness) - as
I am only too aware because of the work in Ravi.

Can't be helped I suppose.

Regards
Dibyendu
Egor Skriptunoff
2018-06-20 20:45:51 UTC
Permalink
Post by Luiz Henrique de Figueiredo
BTW, we also welcome feedback on luac
I'm unable to build luac.exe
luac.obj : error LNK2005: luaP_opmodes already defined in lopcodes.obj
luac.exe : fatal error LNK1169: one or more multiply defined symbols found


And thanks for auto-randomseed!
Luiz Henrique de Figueiredo
2018-06-20 21:43:10 UTC
Permalink
Post by Egor Skriptunoff
I'm unable to build luac.exe
luac.obj : error LNK2005: luaP_opmodes already defined in lopcodes.obj
luac.exe : fatal error LNK1169: one or more multiply defined symbols found
Thanks for the report.

Why did the linker try to resolve symbols in lopcodes.obj since
presumably luac.obj is listed before lua.lib?
How are you building luac.exe?
Egor Skriptunoff
2018-06-21 05:16:27 UTC
Permalink
On Thu, Jun 21, 2018 at 12:43 AM, Luiz Henrique de Figueiredo <
Post by Luiz Henrique de Figueiredo
Post by Egor Skriptunoff
I'm unable to build luac.exe
luac.obj : error LNK2005: luaP_opmodes already defined in lopcodes.obj
luac.exe : fatal error LNK1169: one or more multiply defined symbols
found
Why did the linker try to resolve symbols in lopcodes.obj since
presumably luac.obj is listed before lua.lib?
How are you building luac.exe?
I'm using script based on "etc/luavs.bat" to build Lua under Visual Studio
command prompt.
It worked perfectly with all previous versions of Lua.

@setlocal
@set MYCOMPILE=cl /nologo /MD /O2 /W3 /c /D_CRT_SECURE_NO_DEPRECATE
@set MYLINK=link /nologo
cd src
%MYCOMPILE% /DLUA_BUILD_AS_DLL l*.c
del lua.obj luac.obj
%MYLINK% /DLL /out:lua54.dll l*.obj
%MYCOMPILE% /DLUA_BUILD_AS_DLL lua.c
%MYLINK% /out:lua.exe lua.obj lua54.lib
%MYCOMPILE% l*.c
del lua.obj linit.obj lbaselib.obj ldblib.obj liolib.obj lmathlib.obj
loslib.obj ltablib.obj lstrlib.obj loadlib.obj
%MYLINK% /out:luac.exe *.obj
del *.obj
cd ..

Should I set the correct order of .obj files for linker invocation?
What is this order?
Xavier Wang
2018-06-21 09:06:11 UTC
Permalink
Post by Egor Skriptunoff
On Thu, Jun 21, 2018 at 12:43 AM, Luiz Henrique de Figueiredo <
Post by Luiz Henrique de Figueiredo
Post by Egor Skriptunoff
I'm unable to build luac.exe
luac.obj : error LNK2005: luaP_opmodes already defined in lopcodes.obj
luac.exe : fatal error LNK1169: one or more multiply defined symbols
found
Why did the linker try to resolve symbols in lopcodes.obj since
presumably luac.obj is listed before lua.lib?
How are you building luac.exe?
I'm using script based on "etc/luavs.bat" to build Lua under Visual Studio
command prompt.
It worked perfectly with all previous versions of Lua.
@setlocal
@set MYCOMPILE=cl /nologo /MD /O2 /W3 /c /D_CRT_SECURE_NO_DEPRECATE
@set MYLINK=link /nologo
cd src
%MYCOMPILE% /DLUA_BUILD_AS_DLL l*.c
del lua.obj luac.obj
%MYLINK% /DLL /out:lua54.dll l*.obj
%MYCOMPILE% /DLUA_BUILD_AS_DLL lua.c
%MYLINK% /out:lua.exe lua.obj lua54.lib
%MYCOMPILE% l*.c
del lua.obj linit.obj lbaselib.obj ldblib.obj liolib.obj lmathlib.obj
loslib.obj ltablib.obj lstrlib.obj loadlib.obj
%MYLINK% /out:luac.exe *.obj
del *.obj
cd ..
Should I set the correct order of .obj files for linker invocation?
What is this order?
You should add lopcodes.obj to the “del” line
--
regards,
Xavier Wang.
Egor Skriptunoff
2018-06-22 20:57:13 UTC
Permalink
Post by Egor Skriptunoff
You should add lopcodes.obj to the “del” line
Thanks, it works!
Luiz Henrique de Figueiredo
2018-06-21 10:42:41 UTC
Permalink
Post by Egor Skriptunoff
%MYLINK% /out:luac.exe *.obj
I wonder why this is not as below, which was what I meant:

%MYLINK% /out:luac.exe luac.obj lua54.lib

Is it because lua54.lib is affected by /DLUA_BUILD_AS_DLL ? Indeed,
luac.exe needs to be statically linked.

Anyway, the luaP_opmodes glitch will be fixed for the final Lua 5.4.0
release. Thanks for the report.
Russell Haley
2018-06-25 05:35:52 UTC
Permalink
On Wed, Jun 20, 2018 at 10:16 PM, Egor Skriptunoff <
Post by Egor Skriptunoff
On Thu, Jun 21, 2018 at 12:43 AM, Luiz Henrique de Figueiredo <
Post by Luiz Henrique de Figueiredo
Post by Egor Skriptunoff
I'm unable to build luac.exe
luac.obj : error LNK2005: luaP_opmodes already defined in lopcodes.obj
luac.exe : fatal error LNK1169: one or more multiply defined symbols
found
Why did the linker try to resolve symbols in lopcodes.obj since
presumably luac.obj is listed before lua.lib?
How are you building luac.exe?
I'm using script based on "etc/luavs.bat" to build Lua under Visual Studio
command prompt.
It worked perfectly with all previous versions of Lua.
@setlocal
@set MYCOMPILE=cl /nologo /MD /O2 /W3 /c /D_CRT_SECURE_NO_DEPRECATE
@set MYLINK=link /nologo
cd src
%MYCOMPILE% /DLUA_BUILD_AS_DLL l*.c
del lua.obj luac.obj
%MYLINK% /DLL /out:lua54.dll l*.obj
%MYCOMPILE% /DLUA_BUILD_AS_DLL lua.c
%MYLINK% /out:lua.exe lua.obj lua54.lib
%MYCOMPILE% l*.c
del lua.obj linit.obj lbaselib.obj ldblib.obj liolib.obj lmathlib.obj
loslib.obj ltablib.obj lstrlib.obj loadlib.obj
%MYLINK% /out:luac.exe *.obj
del *.obj
cd ..
Should I set the correct order of .obj files for linker invocation?
What is this order?
I can't believe how long that's been sitting there and I didn't know about
it. I'm going to post this on the winlua github site. Egor, my I add your
name to the contributors list?

Russ
Egor Skriptunoff
2018-06-25 19:34:26 UTC
Permalink
Post by Egor Skriptunoff
I can't believe how long that's been sitting there and I didn't know about
it. I'm going to post this on the winlua github site. Egor, my I add your
name to the contributors list?
I've done nothing to become a contributor.
etc/luavs.bat is contributed by David Manura and Mike Pall
The file for Lua 5.1 is available at
https://www.lua.org/ftp/lua-5.1.5.tar.gz
An obvious modification is required to run it on Lua 5.2+ (print.c was
removed)
Russell Haley
2018-06-25 20:41:04 UTC
Permalink
On Mon, Jun 25, 2018 at 12:34 PM, Egor Skriptunoff <
Post by Egor Skriptunoff
Post by Egor Skriptunoff
I can't believe how long that's been sitting there and I didn't know
about it. I'm going to post this on the winlua github site. Egor, my I add
your name to the contributors list?
I've done nothing to become a contributor.
etc/luavs.bat is contributed by David Manura and Mike Pall
The file for Lua 5.1 is available at https://www.lua.org/ftp/lua-5.
1.5.tar.gz
An obvious modification is required to run it on Lua 5.2+ (print.c was
removed)
There was a copy in the 5.2 folder that I found too. While your
contribution is minor (version bump, etc) I personally believe it should be
recorded as it is pertinent to the state of the code. I'll leave it as-is
unless you reconsider.

Russ
Roberto Ierusalimschy
2018-06-19 12:22:44 UTC
Permalink
Post by Luiz Henrique de Figueiredo
The focus of Lua 5.4.0 is performance. We're especially interested in
feedback about its performance but all feedback is welcome. Thanks.
--lhf
In particular, we are interested in how our jump table implementation
behave in different platforms. It seems to hurt performance in some
systems. You can turn them on/off defining the flag LUA_USE_JUMPTABLE
to 1 or 0. The default is 1 when __GNUC__ is defined. Also, in some
systems you may get better performance with jump tables by turning
off some compiler optimizations, such as crossjumping.

-- Roberto
Dibyendu Majumdar
2018-06-19 22:02:34 UTC
Permalink
On 18 June 2018 at 21:33, Luiz Henrique de Figueiredo
Post by Luiz Henrique de Figueiredo
The focus of Lua 5.4.0 is performance. We're especially interested in
feedback about its performance but all feedback is welcome. Thanks.
I am curious about the changes in how function calls/returns are
handled. Were the drivers for these changes performance related?

Thanks and Regards
Dibyendu
Roberto Ierusalimschy
2018-06-20 12:56:37 UTC
Permalink
Post by Dibyendu Majumdar
I am curious about the changes in how function calls/returns are
handled. Were the drivers for these changes performance related?
Yes.

-- Roberto
dyngeccetor8
2018-06-23 02:24:18 UTC
Permalink
Post by Luiz Henrique de Figueiredo
Lua 5.4.0 (work2) is now available for testing at
http://www.lua.org/work/lua-5.4.0-work2.tar.gz
[...]
The focus of Lua 5.4.0 is performance. We're especially interested in
feedback about its performance but all feedback is welcome. Thanks.
--lhf
5.4.0 vs 5.3.4, heavy table usage, time in seconds (less is better)

5.3.4 / 5.4.0

17.232 / 16.104 -- 5.4_speed = 5.3_speed * 107.0%
1333.64 / 1429.24 -- 5.4_speed = 5.3_speed * 93.3%

For first test case I've done three runs of each version. Times does not
differ. For second test case I've done second run for 5.4.0 version
to assure that execution is really slower. First run time was 1457.46s.

Benchmark is reformatting source code for given set of Lua files.
Formatter spends 30% to 60% of time in each run for merging text
chunks. This produces a lot of "garbage" memory.

First test case contains some typical files, 38 files, ~ 1 MiB total
size. Second case is "survival" test with 2.2k files, ~ 101 MiB total
size. Second test set includes first test set. Contains more
machine-generated Lua files.

OS GNU/Linux, 14.04.1-Ubuntu, kernel 3.19.0-32-generic.

-- Martin
dyngeccetor8
2018-06-24 15:11:05 UTC
Permalink
5.4.0 version [...]
execution is really slower.
I've sketched artificial test to exploit slower (but economical)
garbage collector and give a try for new random().

Can someone retest it?

$ time lua gc_benchmark.lua
Lua 5.3

real 0m38.009s
user 0m36.640s
sys 0m1.384s

$ time ~/Downloads/lua-5.4.0-work2/src/lua gc_benchmark.lua
Lua 5.4

real 1m17.759s
user 1m16.508s
sys 0m1.208s

-- gc_bechmark.lua (
local num_iterations = 1e8
local table_width = 1e8

print(_VERSION)

math.randomseed(os.time())

local t = {}
for i = 1, num_iterations do
local k = math.random(table_width)
t[k] = {}
end
-- )

-- Martin
Russell Haley
2018-06-25 06:00:45 UTC
Permalink
Post by dyngeccetor8
5.4.0 version [...]
execution is really slower.
I've sketched artificial test to exploit slower (but economical)
garbage collector and give a try for new random().
Can someone retest it?
Raw numbers on Windows 10 - Third Gen i7 laptop

Lua 5.3:
https://pastebin.com/ti0MwpxN

Lua 5.4:
https://pastebin.com/ZVrUDZfv

5.4 seems a smidgen faster.

Night,
Russ
Post by dyngeccetor8
$ time lua gc_benchmark.lua
Lua 5.3
real 0m38.009s
user 0m36.640s
sys 0m1.384s
$ time ~/Downloads/lua-5.4.0-work2/src/lua gc_benchmark.lua
Lua 5.4
real 1m17.759s
user 1m16.508s
sys 0m1.208s
-- gc_bechmark.lua (
local num_iterations = 1e8
local table_width = 1e8
print(_VERSION)
math.randomseed(os.time())
local t = {}
for i = 1, num_iterations do
local k = math.random(table_width)
t[k] = {}
end
-- )
-- Martin
dyngeccetor8
2018-06-25 19:35:51 UTC
Permalink
Post by Russell Haley
Raw numbers on Windows 10 - Third Gen i7 laptop
https://pastebin.com/ti0MwpxN
https://pastebin.com/ZVrUDZfv
5.4 seems a smidgen faster.
Night,
Russ
It's strange that on your machine there is no difference.

Idea was to exploit new garbage collector. More specifically,
it's performance in registering new objects. Which gives
two-fold performance drop on my machine. It looks like
on your machine new collector works just same (or disabled
in both cases). I have no ideas to explain this.

-- Martin
Russell Haley
2018-06-25 20:47:58 UTC
Permalink
Post by dyngeccetor8
Post by Russell Haley
Raw numbers on Windows 10 - Third Gen i7 laptop
https://pastebin.com/ti0MwpxN
https://pastebin.com/ZVrUDZfv
5.4 seems a smidgen faster.
Night,
Russ
It's strange that on your machine there is no difference.
Idea was to exploit new garbage collector. More specifically,
it's performance in registering new objects. Which gives
two-fold performance drop on my machine. It looks like
on your machine new collector works just same (or disabled
in both cases). I have no ideas to explain this.
I have 16 GB ram on the computer. Could that explain anything? I'll try it
again on FreeBSD if my "Attention Deficit Disorder" doesn't distract me
tonight (I get about 1.5 - 2 hours at night, it's hard to stay focused on
anything). I see there is possible DTrace support for Lua via this er..
add in.. https://github.com/chrisa/lua-usdt. That would be a fun little
project as I've been looking for a reason to do some more DTrace scripting.

Russ
Post by dyngeccetor8
-- Martin
Russell Haley
2018-06-26 05:20:34 UTC
Permalink
Post by Russell Haley
Post by dyngeccetor8
Post by Russell Haley
Raw numbers on Windows 10 - Third Gen i7 laptop
https://pastebin.com/ti0MwpxN
https://pastebin.com/ZVrUDZfv
5.4 seems a smidgen faster.
Night,
Russ
It's strange that on your machine there is no difference.
Idea was to exploit new garbage collector. More specifically,
it's performance in registering new objects. Which gives
two-fold performance drop on my machine. It looks like
on your machine new collector works just same (or disabled
in both cases). I have no ideas to explain this.
I have 16 GB ram on the computer. Could that explain anything? I'll try
it again on FreeBSD if my "Attention Deficit Disorder" doesn't distract me
tonight (I get about 1.5 - 2 hours at night, it's hard to stay focused on
anything). I see there is possible DTrace support for Lua via this er..
add in.. https://github.com/chrisa/lua-usdt. That would be a fun little
project as I've been looking for a reason to do some more DTrace scripting.
Russ
My results are invalid. I was using the following command line from
powershell:

Measure-Command {lua.exe C:\temp\gc_benchmark.lua}

which was actually calling C:\Program Files\WinLua\Lua\5.3\bin\lua.exe.
All I was actually doing was comparing 5.3 32 bit to 5.3 64 bit. oops!

When run *correctly* Lua 5.3 uses about 14 MB of memory (times still
similar to last night). 5.4 eats a GB of memory on Windows 10 before it's
killed with a 'not enough memory' error:

C:\Users\russh\lua\lua-5.4.0-work2\src> lua.exe C:\temp\gc_benchmark.lua
Lua 5.3
C:\Users\russh\lua\lua-5.4.0-work2\src> .\lua.exe C:\temp\gc_benchmark.lua
Lua 5.4
C:\Users\russh\lua\lua-5.4.0-work2\src\lua.exe: not enough memory

On FreeBSD the results are much much different. BOTH lua53 (installed via
binary from pkg) and lua 5.4 (run from the 5.4 src directory to be sure)
eat all the memory of my virtual machine and then chew up my swap and kill
the entire VM. How cool is that? I'm going to try it on my FreeBSD server
with 24 GB of ram! tee hee...

Russ
Post by Russell Haley
Post by dyngeccetor8
-- Martin
Russell Haley
2018-06-26 05:53:47 UTC
Permalink
Post by Russell Haley
Post by Russell Haley
Post by dyngeccetor8
Post by Russell Haley
Raw numbers on Windows 10 - Third Gen i7 laptop
https://pastebin.com/ti0MwpxN
https://pastebin.com/ZVrUDZfv
5.4 seems a smidgen faster.
Night,
Russ
It's strange that on your machine there is no difference.
Idea was to exploit new garbage collector. More specifically,
it's performance in registering new objects. Which gives
two-fold performance drop on my machine. It looks like
on your machine new collector works just same (or disabled
in both cases). I have no ideas to explain this.
I have 16 GB ram on the computer. Could that explain anything? I'll try
it again on FreeBSD if my "Attention Deficit Disorder" doesn't distract me
tonight (I get about 1.5 - 2 hours at night, it's hard to stay focused on
anything). I see there is possible DTrace support for Lua via this er..
add in.. https://github.com/chrisa/lua-usdt. That would be a fun little
project as I've been looking for a reason to do some more DTrace scripting.
Russ
My results are invalid. I was using the following command line from
Measure-Command {lua.exe C:\temp\gc_benchmark.lua}
which was actually calling C:\Program Files\WinLua\Lua\5.3\bin\lua.exe.
All I was actually doing was comparing 5.3 32 bit to 5.3 64 bit. oops!
When run *correctly* Lua 5.3 uses about 14 MB of memory (times still
similar to last night). 5.4 eats a GB of memory on Windows 10 before it's
C:\Users\russh\lua\lua-5.4.0-work2\src> lua.exe C:\temp\gc_benchmark.lua
Lua 5.3
C:\Users\russh\lua\lua-5.4.0-work2\src> .\lua.exe C:\temp\gc_benchmark.lua
Lua 5.4
C:\Users\russh\lua\lua-5.4.0-work2\src\lua.exe: not enough memory
On FreeBSD the results are much much different. BOTH lua53 (installed via
binary from pkg) and lua 5.4 (run from the 5.4 src directory to be sure)
eat all the memory of my virtual machine and then chew up my swap and kill
the entire VM. How cool is that? I'm going to try it on my FreeBSD server
with 24 GB of ram! tee hee...
Russ
The results on FreeBSD are more in line with your original results.

https://pastebin.com/NCW35L2B

I reported the final memory usage, but it doesn't really tell much of a
story. When watching the memory climb in 'top' the 5.4 interpreter memory
usage seemed to rise more slowly in more measured increments. I noticed in
5.3 that it will allocate big chunks (300 or more MB) of memory at a time,
but also de-allocate big chunks at a time (I saw the system free over 1 GB
of memory once, but often ~400 MB).

Russ
Post by Russell Haley
Post by Russell Haley
Post by dyngeccetor8
-- Martin
dyngeccetor8
2018-06-26 09:39:40 UTC
Permalink
Post by Russell Haley
The results on FreeBSD are more in line with your original results.
https://pastebin.com/NCW35L2B
I reported the final memory usage, but it doesn't really tell much of a
story. When watching the memory climb in 'top' the 5.4 interpreter memory
usage seemed to rise more slowly in more measured increments. I noticed in
5.3 that it will allocate big chunks (300 or more MB) of memory at a time,
but also de-allocate big chunks at a time (I saw the system free over 1 GB
of memory once, but often ~400 MB).
Russ
Thank you for testing!

I used this benchmark on 16GB memory machine, where it ate around 8GB.
For another memory sizes you can change <num_iterations> or <table_width>.

Yesterday I ran this test on Lua 5.2.4 and 5.1.5. From younger to older,
run times and memory consumption increased. 5.3.4 has outstanding run time.
This looks like a curse of local minimum.

-- Martin
Roberto Ierusalimschy
2018-06-25 14:18:54 UTC
Permalink
Post by dyngeccetor8
5.4.0 version [...]
execution is really slower.
I've sketched artificial test to exploit slower (but economical)
garbage collector and give a try for new random().
Can someone retest it?
[...]
local num_iterations = 1e8
local table_width = 1e8
print(_VERSION)
math.randomseed(os.time())
local t = {}
for i = 1, num_iterations do
local k = math.random(table_width)
t[k] = {}
end
This is a particularly nasty example for Lua 5.4. It generates
little garbage (around 25% of the total memory used); and it is all
around one single big table, so nothing relevant ever becomes "old".
In fact, both 5.3 and 5.4 run faster if we turn off the collector,
without using significantly more memory. (But even then 5.4 is still
some 10% slower than 5.3. I will check that.)

As a consolation, luajit also does not perform particularly well
in this benchmark :-) (only ~10% faster than 5.3).

-- Roberto
Roberto Ierusalimschy
2018-06-25 14:22:01 UTC
Permalink
Post by dyngeccetor8
5.4.0 vs 5.3.4, heavy table usage, time in seconds (less is better)
5.3.4 / 5.4.0
17.232 / 16.104 -- 5.4_speed = 5.3_speed * 107.0%
1333.64 / 1429.24 -- 5.4_speed = 5.3_speed * 93.3%
For first test case I've done three runs of each version. Times does not
differ. For second test case I've done second run for 5.4.0 version
to assure that execution is really slower. First run time was 1457.46s.
Benchmark is reformatting source code for given set of Lua files.
Formatter spends 30% to 60% of time in each run for merging text
chunks. This produces a lot of "garbage" memory.
First test case contains some typical files, 38 files, ~ 1 MiB total
size. Second case is "survival" test with 2.2k files, ~ 101 MiB total
size. Second test set includes first test set. Contains more
machine-generated Lua files.
OS GNU/Linux, 14.04.1-Ubuntu, kernel 3.19.0-32-generic.
Do you know how much memory each case used?

-- Roberto
dyngeccetor8
2018-06-25 19:09:12 UTC
Permalink
Post by Roberto Ierusalimschy
Post by dyngeccetor8
5.4.0 vs 5.3.4, heavy table usage, time in seconds (less is better)
5.3.4 / 5.4.0
17.232 / 16.104 -- 5.4_speed = 5.3_speed * 107.0%
1333.64 / 1429.24 -- 5.4_speed = 5.3_speed * 93.3%
[...]
OS GNU/Linux, 14.04.1-Ubuntu, kernel 3.19.0-32-generic.
Do you know how much memory each case used?
-- Roberto
I don't. That were field tests, only time was measured.

-- Martin
Russell Haley
2018-06-23 04:12:31 UTC
Permalink
On Mon, Jun 18, 2018 at 1:33 PM, Luiz Henrique de Figueiredo <
Post by Luiz Henrique de Figueiredo
Lua 5.4.0 (work2) is now available for testing at
http://www.lua.org/work/lua-5.4.0-work2.tar.gz
The checksums are
MD5 3cdf2a4eb84dde6b6aaf5d2d1de07be9 -
SHA1 e8484e61c5c338e3ec2f75dbe0f6703d079fecf9 -
This is a work version. All details may change in the final version.
An updated reference manual is included and also available at
http://www.lua.org/work/doc
The complete diffs from work1 to work2 are available at
http://www.lua.org/work/diffs-lua-5.4.0-work1-work2.html
http://www.lua.org/work/diffu-lua-5.4.0-work1-work2.html
Previous work versions are available at
http://www.lua.org/work/
The focus of Lua 5.4.0 is performance. We're especially interested in
feedback about its performance but all feedback is welcome. Thanks.
--lhf
Thank you for fixing the FreeBSD build! I managed to patch luaossl for 5.4
too. I've updated the port file but haven't posted it yet. Interestingly
when testing with no readline in the build I can still interact with the
interpreter???

https://pastebin.com/TWTPRHdi

Drat. Laptop battery is dead...

Russ
Thijs Schreijer
2018-06-25 21:54:22 UTC
Permalink
Post by Luiz Henrique de Figueiredo
Lua 5.4.0 (work2) is now available for testing at
http://www.lua.org/work/lua-5.4.0-work2.tar.gz
The checksums are
MD5 3cdf2a4eb84dde6b6aaf5d2d1de07be9 -
SHA1 e8484e61c5c338e3ec2f75dbe0f6703d079fecf9 -
This is a work version. All details may change in the final version.
An updated reference manual is included and also available at
http://www.lua.org/work/doc
The complete diffs from work1 to work2 are available at
http://www.lua.org/work/diffs-lua-5.4.0-work1-work2.html
http://www.lua.org/work/diffu-lua-5.4.0-work1-work2.html
Previous work versions are available at
http://www.lua.org/work/
The focus of Lua 5.4.0 is performance. We're especially interested in
feedback about its performance but all feedback is welcome. Thanks.
--lhf
Fyi, just tested this with LuaWinMake as well (unreleased 5.4 branch [1]). With GCC based toolchains it works, with MS toolchains I get the same error as previously reported by Egor Skriptunoff (regarding `luaP_opmodes`)

Thijs

[1] https://github.com/Tieske/luawinmake/pull/14
Matthew Wild
2018-07-11 14:51:46 UTC
Permalink
On 18 June 2018 at 21:33, Luiz Henrique de Figueiredo
Post by Luiz Henrique de Figueiredo
Lua 5.4.0 (work2) is now available for testing at
http://www.lua.org/work/lua-5.4.0-work2.tar.gz
The checksums are
MD5 3cdf2a4eb84dde6b6aaf5d2d1de07be9 -
SHA1 e8484e61c5c338e3ec2f75dbe0f6703d079fecf9 -
This is a work version. All details may change in the final version.
An updated reference manual is included and also available at
http://www.lua.org/work/doc
This URL is currently returning a 404.

Regards,
Matthew
Luiz Henrique de Figueiredo
2018-07-11 14:57:09 UTC
Permalink
Post by Matthew Wild
Post by Luiz Henrique de Figueiredo
An updated reference manual is included and also available at
http://www.lua.org/work/doc
This URL is currently returning a 404.
Sorry. It's working again now. Thanks for the report.
Russell Haley
2018-07-15 06:16:58 UTC
Permalink
On Wed, Jul 11, 2018 at 7:57 AM, Luiz Henrique de Figueiredo <
Post by Luiz Henrique de Figueiredo
Post by Matthew Wild
Post by Luiz Henrique de Figueiredo
An updated reference manual is included and also available at
http://www.lua.org/work/doc
This URL is currently returning a 404.
Sorry. It's working again now. Thanks for the report.
Hi,

This is a repeat announcement from the 5.3.5 discussion:

The port patch for Lua 5.4.0 (work2) on FreeBSD is in review here:

https://reviews.freebsd.org/D14709.

It's largely identical to the 5.3 patch here:

https://reviews.freebsd.org/D13690


Once again, great gobs of Thanks to Andrew Gierth and duarnimator for their
help.

Russ
Egor Skriptunoff
2018-07-27 20:09:16 UTC
Permalink
Post by Luiz Henrique de Figueiredo
The focus of Lua 5.4.0 is performance.
We're especially interested in feedback about its performance.
I've tested the performance of one program under Lua 5.1 / 5.2 / 5.3 / 5.4
The program is just a calculator of SHA512 written for Lua 5.1, all bitwise
operations are emulated with arithmetic.
The code is here: https://stackoverflow.com/a/51561685/6834680
(the code consists of two files: the module and the testing/benchmarking
script)

All Lua executables under test are 64-bit, built with the same script under
the same compiler.
So, the benchmarking is fair.


C:\>lua51.exe sha2for51_test.lua
CPU seconds: 12.371

C:\>lua52.exe sha2for51_test.lua
CPU seconds: 12.513

C:\>lua53.exe sha2for51_test.lua
CPU seconds: 16.801

C:\>lua54.exe sha2for51_test.lua
CPU seconds: 15.6


and one more time (to make sure there are no fluctuations here)


C:\>lua51.exe sha2for51_test.lua
CPU seconds: 12.356

C:\>lua52.exe sha2for51_test.lua
CPU seconds: 12.496

C:\>lua53.exe sha2for51_test.lua
CPU seconds: 16.88

C:\>lua54.exe sha2for51_test.lua
CPU seconds: 15.491


Two notes on this:

1)
Arithmetic computations written for Lua 5.1 appear to be 1.25 times slower
in Lua 5.4 (compared to Lua 5.1)
(Probably, this happened because of heavy mixing of floats with integers in
arithmetic expressions.)

2)
Lua 5.4 is *7-8% faster* than Lua 5.3 !
云风 Cloud Wu
2018-07-30 02:40:20 UTC
Permalink
Post by Egor Skriptunoff
1)
Arithmetic computations written for Lua 5.1 appear to be 1.25 times slower
in Lua 5.4 (compared to Lua 5.1)
(Probably, this happened because of heavy mixing of floats with integers
in arithmetic expressions.)
Maybe one of reason is that `Integer modulus` is slower than `Float
modulus`, See function `luaV_mod` in lvm.c .
Egor Skriptunoff
2018-08-09 19:38:21 UTC
Permalink
Let's benchmark modulo operator.
The code is here: pastebin.com/13XBxdn6
The Lua executables are x64 exe-files built with VS2010.

The results:

C:\>lua51.exe modulo_benchmark.lua
CPU seconds for floating point modulo: 2.377

C:\>lua52.exe modulo_benchmark.lua
CPU seconds for floating point modulo: 2.519

C:\>lua53.exe modulo_benchmark.lua
CPU seconds for integer modulo: 3.84
CPU seconds for floating point modulo: 6.016

C:\>lua54.exe modulo_benchmark.lua
CPU seconds for integer modulo: 3.765
CPU seconds for floating point modulo: 6.241

Modulo operator become slower starting with Lua 5.3.
This is because of Lua 5.2 formula
a%b = a-floor(a/b)*b
was replaced with more slow, but more correct and more precise Lua 5.3
algorithm
m=fmod(a,b); if (m*b<0) m+=b; return m; // for floats
m=a%b; if (m!=0 && a^b<0) m+=b; return m; // for ints

BTW, changing (m!=0 && a^b<0) to (m!=0 && m^b<0) might make the code more
optimizer-friendly:
lifetimes of local variables "a" and "m" are non-overlapping, hence both
these variables could share single CPU register.

Ok, modulo operator has changed its implementation, that's why it become
slower.
Let's benchmark another operator that didn't change its semantics since Lua
5.1.
For example, addition.
Use the same benchmarking code as previously, but replace all percents with
pluses.

The results:

C:\>lua51.exe addition_benchmark.lua
CPU seconds for floating point addition: 1.185

C:\>lua52.exe addition_benchmark.lua
CPU seconds for floating point addition: 1.093

C:\>lua53.exe addition_benchmark.lua
CPU seconds for integer addition: 1.107
CPU seconds for floating point addition: 1.556

C:\>lua54.exe addition_benchmark.lua
CPU seconds for integer addition: 1.452
CPU seconds for floating point addition: 1.717

Why numeric addition (the simplest thing in Lua) become slower starting
with Lua 5.3?
Is it because of detecting operand subtypes (int/float) inside every "ADD"
instruction?
Post by Egor Skriptunoff
Arithmetic computations written for Lua 5.1 appear to be 1.25 times slower
Post by Egor Skriptunoff
in Lua 5.4 (compared to Lua 5.1)
(Probably, this happened because of heavy mixing of floats with integers
in arithmetic expressions.)
Maybe one of reason is that `Integer modulus` is slower than `Float
modulus`, See function `luaV_mod` in lvm.c .
The situation is strange:
As displayed above, on usual computer under x64 Windows integer operations
in Lua are faster than floating point ones.
But on 10-year-old PC under x64 Linux integer operations are slower than
floating point operations:

$ lua -v modulo_benchmark.lua
Lua 5.3.4 Copyright (C) 1994-2017 Lua.org, PUC-Rio
CPU seconds for integer modulo: 7.368922
CPU seconds for floating point modulo: 5.265935

$ lua -v addition_benchmark.lua
Lua 5.3.4 Copyright (C) 1994-2017 Lua.org, PUC-Rio
CPU seconds for integer addition: 1.893307
CPU seconds for floating point addition: 1.762248

So, this contradiction shows all these benchmarks are probably valueless
:-)
Rodrigo Azevedo
2018-08-09 20:50:11 UTC
Permalink
Post by Egor Skriptunoff
Let's benchmark modulo operator.
The code is here: pastebin.com/13XBxdn6
The Lua executables are x64 exe-files built with VS2010.
C:\>lua51.exe modulo_benchmark.lua
CPU seconds for floating point modulo: 2.377
C:\>lua52.exe modulo_benchmark.lua
CPU seconds for floating point modulo: 2.519
C:\>lua53.exe modulo_benchmark.lua
CPU seconds for integer modulo: 3.84
CPU seconds for floating point modulo: 6.016
C:\>lua54.exe modulo_benchmark.lua
CPU seconds for integer modulo: 3.765
CPU seconds for floating point modulo: 6.241
Modulo operator become slower starting with Lua 5.3.
This is because of Lua 5.2 formula
a%b = a-floor(a/b)*b
was replaced with more slow, but more correct and more precise Lua 5.3
algorithm
m=fmod(a,b); if (m*b<0) m+=b; return m; // for floats
m=a%b; if (m!=0 && a^b<0) m+=b; return m; // for ints
BTW, changing (m!=0 && a^b<0) to (m!=0 && m^b<0) might make the code more
lifetimes of local variables "a" and "m" are non-overlapping, hence both
these variables could share single CPU register.
Ok, modulo operator has changed its implementation, that's why it become
slower.
Let's benchmark another operator that didn't change its semantics since
Lua 5.1.
For example, addition.
Use the same benchmarking code as previously, but replace all percents
with pluses.
C:\>lua51.exe addition_benchmark.lua
CPU seconds for floating point addition: 1.185
C:\>lua52.exe addition_benchmark.lua
CPU seconds for floating point addition: 1.093
C:\>lua53.exe addition_benchmark.lua
CPU seconds for integer addition: 1.107
CPU seconds for floating point addition: 1.556
C:\>lua54.exe addition_benchmark.lua
CPU seconds for integer addition: 1.452
CPU seconds for floating point addition: 1.717
Why numeric addition (the simplest thing in Lua) become slower starting
with Lua 5.3?
Is it because of detecting operand subtypes (int/float) inside every "ADD"
instruction?
Post by Egor Skriptunoff
Arithmetic computations written for Lua 5.1 appear to be 1.25 times
slower in Lua 5.4 (compared to Lua 5.1)
(Probably, this happened because of heavy mixing of floats with integers
in arithmetic expressions.)
Maybe one of reason is that `Integer modulus` is slower than `Float
modulus`, See function `luaV_mod` in lvm.c .
As displayed above, on usual computer under x64 Windows integer operations
in Lua are faster than floating point ones.
But on 10-year-old PC under x64 Linux integer operations are slower than
$ lua -v modulo_benchmark.lua
Lua 5.3.4 Copyright (C) 1994-2017 Lua.org, PUC-Rio
CPU seconds for integer modulo: 7.368922
CPU seconds for floating point modulo: 5.265935
$ lua -v addition_benchmark.lua
Lua 5.3.4 Copyright (C) 1994-2017 Lua.org, PUC-Rio
CPU seconds for integer addition: 1.893307
CPU seconds for floating point addition: 1.762248
So, this contradiction shows all these benchmarks are probably valueless
:-)
On a recent Linux 4.15.0-24-generic x86_64
gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3)
Intel(R) Core(TM) i5-6500 CPU

MODULO

lua-5.2.4/src/lua teste_lua54_benchmark.lua

CPU seconds for integer modulo: 1.949047
CPU seconds for floating point modulo: 1.934812

lua-5.3.5/src/lua teste_lua54_benchmark.lua

CPU seconds for integer modulo: 3.228948
CPU seconds for floating point modulo: 1.986498

lua-5.4.0-work2/src/lua teste_lua54_benchmark.lua

CPU seconds for integer modulo: 3.201198
CPU seconds for floating point modulo: 2.24338

OBS: CPU throttling ON

ADDITION

lua-5.2.4/src/lua teste_lua54_benchmark_plus.lua

CPU seconds for integer addition: 0.881306
CPU seconds for floating point addition: 0.88283

lua-5.3.5/src/lua teste_lua54_benchmark_plus.lua

CPU seconds for integer addition: 0.850767
CPU seconds for floating point addition: 0.882885

lua-5.4.0-work2/src/lua teste_lua54_benchmark_plus.lua

CPU seconds for integer addition: 1.04777
CPU seconds for floating point addition: 1.071874

OBS: Lua 5.4 compiles (luac -l) with a lot more "LOADK"
instructions compared to lua 5.2 and 5.3.

Is "LOADK" primarily responsible for the performance gap
between versions 5.3 and 5.4?
--
Rodrigo Azevedo Moreira da Silva
云风 Cloud Wu
2018-08-13 02:55:22 UTC
Permalink
Post by Egor Skriptunoff
Let's benchmark modulo operator.
The code is here: pastebin.com/13XBxdn6
The Lua executables are x64 exe-files built with VS2010.
C:\>lua51.exe addition_benchmark.lua
CPU seconds for floating point addition: 1.185
C:\>lua52.exe addition_benchmark.lua
CPU seconds for floating point addition: 1.093
C:\>lua53.exe addition_benchmark.lua
CPU seconds for integer addition: 1.107
CPU seconds for floating point addition: 1.556
C:\>lua54.exe addition_benchmark.lua
CPU seconds for integer addition: 1.452
CPU seconds for floating point addition: 1.717
Why numeric addition (the simplest thing in Lua) become slower starting
with Lua 5.3?
Is it because of detecting operand subtypes (int/float) inside every "ADD"
instruction?
Lua 5.4 optimize small constant integer ADD by OP_ADDI , and a bit slower
for big constants.

You can change those big integers to small one and test again, you will
find much faster.
Roberto Ierusalimschy
2018-08-13 17:01:49 UTC
Permalink
Post by 云风 Cloud Wu
Lua 5.4 optimize small constant integer ADD by OP_ADDI , and a bit slower
for big constants.
You can change those big integers to small one and test again, you will
find much faster.
That is correct. When choosing the new opcodes, we did some measures of
occurrences of some kinds of operands for some operations over a big
corpus of Lua files. We found that occurrences of additions with "big"
integer constants (actually anything larger than 2) and additions with
any float constant are very rare and therefore it is not worth to have
specific opcodes for those cases.

If these hypotheses are wrong, we will be happy to learn about and
rework our set of opcodes.

-- Roberto
Roberto Ierusalimschy
2018-08-23 17:04:21 UTC
Permalink
Post by Egor Skriptunoff
I've tested the performance of one program under Lua 5.1 / 5.2 / 5.3 / 5.4
The program is just a calculator of SHA512 written for Lua 5.1, all bitwise
operations are emulated with arithmetic.
The code is here: https://stackoverflow.com/a/51561685/6834680
(the code consists of two files: the module and the testing/benchmarking
script)
All Lua executables under test are 64-bit, built with the same script under
the same compiler.
So, the benchmarking is fair.
C:\>lua51.exe sha2for51_test.lua
CPU seconds: 12.371
C:\>lua52.exe sha2for51_test.lua
CPU seconds: 12.513
C:\>lua53.exe sha2for51_test.lua
CPU seconds: 16.801
C:\>lua54.exe sha2for51_test.lua
CPU seconds: 15.6
At least in my machine, this time difference mainly boils down to
the mod operation. If I compile lua 5.4 using the old (but slightly
wrong) definition for '%', Lua 5.4 runs 4% slower than Lua 5.1, which
I think is due to the high number of coercions between integers and
floats.

We can try to improve the implementation of '%' for floats, but
I am not sure it is worth, given that, with integers and bitwise
operations, that kind of use of '%' for floats is much diminished.

-- Roberto
Dirk Laurie
2018-08-24 08:42:09 UTC
Permalink
Given Lua's position as an extension language embedded in a C host,
would not simply invoking fmod from math.h be the obvious thing to do?
Op Do., 23 Aug. 2018 om 19:04 het Roberto Ierusalimschy
Post by Roberto Ierusalimschy
Post by Egor Skriptunoff
I've tested the performance of one program under Lua 5.1 / 5.2 / 5.3 / 5.4
The program is just a calculator of SHA512 written for Lua 5.1, all bitwise
operations are emulated with arithmetic.
The code is here: https://stackoverflow.com/a/51561685/6834680
(the code consists of two files: the module and the testing/benchmarking
script)
All Lua executables under test are 64-bit, built with the same script under
the same compiler.
So, the benchmarking is fair.
C:\>lua51.exe sha2for51_test.lua
CPU seconds: 12.371
C:\>lua52.exe sha2for51_test.lua
CPU seconds: 12.513
C:\>lua53.exe sha2for51_test.lua
CPU seconds: 16.801
C:\>lua54.exe sha2for51_test.lua
CPU seconds: 15.6
At least in my machine, this time difference mainly boils down to
the mod operation. If I compile lua 5.4 using the old (but slightly
wrong) definition for '%', Lua 5.4 runs 4% slower than Lua 5.1, which
I think is due to the high number of coercions between integers and
floats.
We can try to improve the implementation of '%' for floats, but
I am not sure it is worth, given that, with integers and bitwise
operations, that kind of use of '%' for floats is much diminished.
-- Roberto
Hugo Musso Gualandi
2018-08-24 11:23:32 UTC
Permalink
Lua does use fmod but it doesn't have exactly the same semantics. See the comment on luai_nummod on limits.h for details.

On Aug 24, 2018 4:42 AM, Dirk Laurie wrote: > > Given Lua's position as an extension language embedded in a C host, > would not simply invoking fmod from math.h be the obvious thing to do? > Op Do., 23 Aug. 2018 om 19:04 het Roberto Ierusalimschy > geskryf: > > > > > I've tested the performance of one program under Lua 5.1 / 5.2 / 5.3 / 5.4 > > > The program is just a calculator of SHA512 written for Lua 5.1, all bitwise > > > operations are emulated with arithmetic. > > > The code is here:  https://stackoverflow.com/a/51561685/6834680 > > > (the code consists of two files: the module and the testing/benchmarking > > > script) > > > > > > All Lua executables under test are 64-bit, built with the same script under > > > the same compiler. > > > So, the benchmarking is fair. > > > > > > > > > C:\>lua51.exe sha2for51_test.lua > > > CPU seconds:    12.371 > > > > > > C:\>lua52.exe sha2for51_test.lua > > > CPU seconds:    12.513 > > > > > > C:\>lua53.exe sha2for51_test.lua > > > CPU seconds:    16.801 > > > > > > C:\>lua54.exe sha2for51_test.lua > > > CPU seconds:    15.6 > > > > At least in my machine, this time difference mainly boils down to > > the mod operation. If I compile lua 5.4 using the old (but slightly > > wrong) definition for '%', Lua 5.4 runs 4% slower than Lua 5.1, which > > I think is due to the high number of coercions between integers and > > floats. > > > > We can try to improve the implementation of '%' for floats, but > > I am not sure it is worth, given that, with integers and bitwise > > operations, that kind of use of '%' for f
Roberto Ierusalimschy
2018-08-24 18:15:49 UTC
Permalink
Post by Hugo Musso Gualandi
Lua does use fmod but it doesn't have exactly the same semantics. See the comment on luai_nummod on limits.h for details.
More exactly, Lua uses this definition:

{ (m) = l_mathop(fmod)(a,b); if ((m)*(b) < 0) (m) += (b); }

The correction is due to fmod assumes a rounding towards zero in
the division, while Lua rounds towards minus infinite. But the test
adds little to the cost; 'fmod' is the expensive operation here.

Earlier Lua versions used the mathematical definition:

(m) = ((a) - l_mathop(floor)((a)/(b))*(b))

It is *much* faster than 'fmod', but it gives NaN when 'b' is infinite
(it should give 'a'). I am not sure whether there are other problematic
cases besides 'b' being inf or -inf.

-- Roberto
Roberto Ierusalimschy
2018-08-24 20:12:23 UTC
Permalink
Post by Roberto Ierusalimschy
Post by Hugo Musso Gualandi
Lua does use fmod but it doesn't have exactly the same semantics. See the comment on luai_nummod on limits.h for details.
{ (m) = l_mathop(fmod)(a,b); if ((m)*(b) < 0) (m) += (b); }
The correction is due to fmod assumes a rounding towards zero in
the division, while Lua rounds towards minus infinite. But the test
adds little to the cost; 'fmod' is the expensive operation here.
(m) = ((a) - l_mathop(floor)((a)/(b))*(b))
It is *much* faster than 'fmod', but it gives NaN when 'b' is infinite
(it should give 'a'). I am not sure whether there are other problematic
cases besides 'b' being inf or -inf.
-- Roberto
Apparently, the following formula gives the correct answer for all
cases:

{ (m) = l_mathop(floor)((a)/(b)); \
(m) = ((m) == 0) ? ((a) * (b) < 0 ? (a) + (b) : (a)) : (a) - (m)*(b); }

In my machine, it can be up to 5x faster than 'fmod'. ('fmod' time
seems to depend heavily on the parameters.)

-- Roberto
Egor Skriptunoff
2018-08-25 17:28:10 UTC
Permalink
Post by Roberto Ierusalimschy
Post by Roberto Ierusalimschy
{ (m) = l_mathop(fmod)(a,b); if ((m)*(b) < 0) (m) += (b); }
(m) = ((a) - l_mathop(floor)((a)/(b))*(b))
It is *much* faster than 'fmod', but it gives NaN when 'b' is infinite
(it should give 'a'). I am not sure whether there are other problematic
cases besides 'b' being inf or -inf.
Apparently, the following formula gives the correct answer for all
{ (m) = l_mathop(floor)((a)/(b)); \
(m) = ((m) == 0) ? ((a) * (b) < 0 ? (a) + (b) : (a)) : (a) - (m)*(b); }
In my machine, it can be up to 5x faster than 'fmod'. ('fmod' time
seems to depend heavily on the parameters.)
Another problem of "fast modulo" is possibility of negative result due to
precision loss.
Example:
Obviously, (3*2^52+2) is congruent to 2 modulo 3.
Let's test it on Lua 5.2 and 5.3

C:\>lua52.exe -e"local a = 3*2^52+2; local b = 3; print(a%b)"
-2

C:\>lua53.exe -e"local a = 3*2^52+2; local b = 3; print(a%b)"
2.0

Users are expecting (N % M) to be between 0 and (M-1).
Correct modulo operation is not very simple, that's why fmod() is expensive.
Egor Skriptunoff
2018-08-26 03:29:37 UTC
Permalink
Post by Egor Skriptunoff
Correct modulo operation is not very simple
I should clarify what is the "correct modulo".

Rounding errors are inevitable, so instead of exact result
a%b
you are allowed to return
approx( approx(a) % approx(b) )
Each approx() might modify low bits (bits beyond FP precision).
You are not allowed to return arbitrary garbage (such as negative value for
positive b).

In other words, you must return a value which belongs to possible result
range according to interval arithmetic.

Of course, the same requirement applies to all math operations and
functions, not only modulo.
Dirk Laurie
2018-08-26 11:05:40 UTC
Permalink
Op So., 26 Aug. 2018 om 05:29 het Egor Skriptunoff
Post by Egor Skriptunoff
Post by Egor Skriptunoff
Correct modulo operation is not very simple
I should clarify what is the "correct modulo".
Rounding errors are inevitable, so instead of exact result
a%b
you are allowed to return
approx( approx(a) % approx(b) )
Each approx() might modify low bits (bits beyond FP precision).
You are not allowed to return arbitrary garbage (such as negative value for positive b).
In other words, you must return a value which belongs to possible result range according to interval arithmetic.
Of course, the same requirement applies to all math operations and functions, not only modulo.
This seems to imply that it is perfectly OK to return any value from 0
to 4566 for 0x1p1000%4567, instead of the correct value 4567 that Lua
5.3 returns. Correct-shmorrect, I say.
Egor Skriptunoff
2018-08-26 12:51:11 UTC
Permalink
the correct value ... that Lua 5.3 returns.
Indeed, fmod() is doing its job very well.
IMO, fmod() does worth its performance price.
Roberto Ierusalimschy
2018-08-26 20:18:58 UTC
Permalink
Post by Egor Skriptunoff
the correct value ... that Lua 5.3 returns.
Indeed, fmod() is doing its job very well.
IMO, fmod() does worth its performance price.
Going down all the way, the multiplication used in 5.3 to test
the sign of the result may underflow:

$ Lua 5.3.2 Copyright (C) 1994-2015 Lua.org, PUC-Rio
Post by Egor Skriptunoff
-2.1^-1000 % 2^-1000
-5.928787750095e-323

The result should be non negative, in the range [0,2^-1000). It is
better to use explicit tests:

/* m is not zero and m and b have different signs */
if ((m) < 0 && ((m) < 0) != ((b) < 0)) (m) += (b);

-- Roberto
Roberto Ierusalimschy
2018-08-26 20:26:16 UTC
Permalink
Post by Roberto Ierusalimschy
/* m is not zero and m and b have different signs */
if ((m) < 0 && ((m) < 0) != ((b) < 0)) (m) += (b);
if ((m) != 0 && ((m) < 0) != ((b) < 0)) (m) += (b);

-- Roberto
Albert Chan
2018-08-26 22:58:48 UTC
Permalink
Post by Roberto Ierusalimschy
Going down all the way, the multiplication used in 5.3 to test
$ Lua 5.3.2 Copyright (C) 1994-2015 Lua.org, PUC-Rio
Post by Egor Skriptunoff
-2.1^-1000 % 2^-1000
-5.928787750095e-323
The result should be non negative, in the range [0,2^-1000).
-- Roberto
Good catch !

FYI, even with underflow, the sign bit is still there.

double a = -6e-323, b = 0x1p-1000, c = a*b;
printf("c = %g, sign_bit = %#x\n", c, signbit(c));

c = 0, sign_bit = 0x200

However, signbit test is too good, fmod test like this:

if (signbit((m)*(b)) && (m) != 0) (m) += (b);
Albert Chan
2018-08-27 13:35:07 UTC
Permalink
the correct value ... that Lua 5.3 returns.
Indeed, fmod() is doing its job very well.
IMO, fmod() does worth its performance price.
Depends on the application, there might not be a price.

My primepi.lua, which count numbers of primes <= n, uses lots of mod's

https://github.com/achan001/PrimePi

For prime.pi(10^12), it called mod 16,055,541 times.

On my old Pentium 3, LuaJIT 1.1.8 were using fmod to implenent %.
Just to be sure, I edit away the %, replacing with fmod:

luajit -O primepi.lua 1e12 => 37607912018, take 54.8 seconds

Recompile LuaJIT using Lua 5.1 definition of mod for math.fmod,
i.e. ((a) - floor((a)/(b))*(b)), it run slower (64.0 seconds) !?

My guess is if a and b are not extremely far off, fmod is faster,
because it avoided the expensive division, using subtractions instead.

https://eli.thegreenplace.net/2018/computing-remainders-by-doubling/
Albert Chan
2018-08-26 13:10:51 UTC
Permalink
Post by Dirk Laurie
This seems to imply that it is perfectly OK to return any value from 0
to 4566 for 0x1p1000%4567, instead of the correct value 4567 that Lua
5.3 returns. Correct-shmorrect, I say.
+1 to Dirk

Garbage is garbage, even if it is not "arbitrary" garbage.
In fact, I prefer the "bad" garbage, as a heads-up, this is bad ...

Why fix a good % ?
Dirk Laurie
2018-08-26 13:15:41 UTC
Permalink
Post by Dirk Laurie
This seems to imply that it is perfectly OK to return any value from 0
to 4566 for 0x1p1000%4567, instead of the correct value 4567 that Lua
5.3 returns. Correct-shmorrect, I say.
Meh. Cut-and-pasted the wrong number. The correct value is 4335.
Dirk Laurie
2018-08-25 07:31:03 UTC
Permalink
Could the 'for' error messages be like the argument checks?
E.g
bad 'for' initial value: expected number, got string
instead of
'for' initial value must be a number
Op Ma., 18 Jun. 2018 om 22:33 het Luiz Henrique de Figueiredo
Post by Luiz Henrique de Figueiredo
Lua 5.4.0 (work2) is now available for testing at
http://www.lua.org/work/lua-5.4.0-work2.tar.gz
The checksums are
MD5 3cdf2a4eb84dde6b6aaf5d2d1de07be9 -
SHA1 e8484e61c5c338e3ec2f75dbe0f6703d079fecf9 -
This is a work version. All details may change in the final version.
An updated reference manual is included and also available at
http://www.lua.org/work/doc
The complete diffs from work1 to work2 are available at
http://www.lua.org/work/diffs-lua-5.4.0-work1-work2.html
http://www.lua.org/work/diffu-lua-5.4.0-work1-work2.html
Previous work versions are available at
http://www.lua.org/work/
The focus of Lua 5.4.0 is performance. We're especially interested in
feedback about its performance but all feedback is welcome. Thanks.
--lhf
Egor Skriptunoff
2018-09-20 19:50:43 UTC
Permalink
Post by Luiz Henrique de Figueiredo
Lua 5.4.0 (work2) is now available for testing
A warning is displayed while building Lua 5.4 under MinGW64:

gcc -std=gnu99 -O2 -Wall -Wextra -DLUA_COMPAT_5_3 -DLUA_BUILD_AS_DLL -c
-o loadlib.o loadlib.c
loadlib.c: In function 'lsys_sym':
loadlib.c:209:21: warning: cast between incompatible function types from
'FARPROC' {aka 'long long int (*)()'} to 'int (*)(lua_State *)' {aka 'int
(*)(struct lua_State *)'} [-Wcast-function-type]
lua_CFunction f = (lua_CFunction)GetProcAddress((HMODULE)lib, sym);
^
Egor Skriptunoff
2018-09-21 22:45:45 UTC
Permalink
A warning is displayed while building Lua 5.4 on Debian 9.4 x86

ar rcu liblua.a lapi.o lcode.o lctype.o ldebug.o ldo.o ldump.o lfunc.o
lgc.o llex.o lmem.o lobject.o lopcodes.o lparser.o lstate.o lstring.o
ltable.o ltm.o lundump.o lvm.o lzio.o lauxlib.o lbaselib.o lcorolib.o
ldblib.o liolib.o lmathlib.o loadlib.o loslib.o lstrlib.o ltablib.o
lutf8lib.o linit.o
ar: `u' modifier ignored since `D' is the default (see `U')
Ahmed Charles
2018-10-14 08:27:26 UTC
Permalink
Post by Luiz Henrique de Figueiredo
Lua 5.4.0 (work2) is now available for testing at
http://www.lua.org/work/lua-5.4.0-work2.tar.gz
The checksums are
MD5 3cdf2a4eb84dde6b6aaf5d2d1de07be9 -
SHA1 e8484e61c5c338e3ec2f75dbe0f6703d079fecf9 -
This is a work version. All details may change in the final version.
An updated reference manual is included and also available at
http://www.lua.org/work/doc
The complete diffs from work1 to work2 are available at
http://www.lua.org/work/diffs-lua-5.4.0-work1-work2.html
http://www.lua.org/work/diffu-lua-5.4.0-work1-work2.html
Previous work versions are available at
http://www.lua.org/work/
The focus of Lua 5.4.0 is performance. We're especially interested in
feedback about its performance but all feedback is welcome. Thanks.
--lhf
From lmem.c:

void luaM_free_ (lua_State *L, void *block, size_t osize) {
global_State *g = G(L);
lua_assert((block == 0) == (block == NULL));
(*g->frealloc)(g->ud, block, osize, 0);
g->GCdebt -= osize;
}

The lua_assert should (probably) be:

lua_assert((osize == 0) == (block == N
Roberto Ierusalimschy
2018-10-23 15:56:38 UTC
Permalink
Post by Ahmed Charles
void luaM_free_ (lua_State *L, void *block, size_t osize) {
global_State *g = G(L);
lua_assert((block == 0) == (block == NULL));
(*g->frealloc)(g->ud, block, osize, 0);
g->GCdebt -= osize;
}
lua_assert((osize == 0) == (block == NULL));
Sure. Many thanks,

-- Roberto
Egor Skriptunoff
2018-11-15 20:49:15 UTC
Permalink
Post by Luiz Henrique de Figueiredo
Lua 5.4.0 (work2) is now available for testing
The manual says:
for v = e1, e2, e3 do block end
is equivalent to the code:
do
local var, limit, step = tonumber(e1), tonumber(e2), tonumber(e3)
...


If tonumber("1") is integer, why does loop variable contain float value?
Post by Luiz Henrique de Figueiredo
tonumber("1")
1
Post by Luiz Henrique de Figueiredo
for j = "1", 2 do print(j) end
1.0
2.0
Dibyendu Majumdar
2018-11-15 21:01:55 UTC
Permalink
On Thu, 15 Nov 2018 at 20:49, Egor Skriptunoff
Post by Egor Skriptunoff
Post by Luiz Henrique de Figueiredo
Lua 5.4.0 (work2) is now available for testing
for v = e1, e2, e3 do block end
do
local var, limit, step = tonumber(e1), tonumber(e2), tonumber(e3)
...
If tonumber("1") is integer, why does loop variable contain float value?
Post by Luiz Henrique de Figueiredo
tonumber("1")
1
Post by Luiz Henrique de Figueiredo
for j = "1", 2 do print(j) end
1.0
2.0
I think this option might help:

/*
@@ LUA_COMPAT_FLOATSTRING makes Lua format integral floats without a
@@ a float mark ('.0').
** This macro is not on by default even in compatibility mode,
** because this is not really an incompatibility.
*/
/* #define LUA_COMPAT_FLOATSTRING */
Philippe Verdy
2018-11-16 18:06:09 UTC
Permalink
the loop variable is neither an "integer" or a "float", it is just a Lua
"number".
When you "print" the number, it may display ".0" decimals or not, which
makes you think that there's a difference, but the actual number is the
same. There's no segregation of types (such segregation isonly done
internally by the implementation of the VM which may infer some better
datatypes, if it correctly checks the value range and needed precision).
In a "numeric for loop", it's true that the control variable may be
internally compiled as an integer, but only if:
- the start and limit values are integers (no decimals)
- the start and end values fall into the value range of integers
- the needed precision allows storing the difference between each step from
start to end values (i.e. start + step > start and end - step < end, when
these expressions are computed in the chosen "integer" datatype)
Otherwise the control varaible will still need to be able to represent the
whole range and precision of Lua "numbers" (i.e. a 32-float or double,
depening on which datatype the impelmentation uses, but most Lua
implementations should use a 64-bit IEEE double precision number to
represent Lua "numbers".
One "bad" thing about Lua, is that the value range and precision of Lua
"numbers" is not strictly defined (this is the same problem as for "int",
"flaot" and "double" datatypes in C/C++, which are implementation
dependant: that's why POSIC defined a standard library of common
constrained numeric types), so there's a portability problem. Such thing
does not happen in Java and Javascript where numeric datatypes are strictly
defined (except that in Java, Javascript, C, C++ and even Lua, there's
still varability about "strict" computing modes and rounding modes which
can be tweaked locally, by some hints, which are not always mandatory...
except in Java and Javascrip,t where "strict" numeric evaluation is
enforcable by the language itself). Lua also does not formally define the
allowed values for infinites, Nans, denormals, or sign-ness of zeroes
I would really suggest that Lua is extended to define a standard library of
strictly defined numeric types (strict value ranges and precision, strict
evaluation, rounding modes) to allow creating portable programs (even if
this is at the price of speed). Lua for now is (like C/C++) tweaked in
favor of performance, not necessarily portability (and semantics): each
implementation then defines its own instance of the language, they are not
really /stricto sensu/ the same language if each instance has its own
limits. Unfortunately there's no simple way for a Lua program to safely
detect these limits.
This has a consequence: it's not so simple to emulate in Lua an other
environment like it is possible in Java or Javascript (e.g. with
WebAssembly which allows runninig a full Linux OS inside a 100% pure
Javascript program...): you need to specify the supported platforms (i.e.
specific instances of Lua, i.e. the set of supported Lua VMs).
I bet that Lua will later defined such standard library, at which time it
will be able to compete with Javascript, and will effectively be able to
really implement Javascript itself in a Lua program, or implement other
languages like Python, PHP, C/C++ and various kinds of "emulators") and it
will easier to port programs so that they'll run unchanged on other
architectures (making use of GPUs/APUs/VPUs or working over a computing
grid over a network with Lua objects distributed in various hosts, but Lua
will need other concepts, notably "security contexts" like in Java, or in
web protocols, or those standardized now in ECMAScript for Javascript)
Post by Egor Skriptunoff
Post by Luiz Henrique de Figueiredo
Lua 5.4.0 (work2) is now available for testing
for v = e1, e2, e3 do block end
do
local var, limit, step = tonumber(e1), tonumber(e2), tonumber(e3)
...
If tonumber("1") is integer, why does loop variable contain float value?
Post by Luiz Henrique de Figueiredo
tonumber("1")
1
Post by Luiz Henrique de Figueiredo
for j = "1", 2 do print(j) end
1.0
2.0
Sean Conner
2018-11-16 21:37:48 UTC
Permalink
Post by Philippe Verdy
One "bad" thing about Lua, is that the value range and precision of Lua
"numbers" is not strictly defined (this is the same problem as for "int",
"flaot" and "double" datatypes in C/C++, which are implementation
dependant: that's why POSIC defined a standard library of common
constrained numeric types), so there's a portability problem. Such thing
does not happen in Java and Javascript where numeric datatypes are strictly
defined (except that in Java, Javascript, C, C++ and even Lua, there's
still varability about "strict" computing modes and rounding modes which
can be tweaked locally, by some hints, which are not always mandatory...
except in Java and Javascrip,t where "strict" numeric evaluation is
enforcable by the language itself).
Could you point out in the relevant standatds for Java and Javascript
where it says this? I can accept it for Java, but I'm dubious about
Javascript, which uses floating point as the basis for all numeric
operations (like Lua 5.2 and earlier).
Post by Philippe Verdy
Lua also does not formally define the
allowed values for infinites, Nans, denormals, or sign-ness of zeroes
Um ... what? IEEE-754 (which is what most modern floating point is based
on) does define infinities (two, positive and negative), NaNs (only to say
there are two types---quiet and signaling; it says nothing else about then
except that NaN != NaN), supports denormals and gives two zeros (0 and -0).
If you mean for Lua to state "must support IEEE-754" then say so. But Lua
is based upon what C gives, and what C gives is not IEEE-754 (or at least,
does not mandate its use).
Post by Philippe Verdy
I would really suggest that Lua is extended to define a standard library of
strictly defined numeric types (strict value ranges and precision, strict
evaluation, rounding modes) to allow creating portable programs (even if
this is at the price of speed). Lua for now is (like C/C++) tweaked in
favor of performance, not necessarily portability (and semantics): each
implementation then defines its own instance of the language, they are not
really /stricto sensu/ the same language if each instance has its own
limits. Unfortunately there's no simple way for a Lua program to safely
detect these limits.
I have not noticed any differences between PUC Lua (5.1) and LuaJIT. I
have also not noticed any differences between x86-32 bit version of PUC Lua,
x86-64 bit version of PUC Lua, SPARC 32-bit version of PUC Lua, and SPARC
64-bit version of PUC Lua. All have behaved (and current, are behaving) the
same.
Post by Philippe Verdy
This has a consequence: it's not so simple to emulate in Lua an other
environment like it is possible in Java or Javascript (e.g. with
WebAssembly which allows runninig a full Linux OS inside a 100% pure
If WebAssembly can run Linux, then it certainly can run Lua, since it's
written in C, much like Linux.
Post by Philippe Verdy
you need to specify the supported platforms (i.e.
specific instances of Lua, i.e. the set of supported Lua VMs).
I bet that Lua will later defined such standard library, at which time it
will be able to compete with Javascript, and will effectively be able to
really implement Javascript itself in a Lua program, or implement other
languages like Python, PHP, C/C++ and various kinds of "emulators") and it
will easier to port programs so that they'll run unchanged on other
architectures (making use of GPUs/APUs/VPUs or working over a computing
grid over a network with Lua objects distributed in various hosts, but Lua
will need other concepts, notably "security contexts" like in Java, or in
web protocols, or those standardized now in ECMAScript for Javascript)
I'm not sure what you mean. I have myself written a computer emulator in
both C and Lua [1] and both work the same [2].

-spc (What next? That C needs fixing to emulate Javascript?)

[1] Based around the Motorola 6809. I also emulate the 6840 (timer
chip), 6850 (ACIA chip), 6843 (floppy controller) and 6855 (DMA
controller). Mostly for my own amusement.

And yes, I have two versions, one in C, one in Lua.

[2] Nearly the same---the interface is much nicer on the Lua version as
it's easier to manage than in C. But the emulation works
identically, as I can run the same 6809 program on both and get the
same results.
Philippe Verdy
2018-11-16 22:49:56 UTC
Permalink
Post by Philippe Verdy
Post by Philippe Verdy
One "bad" thing about Lua, is that the value range and precision of Lua
"numbers" is not strictly defined (this is the same problem as for "int",
"flaot" and "double" datatypes in C/C++, which are implementation
dependant: that's why POSIC defined a standard library of common
constrained numeric types), so there's a portability problem. Such thing
does not happen in Java and Javascript where numeric datatypes are
strictly
Post by Philippe Verdy
defined (except that in Java, Javascript, C, C++ and even Lua, there's
still varability about "strict" computing modes and rounding modes which
can be tweaked locally, by some hints, which are not always mandatory...
except in Java and Javascrip,t where "strict" numeric evaluation is
enforcable by the language itself).
Could you point out in the relevant standatds for Java and Javascript
where it says this? I can accept it for Java, but I'm dubious about
Javascript, which uses floating point as the basis for all numeric
operations (like Lua 5.2 and earlier).
Javascript also has a variation well documented for its numeric datatype
(yes for now it's limited to floating points, a subset of IEEE 64-bit
"doubles", but it does not require everywhere the enforcement of rounding
modes, or normalization of numbers (this is done only in functions starting
by a pseudo-instruction, a string constant used as a directive which
indicates it must run under "strict" evaluation mode).
Post by Philippe Verdy
Post by Philippe Verdy
Lua also does not formally define the
allowed values for infinites, Nans, denormals, or sign-ness of zeroes
Um ... what? IEEE-754 (which is what most modern floating point is based
on) does define infinities (two, positive and negative), NaNs (only to say
there are two types---quiet and signaling; it says nothing else about then
except that NaN != NaN), supports denormals and gives two zeros (0 and -0).
If you mean for Lua to state "must support IEEE-754" then say so. But Lua
is based upon what C gives, and what C gives is not IEEE-754 (or at least,
does not mandate its use).
You're repeating exactly what I said: this is not defined formally in Lua,
but by the implementation (for *example*, what C gives, but C is not a
requirement of the language for implementing it).
Post by Philippe Verdy
Post by Philippe Verdy
I would really suggest that Lua is extended to define a standard library
of
Post by Philippe Verdy
strictly defined numeric types (strict value ranges and precision, strict
evaluation, rounding modes) to allow creating portable programs (even if
this is at the price of speed). Lua for now is (like C/C++) tweaked in
favor of performance, not necessarily portability (and semantics): each
implementation then defines its own instance of the language, they are
not
Post by Philippe Verdy
really /stricto sensu/ the same language if each instance has its own
limits. Unfortunately there's no simple way for a Lua program to safely
detect these limits.
I have not noticed any differences between PUC Lua (5.1) and LuaJIT. I
have also not noticed any differences between x86-32 bit version of PUC Lua,
x86-64 bit version of PUC Lua, SPARC 32-bit version of PUC Lua, and SPARC
64-bit version of PUC Lua. All have behaved (and current, are behaving) the
same.
That's not true, you give examples where Lua is used as a standalone
application or embedded using the C library. There are other use cases
where the "ABI" in C is not used at all and Lua is integrated with
something else.
Post by Philippe Verdy
This has a consequence: it's not so simple to emulate in Lua an other
Post by Philippe Verdy
environment like it is possible in Java or Javascript (e.g. with
WebAssembly which allows runninig a full Linux OS inside a 100% pure
If WebAssembly can run Linux, then it certainly can run Lua, since it's
written in C, much like Linux.
You consider the exact reverse problem: yes, WebAssembly can certinaly run
Lua, but I doubt that Lua can safely (and portably) run WebAssembly this is
what I wrote) without breaking, except with specific implemantations of Lua.
Post by Philippe Verdy
you need to specify the supported platforms (i.e.
Post by Philippe Verdy
specific instances of Lua, i.e. the set of supported Lua VMs).
I bet that Lua will later defined such standard library, at which time it
will be able to compete with Javascript, and will effectively be able to
really implement Javascript itself in a Lua program, or implement other
languages like Python, PHP, C/C++ and various kinds of "emulators") and
it
Post by Philippe Verdy
will easier to port programs so that they'll run unchanged on other
architectures (making use of GPUs/APUs/VPUs or working over a computing
grid over a network with Lua objects distributed in various hosts, but
Lua
Post by Philippe Verdy
will need other concepts, notably "security contexts" like in Java, or in
web protocols, or those standardized now in ECMAScript for Javascript)
I'm not sure what you mean. I have myself written a computer emulator in
both C and Lua [1] and both work the same [2].
I've also written several emulators, interpretors, compilers since many
years.

My first complete one was done in 1989, then extended in a period up to
2000; and it was really used in actual production and sold, it was for a
data reporting tool, then for a printer emulator, then a Postscript
emulator engine, then ported as a printer driver embedded in the
application, then it was used to emulate terminal protocols as well, with
fully customizable appearances driven by user scripts which were desigend
by users, compiled, stored and reused to generate various reports on very
voluminous data sets (also customizable coming from multiple RDBMS systems
or database file formats or live datasources, with data queries/filters
integrated as well in the user scripts; the application of these were for
example used in billing, accounting, financial analysis, commercial
analysis, ressource management, statistic analysis, scientific data
agregation, measurements). Note only the languages developed was for
processing of incoming data flows, but also for adaptation of results to
various kinds of outputs (not just displays, also printers, or databases,
to be used as a new data source, that could be queried again). The language
was then entirely dedicated to data transforms. It was very fast and
avoided writing many specific programs, and was ported on many systems
(mainframes, MVS, VMS, various flavors of Unix, and even on old versions of
Windows, it was also running in MSDOS or similar, notably in industrial
environements and later rapidly on NT and OS/2.) It did not require any
multiprocessing OS (it was written in C only, not any piece of C++ and care
was taken to make it as portable as possible, using basic POSIX
capabilities and if needed emulating them when they were missing in the
native OS) it was integrable as well in several RDBMS (e.g. via Oracle
PL/SQL, or Sybase/MSSQL stored procedures, or Informix procedures, and much
later on MySQL) and even emulated several SQL variants (rewriting SQL
queries when needed).

The language also had a visual programming interface (not requiring users
to write code: the visual designer generated the code itself from a schema
drawn like in Powerpoint, with boxes, arrows which were freeely movable
with a mouse on a blank design sheet; however I did not develop myself this
visual interface, but adapted the language to cover some of its needs by
adding some syntaxic features). The whole tool contained several language
parsers and generators some dedicated to process input, some to describe
these inputs and filter them, other dedicated to process outputs, filter
them or give them a presentation, but they were all based on a set of
declarations (avoiding imperative ordered instructions) focusing specific
parts of the input dataset, evaluating conditions triggering an execution,
and then acting as filters to produce one or more output. The "ordered
instructions" were voluntarily very limited (there was no need to define
any branch or control of execution, not even a single loop). This was why
it was possible to create a visual designer where time/sequencing was
irrrelevant but all was descriptive and modifiable locally. It was the tool
that finally determined itself the sequencing of actions and produced most
of the transforms needed, evaluating itself all the described conditions.

The complete tool also included a customizable scheduler (based on a
database scheduler and/or a native OS scheduler). My colleagues used it to
design a facet-oriented OO language, I helped them to define a compiler for
it, which could be compiled to native C++ standalone applications working
on the X11 environment and later on Windows (at that time there was still
no standard graphic environment with features-rich sets of visual
components for the UI).

-spc (What next? That C needs fixing to emulate Javascript?)
No I've not said your last statement. Here also you consider the inverse
problem. So this is only what said "the Great Sean Conner"...
William Ahern
2018-11-16 23:46:51 UTC
Permalink
<snip>
Post by Philippe Verdy
Post by Sean Conner
Um ... what? IEEE-754 (which is what most modern floating point is based
on) does define infinities (two, positive and negative), NaNs (only to say
there are two types---quiet and signaling; it says nothing else about then
except that NaN != NaN), supports denormals and gives two zeros (0 and -0).
If you mean for Lua to state "must support IEEE-754" then say so. But Lua
is based upon what C gives, and what C gives is not IEEE-754 (or at least,
does not mandate its use).
You're repeating exactly what I said: this is not defined formally in Lua,
but by the implementation (for *example*, what C gives, but C is not a
requirement of the language for implementing it).
<snip>
Post by Philippe Verdy
Post by Sean Conner
If WebAssembly can run Linux, then it certainly can run Lua, since it's
written in C, much like Linux.
You consider the exact reverse problem: yes, WebAssembly can certinaly run
Lua, but I doubt that Lua can safely (and portably) run WebAssembly this is
what I wrote) without breaking, except with specific implemantations of Lua.
The ease of implementing WebAssembly in Lua would be little different than
implementing it in C on the same platform. If the native C environment used
IEEE 64-bit doubles then that makes implementing WebAssemly semantics easy
in both C and a stock Lua build. If it doesn't then it will be more
difficult in either case. (Even if you can utilize _Float64 in C you still
need to deal with promotion and similar headaches that require careful
treatment.)

And don't forget, WebAssembly doesn't provide 64-bit integers so
implementing Lua (>= 5.3) arithemtic directly in WebAssembly necessarily
requires synthesizing the type. Also, WebAssembly doesn't provide a goto
construct, so implementing continuation-like patterns (e.g. jump tables) is
a headache. Compiling Go to WebAssembly requires horribly inefficient hacks
for this reason (see
https://docs.google.com/document/d/131vjr4DH6JFnb-blm_uRdaC0_Nv3OUwjEY5qVCxCup4/preview#).
By contrast, Lua 5.2 added goto precisely for the convenience of code
generators.

The fact that you can surely compile PUC Lua's C source to WebAssembly using
existing C compilers that target WebAssembly can't be compared to the ease
of hosting WebAssembly directly in Lua. The fair comparison would be
compiling a C-based WebAssembly VM to Emscripten-like Lua code using a
preexisting toolchain. In which case all these points are moot as they'd
already be solved in *both* directions.
Philippe Verdy
2018-11-17 00:16:28 UTC
Permalink
Post by William Ahern
<snip>
Post by Philippe Verdy
Post by Sean Conner
Um ... what? IEEE-754 (which is what most modern floating point is
based
Post by Philippe Verdy
Post by Sean Conner
on) does define infinities (two, positive and negative), NaNs (only to
say
Post by Philippe Verdy
Post by Sean Conner
there are two types---quiet and signaling; it says nothing else about
then
Post by Philippe Verdy
Post by Sean Conner
except that NaN != NaN), supports denormals and gives two zeros (0 and -0).
If you mean for Lua to state "must support IEEE-754" then say so. But
Lua
Post by Philippe Verdy
Post by Sean Conner
is based upon what C gives, and what C gives is not IEEE-754 (or at
least,
Post by Philippe Verdy
Post by Sean Conner
does not mandate its use).
You're repeating exactly what I said: this is not defined formally in
Lua,
Post by Philippe Verdy
but by the implementation (for *example*, what C gives, but C is not a
requirement of the language for implementing it).
<snip>
Post by Philippe Verdy
Post by Sean Conner
If WebAssembly can run Linux, then it certainly can run Lua, since
it's
Post by Philippe Verdy
Post by Sean Conner
written in C, much like Linux.
You consider the exact reverse problem: yes, WebAssembly can certinaly
run
Post by Philippe Verdy
Lua, but I doubt that Lua can safely (and portably) run WebAssembly this
is
Post by Philippe Verdy
what I wrote) without breaking, except with specific implemantations of
Lua.
The ease of implementing WebAssembly in Lua would be little different than
implementing it in C on the same platform.
As I said, this depends if the "C" environment is available, and this is
not the case for all Lua environments where there's NO access at all to it
(blocked completely by security restrictions: and this is the goal of
WebAssembly to avoid this barrier by emulating a stable and predictable
environment in which all limits are known).

Even if it's available, your Lua program will depend on the implementation
and limits of numbers inb this C environment, making it then difficult to
support WebAssembly reliably and portably, without first fixing the C
environment (which is notoriously difficult to make portable for its
partially defined and too permissive datatype system):

You need the modern support of "stdtype" (initially from the ANSI standard,
or at least C99 and later) defining basic native types more strictly than
what basic C types provide.

I know that most modern C environments now have this "stdtype" library; but
using it (and only it) has a known performance cost, and it requires
discipline of C programmers (i.e. not using the "int" datatype at all, and
not even "char", "float" and "double" without an overlay reimplementing if
needed the standard types and the standard artithmetic and mathemetic
operations); or it requires a C compiler that will enforce these limits
using a more precise set of limits for all basic C types (char, short, int,
long, long long, and their signed/unsigned variants, plus bool, float and
double) and defining stricter semantics for or for evaluation order and
allowed "aliasings" caused by pointers and references, as well as exception
handlers (this may also require using specific versions of the standard C
library compiled with these stricter rules).
Sean Conner
2018-11-17 01:16:09 UTC
Permalink
Post by Philippe Verdy
You need the modern support of "stdtype" (initially from the ANSI
standard, or at least C99 and later) defining basic native types more
strictly than what basic C types provide.
No. You'll need at least C89 and code like the following:

#include <limits.h>
#if CHAR_BIT == 8
typedef unsigned char uint8_t;
#else
# error Not a byte-addressible system (at least this is the assumption).
#endif

#if USHRT_MAX == 65536u
typedef unsighed short uint16_t;
#else
# error Can not compile with 16-bit types
#endif

#if UINT_MAX == 4294967295UL
typedef unsigned int uint32_t;
#elif ULONG_MAX == 4294967295UL
typedef unsigned long uint32_t;
#else
# error Can not compile with 32-bit types
#endif

#if ULONG_MAX == 18446744073709551615ULL
typedef unsigned long uint64_t;
#else
# if defined(ULLONG_MAX)
# if ULLONG_MAX == 18446744073709551615ULL
typedef unsigned long long uint64_t;
# else
# error Can not compile with 64-bit types
# endif
# else
# error No 64-bit support.
# endif
#endif

And so on ... Yes, annoying, but possible.
Post by Philippe Verdy
I know that most modern C environments now have this "stdtype" library; but
using it (and only it) has a known performance cost,
^^^^^^^^^^^^^^^^

Only if the exact width integer types don't exist; uint16_t, uint32_t and
uint64_t aren't required, and here, I'll cite my source---ISO/IEC 9899:TC3
(aka, C99), section 7.18.1.1:

3 These types are optional. However, if an implementation provides
integer types with widths of 8, 16, 32, or 64 bits, no padding bits,
and (for the signed types) that have a two’s complement
representation, it shall define the corresponding typedef names.

Required are int_least8_t, uint_least8_t, et. al. (7.18.1.2), and
int_fast8_t, uint_fast8_t et. al. (7.18.1.3).
Post by Philippe Verdy
and it requires
discipline of C programmers (i.e. not using the "int" datatype at all, and
not even "char", "float" and "double" without an overlay reimplementing if
needed the standard types and the standard artithmetic and mathemetic
operations); or it requires a C compiler that will enforce these limits
using a more precise set of limits for all basic C types (char, short, int,
long, long long, and their signed/unsigned variants, plus bool, float and
double) and defining stricter semantics for or for evaluation order and
^^^^^^^^^^^^^^^^^^^^
Post by Philippe Verdy
allowed "aliasings" caused by pointers and references, as well as exception
^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^
Post by Philippe Verdy
handlers (this may also require using specific versions of the standard C
^^^^^^^^
Post by Philippe Verdy
library compiled with these stricter rules).
[Citation needed]

-spc
Philippe Verdy
2018-11-17 13:20:14 UTC
Permalink
It was thus said that the "Great Sean Conner" reversed the statements that
I made only to contradict me with false inferences.
Sean Conner
2018-11-17 00:58:05 UTC
Permalink
Post by Philippe Verdy
Post by Sean Conner
-spc (What next? That C needs fixing to emulate Javascript?)
No I've not said your last statement. Here also you consider the inverse
problem. So this is only what said "the Great Sean Conner"...
That you need to fix Javascript to emulate C? I don't think Javascript
needs fixing---you can port C to darn near everything.

-spc (In fact, isn't Javascript written in C? How can that be?)
Philippe Verdy
2018-11-17 13:14:20 UTC
Permalink
Post by Sean Conner
Post by Philippe Verdy
Post by Sean Conner
-spc (What next? That C needs fixing to emulate Javascript?)
No I've not said your last statement. Here also you consider the inverse
problem. So this is only what said "the Great Sean Conner"...
That you need to fix Javascript to emulate C? I don't think Javascript
needs fixing---you can port C to darn near everything.
I repeat: I've never said that statement.
Post by Sean Conner
-spc (In fact, isn't Javascript written in C? How can that be?)
I repeat I've never said that statement.
Egor Skriptunoff
2018-12-08 07:42:11 UTC
Permalink
Post by Luiz Henrique de Figueiredo
Lua 5.4.0 (work2) is now available for testing at
http://www.lua.org/work/lua-5.4.0-work2.tar.gz
math.randomseed(0); for j=1,10 do print(math.random()) end

Unfortunately, this sequence of random numbers depends on number of binary
digits in the mantissa of a float datatype Lua was built with (despite of
the same seed!).
To fix it, "I2d(Rand64 x)" function should use the highest FIGS bits of x
(instead of the lowest FIGS bits).
I'm talking about this code in "lua-5.4.0-work2/src/lmathlib.c":

#define maskFIG (~(~(Rand64)1 << (FIGS - 1))) /* use FIGS bits */
#define shiftFIG (l_mathop(0.5) / ((Rand64)1 << (FIGS - 1))) /* 2^(-FIGS)
*/

static lua_Number I2d (Rand64 x) {
return (lua_Number)(x & maskFIG) * shiftFIG;
}

Loading...