[ANN] Ravi-Distro 0.5 alpha release

[Dibyendu Majumdar]

Ravi-Distro (https://github.com/dibyendumajumdar/ravi-distro) aims to
be a small and focused distribution of Lua 5.3 (www.lua.org) and Ravi
5.3 (https://github.com/dibyendumajumdar/ravi). In this release binary
distributions of Ravi and Lua 5.3 for Windows 64-bit are provided. The
packages included in this release are:

* lua 5.3.5
* Ravi with OMR JIT backend (updated!)
* luafilesystem
* lua-protobuf (new!)
* luasocket
* lpeglabel
* torch7
* torch7-nn
* torch7-optim
* torch7-autograd
* torch7-cephes
* penlight
* moses
* lua-cjson
* luaffi
* luv (libuv binding)
* nj (JIT engine based on Eclipse OMR)

Please visit the release page below for further information.

https://github.com/dibyendumajumdar/ravi-distro/releases

All feedback welcome!

Thanks and Regards
Dibyendu

[Egor Skriptunoff]

Post by Dibyendu Majumdar
In this release binary
distributions of Ravi and Lua 5.3 for Windows 64-bit are provided.
https://github.com/dibyendumajumdar/ravi-distro/releases
All feedback welcome!

I'm expecting to find ravi-distro-0.5-windows-64bit.zip, but this file is
missing.

[Dibyendu Majumdar]

On Tue, 13 Nov 2018 at 20:11, Egor Skriptunoff

Post by Dibyendu Majumdar
In this release binary
distributions of Ravi and Lua 5.3 for Windows 64-bit are provided.
https://github.com/dibyendumajumdar/ravi-distro/releases
All feedback welcome!

I'm expecting to find ravi-distro-0.5-windows-64bit.zip, but this file is missing.

Yes, apologies will upload tonight.

Thanks and Regards
Dibyendu

[Dibyendu Majumdar]

On Tue, 13 Nov 2018 at 20:11, Egor Skriptunoff

Post by Dibyendu Majumdar
In this release binary
distributions of Ravi and Lua 5.3 for Windows 64-bit are provided.
https://github.com/dibyendumajumdar/ravi-distro/releases
All feedback welcome!

I'm expecting to find ravi-distro-0.5-windows-64bit.zip, but this file is missing.

Hi, this is now available. The reason for the delay was that I found
that the release build was crashing a few tests. I believe the
failures are related to the general problem with stack unwinding when
longjmp is used in a Win64 process; it requires some special actions
by the compiler. However the JIT engines do not tend to set this up
properly. Please let me know if you have noticed such failures.

Thanks and Regards
Dibyendu

[Egor Skriptunoff]

Post by Egor Skriptunoff
I'm expecting to find ravi-distro-0.5-windows-64bit.zip, but this file

is missing.
Hi, this is now available.

OK, let's test it!

1)
A bug: correct Lua 5.3 program doesn't run on Ravi.
C:\Software\ravi\bin>ravi -e"for x = 1, 2 do x = 42.0 end"
ravi: (command line):1: Invalid assignment: integer expected near 'end'

2)
Ravi shows some warnings in JIT mode. (Nevertheless, the program works
correctly despite of the warnings)
buffer:2113:1: warning: constant 4023233417 is so big it is long long
buffer:2113:1: warning: decimal constant 4023233417 is between LONG_MAX
and ULONG_MAX. For C99 that means long long, C90 compilers are very likely
to produce unsigned long (and a warning) here
buffer:2115:1: warning: constant 2562383102 is so big it is long long
buffer:2115:1: warning: decimal constant 2562383102 is between LONG_MAX
and ULONG_MAX. For C99 that means long long, C90 compilers are very likely
to produce unsigned long (and a warning) here
buffer:2119:1: warning: constant 3285377520 is so big it is long long
buffer:2119:1: warning: decimal constant 3285377520 is between LONG_MAX
and ULONG_MAX. For C99 that means long long, C90 compilers are very likely
to produce unsigned long (and a warning) here
buffer:2190:1: warning: constant 4294967296 is so big it is long long
buffer:2370:1: warning: constant 4294967297 is so big it is long long
buffer:2946:1: warning: constant 1099511627776 is so big it is long long

3)
Most frustrating problem: compiled functions run slower than not compiled
(what am I doing wrong?).
Simple example (compiled function takes 28 seconds, non-compiled 13
seconds):

C:\Software\ravi\bin>type program.lua
local function f()
local tm = os.clock()
local o = 0
for j = 1, 1e4 do
local x = 0
for k = 1, 1e5 do
x = x ~ (j + k)
end
o = o | x
end
print(" Result = "..o)
print(" CPU time = "..(os.clock() - tm))
end
print"Benchmarking non-compiled function"
f()
print"Compiling the function"
assert(ravi.compile(f))
print"Benchmarking compiled function"
f()

C:\Software\ravi\bin>ravi program.lua
Benchmarking non-compiled function
Result = 114656
CPU time = 13.425
Compiling the function
Benchmarking compiled function
Result = 114656
CPU time = 28.568

[Dibyendu Majumdar]

On Wed, 14 Nov 2018 at 19:55, Egor Skriptunoff

Post by Egor Skriptunoff
OK, let's test it!
1)
A bug: correct Lua 5.3 program doesn't run on Ravi.
C:\Software\ravi\bin>ravi -e"for x = 1, 2 do x = 42.0 end"
ravi: (command line):1: Invalid assignment: integer expected near 'end'

In this case Ravi is deciding that x is an integer type and hence
doesn't like that you are assigning a floating point value.

Post by Egor Skriptunoff
2)
Ravi shows some warnings in JIT mode. (Nevertheless, the program works correctly despite of the warnings)
buffer:2113:1: warning: constant 4023233417 is so big it is long long
buffer:2113:1: warning: decimal constant 4023233417 is between LONG_MAX and ULONG_MAX. For C99 that means long long, C90 compilers are very likely to produce unsigned long (and a warning) here
buffer:2115:1: warning: constant 2562383102 is so big it is long long
buffer:2115:1: warning: decimal constant 2562383102 is between LONG_MAX and ULONG_MAX. For C99 that means long long, C90 compilers are very likely to produce unsigned long (and a warning) here
buffer:2119:1: warning: constant 3285377520 is so big it is long long
buffer:2119:1: warning: decimal constant 3285377520 is between LONG_MAX and ULONG_MAX. For C99 that means long long, C90 compilers are very likely to produce unsigned long (and a warning) here
buffer:2190:1: warning: constant 4294967296 is so big it is long long
buffer:2370:1: warning: constant 4294967297 is so big it is long long
buffer:2946:1: warning: constant 1099511627776 is so big it is long long

Yes sorry about that - this is coming from the C parser - I will
switch off this warning.

Post by Egor Skriptunoff
3)
Most frustrating problem: compiled functions run slower than not compiled (what am I doing wrong?).
C:\Software\ravi\bin>type program.lua
local function f()
local tm = os.clock()
local o = 0
for j = 1, 1e4 do
local x = 0
for k = 1, 1e5 do
x = x ~ (j + k)
end
o = o | x
end
print(" Result = "..o)
print(" CPU time = "..(os.clock() - tm))
end
print"Benchmarking non-compiled function"
f()
print"Compiling the function"
assert(ravi.compile(f))
print"Benchmarking compiled function"
f()
C:\Software\ravi\bin>ravi program.lua
Benchmarking non-compiled function
Result = 114656
CPU time = 13.425
Compiling the function
Benchmarking compiled function
Result = 114656
CPU time = 28.568

Okay - there are a few things here.
Firstly the JIT backend in ravi-distro is not inlining normal Lua
arithmetic / bitwise operators. Partly because I haven't got around to
doing this, and partly because I feel that inlining normal Lua ops
results in code bloat as each op has several branches.

I have changed your program as follows to help Ravi:

local function f()
local tm = os.clock()
local o: integer = 0
for j = 1, 10000 do
local x: integer = 0
for k = 1, 100000 do
x = x ~ (j + k)
end
o = o | x
end
print(" Result = "..o)
print(" CPU time = "..(os.clock() - tm))
end
print"Benchmarking non-compiled function"
f()
print"Compiling the function"
-- ravi.dumplua(f)
assert(ravi.compile(f))
print"Benchmarking compiled function"
f()

Timings on Mac OSX:

Original code interpreted: 9.063722
Above code interpreted: 6.832995
For some reason the above modified version fails to JIT compile with
the OMR JIT backend; I need to investigate this.
Using LLVM backend original: 6.995143
Using LLVM backend above: 0.230904

So once type annotations are used, the LLVM backend can really improve
performance. The OMR backend should too, except that it is failing to
compile the code for some reason.

I realise that you may not want to annotate your code with types ... I
can try to amend your SHA2 code with types for Ravi if you like. It
will be a good piece of benchmark code for Ravi.

Thank you very much for sharing your findings - it really helps.

Regards
Dibyendu

[Dibyendu Majumdar]

Post by Egor Skriptunoff
local function f()
local tm = os.clock()
local o: integer = 0
for j = 1, 10000 do
local x: integer = 0
for k = 1, 100000 do
x = x ~ (j + k)
end
o = o | x
end
print(" Result = "..o)
print(" CPU time = "..(os.clock() - tm))
end
For some reason the above modified version fails to JIT compile with
the OMR JIT backend; I need to investigate this.

Okay its because in the OMR backend (included in Ravi Distro) I
haven't yet implemented the JITing of bitwise Ravi extension opcodes.
The standard Lua bitwise opcodes are handled via function calls in the
OMR backend, so they degrade performance.
The LLVM backend does handle all of them, although the Lua opcodes do
not tend to optimize well due to excessive branching.

I will look at adding the bitwise opcodes to the OMR backend.

Regards
Dibyendu

[Egor Skriptunoff]

Post by Dibyendu Majumdar
I realise that you may not want to annotate your code with types ... I
can try to amend your SHA2 code with types for Ravi if you like. It
will be a good piece of benchmark code for Ravi.

Yes, I'd like you to amend SHA2 with new branch of implementation for Ravi.
:-)

[Philippe Verdy]

Post by Dibyendu Majumdar
On Wed, 14 Nov 2018 at 19:55, Egor Skriptunoff

Post by Egor Skriptunoff
OK, let's test it!
1)
A bug: correct Lua 5.3 program doesn't run on Ravi.
C:\Software\ravi\bin>ravi -e"for x = 1, 2 do x = 42.0 end"
ravi: (command line):1: Invalid assignment: integer expected near

'end'
In this case Ravi is deciding that x is an integer type and hence
doesn't like that you are assigning a floating point value.

That's because it attempts to optimize the integer loop unsafely: when
compiling x=42.0, which changes the loop variable, it forgets to evaluate
the loop condition (x<=42) which would normally imply that this assignment
to x causes a break to the loop; but as this x variable is local to the
loop and this assigned value is then not used after breaking, the
assignment has no other effect than breaking the loop, so this code would
be nearly like "for x = 1, 2 do break end" i.e. it will do nothing if
compiled properly; if the code is interpreted, the for loop will be
executed only once (local x=1 when entering the first loop which sets
x=42.0, and exiting immediately which deletes the x variable).
There is code where modifying the loop control variable is perfectly valid,
even if in general this is done by assigning a value of the same type.

A simple loop of the form "for x = a,b do... end" does not even require
that "a" and "b" be the same type, it just needs that they are comparable ;
the type assigned to "x" and incremented is the type of "a", not "b", the
loop will compare "x <= b" after incrementing "x" which is the type of "a"
(the type of the control variable "x" is not necessarily an "integer",
actually in Lua it is just a "number");
the optimization to use integers has to be checked: you can optimize the
compilation of "b" to an integer (if "a" is an integer) by compiling "for x
= a,b do... end" as "for x = a,int(ceil(b)) do... end", then by compiling
the inner assignment "x = 42.0" as "x=int(ceil(42.0)", i.e. "x=42".

But beware of the valid range of your optimized integers: Lua numbers have
a wider range and allows a "for x=a,b" control variable to be any "number";
given the precision needed for incrementations to be working, the "a", and
"b" numbers must be strictly between -2^53 and 2^53-1 (if "number" is
compiled as an 64-bit IEEE "double precision floatting point") or between
-2^24 and 2^24 (if "number" is compiled as an 32-bit IEEE "single precision
floatting point"), otherwise the loop will be infinite. If your optimized
code uses just a 32-bit integer, it may overflow when entering the loop or
before reaching the end of the loop, or the end value "b" of the loop could
be incorrectly truncated to be lower than "a" and the loop will never
execute at all!

Writing "for x = (a),(b) do (...) end" is only syntaxic sugar for: "do
local x = (a); while x < (b) do (...); x = x + 1 end end"

[Philippe Verdy]

Post by Philippe Verdy
Writing "for x = (a),(b) do (...) end" is only syntaxic sugar for: "do
local x = (a); while x < (b) do (...); x = x + 1 end end"

More exactly it is the equivalent of: "do local x, __end__ = (a), (b);
while x <= __end__ do (...); x = x + 1 end end"
because (b) is evaluated only once before the first checking of the
condition and then entering the first loop (there was also a missing "=" in
the condition)

A compiler may check the type of the value (a) assigned to the control
variable (x), if it is known (constant), but then it must:
* coerce the value (b) assigned to "__end__" to the same datatype using
ceiling (not floor);
* check that the coercion of (b) in "local x,__end__ = (a),coerce(b)" does
not cause an overflow (if this occurs, the loop cannot be optimized using
integers, it has to use a double floatting point for the control
variable...);
* coerce the incrementation "x = x + 1", as "x = successor(coerce(x))", if
(a) was an integer type, so that the condition "x <= __end__" remains
correct;
* check the possible integer overflow of this incrementation (if overflow
occurs, this must break the loop immediately: an overflow can occur here
only if "successor(__end__)" cannot be warrantied to be inside the valid
range of the chosen datatype, or if the current value of "x" which was
changed in the middle of the loop is outside the inclusive integer range
from (a) to (b));
* compile the code inside the loop so that any reference to "x" will
"uncoerce" its chosen integer type to the full range of "number" when
needed (notably when using the value of "x" in function calls, but as well
in simple expressions like "x*x"...)

A compiler may also want to unwind such loops if the code inside the loop
is small, and if the difference between (a) and (b) is known (both (a) and
(b) are constants) and small (generally not more than 4, but this may
depend on the code size inside the loop; this is complex to do if that Lua
code inside the loop contains other execution controls, requiring the
generation of labels for generated jumps into the generated opcodes, or if
the compiler makes special optimizations for exception handling, i.e. to
optimize invokations in Lua code to "pcall(function,...)").

[Philippe Verdy]

it should also be noted that the Lua specification states that "you should
never change the value of the control variable: The effect of such changes
is unpredictable. If you want to break a forloop before its normal
termination, use break."

What this means is that changing this value "may or may not" have a side
effect on the number of loops executed: a compiler may validly create an
iterator from (a) to (b) inclusively, and then the enumerator will return
values into the local variable "x", ignoring and overwriting any previously
assigned value inside the loop.

So the code "for x=1,3 do x=x+.5; print(x, ' ') end" may print either:
- "1.5 2.5 3.5" (using a prebuilt iterator and ignoring changes to the
control variable, i.e. using a stateless loop) or
- "1.5 3" (taking the current value of the control variable into account,
i.e. using a stateful loop).

As Lua is elusive about the expected behavior, the compiler should emit a
strong warning for such assignments to local control variables (I think
that compilers should behave the best using the first behavior using a
stateless loop).

My opinion is that Lua should have been defined so that only the first
(stateless) behavior is valid, and so "for x=(a),(b) do (...) end" should
always behave like "for x in foriter(a, b) do (...) end"

Post by Philippe Verdy

Post by Philippe Verdy
Writing "for x = (a),(b) do (...) end" is only syntaxic sugar for: "do
local x = (a); while x < (b) do (...); x = x + 1 end end"

More exactly it is the equivalent of: "do local x, __end__ = (a), (b);
while x <= __end__ do (...); x = x + 1 end end"
because (b) is evaluated only once before the first checking of the
condition and then entering the first loop (there was also a missing "=" in
the condition)
A compiler may check the type of the value (a) assigned to the control
* coerce the value (b) assigned to "__end__" to the same datatype using
ceiling (not floor);
* check that the coercion of (b) in "local x,__end__ = (a),coerce(b)" does
not cause an overflow (if this occurs, the loop cannot be optimized using
integers, it has to use a double floatting point for the control
variable...);
* coerce the incrementation "x = x + 1", as "x = successor(coerce(x))",
if (a) was an integer type, so that the condition "x <= __end__" remains
correct;
* check the possible integer overflow of this incrementation (if overflow
occurs, this must break the loop immediately: an overflow can occur here
only if "successor(__end__)" cannot be warrantied to be inside the valid
range of the chosen datatype, or if the current value of "x" which was
changed in the middle of the loop is outside the inclusive integer range
from (a) to (b));
* compile the code inside the loop so that any reference to "x" will
"uncoerce" its chosen integer type to the full range of "number" when
needed (notably when using the value of "x" in function calls, but as well
in simple expressions like "x*x"...)
A compiler may also want to unwind such loops if the code inside the loop
is small, and if the difference between (a) and (b) is known (both (a) and
(b) are constants) and small (generally not more than 4, but this may
depend on the code size inside the loop; this is complex to do if that Lua
code inside the loop contains other execution controls, requiring the
generation of labels for generated jumps into the generated opcodes, or if
the compiler makes special optimizations for exception handling, i.e. to
optimize invokations in Lua code to "pcall(function,...)").

[Dirk Laurie]

it should also be noted that the Lua specification states that "you should never change the value of the control variable: The effect of such changes is unpredictable. If you want to break a forloop before its normal termination, use break."

I cannot find this quotation in the Lua 5.3 Referebce Manual.

[Albert Chan]

My opinion is that Lua should have been defined so that only the first (stateless) behavior is valid, and so "for x=(a),(b) do (...) end" should always behave like "for x in foriter(a, b) do (...) end"

Does any version of Lua *not* do this way ?

[Philippe Verdy]

Not all, according to the Lua doc in section 4.3.4 (Numeric for), the
behavior is unspecified. I bet that the interpreter mode uses the 2nd
approach which is simpler, but may give different results.
This is stated at end of this page:
https://www.lua.org/pil/4.3.4.html

Post by Philippe Verdy

Post by Philippe Verdy
My opinion is that Lua should have been defined so that only the first

(stateless) behavior is valid, and so "for x=(a),(b) do (...) end" should
always behave like "for x in foriter(a, b) do (...) end"
Does any version of Lua *not* do this way ?

[Philippe Verdy]

What is even worse is that the same page states that " Second, the control
variable is a local variable automatically declared by the for statement
and is visible only inside the loop".
Which means that the local variable (visible inside the loop) and the
control variable (for the loop itself, which should be completely hidden in
the iterator, so that the local variable in the loop only gets a *copy* of
the value in the hidden control variable of the iterator) are confused in
that statement as if they were the same (which is an error in my opinion).

I don't know which approach is valid, the doc is clearly ambiguous.

The behavior should be specified clearly in the doc by showing, like in my
example:
for x=1,3 do x = x+0.5; print(x, ' ') end
that it MUST print "1.5 2.5 3.5", and NOT "1.5 3".

This stateless approach will then clearly ease the implementation of all
compilers and interpreters, with predictable results, and no need to track
in the source code of the inner loop if a "control variable" is assigned
(this should have no effect at all on the number of loops executed, so that
the number of loops is fully predictable before the first loop starts, and
only the "break" instruction can interrupt the loops before the iterator
reaches the ending condition).

As long as this behavior is unspecified, a compiler should emit a strong
warning, if ever it finds inside the loop an assignment to the "local
variable" declared by the "for" statement, which is not necessarily the
control variable of the hidden iterator.

I'm not sure that there's really an iterator object instantiated to contain
the control variable, the maximum value to compare with the control
variable, and the stepping value; the doc just indicates that there are two
hidden variables for the end value and the stepping value, and a compiler
may also want to avoid creating a real iterator object, and would prefer
just generating 1 hidden variables (or 2 with the stepping value), leaving
the control variable directly usable as the local variable of the loop).
With the stateless approach, a compiler would need to generate 2 hidden
variables (or 3 with the stepping value) separately from the local variable
visible inside the loop code).

When the stepping value is negative, the completion test is reversed: the
"end value" is a minimal value so the condition is "while __control__>=
__end__ do..." instead of ""while __control__<= __end__ do...", so there
are actually two kinds of numeric iterators generated (forward or backward).

The doc is not clear about the behavior of a numeric for loop, when the
optional stepping value is zero: "for i=1,10,0 do...end" cannot be
determined to be a forward or backward iterator, but logically this is an
infinite loop, given that the values in the two first parameters (1,10) are
both inclusive.

Post by Philippe Verdy
Not all, according to the Lua doc in section 4.3.4 (Numeric for), the
behavior is unspecified. I bet that the interpreter mode uses the 2nd
approach which is simpler, but may give different results.
https://www.lua.org/pil/4.3.4.html

Post by Philippe Verdy

Post by Philippe Verdy
My opinion is that Lua should have been defined so that only the first

(stateless) behavior is valid, and so "for x=(a),(b) do (...) end" should
always behave like "for x in foriter(a, b) do (...) end"
Does any version of Lua *not* do this way ?

[Francisco Olarte]

Philippe:

I'm a little lost in the discussion, due to he bottom quotes and
similars. You seem to complain of unspecified behaviour of numeric
for loops in PIL and friends, and...

Post by Philippe Verdy
What is even worse is that the same page states that " Second, the control
variable is a local variable automatically declared by the for statement and
is visible only inside the loop".
Which means that the local variable (visible inside the loop) and the
control variable (for the loop itself, which should be completely hidden in
the iterator, so that the local variable in the loop only gets a *copy* of
the value in the hidden control variable of the iterator) are confused in
that statement as if they were the same (which is an error in my opinion).
I don't know which approach is valid, the doc is clearly ambiguous.
The behavior should be specified clearly in the doc by showing, like in my
for x=1,3 do x = x+0.5; print(x, ' ') end
that it MUST print "1.5 2.5 3.5", and NOT "1.5 3".

I've read the ref manual ( which IMHO is the place to go for this kind
of things ) ( https://www.lua.org/manual/5.3/manual.html#3.3.5 )

and It gives some equivalent code which seems to clearly define both
the above-quote loop behaviour and the 0-increment case discussed
below.

Post by Philippe Verdy
This stateless approach will then clearly ease the implementation of all
compilers and interpreters, with predictable results, and no need to track
in the source code of the inner loop if a "control variable" is assigned
(this should have no effect at all on the number of loops executed, so that
the number of loops is fully predictable before the first loop starts, and
only the "break" instruction can interrupt the loops before the iterator
reaches the ending condition).

The code in ref manual for 5.3 seems to define exactly this behaviour.

Francisco Olarte

[Andrew Gierth]

Philippe> What is even worse is that the same page states

Why are you reading the online copy of PiL, which is for a long-obsolete
version of Lua, rather than the _actual documentation_?

https://www.lua.org/manual/5.3/manual.html#3.3.5

This makes it perfectly clear that:

- the actual loop iteration variables are hidden and not accessible

- any changes to the visible loop variable in the loop body have no
effect beyond the end of the current iteration

Philippe> The doc is not clear about the behavior of a numeric for
Philippe> loop, when the optional stepping value is zero

The actual manual (as linked above) is precisely clear about this.

[Philippe Verdy]

OK, but basic search on the Internet still refer to the old specification
which is on the Lua.org site itself. What was unspecified is now specified
since Lua 5.3 only (and I know various sites that still use Lua 5.1, and
several JIT compilers that did not use these clear statements for Lua 5.3,
which may now cause incompatible behavior with valid interpretations of the
former less precise specification).

Yes the new spec specifies that a zero step is now a *forward* iterator
only (this is an arbitrary choice of Lua 5.3, in both cases it will cause
either an infinite loop or no loop executed at all, and in both cases the
start value, which will be constant if loops are executed will only be
compared to the limit value: if start >= limit you have an infinite loop).

What is missing (but is implied) in the 5.3 manual is the case of a missing
step value; the sample code says
local ..., step = ..., tonumber(e3)
so it can return nil, then
if not (var and limit and step) then error() end
so a "nil" (unspecified) step would cause an error. The first statement of
the pseudo-code should be:
local ..., step = ..., (e3 == nil and 1 or tonumber(e3))

There's still nothing that indicates clearly that the default value of the
missing third expression in parameter of the numeric for loop is 1 (this
could still be a problem if an implementation chose to set it at -1 if
start > end, but it would break lot of codes assuming that code like:
for i=1,#t do ... --[==[ operation on t[i]... ]==] ... end
where the end expression "#t" could be the limit=0 (which is lower start=1)
and where it is normally expected that there would be no loop performed
(backward direction with an implied step=-1 should be explicitly
prohibited, in the numeric for version) That same loop could be written
using a generic loop (with an iterator):
for i in ipairs(t) do ... --[==[ operation on t[i]... ]==] ... end"
Such precision is present only further down in notes ("If the third
expression (the step) is absent, then a step of 1 is used."), but it should
be integrated in the pseudo-code, unless the following Lua code:
for i in 1,10,nil do ... end
which would generate a runtime error(), but not this one:
for i in 1,10 do ... end

Post by Andrew Gierth
Philippe> What is even worse is that the same page states
Why are you reading the online copy of PiL, which is for a long-obsolete
version of Lua, rather than the _actual documentation_?
https://www.lua.org/manual/5.3/manual.html#3.3.5
- the actual loop iteration variables are hidden and not accessible
- any changes to the visible loop variable in the loop body have no
effect beyond the end of the current iteration
Philippe> The doc is not clear about the behavior of a numeric for
Philippe> loop, when the optional stepping value is zero
The actual manual (as linked above) is precisely clear about this.
--
Andrew.

[Dibyendu Majumdar]

Post by Dibyendu Majumdar
On Wed, 14 Nov 2018 at 19:55, Egor Skriptunoff

Post by Egor Skriptunoff
OK, let's test it!
1)
A bug: correct Lua 5.3 program doesn't run on Ravi.
C:\Software\ravi\bin>ravi -e"for x = 1, 2 do x = 42.0 end"
ravi: (command line):1: Invalid assignment: integer expected near 'end'

In this case Ravi is deciding that x is an integer type and hence
doesn't like that you are assigning a floating point value.

That's because it attempts to optimize the integer loop unsafely: when compiling x=42.0, which changes the loop variable, it forgets to evaluate the loop condition (x<=42) which would normally imply that this assignment to x causes a break to the loop; but as this x variable is local to the loop and this assigned value is then not used after breaking, the assignment has no other effect than breaking the loop, so this code would be nearly like "for x = 1, 2 do break end" i.e. it will do nothing if compiled properly; if the code is interpreted, the for loop will be executed only once (local x=1 when entering the first loop which sets x=42.0, and exiting immediately which deletes the x variable).
There is code where modifying the loop control variable is perfectly valid, even if in general this is done by assigning a value of the same type.

Hi, to be honest the for num loop variable should be treated as
'readonly' inside the loop body as it is a copy of the real index
variable, and changing it inside the body doesn't affect the loop, and
is just bad practice in my view. Anyway I take the liberty of
optimising the for num index here, and require that the variable is
not assigned a non-integer value. I would forbid assignment but this
is harder to do.

Regards
Dibyendu

[Philippe Verdy]

Post by Dibyendu Majumdar
Hi, to be honest the for num loop variable should be treated as
'readonly' inside the loop body
as it is a copy of the real index
variable, and changing it inside the body doesn't affect the loop,

This means that the value provided by the loop iterator can be transformed
as you want to do the operations needed in your loop, you don't necessarily
need to keep it unchanged. A compiler can detect if it's assigned or not;
and if not, it can suppress the copy into from the actual control variable
(invisible, but not readonly) to the local variable.

Post by Dibyendu Majumdar
and
is just bad practice in my view.

Given it is local, this is not bad practice at all, it's jsut a standard
local variable, preinitialized, but still mutable as we want. So you don't
need to declare another local varaible in the loop

Post by Dibyendu Majumdar
Anyway I take the liberty of
optimising the for num index here, and require that the variable is
not assigned a non-integer value. I would forbid assignment but this
is harder to do.

The best you can do is to emit a warning because it may not be compatible
with older versions of Lua. If you compile for Lua 5.3 only, nothing is
needed, the actual control varaible is well protected, and it's just
simpelr to manage all variables the same way everywhere in the code inside
the loop.

Post by Dibyendu Majumdar
Regards
Dibyendu

[Egor Skriptunoff]

Post by Dibyendu Majumdar
Hi, to be honest the for num loop variable should be treated as
'readonly' inside the loop body as it is a copy of the real index
variable, and changing it inside the body doesn't affect the loop, and
is just bad practice in my view.

IMO, it's not a "bad practice".
Mutability of loop variable (without affecting the loop) is quite useful.
Examples:

for name, value in str:gmatch"(%w+)=(%d+)" do
value = tonumber(value)
....
end

for k = 1, 10 do
k = tostring(k)
....
end

[Dirk Laurie]

Op Vr. 16 Nov. 2018 om 07:26 het Egor Skriptunoff

Post by Egor Skriptunoff

Post by Dibyendu Majumdar
Hi, to be honest the for num loop variable should be treated as
'readonly' inside the loop body as it is a copy of the real index
variable, and changing it inside the body doesn't affect the loop, and
is just bad practice in my view.

IMO, it's not a "bad practice".
Mutability of loop variable (without affecting the loop) is quite useful.
for name, value in str:gmatch"(%w+)=(%d+)" do
value = tonumber(value)
....
end
for k = 1, 10 do
k = tostring(k)
....
end

+1.

[Dibyendu Majumdar]

On Wed, 14 Nov 2018 at 19:55, Egor Skriptunoff

Post by Egor Skriptunoff
local function f()
local tm = os.clock()
local o = 0
for j = 1, 1e4 do
local x = 0
for k = 1, 1e5 do
x = x ~ (j + k)
end

What's interesting is that Lua's parser think 1e5 is floating point
... whereas 10000 is integer.
That's why I had to change the limit to 10000 in the type annotated version.

Regards

[Egor Skriptunoff]

Post by Dibyendu Majumdar

Post by Egor Skriptunoff
for j = 1, 1e4 do

That's why I had to change the limit to 10000 in the type annotated version.

In Lua 5.3, in numeric "for"-loop, if both "initial value" and "step" are
integers, then loop variable will have only integer values, even if "limit"
is float.

for j = 1, math.pi, 2 do
print(j, math.type(j))
end

So, when parsing "for j = 1, 1e4 do", Ravi should conclude that "j" is
integer, similar to how Lua 5.3 deduces type of loop variable.
This way there would be no reason to replace 1e4 with 10000

[Philippe Verdy]

Post by Egor Skriptunoff
So, when parsing "for j = 1, 1e4 do", Ravi should conclude that "j" is
integer, similar to how Lua 5.3 deduces type of loop variable.
This way there would be no reason to replace 1e4 with 10000

an integer yes, but in the value range of numbers (i.e. IEEE doubles...).
Just retry with
for j = 1, 1e99 do
and you'll conclude that j cannot be a 32-bit (or even 64-bit) integer
because the upper limit is largely above MAXINT+1.

Note also that this loop would even be infinite in Lua (because adding
incremental step of 1 will cease to be significant when j reaches 2^53 (if
Lua "numbers" are compiled as IEEE 64-bit doubles) like in this example:
for j = 2**53, 2**53+2**15-1 do
which you could expect perform (2**15)=32768 loops exactly, but will
instead loop infinitely with j constantly set to the same start value
(2**53), and the default step=1 has no effect (same here as step=0)... Lua
does not check that (start+step != start) and that (limit-step != limit),
i.e. that there will be an effective linear progression and that the limit
will be reachable.

[Dibyendu Majumdar]

On Wed, 14 Nov 2018 at 19:55, Egor Skriptunoff

Post by Egor Skriptunoff
C:\Software\ravi\bin>type program.lua
local function f()
local tm = os.clock()
local o = 0
for j = 1, 1e4 do
local x = 0
for k = 1, 1e5 do
x = x ~ (j + k)
end
o = o | x
end
print(" Result = "..o)
print(" CPU time = "..(os.clock() - tm))
end
print"Benchmarking non-compiled function"
f()
print"Compiling the function"
assert(ravi.compile(f))
print"Benchmarking compiled function"
f()

Here are LLVM results on my Windows machine:
C:\work\github\ravi\ravi-tests>ravi egor_test.lua
Benchmarking non-compiled function
Result = 114656
CPU time = 11.709
Compiling the function
Benchmarking compiled function
Result = 114656
CPU time = 6.633

C:\work\github\ravi\ravi-tests>ravi egor_test_typed.lua
Benchmarking non-compiled function
Result = 114656
CPU time = 9.832
Compiling the function
Benchmarking compiled function
Result = 114656
CPU time = 0.311

I have uploaded a distribution that uses LLVM backend - please see
https://github.com/dibyendumajumdar/ravi-distro/releases

What you can see from above, and my experience is that, JIT compiling
plain Lua code will make code 2x faster at best; to get big
improvements type annotations are needed.

Regards
Dibyendu