LuaJIT and large tables

[Sean Conner]

I'm not sure if this is the correct place for this, but I found an issue
with LuaJIT (2.0.0 beta 2) and it relates to my insanely large Lua script
[1][2]. The script generates a really huge array (73,011 items). Lua 5.1.4
handles it fine, but when I attempted to use LuaJIT on it, I got the
following:

luajit: default.lua:352835: function at line 2357 has more than 65536
constants
stack traceback:
[C]: in function 'dofile'
showmsg.lua:3: in main chunk
[C]: ?

Lua itself limits a function to around 2^17 or 2^18 constants, so I worked
around that limit. Is the 65,536 limit in LuaJIT a bug, or a design
constaint?

-spc (It's all string data, by the way ... )

[1] http://lua-users.org/lists/lua-l/2009-11/msg00463.html
It's an 82Mb file, by the way.

[2] The Lua bug it exposed has since been fixed
http://www.lua.org/bugs.html#5.1.4-6

[Mike Pall]

Post by Sean Conner
Lua itself limits a function to around 2^17 or 2^18 constants, so I worked
around that limit. Is the 65,536 limit in LuaJIT a bug, or a design
constaint?

LuaJIT has different limits and different design constraints.

For a single table that consists only of constants there's
effectively no limit (memory is the limit).

But your test case has lots of small, nested tables. Every
individual table only occupies a single constant slot (and not one
for each of its keys/values). But there are too many of these
tables to keep them in a single function.

The short term solution is to split the table construction into
smaller pieces, wrapped into individual functions. The long term
solution would be to add a workaround for these limits to LuaJIT.
But I'm not sure about the best way, yet.

Post by Sean Conner
[2] The Lua bug it exposed has since been fixed
http://www.lua.org/bugs.html#5.1.4-6

This problem is not present in LuaJIT 2.0.

--Mike

[Sean Conner]

Post by Mike Pall

Post by Sean Conner
Lua itself limits a function to around 2^17 or 2^18 constants, so I worked
around that limit. Is the 65,536 limit in LuaJIT a bug, or a design
constaint?

LuaJIT has different limits and different design constraints.

It's why I asked 8-)

Post by Mike Pall
The short term solution is to split the table construction into
smaller pieces, wrapped into individual functions. The long term
solution would be to add a workaround for these limits to LuaJIT.
But I'm not sure about the best way, yet.

I cut the number to a quarter of the original size (I kept halving it
until LuaJIT worked), but so far, that's the *only* program I've had to
change in order to get LuaJIT to work (so it's a 99% drop-in replacement for
Lua).

It did cut the processing down a bit (best of three runs):

[spc]lucy:~/.intertwingle>time lua showmsg.lua >/dev/null

real 0m6.825s
user 0m6.318s
sys 0m0.508s
[spc]lucy:~/.intertwingle>time luajit showmsg.lua >/dev/null

real 0m5.881s
user 0m5.380s
sys 0m0.501s

And, oddly enough, just making the table construction smaller made the Lua
code run faster (again, best of three runs; script with the larger table
constructions):

[spc]lucy:~/.intertwingle>time lua showmsg.lua >/dev/null

real 0m7.248s
user 0m6.759s
sys 0m0.490s

-spc (Thanks for the clarification)