Discussion:
Lua that generates executables?
(too old to reply)
Joseph Stewart
2004-11-16 13:49:19 UTC
Permalink
Hello all,

I remember some traffic several months ago about a lua
package/modification that allowed one to generate stand-alone
executables with lua.

I can't seem to come up with the right keyword to search the mailing
list to find this work.

Can someone steer me in the right direction?

Failing that, I'll probably make my own... seems pretty easy to do...
I'm just feeling lazy.

Thanks for this list!

-joe
--
Person who say it cannot be done should not interrupt person doing it.
-- Chinese Proverb
Asko Kauppi
2004-11-16 14:49:32 UTC
Permalink
Eric Tetz, 25th August.

http://tetzfamily.com/eric/code/lua2exe.zip

Hailed be the OS X Mail search tool.. :)



16.11.2004 kello 15:49, Joseph Stewart kirjoitti:

Hello all,
Post by Joseph Stewart
I remember some traffic several months ago about a lua
package/modification that allowed one to generate stand-alone
executables with lua.
I can't seem to come up with the right keyword to search the mailing
list to find this work.
Can someone steer me in the right direction?
Failing that, I'll probably make my own... seems pretty easy to do...
I'm just feeling lazy.
Thanks for this list!
-joe
--
Person who say it cannot be done should not interrupt person doing it.
-- Chinese Proverb
Luiz Henrique de Figueiredo
2004-11-16 15:03:45 UTC
Permalink
Post by Joseph Stewart
I remember some traffic several months ago about a lua
package/modification that allowed one to generate stand-alone
executables with lua.
Perhaps you mean my srlua?
http://www.tecgraf.puc-rio.br/~lhf/ftp/lua/#srlua

--lhf
Joseph Stewart
2004-11-16 16:48:09 UTC
Permalink
That's the one! I'll also check out Asko's suggestion!

-j


On Tue, 16 Nov 2004 13:03:45 -0200, Luiz Henrique de Figueiredo
Post by Luiz Henrique de Figueiredo
Post by Joseph Stewart
I remember some traffic several months ago about a lua
package/modification that allowed one to generate stand-alone
executables with lua.
Perhaps you mean my srlua?
http://www.tecgraf.puc-rio.br/~lhf/ftp/lua/#srlua
--lhf
--
Person who say it cannot be done should not interrupt person doing it.
-- Chinese Proverb
Ivan Kolev
2004-11-16 20:24:55 UTC
Permalink
And what about glueing several Lua modules together into a common .exe? How would they find and load each other?
And what about compressing them into a .zip file and attaching that file to the exe? Maybe http://luaforge.net/projects/luazip/ would help here?

;)

Cheers,
Ivan
Luiz Henrique de Figueiredo
2004-11-16 20:29:32 UTC
Permalink
Post by Ivan Kolev
And what about glueing several Lua modules together into a common .exe? How would they find and load each other?
If you want to load and run all at once, precompile them with luac and glue
the output of luac.
Post by Ivan Kolev
And what about compressing them into a .zip file and attaching that file to the exe? Maybe http://luaforge.net/projects/luazip/ would help here?
Yes, it can be done. But I have slightly different plans in that direction...
(Though not much time to do it. Of course, anyone is welcome to do it.)
--lhf
Ivan Kolev
2004-11-16 21:04:07 UTC
Permalink
Post by Luiz Henrique de Figueiredo
If you want to load and run all at once, precompile them with luac and glue
the output of luac.
Yes, this sounds reasonable for most occasions, I suppose.
Post by Luiz Henrique de Figueiredo
Yes, it can be done. But I have slightly different plans in that direction...
(Though not much time to do it. Of course, anyone is welcome to do it.)
I don't actually need this right now (as Philippe mentioned, there's no point to zip the scripts in most cases), I was trying to hint to something like Java's .jar files, i.e. more along the lines of a robust package system. Maybe Roberto's package proposal is going to address all these topics...

Ivan
Philippe Lhoste
2004-11-16 20:29:38 UTC
Permalink
Post by Ivan Kolev
And what about glueing several Lua modules together into a common .exe?
How would they find and load each other?
This is quite tricky, I guess you should paste the included files into
the main file.
Post by Ivan Kolev
And what about compressing them into a .zip file and attaching that file
to the exe? Maybe http://luaforge.net/projects/luazip/ would help here?
Well, you have to have a quite large Lua script to make it worth
including the unzipping library...
That, or a lot of Lua data, like the guy putting binary image data in
Ascii form.
--
Philippe Lhoste
-- (near) Paris -- France
-- Professional programmer and amateur artist
-- http://Phi.Lho.free.fr
-- -- -- -- -- -- -- -- -- -- -- -- -- --
Quinn Tyler Jackson
2004-11-16 20:33:28 UTC
Permalink
Post by Ivan Kolev
And what about glueing several Lua modules together into a common
.exe? How would they find and load each other?
You could do that by writing a light-weight executable stub in something
like C or C++ that would have the responsibility of searching an index at
the end of itself for a module called "main" or something like that -- the
index would point to pre-compiled Lua chunks that had also been attached to
the stub. The library loading calls of Lua would then have to be modified
slightly to first search the tacked-on-index -- and if it found the package
there, it could load it instead of the chunk.

LuaLink sounds like an interesting project. Using fstreams, it could
probably be done fairly portably.

--
Quinn Tyler Jackson
http://members.shaw.ca/qjackson/
Daniel Quintela
2004-11-16 22:06:01 UTC
Permalink
Post by Ivan Kolev
And what about glueing several Lua modules together into a common
.exe? How would they find and load each other?
And what about compressing them into a .zip file and attaching that
file to the exe?
Hi:

It wasn't my objective to do a general tool when I made Kurmi, but it
seems like you are proposing: a Lua interpreter with a zip "filesystem"
attached, with the possibility of running scripts and "automagically"
extract+loadlib from the zip.
I'm using it as the core of an installer, you can use it for what you
want. ;-)

Regards,
Daniel
Andre Carregal
2004-11-16 22:29:05 UTC
Permalink
Hi,

I have to agree that zipping Lua files usually does not offer much
advantage, unless you are already offering the Zip library, as is the Kepler
case.

We are about to release a new version of LuaZip followed by our Lua ARchive
(LAR) module. We had LAR ready for a while, but with the incoming Compat-5.1
module it seemed a better idea to hold it a bit more.

We are aiming on an easy deployable format for Kepler applications, but LAR
could be used anywhere that needs to make things more compact.

Basically it allows someone to transparently access files (with require,
loadfile and io.open) inside the .lar archive. For example:

io.open ("/a/b/c.lua")

would work for any of

/a/b/c.lua
/a/b.lar (with c.lua zipped)
/a.lar (with b/c.lua zipped)

Andre
-----Original Message-----
Sent: Tuesday, November 16, 2004 6:04 PM
To: Lua list
Subject: Re: Lua that generates executables?
Post by Luiz Henrique de Figueiredo
If you want to load and run all at once, precompile them
with luac and glue
Post by Luiz Henrique de Figueiredo
the output of luac.
Yes, this sounds reasonable for most occasions, I suppose.
Post by Luiz Henrique de Figueiredo
Yes, it can be done. But I have slightly different plans in
that direction...
Post by Luiz Henrique de Figueiredo
(Though not much time to do it. Of course, anyone is
welcome to do it.)
I don't actually need this right now (as Philippe mentioned,
there's no point to zip the scripts in most cases), I was
trying to hint to something like Java's .jar files, i.e. more
along the lines of a robust package system. Maybe Roberto's
package proposal is going to address all these topics...
Ivan
Ivan Kolev
2004-11-16 22:14:33 UTC
Permalink
Post by Andre Carregal
We are aiming on an easy deployable format for Kepler applications, but LAR
could be used anywhere that needs to make things more compact.
Basically it allows someone to transparently access files (with require,
io.open ("/a/b/c.lua")
would work for any of
/a/b/c.lua
/a/b.lar (with c.lua zipped)
/a.lar (with b/c.lua zipped)
Andre
Exactly! I forgot to mention in my previous post that in this case Zip is used not as much for the compression, as for the virtual file system that it creates inside the .zip file.
Another common option for virtual file systems (as used in games, for example) is encryption. So it would be wise to have an option to add some sort of filter to the .LAR reader/creator...
BTW, I talk about Zip because it is a common tool. It could be tar as well (though it's not so common on Windows). But it's not a good idea to have a completely new archive format, because we'll lose the easy archive creation/browsing/modification that a tool like Zip provides. I suppose Java picked Zip for the same reason. (It's a pity that Zip rules the archiving world when there are much better solutions like Rar, but that's a completely different topic).

Just tossing ideas around ;)
Ivan
Asko Kauppi
2004-11-16 22:26:38 UTC
Permalink
If rar is free (not sure if it is, don't remember..) I wouldn't mind
using it.

The point (for the package creator) is that the tool to make the
package is generic, and available on multiple platforms. It does not
have to be the most generic, nor most available tool.

Actually, I took for granted Andre meant something like rar, but.. on
second read.. he doesn't say specifically.

-ak
Post by Ivan Kolev
Post by Andre Carregal
We are aiming on an easy deployable format for Kepler applications, but LAR
could be used anywhere that needs to make things more compact.
Basically it allows someone to transparently access files (with require,
io.open ("/a/b/c.lua")
would work for any of
/a/b/c.lua
/a/b.lar (with c.lua zipped)
/a.lar (with b/c.lua zipped)
Andre
Exactly! I forgot to mention in my previous post that in this case Zip
is used not as much for the compression, as for the virtual file
system that it creates inside the .zip file.
Another common option for virtual file systems (as used in games, for
example) is encryption. So it would be wise to have an option to add
some sort of filter to the .LAR reader/creator...
BTW, I talk about Zip because it is a common tool. It could be tar as
well (though it's not so common on Windows). But it's not a good idea
to have a completely new archive format, because we'll lose the easy
archive creation/browsing/modification that a tool like Zip provides.
I suppose Java picked Zip for the same reason. (It's a pity that Zip
rules the archiving world when there are much better solutions like
Rar, but that's a completely different topic).
Just tossing ideas around ;)
Ivan
David Given
2004-11-16 22:42:06 UTC
Permalink
Post by Asko Kauppi
If rar is free (not sure if it is, don't remember..) I wouldn't mind
using it.
It's not. It's very non-free, actually.

If you want free archiving systems, your best bet is good old fashioned ZIP,
which allows you to cheaply extract any file from the archive. If you want
better compression and don't mind having to decompress everything at once, use
tar compressed with gz or bz2. gz is relatively cheap, bz2 gives excellent
compression but needs massive buffers to decompress.
--
[insert interesting .sig here]
Daniel Quintela
2004-11-16 22:54:32 UTC
Permalink
Post by David Given
If you want free archiving systems, your best bet is good old
fashioned ZIP, which allows you to cheaply extract any file from the
archive. If you want better compression and don't mind having to
decompress everything at once, use tar compressed with gz or bz2. gz
is relatively cheap, bz2 gives excellent compression but needs massive
buffers to decompress.
Remember that the zip file format supports multiple compression method
in a member by member basis.
You can compress some members with the "old" method and others with bz2.

Daniel
David Given
2004-11-16 23:10:57 UTC
Permalink
Daniel Quintela wrote:
[...]
Post by Daniel Quintela
Remember that the zip file format supports multiple compression method
in a member by member basis.
You can compress some members with the "old" method and others with bz2.
Ah, but compressing member by member means that you have to start a new
compression block with every file, which means that if you've got a lot of
very small files there's really no point because of the overhead. Plus, the
directory information is uncompressed, and it very quickly adds up.

Also, bz2 compression in ZIP files is rather new and not particularly well
supported. Standard ZIP files haven't changed much from good old PKZIP 2.04g
(and it's scary the way that version number rolls off the keyboard).
--
[insert interesting .sig here]
Adrián Pérez
2004-11-27 21:40:25 UTC
Permalink
[...]
Post by Daniel Quintela
Remember that the zip file format supports multiple compression
method in a member by member basis.
You can compress some members with the "old" method and others with bz2.
[...] Also, bz2 compression in ZIP files is rather new and not
particularly well supported. Standard ZIP files haven't changed much
from good old PKZIP 2.04g (and it's scary the way that version number
rolls off the keyboard).
What about LZO compression? It has a good compression ratio, comparable
to gzip, and it has fast decompression times using quite small buffers.
There is miniLZO available, too: it's weights only 12 kB (compiled for
MacOS X) and has full decompression support plus minimal support for
compression (only one compression method supported, all decompression
methods available).

The library is GPL.

Webpage:
http://www.oberhumer.com/opensource/lzo/

Tiny Lua miniLZO module:
http://lua-users.org/wiki/LuaModuleLzo

-ap

Danilo Tuler
2004-11-16 23:36:48 UTC
Permalink
Post by Ivan Kolev
Another common option for virtual file systems (as used in
games, for example) is encryption. So it would be wise to
have an option to add some sort of filter to the .LAR
reader/creator...
Yes, filters based on LTN12
(http://lua-users.org/wiki/FiltersSourcesAndSinks) are on the back of our
minds.
Post by Ivan Kolev
BTW, I talk about Zip because it is a common tool. It could
be tar as well (though it's not so common on Windows). But
it's not a good idea to have a completely new archive format,
because we'll lose the easy archive
creation/browsing/modification that a tool like Zip provides.
In fact the first work version of LAR supported zip and tar(.gz).
But I think we are going to stick to one format (zip), instead of supporting
both.
Post by Ivan Kolev
I suppose Java picked Zip for the same reason. (It's a pity
that Zip rules the archiving world when there are much better
solutions like Rar, but that's a completely different topic).
As you said, compression is not the primary concern, so zip works pretty
well.

Danilo
Luiz Henrique de Figueiredo
2004-11-16 23:18:54 UTC
Permalink
Post by Ivan Kolev
I forgot to mention in my previous post that in this case Zip is used not as much for the compression, as for the virtual file system that it creates inside the .zip file.
I had in mind to use the cpio format, which is as simple as it gets. It's even
simpler than tar... But like I said, I don't really have time for this right
now plus srlua can be easily modified to handle any kind of file.
--lhf
Daniel Quintela
2004-11-16 23:43:00 UTC
Permalink
Post by Luiz Henrique de Figueiredo
I had in mind to use the cpio format, which is as simple as it gets. It's even
simpler than tar... But like I said, I don't really have time for this right
now plus srlua can be easily modified to handle any kind of file.
--lhf
Zip, Tar, CPIO... I think any proposal has to include support of dynamic
module loading.
Do you agree ?

Daniel
Luiz Henrique de Figueiredo
2004-11-17 00:05:34 UTC
Permalink
Post by Daniel Quintela
Zip, Tar, CPIO... I think any proposal has to include support of dynamic
module loading.
Actually, srlua already contains a simple support for that: the glue program
accepts a 4th, optional, argument, a single letter which it uses in the glue
signature. The default is 'L', as in "%%glue:L". 'L' is for Lua, of course.
But we could use 'Z' for zip files, 'T' for tar files, etc. srlua.c would have
to be extended to handle these, of course.

The point of all this is orthogonality. glue helps you append arbitrary data
to an executable. The interpretation of what you do with the data is up to
you, but at least glue helps you send this information to the running program.
--lhf
Daniel Quintela
2004-11-17 03:30:25 UTC
Permalink
I see that I didn't explain well what I mean with "dynamic loading
support". :-(
If you have an application which consist of some Lua scripts i.e. using
LuaSocket, your distribution system must extract luasocket.dll to a
temporary place and use the OS to dynload it.
The virtual file system is not enough to make this kind of "packed
application" when the scripts depends on dlls or shared libraries.

Daniel
David Burgess
2004-11-17 00:15:46 UTC
Permalink
On a similar vein. In Win32 we build Lua binary code into resource file format
data (using a Lua script). The resources are then added to a DLL (possiibly only
containing resources). The loading is done with a single function
(named rcload())
this is a rough equivalent to loadfile() except for loading Lua code from DLLs
instead of the file system. We have found that there is little advantage of
compressing the script in the DLL (although we sometimes do it). Loading
scripts from DLL resources is dynamite quick. It offers enormous advantages
in packaging, the package IS the DLL. It has advantages for deployment
simplicity,
it has more user and admin. resilience. There must be a rough equivalent to this
approach on Unix?

PS if anyone wants the binary to resource code and assocaited loader, let
me know and I can put it on the web.
Post by Daniel Quintela
Post by Luiz Henrique de Figueiredo
I had in mind to use the cpio format, which is as simple as it gets. It's even
simpler than tar... But like I said, I don't really have time for this right
now plus srlua can be easily modified to handle any kind of file.
--lhf
Zip, Tar, CPIO... I think any proposal has to include support of dynamic
module loading.
Do you agree ?
Daniel
--
DB
Marius Gheorghe
2004-11-17 14:30:57 UTC
Permalink
Post by David Burgess
PS if anyone wants the binary to resource code and assocaited
loader, let
me know and I can put it on the web.
Yes, please. I have been long thinking about writing my own Lua to RC
converter but never had the time to do it.

Marius
Taj Khattra
2004-11-16 22:48:11 UTC
Permalink
the Tcl/Tk folks have Starkits/Starpacks/Tclkits for packaging Tcl
applications - see http://wiki.tcl.tk/3661

might be a useful source of ideas for folks thinking of similar things
for Lua...
Ashwin Hirschi
2004-11-17 04:33:15 UTC
Permalink
It's nice to see that the Lua ARchive idea is actively being worked on!

My company has developed a near-identical concept to create packaged distributions for easy deployment. Our runtime can load both code & data directly (and indeed transparently) from an archive. It really is a very nice setup and works marvellously.

Previously deploying a product meant unzipping an archive containing the runtime as well as the compiled application and library scripts onto the clients' machine. To be honest, I find this more than acceptable when installing larger systems, like client-server applications, at a customer's site. Especially since someone's driving up there anyway, and you have the additional benefit of amusing your client by installing a heavy-duty client-server product in less than five minutes [and from a floppy disk no less! ;-)].

But since we also develop smaller tools for specialized markets [and don't always enjoy travelling] I started looking into ways to provide even cleaner installs. Our base system takes about 70 small (Lua) library scripts to handle things like the graphical user-interface, multi-threading, communication and the like. Everything's nicely organized into directories, but still... it's not something I'd like to provide to customers without an installer...

So, thinking about it, it was an easy step to take the initial approach of unzipping scripts during install to unzipping the scripts when the system actually needs them.

Our approach is not based on luazip, but it does use zziplib [that luazip relies on as well, I think]. This library seems to work as advertised and also provides interesting features to enable extra processing for de-shrouding and such. Having said that, we are actually in the process of exchanging it for something else, mainly to allow us to also create/update archives.

As far as benefits go, it's true compiled Lua scripts take very little space.

But, looking at our latest product, even small (non-trivial) applications *do* benefit from compression: the deployment set (library and application scripts as well as data files, like GUI definitions and things like colour schemes) takes about 260k. The archived version is only 112k. So, even counting the code to support the transparent unzipping of it all, you win.

Deployment is trivial: since the system is self-configuring [i.e. we create require's LUA_PATH based on - and relative to - the location of the executable], users can simply put the runtime and its archive anywhere they like. And it runs as is. This is somewhat similar to Jean-Claude Wippler's Starkit concept [that I was happy to see got a mention here on the list as well].

A benefit that hasn't been mentioned yet [or has only been hinted at?] is *updating* deployed systems.

Since our deployed software can basically be a small runtime (about 200k) and an archive to hold the actual logic and data, updating the system becomes a matter of swapping the old archive for the new, updated one. You don't, for example, run the danger of losing inter-script integrity: dependencies between the scripts within the archive are clear and contained. And, last but not least, those paranoid of mind can keep the old version by simply hanging on to the original application archive, knowing they can restore the old setup by a simple 1-file replace.

Lastly, somewhat to my initial surprise, there's no actual slow-down while the application starts (from an archive-based configuration). The time it takes to decompress the scripts and data must be offset by the fact there's a lot less file-IO going on, I guess. So no complaints here either!

Anyway, this turned out quite a bit longer than I anticipated... But I hope I managed to illustrate that a well-integrated Lua ARchive mechanism is definitely something that's nice to have. I'm pretty sure many people will enjoy it, once they experience the potential benefits.

Ashwin.
--
no signature is a signature.
Vijay Aswadhati
2004-11-20 16:24:17 UTC
Permalink
Post by Ashwin Hirschi
A benefit that hasn't been mentioned yet [or has
only been hinted at?] is *updating* deployed systems.
Since our deployed software can basically be a
small runtime (about 200k) and an archive to
hold the actual logic and data, updating the
system becomes a matter of swapping the old
archive for the new, updated one. You don't,
for example, run the danger of losing
inter-script integrity: dependencies between
the scripts within the archive are clear and
contained. And, last but not least, those
paranoid of mind can keep the old version
by simply hanging on to the original application
archive, knowing they can restore the old setup
by a simple 1-file replace.
Great feature discussion thread with all the right
key words and lots of references to good and proven
solutions.

For updating deployed software, I wanted to throw
in and encourage others to look at Java Web Start
technology. Perl (ActivePerl may be) has something
similar, with an extra feature to search for new
software modules. A while back this (the feature
to search and download software modules, similar
to Perl) came up in discussing packages and making
them available from LuaForge. (just a note to myself)
Andre Carregal
2004-11-16 23:49:34 UTC
Permalink
As a matter of fact, we considered using tar[.gz] and/or other formats for
the LAR file but we found out that this would make things worse when
concerning portability.

One option would be the use of more than one extension (something like
.ltar, .lrar, .lzip etc) but that would present too much trouble for our
goal.

We settled on the Zip format not for its compression ratio, for sure, but
for its compressors availability in multiple environments.

The virtual file system provided by LAR is indeed a first step. The next one
will be defining a common structure for applications/modules, something like
the WEB-INF directory on .war files for example.

Andre
-----Original Message-----
Sent: Tuesday, November 16, 2004 7:27 PM
To: Lua list
Subject: Re: Lua that generates executables?
If rar is free (not sure if it is, don't remember..) I wouldn't mind
using it.
The point (for the package creator) is that the tool to make the
package is generic, and available on multiple platforms. It does not
have to be the most generic, nor most available tool.
Actually, I took for granted Andre meant something like rar, but.. on
second read.. he doesn't say specifically.
-ak
Post by Ivan Kolev
Post by Andre Carregal
We are aiming on an easy deployable format for Kepler
applications,
Post by Ivan Kolev
Post by Andre Carregal
but LAR
could be used anywhere that needs to make things more compact.
Basically it allows someone to transparently access files (with
require,
io.open ("/a/b/c.lua")
would work for any of
/a/b/c.lua
/a/b.lar (with c.lua zipped)
/a.lar (with b/c.lua zipped)
Andre
Exactly! I forgot to mention in my previous post that in
this case Zip
Post by Ivan Kolev
is used not as much for the compression, as for the virtual file
system that it creates inside the .zip file.
Another common option for virtual file systems (as used in
games, for
Post by Ivan Kolev
example) is encryption. So it would be wise to have an
option to add
Post by Ivan Kolev
some sort of filter to the .LAR reader/creator...
BTW, I talk about Zip because it is a common tool. It could
be tar as
Post by Ivan Kolev
well (though it's not so common on Windows). But it's not a
good idea
Post by Ivan Kolev
to have a completely new archive format, because we'll lose
the easy
Post by Ivan Kolev
archive creation/browsing/modification that a tool like Zip
provides.
Post by Ivan Kolev
I suppose Java picked Zip for the same reason. (It's a pity
that Zip
Post by Ivan Kolev
rules the archiving world when there are much better solutions like
Rar, but that's a completely different topic).
Just tossing ideas around ;)
Ivan
Continue reading on narkive:
Loading...