Inclusion of lpack in Lua? Suggestions for Lua's unpack()

[Majic]

Hello...

A friend of mine recently began trying to reverse-engineer the
[closed] Volano instant messenger protocol in Perl which relied
heavily on Perl's pack() and unpack() functions for dissecting binary
streams. I wanted to rewrite his "framework" in Lua but could not
easily do so without the lpack module, linked here:
http://www.tecgraf.puc-rio.br/~lhf/ftp/lua/5.1/lpack.tar.gz I am in
no way affiliated with the author so I cannot speak for him, and I
must admit that I have not used it very much so far, but it is my
opinion that something like this will become necessary with the
inclusion of bitlib in 5.2. If you do not come from a Perl
background, or are just not familiar with Perl's pack() and unpack()
the behavior can be read about at these links:
http://perldoc.perl.org/functions/pack.html and
http://perldoc.perl.org/functions/unpack.html

Regardless if you think adopting a new pack() and unpack() is a good
idea, Lua already has an unpack() and the semantics are quite
different. Firstly, I believe I heard that redefining __length for
string and tables in 5.2 is acknowledged by functions relying on the
length operator. This is semi-relevant to what I'm getting to,
because unpack() relies on the length operator to return the
iteritable keys, which I'm sure you all know. :)

I wanted to suggest that unpack() be moved to table.unpack() since the
"namespacing" seems to make more sense (at least to me). I also
wanted to suggest that instead of relying on the length operator, it
could return all values in general. Something like:

local sample = { ['key'] = 'value', 'value2' } unpack(sample) would
yield: return 'value', 'value2'

I hope that I have not upset anyone by this, as I have come across a
few strong opinions about the semantics of Lua when discussing it with
friends. If anything, I would feel most strongly about *just* the
move from unpack() to table.unpack()

Regards :)

[Roberto Ierusalimschy]

[...] If anything, I would feel most strongly about *just* the
move from unpack() to table.unpack()

This was done already.

-- Roberto

[Majic]

Oh, hrmm.... guess I need to look through 5.2 again

On Mon, Jan 18, 2010 at 5:48 AM, Roberto Ierusalimschy

Post by Roberto Ierusalimschy

[...]  If anything, I would feel most strongly about *just* the
move from unpack() to table.unpack()

This was done already.
-- Roberto

[David Manura]

Post by Majic
Perl's pack() and unpack() functions for dissecting binary
streams.  ... but it is my opinion that something like this will
become necessary with the inclusion of bitlib in 5.2.

These are very useful functions. It's a fundamental need to
(en/de)code some binary stream. The need also comes up in FFI [1] and
was also mentioned in a similar way in LuaJIT. I wouldn't go as far
to call them "necessary" as in impossible not to have since you can
write them in Lua, and some programs using bitops might not use
packing/unpacking (but do most?). One would also need to decide which
encodings to support (128-bit floats?)

[1] http://alien.luaforge.net/

Post by Majic
I also wanted to suggest that instead of relying on the length operator, it
local sample = { ['key'] = 'value', 'value2' } unpack(sample) would
yield: return 'value', 'value2'

Keys are unordered, unless you impose an ordering like next or
__pairs. It's not clear to me that such an implementation of unpack
would return elements in the order I would need them in. What is the
use case for this?

[Majic]

Yes, I guess I must concede that a true pack()/unpack() for binary
streams would not be a necessity as you can just require() the module
whenever you want, it would just be nice to have it in the core
alongside bitlib for that greater flexibility... Also might make
writing string functions that support unicode completely possible from
Lua :)

In regards to the existing unpack(), I had forgotten about that... now
it makes sense that it only returns the pairs with numeric keys
because it knows how to order them. >.< I personally haven't had a
need to return *every* value, it just seemed like there was a possible
use case missing... Guess I"ll clam up about that as there's no
definitive need.. XD

Post by Majic
Perl's pack() and unpack() functions for dissecting binary
streams.  ... but it is my opinion that something like this will
become necessary with the inclusion of bitlib in 5.2.

These are very useful functions.  It's a fundamental need to
(en/de)code some binary stream.  The need also comes up in FFI [1] and
was also mentioned in a similar way in LuaJIT.  I wouldn't go as far
to call them "necessary" as in impossible not to have since you can
write them in Lua, and some programs using bitops might not use
packing/unpacking (but do most?).  One would also need to decide which
encodings to support (128-bit floats?)
[1] http://alien.luaforge.net/

Post by Majic
I also wanted to suggest that instead of relying on the length operator, it
  local sample = { ['key'] = 'value', 'value2' } unpack(sample) would
  yield: return 'value', 'value2'

Keys are unordered, unless you impose an ordering like next or
__pairs.  It's not clear to me that such an implementation of unpack
would return elements in the order I would need them in.  What is the
use case for this?

[steve donovan]

Post by Majic
Yes, I guess I must concede that a true pack()/unpack() for binary
streams would not be a necessity as you can just require() the module
whenever you want, it would just be nice to have it in the core
alongside bitlib for that greater flexibility...

But we already have lhf's lpack and Roberto's lstruct. (The lstruct in
Alien is a little older but extended to handle userdata as well.)

May be nice to have such a 'builtin' facility, but it does seem like a
job for an extension. Otherwise, how do we know when to stop?

[Majic]

To be honest, I considered bitlib the job for an extension, and I
believe it was... but lpack and bitlib make such a perfect pairing,
and the trust-factor associated with it being in the core would
definitely draw users from other languages who don't currently
consider Lua a contender for jobs like this.

On Mon, Jan 18, 2010 at 10:33 PM, steve donovan

Post by steve donovan

Post by Majic
Yes, I guess I must concede that a true pack()/unpack() for binary
streams would not be a necessity as you can just require() the module
whenever you want, it would just be nice to have it in the core
alongside bitlib for that greater flexibility...

But we already have lhf's lpack and Roberto's lstruct. (The lstruct in
Alien is a little older but extended to handle userdata as well.)
May be nice to have such a 'builtin' facility, but it does seem like a
job for an extension.  Otherwise, how do we know when to stop?

[steve donovan]

Post by Majic
and the trust-factor associated with it being in the core would
definitely draw users from other languages who don't currently
consider Lua a contender for jobs like this.

I hadn't thought about the 'trust-factor' !

I suppose I consider lpack (or lstruct) to be optional bits of Lua,
written by the same people ;)