__tostring and string.format

[HyperHacker]

t={}
setmetatable(t, {__tostring = function(t) return "xxx" end})
print(t)

xxx

print(('%s'):format(t))

stdin:1: bad argument #1 to 'format' (string expected, got table)

Why does string.format not call tostring() on arguments given as %s?
(and for that matter tonumber() for %d/%u, which in turn could call a
__tonumber metamethod...)

[Jonathan Castello]

Post by HyperHacker

t={}
setmetatable(t, {__tostring = function(t) return "xxx" end})
print(t)

xxx

print(('%s'):format(t))

stdin:1: bad argument #1 to 'format' (string expected, got table)
Why does string.format not call tostring() on arguments given as %s?
(and for that matter tonumber() for %d/%u, which in turn could call a
__tonumber metamethod...)
--
Sent from my toaster.

I had never noticed that, but it's a good idea. As for __tonumber, I
was thinking about that too, and I'd like to see it just for parity
with __tostring.

~Jonathan

[steve donovan]

Post by HyperHacker
Why does string.format not call tostring() on arguments given as %s?
(and for that matter tonumber() for %d/%u, which in turn could call a
__tonumber metamethod...)

This bit me recently and I'm now using a more paranoid version:

local format = string.format

function formatx (fmt,...)
local args = {...}
local i = 1
for p in fmt:gmatch('%%s') do
if type(args[i]) ~= 'string' then
args[i] = tostring(args[i])
end
i = i + 1
end
return format(fmt,unpack(args))
end

print(formatx('%s=%s',10,nil))

string.format is one of the important string functions and it would be
good if this kind of behaviour was built-in and hence more efficient.

But then we could ask the very same question about table.concat, which
also doesn't use tostring...

steve d.

[Alexander Gladysh]

Post by steve donovan
string.format is one of the important string functions and it would be
good if this kind of behaviour was built-in and hence more efficient.

If you're not sure of the argument type, you always can call
tostring() on it explicitly, no?

Post by steve donovan
But then we could ask the very same question about table.concat, which
also doesn't use tostring...

Why should it?

Alexander.

[GrayFace]

Post by Alexander Gladysh

Post by steve donovan
But then we could ask the very same question about table.concat, which
also doesn't use tostring...

Why should it?

Because it would be convenient. Are there any reasons for tostring() not
to be called in these 2 cases?

[Roberto Ierusalimschy]

Post by GrayFace

Post by Alexander Gladysh

Post by steve donovan
But then we could ask the very same question about table.concat, which
also doesn't use tostring...

Why should it?

Because it would be convenient. Are there any reasons for tostring()
not to be called in these 2 cases?

Robustness? Several errors (e.g., a nil value) will be silently
supressed by 'tostring'. For '%s' this is not such a problem, as usually
we see the result, so it is easy to spot bugs. In table.concat, however,
it may cause some pain for debugging.

-- Roberto

[Javier Guerra Giraldez]

On Mon, Dec 6, 2010 at 9:39 AM, Roberto Ierusalimschy

Post by Roberto Ierusalimschy

Post by GrayFace

Post by Alexander Gladysh

Post by steve donovan
But then we could ask the very same question about table.concat, which
also doesn't use tostring...

Why should it?

Because it would be convenient. Are there any reasons for tostring()
not to be called in these 2 cases?

Robustness? Several errors (e.g., a nil value) will be silently
supressed by 'tostring'. For '%s' this is not such a problem, as usually
we see the result, so it is easy to spot bugs. In table.concat, however,
it may cause some pain for debugging.

in my book, a function that (within reason) normalizes its input is
more robust than one that fails.

of course, consistently failing is immensely more robust than
producing undesirable output on questionable input.

unfortunately, pre-screening a table before concat()ing it feels
clunky. a mapconcat() function would be nice, but that seems a good
candidate for a user library, not the core one. borrowing from
Scheme, i'd like a fold() that could produce an efficient replacement
to concat() when used with some core-provided string accumulation
function.

[steve donovan]

On Mon, Dec 6, 2010 at 5:02 PM, Javier Guerra Giraldez

Post by Javier Guerra Giraldez
in my book, a function that (within reason) normalizes its input is
more robust than one that fails.

Except for that nil case!

Post by Javier Guerra Giraldez
of course, consistently failing is immensely more robust than
producing undesirable output on questionable input.

Imagine you have generated 1MB of XML - now look for the nil token.

Post by Javier Guerra Giraldez
 a mapconcat() function would be nice,  but that seems a good
candidate for a user library,

In Penlight, List.join() works like that, but that module is trying to
slavishly reproduce the same results as Python...

Of course, it's easy to give out advice. I'm personally finding an
enhanced string.format to be useful since I have objects with
__tostring overriden.

steve d.

[Mark Hamburg]

I certainly find myself being paranoid and invoking tostring a lot when passing things to string.format and this seems less than ideal given that I have just said that I wanted a string. If debugging support is really important, I could see introducing a different escape character to mean "I want a string and you should use tostring to get one if need be". But since many of my uses of string.format are in debugging code anyway, I would like that code to be shorter and clearer about what is being passed in rather than wrapping everything in a layer of tostring().

Mark

[Duncan Cross]

On Mon, Dec 6, 2010 at 2:39 PM, Roberto Ierusalimschy

Post by Roberto Ierusalimschy

Post by GrayFace

Post by Alexander Gladysh

Post by steve donovan
But then we could ask the very same question about table.concat, which
also doesn't use tostring...

Why should it?

Because it would be convenient. Are there any reasons for tostring()
not to be called in these 2 cases?

Robustness? Several errors (e.g., a nil value) will be silently
supressed by 'tostring'. For '%s' this is not such a problem, as usually
we see the result, so it is easy to spot bugs. In table.concat, however,
it may cause some pain for debugging.
-- Roberto

I'm confused by what you mean here. table.concat(), like all the other
table functions like table.sort() and table.insert(), expects to take
a valid Lua array-table and if you pass it one with nils/holes in, it
usually raises no error and just has undefined results. We are
expected to know and understand this for the other functions, why not
table.concat() too?

-Duncan

[Roberto Ierusalimschy]

Post by Duncan Cross
I'm confused by what you mean here. table.concat(), like all the other
table functions like table.sort() and table.insert(), expects to take
a valid Lua array-table and if you pass it one with nils/holes in, it
usually raises no error and just has undefined results. We are
expected to know and understand this for the other functions, why not
table.concat() too?

Sorry, the nil example is not a good one ;) Any other type will do. For
instance, if you add a function to a table and then call 'table.concat',
do you really want to have 'function: 0x9fb7020' in the resulting
string? Or, more probably, you wanted to add the function result but
got it wrong?

If you want to 'tostring' the elements, you can do it while adding
them to the table, no need to retraverse the table later.

-- Roberto

[Duncan Cross]

On Mon, Dec 6, 2010 at 3:37 PM, Roberto Ierusalimschy

Post by Roberto Ierusalimschy

Post by Duncan Cross
I'm confused by what you mean here. table.concat(), like all the other
table functions like table.sort() and table.insert(), expects to take
a valid Lua array-table and if you pass it one with nils/holes in, it
usually raises no error and just has undefined results. We are
expected to know and understand this for the other functions, why not
table.concat() too?

Sorry, the nil example is not a good one ;) Any other type will do. For
instance, if you add a function to a table and then call 'table.concat',
do you really want to have 'function: 0x9fb7020' in the resulting
string?  Or, more probably, you wanted to add the function result but
got it wrong?
If you want to 'tostring' the elements, you can do it while adding
them to the table, no need to retraverse the table later.
-- Roberto

Ah, I see how the nil example was a bit of a red herring.

I have to admit I'm still unconvinced, though. They are both quite
similar ideas: when you call a table function you are expected not to
break the "contract" that the table should not have any holes, and
when you call table.concat() specifically, you are expected not to
break the "contract" that the table only contain strings. If it's OK
to not throw an error and return something the user might not be
expecting when the first "contract" is broken, it seems like it should
be just as OK for the second one. (Particularly since the second
mistake just seems so much less likely to happen in practice - your
example of forgetting to call the function just doesn't feel like
something that I fall into doing even at my most careless - but this
is a completely subjective observation, I know.)

-Duncan

[Roberto Ierusalimschy]

Post by Duncan Cross
I have to admit I'm still unconvinced, though. They are both quite
similar ideas: when you call a table function you are expected not to
break the "contract" that the table should not have any holes, and
when you call table.concat() specifically, you are expected not to
break the "contract" that the table only contain strings. If it's OK
to not throw an error and return something the user might not be
expecting when the first "contract" is broken, it seems like it should
be just as OK for the second one.

I did not say it is not OK. Both behaviors are OK. The point is what
is more useful: error detection x flexibility. I think that, for %s,
flexibility seems more useful; but for table.concat, I guess error
detection is better.

-- Roberto

[Etan Reisner]

Post by steve donovan
local format = string.format
function formatx (fmt,...)
local args = {...}
local i = 1
for p in fmt:gmatch('%%s') do
if type(args[i]) ~= 'string' then
args[i] = tostring(args[i])
end
i = i + 1
end
return format(fmt,unpack(args))
end

That doesn't work for formats with any non-%s specifiers. Any non-%s
specifier will throw off the i counter for later arguments.

formatx('%s(%x)=%s(%x)',1,5,nil,10)
lua: test.lua:13: bad argument #4 to 'format' (string expected, got nil)

It also isn't safe for argument lists with holes as {}/unpack isn't
guaranteed to preserve the full list when presented with holes, but this
is easy enough to fix with select.

-Etan

[Arseny Vakhrushev]

Post by Etan Reisner
It also isn't safe for argument lists with holes as {}/unpack isn't
guaranteed to preserve the full list when presented with holes, but this
is easy enough to fix with select.

Actually, it works well. I mean {...}/unpack() is fully reversible.

// Seny

[Alexander Gladysh]

On Wed, Dec 8, 2010 at 03:16, Arseny Vakhrushev

Post by Arseny Vakhrushev

Post by Etan Reisner
It also isn't safe for argument lists with holes as {}/unpack isn't
guaranteed to preserve the full list when presented with holes, but this
is easy enough to fix with select.

Actually, it works well. I mean {...}/unpack() is fully reversible.

You're just lucky with your data

$ lua -e 'print(select("#", unpack{ 1, nil, 3 }))'
3

$ luajit2 -e 'print(select("#", unpack{ 1, nil, 3 }))'
1

Alexander.

[Arseny Vakhrushev]

Post by Alexander Gladysh

Post by Arseny Vakhrushev
Actually, it works well. I mean {...}/unpack() is fully reversible.

You're just lucky with your data
$ lua -e 'print(select("#", unpack{ 1, nil, 3 }))'
3
$ luajit2 -e 'print(select("#", unpack{ 1, nil, 3 }))'
1

Well, I was talking about vanilla Lua. Anyway, the above difference is very weird because it doesn't allow to postpone the usage of varargs, for instance, in some coroutine create/resume loop where arguments could be anything generic. Fortunately, I rely on this behavior only within a testing framework which is run under plain Lua.

// Seny

[Miles Bader]

Post by Arseny Vakhrushev

Post by Alexander Gladysh

Post by Arseny Vakhrushev
Actually, it works well. I mean {...}/unpack() is fully reversible.

You're just lucky with your data
$ lua -e 'print(select("#", unpack{ 1, nil, 3 }))'
3
$ luajit2 -e 'print(select("#", unpack{ 1, nil, 3 }))'
1

Well, I was talking about vanilla Lua. Anyway, the above difference is
very weird because it doesn't allow to postpone the usage of varargs,
for instance, in some coroutine create/resume loop where arguments could
be anything generic. Fortunately, I rely on this behavior only within a
testing framework which is run under plain Lua.

Sadly, the same is true of vanilla lua:

$ lua -v
Lua 5.1.4 Copyright (C) 1994-2008 Lua.org, PUC-Rio

$ lua -e 'print(select ("#", unpack{ 1, nil, nil, nil, nil, nil,nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, 3 }))'
1

-Miles

[Arseny Vakhrushev]

Post by Miles Bader

Post by Arseny Vakhrushev
Well, I was talking about vanilla Lua. Anyway, the above difference is
very weird because it doesn't allow to postpone the usage of varargs,
for instance, in some coroutine create/resume loop where arguments could
be anything generic. Fortunately, I rely on this behavior only within a
testing framework which is run under plain Lua.

$ lua -v
Lua 5.1.4 Copyright (C) 1994-2008 Lua.org, PUC-Rio
$ lua -e 'print(select ("#", unpack{ 1, nil, nil, nil, nil, nil,nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, 3 }))'
1

Actually, it slightly differs from what I was talking about. Please consider this:

$ lua -e 'function f(...) local t = { ... } ; print(select("#", unpack(t))) end ; f(1, nil, nil, nil, nil, nil,nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, 3)'
17

$ luajit-2.0.0-beta5 -e 'function f(...) local t = { ... } ; print(select("#", unpack(t))) end ; f(1, nil, nil, nil, nil, nil,nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, 3)'
1

The difference here is that arguments are stored in a table constructed with { ... }.

// Seny

[Michal Kottman]

Post by Miles Bader

Post by Arseny Vakhrushev

Post by Alexander Gladysh

Post by Arseny Vakhrushev
Actually, it works well. I mean {...}/unpack() is fully reversible.

You're just lucky with your data
$ lua -e 'print(select("#", unpack{ 1, nil, 3 }))'
3
$ luajit2 -e 'print(select("#", unpack{ 1, nil, 3 }))'
1

Well, I was talking about vanilla Lua. Anyway, the above difference is
very weird because it doesn't allow to postpone the usage of varargs,
for instance, in some coroutine create/resume loop where arguments could
be anything generic. Fortunately, I rely on this behavior only within a
testing framework which is run under plain Lua.

$ lua -v
Lua 5.1.4 Copyright (C) 1994-2008 Lua.org, PUC-Rio
$ lua -e 'print(select ("#", unpack{ 1, nil, nil, nil, nil, nil,nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, 3 }))'
1

You can always use the additional parameters of unpack to specify how
many values you want to unpack:

$ lua -e 'print(select ("#", unpack({ 1, nil, nil, nil, nil, nil,nil,
nil, nil, nil, nil, nil, nil, nil, nil, nil, 3 }, 1, 17)))'
17

What I tend to do is following:

function f(...)
local args = { n=select('#', ...), ... }
-- do something with args
return unpack(args, 1, args.n)
end

[Arseny Vakhrushev]

Post by Arseny Vakhrushev
function f(...)
local args = { n=select('#', ...), ... }
-- do something with args
return unpack(args, 1, args.n)
end

Exactly! But it seems you do not have to have a separate 'n' field in plain Lua because unpack works well with {...}-constructed tables. For some reason, this behavior is different in LuaJIT.

// Seny

[Mike Pall]

Post by Arseny Vakhrushev
Exactly! But it seems you do not have to have a separate 'n'
field in plain Lua because unpack works well with
{...}-constructed tables. For some reason, this behavior is
different in LuaJIT.

The behavior of # is very well defined, please see the Lua manual.

But it doesn't have a unique result for arrays with holes. Any Lua
implementation may return _any_ of the possible results for it
(e.g. this depends on non-integer keys, too). In fact, it might
return a different result every time you call it!

You simply cannot rely on the result of # for arrays with holes.
Period. So better fix your code and stop wondering.

--Mike

[Roberto Ierusalimschy]

Post by Michal Kottman
local args = { n=select('#', ...), ... }

Lua 5.2 now offers table.pack to do that:

local args = table.pack(...)

-- Roberto

[Arseny Vakhrushev]

Post by Michal Kottman
$ lua -e 'print(select ("#", unpack({ 1, nil, nil, nil, nil, nil,nil,
nil, nil, nil, nil, nil, nil, nil, nil, nil, 3 }, 1, 17)))'
17
function f(...)
local args = { n=select('#', ...), ... }
-- do something with args
return unpack(args, 1, args.n)
end

I am sorry! I was wrong. You really need to store the arg count here in either Lua.

This is what the manual says on unpack: "... By default, i is 1 and j is the length of the list, as defined by the length operator."

// Seny

[steve donovan]

On Tue, Dec 7, 2010 at 9:25 PM, Etan Reisner

Post by Etan Reisner
That doesn't work for formats with any non-%s specifiers. Any non-%s
specifier will throw off the i counter for later arguments.

You're right, it's a stupid piece of code. Thanks for the
correction.The gmatch should be '%%.' and then there should be an
explicit match for '%s'.

However, {...} should work, if we work with the format specifiers.

steve d.

[Doug Currie]

Post by HyperHacker

t={}
setmetatable(t, {__tostring = function(t) return "xxx" end})
print(t)

xxx

print(('%s'):format(t))

stdin:1: bad argument #1 to 'format' (string expected, got table)
Why does string.format not call tostring() on arguments given as %s?

For a patch, see:

http://lua-users.org/lists/lua-l/2006-10/msg00020.html

e