Lua Needs

[Soni "They/Them" L.]

I would like to use Lua again, but it falls short on some of my needs:

- The GC is not reentrant, as such it cannot do garbage collection or
call debug hooks inside __gc hooks. This is a dealbreaker, as it has
serious consequences for sandboxing.
- Lua has little to no support for composition-based OOP, whereas it has
many features to support inheritance (__index/__newindex) and other
forms of OOP (self:methods()). This isn't a big deal, tho, as it can be
easily solved with a rather small patch.

These would be nice to have.

[Pierre Chapuis]

Post by Soni "They/Them" L.
- Lua has little to no support for composition-based OOP, whereas it has
many features to support inheritance (__index/__newindex) and other
forms of OOP (self:methods()). This isn't a big deal, tho, as it can be
easily solved with a rather small patch.

Can you give an example of what you would want here?

Because for me, "composition-based OOP" does not need much
language support. For what it's worth I almost never use inheritance
in Lua (or any other language, really).

[Soni "They/Them" L.]

Post by Pierre Chapuis

Post by Soni "They/Them" L.
- Lua has little to no support for composition-based OOP, whereas it has
many features to support inheritance (__index/__newindex) and other
forms of OOP (self:methods()). This isn't a big deal, tho, as it can be
easily solved with a rather small patch.

Can you give an example of what you would want here?
Because for me, "composition-based OOP" does not need much
language support. For what it's worth I almost never use inheritance
in Lua (or any other language, really).

The classic Cratera[1] `foo:[component].method()` (note the colon) syntax.

[1] https://bitbucket.org/TeamSoni/cratera

[Philippe Verdy]

Post by Soni "They/Them" L.
The classic Cratera[1] `foo:[component].method()` (note the colon) syntax.
[1] https://bitbucket.org/TeamSoni/cratera

Write it in Lua as
foo[component].method(foo), or
foo[component]['method'](foo)

I don't see what the "cratera" fork brings which is not already part of
standard Lua...
You can still use an expression to replace any identifier, just replace the
" .identifier " syntax by " ['identifier'] ".
The colon is just a syntaxic sugar in Lua for passing an object (a table)
in the first parameter of a function which is one of the property values of
the object). And there's not even any need to change the VM to support it,
you can always rewrite the expression with colon into a function call.

The interest of the colon however is that this common expression (for
computing the reference of the object) is coimputed only once and
duplicated (kept in an hidden temporary variable or register) before first
dereferencing it to extract one of its property (which is a function) and
then reuse it as the first parameter to call the function.
It only exists because Lua does not "directly" allow using assignments in
the middle of an an expression.

But there's a way to perform an assignment in a local variable in the
middle of an expression in standard Lua. For example the following is not
allowed by the syntax:

y = (local x; sin(x = arbitrary_expression()) + cos(x))

But you can rewrite it using a local function in a closure to perform the
assignment by calling that function:

local x;
y = sin( (function(){x = arbitrary_expression()})() ) + cos(x)

or more equivalently (to limit the scope of "x" inside the expression) as:

y = (function() {local x; return sin( (function(){x =
arbitrary_expression()})() ) + cos(x)})()

Here also you don't need any change to Lua or its interpreter or compiler,
or to the VM. The possibility of performing asignments in the middle of
expressions, and even adding local varaible declaration would just make it
more easier (temporary variables are frequently needed to store the results
of common subexpressions that you MUST NOT evaluate multiple times without
introiducing side-effects, notably if the "common subexpression contains
function calls, for example performing some I/O or modifying other
referenced objects or other objects hiden in the function closure).

Having the possibility of directly declaring local variables in the middle
of expresssions would be useful and productive (and it would perform
various optimizations that are hard to realize in interpreters and
compilers, that have to perform complex matching search to detect common
subexpressions and check if their semantics is really equivalent).

[Egor Skriptunoff]

Post by Soni "They/Them" L.

Post by Soni "They/Them" L.
- Lua has little to no support for composition-based OOP
Can you give an example of what you would want here?

The classic Cratera[1] `foo:[component].method()` (note the colon) syntax.

What OOP languages do support this "composition-based OOP" ?

[Coda Highland]

On Mon, Nov 26, 2018 at 6:58 AM Egor Skriptunoff

Post by Egor Skriptunoff

Post by Soni "They/Them" L.

Post by Pierre Chapuis

Post by Soni "They/Them" L.
- Lua has little to no support for composition-based OOP

Can you give an example of what you would want here?

The classic Cratera[1] `foo:[component].method()` (note the colon) syntax.

What OOP languages do support this "composition-based OOP" ?

Lots of them. C++, Java, C#, Python... Javascript doesn't usually do
it that way but its ability to do dynamic function rebinding would
make it POSSIBLE to do it there (I just don't know anyone that
bothers).

The syntax isn't usually that pretty, though. Python's the only one I
know of off the top of my head that can make it look that clean. In C#
it looks more like foo.GetComponent<Component>().Method().

Lua actually can pull it off with foo[component]:method() if you have
foo's __index metamethod return a binding object, but that comes at
the cost of a fair amount of overhead relative to the syntax patch.

/s/ Adam

[Egor Skriptunoff]

Post by Soni "They/Them" L.

Post by Egor Skriptunoff

Post by Soni "They/Them" L.

Post by Pierre Chapuis

Post by Soni "They/Them" L.
Lua has little to no support for composition-based OOP

Can you give an example of what you would want here?

The classic Cratera[1] `foo:[component].method()` (note the colon)

syntax.

Post by Egor Skriptunoff
What OOP languages do support this "composition-based OOP" ?

Lots of them. C++, Java, C#, Python...
Python's the only one I know of off the top of my head
that can make it look that clean.

How does Python support "composition OOP"?
Do you mean the ability to define __getattr__?
It doesn't look like a "support"; a user has to do all the work manually.
It' very similar to Lua's ability to define __index.

I like the suggestion to make "a:b.c()" a valid Lua syntax.

But the next question from Sony L. would be the following:
How to save intermediate value in this code
a:b.c()
a:b.d()
to rewrite it in optimized way
local x = a:b
x.c()
x.d()

[Coda Highland]

On Wed, Nov 28, 2018 at 4:06 AM Egor Skriptunoff

Post by Egor Skriptunoff

Post by Coda Highland

Post by Egor Skriptunoff

Post by Soni "They/Them" L.

Post by Pierre Chapuis

Post by Soni "They/Them" L.
Lua has little to no support for composition-based OOP

Can you give an example of what you would want here?

The classic Cratera[1] `foo:[component].method()` (note the colon) syntax.

What OOP languages do support this "composition-based OOP" ?

Lots of them. C++, Java, C#, Python...
Python's the only one I know of off the top of my head
that can make it look that clean.

How does Python support "composition OOP"?
Do you mean the ability to define __getattr__?
It doesn't look like a "support"; a user has to do all the work manually.
It' very similar to Lua's ability to define __index.

As I said in another post, by default you don't explicitly call out
the component you want to use. Python accomplishes composition via
multiple inheritance ("mixins"), so you'd just write a.c() instead of
a.b.c().

But you're right, being able to tinker with Python's metaclass
functionality is how Python is able to achieve something analogous to
the specific technique under discussion. And yes, it IS like being
able to define __index, except since Python has a native notion of
classes there's a different attitude about doing so.

Post by Egor Skriptunoff
I like the suggestion to make "a:b.c()" a valid Lua syntax.
How to save intermediate value in this code
a:b.c()
a:b.d()
to rewrite it in optimized way
local x = a:b
x.c()
x.d()

(Nitpick: it would have to be x:c(), not x.c().)

I hinted at the same notion earlier. The problem with doing this at
all is that it's EXPENSIVE -- probably more expensive than just
evaluating the two paths in the first place. You have to allocate an
object to represent the binding, such that using . or : to access its
contents returns b, but using () passes through a. And then you have
to make the function call machinery detect when one of these objects
is being used in order to make sure the right value is passed in the
parameter list. And of course, this allocation will subsequently need
to be garbage-collected... It's really not worth it, especially since
this overhead would apply to ALL uses of : and ALL uses of () even if
it's not needed unless you add EVEN MORE complexity to the parser.

In practice it would be better to write this:

local x = a.b
x.c(a)
x.d(a)

This works with stock Lua or with the proposed a:b.c() syntax patch
and doesn't involve any additional allocations.

/s/ Adam

[Soni L.]

In practice it's better to use:

local a = a
a:b.c()
a:d.e()

Because they're components and as such are a property of the object. They
may change and as such you should not cache them. Consider them part of the
object, rather than separate objects.

It's okay to cache their functions tho, as expected.

Post by Coda Highland
On Wed, Nov 28, 2018 at 4:06 AM Egor Skriptunoff

Post by Egor Skriptunoff

Post by Coda Highland

Post by Egor Skriptunoff

Post by Soni "They/Them" L.

Post by Pierre Chapuis

Post by Soni "They/Them" L.
Lua has little to no support for composition-based OOP

Can you give an example of what you would want here?

The classic Cratera[1] `foo:[component].method()` (note the colon)

syntax.

Post by Egor Skriptunoff

Post by Coda Highland

Post by Egor Skriptunoff
What OOP languages do support this "composition-based OOP" ?

Lots of them. C++, Java, C#, Python...
Python's the only one I know of off the top of my head
that can make it look that clean.

How does Python support "composition OOP"?
Do you mean the ability to define __getattr__?
It doesn't look like a "support"; a user has to do all the work manually.
It' very similar to Lua's ability to define __index.

As I said in another post, by default you don't explicitly call out
the component you want to use. Python accomplishes composition via
multiple inheritance ("mixins"), so you'd just write a.c() instead of
a.b.c().
But you're right, being able to tinker with Python's metaclass
functionality is how Python is able to achieve something analogous to
the specific technique under discussion. And yes, it IS like being
able to define __index, except since Python has a native notion of
classes there's a different attitude about doing so.

Post by Egor Skriptunoff
I like the suggestion to make "a:b.c()" a valid Lua syntax.
How to save intermediate value in this code
a:b.c()
a:b.d()
to rewrite it in optimized way
local x = a:b
x.c()
x.d()

(Nitpick: it would have to be x:c(), not x.c().)
I hinted at the same notion earlier. The problem with doing this at
all is that it's EXPENSIVE -- probably more expensive than just
evaluating the two paths in the first place. You have to allocate an
object to represent the binding, such that using . or : to access its
contents returns b, but using () passes through a. And then you have
to make the function call machinery detect when one of these objects
is being used in order to make sure the right value is passed in the
parameter list. And of course, this allocation will subsequently need
to be garbage-collected... It's really not worth it, especially since
this overhead would apply to ALL uses of : and ALL uses of () even if
it's not needed unless you add EVEN MORE complexity to the parser.
local x = a.b
x.c(a)
x.d(a)
This works with stock Lua or with the proposed a:b.c() syntax patch
and doesn't involve any additional allocations.
/s/ Adam

[Philippe Verdy]

Post by Soni L.
local a = a
a:b.c()
a:d.e()
Because they're components and as such are a property of the object. They
may change and as such you should not cache them. Consider them part of the
object, rather than separate objects.

you could have the case

local a = a
a:b.d()
a:c.d()

where the desire is to have a way to select one of the "d" component
methods (or properties) independantly of the path (possibly complex and
resulting from a function call) from which it is accessed, i.e. something
like:

local a = a
local x = test and a:b or a:c
x:d()

The function+closure based solution I proposed also works to hold the two
references (the base object, and the target function which is any component
accessible from the base object via an unknown path, when this targetr
function needs in its first parameter a reference to the base object) for
this case:

local a = a

Post by Soni L.
local x = function(a, b, c)

local __o___ = a;

Post by Soni L.
return function(...)
return (test and b or c)(__o__, ...)
end
end
x(d)

imagine that "d" is a generic "get" or "set" function used to define some
property or subproperty of an object. Such concept is used in oop based on
facets (I've used it on a L4G programming language where there was no
"method" at all, only dynamic properties.

But such concept is not dead and is widely used for implementing UI
components, that are exposing only properties to define the behavior and
content/visible aspect of the UI component and propagate all side effects
in a component tree, without forcing users to really write any line of
code, only get/set values of properties, these values having simple types :
generally a single string, a numeric or enum value, a color value, or a
locale identifier (with its implied BCP 47 fallback semantics, something
that strings alone don't have; such value having also other subproperties
like the language code, the script code, or sets of other locales that the
locale identifer match during fallbacks). Several "Visual" programming
languages (and thier associated UI designer) use such concept with the
additional benefit that they are generally easy to serialize (objects that
have programmable method facets are much harder to serialize correctly and
safely)

[Coda Highland]

Post by Soni L.
local a = a
local x = test and a:b or a:c
x:d()

I mean...

a:[test and 'b' or 'c'].d()

/s/ Adam

[Philippe Verdy]

Yes, that's exactly what I understood: you have a variable/conditional
"path" to the property of object "a" that interest you, and then "d()" is a
method on that property that must be used where you want to it to act on
the object "a".

My approach in pure Lua, using a closure to enclose the value of "a", plus
a function in Lua to build an intermediate object should work for your case
even if you modified it:

a:[test and 'b' or 'c'].d("additional parameters:", {12, 34}, 'sample')

which is also writable in Lua as:

(function(a)
local __o__ = a;
return function(...)
return (__o__[test and 'b' or 'c']).d(__o__, ...)
end
end)(d,"additional parameters:", {12, 34}, 'sample')

See how I added also the other parameters (which are passed to the internal
function here using a vararg, but the vararg is not mandatory).

The typical usage would be to create dynamic properties in a OOP language
with facets:

value = (some expression selecting an object):[some expression
computing a property name].get();

(some expression selecting an object):[some expression selecting a
property name in the object].set(value);

which can be written as:

value = (function(o)
local __o___ = o;
return function() return __o__.get(__o__[some expression selecting
a property name in the object]) end
end)(get, some expression selecting an object);

(function(o, v)
local __o___ = o;
return function() return __o__.set(__o__[some expression selecting
a property name in the object]) end
end)(some expression selecting an object, value);

There's many variants possible depending on which part is variable or not
or dependant on the object that the "apparently simple" syntax
a:b.c(d)
does not disambiguate clearly. And this would be even worse if we allowed
this:
a:b:c.d(e)
which could be understood differently as any one of:
((a:b):c).d(e)
(a:(b:c)).d(e)
a:((b:c).d)(e)
depending on associativity, and which object must be passed to the final
dynamic method named "d" here.

Post by Coda Highland

Post by Soni L.
local a = a
local x = test and a:b or a:c
x:d()

I mean...
a:[test and 'b' or 'c'].d()
/s/ Adam

[Philippe Verdy]

For this reason, the generalization of the ":" pseudo-operator of Lua is
not easy: it would create lot of overhead at compile-time and runtime to
create many closure objects, that are compiled using a static "prototype"
which is an integer-indexed array mapping the numbered upvalues to their
bound object, and then used at runtime to bind each actual upvalues to
other accessible external objects

The ":" pseudo-operator is a syntaxic sugar of Lua which can always be
rewritten using the "." pseudo-operator.
But as well the "." pseudo-operator is a syntax sugar of Lua which can
always be rewritten using the "[]" table indexing operator (this one is the
only real operator).
Combining the two sugars (pseudo-operators ":" and ".") needs to be limited
to avoid ambiguities.

The only safe way (without needing the overhead of instanciating too many
additional closures) is to declare local variables in separate statements
to precompute the object references you need to pass further down for
forming the function call. You can finally always resolve all cases using
the "[]" table indexing operator only.

Then use the final "()" function call operator where you'll pass the
explicit references by these temporary variables and minimize the number of
needed closures (because they are costly on the heap, and may then be quite
slow if closures are used extensively in tight loops with many repetitions).

---

Note that each closure is an actual object (a simple indexed array, not a
full table with random keys) that is allocated on the heap (then
instanciated by binding the references to the actual variables), so it has
a runtime cost on the garbage collector, each time we call any Lua function
that uses any upvalues or returns a list with more than one value).
This object needs then to be garbage-collected when exiting the closure
(after each function call). No such closure needs to be allocated is the
function has no use of external variables in its lexical scope and uses
only its parameters and the reference to its own metatable which keeps data
across all calls from the same thread.

The function's own metatable is then like a "thread local storage" (TLS)
that does not need to be allocated at each call; this TLS is used only for
data that can be modified, it does not contain static data which is stored
in its constant pool, including the static prototype built by the Lua
compiler and which is used to instanciate the mapping for the closure of
the function to its lexical context). This metatable also contains a
reference to its external dynamic environment (not bound directly by the
closure). C functions are a bit different because instead of a full table,
their environment is an indexed array of userdata, but userdata bound to c
functions are also thread local storage, allocated on the heap only once
per thread. These TLS avoid the extra cost of closures at each call by
limiting a lot the use of heap allocation.

If you have a program that makes many loops across function calls and you
see that it uses lot of heap and garbage collector is too much sollicited,
it may help to see if you can avoid the closures and use TLS instead, i.e.
the environment of functions (which is unique per thread, and kept across
calls withoout being allocated and garbage collected at each call like
closures).

Post by Philippe Verdy
Yes, that's exactly what I understood: you have a variable/conditional
"path" to the property of object "a" that interest you, and then "d()" is a
method on that property that must be used where you want to it to act on
the object "a".
My approach in pure Lua, using a closure to enclose the value of "a", plus
a function in Lua to build an intermediate object should work for your case
a:[test and 'b' or 'c'].d("additional parameters:", {12, 34}, 'sample')
(function(a)
local __o__ = a;
return function(...)
return (__o__[test and 'b' or 'c']).d(__o__, ...)
end
end)(d,"additional parameters:", {12, 34}, 'sample')
See how I added also the other parameters (which are passed to the
internal function here using a vararg, but the vararg is not mandatory).
The typical usage would be to create dynamic properties in a OOP language
value = (some expression selecting an object):[some expression
computing a property name].get();
(some expression selecting an object):[some expression selecting a
property name in the object].set(value);
value = (function(o)
local __o___ = o;
return function() return __o__.get(__o__[some expression
selecting a property name in the object]) end
end)(get, some expression selecting an object);
(function(o, v)
local __o___ = o;
return function() return __o__.set(__o__[some expression
selecting a property name in the object]) end
end)(some expression selecting an object, value);
There's many variants possible depending on which part is variable or not
or dependant on the object that the "apparently simple" syntax
a:b.c(d)
does not disambiguate clearly. And this would be even worse if we allowed
a:b:c.d(e)
((a:b):c).d(e)
(a:(b:c)).d(e)
a:((b:c).d)(e)
depending on associativity, and which object must be passed to the final
dynamic method named "d" here.

Post by Coda Highland

Post by Soni L.
local a = a
local x = test and a:b or a:c
x:d()

I mean...
a:[test and 'b' or 'c'].d()
/s/ Adam

[Philippe Verdy]

Note, my use of "TLS" here is a bit abusive, because it is not strictly
thread-local: objects referenced in closures or function environemnt is
accessible in other threads by communicating them via yield(...) or
resume(...), because they are all allocated on the same heap shared by all
threads/coroutines created from the first one which is started by the
instance of the Lua engine (which runs all its "thread" objects in the same
OS thread).

But "TLS" or "thread local storage" must be understood by interpreting the
"thread" term as being "Lua thread/coroutine", where each one can have its
own separate storage; there's no limitation however for allowing other
threads to use and modify these storages if they are exchanged by
yield(...) and resume(...), passing their references via the stack of each
thread. Stacks are very fast compared to the heap and you can even get much
higher performance by using yield() and resume() to invoke threads running
a service loop, instead of calling functions and passing references by
closures.

Post by Philippe Verdy
For this reason, the generalization of the ":" pseudo-operator of Lua is
not easy: it would create lot of overhead at compile-time and runtime to
create many closure objects, that are compiled using a static "prototype"
which is an integer-indexed array mapping the numbered upvalues to their
bound object, and then used at runtime to bind each actual upvalues to
other accessible external objects
The ":" pseudo-operator is a syntaxic sugar of Lua which can always be
rewritten using the "." pseudo-operator.
But as well the "." pseudo-operator is a syntax sugar of Lua which can
always be rewritten using the "[]" table indexing operator (this one is the
only real operator).
Combining the two sugars (pseudo-operators ":" and ".") needs to be
limited to avoid ambiguities.
The only safe way (without needing the overhead of instanciating too many
additional closures) is to declare local variables in separate statements
to precompute the object references you need to pass further down for
forming the function call. You can finally always resolve all cases using
the "[]" table indexing operator only.
Then use the final "()" function call operator where you'll pass the
explicit references by these temporary variables and minimize the number of
needed closures (because they are costly on the heap, and may then be quite
slow if closures are used extensively in tight loops with many repetitions).
---
Note that each closure is an actual object (a simple indexed array, not a
full table with random keys) that is allocated on the heap (then
instanciated by binding the references to the actual variables), so it has
a runtime cost on the garbage collector, each time we call any Lua function
that uses any upvalues or returns a list with more than one value).
This object needs then to be garbage-collected when exiting the closure
(after each function call). No such closure needs to be allocated is the
function has no use of external variables in its lexical scope and uses
only its parameters and the reference to its own metatable which keeps data
across all calls from the same thread.
The function's own metatable is then like a "thread local storage" (TLS)
that does not need to be allocated at each call; this TLS is used only for
data that can be modified, it does not contain static data which is stored
in its constant pool, including the static prototype built by the Lua
compiler and which is used to instanciate the mapping for the closure of
the function to its lexical context). This metatable also contains a
reference to its external dynamic environment (not bound directly by the
closure). C functions are a bit different because instead of a full table,
their environment is an indexed array of userdata, but userdata bound to c
functions are also thread local storage, allocated on the heap only once
per thread. These TLS avoid the extra cost of closures at each call by
limiting a lot the use of heap allocation.
If you have a program that makes many loops across function calls and you
see that it uses lot of heap and garbage collector is too much sollicited,
it may help to see if you can avoid the closures and use TLS instead, i.e.
the environment of functions (which is unique per thread, and kept across
calls withoout being allocated and garbage collected at each call like
closures).

Post by Philippe Verdy
Yes, that's exactly what I understood: you have a variable/conditional
"path" to the property of object "a" that interest you, and then "d()" is a
method on that property that must be used where you want to it to act on
the object "a".
My approach in pure Lua, using a closure to enclose the value of "a",
plus a function in Lua to build an intermediate object should work for your
a:[test and 'b' or 'c'].d("additional parameters:", {12, 34}, 'sample')
(function(a)
local __o__ = a;
return function(...)
return (__o__[test and 'b' or 'c']).d(__o__, ...)
end
end)(d,"additional parameters:", {12, 34}, 'sample')
See how I added also the other parameters (which are passed to the
internal function here using a vararg, but the vararg is not mandatory).
The typical usage would be to create dynamic properties in a OOP language
value = (some expression selecting an object):[some expression
computing a property name].get();
(some expression selecting an object):[some expression selecting a
property name in the object].set(value);
value = (function(o)
local __o___ = o;
return function() return __o__.get(__o__[some expression
selecting a property name in the object]) end
end)(get, some expression selecting an object);
(function(o, v)
local __o___ = o;
return function() return __o__.set(__o__[some expression
selecting a property name in the object]) end
end)(some expression selecting an object, value);
There's many variants possible depending on which part is variable or not
or dependant on the object that the "apparently simple" syntax
a:b.c(d)
does not disambiguate clearly. And this would be even worse if we allowed
a:b:c.d(e)
((a:b):c).d(e)
(a:(b:c)).d(e)
a:((b:c).d)(e)
depending on associativity, and which object must be passed to the final
dynamic method named "d" here.

Post by Coda Highland

Post by Soni L.
local a = a
local x = test and a:b or a:c
x:d()

I mean...
a:[test and 'b' or 'c'].d()
/s/ Adam

[Coda Highland]

For this reason, the generalization of the ":" pseudo-operator of Lua is not easy: it would create lot of overhead at compile-time and runtime to create many closure objects, that are compiled using a static "prototype" which is an integer-indexed array mapping the numbered upvalues to their bound object, and then used at runtime to bind each actual upvalues to other accessible external objects

I agree, and have agreed since the beginning: the generalization is
inappropriate. Limit it to only using one per function call expression
and don't have it generate any sort of extra binding object; it simply
defines which object in the lookup is the one passed to the "self"
parameter. This is easy, efficient, useful, and most importantly
doesn't introduce overhead for the existing use case.

/s/ Adam

[Soni L.]

I think you're just overcomplicating it tbh

Post by Philippe Verdy
For this reason, the generalization of the ":" pseudo-operator of Lua is
not easy: it would create lot of overhead at compile-time and runtime to
create many closure objects, that are compiled using a static "prototype"
which is an integer-indexed array mapping the numbered upvalues to their
bound object, and then used at runtime to bind each actual upvalues to
other accessible external objects
The ":" pseudo-operator is a syntaxic sugar of Lua which can always be
rewritten using the "." pseudo-operator.
But as well the "." pseudo-operator is a syntax sugar of Lua which can
always be rewritten using the "[]" table indexing operator (this one is the
only real operator).
Combining the two sugars (pseudo-operators ":" and ".") needs to be
limited to avoid ambiguities.
The only safe way (without needing the overhead of instanciating too many
additional closures) is to declare local variables in separate statements
to precompute the object references you need to pass further down for
forming the function call. You can finally always resolve all cases using
the "[]" table indexing operator only.
Then use the final "()" function call operator where you'll pass the
explicit references by these temporary variables and minimize the number of
needed closures (because they are costly on the heap, and may then be quite
slow if closures are used extensively in tight loops with many repetitions).
---
Note that each closure is an actual object (a simple indexed array, not a
full table with random keys) that is allocated on the heap (then
instanciated by binding the references to the actual variables), so it has
a runtime cost on the garbage collector, each time we call any Lua function
that uses any upvalues or returns a list with more than one value).
This object needs then to be garbage-collected when exiting the closure
(after each function call). No such closure needs to be allocated is the
function has no use of external variables in its lexical scope and uses
only its parameters and the reference to its own metatable which keeps data
across all calls from the same thread.
The function's own metatable is then like a "thread local storage" (TLS)
that does not need to be allocated at each call; this TLS is used only for
data that can be modified, it does not contain static data which is stored
in its constant pool, including the static prototype built by the Lua
compiler and which is used to instanciate the mapping for the closure of
the function to its lexical context). This metatable also contains a
reference to its external dynamic environment (not bound directly by the
closure). C functions are a bit different because instead of a full table,
their environment is an indexed array of userdata, but userdata bound to c
functions are also thread local storage, allocated on the heap only once
per thread. These TLS avoid the extra cost of closures at each call by
limiting a lot the use of heap allocation.
If you have a program that makes many loops across function calls and you
see that it uses lot of heap and garbage collector is too much sollicited,
it may help to see if you can avoid the closures and use TLS instead, i.e.
the environment of functions (which is unique per thread, and kept across
calls withoout being allocated and garbage collected at each call like
closures).

Post by Philippe Verdy
Yes, that's exactly what I understood: you have a variable/conditional
"path" to the property of object "a" that interest you, and then "d()" is a
method on that property that must be used where you want to it to act on
the object "a".
My approach in pure Lua, using a closure to enclose the value of "a",
plus a function in Lua to build an intermediate object should work for your
a:[test and 'b' or 'c'].d("additional parameters:", {12, 34}, 'sample')
(function(a)
local __o__ = a;
return function(...)
return (__o__[test and 'b' or 'c']).d(__o__, ...)
end
end)(d,"additional parameters:", {12, 34}, 'sample')
See how I added also the other parameters (which are passed to the
internal function here using a vararg, but the vararg is not mandatory).
The typical usage would be to create dynamic properties in a OOP language
value = (some expression selecting an object):[some expression
computing a property name].get();
(some expression selecting an object):[some expression selecting a
property name in the object].set(value);
value = (function(o)
local __o___ = o;
return function() return __o__.get(__o__[some expression
selecting a property name in the object]) end
end)(get, some expression selecting an object);
(function(o, v)
local __o___ = o;
return function() return __o__.set(__o__[some expression
selecting a property name in the object]) end
end)(some expression selecting an object, value);
There's many variants possible depending on which part is variable or not
or dependant on the object that the "apparently simple" syntax
a:b.c(d)
does not disambiguate clearly. And this would be even worse if we allowed
a:b:c.d(e)
((a:b):c).d(e)
(a:(b:c)).d(e)
a:((b:c).d)(e)
depending on associativity, and which object must be passed to the final
dynamic method named "d" here.

Post by Coda Highland

Post by Soni L.
local a = a
local x = test and a:b or a:c
x:d()

I mean...
a:[test and 'b' or 'c'].d()
/s/ Adam

[Philippe Verdy]

Post by Soni L.
I think you're just overcomplicating it tbh

That's your opinion. It's a fact that "." and ":" are syntaxic sugars of
Lua and are not real operators. They cannot interact freely But all can be
made with closures (for binding the upvalues) and with the "[]" indexing
operator (in tables). And a fact as well that "self" is not a keyword but a
conventional name of the first parameter of a function that can be called
with ":".

"self" is just the *implicit* parameter name used when we DECLARE a
function with the ":" syntaxic sugar. We an do everything in Lua without
ever using "." or ":", but only using "[]" and passing all parameters
explicitly. for this reason "self" is not even a reserved keyword (it can
be renamed as we want if we don't use the ":"-enhanced function
declaration).

Once you understand it, there are three ways to pass parameters to
functions:
- by parameters on stack, which become "registers" just like other local
variables declared with "local" in the function body (the fastest method,
but limited to ~250 variables): this includes the "self" parameter.
- by the function's metatable, which includes its environment (this is fast
too because it is allocated on the heap only once by thread and preserved
across multiple calls in the same thread/coroutine: there's no limit on the
number of parameters): this includes the access to the "global" environment
(actually not global, it's not static like in C but scoped recursively and
specific to each thread; it's sort of "TLS" except that multiple threads
can communicate objects as all these objects are allocated on the same
global heap using the same Allocator, if all threads are created in Lua
from the same main thread instanciated in C by the Lua engine)
- by external variables bound into upvalues with the closure (it is the
slowest method because the closure is allocated on each call and needs to
be garbage collected and it stresses a lot the garbage collector: this woks
for returning multiple values from the function)

The closures are interesting and facilitates a lot programming in Lua, but
it has a significant runtime cost and stresses the GC at each call.

If there's something to do to the Lua engine is to rework the way upvalues
are allocated: they should be on the call stack as much as possible (e.g.
for up to ~250 upvalues) and not on the heap. This would dramatically
increase the performance of closures and reduce a lot the stress on the GC,
they would become as fast as function parameters and local variables,
working like "registers" (with negative index), and would simplify also the
instruction set for the bytecode and the complexity of the compiler, which
would also not need to create static "prototypes" for each closure and make
any run-time binding (but this would require to increase a bit the size of
stacks per thread (stacks are allocated only once on the heap before the
thread starts): this increase would have no cost because we save lot of
uses on the heap and would save lot of work that needs to be made by the
GC... The global memory footprint would be much lower wioth higher
performance too.

[Dirk Laurie]

If there's something to do to the Lua engine is to rework the way upvalues are allocated: they should be on the call stack as much as possible (e.g. for up to ~250 upvalues) and not on the heap.

The way I understand it, upvalues are not on the heap. They are on the
main execution stack below the bottom of the current function's stack
frame.

[Philippe Verdy]

Ask yourself what are LUA_TCLOSURE objects (yes in the heap...) and see how
they are correlated with the static "prototypes" generated by the compiler
which instructs how to bind caller's variables to callee's upvalues. The
actual values are on the stack but at unknown position from the callee, so
a mapping LUA_TCLOSURE object is allocated on the heap.

Ask yourself why these closure objects exist: this is because functions may
be recursive (and not necessarily by trailing calls), so the variables in
closures may refer not to the immediate parent caller frame but to some
ancestor frame at arbitrary number of levels in the calls stack). To avoid
every access to the closure varaible to pass through a chain, there's a
need for a mapping which is created and initialized before each call to
rebind each variable for the next inner call (according to the closure's
prototype): such object is allocated only if there are external variables
used by the function that are not local to the function itself, they are
not directly within the stack window.

This explains also because the bytecode needs separate opcodes for
accessing registers and upvalues: if they could be directly on the stack,
it would be enough to reference upvalues with negative register indexes and
then use the stack as a sliding window, like we do in C/C++ call
conventions (except that stack indexes in C/C++ are negative for
parameters, and positive for local variables, some of the former being
cached in actual registers, but still supported by "shadow" variables on
the stack, allocated at fixed positions in the stack fram by the compiler
or just pushed before the actual parameters and popped to restore these
registers after the call).

There's been performance tests that show that closures are not so fast,
they can create massive amounts of garbage collected objects (with internal
type LUA_TCLOSURE). I think this behavior very curious, and the current
implementation that allocates the LUA_TCLOSURE objects on the heap is not
the best option, the mapping could be allocated directly in the stack of
local variables/registers of the caller (and all these closure objects used
by the caller could be merged to a single one, i.e. as the largest closure
object needed for inner calls, merged like in an union. The closure objects
themselves to not hold any variable value, these are just simple mappings
from a small fixed integer sets (between 1 and the number of upvalues of
the called function) and variable integers (absolute indexes in the
thread's stack where the actual variable is located).

The byte code is not as optimized as it could be: the register numbers are
only positive, the upvalue numbers are also only positive, they could forml
a single set (positive integers for local registers, negative integers for
upvalues, meaning that they are used to index the entries in the closure
object to get access to the actual varaible located anywhere on the stack,
outside of the immediate parent frame). The generated bytecode is not as
optimal as it could be because various operations can only work on
registers or constants (like ADD r1,r2,r3) so temporary registers must be
allocated by the compiler (let's remember that the number of registers is
limited). As well Lua's default engine treat all registers the same, when
most of them will work with a single r0 register (an "accumulator") which
could be implicit for most instructions, and this would reduce the
instruction sizes (currently 32-bit or 64 bits), which is inefficient as it
uses too much the CPU L1 data cache.

I'm convinced that the current approach in the existing Lua VM engine and
its internal instruction set can be largely improved for better
performance, without really changing the language itself, to get better
data locality (smaller instruction sizes, less wasted unused bit fields),
and elimination of uses of the heap for closures (to dramatically reduce
the stress on the garbage collector)

Post by Philippe Verdy
If there's something to do to the Lua engine is to rework the way

upvalues are allocated: they should be on the call stack as much as
possible (e.g. for up to ~250 upvalues) and not on the heap.
The way I understand it, upvalues are not on the heap. They are on the
main execution stack below the bottom of the current function's stack
frame.

[Soni "They/Them" L.]

Sorry, accidentally replied directly instead of reply to list >.<

I still stand by "Closures = overcomplicating it" with regards to
"foo:bar()" and "foo:bar.baz()".

In any case, I'm not sure what your goal is here... ¯\_(?)_/¯

Closures are naturally complicate to understand it you still don't see
that they are themselves objects (even if you cannot create any
closure-type object in Lua directly because they are created by the
compiler and handled by the underlying VM).
Closures are instrinsic internal constructions of the language (which
refers to them by speaking about "lexical scopes", but at runtime the
"lexical" scope is non-sense, it is backed by a effective structure.
Look at Lua sources and search for "Kproto" and "LUA_TCLOSURE" in
these sources: you'll find them only in the sources of the VM engine
itself, and created by the compiler, from the abstract syntaxic tree
AFTER parsing the language source, to determien when and where to
create these objects.
Look at the LuaC interface and the description of the "Lua_Allocator"
type (in C only, not in Lua itself where it is always inaccessible).
closures have a very significant impact and cause lot of work to be
done in the GC, even in the smallest Lua programs that just run a
single function around a large loop and don't allocate any object,
don't modify or add any table entry. Look at how the bytecodes are
builtup, distinshing upvalues and registers with distinct opcodes even
if the value in them are the same type (e.g. number or string).
What is overcomplicating things is the Lua engine itself which is not
optimized as much as it could to avoid most costs on heap usage and
should maximum reuse of objects instead of constantly allocating and
discarding them in the same thread!
Closures = overcomplicating it.

Post by Philippe Verdy
Ask yourself what are LUA_TCLOSURE objects (yes in the heap...)
and see how they are correlated with the static "prototypes"
generated by the compiler which instructs how to bind caller's
variables to callee's upvalues. The actual values are on the
stack but at unknown position from the callee, so a mapping
LUA_TCLOSURE object is allocated on the heap.
Ask yourself why these closure objects exist: this is because
functions may be recursive (and not necessarily by trailing
calls), so the variables in closures may refer not to the
immediate parent caller frame but to some ancestor frame at
arbitrary number of levels in the calls stack). To avoid every
access to the closure varaible to pass through a chain, there's a
need for a mapping which is created and initialized before each
call to rebind each variable for the next inner call (according
to the closure's prototype): such object is allocated only if
there are external variables used by the function that are not
local to the function itself, they are not directly within the
stack window.
This explains also because the bytecode needs separate opcodes
for accessing registers and upvalues: if they could be directly
on the stack, it would be enough to reference upvalues with
negative register indexes and then use the stack as a sliding
window, like we do in C/C++ call conventions (except that stack
indexes in C/C++ are negative for parameters, and positive for
local variables, some of the former being cached in actual
registers, but still supported by "shadow" variables on the
stack, allocated at fixed positions in the stack fram by the
compiler or just pushed before the actual parameters and popped
to restore these registers after the call).
There's been performance tests that show that closures are not so
fast, they can create massive amounts of garbage collected
objects (with internal type LUA_TCLOSURE). I think this behavior
very curious, and the current implementation that allocates the
LUA_TCLOSURE objects on the heap is not the best option, the
mapping could be allocated directly in the stack of local
variables/registers of the caller (and all these closure objects
used by the caller could be merged to a single one, i.e. as the
largest closure object needed for inner calls, merged like in an
union. The closure objects themselves to not hold any variable
value, these are just simple mappings from a small fixed integer
sets (between 1 and the number of upvalues of the called
function) and variable integers (absolute indexes in the thread's
stack where the actual variable is located).
The byte code is not as optimized as it could be: the register
numbers are only positive, the upvalue numbers are also only
positive, they could forml a single set (positive integers for
local registers, negative integers for upvalues, meaning that
they are used to index the entries in the closure object to get
access to the actual varaible located anywhere on the stack,
outside of the immediate parent frame). The generated bytecode is
not as optimal as it could be because various operations can only
work on registers or constants (like ADD r1,r2,r3) so temporary
registers must be allocated by the compiler (let's remember that
the number of registers is limited). As well Lua's default engine
treat all registers the same, when most of them will work with a
single r0 register (an "accumulator") which could be implicit for
most instructions, and this would reduce the instruction sizes
(currently 32-bit or 64 bits), which is inefficient as it uses
too much the CPU L1 data cache.
I'm convinced that the current approach in the existing Lua VM
engine and its internal instruction set can be largely improved
for better performance, without really changing the language
itself, to get better data locality (smaller instruction sizes,
less wasted unused bit fields), and elimination of uses of the
heap for closures (to dramatically reduce the stress on the
garbage collector)
Op Sa. 1 Des. 2018 om 17:08 het Philippe Verdy

Post by Philippe Verdy
If there's something to do to the Lua engine is to rework

the way upvalues are allocated: they should be on the call
stack as much as possible (e.g. for up to ~250 upvalues) and
not on the heap.
The way I understand it, upvalues are not on the heap. They are on the
main execution stack below the bottom of the current
function's stack
frame.

[Philippe Verdy]

This post might be inappropriate. Click to display it.

[Tim Hill]

Post by Dirk Laurie

If there's something to do to the Lua engine is to rework the way upvalues are allocated: they should be on the call stack as much as possible (e.g. for up to ~250 upvalues) and not on the heap.

The way I understand it, upvalues are not on the heap. They are on the
main execution stack below the bottom of the current function's stack
frame.

My understanding was they start that way, but if the owning function exits they are migrated to a distinct structure to lift them “out” of the stack frame that is being discarded. Something like that Roberto probably winced since its been a while since I looked at his paper in this subject.

—Tim

[Sam Pagenkopf]

Post by Egor Skriptunoff
How to save intermediate value in this code
a:b.c()
a:b.d()
to rewrite it in optimized way
local x = a:b
x.c()
x.d()

Okay, but what would you expect from this?
a.b = a:b

[Coda Highland]

Post by Sam Pagenkopf

Post by Egor Skriptunoff
How to save intermediate value in this code
a:b.c()
a:b.d()
to rewrite it in optimized way
local x = a:b
x.c()
x.d()

Okay, but what would you expect from this?
a.b = a:b

Using the implementation I was alluding to, that operation would
BASICALLY be a no-op, because the new value of a.b would be a binding
object that exposes the properties of b but gets passed to methods as
if it were a. It would cause issues if you tried to then use a as a
prototype to construct a2, but in and of itself it wouldn't be an
issue.

/s/ Adam

[Philippe Verdy]

Post by Egor Skriptunoff
How to save intermediate value in this code
a:b.c()
a:b.d()

This idea a very strange object type in Lua creating an object that

Post by Egor Skriptunoff
to rewrite it in optimized way
local x = a:b
x.c()
x.d()

The way I percieve it is that "x" should be a "proxying" function object
(which internally will call function "b"), defined in a closure that holds
the value "a". Something like:

local x = function
local __o___ = a;
return function(...)
return b(__o__, ...)
end
end
x(c)
x(d)

This adds the overhead of one level of function call (even if this is a
trailing call that will not consume stack space)

[Philippe Verdy]

Post by Philippe Verdy

Post by Egor Skriptunoff
How to save intermediate value in this code
a:b.c()
a:b.d()

This idea a very strange object type in Lua creating an object that

Post by Egor Skriptunoff
to rewrite it in optimized way
local x = a:b
x.c()
x.d()

The way I percieve it is that "x" should be a "proxying" function object
(which internally will call function "b"), defined in a closure that holds

Fix:

local x = function(a, b)
local __o___ = a;

Post by Philippe Verdy
return function(...)
return b(__o__, ...)
end
end
x(c)
x(d)

Sorry: I forgot the (a,b) parameters to the first function...

[Russell Haley]

Post by Soni "They/Them" L.

Post by Pierre Chapuis

Post by Soni "They/Them" L.
- Lua has little to no support for composition-based OOP, whereas it has
many features to support inheritance (__index/__newindex) and other
forms of OOP (self:methods()). This isn't a big deal, tho, as it can be
easily solved with a rather small patch.

Can you give an example of what you would want here?
Because for me, "composition-based OOP" does not need much
language support. For what it's worth I almost never use inheritance
in Lua (or any other language, really).

The classic Cratera[1] `foo:[component].method()` (note the colon) syntax.
[1] https://bitbucket.org/TeamSoni/cratera
I looked at cratera once but didn't grasp the benefits because I don't

write games. Months later I was reading Roblox game code and the use case
for cratera became crystal clear. Likewise, Garry's Mod has very deep
component nesting and a library like cratera would simplify development.[1]

[1] I still don't write games so I may be off mark.

Russ

[Gabriel Bertilson]

I haven't done game programming in Lua, but it occurred to me that the
component syntax would allow you to use utf8 functions as string
methods: ('βλαβλα'):utf8.codepoint(1, -1) instead of
utf8.codepoint('βλαβλα', 1, -1). That's neat. The utf8 library would
have to be added as a field inside the __index metafield of the string
metatable.

— Gabriel

Post by Soni "They/Them" L.

Post by Pierre Chapuis

Post by Soni "They/Them" L.
- Lua has little to no support for composition-based OOP, whereas it has
many features to support inheritance (__index/__newindex) and other
forms of OOP (self:methods()). This isn't a big deal, tho, as it can be
easily solved with a rather small patch.

Can you give an example of what you would want here?
Because for me, "composition-based OOP" does not need much
language support. For what it's worth I almost never use inheritance
in Lua (or any other language, really).

The classic Cratera[1] `foo:[component].method()` (note the colon) syntax.
[1] https://bitbucket.org/TeamSoni/cratera

I looked at cratera once but didn't grasp the benefits because I don't write games. Months later I was reading Roblox game code and the use case for cratera became crystal clear. Likewise, Garry's Mod has very deep component nesting and a library like cratera would simplify development.[1]
[1] I still don't write games so I may be off mark.
Russ

[Sam Pagenkopf]

object.component.method(object) => object:component.method()
^ I also tried this syntax (and was sad it didn't work), but it's not so
bad. That strategy makes for cross-cutting communication between
components, which is not the greatest thing. It also assumes that call your
parents, but not your grandparents! Here's my alternative, a "parent"
variable:

Within calls, replace:
object.component.method(object) => object.component:method()

And within component methods:
self => self.parent
self.<<name of this component>> => self

Then, just add a parent member to constructors.

This also makes it easier to do the better thing; preferring self.x over
self.parent.x, which avoids conflict and does some "light" encapsulation.
If you do cross-cutting communication within the parent, rather than
between components, that's also better, and you might not even need a
parent variable at all.

Let me know if that's helpful.

Post by Russell Haley

Post by Soni "They/Them" L.

Post by Pierre Chapuis

Post by Soni "They/Them" L.
- Lua has little to no support for composition-based OOP, whereas it

has

Post by Pierre Chapuis

Post by Soni "They/Them" L.
many features to support inheritance (__index/__newindex) and other
forms of OOP (self:methods()). This isn't a big deal, tho, as it can be
easily solved with a rather small patch.

Can you give an example of what you would want here?
Because for me, "composition-based OOP" does not need much
language support. For what it's worth I almost never use inheritance
in Lua (or any other language, really).

The classic Cratera[1] `foo:[component].method()` (note the colon) syntax.
[1] https://bitbucket.org/TeamSoni/cratera
I looked at cratera once but didn't grasp the benefits because I don't

write games. Months later I was reading Roblox game code and the use case
for cratera became crystal clear. Likewise, Garry's Mod has very deep
component nesting and a library like cratera would simplify development.[1]
[1] I still don't write games so I may be off mark.
Russ

[云风 Cloud Wu]

Post by Sam Pagenkopf
object.component.method(object) => object:component.method()

My alternative is when we add a component to an object, I copy all the
methods of component into object's meta table, and we can use it in
this way :

object:component_method(...) => object.component.method(object, ...)

[Soni L.]

Suggestion: Don't assume (or use) inheritance. Or at least don't mix
inheritance with components.

Rust has traits, which are somewhat similar to components, but it doesn't
have inheritance. That's because mixing traits/components and inheritance
is, at best, messy.

Post by Sam Pagenkopf
object.component.method(object) => object:component.method()
^ I also tried this syntax (and was sad it didn't work), but it's not so
bad. That strategy makes for cross-cutting communication between
components, which is not the greatest thing. It also assumes that call your
parents, but not your grandparents! Here's my alternative, a "parent"
object.component.method(object) => object.component:method()
self => self.parent
self.<<name of this component>> => self
Then, just add a parent member to constructors.
This also makes it easier to do the better thing; preferring self.x over
self.parent.x, which avoids conflict and does some "light" encapsulation.
If you do cross-cutting communication within the parent, rather than
between components, that's also better, and you might not even need a
parent variable at all.
Let me know if that's helpful.

Post by Russell Haley

Post by Soni "They/Them" L.

Post by Pierre Chapuis

Post by Soni "They/Them" L.
- Lua has little to no support for composition-based OOP, whereas it

has

Post by Pierre Chapuis

Post by Soni "They/Them" L.
many features to support inheritance (__index/__newindex) and other
forms of OOP (self:methods()). This isn't a big deal, tho, as it can

be

Post by Pierre Chapuis

Post by Soni "They/Them" L.
easily solved with a rather small patch.

Can you give an example of what you would want here?
Because for me, "composition-based OOP" does not need much
language support. For what it's worth I almost never use inheritance
in Lua (or any other language, really).

The classic Cratera[1] `foo:[component].method()` (note the colon) syntax.
[1] https://bitbucket.org/TeamSoni/cratera
I looked at cratera once but didn't grasp the benefits because I don't

write games. Months later I was reading Roblox game code and the use case
for cratera became crystal clear. Likewise, Garry's Mod has very deep
component nesting and a library like cratera would simplify development.[1]
[1] I still don't write games so I may be off mark.
Russ

[Coda Highland]

Suggestion: Don't assume (or use) inheritance. Or at least don't mix inheritance with components.
Rust has traits, which are somewhat similar to components, but it doesn't have inheritance. That's because mixing traits/components and inheritance is, at best, messy.

So on the one hand, you're completely right. Traits are a much better
model for this than inheritance, and mixing traits and inheritance can
indeed get quite messy.

But on the other hand, that still leaves the matter of "what's the
best way to implement traits in Lua?" as a question to be discussed.

/s/ Adam

[Sam Pagenkopf]

I'd imagine that a trait system in lua would have a set of trait tests that
run before the program, and attach methods. Each trait would be a unique
table of requirements and possibly default implementations. Each object
would have some way of saying it implements said trait, either explicitly
or duck-typed. Instead of verifying input and output parameters, the best
you could probably do is run tests. It doesn't seem massively useful, but
it could be a nice sort of subset of a class framework. If there's
interest, I might try to write one.

Post by Soni L.

Post by Soni L.
Suggestion: Don't assume (or use) inheritance. Or at least don't mix

inheritance with components.

Post by Soni L.
Rust has traits, which are somewhat similar to components, but it

doesn't have inheritance. That's because mixing traits/components and
inheritance is, at best, messy.
So on the one hand, you're completely right. Traits are a much better
model for this than inheritance, and mixing traits and inheritance can
indeed get quite messy.
But on the other hand, that still leaves the matter of "what's the
best way to implement traits in Lua?" as a question to be discussed.
/s/ Adam

[Soni L.]

Post by Soni L.

Post by Soni L.
Suggestion: Don't assume (or use) inheritance. Or at least don't mix

inheritance with components.

Post by Soni L.
Rust has traits, which are somewhat similar to components, but it

doesn't have inheritance. That's because mixing traits/components and
inheritance is, at best, messy.
So on the one hand, you're completely right. Traits are a much better
model for this than inheritance, and mixing traits and inheritance can
indeed get quite messy.
But on the other hand, that still leaves the matter of "what's the
best way to implement traits in Lua?" as a question to be discussed.
/s/ Adam

Sorry, uh, is it okay if I ask what you mean by that?

I mean, I was specifically saying these shouldn't be a thing:

foo:thing.other.more.why() -> foo.thing.other.more.why(foo)

foo.thing.other.more:why() -> the ability to use self.super.super.super to
access foo

Mostly because they look nothing like traits, but also because they look
like some weird attempt at inheritance?

[Coda Highland]

Post by Soni L.

Post by Coda Highland

Suggestion: Don't assume (or use) inheritance. Or at least don't mix inheritance with components.
Rust has traits, which are somewhat similar to components, but it doesn't have inheritance. That's because mixing traits/components and inheritance is, at best, messy.

So on the one hand, you're completely right. Traits are a much better
model for this than inheritance, and mixing traits and inheritance can
indeed get quite messy.
But on the other hand, that still leaves the matter of "what's the
best way to implement traits in Lua?" as a question to be discussed.
/s/ Adam

Sorry, uh, is it okay if I ask what you mean by that?
foo:thing.other.more.why() -> foo.thing.other.more.why(foo)
foo.thing.other.more:why() -> the ability to use self.super.super.super to access foo
Mostly because they look nothing like traits, but also because they look like some weird attempt at inheritance?

First off: This isn't inheritance at all. It's composition. Some
languages do support composition via inheritance (C++, Python), but in
the larger programming world it's not a popular paradigm because of
all of the issues that arise in multiple inheritance.

There's no SPECIFIC reason why foo:bar.baz() -> foo.bar.baz(foo) can't
be supported. I have no complaints about that syntax as long as only a
single : is supported in an expression. It might be a bit ugly to
handle at parse time or at run time, but the fact that Cratera
supports it suggests it's not THAT bad.

self.super.super.super fails HARD, though, because the notion of
"super" there becomes bound up in the specific way the expression is
constructed. Consider this:

local more = foo.thing.other.more
more:why() -- what does self.super mean here?

The "more" object there has no idea that it's contained within
"foo.thing.other" unless you explicitly assign the value of "super" in
the object -- at which point that works just fine in current Lua
without any modifications at all -- or if you do what Python does and
you have . return a binding object instead of the object itself. But
tracking the whole call-time super chain adds an arbitrary amount of
overhead to every function call expression even if it's never going to
be used.

/s/ Adam

[Soni L.]

You lost me, sorry. I'm not sure how python/C++ actually do this, but
Cratera, like Rust, lets you do:

foo:bar.baz()
foo:baz.baz()

Where each do their own thing.

The Rust equivalent is (roughly)

<Foo as Bar>::baz(foo);
<Foo as Baz>::baz(foo);

(there are variations you can use, such as `Bar::baz(foo)` instead of `<Foo
as Bar>::baz(foo)`)

Post by Soni L.

Post by Soni L.

Post by Coda Highland

Post by Soni L.
Suggestion: Don't assume (or use) inheritance. Or at least don't mix

inheritance with components.

Post by Soni L.

Post by Coda Highland

Post by Soni L.
Rust has traits, which are somewhat similar to components, but it

doesn't have inheritance. That's because mixing traits/components and
inheritance is, at best, messy.

Post by Soni L.

Post by Coda Highland
So on the one hand, you're completely right. Traits are a much better
model for this than inheritance, and mixing traits and inheritance can
indeed get quite messy.
But on the other hand, that still leaves the matter of "what's the
best way to implement traits in Lua?" as a question to be discussed.
/s/ Adam

Sorry, uh, is it okay if I ask what you mean by that?
foo:thing.other.more.why() -> foo.thing.other.more.why(foo)
foo.thing.other.more:why() -> the ability to use self.super.super.super

to access foo

Post by Soni L.
Mostly because they look nothing like traits, but also because they look

like some weird attempt at inheritance?
First off: This isn't inheritance at all. It's composition. Some
languages do support composition via inheritance (C++, Python), but in
the larger programming world it's not a popular paradigm because of
all of the issues that arise in multiple inheritance.
There's no SPECIFIC reason why foo:bar.baz() -> foo.bar.baz(foo) can't
be supported. I have no complaints about that syntax as long as only a
single : is supported in an expression. It might be a bit ugly to
handle at parse time or at run time, but the fact that Cratera
supports it suggests it's not THAT bad.
self.super.super.super fails HARD, though, because the notion of
"super" there becomes bound up in the specific way the expression is
local more = foo.thing.other.more
more:why() -- what does self.super mean here?
The "more" object there has no idea that it's contained within
"foo.thing.other" unless you explicitly assign the value of "super" in
the object -- at which point that works just fine in current Lua
without any modifications at all -- or if you do what Python does and
you have . return a binding object instead of the object itself. But
tracking the whole call-time super chain adds an arbitrary amount of
overhead to every function call expression even if it's never going to
be used.
/s/ Adam

[Coda Highland]

Post by Coda Highland
foo:bar.baz()
foo:baz.baz()
Where each do their own thing.
The Rust equivalent is (roughly)
<Foo as Bar>::baz(foo);
<Foo as Baz>::baz(foo);
(there are variations you can use, such as `Bar::baz(foo)` instead of `<Foo as Bar>::baz(foo)`)

In C++ by default you actually DON'T get the ability to preserve a
pointer to the foo; you only get a pointer to the version that's been
cast to bar. There are techniques that are used to circumvent this
problem such as CRTP, but as I said, doing it with multiple
inheritance is unpopular, and in practice this kind of component-based
architecture gets built more explicitly instead of just using the
language's built-in features.

With Python's builtins, you just use foo.baz(), and then "self" will
be foo. You don't explicitly call out bar. Yes, this DOES cause
problems if two of your components have conflicting names. You just
try to avoid doing that. (The language provides some name mangling for
internal-use-only members to help mitigate the issue.) There are
metaclass shenanigans you can use if you want something fancier.

But the point I was trying to make is that you shouldn't try to think
of this as an inheritance scheme, because what you're talking about
isn't inheritance. It's composition. Composition is done using
inheritance in some languages, but there are other alternatives that
don't involve inheritance. The Cratera technique isn't achieving
composition through inheritance; it's achieving it through lexical
binding, which is a perfectly natural way of doing it in Lua since Lua
uses lexical binding for member functions in the first place.
(Javascript also uses lexical binding. Python... sort of does.)

/s/ Adam

[Soni L.]

Umm, you realize I made Cratera? I thought I made that explicit, but maybe
I didn't.

I was replying to someone who had some incorrect ideas about composition.

Sorry for the confusion.

Post by Coda Highland

Post by Soni L.
You lost me, sorry. I'm not sure how python/C++ actually do this, but
foo:bar.baz()
foo:baz.baz()
Where each do their own thing.
The Rust equivalent is (roughly)
<Foo as Bar>::baz(foo);
<Foo as Baz>::baz(foo);
(there are variations you can use, such as `Bar::baz(foo)` instead of

`<Foo as Bar>::baz(foo)`)
In C++ by default you actually DON'T get the ability to preserve a
pointer to the foo; you only get a pointer to the version that's been
cast to bar. There are techniques that are used to circumvent this
problem such as CRTP, but as I said, doing it with multiple
inheritance is unpopular, and in practice this kind of component-based
architecture gets built more explicitly instead of just using the
language's built-in features.
With Python's builtins, you just use foo.baz(), and then "self" will
be foo. You don't explicitly call out bar. Yes, this DOES cause
problems if two of your components have conflicting names. You just
try to avoid doing that. (The language provides some name mangling for
internal-use-only members to help mitigate the issue.) There are
metaclass shenanigans you can use if you want something fancier.
But the point I was trying to make is that you shouldn't try to think
of this as an inheritance scheme, because what you're talking about
isn't inheritance. It's composition. Composition is done using
inheritance in some languages, but there are other alternatives that
don't involve inheritance. The Cratera technique isn't achieving
composition through inheritance; it's achieving it through lexical
binding, which is a perfectly natural way of doing it in Lua since Lua
uses lexical binding for member functions in the first place.
(Javascript also uses lexical binding. Python... sort of does.)
/s/ Adam

[Coda Highland]

Umm, you realize I made Cratera? I thought I made that explicit, but maybe I didn't.
I was replying to someone who had some incorrect ideas about composition.
Sorry for the confusion.

You might have made that explicit and I just missed it. My apologies
on that point.

But you replied to... um... me. And I'm pretty sure I've got my head
on straight when it comes to composition.

So there's definitely some confusion here. Not anything that's an
actual problem, I just think I might have missed a point somewhere.

/s/ Adam

[Soni L.]

The first reply on this side-chain was not directed at you, but at Sam
Pagenkopf.

Post by Soni L.
Umm, you realize I made Cratera? I thought I made that explicit, but

maybe I didn't.

Post by Soni L.
I was replying to someone who had some incorrect ideas about composition.
Sorry for the confusion.

You might have made that explicit and I just missed it. My apologies
on that point.
But you replied to... um... me. And I'm pretty sure I've got my head
on straight when it comes to composition.
So there's definitely some confusion here. Not anything that's an
actual problem, I just think I might have missed a point somewhere.
/s/ Adam

[Coda Highland]

The first reply on this side-chain was not directed at you, but at Sam Pagenkopf.

Oh, that's really weird. I definitely see that gmail's thread history
agrees with you, but the quote history in that post was my post. Very
odd.

/s/ Adam

[Sam Pagenkopf]

I'm very hazy on the benefit of a trait system that relies on
self.trait.subtrait.othersubtrait(self). Is it because each trait can
remain constant? Why is it more important to have methods in a child
operating on a shared pool of variables, when it's possible to put both the
variables and methods in children and avoid relying on a large, shared
state? It just feels like a very arbitrary separation to me, and goes again
the grain of good compositional design.

Post by Soni L.

Post by Soni L.
The first reply on this side-chain was not directed at you, but at Sam

Pagenkopf.
Oh, that's really weird. I definitely see that gmail's thread history
agrees with you, but the quote history in that post was my post. Very
odd.
/s/ Adam

[Magicks M]

foo.component.method(foo, ...) is support for this style of programming.
If you really want it to look better you could wrap the methods in a
function.
Simple example to illustrate my idea:

function redirect(method)
return function(component, ...) return method(component.object, ...) end
end
-- declare this helper to register component actions
-- one could also just wrap the method directly i.e
`redirect(function(object`
function component:action(name)
self[name] = redirect(self[name])
end

function component:something()
-- code here self == the object
end
component:action 'something' --register it as an action on the object the
component is attatched to.
-- `foo.component:something()` will now work assuming components are wired
up to have that .object reference.

While this example is clearly not complete, I hope it illustrates the
suggestion clearly enough. I actually prefer the verbose method though,
since a big part of composition based designs is being able to decouple the
actions and the data (example is using an ECS) so having that clear
reference too foo makes sense to me.

Post by Soni "They/Them" L.

Post by Pierre Chapuis

Post by Soni "They/Them" L.
- Lua has little to no support for composition-based OOP, whereas it has
many features to support inheritance (__index/__newindex) and other
forms of OOP (self:methods()). This isn't a big deal, tho, as it can be
easily solved with a rather small patch.

Can you give an example of what you would want here?
Because for me, "composition-based OOP" does not need much
language support. For what it's worth I almost never use inheritance
in Lua (or any other language, really).

The classic Cratera[1] `foo:[component].method()` (note the colon) syntax.
[1] https://bitbucket.org/TeamSoni/cratera