Patchless modification of Lua source code

[Dirk Laurie]

Over the years, there have been several replies to suggested patches
on this list, saying "no need to patch Lua".

The trick is not described in the "Customization" section of
'doc/readme.html' so this post has the dual purpose of documenting it
here (probably not for the first tine) and pleading that a better (not
Linux-specfic, and also saying how to do it for liblua.a or liblua.dll
as the case may be) description than mine might in future be described
there.

~~~
A customized version of Lua that would affect only non-header files,
i.e. files with names ending in ".c", never requires patching Lua
itself. The job can be done from the command line, illustrated here
for two such files under Linux.

The key points are:

1. All external names used by Lua are declared in header files.
2. If such a name has been resolved by an object file or library
linked earlier, the linker will be satisfied and will not replace it.
3. Any .o files needed by the linker will be built automatically from
.c files by the Makefile rules.

So all that is necessary is:

1. In the Lua source directory, save your modified files with another
prefix, say "my", instead ol "l" starting their names, e.g.
'myctype.c' and 'mylex.c'.
2. When you want a new 'lua' including your most recent modifcations:
rm my*.o # necessary because the Makefile does not know about them
make linux -e "LUA_O= lua.o myctype.o mylex.o"
~~~

[Dirk Laurie]

Post by Dirk Laurie
rm my*.o # necessary because the Makefile does not know about them
make linux -e "LUA_O= lua.o myctype.o mylex.o"

Actually, you probably do not need "rm my*.o". The dependency of .o
files on .c files is automatic.

[Viacheslav Usov]

Post by Dirk Laurie
1. All external names used by Lua are declared in header files.
2. If such a name has been resolved by an object file or library
linked earlier, the linker will be satisfied and will not replace it.
3. Any .o files needed by the linker will be built automatically from
.c files by the Makefile rules.

No. Quoting clause 6.9/5 of the C11 standard:

If an identifier declared with external linkage is used in an expression
(other than as part of the operand of a sizeof or _Alignof operator whose
result is an integer constant), somewhere in the entire program there shall
be exactly one external definition for the identifier; otherwise, there
shall be no more than one.

(end)

The standard has no concept of "linked earlier" and consequently no primacy
of "names linked earlier", and the entire program, including all the
libraries, shall have a function with a given name defined at most once.
Per clause 4/2, "If a ‘‘shall’’ or ‘‘shall not’’ requirement [...] is
violated, the behavior is undefined".

Note that what you do may fail in weird ways even with toolchains that do
have the notion of "linked earlier" - precisely because something has been
linked earlier. Your application may be using the "modified" functions,
while Lua itself and the other libraries use the "original" functions. This
would be a special case of generally undefined behaviour.

Do not do this, except for fun.

Cheers,
V.

[Dirk Laurie]

Post by Dirk Laurie
1. All external names used by Lua are declared in header files.
2. If such a name has been resolved by an object file or library
linked earlier, the linker will be satisfied and will not replace it.
3. Any .o files needed by the linker will be built automatically from
.c files by the Makefile rules.

If an identifier declared with external linkage is used in an expression (other than as part of the operand of a sizeof or _Alignof operator whose result is an integer constant), somewhere in the entire program there shall be exactly one external definition for the identifier; otherwise, there shall be no more than one.
(end)
The standard has no concept of "linked earlier" and consequently no primacy of "names linked earlier", and the entire program, including all the libraries, shall have a function with a given name defined at most once. Per clause 4/2, "If a ‘‘shall’’ or ‘‘shall not’’ requirement [...] is violated, the behavior is undefined".
Note that what you do may fail in weird ways even with toolchains that do have the notion of "linked earlier" - precisely because something has been linked earlier. Your application may be using the "modified" functions, while Lua itself and the other libraries use the "original" functions. This would be a special case of generally undefined behaviour.
Do not do this, except for fun.

What you state so categorically and trenchantly may well be true in
other situations, but I would remind you that we are now dealing with
source files that would have worked if used as patched versions of
standard Lua files. The point of the exercise is simply that patching
can be avoided by this trick, for which I claim no originality.

For the 'linux' target of the Makefile supplied with Lua 5.4 work2,
the link step expands to:

gcc -std=gnu99 -o lua lua.o mylex.o myctype.o liblua.a -lm -Wl,-E -ldl

The way I understand it, is that the only the three object files
lua.o, mylex.o and myctype.o must not contain a duplicate definition
of an external routine. Only some routines of a library are linked in,
only the ones still unresolved. For example, the C99 math library
contains atanh, but the lua source code does not request it, so it is
not linked into the lua executable.

It seems to me that, if those three files contain a name conflict,
then omitting mylex.o and myctype.o and instead replacing llex.c and
lctype.c by mylex.c and myctype.c respectively, as in effect patching
would do, would contain the same name conflict, contradicting the
assumption that patching would have worked.

Admittedly I do not have the C11 standard open in front of me (which
BTW seems to be totally irrelevant when a program is compiled
according to the gnu99 standard) and am relying solely on about thirty
years' experience as a C programmer.

I apologize for omitting, in the mistaken belief that they would be
implicit to any reasonably experienced reader, the final words "from
liblua.a" from my point number 2,

— Dirk

[Viacheslav Usov]

The point of the exercise is simply that patching can be avoided by this

trick, for which I claim no originality.

The point of my message is that the exercise violates the rules of the C
language and is inherently dangerous. It would be a disservice to the
community to promote your method without this clearly stated.

Admittedly I do not have the C11 standard open in front of me (which BTW

seems to be totally irrelevant when a program is compiled according to the
gnu99 standard)

You did say:

[...] pleading that a better (not Linux-specfic, and also saying how to do
it for liblua.a or liblua.dll as the case may be) description than mine
might in future be described there.

(end)

In that broader context your method ought to be supported by a language
standard rather than a particular tool. Even with that tool, though, your
method is dangerous.

The requirement that names with external linkage shall be defined at most
once is not new in C11. It existed already in C89, viz., clause 3.7:

If an identifier declared with external linkage is used in an expression
(other than as part of the operand of a sizeof operator), somewhere in the
entire program there shall be exactly one external definition for the
identifier.

(end)

Observe how the more recent standard is almost a verbatim replica of the
old one in this respect. This is not accidental, because the requirement of
_one_ definition of an external entity is a basic tenet of C and is
traceable back K&R 's book (1978). Your method is fundamentally flawed.

Cheers,
V.

[Dirk Laurie]

Post by Viacheslav Usov
Your method is fundamentally flawed.

This is not "my" method. I have merely filled out the details and
written it up in a thread that is easy to find. The suggestion that
the linker can do the work without the need for patching the Lua core
is due to Luiz.

http://lua-users.org/lists/lua-l/2011-05/msg00543.html

[Luiz Henrique de Figueiredo]

The suggestion that the linker can do the work without the need for
patching the Lua core is due to Luiz.

That has been my experience in many years without exception.

The C standard is talking about the definition of symbols but it never
mentions a linker or how it should work, AFAIR. The linker is just
implementing the requirements in the C standard in a reasonable and
convenient way: it resolves symbols as it sees definitions for them. A
library is not meant to be totally incorporated into the program; only
modules that provide definitions for the program are incorporated.
Again, this is reasonable and convenient, and works extremely well in
practice. It has been like that for many decades.

[Viacheslav Usov]

On Wed, Nov 21, 2018 at 12:12 PM Luiz Henrique de Figueiredo <

Post by Luiz Henrique de Figueiredo
The C standard is talking about the definition of symbols but it never

mentions a linker or how it should work, AFAIR.

This is not true. For example, in C89, clause 2.1.1.2 Translation phases,
number 8:

All external object and function references are resolved. Library
components are linked to satisfy external references to functions and
objects not defined in the current translation.

(end)

This is echoed verbatim in C11. The requirements for external references
and definitions are covered elsewhere in the standard (I cited them
earlier).

Post by Luiz Henrique de Figueiredo
The linker is just implementing the requirements in the C standard in a

reasonable and convenient way:

The linker is either conformant or not. However, the particular requirement
of "one definition only" is one where the standard lets the implementation
(i.e., the linker) behave in an undefined way (because the standard
implicitly recognises that an implementation may not have sufficient means
to enforce conformant behaviour). This merely means that the onus is on the
programmer to ensure that the whole program is conformant, which, in this
case, means to ensure that no function is defined more than once in the
program, which includes all its libraries.

Post by Luiz Henrique de Figueiredo
A library is not meant to be totally incorporated into the program; only

modules that provide definitions for the program are incorporated.

This is, in fact, not a requirement. An implementation is free to
incorporate all of the library. And when a shared library used, this is
exactly what happens.

If the previous "static" linking phase has resolved some of the
dependencies, these dependencies won't be resolved during the "dynamic"
linking phase to the functions incorporated in the shared library. This is

Post by Luiz Henrique de Figueiredo
Again, this is reasonable and convenient, and works extremely well in

practice. It has been like that for many decades.

would be a dangerous idea to entertain.

First, other libraries that the executable depends on will not have been
linked statically to the same "replacement" functions, so they will have to
resort to the "original" dependencies. Your program, as whole, may end up
using different versions of the same set of functions simultaneously. Even
without those other libraries the sole library that you intend to patch
"without patching" can happily continue using its own functions, with the
same result.

Second, if those other libraries resort to the same trick, the situation is
not just proportionally worse, it is much worse because this may end up
with duplicate export symbols and a whole new problem to solve.

Finally, none of this is portable and on some platforms one would have to
go through pretty serious gyrations to convince the linker it is OK to have
duplicate symbols and that these particular ones are to be used statically.
Which is actually good.

Cheers,
V.

[Philippe Verdy]

I disagree. In practice the linker are trying to resolve symbols by
starting from a first unit and using the other modules and libraries one by
one, in the strict order where they are specified in the link command,
until there remains no other unresolved symbols, at which time all other
units may be discarded, when building a standalone application, or still
added to the pack when building a library. But the linker will emit a
warning if it finds a symbol redefined by another unit, because the result
of the link can only keep one of the reference, the first one encountered,
the rest, if it is not reachable at all by any other exported symbol part
of the smallest unit, can be discarded directly by the linker.

So the order by which you specify the compiled units, and then the
libraries, is significant.

E.g a link command specifying "-ll -lm" is not equivalent to the same
command specifying "-lm -ll" instead. The same remark applies to the
ordered list of .o units, which just come as a first group before all
libraries, but are not necessarily all included in the linker module,
except when building a library and the modules kept in the library still
have at least one distinctive symbol.

Post by Viacheslav Usov
On Wed, Nov 21, 2018 at 12:12 PM Luiz Henrique de Figueiredo <

Post by Luiz Henrique de Figueiredo
The C standard is talking about the definition of symbols but it never

mentions a linker or how it should work, AFAIR.
This is not true. For example, in C89, clause 2.1.1.2 Translation phases,
All external object and function references are resolved. Library
components are linked to satisfy external references to functions and
objects not defined in the current translation.
(end)
This is echoed verbatim in C11. The requirements for external references
and definitions are covered elsewhere in the standard (I cited them
earlier).

Post by Luiz Henrique de Figueiredo
The linker is just implementing the requirements in the C standard in a

The linker is either conformant or not. However, the particular
requirement of "one definition only" is one where the standard lets the
implementation (i.e., the linker) behave in an undefined way (because the
standard implicitly recognises that an implementation may not have
sufficient means to enforce conformant behaviour). This merely means that
the onus is on the programmer to ensure that the whole program is
conformant, which, in this case, means to ensure that no function is
defined more than once in the program, which includes all its libraries.

Post by Luiz Henrique de Figueiredo
A library is not meant to be totally incorporated into the program; only

modules that provide definitions for the program are incorporated.
This is, in fact, not a requirement. An implementation is free to
incorporate all of the library. And when a shared library used, this is
exactly what happens.
If the previous "static" linking phase has resolved some of the
dependencies, these dependencies won't be resolved during the "dynamic"
linking phase to the functions incorporated in the shared library. This is

Post by Luiz Henrique de Figueiredo
Again, this is reasonable and convenient, and works extremely well in

practice. It has been like that for many decades.
would be a dangerous idea to entertain.
First, other libraries that the executable depends on will not have been
linked statically to the same "replacement" functions, so they will have to
resort to the "original" dependencies. Your program, as whole, may end up
using different versions of the same set of functions simultaneously. Even
without those other libraries the sole library that you intend to patch
"without patching" can happily continue using its own functions, with the
same result.
Second, if those other libraries resort to the same trick, the situation
is not just proportionally worse, it is much worse because this may end up
with duplicate export symbols and a whole new problem to solve.
Finally, none of this is portable and on some platforms one would have to
go through pretty serious gyrations to convince the linker it is OK to have
duplicate symbols and that these particular ones are to be used statically.
Which is actually good.
Cheers,
V.

[Viacheslav Usov]

Post by Philippe Verdy
I disagree.

With what, exactly?

Post by Philippe Verdy
In practice the linker

What linker is that, exactly?

How in your "I disagree" response did you consider the case of shared
libraries, which I have specifically addressed twice and which are the
whole point of this thread to begin with?

Cheers,
V.

[Philippe Verdy]

shared libaries are handled the same way: an application loading a shared
libary will be linked by the system by resolving names in a defined order:
the order specified in the compiled applications, which lists the shared
libraries it needs in a significant order. If two shard libraries export
the same symbol, only one will be used: the symbol coming from the first
library (the other symbol from a further-listed shared library may still be
used indirectly, if the second library has other symbols used by the
application, and the application then allows the second distinct
implementation of the first symbol used indirectly)

Say that an apps needs symbols ("a","b","c") from "sharedlibary1" and
symbols ("d","e","f") from "sharedlibary2". The "sharedlibrary2" may also
export the symbol "a" (used indirectly within sharedlibary2 to implement
"c"). The app will itself will use directly only "a" from "sharedlibrary1"
because the application was compiled to load and use
"sharedlibary1,sharedlibary2" in that precise order. But when the
application will use "d" from "sharedlibary2", this could cause the
implementation "a" from "sharedlibary2" to be used, even if there's also
the separate implementation of "a" in sharedlibrary1.

In all cases, the order or resolution is always important and needs to be
specified. For this reason, shared libraries are not used in an unordered
collection of library names: there must be a specified order (and typically
shared libaries are loaded and resolved one at a time, and internal uses by
shared libaries themselves are resolved internally by each shared libary
itself, specifying its own internal dependencies (already resolved when
they were compiled), or external dependencies (these additional
dependencies must have NO impact to the application using it, unless the
application loading a shared DLL also instructs that DLL to allow resolving
its own external dependencies using a path order specified by the
application, to override the default path included in the shared library
itself; for most cases, such overrides are not possible but many shared
library loaders are using an external path, such as in the environment, to
indicate where to load the other libraries: a DLL loader may check that the
located libaries match a specific digital signature to accept using it, and
if not, the lookup will need to search on other entries of the environment
path: this can prevent unexpected matches, and it's a solution that allows
"signed" libraries to be specified in random order; if libraries are not
signed and the application or libraries do not specify a specific digital
signature or an absolute path for locating them, then nothing will prevent
duplicate symbols to occur, except the specified order).

Symbol dependies form an oriented graph across all modules, with several
layers of locality, each module defining its own order of dependencies (the
names of shared libaries are themselves also "symbols", like normal symbols
attached to individual objects inside each library or module).

Post by Viacheslav Usov

Post by Philippe Verdy
I disagree.

With what, exactly?

Post by Philippe Verdy
In practice the linker

What linker is that, exactly?
How in your "I disagree" response did you consider the case of shared
libraries, which I have specifically addressed twice and which are the
whole point of this thread to begin with?
Cheers,
V.

[Viacheslav Usov]

Post by Philippe Verdy
shared libaries are handled the same way

Same way as that of an unspecified linker?

Are you claiming that your description is applicable to all the operating
systems and all the linkers in the world?

Post by Philippe Verdy
But when the application will use "d" from "sharedlibary2", this could

cause the implementation "a" from "sharedlibary2" to be used, even if
there's also the separate implementation of "a" in sharedlibrary1.

Post by Philippe Verdy
internal uses by shared libaries themselves are resolved internally by

each shared libary itself, specifying its own internal dependencies
(already resolved when they were compiled), or external dependencies (these
additional dependencies must have NO impact to the application using it

So this seems to agree at least with problem "First" in the message to
which you responded "I disagree".

Again, what exactly do you disagree with?

Cheers,
V.

[Philippe Verdy]

I disagreed with what to say about linkers (I also said that command line
shells in OSes are also exctly like linkers; and variable name binding
implemented in any language is in fact also like a linker). Everywhere
there's an order of priority for name lookups
In programming languages, including Lua, this is called "scope", and a
"closure" is the fact of binding specific sets of variables to specific
scopes.
And that's what I mean by "locality".
Some names may be left unresolved by such linking/binding, leaving
"external" names. Or they may be oinly partially resolved to a small
subset, with a late final binding: this is what occurs with effective
values of parameters of functions (but there's still always a priority
order for completely resolving these links).
I do not know any example where a well defined priority order is not used
by any "linker" (or programming language, or shell using linking/binding)
for resolving "names", "symbols", or "variables", even in languages that
allow symbolic values (e.g. Lisp, or Prolog).

Post by Viacheslav Usov

Post by Philippe Verdy
shared libaries are handled the same way

Same way as that of an unspecified linker?
Are you claiming that your description is applicable to all the operating
systems and all the linkers in the world?

Post by Philippe Verdy
But when the application will use "d" from "sharedlibary2", this could

cause the implementation "a" from "sharedlibary2" to be used, even if
there's also the separate implementation of "a" in sharedlibrary1.

Post by Philippe Verdy
internal uses by shared libaries themselves are resolved internally by

each shared libary itself, specifying its own internal dependencies
(already resolved when they were compiled), or external dependencies (these
additional dependencies must have NO impact to the application using it
So this seems to agree at least with problem "First" in the message to
which you responded "I disagree".
Again, what exactly do you disagree with?
Cheers,
V.

[Sean Conner]

Post by Philippe Verdy
I do not know any example where a well defined priority order is not used
by any "linker" (or programming language, or shell using linking/binding)
for resolving "names", "symbols", or "variables", even in languages that
allow symbolic values (e.g. Lisp, or Prolog).

INRAC.

A computer language with non-deterministic binding (and flow control, but
I digress), used for a few commerical programs that I know of.

-spc (There are *always* exceptions, except for death ... )

[Philippe Verdy]

I said "I do not know", you replied by giving an example you know but is
almost confidential, whose applications using it are rare to find.

So you don't disagree what I said, you confirm it!

Post by Sean Conner

Post by Philippe Verdy
I do not know any example where a well defined priority order is not used
by any "linker" (or programming language, or shell using linking/binding)
for resolving "names", "symbols", or "variables", even in languages that
allow symbolic values (e.g. Lisp, or Prolog).

INRAC.
A computer language with non-deterministic binding (and flow control, but
I digress), used for a few commerical programs that I know of.
-spc (There are *always* exceptions, except for death ... )

[Philippe Verdy]

I cannot even find any reference about INRAC on the net. All I find is the
acronym and brand in France of the "Institut National pour la Retraite
ACtive", a training institute for senior people.

I find a reference in GitHub for the name of a module written in Perl for
reverse engineering of RAC files used in the early 1980s by the RACTER
program, used for chatting, an ancestor of IRC... But I don't know if you
refer to this Perl module and its related language. This is an alpha
module, not tested, not even documented.

Post by Philippe Verdy
I said "I do not know", you replied by giving an example you know but is
almost confidential, whose applications using it are rare to find.
So you don't disagree what I said, you confirm it!

Post by Philippe Verdy

Post by Philippe Verdy
I do not know any example where a well defined priority order is not

used

Post by Philippe Verdy
by any "linker" (or programming language, or shell using

linking/binding)

Post by Philippe Verdy
for resolving "names", "symbols", or "variables", even in languages that
allow symbolic values (e.g. Lisp, or Prolog).

INRAC.
A computer language with non-deterministic binding (and flow control, but
I digress), used for a few commerical programs that I know of.
-spc (There are *always* exceptions, except for death ... )

[Sean Conner]

Post by Philippe Verdy
I cannot even find any reference about INRAC on the net. All I find is the
acronym and brand in France of the "Institut National pour la Retraite
ACtive", a training institute for senior people.
I find a reference in GitHub for the name of a module written in Perl for
reverse engineering of RAC files used in the early 1980s by the RACTER
program, used for chatting, an ancestor of IRC... But I don't know if you
refer to this Perl module and its related language. This is an alpha
module, not tested, not even documented.

Congratulations! RACTER was, in fact, written in INRAC and was
commercially available in tthe mid 80s [1][2]. The RAC files *is* the
program, and the language itself is non-deterministic in resolving names
which I gave as an example to back up the point Viacheslav Usov was making.

-spc

[1] It was even used in the writing of the book _The Policemans' Beard
is Half Constructed_ (https://www.amazon.com/Policemans-Beard-Half-Constructed-Computer/dp/0446380512/)

[2] The INRAC compiler was also available commercially, but is hard to
find.

[Philippe Verdy]

Well this program proves what I said: it uses "late binding", then uses an
undocumented "linking" step to try resolving the references:

- I don't know what it does really, but if it allows using any candidate,
that it will enumerate all possible solutions, by explictly instanciating
all the possible bindings, so this will still effectively a linking step,
applied to each instance.

- If it just randomly select one candidate, then the results are
unpredictable, and the language itself has NO practival use, it is
fundamentally flawed, unsecure, non-portable at all: an implementation of
the language will necessrily have to make arbitrary choices... and document
them! This will create as many distinct "formal" languages as there are
arbitrary choices, even if all of them use the same basic syntax. Such
language is not formal, it is then actually a (possibly very large) family
of languages with distinct applications working only for one of its
instances.

Still we are back to the final linking step which is necessarily
deterministic (otherwise it is not implementable at all an any
deterministic computer or Turing machine, this informal language would be a
"blackbox", an "oracle", we could refer by "only God knows").

Post by Sean Conner

Post by Philippe Verdy
I cannot even find any reference about INRAC on the net. All I find is

the

Post by Philippe Verdy
acronym and brand in France of the "Institut National pour la Retraite
ACtive", a training institute for senior people.
I find a reference in GitHub for the name of a module written in Perl for
reverse engineering of RAC files used in the early 1980s by the RACTER
program, used for chatting, an ancestor of IRC... But I don't know if you
refer to this Perl module and its related language. This is an alpha
module, not tested, not even documented.

Congratulations! RACTER was, in fact, written in INRAC and was
commercially available in tthe mid 80s [1][2]. The RAC files *is* the
program, and the language itself is non-deterministic in resolving names
which I gave as an example to back up the point Viacheslav Usov was making.
-spc
[1] It was even used in the writing of the book _The Policemans' Beard
is Half Constructed_ (
https://www.amazon.com/Policemans-Beard-Half-Constructed-Computer/dp/0446380512/
)
[2] The INRAC compiler was also available commercially, but is hard to
find.

[Sean Conner]

Post by Philippe Verdy

Post by Sean Conner

Post by Philippe Verdy
I do not know any example where a well defined priority order is not used
by any "linker" (or programming language, or shell using linking/binding)
for resolving "names", "symbols", or "variables", even in languages that
allow symbolic values (e.g. Lisp, or Prolog).

INRAC.
A computer language with non-deterministic binding (and flow control, but
I digress), used for a few commerical programs that I know of.

I said "I do not know", you replied by giving an example you know but is
almost confidential, whose applications using it are rare to find.
So you don't disagree what I said, you confirm it!

You do not speak for me, and I did not "confirm" what you said.

You also did not do a search, because while I said "used for a few
commerical programs" I said nothing about the difficulty of finding the
programs.

-spc

[Philippe Verdy]

I can pretend that you reconfirm once again what I said, by confirming that
your example is hard to find (if we cannot find and evaluate it, then your
reply is justifying nothing at all, it just proves that you cannot justify
your opposition by verifiable arguments you could have invented only to
give a contradiction...)

Yes there are programming languages that use "late binding" (i.e. where
bionding is not solved at all imemdiately, allowing multiple names/symbols
to coexist and remain unfiltered), but this does not prove that these
languages are usable for anything, without finally using a final "linker"
that will resolve the references to names using a well defined and
predictable order with additional data (or environment) specifying the
expected ordering rules.

Post by Philippe Verdy

Post by Philippe Verdy

Post by Sean Conner

Post by Philippe Verdy
I do not know any example where a well defined priority order is not

used

Post by Philippe Verdy

Post by Sean Conner

Post by Philippe Verdy
by any "linker" (or programming language, or shell using

linking/binding)

Post by Philippe Verdy

Post by Sean Conner

Post by Philippe Verdy
for resolving "names", "symbols", or "variables", even in languages

that

Post by Philippe Verdy

Post by Sean Conner

Post by Philippe Verdy
allow symbolic values (e.g. Lisp, or Prolog).

INRAC.
A computer language with non-deterministic binding (and flow

control, but

Post by Philippe Verdy

Post by Sean Conner
I digress), used for a few commerical programs that I know of.

I said "I do not know", you replied by giving an example you know but is
almost confidential, whose applications using it are rare to find.
So you don't disagree what I said, you confirm it!

You do not speak for me, and I did not "confirm" what you said.
You also did not do a search, because while I said "used for a few
commerical programs" I said nothing about the difficulty of finding the
programs.
-spc

[Sean Conner]

Post by Philippe Verdy
I can pretend that you reconfirm once again what I said, by confirming that
your example is hard to find (if we cannot find and evaluate it, then your
reply is justifying nothing at all, it just proves that you cannot justify
your opposition by verifiable arguments you could have invented only to
give a contradiction...)
Yes there are programming languages that use "late binding" (i.e. where
bionding is not solved at all imemdiately, allowing multiple names/symbols
to coexist and remain unfiltered), but this does not prove that these
languages are usable for anything, without finally using a final "linker"
that will resolve the references to names using a well defined and
predictable order with additional data (or environment) specifying the
expected ordering rules.

You know what? You're right. I'm wrong. I'm outta here.

-spc

[Philippe Verdy]

Note: checking digital signatures of shared libraries is rarely used.
Instead the common practice is to specify a version number: a module may
not just indicate the name of a library, but also the allowed version(s) in
which it may be linked. This generally solves quite well the dependencies
on an external (unpredictable) path.

If you want to create a specific "patched" version of Lua, the best to it
it to assign to your path a subversion number indicating that it derives
from a base "standard" version.

Say you create a patched version based in Lua version "5.3", your new patch
will be versioned for example as "5.3-mypatch-1.0". The DLL loader for an
application depending on your patch, can be instructed to match
"5.3-mypatch-1.0" (or higher) and not just "5.3", and it will not match
version "5.4" because it is in another branch which does not include the
"mypatch-1.0" base of the derived branch which itself depends on the base
"5.3" branch, which is not replacable in this case.

This is in fact a basic scheme of all "sane" versioning systems where each
branch must be precisely identifiable.

But appliatiosn that want very secure dependencies, should not just match
version numbers (not even a single one excluding higher ones), but should
match digital signatures (this is possible with the Windows DLLs, not sure
that it works with all dynamic library loaders on Linux/Unix, which may not
have the feature needed to check exact implementations, but may just use
whatever "simple" symbols or names are found first in their loader path, in
which case the result will not be predictable if the environment path can
vary: the behavior will be fully driven and defined by the order specified
in the environment path, just like with shells for resolving simple command
names to actual locations of their implementations in applications stored
on the filesystem, where the order of entries in the environment path is
also very significant).

Post by Viacheslav Usov

Post by Philippe Verdy
I disagree.

With what, exactly?

Post by Philippe Verdy
In practice the linker

What linker is that, exactly?
How in your "I disagree" response did you consider the case of shared
libraries, which I have specifically addressed twice and which are the
whole point of this thread to begin with?
Cheers,
V.

[pocomane]

At my undestanding this is not related to the C standard

All external object and function references are resolved. Library components are linked to satisfy external references to functions and objects not defined in the current translation.

Infact, there are 2 translation units in the poposed scenario: one for
the executable, one for the shared library. In both, each symbol is
defined just once.

That references are resolved when the program is loaded in memory for
the execution. So the C compiler/linker/whatever is NOT in charge to
resolve the references. On linux machine, ld-linux-blabla.so is in
charge, i.e the the os dynamic linker/loader.

Now, I am not saying that the proposed procedure is a good pratice,
neither I am saying it is not. For sure it is not portable... I
mean... some platforms may even not have the support for the shared
libraries!

[Philipp Janda]

[...] A library is not meant to be totally incorporated into the
program; only modules that provide definitions for the program are
incorporated. Again, this is reasonable and convenient, and works
extremely well in practice. It has been like that for many decades.

The manual for GNU ld seems to support that. Extract from the

The linker will search an archive only once, at the location where it
is specified on the command line. If the archive defines a symbol
which was undefined in some object which appeared before the archive
on the command line, the linker will include the appropriate file(s)
from the archive.

And a small test:


$ cat a.c
#include <stdio.h>

void a(void) {
puts("A");
}


$ cat b.c
#include <stdio.h>

void a(void) {
puts("A");
}

void b(void) {
puts("B");
}


$ cat main.c
extern void a(void);
extern void b(void);

int main(void) {
a();
b();
return 0;
}


$ gcc -c a.c

$ gcc -c b.c

$ ar r liba.a a.o

$ ar r libb.a b.o

$ gcc -o main main.c a.o b.o
b.o: In function `a':
b.c:(.text+0x0): multiple definition of `a'
a.o:a.c:(.text+0x0): first defined here
collect2: error: ld returned 1 exit status

$ gcc -o main main.c b.o a.o
a.o: In function `a':
a.c:(.text+0x0): multiple definition of `a'
b.o:b.c:(.text+0x0): first defined here
collect2: error: ld returned 1 exit status

$ gcc -o main main.c a.o -L. -lb
./libb.a(b.o): In function `a':
b.c:(.text+0x0): multiple definition of `a'
a.o:a.c:(.text+0x0): first defined here
collect2: error: ld returned 1 exit status

$ gcc -o main main.c b.o -L. -la


So it seems to work in practice on Linux using my current version of the
GNU linker if the original object file is in an archive, the modified
object file is listed before the archive on the command line, and the
modified object file exports at least all the same symbols as the
original object file.


Philipp

[Philippe Verdy]

Post by Philipp Janda

The linker will search an archive only once, at the location where it
is specified on the command line. If the archive defines a symbol
which was undefined in some object which appeared before the archive
on the command line, the linker will include the appropriate file(s)
from the archive.

So it seems to work in practice on Linux using my current version of the
GNU linker if the original object file is in an archive, the modified
object file is listed before the archive on the command line, and the
modified object file exports at least all the same symbols as the
original object file.

This does not contradict what I said: having a priority order does not mean
that a "past" library will necessarily be read again: the linker just needs
to keep it (at least partly: the minimum is its table of symbols and the
module in which it is defined). Then it still resolved the symbols from the
set of "active" modules in order to determine the list of modules to keep,
and assemble new ones until there remains no unresoved symbols, but any
module that does not define any needed symbol for the assemblee can be
dropped from the assemblee (only its list of external symbols is kept); if
there remains unresolved symbols, the linker can only generate a partially
linked module (containing all their exported symbols). In all cases, the
priority order is preserved. But not all specifiied modules will be packed
together in the assemblee, but only what is needed (and notably not the
full set of unneeded modules that are packed in a library). The exception
is when building a library (all modules are kept), or a shared library
(only modules that contain exported symbols are kept along with their
dependant modules definining non-exported symbols needed by other modules).
If the result of linking must be an executable or a shared library, the
link will fail if there remains unresolved symbols, or what will be
produced will be a file in a intermediate object format which is not
executable, and contains the list of unresolved symbols (because such
format may still allow later binding in a next pass of the linker)

[Sean Conner]

Having consumed an inordiate amount of food earlier today [1] and
everything is fairly quiet and not much is going on, I thought I would
investigate this matter a bit further.

So without further ado ...

Post by Viacheslav Usov

Post by Dirk Laurie
1. All external names used by Lua are declared in header files.
2. If such a name has been resolved by an object file or library
linked earlier, the linker will be satisfied and will not replace it.
3. Any .o files needed by the linker will be built automatically from
.c files by the Makefile rules.

The standard has no concept of "linked earlier" and consequently no primacy
of "names linked earlier", and the entire program, including all the
libraries, shall have a function with a given name defined at most once.
Per clause 4/2, "If a ‘‘shall’’ or ‘‘shall not’’ requirement [...] is
violated, the behavior is undefined".

I'm not convinced. I'm quoting the C99 standard. First:

5.1.1.1:

1 ... Previously translated translation units may be preserved
individually or in libraries.... Translation units may be
separately translated and then later linked to produce an
executable program.

All this says is that compiled files (or "translation units" if you want)
can be saved as files, or stored in a "library" (which is not defined in the
standard, but is probably left undefined to give leeway to the
implementation) to avoid unnecessary compilation.

6.2.2:

2 In the set of translation units and libraries that constitutes an
entire program, each declaration of a particular identifier with
external linkage denotes the same object or function....

So here we see that a program can consist of individual translation units
and "libraries". This also says that each external declaration to a given
name will eventually resolve to the same object or function. Now, onto the
point that appears to have the concept of "linked earlier":

5.1.1.2:

8 All external object and function references are resolved. Library
components are linked to satisfy external references to functions
and objects not defined in the current translation. All such
translator output is collected into a program image which contains
information needed for execution in its execution environment.

I interpret this to mean, "if the given object files do not include an
object or function with external linkage, then said objects or functions can
be pulled from a "library". So let's go over the critical step in the
procedure:

make linux -e "LUA_O=lua.o myctype.o mylex.o"

Going through the makefiles for Lua 5.3, I find that LUA_O is defined as:

LUA_O= lua.o

Untangling the makefile (and I'll be following the Linux build since
that's what I'm using, but the others are similar) that when I do a

% make linux

the build will make the targets defined in ALL_T, which is defined as LUA_A,
LUA_T and LUAC_T. These are, respectively:

LUA_A= liblua.a
LUA_T= lua
LUAC_T= luac

These, in turn, depend upon

lua : lua.o
luac : luac.o
liblua.a : <object files except for the two listed above>

This eventually will turn into the command line:

gcc -std=gnu99 -o lua lua.o liblua.a -lm -Wl,-E -ldl -lreadline -lncurses

On Unix, files with a '.a' extension are a "library". The "-l" option is
just a shortcut to specifying other libraries one may link against, and
expanding everything out results in:

gcc -std=gnu99 -o lua lua.o liblua.a /usr/lib/libm.so -Wl,-E /usr/lib/libdl.so /usr/lib/libreadline.so /usr/lib/libncurses.so

(also on Unix, files with a '.so' extension are also a "library" but the
difference between the two is not important for this disussion)

Try to compile lua without specifying liblua.a and you get complaints
about missing external references (the first one is lua_sethook() by the
way). Adding back liblua.a and everything compiles, since the missing
functions are found in a library per 5.1.1.2/8.

Now, to the command in question:

% make linux -e "LUA_O= lua.o myctype.o mylex.o"

This will be (effectively) expanded out to:

gcc -std=gnu99 -o lua lua.o myctype.o mylex.o liblua.a /usr/lib/libm.so -Wl,-E /usr/lib/libdl.so /usr/lib/libreadline.so /usr/lib/libncurses.so

The file 'mylex.c' will presumedly provide definitions for the following
functions:

luaX_init()
luaX_setinput()
luaX_newstring()
luaX_next()
luaX_lookahead()
luaX_syntaxerror()
luaX_token2str()

The three "translation units" provide a number of external functions---the
missing ones will therefore be pulled in from liblua.a, so in this case,
yes, there *IS* a form of the concept of "linked earlier". The key sentence
from 5.1.1.2/8 is:

Library components are linked to satisfy external references to
functions and objects not defined in the current translation.

I would claim that lua.o, myctype.o and mylex.o constitude "the current
translation(s)" and that the functions in mylex.c will be used and the ones
that exist in the liblua.a "library" will not be, given the order in the key
sentance above.

Post by Viacheslav Usov
Note that what you do may fail in weird ways even with toolchains that do
have the notion of "linked earlier" - precisely because something has been
linked earlier. Your application may be using the "modified" functions,
while Lua itself and the other libraries use the "original" functions. This
would be a special case of generally undefined behaviour.

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Funny you mention that---I checked the C99 standard and found *nothing*
about this. It's not unspecified, it's not undefined, it's not
implementation defined, its' not locale specific, *nothing*. So yes, I
think this might be charitably noted as "generally undefined behavior" but
the C99 standard says nothing else about it.

But given the wording of 5.1.1.2/8, I don't think your concern is
valid---translations units are searched prior to libraries for external
references so any reference to a function X found in a translation unit,
regardless of where it's used (in the current translation units or a
library), will be used.

-spc (If at this point you don't think this interpretation holds, I'd like to
hear the argument against it)

[1] Okay, technically yesterday by the time this is going out, but I
still consider it the same day. Also, today is Thanksgiving Day in
the United States of America, a holiday [2] typically celebrated by
eating a large meal with friends and family and falling asleep
shortly thereafter.

[2] It's this weird quasi-religious/secular holiday were the purpose is
to give thanks for what you have to whatever deity or non-deity of
your (singular or plural) choice.

[Viacheslav Usov]

Post by Sean Conner
I interpret this to mean, "if the given object files do not include an

object or function with external linkage, then said objects or functions
can be pulled from a "library".

I cannot see how you reach this conclusion and this is exactly what the
rest of your argument hinges on. I'd say this also exactly where the
fundamental disagreement is.

5.1.1.2 #8 and 6.2.2 #2 that you quoted do not say that libraries are only
consulted if something was not resolved in some "primary" translation
units, and, as I said earlier, no such primacy is assigned to any
translation units by the standard.

Nor do they say anything like "parts of libraries". An "entire program"
includes full libraries. And here, you dismissed an important distinction
between static libraries and shared libraries. Static libraries (archives)
are typically just loose collections of object files not linked together.
So (we kiss good-buy to portability and conformance at this point) it kind
of makes sense to say that one of those object files can be pulled back
from the archive "on demand" and linked to the program without pulling in
other object files, so those other object files are kind of not really part
of the program, so whatever external symbols they have is irrelevant. This
behaviour is exactly what you and the other advocates of patchless patching
exploit - where it works, and I have not seen any attempt to demonstrate
that it works except with the GNU linker.

However, the whole point of patchless patching is about shared libraries,
because it is not that difficult to modify a static library and link your
application against this modified library. Shared libraries are not loose
collections of object files. They are pre-linked executables and they most
certainly bring with them all of their external symbols into the program. I
explained how this could result in complications previously and won't
repeat that. As far as I can see, no one in this entire thread has come up
with a show case of patchless patching for a shared library, while claims
like "the difference between [shared and static libraries] is not important
for this disussion [sic]" have been made.

Post by Sean Conner
Funny you mention that---I checked the C99 standard and found *nothing*

about this. It's not unspecified, it's not undefined, it's not
implementation defined, its' not locale specific, *nothing*.

The very first message of mine in this thread explained how having multiple
definitions of external linkage identifiers in an "entire program" is
undefined behaviour, quoting the standard. Given your interpretation as to
what an "entire program" is, you might indeed have difficulty seeing that.

Cheers,
V.

[Sean Conner]

Post by Sean Conner

Post by Sean Conner
I interpret this to mean, "if the given object files do not include an

object or function with external linkage, then said objects or functions
can be pulled from a "library".
I cannot see how you reach this conclusion and this is exactly what the
rest of your argument hinges on. I'd say this also exactly where the
fundamental disagreement is.
5.1.1.2 #8 and 6.2.2 #2 that you quoted do not say that libraries are only
consulted if something was not resolved in some "primary" translation
units, and, as I said earlier, no such primacy is assigned to any
translation units by the standard.

Again, the key sentence from 5.1.1.2 #8:

Library components

That is, translations units pre-compiled and stored in a library. The C
standard (as far as I can see) does not define "library", but from the
language I've read, it can store translation units and these translation
units can be referenced at a later time.

are linked

Again, "linked" (and "linkage" and "linking") are not defined by the C
standard, but the implication is that different translation units are
somehow combined so that all external references are satisfied.

to satisfy external references to functions and objects

As in right here.

not defined in the current translation.

And I think this is where we have our differenes. If the "current
translation" does not contain function foo(), *then* such a function is
looked for in any given libraries [1]. I ask, what does "not defined in the
current translation" mean to you?

Post by Sean Conner
Nor do they say anything like "parts of libraries".

The first two words of 5.1.1.2 #8---"Library components". I looked up
"component" in the Oxford English Dictionary and I found:

A constituent element or part

Post by Sean Conner
An "entire program" includes full libraries.

Citation please. The phrase "entire program" appears in 6.2.2#2:

In the set of translation units and libraries that constitutes an
entire program ...

I could not find the phrase "full library" (or variations) in the C99
Standard.

Post by Sean Conner
And here, you dismissed an important distinction
between static libraries and shared libraries. Static libraries (archives)
are typically just loose collections of object files not linked together.
So (we kiss good-buy to portability and conformance at this point) it kind

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Citation please, from the C standard (any one of C89, C99, or C11). It's
odd to think of GNU C as not being conformant. Or any of the commercial
compilers (don't worry, I'll address one of those in a bit).

Post by Sean Conner
of makes sense to say that one of those object files can be pulled back
from the archive "on demand" and linked to the program without pulling in
other object files, so those other object files are kind of not really part
of the program, so whatever external symbols they have is irrelevant. This
behaviour is exactly what you and the other advocates of patchless patching
exploit - where it works, and I have not seen any attempt to demonstrate
that it works except with the GNU linker.

I did some experiments last night on this very subject. At first, I used
Linux but curious, I went and did the same experiments on Solaris, so I'll
use the results from that. On the Solaris system in question, I used the
Sun Compiler Suite, so no GNU tools. And just to make sure:

% cc -V ; ld -V
cc: Sun C 5.12 SunOS_sparc 2011/11/16
ld: Software Generation Utilities - Solaris Link Editors: 5.10-1.1512

(and again, it would be odd to think of the Sun compiler as not being
conformant)

The experiment is a very small program---the function main() calls
func1(), which calls func2(). Each function prints it's been called and a
sample output would look like:

Hello from main
Hello from func1
Hello from func2

Experiment 1---main() is in on translation unit; func1() and func2() are in
another translation unit, which is stored in a library. Compile and run:

% cc -c -o main.o main.c

main() is compiled.

% cc -c -o func.o func.c
% ar rv libfuncall.a func.o
a - func.o
ar: creating libfuncall.a
ar: writing libfuncall.a

func1() and func2() in the same translation unit is compiled into a library.

% cc-o main1 main.o libfuncall.a

The two are linked.

% ./main1
Hello from main
Hello from func1
Hello from func2

And the output. This is the status quo. Next up, the "patchless patching
exploit" (to use your terms)---we have our own version of func2() which,
when called, will print "Hello from myfunc2":

% cc -c -o main.o main.c
% cc -c -o myfunc2.o myfunc2.c
% cc -c -o func.o func.c
% ar rv libfuncall.a func.o
a - func.o
ar: creating libfuncall.a
ar: writing libfuncall.a
% cc -o main2 main.o myfunc2.o libfuncall.a
ld: fatal: symbol 'func2' is multiply-defined:
(file myfunc2.o type=FUNC; file libfuncall.a(func.o) type=FUNC);
ld: fatal: file processing errors. No output written to main2

And this result does back your position---that we have two definitions of
the external function func2() (and for the record, I got this result on
Linux as well). Before I interpret this result, let's go on to experiments
three and four. In these two cases, func1() and func2() are in separate
translation units and both are stored in a library. Experiment 3---the base
line:

% cc -c -o main.o main.c
% cc -c -o func1.o func1.c
% cc -c -o func2.o func2.c
% ar rv libfunc.a func1.o func2.o
a - func1.o
a - func2.o
ar: creating libfunc.a
ar: writing libfunc.a
% cc -o main3 main.o libfunc.a
% ./main3
Hello from main
Hello from func1
Hello from func2

Nothing unusual here. Now, onto the "patchless patching exploit":

% cc -c -o main.o main.c
% cc -c -o myfunc2.o myfunc2.c
% cc -c -o func1.o func1.c
% cc -c -o func2.o func2.c
% ar rv libfunc.a func1.o func2.o
a - func1.o
a - func2.o
ar: creating libfunc.a
ar: writing libfunc.a
% cc -o main4 main.o myfunc2.o libfunc.a

That's odd---it compiled!

% ./main4
Hello from main
Hello from func1
Hello from myfunc2

And ran!

So, how do I interpret these results? In experiment 2, the translation
unit main had an unsatisfied external function, func1(). func1() is not
defined in the translation unit myfunc2, so we then examine the external
functions stored in the library libfuncall.a. There is a component of
libfuncall.a that has the external function func1(), but said component
*also* has external function func2(). When it comes time to link the
translation unit myfunc2, it also had an external function func2() and thus,
an error, because 6.9#5 states:

... somewhere in the entire program there shall be exactly one
external definition for the identifier; otherwise, there shall be no
more than one.

But now we get to experiment 4. This time, the translation unit main had
an unsatisfied external function func1(). func1() is not defined in the
translation unit myfunc2, so we then examine the external functions stored
in the library libfunc.a. There is a component of libfunc.a that has the
external function func1(), so that component is pulled in. That component
has an unsastified external function func2(). Said external function is not
defined in translation unit main, but it *is* defined in translation unit
myfunc2. There are no more unresolved external functions (or objects for
that matter) so we get the final program that works as expected (and again,
for the record, I got the same result on Linux).

In fact, these results seem (in my opinion) to be consistent with the
langauge in the C99 standard. I would also conjecture that you will find
the same results for static compilation across all C compilers.

I will conceed that there might exist a C compiler out there that does not
conform to these behaviors, but it would be as rare as coming across a C
compiler for a sign-magnitude or 1's-complement system [2].

Post by Sean Conner
However, the whole point of patchless patching is about shared libraries,
because it is not that difficult to modify a static library and link your
application against this modified library. Shared libraries are not loose
collections of object files. They are pre-linked executables

They are not pre-linked executables. Well, mostly. I know you can
execute libc under Linux:

[spc]lucy:/lib>./libc.so.6
GNU C Library stable release version 2.3.4, by Roland McGrath et al.
Copyright (C) 2005 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 3.4.6 20060404 (Red Hat 3.4.6-11).
Compiled on a Linux 2.4.20 system on 2010-04-18.
Available extensions:
GNU libio by Per Bothner
crypt add-on version 2.1 by Michael Glad and others
linuxthreads-0.10 by Xavier Leroy
The C stubs add-on version 2.1.2.
BIND-8.2.3-T5B
NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
Glibc-2.0 compatibility add-on by Cristian Gafton
GNU Libidn by Simon Josefsson
libthread_db work sponsored by Alpha Processor Inc
Thread-local storage support included.
For bug reporting instructions, please see:
<http://www.gnu.org/software/libc/bugs.html>.

but that's actually rare. If you try some other shared object (Linux):

[spc]lucy:/usr/lib>./libcrypt.so
Segmentation fault

or Solaris:

[spc]sol:/usr/lib>./libcrypt.so
Illegal Instruction

So I politely disagree with them being "pre-linked executables."

Post by Sean Conner
and they most
certainly bring with them all of their external symbols into the program. I
explained how this could result in complications previously and won't
repeat that. As far as I can see, no one in this entire thread has come up
with a show case of patchless patching for a shared library, while claims
like "the difference between [shared and static libraries] is not important
for this disussion [sic]" have been made.

Let me rectify that then. The same four experiments as above, in that
order, but this time with shared libraries. Again. on Solaris:

% cc -V ; ld -V
cc: Sun C 5.12 SunOS_sparc 2011/11/16
ld: Software Generation Utilities - Solaris Link Editors: 5.10-1.1512

Experment 1:

% cc -shared -xcode=pic32 -c -o func.ss func.c
% cc -shared -o libfuncall.so func.ss
% cc -Wl,-R/lusr/home/spc/foo -o smain1 main.o -L/lusr/home/spc/foo -lfuncall
% ldd ./smain1
libfuncall.so => ./libfuncall.so
libc.so.1 => /usr/lib/libc.so.1
libm.so.2 => /usr/lib/libm.so.2
/platform/SUNW,Sun-Fire-T1000/lib/libc_psr.so.1

This is to ensure we have an executable that loads our library at runtime.
It is, so let's run it:

% ./smain1
Hello from main
Hello from func1
Hello from func2

Now experiemnt two, the "patchless patching exploit."

% cc -c -o myfunc2.o myfunc2.c
% cc -Wl,-R/lusr/home/spc/foo -o smain2 main.o myfunc2.o -L/lusr/home/spc/foo -lfuncall
% ldd ./smain2
libfuncall.so => ./libfuncall.so
libc.so.1 => /usr/lib/libc.so.1
libm.so.2 => /usr/lib/libm.so.2
/platform/SUNW,Sun-Fire-T1000/lib/libc_psr.so.1

It compiled, unlike the second experiment with static libraries. But
let's see how it runs:

% ./smain2
Hello from main
Hello from func1
Hello from myfunc2

Wow! It worked! Even with func1() and func2() in the same translation
unit, func1() is calling func2() from translation unit myfunc2. And it's
not Linux! (for the record, it worked under Linux). And just to be
complete, experiments three and four. I won't comment much on these as they
too, work as the static version (even on Linux):

Experiment 3:

% cc -shared -xcode=pic32 -c -o func1.ss func1.c
% cc -shared -xcode=pic32 -c -o func2.ss func2.c
% cc -shared -o libfunc.so func1.ss func2.ss
% cc -Wl,-R/lusr/home/spc/foo -o smain3 main.o -L/lusr/home/spc/foo -lfunc
% ldd ./smain3
libfunc.so => ./libfunc.so
libc.so.1 => /usr/lib/libc.so.1
libm.so.2 => /usr/lib/libm.so.2
/platform/SUNW,Sun-Fire-T1000/lib/libc_psr.so.1
% ./smain3
Hello from main
Hello from func1
Hello from func2

Experiment 4:

% cc -Wl,-R/lusr/home/spc/foo -o smain4 main.o myfunc2.o -L/lusr/home/spc/foo -lfunc
% ldd ./smain4
libfunc.so => ./libfunc.so
libc.so.1 => /usr/lib/libc.so.1
libm.so.2 => /usr/lib/libm.so.2
/platform/SUNW,Sun-Fire-T1000/lib/libc_psr.so.1
% ./smain4
Hello from main
Hello from func1
Hello from myfunc2

Post by Sean Conner

Post by Sean Conner
Funny you mention that---I checked the C99 standard and found *nothing*

about this. It's not unspecified, it's not undefined, it's not
implementation defined, its' not locale specific, *nothing*.
The very first message of mine in this thread explained how having multiple
definitions of external linkage identifiers in an "entire program" is
undefined behaviour, quoting the standard.

That was 6.9#5, which I quoted a portion of, but here's the full quote:

5 An external definition is an external declaration that is also a
definition of a function (other than an inline definition) or an
object. If an identifier declared with external linkage is used in
an expression (other than as part of the operand of a sizeof
operator whose result is an integer constant), somewhere in the
entire program there shall be exactly one external definition for
the identifier; otherwise, there shall be no more than one.

I'm not reading "undefined behavior" there, I see "error" there. Annex J
of the C99 standard lists all the unspecified, undefined,
implementation-defined and locale-specific behaviors. Nowhere is this
addresses.

Look, I recognise you don't like this and think it's a violation of the C
Standard. I don't see it as a violation of the C Standard, but I'll grant
that is may be an unusual interpetation of the C Standard.

Post by Sean Conner
Given your interpretation as to
what an "entire program" is, you might indeed have difficulty seeing that.
Cheers,
V.

-spc (So did I use two non-comformant compilers for this experiment then?)

[1] In every C compiler I've used over the past 30 years, there is no
need to specify the Standard C library. There have been options to
tell the compiler *not* to reference the Standard C library, but
again, the C Standard is silent on that point.

[2] They exist, but are so rare that I would be surprised if anyone on
this list has used such a system.

[Philipp Janda]

Post by Sean Conner

Post by Viacheslav Usov
The very first message of mine in this thread explained how having multiple
definitions of external linkage identifiers in an "entire program" is
undefined behaviour, quoting the standard.

5 An external definition is an external declaration that is also a
definition of a function (other than an inline definition) or an
object. If an identifier declared with external linkage is used in
an expression (other than as part of the operand of a sizeof
operator whose result is an integer constant), somewhere in the
entire program there shall be exactly one external definition for
the identifier; otherwise, there shall be no more than one.
I'm not reading "undefined behavior" there, I see "error" there. Annex J
of the C99 standard lists all the unspecified, undefined,
implementation-defined and locale-specific behaviors. Nowhere is this
addresses.

This one is easy. Very first bullet point in Annex J.2.1:
J.2 Undefined behavior
1 The behavior is undefined in the following circumstances:
— A ‘‘shall’’ or ‘‘shall not’’ requirement that appears outside
of a constraint is violated (clause 4).

Which refers to 4.2 in the normative part of the standard:

2 If a ‘‘shall’’ or ‘‘shall not’’ requirement that appears
outside of a constraint is violated, the behavior is undefined.
Undefined behavior is otherwise indicated in this International
Standard by the words ‘‘undefined behavior’’ or by the omission
of any explicit definition of behavior. There is no difference
in emphasis among these three; they all describe ‘‘behavior that
is undefined’’.

Post by Sean Conner
-spc (So did I use two non-comformant compilers for this experiment then?)

Philipp

[Sean Conner]

Post by Philipp Janda

Post by Sean Conner

Post by Viacheslav Usov
The very first message of mine in this thread explained how having
multiple definitions of external linkage identifiers in an "entire
program" is undefined behaviour, quoting the standard.

5 An external definition is an external declaration that is also a
definition of a function (other than an inline definition) or an
object. If an identifier declared with external linkage is used in
an expression (other than as part of the operand of a sizeof
operator whose result is an integer constant), somewhere in the
entire program there shall be exactly one external definition for
the identifier; otherwise, there shall be no more than one.
I'm not reading "undefined behavior" there, I see "error" there. Annex
J of the C99 standard lists all the unspecified, undefined,
implementation-defined and locale-specific behaviors. Nowhere is this
addresses.

J.2 Undefined behavior
1 The behavior is undefined in the following circumstances: — A
‘‘shall’’ or ‘‘shall not’’ requirement that appears outside of a
constraint is violated (clause 4).
2 If a ‘‘shall’’ or ‘‘shall not’’ requirement that appears outside
of a constraint is violated, the behavior is undefined. Undefined
behavior is otherwise indicated in this International Standard by
the words ‘‘undefined behavior’’ or by the omission of any explicit
definition of behavior. There is no difference in emphasis among
these three; they all describe ‘‘behavior that is undefined’’.

Fair enough, I stand corrected on that point.

Post by Philipp Janda

Post by Sean Conner
-spc (So did I use two non-comformant compilers for this experiment then?)

Philipp

No answer to my question though?

-spc

[Philipp Janda]

Post by Sean Conner
No answer to my question though?

I'm not sure which question you mean, so I'll try to answer all
questions I can find in your last message.

Post by Sean Conner
I ask, what does "not defined in the
current translation" mean to you?

Probably the same as you, with the exception that I think "current
translation" refers to a single translation unit as in the descriptions
of the other translation phases.
But then we do agree on the fact that unresolved external symbols may be
looked up in a library. What we don't agree on is what is supposed to
happen if a library contains an external symbol that already is defined
somewhere else (either in a standalone translation unit or in a library).

Post by Sean Conner
So, how do I interpret these results?

You answered this one yourself.

Post by Sean Conner
So did I use two non-comformant compilers for this experiment
then?

Your compilers are probably fine. As soon as undefined behavior is
involved (which I claim is the case if you link multiple external
definitions), a compiler can do almost anything, and that includes
ignoring some symbols from libraries if there are definitions for those
symbols in other translation units.

Post by Sean Conner
-spc

Philipp

[Andrew Gierth]

Sean> -spc (So did I use two non-comformant compilers for this
Sean> experiment then?)

If you invoke undefined behavior, then the compiler is allowed to do
anything, where "anything" includes things like:

- the compiler did exactly what you expected
- the compiler did exactly what you expected, but only because of
defaults you don't know about, and changing those defaults to make
something else work will cause this to fail
- the compiler did something close enough to what you expected to
fool you into thinking it worked, but in fact it inserted code
to make it fail on alternate full moons
- the compiler made the thing you were trying to do work as you
expected, but subtly broke something somewhere else
- error messages
- nasal demons

Sean> The experiment is a very small program---the function main()
Sean> calls func1(), which calls func2(). Each function prints it's
Sean> been called and a sample output would look like:

Sean> Hello from main
Sean> Hello from func1
Sean> Hello from func2

[snip various experiments]

Sean> Now experiemnt two, the "patchless patching exploit."

Sean> % cc -c -o myfunc2.o myfunc2.c
Sean> % cc -Wl,-R/lusr/home/spc/foo -o smain2 main.o myfunc2.o -L/lusr/home/spc/foo -lfuncall
Sean> % ldd ./smain2
Sean> libfuncall.so => ./libfuncall.so
Sean> libc.so.1 => /usr/lib/libc.so.1
Sean> libm.so.2 => /usr/lib/libm.so.2
Sean> /platform/SUNW,Sun-Fire-T1000/lib/libc_psr.so.1

Sean> It compiled, unlike the second experiment with static
Sean> libraries. But let's see how it runs:

Sean> % ./smain2
Sean> Hello from main
Sean> Hello from func1
Sean> Hello from myfunc2

Sean> Wow! It worked! Even with func1() and func2() in the same
Sean> translation unit, func1() is calling func2() from translation
Sean> unit myfunc2. And it's not Linux! (for the record, it worked
Sean> under Linux).

It's worth noting here that linux and Solaris have very similar shared
library implementations, because the Solaris model was the one adopted
by GNU ld. However, other models do exist (for example the Windows
model), and not all of them will behave this way.

Furthermore, the ability to override functions in libraries this way is
often seen as a bug rather than a feature, especially when doing dynamic
loading. Accordingly, there are ways to change it: either by linking the
shared library with -Bsymbolic, or providing a link map, symbol version
file, exported symbol list, or explicit declaration that hides func2()
in the library build.

(now go and look at the definition of LUAI_FUNC in luaconf.h and
consider what that means for your approach)

[Viacheslav Usov]

Post by Sean Conner
And I think this is where we have our differenes. If the "current

translation" does not contain function foo(), *then* such a function is
looked for in any given libraries [1].

Correct. As long as there is exactly one definition of foo() in the entire
program, this is well-defined.

Having bar() defined in the "current translation" and some other library
would cause undefined behaviour, where your reasoning would no longer apply.

Post by Sean Conner
Again, the key sentence from 5.1.1.2 #8: Library components

This is not where an "entire program" is referenced.

Post by Sean Conner
I could not find the phrase "full library" (or variations) in the C99

Standard.

I used the word "full" to emphasise that libraries are not considered
piece-wise by the standard when it talks about an "entire program".

Post by Sean Conner
Citation please, from the C standard (any one of C89, C99, or C11).

Given that neither the notion of static vs shared libraries, not their
organisation, is addressed by the standard, any discussion at this level is
not portable, and any use of implementation details, in situations that
cause undefined behaviour in the standard, even if it is consistent within
the Implementation, is non-conformant. I stress the "use" not the "details".

Post by Sean Conner
It's odd to think of GNU C as not being conformant.

I did not say GNU C was non-coformant (even though it may be). As I
explained in earlier messages in this thread, it is the "program", not the
"tool", that is non-conformant if the one definition requirement is
violated.

Post by Sean Conner
I did some experiments last night on this very subject.

And, as far as I can tell, they were consistent with my description,
wherein the linker pulls previously archived object files from a static
library, ignoring other object files whenever it can. Which is perfectly
fine when there is no more than one definition of any external symbol in
the entire program, and is a trivially acceptable form of undefined
behaviour otherwise.

Post by Sean Conner
At first, I used Linux but curious, I went and did the same experiments

on Solaris

As mentioned by others, Linux and Solaris have a common Unix heritage and
adhere to other common standards, so one should expect similar behaviours.
So as a demonstration of portability, this is quite weak.

We have an elephant in the room here, and let me just name it: Windows.
Once somebody demonstrates similar techniques on Windows, we could talk
about real-world portability.

Post by Sean Conner
They are not pre-linked executables. Well, mostly. I know you can

execute libc under Linux:

To execute means more than just run something like a program in a shell.
And an executable means more than the main executable program file. Any,
anyway, the emphasis should have been on "pre-linked" rather than
"executable".

Post by Sean Conner
Wow! It worked! Even with func1() and func2() in the same translation

unit, func1() is calling func2() from translation unit myfunc2. And it's
not Linux! (for the record, it worked under Linux).

Great. Now you might go back to my response to Luiz and address concerns
"First" and "Second" there.

Post by Sean Conner
I'm not reading "undefined behavior" there, I see "error" there.

There is no "error" in the standard. What you call an "error", is
"undefined behaviour", because "If a ‘‘shall’’ or ‘‘shall not’’ requirement
[...] is violated, the behavior is undefined", which I said in my first
message in this thread.

Cheers,
V.

[Philippe Verdy]

consider these:

main.c:
#include<stdio.h>
main(){printf("Hello from main\n");func1();printf("Back to
main\n");func2();printf("Back to main\n");}

func1.c:
#include<stdio.h>
func1(){printf("\tHello from func1\n");func2();printf("\tBack to func1\n");}

func2.c:
#include<stdio.h>
func2(){printf("\t\tHello from func2\n");}

myfunc2.c:
#include<stdio.h>
func2(){printf("\t\Hello from myfunc2\n");}

Try creating a non-shared library containing func1.o and func2.o : OK, no
problem, there's no duplicate symbol. You can then link main.o with it to
create the main program.
Try creating a non-shared library containing func1.o and func2.o and
myfunc2.o : error, duplicate symbol (func2). You cannot link main.o with it
to create the main program.

Try creating a **shared** library "mylib.so" containing func1.o and func2.o
: OK, no problem, there's no duplicate symbol. You can then link main.o
with it to create the main program.
Now try linking main.o with myfunc2.o and mylib.so, what you get is this:

Hello from main
Hello from func1
Hello from func2 (sic!)
Back to func1
Back to main
Hello from myfun2 (sic!)
Back to main

In other words, the program is linked with two distinct versions of
func2(), which exist simultaneously! One version is statically used by
func1() from inside the sharedlibrary where func1 is used by main. The
second version (myfunc1.o) is however is now used by main but does **not**
replace the version used internally by func1() inside the shared library
(even if mylib.so also exports the symbol for "func2", this export is not
used when linking main because you've specified myfunc2.o **before**
mylib.so when linking main).

You cannot then replace the internal use made inside the shared libary
because the binding from func1 to func2 is already resolved internally in
the shared library, and the dependency of func1 with func2 is not exposed:
the shared libary only exposes a set of exported symbols.

You'd have the same situation with DLLs on Windows.

DLLs or shared libary are much more flexible and predictable than simple
libaries where no early binding occurs between units. Basically, the simple
library is just an archive containing multiple units, in an unspecified
order, so they cannot export the same symbol multiple times. If the archive
format allows it, it's just because it ignores compeltley the exported
symbols in each unit but only distinguish the unit names packed in the
library. The situation is different with shared libaries whose internal
links between units are already resolved by early binding, and these early
binding cannot be replaced.

What is not predictable is the order of units inside **non-shared**
libraries (that's why a shared libary cannot be made containing the same
symbol multiple times from distinct units in the library). But there's no
problem for the order of units inside a call to a linker. An no problem
inside shared libaries, because they are warrantied to export the same
symbol only once and have their dependencies already resolved inside it, so
they cannot export the same symbol multiple times.

A shared libary (or executable) however can be build to contain the same
symbol multiple times, but only one of these symbols is really exported:
the first one specified when linking it, but it does not mean that there
are not multiple implementations the shared library (or program) however
becomes an unbreakable new unit (that Windows calls a "module" if it is
executable, but this also applies to shared libaries on Linux using the ELF
binary format or similar, where early binding of internal symbols is
already made inside it and where sets of exported symbols are already
merged to a single set containing no duplicate, even if there are multiple
internal implementations).

Post by Sean Conner

Post by Sean Conner
And I think this is where we have our differenes. If the "current

translation" does not contain function foo(), *then* such a function is
looked for in any given libraries [1].
Correct. As long as there is exactly one definition of foo() in the entire
program, this is well-defined.
Having bar() defined in the "current translation" and some other library
would cause undefined behaviour, where your reasoning would no longer apply.

Post by Sean Conner
Again, the key sentence from 5.1.1.2 #8: Library components

This is not where an "entire program" is referenced.

Post by Sean Conner
I could not find the phrase "full library" (or variations) in the C99

Standard.
I used the word "full" to emphasise that libraries are not considered
piece-wise by the standard when it talks about an "entire program".

Post by Sean Conner
Citation please, from the C standard (any one of C89, C99, or C11).

Given that neither the notion of static vs shared libraries, not their
organisation, is addressed by the standard, any discussion at this level is
not portable, and any use of implementation details, in situations that
cause undefined behaviour in the standard, even if it is consistent within
the Implementation, is non-conformant. I stress the "use" not the "details".

Post by Sean Conner
It's odd to think of GNU C as not being conformant.

I did not say GNU C was non-coformant (even though it may be). As I
explained in earlier messages in this thread, it is the "program", not the
"tool", that is non-conformant if the one definition requirement is
violated.

Post by Sean Conner
I did some experiments last night on this very subject.

And, as far as I can tell, they were consistent with my description,
wherein the linker pulls previously archived object files from a static
library, ignoring other object files whenever it can. Which is perfectly
fine when there is no more than one definition of any external symbol in
the entire program, and is a trivially acceptable form of undefined
behaviour otherwise.

Post by Sean Conner
At first, I used Linux but curious, I went and did the same experiments

on Solaris
As mentioned by others, Linux and Solaris have a common Unix heritage and
adhere to other common standards, so one should expect similar behaviours.
So as a demonstration of portability, this is quite weak.
We have an elephant in the room here, and let me just name it: Windows.
Once somebody demonstrates similar techniques on Windows, we could talk
about real-world portability.

Post by Sean Conner
They are not pre-linked executables. Well, mostly. I know you can

To execute means more than just run something like a program in a shell.
And an executable means more than the main executable program file. Any,
anyway, the emphasis should have been on "pre-linked" rather than
"executable".

Post by Sean Conner
Wow! It worked! Even with func1() and func2() in the same translation

unit, func1() is calling func2() from translation unit myfunc2. And it's
not Linux! (for the record, it worked under Linux).
Great. Now you might go back to my response to Luiz and address concerns
"First" and "Second" there.

Post by Sean Conner
I'm not reading "undefined behavior" there, I see "error" there.

There is no "error" in the standard. What you call an "error", is
"undefined behaviour", because "If a ‘‘shall’’ or ‘‘shall not’’ requirement
[...] is violated, the behavior is undefined", which I said in my first
message in this thread.
Cheers,
V.

[Ivan Krylov]

On Sat, 24 Nov 2018 14:18:07 +0100

Post by Philippe Verdy
You'd have the same situation with DLLs on Windows.

Thank you for your example, I have added two __declspec(dllexport)'s

Post by Philippe Verdy
cl /c main.c
cl /c func1.c
cl /c func2.c
cl /c myfunc2.c
link /dll /out:mylib.dll func1.obj func2.obj

Creating library mylib.lib and object mylib.exp

Post by Philippe Verdy
link /out:main.exe main.obj myfunc2.obj mylib.lib
main

Hello from main
Hello from func1
Hello from func2
Back to func1
Back to main
Hello from myfunc2
Back to main

[Viacheslav Usov]

Post by Sean Conner
Hello from main
Hello from func1
Hello from func2
Back to func1
Back to main
Hello from myfunc2
Back to main

Note that the same code but with the static func.lib produces

Hello from main
Hello from func1
Hello from myfunc2
Back to func1
Back to main
Hello from myfunc2
Back to main

Which demonstrates how undefined behaviour can produce different results
even in one and the same implementation.

This also illustrates the invalidity of the claim that "the difference
between [shared and static libraries] is not important for this disussion
[sic]", and why "pre-linked" is an important aspect of shared libraries.

Cheers,
V.

[Philippe Verdy]

If you use non-shared libraries, in fact the result is unpredictable, you
could as well have seen:

Hello from main
Hello from func1
Hello from func2
Back to func1
Back to main
Hello from func2
Back to main

That's what I was demonstrating: there's undefined behavior ONLY if you use
non-shared libraries because the order of units in these libraries is NOT
significant.
And in fact makes the use of "pre-linked" even more relevant to this
discussion.

Only non-shared libraries are not portable if you have multiple units
defining the same symbol.
But you are not required to use them if, instead, you use a linker directly
and specify a full list of units names (or shared library names) where the
order of resolution is entirely determined.

[Philippe Verdy]

Note that Windows and the ELF format in general allows a shared library to
be "pre-linked" within its set of internal references, but still be
replaceable by a linker: the "pre-link" is kept as a default if there's no
other implementation specified but this default is overridable if you
specify a prior unit taking the priority, in which case that unit will
become the implementation used by the shared libary itself at run-time (the
default implementation will still be part of the modules possibly loaded in
memory but they won't be used (and a VMM may not even need to page in these
unused parts if they are larger than a single memory page, so this won't
take too much memory; this unused implementation will still use some amount
of virtual memory)...
For that case, the same symbol may be present in the list of imports and
the list of exports in a "pre-linked" module (executable file, or
shared-library)...

A "pre-linked" module may also specify restrictions in their list of
imports, such as specifying that the symbol must be loaded from specific
module names, or specifying that it must be in modules matching a specific
digital signature, or other security constraints (like version number). The
module also specifies also for its list of exports a set of properties,
that allows other modules to check these constraints, notably the export
list contains the name of the module itself, its version, and some data
signature or globally unique identifier.

Post by Philippe Verdy
If you use non-shared libraries, in fact the result is unpredictable, you
Hello from main
Hello from func1
Hello from func2
Back to func1
Back to main
Hello from func2
Back to main
That's what I was demonstrating: there's undefined behavior ONLY if you
use non-shared libraries because the order of units in these libraries is
NOT significant.
And in fact makes the use of "pre-linked" even more relevant to this
discussion.
Only non-shared libraries are not portable if you have multiple units
defining the same symbol.
But you are not required to use them if, instead, you use a linker
directly and specify a full list of units names (or shared library names)
where the order of resolution is entirely determined.

[Philippe Verdy]

Also note that a POSIX system does not need any specific file format for
**non-shared** libraries: they can just be represented simply as a simple
archive (similar to a .zip file) or as a directory. In both cases, trhe
units stored in these archives or directories are in random order (which is
not warrantied to be stable across time, notably if you add or remove units
or any other files from the archive or directory). The archive or directory
may expose the list of their contents in arbitrary random order, or ordered
in some arbitry collation order (depending on user's locale), so this is
not stable and portable.

But you can make the order of units predictable by adding a supplementary
"manifest" file in this directory or archive, to specify the explicit order
in which units present in that directory or archive must be resolved.

Post by Philippe Verdy
Note that Windows and the ELF format in general allows a shared library to
be "pre-linked" within its set of internal references, but still be
replaceable by a linker: the "pre-link" is kept as a default if there's no
other implementation specified but this default is overridable if you
specify a prior unit taking the priority, in which case that unit will
become the implementation used by the shared libary itself at run-time (the
default implementation will still be part of the modules possibly loaded in
memory but they won't be used (and a VMM may not even need to page in these
unused parts if they are larger than a single memory page, so this won't
take too much memory; this unused implementation will still use some amount
of virtual memory)...
For that case, the same symbol may be present in the list of imports and
the list of exports in a "pre-linked" module (executable file, or
shared-library)...
A "pre-linked" module may also specify restrictions in their list of
imports, such as specifying that the symbol must be loaded from specific
module names, or specifying that it must be in modules matching a specific
digital signature, or other security constraints (like version number). The
module also specifies also for its list of exports a set of properties,
that allows other modules to check these constraints, notably the export
list contains the name of the module itself, its version, and some data
signature or globally unique identifier.

Post by Philippe Verdy
If you use non-shared libraries, in fact the result is unpredictable, you
Hello from main
Hello from func1
Hello from func2
Back to func1
Back to main
Hello from func2
Back to main
That's what I was demonstrating: there's undefined behavior ONLY if you
use non-shared libraries because the order of units in these libraries is
NOT significant.
And in fact makes the use of "pre-linked" even more relevant to this
discussion.
Only non-shared libraries are not portable if you have multiple units
defining the same symbol.
But you are not required to use them if, instead, you use a linker
directly and specify a full list of units names (or shared library names)
where the order of resolution is entirely determined.

[Viacheslav Usov]

Post by Philippe Verdy
That's what I was demonstrating: there's undefined behavior ONLY if you

use non-shared libraries because the order of units in these libraries is
NOT significant.

The C standard has no notion of shared (or not) libraries, so this
distinction is irrelevant. Per the standard, multiple external symbols are
undefined behaviour no matter how that is managed in any given
implementation. The standard has no notion of the "order of units", so what
you say here and later about it is meaningless per the standard.

Post by Philippe Verdy
Only non-shared libraries are not portable if you have multiple units

defining the same symbol.

Again, multiple definitions of an external symbol cause undefined behaviour
per the C standard, so there is no portability under the standard.

Speaking of real-word portability, so far it has only been demonstrated
that one gets consistent results using shared libraries on Windows and
(probably) Linux (and, pedantically, it is unknown if this is stable if we
use different linkers). The behaviour on Solaris, while consistent with
itself, seems different, wherein the patched function seems to be used even
when called indirectly, while the former platforms use it only when it is
directly called. It is possible that some linker option magic can make the
behaviour consistent among all three platforms, but that is yet to be
demonstrated.

For the record, the demonstrated behaviour on Solaris seems more useful for
patching than that on Linux and Windows, but I suspect there ain't no such
thing as a free lunch so we are yet to see what tradeoffs that really
involves.

Cheers,
V.

[Sean Conner]

Post by Viacheslav Usov
Speaking of real-word portability, so far it has only been demonstrated
that one gets consistent results using shared libraries on Windows and
(probably) Linux (and, pedantically, it is unknown if this is stable if we
use different linkers).

No, the output I saw under Linux was consistent with Solaris, NOT Windows.

Post by Viacheslav Usov
The behaviour on Solaris, while consistent with
itself, seems different, wherein the patched function seems to be used even
when called indirectly, while the former platforms use it only when it is

^^^^^^^^^^^^^^^^

Post by Viacheslav Usov
directly called.

Again, Windows only exhibited that behavior, not Linux.

Post by Viacheslav Usov
For the record, the demonstrated behaviour on Solaris

and Linux

Post by Viacheslav Usov
seems more useful for
patching than that on Linux and Windows, but I suspect there ain't no such
thing as a free lunch so we are yet to see what tradeoffs that really
involves.

-spc

[Viacheslav Usov]

Post by Sean Conner
No, the output I saw under Linux was consistent with Solaris, NOT Windows.

Since you did not say that in the original message that had only the output
on Solaris, I could not not assume that.

Instead, I relied on the message from Philippe Verdy which had this output:

Hello from main
Hello from func1
Hello from func2 (sic!)
Back to func1
Back to main
Hello from myfun2 (sic!)
Back to main

His message did not say that it was done on Linux, that was my guess. And
it did not show the command line options he used to get the result, nor did
it specify the details of the toolchain.

ELF shared libraries allow for symbol interposition, so that a shared
library may end up calling a function defined elsewhere even if it is also
defined in the shared library. This is I think what your results
demonstrate. However, that is both inefficient and questionable from the
security standpoint so there are ways controlled by a bunch of different
options to suppress this behaviour. Even default behaviour in this respect
may have changed over the years and tool versions, so it is not completely
surprising that you and Philippe got different results.

On the other hand, I do not think such behaviour is available with PE DLLs.
I am not sure about MacOS.

I'd say all this demonstrates "undefined behaviour" quite convincingly.

Cheers,
V.

[Sean Conner]

Post by Viacheslav Usov

Post by Sean Conner
No, the output I saw under Linux was consistent with Solaris, NOT

Windows.
Since you did not say that in the original message that had only the output
on Solaris, I could not not assume that.

I did say that. Twice. Granted, they were in parentheticals but they
are there. And the reason I reported the Solaris results is that you
explicitely called out the GNU tools as a possible reason, and I wanted to
get a counter to that.

I could have probably been clearer on that point though.

Post by Viacheslav Usov
Hello from main
Hello from func1
Hello from func2 (sic!)
Back to func1
Back to main
Hello from myfun2 (sic!)
Back to main
His message did not say that it was done on Linux, that was my guess. And
it did not show the command line options he used to get the result, nor did
it specify the details of the toolchain.
ELF shared libraries allow for symbol interposition, so that a shared
library may end up calling a function defined elsewhere even if it is also
defined in the shared library. This is I think what your results
demonstrate. However, that is both inefficient and questionable from the
security standpoint so there are ways controlled by a bunch of different
options to suppress this behaviour. Even default behaviour in this respect
may have changed over the years and tool versions, so it is not completely
surprising that you and Philippe got different results.
On the other hand, I do not think such behaviour is available with PE DLLs.

Ivan Krylov did the experiment under Windows (gave the commands, etc). He
was able to replicate my results with static compilation (with separate
translation units in the library), but the DLL version behaved differently
than Solaris/Linux.

Post by Viacheslav Usov
I am not sure about MacOS.

I am not sure either.

Post by Viacheslav Usov
I'd say all this demonstrates "undefined behaviour" quite convincingly.

True. But I did find this, written by Microsoft, about the Microsoft LINK
program (about statically linking):

For example, imagine that a C programmer has written two versions of
a function named _myfunc()_ that is called by the program MYPROG.C.
One version of _myfunc()_ is for debugging; its object module is
found in MYFUNC.OBJ. The other is a production version whose object
module resides in MYLIB.LIB. Under normal circumstances, the
programmer links the production version of _myfunc()_ by using
MYLIB.LIB. To use the debugging version of _myfunc()_, the
programmer explicitly includes its object module (MYFUNC.OBJ) when
LINK is executed. This causes LINK to build the debugging version
of _myfunc()_ into the executable file because it encounters the
debugging version in MYFUNC.OBJ before it finds the other version in
MYLIB.LIB.

To exploit the order in which LINK resolves external references, it
is important to know LINK's library search strategy: Each individual
library is search repeatedly (from first library to last, in the
sequence in which they are input to LINK) until no further external
references can be resolved.

_The MS-DOS Encyclopedia_

Given Microsoft's history of backwards compatability, I'm sure this is
still true to this day.

-spc

[Viacheslav Usov]

Post by Sean Conner
I did say that. Twice. Granted, they were in parentheticals but they

are there.

One of them was "(for the record, it worked under Linux)", whence I could
not deduce it produced the same output.

Post by Sean Conner
_The MS-DOS Encyclopedia_

When it comes to static linking, you might as well use a contemporary
source:

"Object files on the command line are processed in the order they appear on
the command line. Libraries are searched in command line order as well,
with the following caveat: [...]"

https://docs.microsoft.com/en-us/cpp/build/reference/link-input-files?view=vs-2017

My point was about PE DLLs, though. In PE, there is a clear distinction
between imports and exports. To create an equivalent of an elfian public
symbol in a shared library, a DLL export would also have to be an import
and would need a body, which to my knowledge is impossible using
Microsoft's tools. I am not sure whether one could do that with a special
tool exploiting PE's dark corners, but then we would end up with the
"original" DLL's having to be built in some weird (for Windows) way, which
kind of nullifies the whole point of the exercise.

Cheers,
V.

[Philippe Verdy]

DLLs in Windows have a dllEntry point which can be used for various
customization of dynamic linking and resolution of dependencies. DllEntry
points are normal C functions, comparable to the main function in C, but
taking other arguments (not from a command line, but that can still use the
environment if needed and can modify it, however it cannot load other
modules/libaries itself due to possible deadlock conditions; as well it
cannot start or terminate any thread or process, instead it returns a
status that allows the loader to refuse to create or terminate a thread or
process).

It is called when first loading the DLL but before actual linking to other
programs (actually this is used not only at the per-process level but also
for each thread). If the DLL does not need per-thread initialization (e.g.
if it does not allocate any TLS data for each thread to keep track of
external resources used by the thread when they use the exported DLL
functions, they can disable these notifications and the DLL entry will only
be called on process termination). The DLL entry point can control the
search path and load other libraries itself.

Post by Sean Conner

Post by Sean Conner
I did say that. Twice. Granted, they were in parentheticals but they

are there.
One of them was "(for the record, it worked under Linux)", whence I could
not deduce it produced the same output.

Post by Sean Conner
_The MS-DOS Encyclopedia_

When it comes to static linking, you might as well use a contemporary
"Object files on the command line are processed in the order they appear
on the command line. Libraries are searched in command line order as well,
with the following caveat: [...]"
https://docs.microsoft.com/en-us/cpp/build/reference/link-input-files?view=vs-2017
My point was about PE DLLs, though. In PE, there is a clear distinction
between imports and exports. To create an equivalent of an elfian public
symbol in a shared library, a DLL export would also have to be an import
and would need a body, which to my knowledge is impossible using
Microsoft's tools. I am not sure whether one could do that with a special
tool exploiting PE's dark corners, but then we would end up with the
"original" DLL's having to be built in some weird (for Windows) way, which
kind of nullifies the whole point of the exercise.
Cheers,
V.

[Viacheslav Usov]

Post by Philippe Verdy
In other words, the program is linked with two distinct versions of

func2(), which exist simultaneously! One version is statically used by
func1() from inside the sharedlibrary where func1 is used by main. The
second version (myfunc1.o) is however is now used by main but does **not**
replace the version used internally by func1() inside the shared library
(even if mylib.so also exports the symbol for "func2", this export is not
used when linking main because you've specified myfunc2.o **before**
mylib.so when linking main).

Which is part of concern "First" in my response to Luiz earlier in this
thread, to which you responded with "I disagree". I still do not know what
exactly you disagree with, because you never answered my follow-on
questions specifically enough.

You might also want to experiment with concern "Second".

Cheers,
V.

[Ivan Krylov]

Hi!

On Sat, 24 Nov 2018 13:04:50 +0100

Windows. Once somebody demonstrates similar techniques on Windows, we
could talk about real-world portability.
cl /c main.c
cl /c func1.c
cl /c func2.c
cl /c myfunc2.c
lib /out:func.lib func1.obj func2.obj
link /out:main.exe main.obj myfunc2.obj func.lib
main.exe

main
func1
myfunc2

First, other libraries that the executable depends on will not have
been linked statically to the same "replacement" functions, so they
will have to resort to the "original" dependencies. Your program, as
whole, may end up using different versions of the same set of
functions simultaneously.

I've certainly encountered such nasty cases in the past when I was
learning C, but right now I cannot produce such an example that would
also be concise and easy to reproduce. If you have one handy, I would be
glad to test it on Windows, too.

[Philippe Verdy]

There's no elephant here, Windows is not different from Unix or other
POSIX-compatible systems when it handles simple libraries or shared
libraries (DLLs). Not all POSIX systems support shared libraries, but in
that case, it is only the possibility of created **non-shared** libraries
that can produce unspecified behavior, because **non-shared** libaries are
unordered collections of units (the order is unpredictable), so they cannot
contain multiple distinct units defining the same exported symbol.
This problem does not occur with shared libaries/DLLs whose internal
dependencies are fully resolved and the rest of their unresolved symbols
(that are not part of the list of their own exported symbols) is restricted
to be a set of unique identifiers: the order or resolution is fully
specified with shared libraries.

Post by Ivan Krylov
Hi!
On Sat, 24 Nov 2018 13:04:50 +0100

Windows. Once somebody demonstrates similar techniques on Windows, we
could talk about real-world portability.
cl /c main.c
cl /c func1.c
cl /c func2.c
cl /c myfunc2.c
lib /out:func.lib func1.obj func2.obj
link /out:main.exe main.obj myfunc2.obj func.lib
main.exe

main
func1
myfunc2

First, other libraries that the executable depends on will not have
been linked statically to the same "replacement" functions, so they
will have to resort to the "original" dependencies. Your program, as
whole, may end up using different versions of the same set of
functions simultaneously.

I've certainly encountered such nasty cases in the past when I was
learning C, but right now I cannot produce such an example that would
also be concise and easy to reproduce. If you have one handy, I would be
glad to test it on Windows, too.
--
Best regards,
Ivan

[Sean Conner]

Post by Viacheslav Usov
We have an elephant in the room here, and let me just name it: Windows.
Once somebody demonstrates similar techniques on Windows, we could talk
about real-world portability.

Ivan Krylov ran a similar experiment on Windows, and with static linking,
each function in its own translation unit and turned into a library. Then

Post by Viacheslav Usov
cl /c main.c
cl /c func1.c
cl /c func2.c
cl /c myfunc2.c
lib /out:func.lib func1.obj func2.obj
link /out:main.exe main.obj myfunc2.obj func.lib
main.exe

main
func1
myfunc2

Which matches what I saw on Linux and Solaris in the same scenario. He
then did it with shared objects (DLLs), each function in its own translation

Post by Viacheslav Usov
cl /c main.c
cl /c func1.c
cl /c func2.c
cl /c myfunc2.c
link /dll /out:mylib.dll func1.obj func2.obj

Creating library mylib.lib and object mylib.exp

Post by Viacheslav Usov
link /out:main.exe main.obj myfunc2.obj mylib.lib
main

Hello from main
Hello from func1
Hello from func2
Back to func1
Back to main
Hello from myfunc2
Back to main

Which is different from what I saw on Linux and Solaris in the same
scenario:

Hello from main
Hello from func1
Hello from myfunc2

-spc

[Philippe Verdy]

Post by Sean Conner

Post by Viacheslav Usov
We have an elephant in the room here, and let me just name it: Windows.
Once somebody demonstrates similar techniques on Windows, we could talk
about real-world portability.

Ivan Krylov ran a similar experiment on Windows, and with static linking,
each function in its own translation unit and turned into a library. Then

Post by Viacheslav Usov
cl /c main.c
cl /c func1.c
cl /c func2.c
cl /c myfunc2.c
lib /out:func.lib func1.obj func2.obj
link /out:main.exe main.obj myfunc2.obj func.lib
main.exe

main
func1
myfunc2
Which matches what I saw on Linux and Solaris in the same scenario. He
then did it with shared objects (DLLs), each function in its own translation

Post by Viacheslav Usov
cl /c main.c
cl /c func1.c
cl /c func2.c
cl /c myfunc2.c
link /dll /out:mylib.dll func1.obj func2.obj

Creating library mylib.lib and object mylib.exp

Post by Viacheslav Usov
link /out:main.exe main.obj myfunc2.obj mylib.lib
main

Hello from main
Hello from func1
Hello from func2
Back to func1
Back to main
Hello from myfunc2
Back to main
Which is different from what I saw on Linux and Solaris in the same
Hello from main
Hello from func1
Hello from myfunc2

This last is certainly not the same scenario ! My scenario made it clear
which version was used by testing not the two calls only from main, but
also testing the second function when it is called from the 1st one.

This is exactly where you can see that simple (unlinked) libraries do not
behave like prelinked libaries, and that it is the use of simple libaries
which is completely inconsistant (independantly of the language you used to
create them, it may be C or anothy other).

Simple libraries ARE NOT suitable and NOT compatible with the C standard.
They can never be strictly portable. Shared libaries (prelinked) is
necessary for portability beause this ensures a strict and stable order for
resolving external dependencies. In summary don't use imple libaries at all
with C! Instead use the linker and provide the **full list** of the
libraries you want to link to, in which you'll specify their expected order
or resolution

The basic syntax like "-lm" for invokiner the linker or the C compiler with
linker capabilities is inconsistant. You can never predict which unit will
be included, unless the ".a" libraries are built to ensure that they will
NEVER contain any pair of units defining the same symbol (i.e. each symbol
defined in the library belongs to one and only one unit of the library).

The old "ar" tool of Unix (or "tar" or "shar", which give equivalent
results) is clearly buggy, it's not different from a simple ".zip" file, or
from a filesystem directory containing all compiled .or units listed in an
unspecified/unstable order (and not containing any additional metadata file
like a "manifest" to specify the order of resolution) ! The old "*ar" tools
have never been seriously designed to be suitable for programming, they
were made only for backing up and transfering collections of files of any
type between systems by packing them ins a single file transferable in a
single session (e.g. by FTP) !

There's no such limitation and portability issue when you don't use ANY
static library but only use "prelinked" shared libraries (like DLL on
windows or ELF libraries on Linux.. note that executable and DLL modules on
Windows, as well as on OS/2, are based on directly on the ELF format) !

Let's be serious! Prelinked library formats (or archives containing
manifests) are the only suitable formats for developping in any language
allowing separate compilations of units (now almost all of them); the
alternative is to use unit names with their full pathname (and not just the
unit name) so that you can choose between multiple possible implementations
in separate modules (with distinct paths) defining the same exported symbol.

[Sean Conner]

Post by Philippe Verdy

Post by Ivan Krylov

cl /c main.c
cl /c func1.c
cl /c func2.c
cl /c myfunc2.c
link /dll /out:mylib.dll func1.obj func2.obj

Creating library mylib.lib and object mylib.exp

link /out:main.exe main.obj myfunc2.obj mylib.lib
main

Hello from main
Hello from func1
Hello from func2
Back to func1
Back to main
Hello from myfunc2
Back to main
Which is different from what I saw on Linux and Solaris in the same
Hello from main
Hello from func1
Hello from myfunc2

This last is certainly not the same scenario ! My scenario made it clear
which version was used by testing not the two calls only from main, but
also testing the second function when it is called from the 1st one.

Okay.

% uname
Linux
% cc -c -o main.o main.c
% cc -c -o myfunc2.o myfunc2.c
% cc -shared -fPIC -o func.ss func.c
% ar rv libfuncall.so func.ss
% ar: creating libfuncall.so
% a - func.ss
% cc -Wl,-rpath,/tmp/foo -o smain2 main.o myfunc2.o libfuncall.so
% ./smain2
Hello from main
Hello from func1
Hello from myfunc2
Back to func1
Back to main
Hello from myfunc2
Back to main

Happy now?

Post by Philippe Verdy
There's no such limitation and portability issue when you don't use ANY
static library but only use "prelinked" shared libraries (like DLL on
windows or ELF libraries on Linux.. note that executable and DLL modules on
Windows, as well as on OS/2, are based on directly on the ELF format) !

Citation needed.

-spc

[Philippe Verdy]

Post by Sean Conner
% uname
Linux
% cc -c -o main.o main.c
% cc -c -o myfunc2.o myfunc2.c
% cc -shared -fPIC -o func.ss func.c
% ar rv libfuncall.so func.ss
% ar: creating libfuncall.so
% a - func.ss
% cc -Wl,-rpath,/tmp/foo -o smain2 main.o myfunc2.o libfuncall.so
% ./smain2
Hello from main
Hello from func1
Hello from myfunc2
Back to func1
Back to main
Hello from myfunc2
Back to main
Happy now?

That's what I wanted. And demonstrates what I wanted to show: this is the
only portable and expected behavior !

Post by Sean Conner

Post by Philippe Verdy
There's no such limitation and portability issue when you don't use ANY
static library but only use "prelinked" shared libraries (like DLL on
windows or ELF libraries on Linux.. note that executable and DLL modules on
Windows, as well as on OS/2, are based on directly on the ELF format) !

Citation needed.

I gave citations as examples, because there's nowhere any other counter
example using shared/prelinked libraries (or archive formats containing an
explicit manifest of the expected link order) any example I can find where
this is not true.

So instead I ask you to demonstrate the existence of any counter-example !

The C standard itself does not really indicate how we will link units into
a working program: it does not even need the existence of "libraries", it
just speaks the possibility of creating programs using collections of units
compilable separately (that's thbe meaning of the "extern" keyword in C),
and then it needs to use an external linker and to specify the units you
need for a working program in a well defined resolution order (and basic
libaries which are only collections of units in unspecified order are not
compatible with this goal).

But shared libaries/ prelinked libraries are compatible with this goal as
their role is to prelink them partially, using also the same linker and the
same specified order, which then gives a predictable resolution order for
symbols in the library (something impossible to reach using only basic
static libraries containining arbitrary number of object units in random
order).

It's not a basic "librarian" tool that resolves symbols (the librarian is
just a way to archive and pack multiple files into a single one, because it
is generally faster to process than large collections of small files, from
which they can be extracted without modification; the speed of processing
was certinaly true with past filesystems, but it's no longer the case with
modern filesystems). Only a true linker (actually made for building
programs) does that work of resolving symbols accurately and predictably!

[Philippe Verdy]

In fact the C standar NEVER forbids different compiled units to redefine
the same external symbols.

In fact it even **encourages** such use, which is required for example so
that all C programs can have their own implementation of the "main()"
function which is externalized to the same symbol once compiled with the
same "C" linkage by default.

Then it's not the C language or the compiler that instruct themselves how
these compiled units will be linked together. This is only specified at the
linker level (which is not part of the C standard itself). But all linkers
require you specify unambiguously the order of units.

If **any** linker allows you to use "libraries" (containing multiple units
in random order), it can accept to process them ONLY if there's no pair of
distinct units in the same library that define the same symbol (otherwise
the result would be completely unpredictable).

Shared libraries (or libraries and directories containing an ordering
manifest file) are assured to respect this constraint because they are
already the result of a prevcious pass of the linker for which the order or
resolution was already specified and checked.

Shared libraries are not a requirement for any POSIX system compatible with
C: these systems can work perfectly without supporting them (it's enough on
these system to have a linker tool that supports a manifest in their
supported archive format, or that will look for the existence of a manifest
file if libraries are represented as a filesystem directory containing all
compiled units).

In practice, archive formats recognized by linkers **already look for a
manifest file** containing not just the order of units, but a full mapping
index of symbols, associating them with the name of the unit in which they
are defined and exported, because it allows much faster linking, without
having to process completely each unit file inside the library: such unique
mapping index cannot be built and inserted inside the library if there are
two units in the library definining the same symbol.

But:
- old archiving tools (like old versions of "ar" on old versions of Unix)
did not check that and did not build this index/manifest, it had to be
built and added separately to the archive, and updated each time you
added/updated/removed a unit from the library. The old "ar" tool is now
completely deprecated. For backup/archiving purpose, "tar" is much better
and universally supported on all Unix/Linux variants, and now most archives
are compressed ("taz", "tgz, "zip", "gz", "xz", you have the choice...)
- archive formats containing a manifest or index solve the problem for
linkers (this solution is used by linkers inside virtual machines like
Java, .Net, Perl, Python... (Even Lua uses such solution even if it's
hidden behind the concept of "loaders", which are actually linkers that
programs themselves can control and tune for their needs)
- shared libaries are in fact much cleaner, more compact, and much faster
to process to create native programs: the ELF format (or similar variants)
is now almost universal between all Unix/Linux/Windows and many other
systems (and they still allow compiled programs to invoke or use the system
linker system themselves using "loaders")

This solution based on shared libraries (or archives with manifests) and
the concept of generic "loaders" is not just for linking standalone
programs, but it also exists in scripting languages and programs runing on
virtual machines: these programs can control themselves the resolution
order, control themselves the environment path in which external libraries
or units will be found, they can check contraints like security
requirements, access rights, digital signatures; they can use network
services to download the units; they can use conditions like the user's
locale and other preferences, they can try to best match the architecture
such as i386 vs x64 vs. i686 when they have the choice, they can perform
comparative benchmark tests before deciding which implementation to use...

**Absolutely nothing in the C standard** forbids units needed or used in
the same program to be limited to sets of unique symbols !

All what the C standard says, is that a separate unit will never be created
by the compiler such that it contains multiple instances of the same extern
symbol with the "C" linkage (Some compilers for specific systems may be
exceptions to this rule : they may still create **simultaneously** multiple
implementations of the same source, compiled for different architectures,
or for different goals such as debugging purpose, or different levels or
methods of optimization which may not be safe in all situations such as
relocatable vs. reentrant versions for multithreading; in which case the
object format will actuall be like an archive with several distinct entry
points for the same symbol but distinguished by some encoded goals, or by
some "decoration" in the encoded exported symbols; but this is equivalent
to exporting distinct symbols; this works only if the linker recognizes the
multiple encoded symbols and knows the rules to locate them and to decide
predicatably which implementation are the most suitable)

Post by Philippe Verdy

Post by Sean Conner
% uname
Linux
% cc -c -o main.o main.c
% cc -c -o myfunc2.o myfunc2.c
% cc -shared -fPIC -o func.ss func.c
% ar rv libfuncall.so func.ss
% ar: creating libfuncall.so
% a - func.ss
% cc -Wl,-rpath,/tmp/foo -o smain2 main.o myfunc2.o
libfuncall.so
% ./smain2
Hello from main
Hello from func1
Hello from myfunc2
Back to func1
Back to main
Hello from myfunc2
Back to main
Happy now?

That's what I wanted. And demonstrates what I wanted to show: this is the
only portable and expected behavior !

Post by Sean Conner

Post by Philippe Verdy
There's no such limitation and portability issue when you don't use ANY
static library but only use "prelinked" shared libraries (like DLL on
windows or ELF libraries on Linux.. note that executable and DLL

modules on

Post by Sean Conner

Post by Philippe Verdy
Windows, as well as on OS/2, are based on directly on the ELF format) !

Citation needed.

I gave citations as examples, because there's nowhere any other counter
example using shared/prelinked libraries (or archive formats containing an
explicit manifest of the expected link order) any example I can find where
this is not true.
So instead I ask you to demonstrate the existence of any counter-example !
The C standard itself does not really indicate how we will link units into
a working program: it does not even need the existence of "libraries", it
just speaks the possibility of creating programs using collections of units
compilable separately (that's thbe meaning of the "extern" keyword in C),
and then it needs to use an external linker and to specify the units you
need for a working program in a well defined resolution order (and basic
libaries which are only collections of units in unspecified order are not
compatible with this goal).
But shared libaries/ prelinked libraries are compatible with this goal as
their role is to prelink them partially, using also the same linker and the
same specified order, which then gives a predictable resolution order for
symbols in the library (something impossible to reach using only basic
static libraries containining arbitrary number of object units in random
order).
It's not a basic "librarian" tool that resolves symbols (the librarian is
just a way to archive and pack multiple files into a single one, because it
is generally faster to process than large collections of small files, from
which they can be extracted without modification; the speed of processing
was certinaly true with past filesystems, but it's no longer the case with
modern filesystems). Only a true linker (actually made for building
programs) does that work of resolving symbols accurately and predictably!

[Sean Conner]

Post by Philippe Verdy

Post by Sean Conner
% uname
Linux
% cc -c -o main.o main.c
% cc -c -o myfunc2.o myfunc2.c
% cc -shared -fPIC -o func.ss func.c
% ar rv libfuncall.so func.ss
% ar: creating libfuncall.so
% a - func.ss
% cc -Wl,-rpath,/tmp/foo -o smain2 main.o myfunc2.o libfuncall.so
% ./smain2
Hello from main
Hello from func1
Hello from myfunc2
Back to func1
Back to main
Hello from myfunc2
Back to main
Happy now?

That's what I wanted. And demonstrates what I wanted to show: this is the
only portable and expected behavior !

I am confused.

Here's the Windows example. This has func1() and func2() in separate C
files that are compiled, and both files are used to create the shared

Post by Philippe Verdy
cl /c main.c
cl /c func1.c
cl /c func2.c
cl /c myfunc2.c
link /dll /out:mylib.dll func1.obj func2.obj

Creating library mylib.lib and object mylib.exp

Post by Philippe Verdy
link /out:main.exe main.obj myfunc2.obj mylib.lib
main

Hello from main
Hello from func1
Hello from func2 ********
Back to func1
Back to main
Hello from myfunc2
Back to main

Look closely at the line marked with '********' (which I just added by hand,
to the output). Notice the function that was called---func2(). NOT
myfunc2(). func2().

Here's the same, but under Linux:

% cc -c -o main.o main.c
% cc -c -o myfunc2.o myfunc2.c
% cc -shared -fPIC -o func1.ss func1.c
% cc -shared -fPIC -o func2.ss func2.c
% cc -shared -o libfunc.so func1.ss func2.ss
% cc -Wl,-rpath,/tmp/foo -o smain4 main.o myfunc2.o libfunc.so
% ./smain4
Hello from main
Hello from func1
Hello from myfunc2 ********
Back to func1
Back to main
Hello from myfunc2
Back to main

Again, look closely at the line marked with '********'. Notice the function
that was called---myfunc2(). NOT func2(). myfunc2().

Do you notice the difference between the two?

BOTH examples used shared libraries. So what is the "portable and expected"
answer? What should the output be for shared libraries?

-spc (Before this, I would have said the Linux version, but I was not familiar
with Windows, so I was wrong in that regard)

[Philippe Verdy]

Post by Sean Conner

Post by Philippe Verdy

Post by Sean Conner
% uname
Linux
% cc -c -o main.o main.c
% cc -c -o myfunc2.o myfunc2.c
% cc -shared -fPIC -o func.ss func.c
% ar rv libfuncall.so func.ss
% ar: creating libfuncall.so
% a - func.ss
% cc -Wl,-rpath,/tmp/foo -o smain2 main.o myfunc2.o

libfuncall.so

Post by Philippe Verdy

Post by Sean Conner
% ./smain2
Hello from main
Hello from func1
Hello from myfunc2
Back to func1
Back to main
Hello from myfunc2
Back to main
Happy now?

That's what I wanted. And demonstrates what I wanted to show: this is the
only portable and expected behavior !

I am confused.
Here's the Windows example. This has func1() and func2() in separate C
files that are compiled, and both files are used to create the shared

Post by Philippe Verdy
cl /c main.c
cl /c func1.c
cl /c func2.c
cl /c myfunc2.c
link /dll /out:mylib.dll func1.obj func2.obj

Creating library mylib.lib and object mylib.exp

Post by Philippe Verdy
link /out:main.exe main.obj myfunc2.obj mylib.lib
main

Hello from main
Hello from func1
Hello from func2 ********
Back to func1
Back to main
Hello from myfunc2
Back to main
Look closely at the line marked with '********' (which I just added by hand,
to the output). Notice the function that was called---func2(). NOT
myfunc2(). func2().
% cc -c -o main.o main.c
% cc -c -o myfunc2.o myfunc2.c
% cc -shared -fPIC -o func1.ss func1.c
% cc -shared -fPIC -o func2.ss func2.c
% cc -shared -o libfunc.so func1.ss func2.ss
% cc -Wl,-rpath,/tmp/foo -o smain4 main.o myfunc2.o libfunc.so
% ./smain4
Hello from main
Hello from func1
Hello from myfunc2 ********
Back to func1
Back to main
Hello from myfunc2
Back to main
Again, look closely at the line marked with '********'. Notice the function
that was called---myfunc2(). NOT func2(). myfunc2().

You've used the PIC option which means that the shared library does not
call directly the functions, but passes through a "call gate" to perform
the indirection.
Call gates ARE overridable when you link a DLL with other units specified
before it, because these call gates are NOT part of the linked module but
are part of a table indexed by the exported functions.
This is the difference with direct calls that, once linked, are not
overriden as the target is resolved internally in the code sections.

PIC code used via call gates are in fact NOT resolved internally by these
shared library formats. So they can be overriden. Still, the order of
resolution is predicatable (this is not like with classic libraries): the
"pre-linked" shared library in that cases is warrantied to resolve all
internal calls to use the same call gate. But function calls via call gates
suffer from an additiona indirection (which causes runtime performance
penalties). However they have the advantage that the PIC code does not need
to be "patched" in paged memory so this PIC code can be in shared read-only
pages; only the memory segment containining the call gates is private per
process: using PIC code is then only interesting if a shared library is
likely to be used by LOT of concurrent processes (e.g. for the standard C
library); there's no advantage when shared libaries are used by very few
processes (the total number of threads using it does not matter as all
threads in the same process shared the same virtual memory, except their
TLS data and interrupt stack; the normal stack is usually allocated in the
shared heap unless the program uses TLS allocation to create new threads).

[Sean Conner]

Post by Philippe Verdy

Post by Sean Conner
% cc -c -o main.o main.c
% cc -c -o myfunc2.o myfunc2.c
% cc -shared -fPIC -o func1.ss func1.c
% cc -shared -fPIC -o func2.ss func2.c
% cc -shared -o libfunc.so func1.ss func2.ss
% cc -Wl,-rpath,/tmp/foo -o smain4 main.o myfunc2.o libfunc.so
% ./smain4
Hello from main
Hello from func1
Hello from myfunc2 ********
Back to func1
Back to main
Hello from myfunc2
Back to main
Again, look closely at the line marked with '********'. Notice the function
that was called---myfunc2(). NOT func2(). myfunc2().

You've used the PIC option which means that the shared library does not
call directly the functions, but passes through a "call gate" to perform
the indirection.

% cc -c -o main.o main.c
% cc -c -o myfunc2.o myfunc2.c
% cc -shared -o func1.ss func1.c *** < NOTE LACK OF -fPIC
% cc -shared -o func2.ss func2.c *** < NOTE LACK OF -fPIC
% cc -shared -o libfunc.so func1.ss func2.ss
% cc -Wl,-rpath,/tmp/foo -o smain4 main.o myfunc2.o libfunc.so
% ./smain4
Hello from main
Hello from func1
Hello from myfunc2 *** < NOTE myfunc2 called!
Back to func1
Back to main
Hello from myfunc2
Back to main

And to further head off that I used "-shared" when compiling func1.c and
func2.c:

% cc -c -o main.o main.c
% cc -c -o myfunc2.o myfunc2.c
% cc -c -o func1.ss func1.c *** < NOTE LACK OF -shared -fPIC
% cc -c -o func2.ss func2.c *** < NOTE LACK OF -shared -fPIC
% cc -shared -o libfunc.so func1.ss func2.ss
% cc -Wl,-rpath,/tmp/foo -o smain4 main.o myfunc2.o libfunc.so
% ./smain4
Hello from main
Hello from func1
Hello from myfunc2 *** < NOTE myfunc2 called!
Back to func1
Back to main
Hello from myfunc2
Back to main
% ldd ./smain4
libfunc.so => /tmp/foo/libfunc.so (0x00a9c000) *** < NOTE this is loading it at runtime!
libc.so.6 => /lib/tls/libc.so.6 (0x00b90000)
/lib/ld-linux.so.2 (0x00b76000)

And to head off that this is Linux specific, I'll do this for Solaris as
well:

% cc -c -o main.o main.c
% cc -c -o myfunc2.o myfunc2.c
% cc -c -o func1.ss func1.c *** < NOTE LACK OF -shared -xcode=pic32
% cc -c -o func2.ss func2.c *** < NOTE LACK OF -shared -xcode=pic32
% cc -shared -o libfunc.so func1.ss func2.ss
% cc -Wl,-R/lusr/home/spc/foo -o smain4 main.o myfunc2.o -L/lusr/home/spc/foo -lfunc
% ldd smain4
libfunc.so => ./libfunc.so
libc.so.1 => /usr/lib/libc.so.1
libm.so.2 => /usr/lib/libm.so.2
/platform/SUNW,Sun-Fire-T1000/lib/libc_psr.so.1
% ./smain4
Hello from main
Hello from func1
Hello from myfunc2 *** < NOTE myfunc2 called!
Back to func1
Back to main
Hello from myfunc2
Back to main

[ rest of possibly wrong post snipped ]

-spc (I guess you coudn't bother to at least try it before writing ... )

[Andrew Gierth]

Compare these cases (this is freebsd, clang, GNU ld):

% cc -c -fPIC func.c func1.c func2.c main.c myfunc2.c
% cc -shared -o libfunc.so func1.o func2.o
% cc -Wl,-rpath,$PWD -o smain main.o myfunc2.o libfunc.so
% ./smain
Hello from main
Hello from func1
Hello from myfunc2
Back to func1
Back to main
Hello from myfunc2
Back to main

So far so good; we overrode the library version of the function.

Alternative #1:

% cc -shared -Wl,-Bsymbolic -o libfunc.so func1.o func2.o
% ./smain
Hello from main
Hello from func1
Hello from func2
Back to func1
Back to main
Hello from myfunc2
Back to main

Now we have two func2's, one visible to the library and one to the main
program.

Or we could do:

% printf '{ global: func1; local: func2; };\n' >libfunc.x
% cc -shared -Wl,--version-script,libfunc.x -o libfunc.so func1.o
func2.o
% ./smain
Hello from main
Hello from func1
Hello from func2
Back to func1
Back to main
Hello from myfunc2
Back to main

Again, two versions.

Let's do it without a version script (NOTE: this is what lua does on
platforms that support it, see LUAI_FUNC)

% printf '/void func2/s/^/__attribute__((visibility("hidden"))) /\nwq\n' | ed func2.c
% cc -c -fPIC func2.c
% cc -shared -o libfunc.so func1.o func2.o
% ./smain
Hello from main
Hello from func1
Hello from func2
Back to func1
Back to main
Hello from myfunc2
Back to main

Two versions again.

The PostgreSQL project went through a more involved version of this mess
recently, to the extent that we now have a source file that exists for
no reason other than to detect mis-linking. (Our case is more complex
because it's to do with dynamic loading, but similar principles apply.)

[Sean Conner]

Let me try this one more time ...

Post by Philippe Verdy
note that executable and DLL modules on
Windows, as well as on OS/2, are based on directly on the ELF format) !

I would like for you to provide a citation that states the Windows
exectuable format, known as PE (for "Portable Executable") is based directly
on ELF (for "Executable/Linker Format"). They are two different formats.

-spc

[Philippe Verdy]

"Based on" does not mean they are the same... There are other formats,
working on the same principles (for example COFF)...
And no! The PE format is not used directly when building, it's only one of
the final formats supported by the native loader/linker.
Note that the Windows kernel (as well as several Unix/Linux kernels) also
supports other formats (PE is not the exclusive one for Windows), and so it
implements several native loaders.
I just took one example.

Post by Sean Conner
Let me try this one more time ...

Post by Philippe Verdy
note that executable and DLL modules on
Windows, as well as on OS/2, are based on directly on the ELF format) !

I would like for you to provide a citation that states the Windows
exectuable format, known as PE (for "Portable Executable") is based directly
on ELF (for "Executable/Linker Format"). They are two different formats.
-spc

[Philippe Verdy]

So you just demonstrater that you just want to polemicate, you don't
provide any useful help with your very shortsighted view when the
discussion was much more generic...
We were giving some examples, now you interpret the examples as being the
only available options (which they are not, programmers will always invent
new alternatives).

Post by Philippe Verdy
"Based on" does not mean they are the same... There are other formats,
working on the same principles (for example COFF)...
And no! The PE format is not used directly when building, it's only one of
the final formats supported by the native loader/linker.
Note that the Windows kernel (as well as several Unix/Linux kernels) also
supports other formats (PE is not the exclusive one for Windows), and so it
implements several native loaders.
I just took one example.

Post by Sean Conner
Let me try this one more time ...

Post by Philippe Verdy
note that executable and DLL modules on
Windows, as well as on OS/2, are based on directly on the ELF format) !

I would like for you to provide a citation that states the Windows
exectuable format, known as PE (for "Portable Executable") is based directly
on ELF (for "Executable/Linker Format"). They are two different formats.
-spc

[Sean Conner]

Post by Philippe Verdy
So you just demonstrater that you just want to polemicate,

You said, and I'm quoting you here, "DLL modules on Windows, as well as on
OS/2, are based on directly on the ELF format," which runs counter to how I
understand computing history.

Microsoft first created DLLs for Windows, starting with version 1.0
(release in 1985). OS/2 (released in 1987) inherited the concept from
Windows (OS/2 being a joint project between IBM and Microsoft at the time).

System V, Release 4 (released in 1988), a combined project between AT&T
and Sun Microsystems, was when the ELF format was introduced. At best, one
could say that ELF was inspired by Windows DLLs, but what you said is ...
not based upon facts. That in turn, makes me discount what you say due to
the inaccuracy.

Post by Philippe Verdy
you don't
provide any useful help with your very shortsighted view when the
discussion was much more generic...
We were giving some examples, now you interpret the examples as being the
only available options (which they are not, programmers will always invent
new alternatives).

I initally four examples on two platforms, Linux and Solaris. Then Ivan
provided some of the examples on Windows. The results were:

mail() calls func1() calls func2()

func1(),func2() in same translation unit, in a library, providing myfunc2(),
static compilation:
Linux: failed to link
Solaris: failed to link
Windows: EXAMPLE NOT PROVIDED

func1(),func2() in different translations units, in a library, providing
myfunc2(), static compilation:
Linux: linked, run, myfunc2() called.
Solaris: linked, run, myfunc2() called.
Windows: lihked, run, myfunc2() called.

func1(),func2() in same translation unit, as a shared library, providing
myfunc2():
Linux: linked, run, myfunc2() called.
Solaris: linked, run, myfunc2() called.
Windows: EXAMPLE NOT PROVIDED

func1(),func2() in different translations units, as a shared library,
providing myfunc2():
Linux: linked, run, myfunc2() called.
Solaris: linked, run, myfunc2() called.
Windows: linked, run, func1() called (NOTE DIFFERENCE!)

So, what damn examples did YOU give that I interpreted as being the only
available options? Because I've seen very little in the way of actual
examples given, or citations to references, or anything from you.

-spc

[Philippe Verdy]

Post by Sean Conner

Post by Philippe Verdy
So you just demonstrater that you just want to polemicate,

You said, and I'm quoting you here, "DLL modules on Windows, as well as on
OS/2, are based on directly on the ELF format," which runs counter to how I
understand computing history.
Microsoft first created DLLs for Windows, starting with version 1.0
(release in 1985). OS/2 (released in 1987) inherited the concept from
Windows (OS/2 being a joint project between IBM and Microsoft at the time).
System V, Release 4 (released in 1988), a combined project between AT&T
and Sun Microsystems, was when the ELF format was introduced. At best, one
could say that ELF was inspired by Windows DLLs, but what you said is ...
not based upon facts. That in turn, makes me discount what you say due to
the inaccuracy.

Post by Philippe Verdy
you don't
provide any useful help with your very shortsighted view when the
discussion was much more generic...
We were giving some examples, now you interpret the examples as being the
only available options (which they are not, programmers will always

invent

Post by Philippe Verdy
new alternatives).

I initally four examples on two platforms, Linux and Solaris. Then Ivan
mail() calls func1() calls func2()
func1(),func2() in same translation unit, in a library, providing myfunc2(),
Linux: failed to link
Solaris: failed to link
Windows: EXAMPLE NOT PROVIDED
func1(),func2() in different translations units, in a library, providing
Linux: linked, run, myfunc2() called.
Solaris: linked, run, myfunc2() called.
Windows: lihked, run, myfunc2() called.
func1(),func2() in same translation unit, as a shared library, providing
Linux: linked, run, myfunc2() called.
Solaris: linked, run, myfunc2() called.
Windows: EXAMPLE NOT PROVIDED
func1(),func2() in different translations units, as a shared library,
Linux: linked, run, myfunc2() called.
Solaris: linked, run, myfunc2() called.
Windows: linked, run, func1() called (NOTE DIFFERENCE!)
So, what damn examples did YOU give that I interpreted as being the only
available options? Because I've seen very little in the way of actual
examples given, or citations to references, or anything from you.

Hmmmm... your examples are visibly fabriquek in your head. The description
you make here even contain errors (results that are impossible to get).
There's no source code given explicitly. The link options are not clearly
given... All this is highly suspect (notably the last "example" shown
supposed for Windows; otheres are not even tested in Windows, those for
Linux/Solaris are not even correctly described).

And as well you don't seem to understand call gates and you seem to
generalize the use of PIC code for everything which is not the default
option (and in fact not the best one except for specific cases, like the
CRT library that is extremely unlikely to be replaced safely unit by unit
as its internal dependencies are very complex). But even in this case, call
gates and PIC are unnecessary (except if you want to rebase the CRTDLL for
"ASLR", something that is actually now proven to not offer real protection
against external attacks by injection of code via buffer overflows on the
stack causing return addresses to known RVA addresses instead of causing
segmentation faults: this is the only thing that ASLR protects, but at the
price of high paging because no DLL code is then actually shared across
processes).

Call gates are not necessary to perform system calls (they are implemnted
by software interrupts instead, without passing through virtual memory
addresses, and implemented in different protection rings and the system
itself is generally not pages out, or uses a different paging pool,
independant of pageing pools used by user processes).

ELF and COFF have a common history, both have been used under Unix and even
Linux; PE is also used in Linux ! Windows can supports ELF (MZ+PE is not
the only option), ir also supports MZ+NE, MZ+LE and some variants still
used for VXD drivers. COFF has many variants. ELF itself has its own
variants (some extensions tried in Linux, then abandonned in Linux...).
Anyway there's no funcamental differences between today's variants of ELF
and COFF (the majhor difference is in how they support debugging
information and Runtime type info; and ELF was created to support more
architectures, and this was done as well in COFF then NE, LE, PE, PE64)
Various user programs or libaries can still use or implement their own
loaders for supporting the binary format they prefer, or just to emulate
other platforms.

I started with just one example but you derived on this topic which was
compeltely independant of what C compilers do, because you still maintain
the confusion between the C standard and what linkers must do which is
absolutely not defined by the C standard itself.

The C standard only defines the "linkage" options that you can use to bind
non local symbols (i.e. variables not declared as auto or register, and not
on the stack) it describes the "static" linkage (which is inviaible to
external linkers and creates a scope local to the compiled unit), and the
"extern" linkage possibly decarated by a "C" or "C++" label to indicate the
name mangling convention that allows distinguishing or not the symbols used
from other compiled units.

[Sean Conner]

Post by Philippe Verdy
Hmmmm... your examples are visibly fabriquek in your head. The description
you make here even contain errors (results that are impossible to get).
There's no source code given explicitly. The link options are not clearly
given... All this is highly suspect (notably the last "example" shown
supposed for Windows; otheres are not even tested in Windows, those for
Linux/Solaris are not even correctly described).

Fine. Source code:

/* main.c */
#include <stdio.h>

extern void func1(void);

int main(void)
{
puts("Hello from main");
func1();
puts("Back to main");
func2();
puts("Back to main");
return 0;
}

/* myfunc2.c */
#include <stdio.h>

void func2(void)
{
puts("\t\tHello from myfunc2");
}

/* func.c */
#include <stdio.h>

void func2(void)
{
puts("\t\tHello from func2");
}

void func1(void)
{
puts("\tHello from func1");
func2();
puts("\tBack to func1");
}

/* func1.c */
#include <stdio.h>

extern void func2(void);

void func1(void)
{
puts("\tHello from func1");
func2();
puts("\tBack to func1");
}

/* func2.c */
#include <stdio.h>

void func2(void)
{
puts("\t\tHello from func2");
}

I've given the commands used. RUN THE DAMN PROGRAMS AND SEE FOR YOURSELF!
I'll believe my eyes over your mad ramblings. The results I got for Linux
and Solaris I RAN MYSELF! I do not have Windows, so I am trusting the
results I got from Ivan.

-spc

[Roberto Ierusalimschy]

[...]

As this discussion seems quite private between a very few members of
this list, and has almost no relationship to Lua, would you all mind
moving it to some other forum?

Many thanks,

-- Roberto

[Philipp Janda]

Post by Sean Conner
[...]
8 All external object and function references are resolved. Library
components are linked to satisfy external references to functions
and objects not defined in the current translation. All such
translator output is collected into a program image which contains
information needed for execution in its execution environment.
I interpret this to mean, "if the given object files do not include an
object or function with external linkage, then said objects or functions can
be pulled from a "library".

Which library if there were multiple ones containing the same external
definition?

I interpret it differently. I think "translation" still means the
translation of a single translation unit (i.e. a single object file) as
in all other translation phases before. So you resolve as many undefined
external references in the object file as you can using libraries, and
the remaining ones are resolved from other compiled translation units
when "all such translator output is collected into a program image".

Anyway, the effect is exactly the same as with your interpretation as
long as "In the set of translation units and libraries that constitutes
an entire program, each declaration of a particular identifier with
external linkage denotes the same object or function" (6.2.2) and
"somewhere in the entire program there shall be exactly one external
definition for the identifier; otherwise, there shall be no more than
one" (6.9.5).


Btw., GNU ld is in violation of your interpretation of the C standard as

Post by Sean Conner

Post by Philipp Janda
$ gcc -o main main.c a.o -L. -lb
b.c:(.text+0x0): multiple definition of `a'
a.o:a.c:(.text+0x0): first defined here
collect2: error: ld returned 1 exit status

Function `a()` shouldn't have been pulled from `libb.a` because `a.o`
already contains an external definition for that identifier.

GNU ld doesn't pull individual functions or objects from the library. It
just filters out those object files in the library that it thinks it
doesn't need.

Post by Sean Conner
[...]
-spc (If at this point you don't think this interpretation holds, I'd like to
hear the argument against it)

Philipp

[99]: http://lua-users.org/lists/lua-l/2018-11/msg00258.html

[Sean Conner]

Post by Philipp Janda

Post by Sean Conner
[...]
8 All external object and function references are resolved. Library
components are linked to satisfy external references to functions
and objects not defined in the current translation. All such
translator output is collected into a program image which contains
information needed for execution in its execution environment.
I interpret this to mean, "if the given object files do not include an
object or function with external linkage, then said objects or functions
can be pulled from a "library".

Which library if there were multiple ones containing the same external
definition?

I don't think that's covered in the C Standard. I know that the C tool
chains I've used over the years have all defaulted to "seach each library,
in order specified on the command line" which seems a reasonable thing to
do.

Post by Philipp Janda
I interpret it differently. I think "translation" still means the
translation of a single translation unit (i.e. a single object file) as in
all other translation phases before. So you resolve as many undefined
external references in the object file as you can using libraries, and the
remaining ones are resolved from other compiled translation units when
"all such translator output is collected into a program image".

Okay, given this example:

% cc -o program main.o other.o -lsomelibrary

So, are you saying that during the linking phase, main.o is scanned first,
and any unresolved external references are scanned for in somelibrary to be
resolved, and then we move on to other.o and repeat the process?

Huh ... I suppose that is a valid interpretation as well, even if I
haven't encountered a C compiler that does that (at least eight if I
recalled them all correctly).

Post by Philipp Janda
Btw., GNU ld is in violation of your interpretation of the C standard as

In my other email I covered this with both static and dynamic libraries.
It was only in the case of a single translation unit containing all
functions where the link failed, which I think you would take as support for
your interpretation. But the link succeeded when each function was in its
own translation unit in the library, which supports my interpretation (for
the two different tool chains I used).

So who is correct here?

Post by Philipp Janda
Function `a()` shouldn't have been pulled from `libb.a` because `a.o`
already contains an external definition for that identifier.
GNU ld doesn't pull individual functions or objects from the library. It
just filters out those object files in the library that it thinks it
doesn't need.

I think most linkers work this way.

-spc

Post by Philipp Janda
[99]: http://lua-users.org/lists/lua-l/2018-11/msg00258.html

[Philipp Janda]

Post by Sean Conner

Post by Philipp Janda

Post by Sean Conner
[...]
8 All external object and function references are resolved. Library
components are linked to satisfy external references to functions
and objects not defined in the current translation. All such
translator output is collected into a program image which contains
information needed for execution in its execution environment.
I interpret this to mean, "if the given object files do not include an
object or function with external linkage, then said objects or functions
can be pulled from a "library".

Which library if there were multiple ones containing the same external
definition?

I don't think that's covered in the C Standard.

Well, the C standard does (un-)define it for an entire program (the set
of translation units and libraries). You are the one who wants to have
certain parts of those libraries excluded from the entire program based
on a linking order.

Post by Sean Conner
I know that the C tool
chains I've used over the years have all defaulted to "seach each library,
in order specified on the command line" which seems a reasonable thing to
do.

This order is actually required by POSIX[88], so most UNIX compilers
will behave this way. One exception is apparently clang[89]. But,
AFAICS, unfortunately even POSIX does not define whether multiple
definitions are allowed or not.

Post by Sean Conner
[...]

Post by Philipp Janda
-spc

Philipp


[88]:
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/c99.html#tag_20_11_04
[89]: https://lld.llvm.org/NewLLD.html#key-concepts

[Sean Conner]

Post by Philipp Janda
Well, the C standard does (un-)define it for an entire program (the set
of translation units and libraries). You are the one who wants to have
certain parts of those libraries excluded from the entire program based
on a linking order.

I'm not alone in this, Dirk Laurie (who started this thread) mentioned a
techique originally given by Luiz Henrique de Figueiredo (one of the
maintainers of Lua). So I feel like I'm in good company here.

Post by Philipp Janda

Post by Sean Conner
I know that the C tool
chains I've used over the years have all defaulted to "seach each library,
in order specified on the command line" which seems a reasonable thing to
do.

This order is actually required by POSIX[88], so most UNIX compilers
will behave this way. One exception is apparently clang[89]. But,
AFAICS, unfortunately even POSIX does not define whether multiple
definitions are allowed or not.
[88]: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/c99.html#tag_20_11_04
[89]: https://lld.llvm.org/NewLLD.html#key-concepts

Thank you for the references.

-spc (Who will readily admit to writing C code assuming it will run on
byte-oriented, 2's complement machines ... )

[Dibyendu Majumdar]

Post by Dirk Laurie
Over the years, there have been several replies to suggested patches
on this list, saying "no need to patch Lua".
1. In the Lua source directory, save your modified files with another
prefix, say "my", instead ol "l" starting their names, e.g.
'myctype.c' and 'mylex.c'.
rm my*.o # necessary because the Makefile does not know about them
make linux -e "LUA_O= lua.o myctype.o mylex.o"

Hi I think this is a bad idea; this is okay for quick hacks but not
the best way to build production software. I think the issues have
been discussed in detail so I won't go into that. But here is an
interesting take on how to build software:

Jonathan Blow on Libraries for his new games programming language:

Regards
Dibyendu