Discussion:
shared memory access from within a Lua script..
Gene Buckle
2004-03-11 19:09:06 UTC
Permalink
I am _very_ new to Lua (I'd never heard of it before today) and I find
myself needing to write a Lua script for a project I'm working on.

I need to be able to move a fairly large amount of data from within a Lua
script to the "outside" world where I can deal with it using another
program. The only way I know to do this is by constructing a shared
memory segment.

Has anyone ever done this before? If so, could someone point me to an
example?

Thanks!

g.
--
Proud Owner of 80-0007
http://www.f15sim.com - The only one of its kind.
Luiz Henrique de Figueiredo
2004-03-11 19:07:08 UTC
Permalink
Post by Gene Buckle
I am _very_ new to Lua (I'd never heard of it before today) and I find
myself needing to write a Lua script for a project I'm working on.
Welcome!
Post by Gene Buckle
I need to be able to move a fairly large amount of data from within a Lua
script to the "outside" world where I can deal with it using another
program. The only way I know to do this is by constructing a shared
memory segment.
If I understand you correctly, all you need to do is to export a C function
to Lua that writes to the shared memory segment. You'll need to work in C
to setup this shared memory segment. The C function exported to Lua would
get a Lua string and an offset into the shared memory segment and simply
copy the Lua string to that position. Note that Lua string can hold arbitrary
binary data.

--lhf
Gene Buckle
2004-03-11 20:32:15 UTC
Permalink
Post by Luiz Henrique de Figueiredo
Post by Gene Buckle
I am _very_ new to Lua (I'd never heard of it before today) and I find
myself needing to write a Lua script for a project I'm working on.
Welcome!
Thanks!
Post by Luiz Henrique de Figueiredo
Post by Gene Buckle
I need to be able to move a fairly large amount of data from within a Lua
script to the "outside" world where I can deal with it using another
program. The only way I know to do this is by constructing a shared
memory segment.
If I understand you correctly, all you need to do is to export a C function
to Lua that writes to the shared memory segment. You'll need to work in C
to setup this shared memory segment. The C function exported to Lua would
get a Lua string and an offset into the shared memory segment and simply
copy the Lua string to that position. Note that Lua string can hold arbitrary
binary data.
Yep, this pretty much sums up what I need to do. Unfortunately, I suck at
C. I have a fairly good handle on what code on the C end I need to call
to support the shared memory segment (CreateFileMapping() ,etc), but I
have zero clue as to how to accomplish this. I don't even know if it's
possiblee to call the Win32 API from winthin a library that's not linked
to a "main()" of some kind. This is one of those things where I basically
need a lot of hand holding. :(

I'd like to be able to use sockets as an either/or option, but since it's
not built into Lua by default, I can't guarantee that it'll be available
in the product I have to interface to.

Am I correct in assuming that it's possible to cast the Lua string into a
struct on the side? Would I have to use a table for this instead?

Thanks!

g.
Brett Kapilik
2004-03-11 19:07:06 UTC
Permalink
I changed lmem.c to look like this:

------------- code block start ----------------

/*
** $Id: lmem.c,v 1.61 2002/12/04 17:38:31 roberto Exp $
** Interface to Memory Manager
** See Copyright Notice in lua.h
*/


#include <stdlib.h>

#define lmem_c

#include "lua.h"

#include "ldebug.h"
#include "ldo.h"
#include "lmem.h"
#include "lobject.h"
#include "lstate.h"
#include "Windows.h"


/*
** definition for realloc function. It must assure that l_realloc(NULL,
** 0, x) allocates a new block (ANSI C assures that). (`os' is the old
** block size; some allocators may use that.)
*/
#ifndef l_realloc
//#define l_realloc(b,os,s) realloc(b,s)
#endif

/*
** definition for free function. (`os' is the old block size; some
** allocators may use that.)
*/
#ifndef l_free
#define l_free(b,os) HeapFree(GetProcessHeap(),0,b);
//#define l_free(b,os) free(b)
#endif


#define MINSIZEARRAY 4


void *luaM_growaux (lua_State *L, void *block, int *size, int size_elems,
int limit, const char *errormsg) {
void *newblock;
int newsize = (*size)*2;
if (newsize < MINSIZEARRAY)
newsize = MINSIZEARRAY; /* minimum size */
else if (*size >= limit/2) { /* cannot double it? */
if (*size < limit - MINSIZEARRAY) /* try something smaller... */
newsize = limit; /* still have at least MINSIZEARRAY free places */
else luaG_runerror(L, errormsg);
}
newblock = luaM_realloc(L, block,
cast(lu_mem, *size)*cast(lu_mem, size_elems),
cast(lu_mem, newsize)*cast(lu_mem, size_elems));
*size = newsize; /* update only when everything else is OK */
return newblock;
}


/*
** generic allocation routine.
*/
void *luaM_realloc (lua_State *L, void *block, lu_mem oldsize, lu_mem size) {
lua_assert((oldsize == 0) == (block == NULL));
if (size == 0) {
if (block != NULL) {
l_free(block, oldsize);
block = NULL;
}
else return NULL; /* avoid `nblocks' computations when oldsize==size==0 */
}
else if (size >= MAX_SIZET)
luaG_runerror(L, "memory allocation error: block too big");
else {
//block = l_realloc(block, oldsize, size);
if(block == NULL)
{
block = HeapAlloc(GetProcessHeap(),0,size);
} else
{
block = HeapReAlloc(GetProcessHeap(),0,block,size);
}
if (block == NULL) {
if (L)
luaD_throw(L, LUA_ERRMEM);
else return NULL; /* error before creating state! */
}
}
if (L) {
lua_assert(G(L) != NULL && G(L)->nblocks > 0);
G(L)->nblocks -= oldsize;
G(L)->nblocks += size;
}
return block;
}

------------- code block start ----------------

With those changes it works great with shared memory in Windows when linking Lua statically. I have been using this is a commercial product for a while now with no problems. If you do this, other Lua DLL modules may not work properly but if working with a closed system it works like a charm. You can link Lua statically in your app and in DLLs and they will not have memory issues this way.

- Brett
-----Original Message-----
Sent: Thursday, March 11, 2004 1:09 PM
Subject: shared memory access from within a Lua script..
I am _very_ new to Lua (I'd never heard of it before today) and I find
myself needing to write a Lua script for a project I'm working on.
I need to be able to move a fairly large amount of data from
within a Lua
script to the "outside" world where I can deal with it using another
program. The only way I know to do this is by constructing a shared
memory segment.
Has anyone ever done this before? If so, could someone point me to an
example?
Thanks!
g.
--
Proud Owner of 80-0007
http://www.f15sim.com - The only one of its kind.
Gene Buckle
2004-03-11 20:41:26 UTC
Permalink
Post by Brett Kapilik
With those changes it works great with shared memory in Windows when
linking Lua statically. I have been using this is a commercial product
for a while now with no problems. If you do this, other Lua DLL modules
may not work properly but if working with a closed system it works like
a charm. You can link Lua statically in your app and in DLLs and they
will not have memory issues this way.
- Brett
Brett, my problem is that I can't change Lua at all. It's built into a
commercial product that I don't have the sources for.

My task is this - The product I'm working with (a game) will provide some
special Lua functions for reading data from the game internals and
functions for setting states within the game engine. I have to use those
functions to interface with the real world in the simplest and most
efficient manner possible. I also have to accomplish this without
changing the Lua interpreter from what you'd normally find in a default
installation.

I spend all day writing Delphi and VB database code. I'm _not_ a C
programmer by any stretch of the word. I can read it fairly well, but I
have little experience writing it. You can understand why I find this a
daunting task. :)

g.
Virgil Smith
2004-03-11 21:51:34 UTC
Permalink
This sounds like a far more daunting task than your original post suggested.

If you have to....

1. Write scripts to interact with an existing (black box) program that is
using some unknown method of linking to / C-side controlling a Lua state.
2. Through these scripts you must interface to a second program/process.

As a first blind stab here, I have to ask....

Does the program with the Lua interface provide access to the Lua io
library?
If so then you can make a quick solution (albeit not necessarily a terribly
efficient one) by writing the data out to files and reading from them in the
other program.

If not then you are indeed going to face some serious difficulties.

If the "engine" you are connecting to allows Lua scripts to use loadlib to
load external C libraries then you can begin to consider writing C-code to
work with your scripts. However, your main problem (other than the fact
that you need to go study the Lua manual for a little while) is that you are
going to have to risk using more than one instance of the Lua code (i.e. you
will have to link it in to your DLL so that you can use the C-API). These
instances may not be identical, and even if they were, they won't use the
same C-runtime data. Mainly this means that you are likely to get crashes
in calls to free due to the fact that the memory to be freed can't be found
in the heap of that particular instance of the C-runtime. This will of
coarse come up at GC time.

Someone may have a much brighter idea, but it sounds to me like you are
"doomed" to end up modifying the Lua sources to attempt to somehow work
around this issue.


In an optimistic light, it is Of course completely possible that this
"engine" offers some helpful information / hooks for this kind of extending.




-----Original Message-----
From: lua-***@bazar2.conectiva.com.br
[mailto:lua-***@bazar2.conectiva.com.br]On Behalf Of Gene Buckle
Sent: Thursday, March 11, 2004 2:41 PM
To: Lua list
Subject: RE: shared memory access from within a Lua script..
Post by Brett Kapilik
With those changes it works great with shared memory in Windows when
linking Lua statically. I have been using this is a commercial product
for a while now with no problems. If you do this, other Lua DLL modules
may not work properly but if working with a closed system it works like
a charm. You can link Lua statically in your app and in DLLs and they
will not have memory issues this way.
- Brett
Brett, my problem is that I can't change Lua at all. It's built into a
commercial product that I don't have the sources for.

My task is this - The product I'm working with (a game) will provide some
special Lua functions for reading data from the game internals and
functions for setting states within the game engine. I have to use those
functions to interface with the real world in the simplest and most
efficient manner possible. I also have to accomplish this without
changing the Lua interpreter from what you'd normally find in a default
installation.

I spend all day writing Delphi and VB database code. I'm _not_ a C
programmer by any stretch of the word. I can read it fairly well, but I
have little experience writing it. You can understand why I find this a
daunting task. :)

g.
Gene Buckle
2004-03-11 22:29:44 UTC
Permalink
Post by Virgil Smith
Does the program with the Lua interface provide access to the Lua io
library?
I would assume so.
Post by Virgil Smith
If so then you can make a quick solution (albeit not necessarily a terribly
efficient one) by writing the data out to files and reading from them in the
other program.
If I didn't need to have an update rate of at least 30Hz, this would work.
:)
Post by Virgil Smith
Someone may have a much brighter idea, but it sounds to me like you are
"doomed" to end up modifying the Lua sources to attempt to somehow work
around this issue.
In an optimistic light, it is Of course completely possible that this
"engine" offers some helpful information / hooks for this kind of extending.
I may have a chance at getting at least LuaSocket stuffed into the build
so I can use that to get data out the door. If I use the loopback
interface to communicate with another program on the same host, it should
be quick enough for my purposes. I have at most, 150ms of time to
process, transmit and act upon the data coming out of the game engine
before I need to process the next frame.

I'm getting the impression that trying for a real shared memory segment is
going to be foolhardy at best, so I'll press for the socket option.

Thanks!

g.
JK
2004-03-12 11:52:57 UTC
Permalink
Hi All,
Hi All,

I found this question with many answers regarding the language Ruby. /*
After I do bunch of computations and create a huge array with bunch of
numbers, I like to use the already-computed array in another program
rather than re-computing it each time I run the program. How can I
save the array object in one program and read it from another program
in Ruby?
*/
and I found this answer the simplest one:

You can use marshalling:

array = ...
File.open("array", "wb") do |f|
Marshal.dump(array, f)
end


To reload:

array = nil
File.open("array", "rb") do |f|
array = Marshal.load(f)
end

Now, I need the same thing in Lua. Is it possibile to do this elegantly in
just 8 lines in Lua?

Thanks!

--
jilani
Reuben Thomas
2004-03-12 12:01:59 UTC
Permalink
Post by JK
After I do bunch of computations and create a huge array with bunch of
numbers, I like to use the already-computed array in another program
rather than re-computing it each time I run the program. How can I
save the array object in one program and read it from another program
...?
Is it possibile to do this elegantly in just 8 lines in Lua?
If you're using the stdlib libraries from the lua-users stdlib project,
this is a matter of:

t = {1,2,3,4,5}
h = io.open("foo","w")
h:write(pickle(t))
h:close()

h = io.open("foo", "r")
t = eval(h:read("*a"))
h:close()

The two useful functions that you get from the libraries here are pickle
and eval. The latter is trivial:

-- @func eval: Evaluate a string
-- @param s: string
-- @returns
-- @param v: value of string
function eval (s)
return loadstring ("return " .. s)()
end

The former is rather more involved, and currently doesn't work for
recursive tables (doesn't sound as though it's a problem in your case). In
any case, I've already added a tostring that works on recursive tables, so
rewriting pickle in the same way wouldn't be a problem.
--
http://www.mupsych.org/~rrt/ | plagiarism, n. the mind burgles
Philippe Lhoste
2004-03-12 14:28:35 UTC
Permalink
Post by JK
I found this question with many answers regarding the language Ruby. /*
After I do bunch of computations and create a huge array with bunch of
numbers, I like to use the already-computed array in another program
rather than re-computing it each time I run the program. How can I
save the array object in one program and read it from another program
in Ruby?
*/
array = ...
File.open("array", "wb") do |f|
Marshal.dump(array, f)
end
array = nil
File.open("array", "rb") do |f|
array = Marshal.load(f)
end
Now, I need the same thing in Lua. Is it possibile to do this elegantly
in just 8 lines in Lua?
I can do that in 1 line only, but it will not be very readable...
A number of methods to serialize tables have been given here over the
years, none perfect because Lua can store all kind of strange data
(userdata for example) as keys or values.

Since you have very simple needs (manage a simple array of numbers
only), I can give a very simple code:

local arrayFile = "Array.txt"
local array = { 1, 5, 12.3, 0.25, -45.01, 7 }
local f = io.open(arrayFile, "w")
f:write("return {\n")
for i, v in ipairs(array) do
f:write("[" .. i .. "]=" .. v .. ",") -- Add \n if you want the file
to be readable by humans
end
f:write("}\n")
f:close()

And read back the values with:

local array = dofile(arrayFile)
--
--=#=--=#=--=#=--=#=--=#=--=#=--=#=--=#=--=#=--
Philippe Lhoste (Paris -- France)
Professional programmer and amateur artist
http://jove.prohosting.com/~philho/ (outdated)
http://philho.multimania.com (in French, for files to download)
--=#=--=#=--=#=--=#=--=#=--=#=--=#=--=#=--=#=--
Juergen Fuhrmann
2004-03-12 14:54:56 UTC
Permalink
Fri, 12 Mar 2004 15:28:35 +0100
Post by JK
I found this question with many answers regarding the language Ruby. /*
After I do bunch of computations and create a huge array with bunch of
numbers, I like to use the already-computed array in another program
rather than re-computing it each time I run the program. How can I
save the array object in one program and read it from another program
in Ruby?
*/
array = ...
File.open("array", "wb") do |f|
Marshal.dump(array, f)
end
array = nil
File.open("array", "rb") do |f|
array = Marshal.load(f)
end
Now, I need the same thing in Lua. Is it possibile to do this elegantly
in just 8 lines in Lua?
I can do that in 1 line only, but it will not be very readable...
A number of methods to serialize tables have been given here over the
years, none perfect because Lua can store all kind of strange data
(userdata for example) as keys or values.
Since you have very simple needs (manage a simple array of numbers
local arrayFile = "Array.txt"
local array = { 1, 5, 12.3, 0.25, -45.01, 7 }
local f = io.open(arrayFile, "w")
f:write("return {\n")
for i, v in ipairs(array) do
f:write("[" .. i .. "]=" .. v .. ",") -- Add \n if you want the file
to be readable by humans
end
f:write("}\n")
f:close()
local array = dofile(arrayFile)
Hi,

This becomes interesting for me:
when I read with dofile, the array data several times are in main memory:

1) when I read the chunk (this could be modified, though)
2) in the bytecode
3) in the table.
If, additionally, the arrays in fact are userdata objects in order to pass
them to C, we have one more time...

Or am I wrong ?

The solution I implemented is something like

a=CreateArray(100000)
data(a)
$
1
2
...
100000
$

where the $'s are chunk separators which might be replaces by XML tags in
a future version.

The data statement triggers the parser for the next chunk, it could
be even fread.

Another solution would just using Lua for indexing a binary file.

In the baseline, this is a marshalling solution like in ruby. Not
very elegant, but it works. Of course I would like some solution which
is more lua'ish.

Juergen
Virgil Smith
2004-03-12 16:10:08 UTC
Permalink
Post by Juergen Fuhrmann
In the baseline, this is a marshalling
solution like in ruby. Not very elegant,
but it works. Of course I would like some
solution which is more lua'ish.
Lua is very adaptable to serialization constructs. If you are very
interested then check the Wiki for "serialization", "XML", "tostring", and
probably several other key phrases dealing with particular representation
standards would be good things to search for.

Lua's table structure translates well to hiearchical data representations.
Userdata and tables with metatables present some complications that I
imagine different people have come up with fairly different approaches to
handling. Recursive references slow things down slightly and cause some
slight extra memory consumption in the serialization process, but are easy
to overcome using Lua. To overcome recursive references one just needs to
introduce an extra "type" into the stream representing references to objects
serialized earlier in the stream. Constructing these references during the
serialization process or extracting them during the deserialization process
simply requires the formation of a table of already processed objects.
Since Lua can use any object/value as a table key you don't have to write
any table searching code yourself. Just use the objects being serialized as
the keys when serializing, and the reference values as keys when
deserializing.

R***@oxfam.org.pe
2004-03-12 15:15:03 UTC
Permalink
Post by Philippe Lhoste
local arrayFile = "Array.txt"
local array = { 1, 5, 12.3, 0.25, -45.01, 7 }
local f = io.open(arrayFile, "w")
f:write("return {\n")
for i, v in ipairs(array) do
f:write("[" .. i .. "]=" .. v .. ",")
-- Add \n if you want the file to be readable by humans
end
f:write("}\n")
f:close()
even shorter, although it uses more memory:

function dumpArray(file, array)
local f = assert(io.open(file, "w"))
assert(f:write("return {", table.concat(array, ","), "}"))
assert(f:close())
end
Philippe Lhoste
2004-03-12 15:42:37 UTC
Permalink
Post by R***@oxfam.org.pe
Post by Philippe Lhoste
local arrayFile = "Array.txt"
local array = { 1, 5, 12.3, 0.25, -45.01, 7 }
local f = io.open(arrayFile, "w")
f:write("return {\n")
for i, v in ipairs(array) do
f:write("[" .. i .. "]=" .. v .. ",")
-- Add \n if you want the file to be readable by humans
end
f:write("}\n")
f:close()
function dumpArray(file, array)
local f = assert(io.open(file, "w"))
assert(f:write("return {", table.concat(array, ","), "}"))
assert(f:close())
end
I forgot that concat can use a separator...
Note that if my routine was written with pairs instead of ipairs, it
will works even if there are holes in the array...
Althought data will be no longer in increasing key values.
--
--=#=--=#=--=#=--=#=--=#=--=#=--=#=--=#=--=#=--
Philippe Lhoste (Paris -- France)
Professional programmer and amateur artist
http://jove.prohosting.com/~philho/ (outdated)
http://philho.multimania.com (in French, for files to download)
--=#=--=#=--=#=--=#=--=#=--=#=--=#=--=#=--=#=--
Loading...