[LuaJIT FFI] [Windows] "bad callback" error in MsgWaitForMultipleObjects() proof-of-concept

[Duncan Cross]

I am attempting a simple loop using MsgWaitForMultipleObjects() on
Windows 7 (64-bit, but LuaJIT is compiled as 32-bit, in MSVC++08
Express.)

All I have at the moment is an empty window and a timer event object.

What's happening is that the application is giving a "bad callback"
error shortly after the application starts, especially when moving the
mouse over the window.

If I turn off JIT with jit.off(), it seems to stop happening.

Here is a minimal test case with FFI bindings included:

<http://pastebin.com/XAyU1srh>

(At the moment I don't seem to be able to reach the LuaJIT git repo,
so I can't be certain that this hasn't already been fixed, but I know
I'm running at least beta-9.)

Any help would be greatly appreciated!

-Duncan

[Mike Pall]

Post by Duncan Cross
What's happening is that the application is giving a "bad callback"
error shortly after the application starts, especially when moving the
mouse over the window.
If I turn off JIT with jit.off(), it seems to stop happening.

The problem is that a FFI callback cannot safely be called from a
C function which is itself called via the FFI from JIT-compiled
code. In your case this is the call to MsgWaitForMultipleObjects.

I've put in a lot of heuristics to detect this, and it usually
succeeds in disabling compilation for such a function. However in
your case the loop is compiled before the callback is ever called,
so the detection fails.

The straighforward solution is to put the message loop into an
extra Lua function and use jit.off(func).

--Mike

[Ignacio Burgueño]

Post by Mike Pall
The problem is that a FFI callback cannot safely be called from a
C function which is itself called via the FFI from JIT-compiled
code. In your case this is the call to MsgWaitForMultipleObjects.

Sorry Mike. I might be missing something obvious but what would be the call
sequence in this case? I can't see any callbacks in the code Duncan sent.

[Mike Pall]

Post by Ignacio Burgueño

Post by Mike Pall
The problem is that a FFI callback cannot safely be called from a
C function which is itself called via the FFI from JIT-compiled
code. In your case this is the call to MsgWaitForMultipleObjects.

Sorry Mike. I might be missing something obvious but what would be the call
sequence in this case? I can't see any callbacks in the code Duncan sent.

MsgWaitForMultipleObjects
==>
TestWindowClass
==> *lpfnWndProc
Lua function at line 142

--Mike

[Ignacio Burgueño]

Post by Duncan Cross
MsgWaitForMultipleObjects
==>
TestWindowClass
==> *lpfnWndProc
Lua function at line 142

Oh, I've totally missed that. Thanks.

Does this mean that there's no way to code a Windows application with
LuaJIT and have the code be jit-compiled? Since all the actual work will be
done in lpfnWndProc or some other function called from there.

Regards,
Ignacio

[Mike Pall]

Post by Ignacio Burgueño
Does this mean that there's no way to code a Windows application with
LuaJIT and have the code be jit-compiled? Since all the actual work will be
done in lpfnWndProc or some other function called from there.

Err, no. Read my message again. It's about the outer message loop
itself, which takes close to zero time. lpfnWndProc can and will
be compiled.

--Mike

[Ignacio Burgueño]

Post by Mike Pall

Post by Ignacio Burgueño
Does this mean that there's no way to code a Windows application with
LuaJIT and have the code be jit-compiled? Since all the actual work will

be

Post by Ignacio Burgueño
done in lpfnWndProc or some other function called from there.

Err, no. Read my message again. It's about the outer message loop
itself, which takes close to zero time. lpfnWndProc can and will
be compiled.

Ah, now I get it. I thought the whole call chain couldn't be jitted. Sorry
for the noise.

Thank you.

[Duncan Cross]

Post by Mike Pall

Post by Duncan Cross
What's happening is that the application is giving a "bad callback"
error shortly after the application starts, especially when moving the
mouse over the window.
If I turn off JIT with jit.off(), it seems to stop happening.

The problem is that a FFI callback cannot safely be called from a
C function which is itself called via the FFI from JIT-compiled
code. In your case this is the call to MsgWaitForMultipleObjects.
I've put in a lot of heuristics to detect this, and it usually
succeeds in disabling compilation for such a function. However in
your case the loop is compiled before the callback is ever called,
so the detection fails.
The straighforward solution is to put the message loop into an
extra Lua function and use jit.off(func).
--Mike

Hi Mike,

Thanks very much for this prompt explanation!

Are you certain it's MsgWaitForMultipleObjects() calling that WNDPROC
callback itself though? My reading of the documentation for that
function [1] is that it only waits until there is an event to process,
it doesn't do anything about it - hence the usual calls to
TranslateMessage/DispatchMessageA afterwards in the loop. I tried
printing debug.traceback() in that callback - I see instances of
CreateWindowExA(), ShowWindow(), UpdateWindow(), PeekMessageA() and
DispatchMessageA() on the stack, but not MsgWaitForMultipleObjects().

(I realise this may not affect the explanation/solution much, but in
case it is significant I thought I better query this.)

[1] http://msdn.microsoft.com/en-us/library/windows/desktop/ms684242(v=vs.85).aspx

-Duncan

[Mike Pall]

Post by Duncan Cross
Are you certain it's MsgWaitForMultipleObjects() calling that WNDPROC
callback itself though? My reading of the documentation for that
function [1] is that it only waits until there is an event to process,
it doesn't do anything about it - hence the usual calls to
TranslateMessage/DispatchMessageA afterwards in the loop. I tried
printing debug.traceback() in that callback - I see instances of
CreateWindowExA(), ShowWindow(), UpdateWindow(), PeekMessageA() and
DispatchMessageA() on the stack, but not MsgWaitForMultipleObjects().

Ah, yes, it's DispatchMessageA(). I haven't checked the Windows
docs for details.

Post by Duncan Cross
(I realise this may not affect the explanation/solution much, but in
case it is significant I thought I better query this.)

Well, it was clear it's one of the functions in the message loop.
Doesn't really matter which one, though.

--Mike

[Duncan Cross]

Post by Mike Pall

Post by Duncan Cross
Are you certain it's MsgWaitForMultipleObjects() calling that WNDPROC
callback itself though? My reading of the documentation for that
function [1] is that it only waits until there is an event to process,
it doesn't do anything about it - hence the usual calls to
TranslateMessage/DispatchMessageA afterwards in the loop. I tried
printing debug.traceback() in that callback - I see instances of
CreateWindowExA(), ShowWindow(), UpdateWindow(), PeekMessageA() and
DispatchMessageA() on the stack, but not MsgWaitForMultipleObjects().

Ah, yes, it's DispatchMessageA(). I haven't checked the Windows
docs for details.

Post by Duncan Cross
(I realise this may not affect the explanation/solution much, but in
case it is significant I thought I better query this.)

Well, it was clear it's one of the functions in the message loop.
Doesn't really matter which one, though.
--Mike

Great, all is clear now - thanks again.

-Duncan