Tuesday, January 18, 2011

Building LuaSocket for LuaJIT on Windows with MinGW. Oh my!

Lua is a lovely little language, somewhat minimalistic like Scheme, with a powerful and concise syntax somewhat like Ruby and Python, but it was never "batteries included", meaning that significant effort must be expended to accomplish some seemingly-trivial tasks.
Those defensive of Lua will counter that this is precisely the point of Lua - to be small and carry as few dependencies as possible. This may be true, but it's nice to offer some easily-installed extras, especially for basic tasks. It would be quite painful if everyone had to rewrite and link to C code just to get the time in milliseconds or open a socket.

Lua is very popular with the embedded and scripting crowd; especially game scripting, with WoW and a bunch of other games allowing users to write little Lua programs with some API to access game functions. Even NMap also allows Lua scripting now.

I'm interested in the use of Lua for more general programming tasks and projects, where re-using existing libraries is important to cut down on extra work. This is especially the case for Lua newbies like myself, and given the minimalism present in Lua's design and organisation, even quite trivial tasks require the use of external libraries.
One example of this is getting the current time in milliseconds - this is useful for writing simple benchmarking scripts, for example. Since Lua tries to stick almost exclusively (apart from some dynamic linking aspects, according to the canonical reference tome, PiL) to the ANSI C specification, you can't get access to time information at finer than per second resolution.

There is a networking library named LuaSocket which happens to provide such a function by calling into the standard C library:
require 'socket'
> =socket
table: 0x0026b680
> =socket.gettime
function: 0x0026ba18
> =socket.gettime()
1295358928.145

Rather than compile the library from source, which can be problematic - especially on Windows and double-especially for those not familiar with C - there are two promising software repository efforts for Lua: LuaRocks and LuaDist.

Unfortunately, neither of these managed to successfully install LuaSocket:
  • LuaDist failed to find anything at all on its repository - hopefully a temporary problem.

  • LuaRocks first failed outright due to wget not being present on my Windows system. After installing a Gnuwin port and copying it into the LuaRocks dir, some faffing about was required to have it use our network proxy, and it finally downloaded the appropriate .lua and prebuilt .dll files. The DLLs crash LuaJIT, the JITted interpiler I'm using.

Apparently the dynamic libraries provided by such extensions must be compiled against your interpreter's lua51.dll.

To get LuaRocks to use a network proxy, you must add an (undocumented) entry to the config.lua file:
proxy = "http://proxy:port"


Since the prebuilt DLLs didn't work, I tried to build luasocket from source, linking against LuaJIT's lua51.dll. For this, LuaRocks tried to use what I assume are Visual Studio commands:
...
Extracting luasocket-2.0.2\test\testclnt.lua
Extracting luasocket-2.0.2\test\testsrvr.lua
Extracting luasocket-2.0.2\test\testsupport.lua

Everything is Ok

Folders: 6
Files: 89
Size: 477216
Compressed: 552960
'msbuild' is not recognized as an internal or external command,
operable program or batch file.
cp: src/mime.dll: No such file or directory
cp: src/socket.dll: No such file or directory

Error: Build error: Failed building.


Oh well, it made a pretty decent effort. There was an option to install LuaRocks using the MinGW compiler toolchain, and to use LuaJIT as the interpreter. This crashed. So I tried to build LuaSocket from source myself, but was stymied because the provided Makefiles/solution files are for Linux or assume Visual Studio is available on Windows.

After an lengthy period of head-banging, the following Makefile (in luasocket's src dir) produced DLLs that work in LuaJIT for me (after calling mingw32-make, the resulting mime.dll and socket.dll can be moved into luajit-dir\mime\core.dll and lj-dir\socket\core.dll).

#------
# LuaSocket makefile configuration
#

#------
# Output file names
#
EXT=dll
SOCKET_V=2.0.2
MIME_V=1.0.2
SOCKET_SO=socket.$(EXT)
MIME_SO=mime.$(EXT)

CC="C:\MinGW\bin\mingw32-gcc.exe"
CINC=-I"C:\MinGW\include"

LD=$(CC)
LDFLAGS=-L "C:\MinGW\lib" -lmingw32 -lkernel32 -lcrtdll -lwsock32 -shared
CFLAGS= $(LUAINC) $(CINC) $(DEF) -pedantic -Wall -O2
#------
# Lua includes and libraries

LUAINC=-I"C:\Program Files\Lua\5.1\include"
LUADLL="C:\code\luajit2\lua51.dll"

SOCKET_OBJS:= \
luasocket.o \
timeout.o \
buffer.o \
io.o \
auxiliar.o \
options.o \
inet.o \
tcp.o \
udp.o \
except.o \
select.o \
wsocket.o

#------
# Modules belonging mime-core
#
#$(COMPAT)/compat-5.1.o \

MIME_OBJS:=\
mime.o

all: $(SOCKET_SO) $(MIME_SO)

$(SOCKET_SO): $(SOCKET_OBJS)
$(LD) -o $@ $(SOCKET_OBJS) $(LUADLL) $(LDFLAGS)

$(MIME_SO): $(MIME_OBJS)
$(LD) -o $@ $(MIME_OBJS) $(LUADLL) $(LDFLAGS)

#------
# List of dependencies
#
auxiliar.o: auxiliar.c auxiliar.h
buffer.o: buffer.c buffer.h io.h timeout.h
except.o: except.c except.h
inet.o: inet.c inet.h socket.h io.h timeout.h wsocket.h
io.o: io.c io.h timeout.h
luasocket.o: luasocket.c luasocket.h auxiliar.h except.h timeout.h \
buffer.h io.h inet.h socket.h wsocket.h tcp.h udp.h select.h
mime.o: mime.c mime.h
options.o: options.c auxiliar.h options.h socket.h io.h timeout.h \
wsocket.h inet.h
select.o: select.c socket.h io.h timeout.h wsocket.h select.h
tcp.o: tcp.c auxiliar.h socket.h io.h timeout.h wsocket.h inet.h \
options.h tcp.h buffer.h
timeout.o: timeout.c auxiliar.h timeout.h
udp.o: udp.c auxiliar.h socket.h io.h timeout.h wsocket.h inet.h \
options.h udp.h
wsocket.o: wsocket.c socket.h io.h timeout.h wsocket.h

clean: rm -f $(SOCKET_SO) $(SOCKET_OBJS)
rm -f $(MIME_SO) $(UNIX_SO) $(MIME_OBJS) $(UNIX_OBJS)
#------
# End of makefile configuration
#

I'm hopeful that LuaDist (if it's not defunct - the webpage is not encouraging) and LuaRocks will improve and become more useful on Windows systems soon. Hopefully the same process will be easier when I try it at home on my old Macbook... but argh, all of this just to get the time in milliseconds!

Monday, January 10, 2011

Minigotcha: escaping path strings for cmd.exe

There's a very handy plugin for (g)Vim called netrw, which provides the capability to edit files over ssh/scp/etc. To set it up with Putty on Windows, the suggested lines to add to $MYVIMRC were:
let g:netrw_cygwin = 0
let g:netrw_scp_cmd = "\"C:\\Program Files\\PuTTY\\pscp.exe\" -pw mypasswd "


I'm not sure why this worked for them and not me, but perhaps I'm using an older (or newer) version of netrw, or the cmd.exe behaves differently in Vista. In any case, it didn't work and gave the infuriating output of:
'C:\Program' is not recognized as an internal or external command,
operable program or batch file.

This is annoying since the path for pscp is obviously escaped. However, it turns out that cmd.exe follows a slightly strange, arbitrary protocol for escaping space pathstrings - it uses another " character (ack, ugly and confusing... why? The argument is already in quotes...):
C:\Windows\system32\cmd.exe /c "C:\Program" Files\PuTTY\pscp.exe"

PuTTY Secure Copy client
Release 0.60

Okay then! So this works:
let g:netrw_scp_cmd ="\"C:\\Program\" Files\\PuTTY\\pscp.exe\""