Bluish Coder: javascript

Prototype Based Programming Languages

2009-07-16T13:11:00+12:00

I've been reading up on protoype based programming languages recently. Mainly using the Io Programming Language and Self but also looking at LambdaMOO and similar languages. A good overview of using the prototype based approach to building programs is Organizing Programs without Classes. This post is based on examples from that paper and from Attack of the Clones which covers design patterns using Self.

Self

In the Self programming language objects are created using a literal object syntax. This syntax defines code and slots within the (| and |) delimiters. An example object looks like:

point = (|
  parent* = traits point.
  x.
  y.
|)

This creates an object with 3 slots. The slots are 'parent', 'x' and 'y'. The 'parent' slot is assigned an inital value of 'traits point' which is the traits object for points (more on what this is later). The '*' that is suffixed to the 'parent' slot means it is a prototype slot and used in the prototype slot lookup chain.

This means when a message is sent to the 'point' object the lookup starts with the 'point' object. If a slot with the messages name is not found in that object then each prototype slot (those suffixed with '*') are searched looking for a slot with that name.

So in the 'point' case, looking for 'x' will find it immediately in the point object. Looking for 'print' will not so it will look for it in the 'traits point' object. If it's not there it will look in that objects prototype slots and so on until it is found or the search is exhausted.

The idiom in Self (and other prototype based languages) is to create global objects like these and use 'clone' to create copies of it. So creating two different points would look like:

a = point clone.
b = point clone.

Notice that 'point' has no method slots. Only the 'x' and 'y' which contain data. The methods are defined in the 'traits point' object. The definition of that could look something like:

traits point = (|
  parent* = traits clonable.
  print = x println y println.
|)

This provides a method to print the 'x' and 'y' values of the object and another parent that provides other basic functionality like the ability to clone. The 'traits point' object doesn't define any data slots. It defines only methods. However it uses 'x' and 'y' messages that aren't defined. It expects to be used in a prototype slot of another object that defines the 'x' and 'y' slots (like our 'point' example earlier.

Separating the code out into data objects and trait objects allows the trait object to be reused in other objects. For example, an object that computes the 'x' and 'y' values rather than storing them can re-use the traits object:

computed_point = (|
  parent* = traits point.
  x = ..compute x..
  y = ..compute y..
|)

This 'computed_point' can be used anywhere a point is expected. The traits object sends the 'x' and 'y' messages when it needs their value and it doesn't matter if there stored as data as in the 'point' object, or as methods that calculate the value as in the 'computed_point' object. Each 'point' and 'computed_point' object shares a single trait object instance. This avoids the need to have multiple copies of the methods in each point object instance.

The prototype slots of an object effectively form an inheritance relationship. The way to subclass objects in Self is to create a new object and set a prototype slot to an instance of the parent object. Usually it's the trait objects that map the subclass relationship since it is those objects that contain the object behaviour (ie. the methods and no data). An example follows in Self of how a 'polygon' and 'filled_polygon' object can be modelled (this is from the 'Organizing Programs without Classes' paper):

polygon_traits = (|
  draw = ...draw using 'vertices' slot to get points...
|)

polygon = (|
  parent* = polygon_traits.
  vertices.
|)

filled_polygon_traits = (|
  parent* = polygon_traits.
  draw = ...draw using 'vertices' and 'fillPattern'...
|)

filled_polygon = (|
  parent* = filled_polygon_traits.
  vertices.
  fillPattern;
|)

Cloning the 'polygon' object and sending the 'draw' message will draw the polygon using the list of points held in the 'vertices' slot. Cloning the 'filled_polygon' object and sending the 'draw' message will draw using the specialized 'draw' method that also uses the 'fillPattern' slot to fill the polygon when drawing. This can re-use the 'draw' method in the 'polygon_traits' object if needed.

The new 'filled_polygon' object did require defining a new 'vertices' slot. Self allows multiple prototype slots, each of which is involved in the lookup for slot names. We can share the 'vertices' from the 'polygon' object by making that an additional prototype slot in 'filled_polygon'. This is often termed a 'data parent':

filled_polygon = (|
  parent* = filled_polygon_traits.
  dataParent* = polygon clone.
  fillPattern;
|)

Notice the 'dataParent' slot is a prototype slot (suffixed with a '*'). This means it participates in the slot lookup process. The data parent approach has an advantage over the previous example in that if we change the representation of the 'polygon' object then all new 'filled_polygon' instances will get this new representation. We don't need to edit the 'filled_polygon' definition for the modified or additional slots.

In the 'filled_polygon' example we re-used the 'vertices' slot from 'polygon'. We can also define subclasses that implement 'vertices' differently than 'polygon'. For example, a rectangle that stores the four corners and computes the vertices from this. Due to the seperation of state and behaviour this can be modelled easily:

rectangle_traits = (|
  parent* = polygon_traits.
  draw = ...draw rectangle using left, right, top, bottom...
  vertices = ...compute vertices list using left, right, etc...
|)

rectangle = (|
  parent* = rectangle_traits.
  left.
  right.
  top.
  bottom.
|)

Inheritance in Self can be dynamic if the prototype slots are made assignable. This means, at runtime, we can change the value of the slot used during message lookup, resulting in different behaviour. This can be used for objects that can be in different states.

An example is a 'file' object. It can be opened or closed. Some methods in 'file' can only be used when the file is open, and some only when it is closed. This could be managed by conditional checks in each method. Or the parent of the object could be changed to a different traits object depending on the state - this avoids the need for each method to check if the file is in the open or closed state:

open_file_traits = (|
  read = ...
  close = setParent: close_file_traits.
|)

closed_file_traits = (|
  open = setParent: open_file_traits.
|)

file = (|
  parent* = closed_file_traits.
|)

Methods like 'open' are only available on closed files. 'read' can only be called on opened files. This is basically the Strategy pattern made easy using Self's dynamic inheritance.

Io

Whereas Self defines the prototype lookup chain to be that of the prototype slots in an object, Io instead has a slot called 'protos' which is a list of all objects in the prototype chain. Instead of creating slots with a name suffixed with '*' you append to the existing 'protos' list.

The 'protos' list is initially populated when you clone an object with the object that you cloned. This is unlike Self where copying an object does a shallow copy of all the slots of that object. In Io you get a 'differential inheritance' model where your newly created object has no slots, just a 'protos' field that contains the original object that was cloned. The Self 'point' example I used earlier looks like:

Point := Object clone do(
  x := 0
  y := 0
)

Calling 'clone' on this new 'Point' object results in a new object that does not contain it's own 'x' and 'y' values. Instead its 'protos' field points to the 'Point' object which contains the values. When you set the 'x' value on the clone it will then create its own 'x' slot rather than changing the prototypes. In this way clones of big objects where relatively few slots are changed will save some memory:

a := Point clone
b := Point clone
a x = 5
a x println
 => 5
b x println
 => 0
a protos first == Point
 => true
b protos first == Point
 => true

This does have an interesting side effect in that if you clone a clone then you can end up with a longish prototype chain for the method lookup:

a := Point clone
b := a clone
c := b clone
c protos first == b
 => true
c protos first protos first == a
 => true
c protos first protos first protos first == Point
 => true

Inheritance is handled in much the same manner as Self but you need to manipulate the 'protos' slot instead of having multiple prototype slots. The filled polygon example looks like:

PolygonTraits = Object clone do(
  draw := method(vertices foreach(v, v println))
)

Polygon := PolygonTraits clone do(
  vertices := list(1,2,3,4)
)

FilledPolygonTraits := PolygonTraits clone do(
  draw := method(resend; fillPattern println)
)

FilledPolygon := FilledPolygonTraits clone do(
  appendProto(Polygon clone)
  fillPattern := "solid"
)

Polygon clone draw
  => 1
     2
     3
FilledPolygon clone draw
  => 1
     2
     3
     "solid"

'appendProto' appends an object to the prototype chain which is initially 'FilledPolygonTraits' in this examples as that was the initial object we cloned. The dynamic inheritance example can also be done in Io:

OpenFileTraits := Object clone do(
  read := method(n, "Reading #{n} bytes" interpolate println)
  close := method(
    self removeProto(OpenFileTraits)
    self appendProto(ClosedFileTraits)
  )
)

ClosedFileTraits := Object clone do(
  open := method(
    self removeProto(ClosedFileTraits)
    self appendProto(OpenFileTraits)
  )
)

File := Object clone do(
  init := method(self appendProto(ClosedFileTraits))
)

f := File clone
f read(255)
  => Exception: File does not respond to 'read'
f open
f read(255)
  => reading 255 bytes
f open
  => Exception: File does not respond to 'open'
f close

It's a bit more work than in Self to manually manage the prototype chain but does work.

JavaScript

JavaScript is also a prototype based programming language. Unlike Self or Io it only allows one object to be used as the prototype in any given object. This is stored in a hidden 'proto' member and cannot be updated once set on construction (some implementations allow changing it however). Objects are created by using the 'new' keyword on a constructor function that initializes the object. For now I'll leave it as an exercise for the reader to implement the examples above in JavaScript. I'd be interested in the approaches people take.

Reading Ogg files with JavaScript

2009-06-05T15:09:00+12:00

On tinyvid.tv I do quite a bit of server side reading of Ogg files to get things like duration and bitrate information when serving information about the media. I wondered if it would be possible to do this sort of thing using JavaScript running in the browser.

The format of the Ogg container is defined in RFC 3533. The difficulty comes in reading binary data from JavaScript. The XMLHttpRequest object can be used to retrieve data via a URL from JavaScript in a page but processing the binary data in the Ogg file is problematic. The response returned by XMLHttpRequest assumes text or XML (in Firefox at least).

One way of handling binary data is described in this Mozilla Developer article. Trying this method out works in Firefox and I can download and read the data in the Ogg file.

Ideally I don't want to download the entire file. It might be a large video. I thought by handling the 'progress' event or ready state 3 (data received) I'd be able to look at the data currently retrieved. This does work but on each call to the 'responseText' attribute in these events Firefox copies its internal copy of the downloaded data into a JavaScript array. Doing this every time a portion of the file is downloaded results in major memory use and slow downs proving impractical for even small files.

I think the only reliable way to process the file in chunks is to use byte range requests and do multiple requests. Is there a more reliable way to do binary file reading via JavaScript using XMLHttpRequest? I'd like to be able to process the file in chunks using an Iteratee style approach.

I put up a rough quick demo of loading the first 100Kb of a video and displaying information from each Ogg packet. This probably works in Firefox only due to the workaround needed to read binary data. Click on the 'Go' button in the demo page. This will load transformers320.ogg and display the contents of the first Ogg physical page.

I decode the header packets for Theora and Vorbis. So the first page shown will show it is for a Theora stream with a given size and framerate. Clicking 'Next' will move on to the Next page. This is a Vorbis header with the rate and channel information. Clicking 'Next' again gets the comment header for the Theora stream. The demo reads the comments and displays them. The same for the Vorbis comment records. As you 'Next' through the file it displays the meaning of the granulepos for each page. It shows whether the Theora data is for a keyframe, what time position it is, etc.

Something like this could be used to read metadata from Ogg files, read subtitle information, show duration, etc. More interesting would be to implement a Theora and/or Vorbis decoder in JavaScript and see how it performs.

The main issues with doing this from JavaScript seem to be:

Handling binary data using XMLHttpRequest in a cross browser manner
Processing the file in chunks so the entire file does not need to be kept in memory
Files need to be hosted on the same domain as the page. tinyvid.tv adds the W3C Access Control headers so they can be accessed cross domain but it also hosts some files on Amazon S3 where these headers can't be added. As a result even tinyvid itself can't use XMLHttpRequest to read these files.

JavaScript Space Invaders Emulator

2008-09-11T11:15:00+12:00

Ajaxian recently posted about a fun JavaScript implementation of PacMan. After spending way too much time on it I wondered how well an emulation of the old arcade game hardware would go in JavaScript.

I've written a few 8080 arcade game emulations before in different languages so I had a go at implementing it in JavaScript. You can try the work in progress at my JavaScript 8080 Emulation page. It runs surprisingly well on modern JavaScript engines.

The page first loads with the arcade game Space Invaders loaded. You can run a set number of instructions, or step through one at a time. It displays the disassembled output. Pressing 'Animate' will run the game in a timer and it can be played. It is a general 8080 arcade game emulator, for the games that use similar hardware to Space Invaders. The buttons at the top load the code for Space Invaders, Lunar Rescue or Balloon Bomber.

If you have a bleeding edge version of Firefox 3.1 with Ogg Vorbis <audio> support, pressing the 'Enable Audio' button will enable sound. The sound support uses <audio> to play the samples when requested by the emulator. This turns out to make a good test case for my audio support and it may need the fixes from bug 449159 and 454364 to work.

If you're interested in the other emulators I've done:

The implementation does not get traced with the TraceMonkey tracing JIT yet. I'll look into the reasons why and as TraceMonkey and my implementation improves it'll get faster I'm sure. Even so, it runs very close to full speed.

This implementation uses Canvas, audio for sound and should work on browsers with a fast JS engine and these technologies.

For the emulator loop I run a set number of instructions during a timer that is run via 'setInterval' to prevent the 'script is running too long' message. One thought that Robert O'Callahan suggested was to run the emulator in a worker thread and have communication for input/output via messages to the browser. I'll play with this idea and see how it goes - it'll give me a chance to try out Firefox 3.1 worker threads implementation.

The emulator can be run (without the GUI) from a JavaScript shell for testing purposes. I used the shell to test the implementation by running my Factor version and logging all the state of the emulated CPU, doing the same with the JavaScript version, and making sure the output was the same.

Although it's not quite perfect, it's currently playable, and shows that the types of games that are written as Java or Flash applets can be done in standard HTML and JavaScript in the latest browsers.

Implementing Native Methods in Tamarin Tracing

2008-05-20T16:47:00+12:00

2008-05-20: Minor update to get things working with latest Tamarin Tracing code, and updated times for test runs.

Tamarin Tracing can be extended by creating native methods. These are methods of a class where the implementation is in C rather than JavaScript.

For this example I'll use a native implementation of the fibonacci function and compare it to the JavaScript version in my previous post.

A JavaScript function that is implemented in C using the 'native' modifier in the JavaScript source. For example, a natively implemented 'fib' function would be declared in JavaScript as:

public native function fib(n:int):int;

Notice that this includes the type of the arguments and the return type. This is so the compiler can produce the correct C types in the C stub code it generates.

The native method must be implemented in C and linked into the final executable. The name of the function is in the following form:

[class]_[visibility]_[name]

In this fib example there is no class, so 'null' is used for that part of the name and visibility is public so that part of the name is left out. The end result is a native C function called null_fib needs to be implemented.

As part of the compilation process the compiler generates a C structure that will be accessed by the native implementation to extract the arguments passed to it. This structure looks like:

struct null_fib_args
{
    public: ScriptObjectp /*global1*/ self; private: int32_t self_pad; 
    public: int32_t n; private: int32_t n_pad; 
    public: StatusOut* status_out;
};

The 'n' field of the structure is the argument passed from JavaScript callers. The native implementation, which we need to write, looks like this:

int32_t native_fib(int32_t n) {
    if(n <= 1)
      return 1;
    else
      return native_fib(n-1)+native_fib(n-2);
  }

  AVMPLUS_NATIVE_METHOD(int32_t, null_fib)
  {
    return native_fib(args->n);
  }

First there is the native_fib C function that we want to call from JavaScript. The AVMPLUS_NATIVE_METHOD macro is used to declare the wrapper function that implements the 'native function fib' we declared in the JavaScript file. This receives an 'args' object that is an instance of the null_fib_args C structure mentioned previously. This is used in our example to extract the passed integer value and call the native C function and return the result.

Native function implementations must be linked into the tamarin tracing executable. It's not possible to compile a JavaScript file containing a native declaration and run it using the tamarin tracing 'avmshell' program. To integrate the fib code into 'avmshell' I modify the shell code to compile and link in the native implementation. We can then write JavaScript code that calls it and run it with 'avmshell'.

The first thing to do is write the JavaScript side of the 'fib' code. In a 'fib.as' file in the 'shell' directory of tamarin tracing I have the following code:

package testing {
  public function fib(n) {
    if(n <= 1)
      return 1;
    else
      return fib(n-1) + fib(n-2);
  }

  public native function fib2(n:int):int;
}

This provides a JavaScript implementation of fibonacci and one called 'fib2', intended to be implemented with C code so I can compare the speed.

This file needs to be compiled to abc bytecode and have the args structure generated in a C header file. There is a script, shell.py, in the 'shell' subdirectory that does this for the other avmshell classes. Changing the line following the comment 'compile builtins' so it includes the 'fib.as' file just created will result in it being included in the build.

What this line in shell.py does is compile the JavaScript files using the Flex SDK compiler (See later about where to get this and where to put it). The command it runs is something like:

java -jar asc.jar -import builtin_full.abc ... fib.as

This produces the abc bytecode for our fibonacci code, as outlined in my previous post.

The next command run by 'shell.py' is the Flex Global Optimizer. This takes all the abc bytecode files for the shell, optimizes them, and produces a C header and implementation file. It is these C files that contain the generated arguments structure, and the implementation file actually contains a C array of the optimized bytecode. The output of this step will be compiled by a C compiler and linked into the 'avmshell' executable.

The native C implementation of the 'fib2' function should be placed in a file in the 'shell' subdirectory and that file added to the 'manifest.mk' makefile. The contents of this file for this example is:

#include "avmshell.h"
#include <stdlib.h>

namespace avmplus
{
  int32_t native_fib(int32_t n) {
    if(n <= 1)
      return 1;
    else
      return native_fib(n-1)+native_fib(n-2);
  }

  AVMPLUS_NATIVE_METHOD(int32_t, null_fib2)
  {
    return native_fib(args->n);
  }
}

I called this 'fibimpl.cpp' and added it to manifest.mk. You'll see in the 'shell' subdirectory various implementations of native methods in [foo]Class.cpp files, where [foo] is the JavaScript class being implemented. There are also [foo].as files which have the JavaScript side of the implementation.

To build our new 'avmshell' which is able to call our native fibonacci implementation, run 'shell.py', and do the configure and make steps as outlined previously:

$ mkdir mybuild
$ cd mybuild
$ ../tamarin-tracing/configure --enable-shell
$ make

I wrote two simple test files to test the 'fib' and 'fib2' functions:

$ cat fib.as
import testing.*;
print("fib 30 = " + fib(30));
$ cat fib2.as
import testing.*;
print("fib 30 = " + fib2(30));

Here are some simple timings on my machine with the tracing jit enabled and disabled:

$ time ./shell/avmshell fib.abc
fib 30 = 1346269

real    0m0.417s
user    0m0.384s
sys     0m0.020s
$ time ./shell/avmshell fib2.abc
fib 30 = 1346269

real    0m0.092s
user    0m0.060s
sys     0m0.020s

$ time ./shell/avmshell -interp fib.abc
fib 30 = 1346269

real    0m7.496s
user    0m7.448s
sys     0m0.004s
$ time ./shell/avmshell -interp fib2.abc
fib 30 = 1346269

real    0m0.070s
user    0m0.060s
sys     0m0.004s

Another way of extending tamarin tracing is via forth. I'll cover that in a later post.

I mentioned earlier about needing the Flex ActionScript compiler and global optimizer from their asc.jar file. Unfortunately tamarin tracing needs a bleeding edge version of this to generate the correct C code. A recent version can be obtained from Mozilla public ftp. This should be placed in the 'utils' subdirectory to be picked up by the scripts. Even more unfortunately this version is out of date for the latest mercurial repository code. Hopefully this situation will be rectified soon, but in the meantime you can go back to changeset 302 from the mercurial repository. I tested the current asc.jar against that.

There are some interesting things from the Tamarin summit about the generated arguments structure. You'll notice it has some padding fields in it. When the native implementation function is called from Forth, the layout of the Forth stack looks like (in Forth stack format):

( obj arg1 ... argn status -- )

Each value on the Forth stack is a 64 bit value. The generated structure type exactly matches the Forth stack layout.

This means that when the Forth stack is ready for the native call, the argument object is actually a pointer to a location on the stack. There is no intermediate argument object actually allocated. The padding fields are to enable exactly matching up with the items on the stack.

Interestingly, if I recall correctly from the Tamarin summit, calling native methods from the tracing jit is actually less efficient than calling it from the interpreter. This is because the interpreter uses the stack layout trick for the arguments object above. But for the tracing jit the argument values are often stored in registers or other memory locations. These must be copied into an arguments object and then the native function called. This is a slight overhead.

Please feel free to leave a comment or email me if you have any questions or corrections to the above. It represents my understanding from attending the summit and playing with the code and may not necessarily be the best way of doing things, or may be incorrect in places.

A Quick Introduction to Tamarin Tracing

2008-05-20T16:14:00+12:00

2008-05-20: Fixed some breakage due to changes to the latest Tamarin Tracing source, and updated more recent timing.

I attended the Tamarin Tech summit at Adobe on Friday. My main interest for attending was to learn more about the tamarin-tracing project. The goal of Tamarin is to produce a high performance ECMAScript 4 implementation.

'Tamarin Tracing' is an implementation that uses a 'tracing jit'. This type of 'just in time compiler' traces code executing during hotspots and compiles it so when those hotspots are entered again the compiled code is run instead. It traces each statement executed, including within other function calls, and this entire execution path is compiled. This is different from compiling individual functions. You can gain more information for the optimizer to operate on, and remove some of the overhead of the calls. Anytime the compiled code makes a call to code that has not been jitted, the interpreter is called to continue.

Apparently the JIT for Lua is also being written using a tracing jit method and a post by Mike Pall describes the approach they are taking in some detail and lists references. A followup post provides more information and mentions Tamarin Tracing.

'Tamarin Tracing' is open source and can be obtained from the mercurial repository:

$ hg clone http://hg.mozilla.org/tamarin-tracing/

To build the source you create a directory to hold the build files, change to it, and run the configure script:

$ $ mkdir mybuild
$ cd mybuild
$ ../tamarin-tracing/configure --enable-shell
$ make

The 'enable-shell' option is required to produce the 'avmshell' binary that executes the bytecode. At the end of the build you'll see the avmshell binary in the shell subdirectory:

$ shell/avmshell
avmplus shell 1.0 build cyclone

usage: avmplus [options] scripts [--] script args
          -Dtimeout            enforce maximum 15 seconds 
                               execution
          -error               crash opens debug dialog, 
                               instead of dumping
          -suppress_stdout     don't emit anything to 
                               stdout (debug messages only)
          -interp              disable the trace optimizer 
                               and nanojit
          -Dnoloops            disable loop invariant hoisting
          -Dnocse              disable common subexpression 
                               elimination
          -Dnosse              disable SSE2 instructions
          -log                 send verbose output to 
                               <script>.log

'avmshell' operates on files containing bytecode not JavaScript. To use it you'll need to have a front end that compiles JavaScript to the 'abc' bytecode format it uses. The bytecode is the ActionScript bytecode. You'll need a compiler that generates this. This can be obtained from the Flex SDK. This is a free download from Adobe. You can also use any other tool that generates the correct bytecode.

Included with Tamarin Tracing is the source for 'esc'. This is a work-in-progress implementation of an ECMAScript 4 compiler written in ECMAScript. It generates the 'abc' bytecode but is (I think) not quite ready for prime time. In this post I'm using the 'asc' compiler from the Flex 2 SDK on Linux. This compiler is written in Java and is in the 'lib/asc.jar' file in the SDK.

A quick test that the avmshell program works:

$ echo "print('hello world!');" >>hello.as
$ java -jar asc.jar hello.as
hello.abc, 86 bytes written
$ shell/avmshell hello.abc
hello world!

'avmshell' has a number of debugging options that are only available when configuring the build with '--enable-debugger'. This allows you to get some information about the trace jit. Here's the build process with a debug enabled build and the available options:

$ mkdir mybuild
$ cd mybuild
$ ../tamarin-tracing/configure --enable-shell --enable-debugger
$ make
$ shell/avmshell
avmplus shell 1.0 build cyclone

usage: avmplus [options] scripts [--] script args
          -d                   enter debugger on start
          -Dnogc               don't collect
          -Dgcstats            generate statistics on gc
          -Dnoincgc            don't use incremental collection
          -Dastrace N          display AS execution information, 
                               where N is [1..4]
          -Dverbose            trace every instruction (verbose!)
          -Dverbose_init       trace builtins too
          -Dverbose_opt_exits  trace optimizer exit instructions
          -Dverbose_opt_detail extreme optimizer verbosity 
          -Dquiet_opt          disable verbosity for optimizer
          -Dstats              display various optimizer 
                               statistics 
          -Dsuperwords         dump basic block usage to stderr 
                               (use with -interp; 
                                2> to save to file, then 
                                superwords.py) 
          -Dtimeout            enforce maximum 15 seconds 
                               execution
          -error               crash opens debug dialog, instead of 
                               dumping
          -suppress_stdout     don't emit anything to stdout 
                               (debug messages only)
          -interp              disable the trace optimizer and 
                               nanojit
          -Dnoloops            disable loop invariant hoisting
          -Dnocse              disable common subexpression 
                               elimination
          -Dnosse              disable SSE2 instructions
          -log                 send verbose output to 
                               <script>.log

To demonstrate some of the output I'll use a simple fibonacci benchmark. This is the contents of fib.as:

function fib(n) {
 if(n <= 1)
  return 1;
 else
  return fib(n-1) + fib(n-2);
}

print("fib 30 = " + fib(30));

A comparison of times with and without the tracing jit enabled:

$ time ./shell/avmshell -interp fib.abc
fib 30 = 1346269

real    0m7.550s
user    0m7.504s
sys     0m0.004s
$ time ./shell/avmshell fib.abc
fib 30 = 1346269

real    0m0.391s
user    0m0.360s
sys     0m0.016s

A complete verbose log is very large and shows the execution of the program, the trace and the assembly code generated:

$ shell/avmshell -Dverbose fib.abc
...
  interp global$init()
  0:getlocal0
  1:pushscope ( global@20c1e61 )
  2:newfunction method_id=0 
  4:getglobalscope
  5:swap ( Function-0 global@20c1e61 )
  6:setslot 1 ( global@20c1e61 Function-0 )
 ...
 interp ()
  0:getlocal1
  1:pushbyte 1
  3:ifnle 10 ( 30 1 )
  10:getglobalscope
  11:nop
  12:getlocal1
  13:pushbyte 1
  15:subtract ( 30 1 )
  16:callproperty {public,fib.as$0}::fib 1 ( global@20c1e61 29 )
...
 10:getglobalscope
  11:nop
  12:getlocal1
  13:pushbyte 1
  15:subtract ( 28 1 )
  16:callproperty {public,fib.as$0}::fib 1 ( global@20c1e61 27 )
  interp ()
SOT  pc 107D148 ip D9DD5 sp 10100FC rp 10082E4
         trace 4314 (10DA000)
       1 in    ecx
       3 int   #20D8940
       4 arg   3
       5 arg   1
       6 call  fragenter
         reference to rp
       7 imm   #16
       8 ld    7(1)
...
   GG: pc 107D148 ip D9DD5 sp 101010C rp 100832C
 assembling pass 1 from 4311:62
       1 in    ecx
         010DF786  mov ecx,-4(ebp)                  ecx(1)
       3 int   #20D8940
       4 arg   3
         010DF789  mov edx,34441536                 ecx(1)
       5 arg   1
       6-call  fragenter
         010DF78E  call 2E96E:fragenter                
         010DF793  mov ecx,-4(ebp)                  ecx(1)
       7 imm   #16
       8-ld    7(1)
         010DF796  mov edi,16(ecx)                  ecx(1)
         010DF799  mov -12(ebp),edi                 ecx(1) edi(8)
 ...

There's a lot of other interesting stuff in the Tamarin Tracing source that I hope to dive into. For example:

the interpreter is written in Forth. There are .fs files in the 'core' subdirectory that contains the Forth source code. Each 'abc' bytecode is implemented in lower level instructions which are implemented in Forth. The tracing jit operates on these lower level instructions. The system can be extended with Forth code to call native C functions. The compiler from Forth to C++ is written in Python and is in 'utils/fc.py'
The jit has two backends. One for Intel x86 32 bit, and the other for ARM. See the 'nanojit' subdirectory.
The complete interpreter source can be rebuilt from the Forth using 'core/builtin.py'. This requires 'asc.jar' to be placed in the 'utils' subdirectory of Tamarin Tracing.

At the summit there was an in-depth session of the internals of the Forth code and how to extend it. I'll write more about that later when/if I get a chance to dig into it.

ECMAScript 4 Reference Implementation Updated

2007-11-02T23:47:00+13:00

The reference implementation for ECMAScript 4 has been updated to M1 and is available for download. The source is available (written in SML), as well as binaries for Windows, OS X, and Linux.

Javascript Packrat Parser

2007-10-31T01:01:00+13:00

I've made some changes to my parser combinator library written in Javascript that I wrote about previously.

I was extending the example code from that post to parse more of the ECMAScript 3 grammar to test it out and was hitting performance issues. I ended up modifying the combinator library to be a Packrat style parser. The Parsing Expression Grammar Wikipedia page has a description of PEGs and what a Packrat parser does. Basically results of each parse step are memoized to prevent reparsing after backtracing, sacrificing memory for speed:

Any parsing expression grammar can be converted directly into a recursive descent parser[citation needed]. Due to the unlimited lookahead capability that the grammar formalism provides, however, the resulting parser could exhibit exponential time performance in the worst case.

By memoizing the results of intermediate parsing steps and ensuring that each parsing function is only invoked at most once at a given input position, however, it is possible to convert any parsing expression grammar into a packrat parser, which always runs in linear time at the cost of substantially greater storage space requirements.

A packrat parser[1] is a form of parser similar to a recursive descent parser in construction, except that during the parsing process it memoizes the intermediate results of all invocations of the mutually recursive parsing functions. Because of this memoization, a packrat parser has the ability to parse many context-free grammars and any parsing expression grammar (including some that do not represent context-free languages) in linear time.

I changed a few other things in the library to more closely map to the vocabulary of PEGs. 'alternative' is now called 'choice' for example. There are still quite a few loose ends to tidy up and documentation of course.

The updated library can be retrieved from my git repository:

git clone git://github.com/doublec/jsparse.git

It can run in a browser but I've mostly been testing it using Mozilla Rhino.

I've included three basic examples. They all operate on the example expression grammar from the wikipedia article:

Value   := [0-9]+ / '(' Expr ')'
Product := Value (('*' / '/') Value)*
Sum     := Product (('+' / '-') Product)*
Expr    := Sum

The first, example1.js, is a direct translation of that grammer. It produces a pretty ugly default Abstract Syntax Tree however:

var Value = choice(repeat1(range('0','9')), Expr);
var Product = sequence(Value, repeat0(sequence(choice('*', '/'), Value)));
var Sum = sequence(Product, repeat0(sequence(choice('+', '-'), Product)));
var Expr = Sum;

The Expr parser can be called by passing it a string to be parsed wrapped in a 'ParserState' object. This object is used to keep track of the current parse position and the memoized results. A helper function, 'ps', can be used to construct it:

var result = Expr(ps("1+2*3"));

The second example, example2.js adds to this to produce a better AST. It also uses the 'chainl' parser combinator to handle grouping correctly. A quick online page demonstrating this example is here. Enter an expression matching the grammar (there is no error checking yet), and press the button to see the AST in JSON format.

The third example, example3.js, evaluates the expression as it parses instead of generating an AST. This is also available online to try.

I've also included in the repository the work in progress of the ECMAScript 3 grammar. It is not complete or correct yet but I use it for testing the library.

Based on what I've learnt from doing this I plan to revisit the way I did Parser Combinators in Factor.

ECMAScript 4 Overview Paper

2007-10-23T10:09:00+13:00

An overview paper describing the ECMAScript 4 language features was announced on the es4-discuss mailing list:

I'm pleased to present you with an overview paper describing ES4 as the language currently stands. TG1 is no longer accepting proposals, we're working on the ES4 reference implementation, and we're expecting the standard to be finished in October 2008. ... This paper is not a spec, it is just a detailed overview. Some features may be cut, others may be changed, and numerous details remain to be worked out, but by and large this is what TG1 expects the language to look like. Your comments on the paper are very much welcome. Please send bug reports directly to me, everything else to the list.

The document is available on the ECMAScript 4 language site in overview.pdf.

Javascript Parser Combinators

2007-10-03T01:19:00+13:00

Cleaning up my hard drive I came across some old libraries I'd written. One of them was a simple set of parser combinators written in Javascript. I put it in a git repository in case they prove useful to someone:

git clone git://github.com/doublec/jsparse.git

The library pulls ideas from parser combinators written in various languages and is pretty simple. But even though it is a small amount of code it works quite well. The repository includes an example of parsing numbers and strings based on the grammar in the EcmaScript specification.

A parser in this library is a function that takes an input string and returns a result object. The result object contains three fields. They are:

remaining: the remaining part of the input string to be parsed
matched: the part of the input string that was successfully parsed by the parser
ast: The abstract syntax tree produced by the parsr

Parser combinators combine parsers together to enable parsing more complex grammars. A number of standard parsers and combinators are provided.

'token' is a combinator that takes a string and returns a parser that will successfully parse an instance of that string:

js> load("jsparse.js")
js> var p = token("begin")
js> uneval(p("begin ... end"))
({remaining:" ... end", matched:"begin", ast:"begin"})

The AST produced by parser is the string that it parsed.

'range' returns a parser that parses single characters within the range of the upper and lower character bounds (inclusive) given:

js> var p = range("0", "9")
js> uneval(p("5"))
({remaining:"", matched:"5", ast:"5"})
js> uneval(p("a"))
false

'negate' takes an existing single character parser, and returns once which parses anything but that which the original parser parsed. For example, 'negate(range("a", "z"))' will return a parser which parses anything except the letters from a to z inclusive:

js> var p = negate(range("a", "z"))
js> uneval(p("g"))
false
js> uneval(p("5"))
({remaining:"", matched:"5", ast:"5"})

'sequence' takes any number of parsers as arguments and returns a parser which suceeds if all the given parsers succeed in order. The AST it returns is an array of the results of each of the parsers.

js> var p = sequence(token("a"), token("b"), token("c"))
js> uneval(p("abcdef"))
({remaining:"def", matched:"abc", ast:["a", "b", "c"]})
js> uneval(p("abdef"))
false

'alternate' provides choice between parsers. It takes any number of parsers as arguments and will try each of them in order. The first one that succeeds results in a successful parse, and its result is the AST:

js> var p = alternate(token("a"), token("b"), token("c"))
js> uneval(p("a123"))
({remaining:"123", matched:"a", ast:"a"})
js> uneval(p("b123"))
({remaining:"123", matched:"b", ast:"b"})
js> uneval(p("c123"))
({remaining:"123", matched:"c", ast:"c"})
js> uneval(p("d123"))
false

'repeat0' does the equivalent of * in regular expressions. It takes a parser and returns a parser which will parse zero or more occurrences of the original parser. The AST is an array containing the AST result of the original parser for each successful occurrence:

 js> var p = repeat0(range("0", "9"))
 js> uneval(p("12345abcd"))
 ({remaining:"abcd", matched:"12345", ast:["1", "2", "3", "4", "5"]})
 js> uneval(p("123abcd"))
 ({remaining:"abcd", matched:"123", ast:["1", "2", "3"]})
 js> uneval(p("abcd"))
 ({remaining:"abcd", matched:"", ast:[]})

'repeat1' does the equivalent of '+' in regular expressions. It takes a parser and results one which will parse one or more occurences of the original parser:

js> var p = repeat1(range("0", "9"))
js> uneval(p("12345abcd"))
({remaining:"abcd", matched:"12345", ast:["1", "2", "3", "4", "5"]})
js> uneval(p("abcd"))
false

'optional' takes a parser and returns one which matches exactly zero or one instances of the original. The AST result is 'false' for the case where there is no match or the result of the original parser if there is a match:

js> var p = sequence(optional(alternate("+", "-")), repeat1(range("0", "9")))
js> uneval(p("1234"))
({remaining:"", matched:"1234", ast:[false, ["1", "2", "3", "4"]]})
js> uneval(p("-1234"))
({remaining:"", matched:"-1234", ast:["-", ["1", "2", "3", "4"]]})
js> uneval(p("+1234"))
({remaining:"", matched:"+1234", ast:["+", ["1", "2", "3", "4"]]})
js> uneval(p("*1234"))
false

You'll notice in this example that I pass "+" and "-" directly instead of token("+") and token("-"). The parsers in this library will automatically convert strings to parsers where needed to make for terser and more readable code.

The AST produced by some of the generated parsers can be non-optimal. For example, a simple parser will produce an array of strings for each digit:

js> var p = repeat1(range("0", "9"))
js> p("123").ast
1,2,3
js> uneval(p("123").ast)
["1", "2", "3"]

The 'action' combinator takes a parser and a function. The function is called with the result of the AST produced by the parser, and the result of the function becomes the new AST. For example, compare these two:

js> var p = action(range("0", "9"), function(ast) { return parseInt(ast) })
js> uneval(p("1"))
({remaining:"", matched:"1", ast:1})
js> var p = range("0", "9")
js> uneval(p("1"))
({remaining:"", matched:"1", ast:"1"})

In the first the result is an actual number, in the second it is a string. Any object can be returned and used for the AST. A abstract syntax tree for the parsed language for example.

There are other combinators provided in the library but these basics do most of what is needed.

The 'example1.js' file shows a translation of some of the grammar productions in the EcmaScript grammar:

var zero 
  = action("0", function(ast) { return 0; });
var decimal_digit 
  = action(range("0", "9"), function(ast) { return parseInt(ast); });
var non_zero_digit 
  = action(range("1", "9"), function(ast) { return parseInt(ast); });
var decimal_digits 
  = repeat1(decimal_digit); 
var decimal_integer_literal 
  = alternate(zero, sequence(non_zero_digit, optional(decimal_digits)));
var signed_integer 
  = alternate(decimal_digits, 
              sequence("+", decimal_digits), 
              sequence("-", decimal_digits));
var exponent_indicator 
  = alternate("e", "E");
var exponent_part 
  = sequence(exponent_indicator, signed_integer);
var decimal_literal = 
  alternate(sequence(decimal_integer_literal, 
                     ".", 
                     optional(decimal_digits), 
                     optional(exponent_part)),
            sequence(".", 
                     decimal_digits, 
                     optional(exponent_part)),
            sequence(decimal_integer_literal, 
                     optional(exponent_part)));

var hex_digit 
  = alternate(range("0", "9"), 
              range("a", "f"), 
              range("A", "F"));
var hex_integer_literal 
  = sequence(alternate("0x", "0X"), 
             repeat1(hex_digit));

var numeric_literal 
  = alternate(hex_integer_literal, decimal_literal);

var single_escape_character 
  = alternate("'", "\"", "\\", "b", "f", "n", "r", "t", "v");
var non_escape_character 
  = negate(single_escape_character);
var character_escape_sequence 
  = alternate(single_escape_character, non_escape_character);
var hex_escape_sequence 
  = sequence("x", hex_digit, hex_digit);
var unicode_escape_sequence 
  = sequence("u", hex_digit, hex_digit, hex_digit, hex_digit);
var escape_sequence 
  = alternate(hex_escape_sequence, 
              unicode_escape_sequence, 
              character_escape_sequence);
var single_string_character 
  = alternate(negate(alternate("\'", "\\", "\r", "\n")),
              sequence("\\", escape_sequence));
var double_string_character 
  = alternate(negate(alternate("\"", "\\", "\r", "\n")),
              sequence("\\", escape_sequence));
var single_string_characters 
  = repeat1(single_string_character);
var double_string_characters 
  = repeat1(double_string_character);
var string_literal 
  = alternate(sequence("\"", optional(double_string_characters), "\""),
              sequence("'", optional(single_string_characters), "'"));

var null_literal 
  = token("null");
var boolean_literal 
  = alternate("true", "false");

var literal 
  = alternate(null_literal, 
              boolean_literal, 
              numeric_literal, 
              string_literal);

I'd like to extend the library a bit and provide more examples. Any comments or ideas would be appreciated.

Standard ML to Javascript compiler

2007-10-01T15:53:00+13:00

smltojs is a compiler from Standard ML to Javascript. According to the page it has support for all of Standard ML.

Since the reference implementation of Ecmascript 4 is written in Standard ML it would be interesting to see if it can be built using this compiler. That would provide an es4 implementation that runs in the browser based off the reference implementation.