Bluish Coder

Distributed Concurrency in Factor

One of the advantages (and pretty much the only one) of being down with the flu recently is I've been able to hack away at Factor code, like the serialization library mentioned previously.

I've just added simple distributed message passing support to the concurrency library. Processes now belong to 'nodes'. These are individual Factor instances running on a machine. Messages can be sent between Factor nodes using the same foramt as sending from processes within a Factor instance.

So far the support I've added is very basic and not at all optimized but it works. Messages can be any Factor type supported by the serialization library.

I've extended the serialization library by added a serializer for local processes that serializes it as a 'remote-process'. This holds the node details (hostname and port) and the process identifier (known as the pid). This allows you to send a local process to a remote process, and that remote process can send a reply back to the local one by sending a message to the remote-process object it receives.

A possible future extension to this might be to serialize proxies for other types. For example, sending a stream in a message can serialize it as a proxy stream so that writes to it from the remote system are sent back to the stream on the local system.

The current state of the system is in my repository:

darcs get http://www.bluishcoder.co.nz/repos/factor

The 'start-node' word is used to start up a TCP listener to handle requests from remote processes. This is required for distributed message passing to work. If you don't use 'start-node' only local message sends will be enabled. Here's a simple example that sends a message to a remote process, and it sends a reply back to the caller:

#! On Machine 1
"concurrency" require
USE: concurrency

"machine1.com" 9000 start-node

: process1 ( -- )
  receive "Message Received!" reply process1 ;

[ process1 ] spawn "process1" swap register-process

This starts the node up with hostname 'machine1.com' on port 9000. A word called 'process1' is defined that blocks until it receives a message (either from another local process or a remote process) and replies to the message sender with the string "Message Received!". It then calls itself to restart the blocking on message receive.

This word is spawned as a process and registered in that nodes global register of named processes as 'process1'.

#! On Machine 2
"concurrency" require
USE: concurrency

"machine2.com" 9000 start-node

[ "machine1.com" 9000 <node> "process1" <remote-process> 
  "test-message" swap send-synchronous .
] spawn

This code, run in a Factor instance on Machine 2, starts a node with hostname 'machine2.com' on port 9000. It spawns a process which sends a message to the process named 'process1' on the node at hostname machine1.com, port 9000.

This example sends a synchronous message. It is the equivalent of Termite Scheme's '!?' operator or Joe Armstrongs '!!' proposed Erlang extension - basically an RPC call. The message is sent to process one and blocks waiting for a reply to that specific message. On the reply it displays it (using '.') which results in 'Message Received!' being printed.

Asynchronous sends work too. For example, a logger process on machine 1:

#! On Machine 1

: logger ( -- )
  receive print logger ;

[ logger ] spawn "logger" swap register-process

Messages can be sent to this process from any machine with:

#! On Machine 2

[ "machine1.com" 9000 <node> "logger" <remote-process> 
  "Log Message!" swap send
] spawn

After this message send 'Log Message!' will be printed on machine 1.

Messages can be sent to named processes, registered in the nodes global registry, as these examples show, or they can be sent to any process given the pid - a unique identifier for that process. They can also be sent to remote processes given a deserialized reference to the process object. You could store on a file system or web server deserialized references to important processes that clients can send messages to.

I'm still working on the public API and making the performance better. Currently all message sends open and close the TCP socket to the remote server per message. There is also no security. The server connection for the node is accessable by anyone. Initially I may follow the Erlang 'magic cookie' approach to prevent unauthorised message sends but keen to look at better options.

My motivation for working on this is to add to my in-progress web framework the ability to have the server side processes distributed across Factor instances or machines. This is one way to enable Factor to use multiple processors in a machine for example.

Object Serialization with Factor

As Slava noted in his weblog, I've tidied up a previous serialization library to get it working with current Factor versions in the hopes of using it to get distributed concurrency working.

The serialisation supports most Factor types and is extensible by adding methods to generic functions. You can even serialize quotations containing words, and so long as those words exist in the target system, the deserialization will link them correctly and the quotation is callable. Actual code doesn't serialize at this stage though.

An example of usage:

"serialize" require
USE: serialize

[
  [ "Hello World!" serialize ] with-serialized
] string-out
  => ...serialized format as a string...

[
  [ deserialize ] with-serialized
] string-in 
  => "Hello World!"

The 'string-out' and 'string-in' shown above are standard words to direct all output to go to and from strings respectively. You can also serialize to files and deserialize them on any other Factor system.

Transterpreter source available

The source for the Transterpreter, a portable Occam implementation, has been made available.

The Transterpreter is a small (2000 lines of code), portable (strict ANSI C), open-source runtime for a growing family of massively concurrent programming languages. Capable of supporting thousands of threads on small devices, it is well suited for embedded and ubiquitous systems development.

Cells style gadget updating in Factor

Factor has a Cells like mechanism for propogating changes in models to their GUI representation (called Gadgets). It's like a lightweight functional reactive programming system. Using the latest darcs Factor code, which includes a Calendar library, you can quite easily get a gadget displaying with a dynamically updated date.

To have a gadget dynamically updated it must reference a 'model' which wraps the value to be displayed. Whenever the data wrapped by the model is changed, the gadget displaying it automatically updates.

In this example I create a global value called 'time' that holds a model wrapping the current date and time:

SYMBOL: time
f <model> time set-global

The model here wraps the value 'f' which is false indicating no value currently set. To have the value change regularly I start a background thread which sets the value every second:

: update-time ( -- )
  now time get set-model 1000 sleep update-time ;

[ update-time ] in-thread

The 'update-time' words gets the current date and time (using 'now') and sets the model's value to it. It does this every 1,000 milliseconds.

We can confirm that this value is updating by doing a 'get' of the 'time' variable and see that it changes between calls:

time get model-value . 
  => T{ timestamp f 2006 8 7 12 52 8.98 -12 }

time get model-value . 
  => T{ timestamp f 2006 8 7 12 52 49.14 -12 }

A label gadget that displays this value, and is updated as it changes, can be created and displayed with:

time get [ timestamp>http-string ] <filter> <label-control> gadget.

The <filter> object takes a model (retrieved with 'time get') and quotation that it will call on the model when its value changes to get a string to display. In this case I'm using the calendar libraries 'timestamp>http-string' word to convert the timestamp held by the model into a string representation. A label gadget is created using this filter as the displayed text.

The 'gadget.' word is what displays the gadget in the GUI. There should appear a date that is updated every second. You can display multiple gadgets on the same model, scroll the window, etc and it will be updated. You can see a screenshot here. You'll have to imagine the seconds value in the screenshot updating every second!

JSON parser for Factor

I've implemented a simple JSON parser for Factor following RFC 4627. It will be available soon from the standard Factor repository and is also available from my darcs repository:

darcs get http://www.bluishcoder.co.nz/repos/factor

The three main words in the 'json' vocabulary are '>json', 'json-print' and 'json>'. An example of use is:

"json" require
USE: json
TUPLE: foo a b ;
"hi" "there" <foo> >json print
  => "{"a" : "hi", "b" : "there"}
"[ 10, 20, \"foo\", { \"value\" : 1e2 } ]" json> .
  => { 10 20 "foo" H{ { "value" 100 } } }

Factor tuples are converted to Javascript objects. Going the other way, Javascript objects are converted to Factor hashtables.