Bluish Coder: factor

Factor Web Framework Example

2008-10-11T12:57:00+13:00

The Factor web framework was recently redeveloped and my previous post on the topic is out of date. I'm re-learning how to write Factor web applications as I work through a simple app I'm building.

At first I was overwhelmed by the large number of features in the new framework, but once I got a bit of an understanding I was pleasantly surprised how easy it was to put an application together.

Starting the web server hasn't changed much. Start a web server running on port 8888 in a background thread:

USING: http.server threads ;
[ 8888 httpd ] in-thread

A web application is written as a bunch of responders. Each responder handles a request from the client browser. An easy way of creating a responder is to create an 'action' object. An 'action' object has a number of slots, one of which is called 'display'. Setting 'display' to a quotation will cause that quotation to be called when the client browser makes a request. The quotation should return a 'response' object. The 'response' object defines what gets sent back to the client browser. The 'body' slot of the 'response' object can be a string which is returned to the browser as the contents of the request. So a basic responder is:

<action> [
  <response>
    200 >>code
    "text/plain" >>content-type
    "Hello World" >>body
] >>display

The web browser uses a global variable called 'main-responder' as the first responder to try when a request is made. If we set 'main-responder' to the value of our responder then it immediately becomes available to a web request. So our entire simple web application is:

USING: http http.server threads furnace.actions ;
[ 8888 httpd ] in-thread
<action> [
  <response>
    200 >>code
    "text/plain" >>content-type
    "Hello World" >>body
] >>display main-responder set-global

Visiting http://localhost:8888/ should display a page with 'Hello World' as the text.

A responder that works just from the root of the URL is not very practical. A dispatcher is an object that can acts as a responder that calls other responders based on part of the URL. Here's a dispatcher that calls one responder when /hello is requested, and another when /goodbye is requested.

USING: 
  furnace.actions 
  http 
  http.server 
  http.server.dispatchers 
  threads 
;
[ 8888 httpd ] in-thread

TUPLE: my-dispatcher < dispatcher ;

my-dispatcher new-dispatcher
  <action> [
    <response>
      200 >>code
      "text/plain" >>content-type
      "Hello World" >>body
  ] >>display "hello" add-responder 
  <action> [
    <response>
      200 >>code
      "text/plain" >>content-type
      "Goodbye World" >>body
  ] >>display "goodbye" add-responder 
main-responder set-global

First I create a tuple called 'my-dispatcher' that inherits from the base dispatcher class. The I create an instance of this using 'new-dispatcher', create my actions and add them under the paths 'hello' and 'goodbye'.

Writing response objects gets a bit tedious. Factor provides two template systems that can be used to make writing web applications easier. One is 'fhtml' which existed in the previous web framework incarnation. The other, chloe, is an XML based templating language, and is covered quite well in the Factor documentation.

To create an action that uses a Chloe template, create a 'page-action' instead of an 'action'. Set the 'template' slot of the page-action to be a two element array containing the dispatcher class and the template filename (without the .xml extension):

USING: 
  furnace.actions 
  html.templates.chloe
  http 
  http.server 
  http.server.dispatchers 
  threads 
;
[ 8888 httpd ] in-thread

TUPLE: my-dispatcher < dispatcher ;

my-dispatcher new-dispatcher
  <page-action>
    { my-dispatcher "main" } >>template
  "foo" add-responder 
  main-responder set-global

When this '/foo' is accessed on the webserver, this action will run. It will look for main.xml which needs to be a chloe template file. Here's a simple example template:

<?xml version='1.0' ?>
<t:chloe xmlns:t="http://factorcode.org/chloe/1.0">
  <html>
    <body>
      <p>Hello!</p>
    </body>
  </html>
</t:chloe>

The chloe help goes into a lot of detail about how to use the template to substitute variable values, etc. I'll cover that in a later post.

The web framework has a lot of other features. Sessions, Authentication, Database, Validation, etc. Hopefully this will help you get started. All the example above can be entered and run (barring typo's and unescaped html entities) in a Factor listener. Download it and try it!

Update: There is a wiki page on the concatenative wiki with more information about the web framework.

Revisiting the Linear recursion combinator in Factor four years on

2008-10-11T04:09:00+13:00

The Joy programming language has a linrec combinator for performing linear recursion. Some time ago I took a stab at implementing linrec in Factor. Slava also posted his version that was originally a part of Factor.

As can be seen from Slava's version, juggling the stack when four quotations are involved can be problematic. I thought I'd revisit linrec using Factor's locals library and see how it compares. This is my attempt at an implementation using locals:

:: linrec 
 ( if-quot:    ( -- ? ) 
   then-quot:  ( -- ) 
   else1-quot: ( -- ) 
   else2-quot: ( -- ) 
   -- )
  if-quot call [ 
    then-quot call 
  ] [ 
    else1-quot call 
    if-quot then-quot else1-quot else2-quot linrec
    else2-quot call
  ] if ; inline recursive

This is pretty readable compared to the solutions in the prior posts. Because linrec is a combinator it needs to be declared 'inline'. It recursively calls itself so needs to be declared 'inline recursive' to enable calls of linrec to be compiled. More details about this can be found in the Factor documentation. Usage of the linrec word looks like:

5 [ dup 1 = ] [ ] [ dup 1- ] [ * ] linrec .
 => 120

[ 5 [ dup 1 = ] [ ] [ dup 1- ] [ * ] linrec ] infer.
 => ( -- object )

{ 1 2 3 } [ dup rest empty? ] [ first ] [ rest ] [ ] linrec
  => 3

[ { 1 2 3 } [ dup rest empty? ] [ first ] [ rest ] [ ] linrec ] infer.
  => ( -- object )

[ 1000 [ dup 1 = ] [ ] [ dup 1- ] [ * ] linrec drop ] time
 => 22ms

: fac ( n -- n ) dup 1 = [ dup 1- fac * ] unless ;
[ 1000 fac drop ] time
 => 19ms

It's nice that the 'fac' word and the linrec definition seem to compile down to the same code:

[ [ dup 1 = ] [ ] [ dup 1- ] [ * ] linrec ] optimized.
  => [
      \ ( gensym ) [
          dup 1 dupd eq?
          [ drop t ]
          [ dup 1 swap tag eq? [ 1 bignum= ] [ drop f ] if ] if
          [ ] [ dup 1 - ( gensym ) * ] if
      ] label
  ]

\ fac optimized.
  => [
      dup 1 dupd eq?
      [ drop t ]
      [ dup 1 swap tag eq? [ 1 bignum= ] [ drop f ] if ] if
      [ ] [ dup 1 - fac * ] if
  ]

One confusing aspect of this linrec definition is having to declare the stack effects of the quotations passed to linrec. Depending on the linear recursion task the stack effects may be different. Compare factorial vs finding the last element of a sequence:

{ [ dup 1 = ] 
  [ ] 
  [ dup 1- ] 
  [ * ] 
} [ infer. ] each
  => ( object -- object object )
     ( -- )
     ( object -- object object )
     ( object object -- object)

{ 
   [ dup rest empty? ] 
   [ first ] 
   [ rest ] 
   [ ] 
} [ infer. ] each
  => ( object -- object object )
     ( object -- object )
     ( object -- object )
 ( -- )

Both usages of linrec still infer and compile so I guess the fact that the stack effects balance out allows this. I'm not sure whether it's just a side effect of implementation or intended.

Working on this example I can say that, for me, Factor programming has become easier compared to four years ago.

Parsing JavaScript with Factor

2008-06-22T00:07:00+12:00

I've made some more changes to the Parsing Expression Grammar library in Factor. Most of the changes were inspired by things that OMeta can do. The grammar I used for testing is an OMeta-JS grammar for a subset of JavaScript. First the list of changes.

Actions in the EBNF syntax receive an AST (Abstract Syntax Tree) on the stack. The action quotation is expected to have stack effect ( ast -- ast ). It modifies the AST and leaves the new version on the stack. This led to code that looks like this:

expr = lhs '+' rhs => [[ first3 nip + ]]

Nothing wrong with that, but a later change added variables to the EBNF grammar to make it more readable:

expr = lhs:x '+' rhs:y => [[ drop x y + ]]

Code that uses variables a lot end up with a lot of 'drop' usage as the first word. I made a change recommended by Slava to have the action add the drop automatically depending on the stack effect of the action. So now this code works:

expr = lhs:x '+' rhs:y => [[ x y + ]]

So now if you use variables in a rule, the stack effect of the action should be ( -- ast ). If you don't, it should be ( ast -- ast ).

I added a way for one EBNF parser to call rules defined in another. This allows creating grammars which are hybrids of existing parsers. Or just to reuse common things like string handling expressions. These calls are called 'foreign' calls and appear on the right hand side of a rule in angle brackets. Here is a parser that parses strings between double quotation marks:

EBNF: parse-string
StringBody = (!('"') .)* 
String= '"' StringBody:b '"' => [[ b >string ]]
;EBNF

To call the 'String' rule from another parser:

EBNF: parse-two-strings
TwoStrings = <foreign parse-string String> <foreign parse-string String>
;EBNF

The <foreign> call in this example takes two arguments. The first is the name of an existing EBNF: defined parser. The second is the rule in that parser to invoke. It can also be used like this:

EBNF: parse-two-strings
TwoString = <foreign parse-string> <foreign parse-string>
;EBNF

If the first argument is the name of an EBNF: defined parser and no second argument is given, then the main rule of that parser is used. The main rule is the last rule in the parser body. A final way foreign can be used:

: a-token ( -- parser ) "a" token ;

EBNF: parse-abc
abc = <foreign a-token> 'b' 'c'
;EBNF

If the first argument given to foreign is not an EBNF: defined parser, it is assumed that it has stack effect ( -- parser ) and it will be called to return the parser to be used.

It is now possible to override the tokenizer in an EBNF defined parser. Usually the sequence to be parsed is an array of characters or a string. Terminals in a rule match successive characters in the array or string. For example:

EBNF: foo
rule = "++" "--"
;EBNF

This parser when run with the string "++--" or the array { CHAR: + CHAR: + CHAR: - CHAR: - } will succeed with an AST of { "++" --" }. If you want to add whitespace handling to the grammar you need to put it between the terminals:

EBNF: foo
space = (" " | "\r" | "\t" | "\n")
spaces = space* => [[ drop ignore ]]
rule = spaces "++" spaces "--" spaces
;EBNF

In a large grammar this gets tedious and makes the grammar hard to read. Instead you can write a rule to split the input sequence into tokens, and have the grammar operate on these tokens. This is how the previous example might look:

EBNF: foo
space = (" " | "\r" | "\t" | "\n")
spaces = space* => [[ drop ignore ]]
tokenizer = spaces ( "++" | "--" )
rule = "++" "--"
;EBNF

'tokenizer' is the name of a built in rule. Once defined it is called to retrieve the next complete token from the input sequence. So the first part of 'rule' is to try and match "++". It calls the tokenizer to get the next complete token. This ignores spaces until it finds a "++" or "--". It is as if the input sequence for the parser was actually { "++" "--" } instead of the string "++--". With the new tokenizer "...." sequences in the grammar are matched for equality against the token, rather than a string comparison against successive items in the sequence. This can be used to match an AST from a tokenizer:

TUPLE: ast-number value ;
TUPLE: ast-string value ;

EBNF: foo-tokenizer
space = (" " | "\r" | "\t" | "\n")
spaces = space* => [[ drop ignore ]]

number = [0-9]* => [[ >string string>number ast-number boa ]]
string = <foreign string-parser String> => [[ ast-string boa ]]
operator = ("+" | "-")

token = spaces ( number | string | operator )
tokens = tokenizer*
;EBNF

ENBF: foo
tokenizer = <foreign foo-tokenizer token>

number = . ?[ ast-number? ]? => [[ value>> ]]
string = . ?[ ast-string? ]? => [[ value>> ]]

rule = string:a number:b "+" number:c => [[ a b c + 2array ]]
;EBNF

In this example I split the tokenizer into a separate parser and use 'foreign' to call it from the main one. This allows testing of the tokenizer separately:

"123 456 +" foo-tokenizer ast>> .
=> { T{ ast-number f 123 } T{ ast-number f 456 } "+" }

The '.' EBNF production means match a single object in the source sequence. Usually this is a character. With the replacement tokenizer it is either a number object, a string object or a string containing the operator. Using a tokenizer in language grammars makes it easier to deal with whitespace. Defining tokenizers in this way has the advantage of the tokenizer and parser working in one pass. There is no tokenization occurring over the whole string followed by the parse of that result. It tokenizes as it needs too. You can even switch tokenizers multiple times during a grammar. Rules use the tokenizer that was defined lexically before the rule. This is usefull in the JavaScript grammar:

EBNF: javascript
tokenizer         = default 
nl                = "\r" "\n" | "\n"

tokenizer         = <foreign tokenize-javascript Tok>
...
End                = !(.)
Name               = . ?[ ast-name?   ]?   => [[ value>> ]] 
Number             = . ?[ ast-number? ]?   => [[ value>> ]]
String             = . ?[ ast-string? ]?   => [[ value>> ]]
RegExp             = . ?[ ast-regexp? ]?   => [[ value>> ]]
SpacesNoNl         = (!(nl) Space)* => [[ ignore ]]
Sc                 = SpacesNoNl (nl | &("}") | End)| ";"

Here the rule 'nl' is defined using the default tokenizer of sequential characters ('default' has the special meaning of the built in tokenizer). This is followed by using the JavaScript tokenizer for the remaining rules. This tokenizer strips out whitespace and newlines. Some rules in the grammar require checking for a newline. In particular the automatic semicolon insertion rule (managed by the 'Sc' rule here). If there is a newline, the semicolon can be optional in places.

"do" Stmt:s "while" "(" Expr:c ")" Sc    => [[ s c ast-do-while boa ]]

Even though the JavaScript tokenizer has removed the newlines, the 'nl' rule can be used to detect them since it is using the default tokenizer. This allows grammars to mix and match the tokenizer as required to make them more readable.

The JavaScript grammar is in the peg.javascript.parser vocabulary. The tokenizer is in peg.javascript.tokenizer. You can run it using the 'parse-javascript' word in peg.javascript:

USE: peg.javascript
"var a='hello'; alert(a);" parse-javascript ast>> pprint
T{ ast-begin f
  V{
      T{ ast-var f "a" T{ ast-string f "hello" } }
      T{ ast-call f
        T{ ast-get f "alert" } V{ T{ ast-get f "a" } } }
  }
}

The grammar is only for a subset of a JavaScript like language. It doesn't handle all of JavaScript yet. I'll continue to work on it as a testbed to improve EBNF. One thing I need to add next is decent error handling of a failed parse.

Collection of Factor Articles in PDF

2008-04-04T12:45:00+13:00

Factor has experienced some rapid change in the libraries and language over the past few years. I've written a few blog posts about Factor in the past and many of them suffer from bitrot due to this, making it hard to try out the examples in the latest Factor versions.

I've collected some of these articles, put them in a PDF document, and am slowly working through them so they are up to date with recent Factor versions. The document is split into sections for the articles that should work, and those that don't. Even the out of date articles make intersting reading for examples on how to use Factor.

There's nothing new in the document since it's a collection of things available from my blog posts, but having the central document makes it easier to update, print out, and read.

You can download it from factor-articles.pdf.

The git repository with the LaTeX source is hosted on github at http://github.com/doublec/factor-articles.

Comments and suggestions welcome.

Factor Parsing DSL

2008-04-04T00:24:00+13:00

I've been doing some experimenting with the emedded grammar code I wrote for Factor, trying to make it easier to use and a bit more useful for real world projects.

My inspiration for the changes has been seeing the kinds of things OMeta can do and the examples in the Steps towards the reinvention of programming from the Viewpoints Research Institute.

The original parsing expression grammar code (in the vocab 'peg') produced a data structure composed of Factor tuples that was interpreted at runtime via a call to the word 'parse'. It still has the data structure of tuples but now it can be compiled into Factor quotations, which are then compiled to native machine code via the Factor compiler. The 'parse' word calls 'compile' on the datastructure and calls it.

I created a parsing word that allows you to embed the peg expression directly in code, have it compiled to a quotation at parse time, and then called at runtime. Usage looks like:

"1+2" [EBNF expr=[0-9] '+' [0-9] EBNF] call

The older peg code had an <EBNF ... EBNF> embedded language and each rule in the language was translated to a Factor word. That's now changed to an EBNF: definition. An example:

EBNF: expr 
digit    = [0-9]            => [[ digit> ]]
number   = (digit)+         => [[ 10 digits>integer ]]
value    =   number 
           | ("(" exp ")")  => [[ second ]]

fac      =   fac:x "*" value:y  => [[ drop x y * ]]
           | fac:x "/" value:y  => [[ drop x y / ]]
           | number

exp      =   exp:x "+" fac:y    => [[ drop x y + ]]
           | exp:x "-" fac:y    => [[ drop x y - ]]
           | fac
;EBNF

This creates a word, 'expr', that runs the last rule in the embedded language (the 'exp' rule in this case) on the string passed to it:

"1+2*3+4" expr parse-result-ast .
 => 11

If you've used peg.ebnf before you'll see some new features in this code:

You can do character ranges using code like [a-zA-Z] to match upper and lowercase characters, etc.
Factor quotations can be embedded to process the results of choices. Anything between the [[ ... ]] will be run when that choice succeeds and the result put in the abstract syntax tree for that rule. The default AST produced by the rule is on the stack of the quotation. The example above drops this in some cases, and transforms it in others.
Rule elements can have variable names suffixed to it and these can be referenced in the action quotations. This is implemented using locals. This can be seen in EBNF code like this:

exp:x "+" exp:y => [[ drop x y + ]]

Usually that rule produces an AST consisting of a 3 element sequence, each element being the AST produced by the rules elements. The action quotation is transformed into:

[let* | x [ 0 over nth ] 
        y [ 2 over nth ] |
  drop x y + 
] with-locals

This is efficient and makes the grammar easier to read. * Another major new feature is grammars now handle direct and indirect left recursion. I implemented this from the VPRI paper Packrat Parsers Can Support Left Recursion. It makes converting existing grammars to peg grammars much easier. * Semantic actions have been added. These are like normal [[ ... ]] actions except they have stack effect ( ast -- bool ). Given an abstract syntax tree from the preceding element, it should return a boolean indicating whether the parse succeeded or not. For example:

odd= [0-9] ?[ digit> odd? ]?

Some of the syntax has changed. Previously { ... } was used for repeating zero or more times and [ ... ] was for optional. Now I use (...)*, (...)+, (...)?, for zero or more, one or more, and optional respectively. The dot (.) is used to match anything at that point. Terminals can be enclosed in single or double quotations.
There is a 'rule' word that can be used to get a single rule from a compiled grammar:

EBNF: foo number=([0-9])+ expr = number '+' number ;EXPR "1+2" foo parse-result-ast => { "1" "+" "2" } "1+2" "number" \ foo rule parse parse-result-ast => "1"

Notice the 'rule' word returns the parser object rather than the compiled quotation. This is useful for testing and further combining with other parsers.

These changes are in the main Factor repository. There is the peg.pl0 and peg.expr vocabs as examples. The peg.ebnf code is in an experimental state and is likely to change a lot until I'm satisfied with it so be aware that it might not be wise to use it in stable code unless you're happy with tracking the changes. I welcome feedback and ideas though.

One feature I'm currently working on but haven't put in the main repository yet is the ability to use the embedded grammar DSL to operate over Factor arrays and tuples. This allows writing an embedded grammar to produced an AST, and another embedded grammar to transform that AST into something else. Here's what code to transform an AST currently looks like:

TUPLE: plus lhs rhs ;

EBNF: adder
num   = . ?[ number? ]? 
plus  = . ?[ plus? ]? expr:a expr:b => [[ drop a b + ]]
expr  =  plus | num
;EBNF

T{ plus f 1 T{ plus f 2 3 } } adder parse-result-ast .

=> 6

This uses features I've already discussed, like semantic actions, to detect the type of the object. The difference is that the parser produced by EBNF: operates not on strings, but on an abstract sequence type that is implemented for strings, sequence, and tuples. I'm still playing around with ideas for this but I think it'll be a useful way to reuse grammars and string them together to perform tasks.

Embedded Grammars in Factor

2007-11-28T23:11:00+13:00

I'm working on a new parser combinator library for Factor based on Parsing Expression Grammars and Packrat parsers. This is based on what I learnt from writing a packrat parser in Javascript.

It's progressing quite well and already fixes some problems in the existing parser combinator library I wrote. The main issue with that one is it's not tail recursive and some combinations of parsers can run out of call stack.

The new library is in the 'peg' vocabulary. I haven't yet implemented the packrat side of things though so it is slow on large grammars and inputs.

I've also done a proof of concept of something I've been meaning to do for awhile. That is writing a parser for EBNF (or similar) that produces Factor code in the form of parser combinators to implement the described grammar. The code for this is in the 'peg.ebnf' vocabulary. It allows you to embed an EBNF-like language and have Factor words generated for each rule:

<EBNF
digit  = '1' | '2' | '3' | '4' .
number = digit { digit } .
expr   = number {('+' | '-' ) number } .
EBNF>

This example would create three Factor words. 'digit', 'number' and 'expr'. These words return parsers that can be used as normal:

"123" number parse
"1" digit parse
"1+243+342" expr parse

The EBNF accepted allows for choice, zero or more repetition, optional (exactly 0 or 1), and grouping. The generated AST is pretty ugly so by default it works best as a syntax checker. You can modify the generated AST with action productions:

<EBNF
digit  = '1' | '2' | '3' | '4' => convert-to-digit .
number = digit { digit }       => convert-to-number .
expr   = number '+' number     => convert-to-expr .
EBNF>

An action is a factor word after the '=>'. The word receives the AST produced from the rule on the stack and it can replace that with a new value that will be used in the AST. So 'convert-to-expr' above might produce a tuple holding the expression values (by default, a sequence of terms in the rule are stored in a vector):

TUPLE: ast-expr lhs operator rhs ;
C: <ast-expr> ast-expr
: convert-to-expr ( old -- new )
  first3 <ast-expr> ;

The generated code is currently pretty ugly, mainly due to it being a quick proof of concept. I'll try doing a few grammars and tidy it up, changing the interface if needed, as I go along.

As an experiment I did a grammar for the PL/0 programming language. It's in 'peg.pl0'. The grammar from the wikipedia article is:

program = block "." .

block = [ "const" ident "=" number {"," ident "=" number} ";"]
        [ "var" ident {"," ident} ";"]
        { "procedure" ident ";" block ";" } statement .

statement = [ ident ":=" expression | "call" ident |
            "begin" statement {";" statement } "end" |
            "if" condition "then" statement |
            "while" condition "do" statement ].

condition = "odd" expression |
            expression ("="|"#"|"<"|"<="|">"|">=") expression .

expression = [ "+"|"-"] term { ("+"|"-") term}.

term = factor {("*"|"/") factor}.

factor = ident | number | "(" expression ")".</pre>The Factor grammar is very similar:<pre>: ident ( -- parser )
  CHAR: a CHAR: z range 
  CHAR: A CHAR: Z range 2array choice repeat1 
  [ >string ] action ;

: number ( -- parser )
  CHAR: 0 CHAR: 9 range repeat1 [ string>number ] action ;

<EBNF
program = block '.' .
block = [ 'const' ident '=' number { ',' ident '=' number } ';' ]
        [ 'var' ident { ',' ident } ';' ]
        { 'procedure' ident ';' [ block ';' ] } statement .
statement = [ ident ':=' expression | 'call' ident |
              'begin' statement {';' statement } 'end' |
              'if' condition 'then' statement |
              'while' condition 'do' statement ] .
condition = 'odd' expression |
            expression ('=' | '#' | '<=' | '<' | '>=' | '>') expression .
expression = ['+' | '-'] term {('+' | '-') term } .
term = factor {('*' | '/') factor } .
factor = ident | number | '(' expression ')'
EBNF>

This grammar as defined works and can parse PL/0 programs. I'll extend this as I improve the EBNF routines, adding actions, etc to generated a decent AST.

Writing Web Applications in Factor

2007-11-21T16:33:00+13:00

There was a request on #concatenative for information on how to write web applications in Factor. I went through a few steps on how to get started. I'm repeating it here for others that might be interested.

There are a number of different ways of writing web applications in Factor but for this approach I'm using the furnace framework.

The first step is to start the web server. This lives in the vocab 'http.server':

USE: http.server
[ 8888 httpd ] in-thread

This will start an instance of the server on port 8888 in another thread, to allow us to continue to enter commands in the listener.

By default web applications are accessed on the URL path /responder/name, where 'name' is the name of the web application.

Accessing the web application path runs an 'action'. An action produces HTML output which gets sent back to the client browser. A web application has a default 'action' that gets run (the equivalent of an index.html), and can have other actions that are specified in the URL. Some examples:

http://localhost:8888/responder/foo: Runs the default action for the 'foo' web application
http://localhost:8888/responder/foo/doit: Runs the 'doit' action
http://localhost:8888/responder/foo/hello?name=chris: Runs the 'hello' action giving the argument 'name' with the value 'chris'

The syntax for furnace URL's is therefore http://servername:port/responder/{webappname}/{action}?{arguments}

Furnace web application must exist under the 'webapps' vocabulary. So accessing /responder/foo will look for furnace details in the vocabulary 'webapps.foo'.

A furnace web application is registered with the http server using the 'web-app' word. It takes three arguments on the stack:

\ web-app effect-in . 
=> { "name" "default" "path" }

The 'name' is the vocabulary name of the web application with out the 'webapps.' prefix. 'default' is the name of the action that gets run when the web application URL is accessed. 'path' is the location of any template files the web application uses.

An action is a word that outputs data to be sent to the browser. It can be as simple as:

: doit ( -- ) serving-text "I am here" print ;

The word must be registered as an action:

\ doit { } define-action

Now accessing the URL for the web application with 'doit' at the end of the path will result in 'I am here' being sent to the browser. Note the 'serving-text' call. That outputs the headers for the mime type and the standard HTTP response. There is also a 'serving-html', or you could write the headers manually.

Actions can take arguments. These are placed on the stack for the word that is called:

: hello ( name -- ) serving-text "Hello " write print ;
\ hello { { "hello" } } define-action

So the complete code for the simplest of web applications is:

USE: http.server
[ 8888 httpd ] in-thread
IN: webapps.test
USE: furnace

: index serving-text "We're alive!" print ;
\ index { } define-action 

: hello ( name -- ) serving-text "Hello " write print ;
\ hello { { "name" } } define-action

"test" "index" "." web-app

Accessing http://localhost:8888/responder/test will run the 'index' action. This is what we passed as the 'default' parameter on the stack to the 'web-app' word. Accessing http://localhost:8888/responder/test/hello?name=chris will run the 'hello' action.

There is also the facility to have template files, very much like JSP. The 'path' parameter to 'web-app' defines the location of these. Inside your action word you can call 'render-template' to run the template and have it sent to the browser:

: runme ( -- ) f "page" "Title" render-template ;
\ runme { } define-action

This will load the 'page.furnace' file in the path given to 'web-app'. It should contain standard HTML with embedded Factor code inside <% and %> tags. It will be run and sent to the client. The 'f' passed in this example can be an instance of a tuple (an object) and the template can access the slots of that instance to display data, etc.

There is quite a bit more that can be done. There is a continuation based workflow system, validators for actions, etc. There is also much more that needs to be done. handling sessions, cookies, etc. Hopefully this post gives a quick introduction and allows you to get started.

How to publish a Git repository

2007-09-22T22:44:00+12:00

This post is aimed mainly at Factor developers who need to make their repository accessible to Slava to retrieve patches. Factor has recently moved to using git as the version control system.

To set up a repository on a server you should clone the existing Factor repository using the '--bare' option:

git clone --bare http://www.factorcode.org/git/factor.git factor.git

A bare repository is one without a checked out working copy of the code. It only contains the git database. As a general rule you should never push into a repository that contains changes in the working copy. To ensure this doesn't happen, we're making the server repository a bare repository - it has no working copy.

Copy the 'factor.git' directory onto your server. I put it in '/git/factor.git'. Now if you have changes on your local machine that you want to push to your repository you can use something like:

git push yourname@yourserver.com:/git/factor.git

If you want to push changes from a specific branch in your local repository:

git push yourname@yourserver.com:/git/factor.git mybranch:master

To publish the remote repository you have two options. You can publish via the HTTP protocol, or via the git protocol. The first is slower but usable by people behind restrictive firewalls, while the second is more efficient but requires an open port. I suggest doing both.

To publish via HTTP, you must make the file 'hooks/post-update' executable:

chmod +x /git/factor.git/hooks/post-update

This gets executed whenever something is pushed to the repository. It runs a command 'git-update-server-info' which updates some files that makes the HTTP retrieval work. You should also run this once manually:

cd /git/factor.git
git-update-server-info

Now make the /git directory published via your webserver (I symbolic link to it in the server's doc-root). People can pull from the repository with:

git pull http://yourserver.com/git/factor.git

To set up the git protocol you need to run the 'git-daemon' command. You pass it a directory which is the root of your git repositories. It will make public all git repositories underneath that root that have the file 'git-daemon-export-ok' in it. So first create this file:

touch /git/factor.git/git-daemon-export-ok

Run the daemon with:

git-daemon --verbose /git

The '--verbose' will give you output showing the results of connecting to it. I run this from within a screen session. You can set it up to automatically run using whatever features your server OS has. Now people can retrieve via the git protocol:

git pull git://yourserver.com/git/factor.git

My repository is accessible from both protocols:

git clone http://www.double.co.nz/git/factor.git
git clone git://double.co.nz/git/factor.git

You can also browse it using gitweb.

Distributed Channels in Factor

2007-09-12T12:58:00+12:00

Following on from my Channels implementation, I've now added 'Remote Channels'. These are distributed channels that allow you to access channels in separate Factor instances, even on different machines on the network. It's based on my Distributed Concurrency work.

A channel can be made accessible by remote Factor nodes using the 'publish' word. Given a channel this will return a long Id value that can be used by remote nodes to use the channel. For example:

<channel> [ sieve ] spawn drop publish .
 => "ID12345678901234567890....."

From a remote node you can create a <remote-channel> which contains the hostname and port of the node containing the channel, and the Id of that channel:

"foo.com" 9000 <node> "ID1234..." <remote-channel>

You can use 'from' and 'to' on the remote channel exactly as you can on normal channels. The data is marshalled over the network using the serialization library. Remote channels are implemented using distributed concurrency so you must start a node on the Factor instance you are using. This is done with 'start-node' giving the hostname and port:

"foo.com" 9000 start-node

Once this is done all published channels become available. Note that the hostname and port must be accessible by the remote machine so it can connect to send the data you request.

As an experiment I published the prime number sieve example mentioned in my last post. It's running on one of my servers. To make it easy to create a <remote-channel> without needing to know the hostname and port I serialized the <remote-channel> instance, saved it in a file and made it available as [...server down sorry...].

You can load this file into Factor, deserialize it and get the <remote-channel> instance. You can then call 'from' on it to get the next prime number in the series. Until they get so big that my Factor instance is DOS'd of course! The code to do this is:

USING: serialization http.client 
       channels.remote concurrency.distributed ;

"yourhostname-or-ip-address.com" 9000 start-server
"[server-down-sorry]/prime.ser" http-get-stream 2nip 
[ deserialize ] with-stream
dup from .
dup from .
...etc...

The '9000' can be any port number openly accessible on your machine. A current Factor bug means you may get an error in 'start-server' about an address already assigned if you run Linux. This is due to an interaction with ipv6 - you can ignore it, the server will start fine. 'start-server' needs to be run whenever you start our Factor instance.

The 'dup from .' duplicates the <remote-channel>, gets the next number from it and prints it. It may not be in sequence as other users may have gotten the next number before you.

There is a lot of room for improvement and additions to the code. Feel free to hack at it and send in patches. Let me know some ideas on how this could be used in 'real world' applications.

Concurrent Channels for Factor

2007-09-05T10:28:00+12:00

Rob Pike gave a talk at Google about the NewSqueak programming language, and specifically how it's concurrency features work. NewSqueak uses channels and is based on the concepts of Communicating Sequential Processes.

To play around with some of the ideas in the video I created a quick implementation of channels for Factor and converted a couple of the NewSqueak examples.

The <channel> word creates a channel that threads can send data to and receive data from. The 'to' word sends data to a channel. It is a synchronous send and blocks waiting for a receiver if there is none. The 'from' word receives data from a channel. It will block if there is no sender.

There can be multiple senders waiting, and if a thread then receives on the channel, a random sender will be released to send its data. There can also be multiple receivers blocking. A random one is selected to receive the data when a sender becomes available.

I'm not sure if I've got the best stack effect ordering for the words but it gives something to experiment with and work out the best interface. I've not yet done a 'select' or 'mux' words.

Here is the 'counter' example ported from NewSqueak:

: (counter) ( channel n -- )
    [ swap to ] 2keep 1+ (counter) ;

: counter ( channel -- )
    2 (counter) ;    

: counter-test ( -- n1 n2 n3 )
    <channel> [ counter ] spawn drop 
    [ from ] keep [ from ] keep from ;

Given a channel, the 'counter' word will send numbers to it, incrementing them by one, starting from two. 'counter-test' creates a channel, spawns a process to run 'counter', and then receives a few values from the channel.

It'll be interesting to compare how well CSP works in a concatenative language vs the message passing concurrency library I implemented previously. The CSP implementation is simple enough that I should be able to get it going in the Factor->Javascript system fairly easily. That would give CSP on the browser.