Bluish Coder

Programming Languages, Martials Arts and Computers. The Weblog of Chris Double.


2016-05-04

Bang, Hat and Arrow in Pony

If you've looked at Pony programming language code you've probably seen use of punctuation symbols in places and wondered what they meant. This post is an attempt to explain three of those - the bang, hat and arrow (!, ^ and -> respectively). Note that this is my understanding based on usage, reading the tutorials and watching videos so there may be errors. I welcome correction!

Bang

The bang symbol (otherwise known as an exclamation mark) combined with a type name can be thought of as the type of an alias of the given type. Having an alias of an object means having another reference to that object. So an alias to a String iso is of type String iso!. This matters mostly in generic code which will be explained later but it does come up in error messages.

If you see ! in an error message like "iso! is not a subtype of iso" this means you are probably trying to assign an object that cannot be aliased without first consuming it.

If you see ! in a type declaration in code like "let foo: A!" then you can read this as "replace A! with a type that can safely hold an alias to A". If A is a String iso then A! would be a String trn for example (following the rules for aliased substitution.

Bang in errors

The following code demonstrates something that is often encountered by first time Pony users:

class Something

actor Main
  new create(env: Env) =>
    let a = recover iso Something end
    Foo(a)

actor Foo
  new create(s: Something iso) =>
    None

Here we have a class called Something. A new instance of it is created in the Main actor with reference capability iso. A new Foo actor is created passing this instance to it. This will fail to compile as we are aliasing the Something object held in a. a holds a reference to it and the variable s holding the argument to the Foo constructor is holding a reference to it at the same time. Objects with a reference capability of iso cannot have more than one reference to it. The error from the compiler will look like:

Error:
e1/main.pony:6:9: argument not a subtype of parameter
    Foo(a)
        ^
    Info:
    e1/main.pony:9:14: parameter type: Something iso
      new create(s: Something iso) =>
                 ^
    e1/main.pony:6:9: argument type: Something iso!
        Foo(a)
            ^
    e1/main.pony:1:1: Something iso! is not a subtype of Something iso: iso! is not a subtype of iso

This error states that the expected type of the parameter for the Foo constructor is of type Something iso but the type that we passed is a Something iso!. It further explains things by noting that Something iso! is not a subtype of Something iso because iso! is not a subtype of iso.

Armed with the knowledge that the bang symbol means the type for an alias this can be read as the argument passed was an alias to a Something iso. This is an error as iso cannot be aliased - this is what iso! is not a subtype of iso means. The subtyping relationship for aliases is outlined in the Capability Subtyping section of the tutorial.

The code can be fixed by consuming the a so it is no longer aliased:

let a = recover iso Something end
Foo(consume a)

Bang in generics

The other place where you'll see the alias type is in generic code. The following non-generic code compiles fine:

class Something
  let a: U8

  new create(x: U8) =>
    a = x

actor Main
  new create(env: Env) =>
    let aint = Something(42)

U8 defaults to val reference capability which can be aliased. This allows the assignment to the field a in Something which is aliasing the x object. If we make this a generic so that any type can be used then it fails to compile:

class Something[A]
  let a: A

  new create(x: A) =>
    a = x

actor Main
  new create(env: Env) =>
    let aint = Something[U8](42)

The error is:

Error:
e3/main.pony:5:7: right side must be a subtype of left side
    a = x
      ^
    Info:
    e3/main.pony:4:17: right side type: A #any !
      new create(x: A) =>
                    ^
    e3/main.pony:5:5: left side type: A #any
        a = x
        ^
    e3/main.pony:4:17: A #any ! is not a subtype of A #any: the subtype has no constraint
      new create(x: A) =>
                ^

For now, ignore the #any in the error message. I'll expand on this later but it's informing us that the type A is unconstrained and can have any reference capability.

The error states that x is an A! but a is an A and A! is not a subtype of A so the assignment cannot happen.

This occurs Because A is unconstrained. It can be any reference capability. Therefore the code must be able to be compiled under the assumption that the most restrictive reference capability can be used. It works fine with val, which can be aliased, but not with iso which cannot. Therefore the generic code cannot be compiled. You can see how iso would fail by expanding a version using String iso:

class Something
  let a: String  iso

  new create(x: String iso) =>
    a = x

actor Main
  new create(env: Env) =>
    let aint = Something(recover iso String end)

The error is:

Error:
e5/main.pony:5:7: right side must be a subtype of left side
    a = x
      ^
    Info:
    e5/main.pony:4:17: right side type: String iso!
      new create(x: String iso) =>
                    ^
    e5/main.pony:2:10: left side type: String iso
      let a: String  iso
             ^
    e5/main.pony:4:17: String iso! is not a subtype of String iso: iso! is not a subtype of iso
      new create(x: String iso) =>
                    ^

This is the same error that the generic code is giving us. The generic code can be fixed in a few ways. The first is to constrain the type so that it is a specific reference capability that works. Here it is changed to val:

class Something[A: Any val]
  let a: A

  new create(x: A) =>
    a = x

actor Main
  new create(env: Env) =>
    let aint = Something[U8](42)

The A: Any val syntax constrains the type parameter to be a subtype of the type after the :. In this case, any type with a reference capability of val. This won't work if you want to be able to use any aliasable type (eg ref as well as val):

class Something[A: Any val]
  let a: A

  new create(x: A) =>
    a = x

actor Main
  new create(env: Env) =>
    let aint = Something[U8](42)
    let bint = Something[String ref](recover ref String end)

The error here is obvious in that we are trying to pass a ref parameter to a function expecting a val. Pony generics solves this by allowing code to be polymorphic over the reference capability. There are specific annotations for classes of reference capabilities. They are:

#read  = { ref, val, box }                = Anything you can read from
#send  = { iso, val, tag }                = Anything you can send to an actor
#share = { val, tag }                     = Anything you can send to more than one actor
#any   = { iso, trn, ref, val, box, tag } = Default of a constraint
#alias = {ref,val, box, tag}              = Set of capabilities that alias as themselves (used by compiler)

A version that will work for ref, val and box becomes:

class Something[A: Any #read]
  let a: A

  new create(x: A) =>
    a = x

actor Main
  new create(env: Env) =>
    let aint = Something[U8](42)
    let bint = Something[String ref](recover ref String end)

But what if you want it to work with non-aliasable types like iso? A solution is to consume the parameter:

class Something[A]
  let a: A

  new create(x: A) =>
    a = consume x

actor Main
  new create(env: Env) =>
    let aint = Something[U8](42)
    let bint = Something[String ref](recover ref String end)
    let cint = Something[String iso](recover iso String end)

Another solution is to declare the field type to be A! instead of A. In the String iso case using A means String iso which cannot hold an alias. Using A! means String iso! which should be read as "a type that can safely alias a String iso". Looking at the Aliased substitution table this is a tag:

class Something[A]
  let a: A!

  new create(x: A) =>
    a = x

actor Main
  new create(env: Env) =>
    let aint = Something[U8](42)
    let bint = Something[String ref](recover ref String end)
    let cint = Something[String iso](recover iso String end)

In this case we are using ! to tell the compiler to use a reference capability that works for whatever the type of A is. An iso becomes a tag, a trn becomes a box, a ref stays a ref, etc.

Hat

The hat symbol (or ^) is an ephemeral type. It's the type of an object that is not assigned to a variable. consume x is used to prevent aliasing of x but at the point of being consumed and before it is assigned to anything else, what type is it? If x is type A then the type of consume x is A^. Constructors always return an ephemeral type as they create objects and return them but they aren't yet assigned to anything.

The following example creates a Box type that acts like single instance array of String iso objects. A value can be stored and updated. A utility function Foo.doit takes a String iso as an argument. It's stubbed out since it doesn't need to do anything for the example. The main code creates a Box, updates it, and calls the utility function on it.

class Box
  var a: String iso

  new create(x: String iso) =>
    a = consume x

  fun ref update(x: String iso): String iso =>
    let b = a = consume x
    consume b

primitive Foo
  fun doit(s: String iso) =>
    None

actor Main
  new create(env: Env) =>
    let a = Box(recover iso String end)
    let b = a.update(recover iso String end)
    Foo.doit(consume b)

Some things to note based on prior discussion. The create method consumes the argument to prevent aliasing. The update function also consumes the argument to prevent aliasing. It uses the destructive read syntax to assign the argument x to the field a and assign the old value of a to b to avoid aliasing. Unfortunately this example fails to compile:

Error:
f3/main.pony:19:14: argument not a subtype of parameter
    Foo.doit(consume b)
             ^
    Info:
    f3/main.pony:12:12: parameter type: String iso
      fun doit(s: String iso) =>
               ^
    f3/main.pony:19:14: argument type: String iso!
        Foo.doit(consume b)
                 ^
    f3/main.pony:7:34: String iso! is not a subtype of String iso: iso! is not a subtype of iso
      fun ref update(x: String iso): String iso =>
                                     ^

From the discussion previously on ! this error tells us that we are aliasing b. We can narrow it down by explicitly declaring the type of b:

let b: String iso = a.update(recover iso String end)

The error is due to update returning a String iso. We are consuming b and returning it as a String iso which then gets aliased when assigned to b in the main routine. Changing the return type to use hat resolves the issue. consume b returns the ephmeral type which is an object with no variable referencing it. It is safe to assign to a String iso so the change compiles:

class Box
  var a: String iso

  new create(x: String iso) =>
    a = consume x

  fun ref update(x: String iso): String iso^ =>
    let b = a = consume x
    consume b

primitive Foo
  fun doit(s: String iso) =>
    None

actor Main
  new create(env: Env) =>
    let a = Box(recover iso String end)
    let b = a.update(recover iso String end)
    Foo.doit(consume b)

Another approach would be to return a String iso but change doit to be a String tag (the type that can alias a String iso). This compiles but because doit now takes String tag it is limited in what it can do with the string. The approach of using an ephemeral type allows obtaining the mutable object from the Box.

Hat in parameters

Sometimes you'll see hat in parameter lists. The Array builtin has an init constructor that looks like:

new init(from: A^, len: USize)

This initializes an array so that all elements are the from value. To explore how this works, here's a smaller example that does something similar:

class Box[A]
  var a: A
  var b: A

  new create(x: A^) =>
    a = x
    b = x

Without the hat in A^ there is an error due to aliasing. We can't assign x to both a and b in case A is an iso. With the hat it compiles. The is because an epehemeral reference capability is a way of saying "a reference capability that, when aliased, results in the base reference capability". So a String iso^ can be assigned to a String iso, a String ref^ can be assigned to a String ref, etc. This means the generic class itself compiles but using it for a String iso will fail due to aliasing but it can be used for other reference capability types. Compare this to a plain x: A where the generic class itself won't compile since a String iso can't be assigned to another String iso due to aliasing.

Code demonstrating this is:

actor Main
  new create(env: Env) =>
    let a = Box[String iso](recover iso String end)
    let c = Box[String ref](recover ref String end)
    let e = Box[String val](recover val String end)

Arrow

The arrow syntax (or ->) is known as viewpoint adapter types and is related to viewpoint adaption.

Arrow in error messages

Viewpoint adaption defines what the reference capability of a field looks like to some caller based on the reference capability of the object the field is being read from. This is important to maintain the reference capability guarantees. A val object should not be able to access an iso field as iso or it breaks the constraints of val - it should be immutable but obtaining it as iso allows mutation of the field. There is a table in viewpoint adaption that shows what the mapping is.

An example of an error that can occur by ignoring viewpoint adaption is in the following code:

class Something
  var a: String iso

  new create() =>
    a = recover iso String end

  fun doit(s: String) =>
    a.append(s) 

actor Main
  new create(env: Env) =>
    let a = Something
    a.doit("hello")

The error here is calling append on a the a field in the doit method. By default methods have a receiver reference capability of box. Anything that happens inside the method cannot affect the state of the object. This is why you see methods that modify object fields start with fun ref - it's to change the receiver reference capability to something mutable. Even though the field a is iso and therefore mutable because we are inside a box method it appears as a non-mutable reference capability. The viewpoint adaption table shows that a box origin with an iso field gives a tag type. So a looks like a String tag within the method. The compiler gives:

Error:
v/main.pony:8:13: receiver type is not a subtype of target type
    a.append(s) 
            ^
    Info:
    v/main.pony:8:5: receiver type: this->String iso!
        a.append(s) 
        ^
    ny/ponyc/packages/builtin/string.pony:622:3: target type: String ref
      fun ref append(seq: ReadSeq[U8], offset: USize = 0, len: USize = -1)
      ^
    v/main.pony:2:10: String tag is not a subtype of String ref: tag is not a subtype of ref
      var a: String iso
             ^

The 'receiver type' of this->String iso! is an example of arrow usage. It's saying that a object of type String iso! (an alias to a String iso) as seen by an origin of this. The reference capability of this in a method is that of the receiver reference capability on the function - in this case box. So this->String iso! is String tag. That's why the last error description line refers to String tag.

The solution here is to change the reference capability for the method to something that allows mutation:

fun ref doit(s: String) =>

Arrow in type declarations

When writing generic code it's sometimes required to be explicit in what viewpoint adaption to use for generic types. Returning to the Box example used previously we'll make it generic and make it usable for any reference capability:

class Box[A]
  var a: A

  new create(x: A) =>
    a = consume x

  fun apply(): this->A! =>
    a

  fun ref update(x: A): A^ =>
    let b = a = consume x
    consume b

  fun clone(): Box[this->A!] =>
    Box[this->A!](a)

Notice the use of this->A! in the return type of apply. We want to return what is held in the Box. If it is a Box[String val] val then we can return a String val since it is immutable and the box is immutable. If it is a Box[String ref] val we still want to return a String val, not a String ref. The latter would allow modifying an immutable box. If it's a Box[String ref] ref then it's safe to return a String ref. This is what the arrow type handles for us. The this refers to the reference capability of the object. The A! refers to the field type - note that it is being aliased here so we want a type that can hold an alias to an A. The viewpoint adaption gives the resulting reference capability of the type.

Looking up the table of viewpoint adaption gives:

Box[String val] val => val->val => val => String val
Box[String ref] val => val->ref => val => String val
Box[String ref] ref => ref->ref => ref => String ref
Box[String iso] ref => ref->iso => iso => String iso
Box[String ref] iso => iso->ref => tag => String tag

That last one is interesting in that the Box[String ref] iso says that only one reference of the Box can exist. If we allow a String ref to be obtained from it then it breaks this condition since both the original reference to the box can modify the string and so can the returned reference. This is why the viewpoint adaption gives a String tag. A tag only allows identity operations so it's safe to have this type of alias of an iso.

Note that the table above gives the mapping for this->A. Because it's a this->A! it has to be a type that can hold an alias to the type of the table. So we have another mapping:

String val! => String val
String ref! => String ref
String iso! => String tag

In this way a Box[String iso] ref will give out a String tag - the only safe way of aliasing the original string in the box.

The other use of an arrow type in this example is in the clone function. This must do a shallow copy of the object. It returns a new Box holding a reference to the same value. Because we need to alias the value the same constraints as described for the apply method exist. We want to return a Box[this->A!] to ensure the value object for that box instance is a safe alias to the original. For a Box[String iso] ref this returns a Box[String tag] for example.

The following code can be used with the Box class above to test it:

primitive Foo
  fun doit(s: String tag) =>
    None

actor Main
  new create(env: Env) =>
    let a = Box[String iso](recover iso String end)
    let b = a.clone()
    Foo.doit(b())

    let c = Box[String ref](recover ref String end)
    let d = c.clone()
    Foo.doit(d())

    let e = Box[String val](recover val String end)
    let f = e.clone()
    Foo.doit(f())

A->B arrows

Arrow types don't need to always use this on the receiver side. They can use an explicit reference capability like box->A or they can use another parameterized type. Examples of this are in some of the library code:

class ArrayValues[A, B: Array[A] #read] is Iterator[B->A]

An ArrayValues is returned by the values method on Array. It's an iterator over the objects in the array. The B->A syntax means that the type of the generic argument to Iterator is of type "A as seen by B" using viewpoint adaption. It's not an iterator over A, it's an iterator over "A as seen by B". This allows iteration over arrays whether they are val or ref and produces a compatible type for the Iterator that works with both.

Conclusion

Most of the functionality described here is some of the more esotoric Pony functionality. It is mainly hit when using generics. The best current reference for generics is a video by Sylvan Clebsch for the virtual Pony users group - Writing Generic Code.

A good way to learn is to try some of the examples in this post and play around with them. Try aliasing, using different types, different reference capabilities and see what happens. The Pony library code, Array.pony for example, is a useful reference.

Tags: pony 

2016-03-15

Closures in Pony

The Pony programming language has support for closures but the documentation hasn't caught up with changes to the implementation. This post describes some of the features of Pony closures with examples. These examples require the latest version of Pony from the github repository. I used commit e4f6f91d.

Lambda

Closures are functions combined with an environment mapping names to values. Pony has a lambda keyword that can be used to create them. It's possible to create a function that captures from the lexical scope and to create a function with no capturing at all.

The following example shows using a lambda that does not capture anything (note that this is an artificial example - the List type already has various mapping and fold functions on it):

use "collections"

actor Main
  new create(env:Env) =>
    let l = List[U32]
    l.push(10).push(20).push(30).push(40)
    let r = reduce(l, 0, lambda(a:U32,b:U32): U32 => a + b end)
    env.out.print("Result: " + r.string())

  fun reduce(l: List[U32], acc: U32, f: {(U32, U32): U32} val): U32 =>
    try
      let acc' = f(acc, l.shift())
      reduce(l, acc', f)
    else
      acc
    end

The reduce function takes an anonymous function as a parameter, named f. This is is defined as {(U32, U32): U32}. This syntax means that it must take two U32 arguments and returns a U32. A U32 is a 32-bit unsigned integer.

The val is a reference capability annotation. It states that f is an immutable value. Since nothing inside f can be mutated it is safe for f to be shared with other actors. Lambda functions that do not close over other variables default to val as they cannot change anything outside of the function. We need to explicitly annotate the val as {...} declarations default to ref. A compile error would result without it due to passing a val object to a ref parameter.

Compiling and running produces the result 100 as expected:

$ ponyc example
$ ...
$ ./example1
  Result: 100

Lambda with environment

The following example closes over the env variable so that it can be accessed within the lambda to print the String contained in the List:

use "collections"

actor Main
  new create(env:Env) =>
    let l = List[String]
    l.push("hello").push("world")
    for_each(l, lambda(s:String)(env) => env.out.print(s) end)

  fun for_each(l: List[String], f: {(String)}) =>
    try
      f(l.shift())
      for_each(l, f)
    end

Notice that the lambda has an additional set of parameters, the (env). This is an argument list of the variables that are being closed over. Variables listed there can be accessed from the body of the lambda. Pony requires you to be explicit about what gets captured vs many other languages that implicitly capture variables in scope. Variables can be renamed in this parameter list using the following syntax:

for_each(l, lambda(s:String)(myenv=env) => myenv.out.print(s) end)

The lambda here returns None which is the default so it is left off the declaration {(String)}. This declaration requires a function that takes a single String argument and returns None. The return value is also left off the actual lambda expression. No reference capability annotations are required here as {...} defaults to ref and a lambda that closes over variables also defaults to ref.

Lambda modifying closed variables

A modification of the above example might be to keep a count of each time the lambda is called and display the count:

actor Main
  new create(env:Env) =>
    let l = List[String]
    l.push("hello").push("world")
    var count = U32(0)
    for_each(l, lambda ref(s:String)(env,count) =>
                    env.out.print(s)
                    count = count + 1
                end)
    // Displays '0' as the count
    env.out.print("Count: " + count.string())

fun for_each(l: List[String], f: {ref(String)}) =>
  try
    f(l.shift())
    for_each(l, f)
  end

The main thing to note here is that both the lambda definition and the declaration have a ref prefixed to the argument list. This signifies a lambda that might mutate itself when called. The effect of this is that within the body of the lambda the receiver of method calls and field access (ie. the this of the lambda object) is a ref vs the default of box. A box receiver won't allow ref functions to be called and mutating requires a ref receiver.

There is an issue with this example as noted by the comment. The final count is displayed as zero. The assignment of count within the lambda changes the count field within the lambda object. The count within the lambda increments and could be displayed showing the increasing values. The count outside of this does not. There is no reference or pointer to the closed over variable. Indirection can be used to do this by using a one element array:

var count : Array[U32] = Array[U32].init(0, 1)
for_each(l, lambda ref(s:String)(env,count) =>
              try
                env.out.print(s)
                count(0) = count(0) + 1
              end
            end)
try
  env.out.print("Count: " + count(0).string())
end

The array access is wrapped with try as it is a partial function and can fail. Another approach could be to move the act of counting into an Actor:

use "collections"

actor Counter
  var n: U32 = 0

  be increment() =>
    n = n + 1

  be print(env:Env) =>
    env.out.print("Count: " + n.string())

actor Main
  new create(env:Env) =>
    let l = List[String]
    l.push("hello").push("world")
    let counter = Counter
    for_each(l, lambda(s:String)(env,counter) =>
                  env.out.print(s)
                  counter.increment()
                end)
    counter.print(env)

  fun for_each(l: List[String], f: {(String)}) =>
    try
      f(l.shift())
      for_each(l, f)
    end

Actors have the tag reference capability and are easier to pass around to other objects, including lambda.

The example here is contrived in that it can be done without modifying a captured variable. Sylvan produced the following approach, using the methods in the collections package:

use "collections"

actor Main
  new create(env:Env) =>
    let l = List[String]
    l.push("hello").push("world")

    let count = l.fold[U32](
      lambda(x: U32, s: String)(env): U32 =>
        env.out.print(s)
        x + 1
      end, 0)

    env.out.print("Count: " + count.string())

Object Literals

Object literals are a way to create objects inline without having to name a class or actor. They look like:

object
  let x = 10
  fun foo() =>
    ...do something...
end

Lambda's are actually syntactic sugar on top of object literals. A lambda expansion looks like:

// Lambda
lambda (s:String)(env) => env.out.print(s) end

// Expands to:
object
  var env:Env = env
  fun apply(s:String) =>
    env.out.print(s)
end

The apply method of an object is called when the () function call syntax is used. Keeping this syntactic transformation in mind can help solve errors that occur when using lambda. The example earlier where the count total didn't account for the increments in the lambda becomes obvious here. The transformation would be:

var count = U32(0)
for_each(l, object
              var env:Env = env
              var count:U32 = count
              fun ref apply(s:String) =>
                env.out.print(s)
                count = count + 1
            end)
env.out.print("Count: " + count.string())

Note the ref annotation in the apply method. This is what the lambda ref(...) syntax results in. Without the ref in the lambda syntax the syntactic expansion is fun apply(...). The default reference capability for fun in objects is box.

The object literal expansion helps show the difference between {(string)} ref and {ref(String))}. The first requires an object with the given reference capability to match the type. Although the object is a ref, the apply method is the default, a box. This means it cannot modify any fields in the object. In the case of a lambda it won't be able to modify the captured variables:

object ref
  fun apply(...)
end

To match the second type definition it requires an object where the apply method itself has the ref capability. This allows apply to modify fields of the object:

object
  fun ref apply(...)
end

Partial Application

Related to lambda is partial application. It allows supplying some arguments to a function (or constructor or behaviour) and returns something that allows supplying the other arguments later.

In cases where you would use a lambda to return a function that sets the value of some arguments you can use partial application instead. The following example creates a lambda that adds five to another number:

actor Main
  new create(env:Env) =>
    let add5 = lambda(a:U32): U32 => 5 + a end
    env.out.print(add5(10).string())
    env.out.print(add5(20).string()) 

With partial application this becomes:

actor Main
  new create(env:Env) =>
    let add5 = U32(5)~add()
    env.out.print(add5(10).string())
    env.out.print(add5(20).string())

The use of + is an alias for the add method on an object. In the example above the ~ operator represents partial application. We are binding the receiver of the add method to be the 32-bit unsigned value 5. The resulting object can be called with the remaining arguments.

Partial application allows binding any arguments, by position or by name. The tutorial goes into a lot of detail.

Accumulator Generator

Paul Graham has an accumulator generator problem with examples in various languages. The problem is defined as:

Write a function foo that takes a number n and returns a function that takes a number i, and returns n incremented by i.

A Pony implementation, using closures, could be:

actor Main
  fun foo(n:U32): {ref(U32): U32} =>
    var s: Array[U32] = Array[U32].init(n, 1)
    lambda ref(i:U32)(s): U32 =>
      try
        s(0) = s(0) + i
        s(0)
      else
        0 
      end
    end

  new create(env:Env) =>
    var f = foo(5)
    env.out.print(f(10).string())
    env.out.print(f(20).string()) 

This uses the one element array trick to enable modifying the captured variable due to the requirement that the number n is incremented. It works only for U32 rather than any number which is also part of the problem definition. Pony has generics which would allow solving this but I'll leave that as an exercise or a future post.

Further reading

Some links to documentation and posts about lambda and reference capabilities to dive into more detail:

Tags: pony 

2015-12-17

Cross compile Pony programs for Android

Support for compiling Pony programs on ARM and 32-bit x86 landed recently. This allows compiling and running Pony on Raspberry Pi and other ARM devices. I was curious if it would be possible to compile Pony programs to run on Android and this post outlines how I got a "Hello World" example working.

The Pony compiler, ponyc, does not currently support cross compilation. It uses the C pre-processor to generate code for the platform it is running on. This has hardcoded assumptions for byte size (32 vs 64 bit) and processor support (ARM vs x86). Note that this is a proof of concept and hacks the compiler and runtime source to get things working. From this I hope to learn more elegant ways of supporting cross compiling.

Ponyc Android Cross Compiler

The first step was to build a version of ponyc that would generate Android compatible ARM code without any host platform specific code being included. I built ponyc for 32-bit x86, modified to not include any x86 specific code generation and to allow selecting the non-native LLVM backends. This ensures that the hard coded assumptions for the 32-bit size matches my target, 32-bit Android.

The changes for this are in the android_ponyc_cross branch of my github repository. The changes were:

  1. Modify the Makefile to use 32-bit flags to the compiler.
  2. Add some LLVM initialization to allow selection of non-native backends so ARM can be generated on x86 hosts.
  3. Comment out some PLATFORM_IS_X86 preprocessor statements to prevent x86 specific code being generated.

For a real cross compiler these would be changed to be runtime selectable in some way. For the proof of concept it suffices to create a hardcoded compiler specific for this example.

For me it was necessary to build a version of LLVM for 32-bit as I'm on a 64-bit host. This was done by doing the following:

$ sudo apt-get install libicu-dev:i386 libncurses5-dev:i386 libxml2-dev:i386
$ tar xvf /tmp/llvm-3.6.2.src.tar.xz
$ cd llvm-3.6.2.src/tools
$ tar xvf /tmp/cfe-3.6.2.src.tar.xz
$ mv cfe-3.6.2.src clang
$ cd ..
$ mkdir build
$ cd build
$ cmake -DLLVM_BUILD_32_BITS=ON -DCMAKE_INSTALL_PREFIX=/tmp/llvm ..      
$ make
$ make install

Building the modified ponyc used these steps:

$ git clone -b android_ponyc_cross https://github.com/doublec/ponyc android_ponyc_cross
$ cd android_ponyc_cross
$ LLVM_CONFIG=/tmp/llvm/bin/llvm-config CXX="g++ -m32" make config=release ponyc

This generates a ponyc executable in the build/release directory.

Android compatible Pony runtime

An Android compatible libponyrt library is needed to link. The changes to build this are on the android_libponyrt branch. This must be built using an Android NDK standalone toolkit. A compatible standalone toolkit can be created with the following run from the NDK root directory:

$ ./build/tools/make-standalone-toolchain.sh --arch=arm \
     --platform=android-21 --install-dir=/tmp/stk21

Add the resulting installation directory to the PATH:

$ export PATH=/tmp/stk21/bin:$PATH

Make sure a recent ponyc compiler is on the PATH. This should not be the cross compiler built previously but a ponyc compiler for the host platform. Build the runtime library with:

$ git clone -b android_libponyrt https://github.com/doublec/ponyc android_libponyrt
$ cd android_libponyrt
$ CC=arm-linux-androideabi-gcc make config=release libponyrt

The resulting library is in the directory build/release/libponyrt.a.

Compiling to Android

With the above tasks complete it's now possible to build an Android compatible binary. I tested with a "Hello World" example:

actor Main
  new create(env:Env) =>
    env.out.print("Hello Android")

With that in a main.pony file in a hello directory, build with:

$ /tmp/android_ponyc_cross/build/release/ponyc --triple arm-linux-androideabi -rir hello
...ignore warnings about core-avx2 feature here...
$ llvm-as hello.ll
$ llc -mtriple=arm-linux-androideabi hello.bc -o hello.o -filetype=obj
$ arm-linux-androideabi-gcc -fPIE -pie hello.o -o hello1 -L/tmp/android_libponyrt/build/release/ -lponyrt 

The first command instructs our cross compiler to generate code for Android and to only produce the LLVM assembly listing. This is compiled to bytecode with llvm-as and then to an object file with llc. The Android version of gcc is used to link with the Android version of libponyrt that was created earlier. The hello binary that is produced can be copied to an Android device (I used a Note 3 running Lollipop) and run:

$ adb push hello1 /data/local/tmp/
$ adb shell
$ cd /data/local/tmp
$ ./hello1
Hello Android

Conclusion

I haven't tested any other features running on Android yet but this is a promising start. Using the Pony FFI and JNI it is hopefluly possible to write native Android applications.

The steps I used for getting ponyc to generate LLVM instructions and using the LLVM tools to compile it were obtained from a gist by Darach Ennis. Similar steps would probably work on other LLVM compatible platforms.

Tags: pony 

2015-12-16

C linkable libraries with Pony

My quick look at Pony post covered how to use the FFI to call C code from a Pony program. It's also possible to compile Pony code to a C library that can be linked into a C program. This allows integrating Pony into existing C projects without having to convert the C project into a library to be called by Pony.

I've put a small example of how to do this on github at https://github.com/doublec/pony-clib-example.

Pony has a special type of actor that will result in a C compatible interface being generated. The syntax for this is actor@. The example in the project is:

actor@ Foo
  new create() =>
    None

  be hi() =>
    @printf[I32]("Hi\n".cstring())

This creates a C compatible actor called Foo. It has a constructor create and a behaviour hi. The constructor does nothing and only exists in the example to show how to call it from C. The hi behaviour calls the C function printf using the Pony FFI. With a file main.pony containing this code in a clib directory it can be built with:

$ ponyc -l clib
Building builtin -> packages/builtin
Building clib -> clib
Generating
Optimising
Writing ./clib.o
Archiving ./libclib.a

The -l switch to ponyc results in a static library being built instead of an executable. A libclib.a is generated containing the compiled Pony code. A clib.h is also generated containing the C function declarations for creating and interacting with any actor@ instances. For the Foo actor this looks like:

typedef struct Foo Foo;
Foo* Foo_Alloc();
Foo* Foo_create__send(Foo* self);
Foo* Foo_hi__send(Foo* self);

Pony has a runtime that needs to be initialized from C and a scheduler that needs to be started. The definitions for these runtime functions can be found in the ponyc source in libponyrt/pony.h. This needs to be included by the C program. A basic C driver for the example project is:

#include <stdio.h>
#include "clib.h"
#include "pony.h"

int main(int argc, char** argv) {
  pony_init(argc, argv);

  Foo* x = Foo_Alloc();
  Foo_create__send(x);
  Foo_hi__send(x);

  pony_start(0);
  return 0;
}

This sould be compiled and linked against libclib.a and libponyrt.a:

export PONYRT_INCLUDE=/path/to/ponyc/src/libponyrt/
export PONYRT_LIB=/path/to/ponyc/build/release/libponyrt.a
gcc -o test -I. -I $(PONYRT_INCLUDE) -g -mcx16 test.c libclib.a $(PONYRT_LIB) -lpthread -ldl

The C code needs to call pony_init to initialize the Pony runtime. It gets passed the standard C argc and argv arguments that main receives so it can handle any Pony runtime specific switches. After this any Pony actors can be allocated and methods or behaviours called on them. The order for actor creation is to call the _Alloc function to allocate the memory for the actor and then the constructor method. Pony names the constructor method based on the name in the Pony code with __send appended.

Actors are asynchronous so just like in Pony code behaviours aren't run immediately. They are queued for execution and will run when the scheduler is started. This is what pony_start does. The argument passed to pony_start is a boolean indicating whether to block until all Pony actors have completed (by passing a false value) or to return immediately (by passing a true value). If the latter is done then the scheduler runs asynchronously. It can be stopped with a later call to pony_stop. It's possible to continue calling Pony functions after the pony_start and before pony_stop. For example:

pony_init(argc, argv);

Foo* x = Foo_Alloc();
Foo_create__send(x);
Foo_hi__send(x);

pony_start(1);

Foo* y = Foo_Alloc();
Foo_create__send(y);
Foo_hi__send(y);

pony_stop();

In this snippet another actor is created after pony_start and behaviours are called on it. These are queued and run as expected. pony_stop blocks until all Pony actors have completed.

There are other Pony runtime functions in pony.h that allow allocating memory on the current actor heap, unscheduling an actor so that no behaviours will be run until rescheduled, making a C thread 'become' an actor so it can call Pony runtime functions, poll for messages on an unscheduled actor, etc. It's possible to allocate an actor, unschedule it, and then use a C thread to handle its message queue using pony_become. When you aren't handling messages to the unscheduled actor you can call functions to it from C with no race conditions.

None of this is currently documented and it will probably change as usage develops. In the meantime it's a useful feature for using Pony functionality in existing applications.

Tags: pony 

2015-11-19

Exporting Self objects to source files using the Transporter

Following on from my previous video on using the Self GUI to create objects, I've done another that demonstrates how to export the objects to Self source files where they can be managed with a version control system and shared with other developers.

The video is available as self_transporter.webm, self_transporter.mp4 or on YouTube.

More information on using the transporter tool can be obtained from:

Tags: self 


This site is accessable over tor as hidden service mh7mkfvezts5j6yu.onion, or Freenet using key:
USK@1ORdIvjL2H1bZblJcP8hu2LjjKtVB-rVzp8mLty~5N4,8hL85otZBbq0geDsSKkBK4sKESL2SrNVecFZz9NxGVQ,AQACAAE/bluishcoder/-25/


Tags

Archives
Links