Bluish Coder

The 'TL;DR' of this post on how to borrow internal fields of iso objects in Pony is:

To borrow fields internal to an iso object, recover the object to a ref (or other valid capability) perform the operations using the field, then consume the object back to an iso.

Read on to find out why.

In this post I use the term borrowing to describe the process of taking a pointer or reference internal to some object, using it, then returning it. An example from C would be something like:

void* new_foo();
void* get_bar(foo* f);
void  delete_foo(foo* f);

...
void* f = new_foo();
void* b = get_bar(f);
...
delete_foo(f);

Here a new foo is created and a pointer to a bar object returned from it. This pointer is to data internal to foo. It's important not to use it after foo is deleted as it will be a dangling pointer. While holding the bar pointer you have an alias to something internal to foo. This makes it difficult to share foo with other threads or reason about data races. The foo object could change the bar data without the holder of the borrowed pointer to bar knowing making it a dangling pointer, or invalid data, at any time. I go through a real world case of this in my article on using C in the ATS programming language.

Pony has the concept of a reference to an object where only one pointer to that object exists. It can't be aliased and nothing else can read or write to that object but the current reference to it. This is the iso reference capability. Capabilities are 'deep' in pony, rather than 'shallow'. This means that the reference capability of an alias to an object affects the reference capabilities of fields of that object as seen by that alias. The description of this is in the viewpoint adaption section of the Pony tutorial.

The following is a Pony equivalent of the previous C example:

class Foo
  let bar: Bar ref
...
let f: Foo ref = Foo.create()
let b: Bar ref = f.bar

The reference capability of f determines the reference capability of bar as seen by f. In this case f is a ref (the default of class objects) which according to the viewpoint adaption table means that bar as seen by f is also a ref. Intuitively this makes sense - a ref signifies multiple read/write aliases can exist therefore getting a read/write alias to something internal to the object is no issue. A ref is not sendable so cannot be accessed from multiple threads.

If f is an iso then things change:

class Foo
  let bar: Bar ref
...
let f: Foo iso = recover iso Foo.create() end
let b: Bar tag = f.bar

Now bar as seen by f is a tag. A tag can be aliased but cannot be used to read/write to it. Only object identity and calling behaviours is allowed. Again this is intuitive. If we have a non-aliasable reference to an object (f being iso here) then we can't alias internally to the object either. Doing so would mean that the object could be changed on one thread and the internals modified on another giving a data race.

The viewpoint adaption table shows that given an iso f it's very difficult to get a bar that you can write to. The following read only access to bar is ok:

class Foo
  let bar: Bar val
...
let f: Foo iso = recover iso Foo.create() end
let b: Bar val = f.bar

Here bar is a val. This allows multiple aliases, sendable across threads, but only read access is provided. Nothing can write to it. According to viewpoint adaption, bar as seen by f is a val. It makes sense that given a non-aliasable reference to an object, anything within that object that is immutable is safe to borrow since it cannot be changed. What if bar is itself an iso?

class Foo
  let bar: Bar iso = recover iso Bar end
...
let f: Foo iso = recover iso Foo.create() end
let b: Bar iso = f.bar

This won't compile. Viewpoint adaption shows that bar as seen by f is an iso. The assignment to b doesn't typecheck because it's aliasing an iso and iso reference capabilities don't allow aliasing. The usual solution when a field isn't involved is to consume the original but it won't work here. The contents of an objects field can't be consumed because it would then be left in an undefined state. A Foo object that doesn't have a valid bar is not really a Foo. To get access to bar externally from Foo the destructive read syntax is required:

class Foo
  var bar: Bar iso = recover iso Bar end
...
let f: Foo iso = recover iso Foo.create() end
let b: Bar iso = f.bar = recover iso Bar end

This results in f.bar being set to a new instance of Bar so it's never in an undefined state. The old value of f.bar is then assigned to b. This is safe as there are no aliases to it anymore due to the first part of the assignment being done first.

What if the internal field is a ref and we really want to access it as a ref? This is possible using recover. As described in the tutorial, one of the uses for recover is:

"Extract" a mutable field from an iso and return it as an iso.

This looks like:

class Foo
  let bar: Bar ref
... 
let f: Foo iso = recover iso Foo end
let f' = recover iso
           let f'': Foo ref = consume f
           let b: Bar ref = f''.bar
           consume f''
         end

Inside the recover block f is consumed and returned as a ref. The f alias to the object no longer exists at this point and we have the same object but as a ref capability in f''. bar as seen by f'' is a ref according to viewpoint adaption and can now be used within the recover block as a ref. When the recover block ends the f'' alias is consumed and returned out of the block as an iso again in f'.

This works because inside the recover block only sendable values from the enclosing scope can be accessed (ie. val, iso, or tag). When exiting the block all aliases except for the object being returned are destroyed. There can be many aliases to bar within the block but none of them can leak out. Multiple aliases to f' can be created also and they are not going to leaked either. At the end of the block only one can be returned and by consuming it the compiler knows that there are no more aliases to it so it is safe to make it an iso.

To show how the ref aliases created within the recover block can't escape, here's an example of an erroneous attempt to assign the f' alias to an object in the outer scope:

class Baz
  var a: (Foo ref | None) = None
  var b: (Foo ref | None) = None

  fun ref set(x: Foo ref) =>
    a = x
    b = x

class Bar

class Foo
  let bar: Bar ref = Bar

var baz: Baz iso = recover iso Baz end
var f: Foo iso = recover iso Foo end
f = recover iso
      let f': Foo ref = consume f
      baz.set(f')
      let b: Bar ref = f'.bar
      consume f'
    end

If this were to compile then baz would contain two references to the f' object which is then consumed as an iso. f would contain what it thinks is non-aliasable reference but baz would actually hold two additional references to it. This fails to compile at this line:

main.pony:20:18: receiver type is not a subtype of target type
          baz.set(f')
                 ^
Info:
main.pony:20:11: receiver type: Baz iso!
              baz.set(f')
              ^
main.pony:5:3: target type: Baz ref
      fun ref set(x: Foo ref) =>
      ^
main.pony:20:18: this would be possible if the arguments and return value were all sendable
              baz.set(f')
                     ^

baz is an iso so is allowed to be accessed from within the recover block. But the set method on it expects a ref receiver. This doesn't work because the receiver of a method of an object is also an implicit argument to that method and therefore needs to be aliased. In this way it's not possible to store data created within the recover block in something passed into the recover block externally. No aliases can be leaked and the compiler can track things easily.

There is something called automatic receiver recovery that is alluded to in the error message ("this would be possible...") which states that if the arguments were sendable then it is possible for the compiler to work out that it's ok to call a ref method on an iso object. Our ref arguments are not sendable which is why this doesn't kick in.

A real world example of where all this comes up is using the Pony net/http package. A user on IRC posted the following code snippet:

use "net/http"
class MyRequestHandler is RequestHandler

  let env: Env

  new val create(env': Env) =>
    env = env'

  fun val apply(request: Payload iso): Any =>
    for (k, v) in request.headers().pairs() do
      env.out.print(k)
      env.out.print(v)
    end

    let r = Payload.response(200)
    r.add_chunk("Woot")
    (consume request).respond(consume r)

The code attempts to iterate over the HTTP request headers and print them out. It fails in the request.headers().pairs() call, complaining that tag is not a subtype of box in the result of headers() when calling pairs(). Looking at the Payload class definition shows:

class iso Payload
  let _headers: Map[String, String] = _headers.create()

  fun headers(): this->Map[String, String] =>
    _headers

In the example code request is an iso and the headers function is a box (the default for fun). The return value of headers uses an arrow type. It reads as "return a Map[String, String] with the reference capability of _headers as seen by this". In this example this is the request object which is iso. _headers is a ref according to the class definition. So it's returning a ref as seen by an iso which according to viewpoint adaption is a tag.

This makes sense as we're getting a reference to the internal field of an iso object. As explained previously this must be a tag to prevent data races. This means that pairs() can't be called on the result as tag doesn't allow function calls. pairs() is a box method which is why the error message refers to tag not being a subtype of box.

To borrow the headers correctly we can use the approach done earlier of using a recover block:

fun val apply(request: Payload iso): Any =>
  let request'' = recover iso
    let request': Payload ref = consume request
    for (k, v) in request'.headers().pairs() do
      env.out.print(k)
      env.out.print(v)
    end
    consume request'
  end
  let r = Payload.response(200)
  r.add_chunk("Woot")
  (consume request'').respond(consume r)

In short, to borrow fields internal to an iso object, recover the object to a ref (or other valid capability) perform the operations using the field, then consume the object back to an iso.