I've written about reference capabilities in the Pony programming language spread across some of my other posts but haven't written about them directly. This post is my attempt to provide an intuitive understanding of reference capabilities and when to use
recover. Hopefully this reduces the confusion when faced with reference capability compilation errors and the need to memorize capability tables.
Reference capabilities are used to control aliasing. Controlling aliasing is important when sharing data to avoid data races in the presence of multiple threads. An example of aliasing is:
let a: Object = create_an_object() let b: Object = a
In that snippet, 'b' is an alias to the object referenced by 'a'. If I pass 'b' to a parallel thread then the same object can be mutated and viewed by two different threads at the same time resulting in data races.
Reference capabilities allow annotating types to say how they can be aliased. Only objects with a particular set of reference capabilities can be shared across actors.
tag - opaque reference
tag reference capability is an opaque reference to an object. You cannot read or write fields of the object referenced through a
tag alias. You can store
tag objects, compare object identity and share them with other actors. The sharing is safe since no reading or writing of state is allowed other than object identity.
let a: Object tag = create_a_tag() let b: Object tag = a if a is b then ...we're the same object... end
There is also the
digestof operator that returns a unique unsigned 64 bit integer value of an object. This is safe to use on
let a: Object tag = create_a_tag() env.out.print("Id is: " + (digestof a).string())
tag reference capability is most often used for actors and passing references to actors around. Actor behaviours (asynchronous method calls) can be made via
val - immutable and sharable
val reference capability on a variable means that the object is immutable. It cannot be written via that variable. Only read-only fields and methods can be used. Aliases can be shared with other actors because it cannot be changed - there is no issue of data races when accessed from multiple threads. There can be any number of aliases to the object but they all must be
let a: Object val = create_an_object() let b: Object val = a let c: Object val = a call_some_function(b) send_to_an_actor(c)
All the above are valid uses of
val. Multiple aliases can exist within the same actor or shared with other actors.
ref - readable, writable but not sharable
ref reference capability means the object is readable and writable but only within a single actor. There can exist multiple aliases to it within an actor but it cannot be shared with other actors. If it were sharable with other actors then this would allow data races as multiple threads of control read or write to it in a non-deterministic manner.
let a: Object ref = create_a_ref_object() let b: Object ref = a call_some_function(b)
The above are valid uses of
ref if they are occuring within a single actor. The following are invalid uses - they will result in a compile error:
let a: Object ref = create_a_ref_object() send_to_an_actor(a) let b: Object val = a
send_to_an_actor call would result in an alias of
a being accessible from another thread. This would cause data races so is disallowed and results in a compilation error. The assignment to an
Object val is also a compilation error. The reasoning for this is access via
b would assume that the object is immutable but it could be changed through the underlying alias
b were passed to another actor then changes made via
a will cause data races.
iso - readable, writable uniquely
iso reference capability means the object is readable and writable but only within a single actor - much like
ref. But unlike
ref it cannot be aliased. There can only be one variable holding a reference to an
iso object at a time. It is said to be 'unique' because it can only be written or read via that single unique reference.
let a: Object iso = create_an_iso_object() let b: Object iso = a call_some_function(a)
The first line above creates an
iso object. The other two lines are compilation errors. The assignment to
b attempts to alias
a. This would enable reading and writing via
b which breaks the uniqueness rule.
The second line calls a function passing
a. This is an implicit alias of
a in that the parameter to
call_some_function has aliased. It is readable and writable via
a and the parameter in
When it comes to fields of objects things get a bit more complicated. Reference capabilities are 'deep'. This means that the capability of an enclosing object affects the capability of the fields as seen by an external user of the object. Here's an example that won't work:
class Foo var bar: Object ref = ... let f: Foo iso = create_a_foo() let b: Object ref = create_an_object() f.bar = b
If this were to compile we would have a
ref alias alive in
b and another alias to the same object alive in the
bar field of
f. We could then pass our
f iso object to another actor and that actor would have a data race when trying to use
bar since the original actor also has an alias to it via
The uniqueness restriction would seem to make
iso not very useful. What makes it useful is the ability to mark aliases as no longer used via the
consume - I don't want this alias
consume keyword tells Pony that an alias should be destroyed. Not the object itself but the variable holding a reference to it. By removing an alias we can pass
iso objects around.
let a: Object iso = create_an_iso_object() let b: Object iso = consume a call_some_function(consume b)
This snippet creates an
iso object referenced in variable
consume in the second line tells Pony that
a should no longer alias that object. It's floating around with nothing pointing to it now. Pony refers to this state as being 'ephemeral'. At this point the variable
a doesn't exist and it is a compile error to use it further. The object has no aliases and can now be assigned to another variable, in this case
b. This meets the requirements of
iso because there is still only one reference to the object, via
The function call works in the same manner. The
consume b makes the object ephemeral and can then be assigned to the parameter for the function and still meet the uniqueness requirements of
iso objects can be sent to other actors. This is safe because there is only a single alias. Once it has been sent to another actor, the alias from the original actor cannot read or be written to because the alias it had was consumed:
let a: Object iso = create_an_iso_object() send_to_an_actor(consume a)
iso reference that is consumed, can that ephemeral object be assigned to other reference capabilities other than
iso? The answer to that is yes.
Intuitively this makes sense. If you have no aliases to an object then when you alias that object you can make it whatever capability you want - it is like having created a new object, nothing else references it until you assign it. From that point on what you can do with it is restricted by the reference capability of the variable you assigned it to.
let a: Object iso = create_an_iso() let b: Object val = consume a let c: Object iso = create_an_iso() let d: Object ref = consume c;
The above are examples of valid conversions. You can have an
iso, make changes to the object, then consume the alias to assign to some other reference capability. Once you've done that you are restricted by that new alias:
let c: Object iso = create_an_iso() let d: Object ref = consume c; send_to_an_actor(d)
That snippet is an error as
ref cannot be sent to another actor as explained earlier. This is also invalid:
let c: Object iso = create_an_iso() let d: Object ref = consume c; send_to_an_actor(c)
Here we are trying to use
c after it is consumed. The
c alias no longer exists so it is a compile error.
What if you want to go the other way and convert a
val to an
let a: Object val = create_a_val() let b: Object iso = consume a
This is an error. Consuming the
a alias does not allow assigning to another reference capability. Because
val allows multiple aliases to exist the Pony compiler doesn't know if
a is the only alias to the object. There could be others aliases elsewhere in the program.
iso requires uniqueness and the compiler can't guarantee it because of this. The same reasoning is why the following is an error:
let a: Object val = create_a_val() let b: Object ref = consume a
Intuitively we can reason why this fails.
ref allows reading and writing within an actor.
val requires immutability and can have multiple aliases. Even though we consume
a there may be other aliases around, like the
iso example before. Writing to the object via the
b alias would break the guarantee of any other aliases to
Given this, how do you do this type of conversion? This is what the
recover expression is used for.
recover - restricted alias conversion
recover expression provides a block scope where the variables that can enter the scope are restricted based on their reference capability. The restriction is that only objects that you could send to an actor are allowed to enter the scope of the
recover expression. That is
recover expression you can create objects and return them from the expression as a different reference capability than what you created them as. This is safe because the compiler knows what entered the block, knows what was created within the block, and can track the aliases such that it knows it's safe to perform a particular conversion.
let a: Object val = recover val let b: Object ref = create_a_ref() ...do something... b end
In this snippet we create a
ref object within the
recover block. This can be returned as a
val because the compiler knows that all aliases to that
ref object exist within the
recover block. When the block scope exits those aliases don't exist - there are no more aliases to the object and can be returned as the reference type of the
How does the compiler know that the
b object wasn't stored elsewhere within the
recover block? There are no global variables in Pony so it can't be stored globally. It could be passed to another object but the only objects accessable inside the block are the restricted ones mentioned before (
tag). Here's an attempt to store it that fails:
var x: Object ref = create_a_ref() let a: Object val = recover val let b: Object ref = create_a_ref() x = b b end
This snippet has a
ref object created in the enclosing lexical scope of the
recover expression. Inside the
recover an attempt is made to assign the object
b to that variable
x. Intuitively this should not work - allowing it would mean that we have a readable and writeable alias to the object held in
x, and an immutable alias in
a allowing data races. The compiler prevents this by not allowing a
ref object from the enclosing scope to enter a
Can we go the other way and convert a
val to a
recover? Unfortunately the answer here is no.
let a: Object ref = recover ref let b: Object val = create_a_ref() ...do something... b end
This results in an error. The reason is a
val can be stored in another
val variable in the enclosing scope because
val objects are safely shareable. This would make it unsafe to return a writeable alias to the
val if it is stored as an immutable alias elsewhere. This code snippet shows how it could be aliased in this way:
let x: Object val = create_a_val() let a: Object val = recover val let b: Object val = create_a_ref() x = b b end
We are able to assign
b to a variable in the enclosing scope as the
x variable is a
val which is one of the valid reference capabilities that can be accessed from within the
recover block. If we were able to recover to a
ref then we'd have a writeable and an immutable alias alive at the same time so that particular conversion path is an error.
A common use for
recover is to create objects with a reference capability different to that defined by the constructor of the object:
class Foo new ref create() => ... let a: Foo val = recover val Foo end let b: Foo iso = recover iso Foo end
The reference capability of the recover expression can be left out and then it is inferred by the capability of the variable being assigned to:
let a: Foo val = recover Foo end let b: Foo iso = recover Foo end
Two more reference capabilities to go. They are
box - allows use of val or ref
box reference capability provides the ability to write code that works for
ref objects. A
box alias only allows readonly operations on the object but can be used on either
let a: Object ref = create_a_ref() let b: Object val = create_a_val() let c: Object box = a let d: Object box = b
This is particularly useful when writing methods on a class that should work for a receiver type of
class Bar var count: U32 = 0 fun val display(out:OutStream) => out.print(count.string()) actor Main new create(env:Env) => let b: Bar val = recover Bar end b.display(env.out)
This example creates a
val object and calls a method
display that expects to be called by a
val object (the "fun val" syntax). The
this from within the
display method is of reference capability
val. This compiles and works. The following does not:
let b: Bar ref = recover Bar end b.display(env.out)
Here the object is a
display expects it to be
val. We can change
display to be
ref and it would work:
fun ref display(out:OutStream) => out.print(count.string())
But now we can't call it with a
val object as in our first example. This is where
box comes in. It allows a
ref or a
val object to be assigned to it and it only allows read only access. This is safe for
val as that is immutable and it is safe for
ref as an immutable view to the
fun box display(out:OutStream) => out.print(count.string())
box by default so can be written as:
fun display(out:OutStream) => out.print(count.string())
As an aside, the default of
box is the cause for a common "new to Pony" error message where an attempt to mutate a field in an object fails with an "expected box got ref" error:
fun increment() => count = count + 1
This needs to be the following as the implicit
box makes the
this immutable within the method:
fun ref increment() => count = count + 1
trn - writeable uniquely, consumable to immutable
trn reference capability is writeable but can be consumed to an immutable reference capability,
val. This is useful for cases where you want to create an object, perform mutable operations on it and then make it immutable to send to an actor.
let a: Array[U32] trn = recover Array[U32] end a.push(1) a.push(2) let b: Array[U32] val = consume a send_to_actor(b)
ref methods can be called on
class Bar var count: U32 = 0 fun box display(out:OutStream) => out.print(count.string()) fun ref increment() => count = count + 1 actor Main new create(env:Env) => let a: Bar trn = recover Bar end a.increment() a.display(env.out)
This provides an alternative to the "How do I convert a
ref to a
val?" question. Instead of starting with a
ref inside a
recover expression you can use
consume to a
You can use
iso in place of
trn in these examples. Where
trn is useful is passing it to
box methods to perform readonly operations on it. This is difficult with
iso as you have to consume the alias everytime you pass it around, and the methods you pass it to have to return it again if you want to perform further operations on it. With
trn you can pass it directly.
actor Main let out: OutStream fun display(b: Bar box) => b.display(out) new create(env:Env) => out = env.out let a: Bar trn = recover Bar end display(a) let b : Bar val = consume a send_to_actor(b)
The equivalent with
iso is more verbose and requires knowledge of ephemeral types (the hat,
actor Main let out: OutStream fun display(b: Bar iso): Bar iso^ => b.display(out) consume b new create(env:Env) => out = env.out let a: Bar iso = recover Bar end let b: Bar iso = display(consume a) let c: Bar val = consume b send_to_actor(c)
I've tried to use a lot of examples to help gain an intuitive understanding of the capability rules. The Pony Tutorial has a Capability Subtyping page that gives the specific rules. Although technical seeming the rules there encode our implicit understanding. This section is a bit more complex and isn't necessary for basic Pony programming if you have a reasonable grasp of it intuitively. It is however useful for working out tricky capability errors and usage.
The way to read those rules are that "
<:" means "is a subtype of" or "can be substituted for". So "
box" means that a
ref object can be assigned to a
let a: Object ref = create_a_ref() let b: Object box = a
The effects are transitive. So if "
iso" and "
trn" and "
ref" then "
let a: Object iso = create_an_iso() let b: Object ref = consume a
Notice we start with
iso^ which is an ephemeral reference capability. We get ephemeral types with
consume. So consuming the
iso gives an
iso^ which can be assigned to a
ref due to the transitive subtyping path above.
Why couldn't we assign the
iso directly without the
consume? This is explained previously using inutition but following the rules on the subtyping page we see that "
! is for an aliased reference capability. When we do "something = a" we are aliasing the
iso and the type of that
a in that expression is
iso!. This can only be assigned to a
tag according to that rule:
let a: Object iso = create_an_iso() let b: Object tag = a
Notice there is no "
iso" which tells us that an alias to an
iso cannot be assigned to an
iso which basically states the rule that
iso can't be aliased.
In a previous section I used an ephemeral type in a method return type:
fun display(b: Bar iso): Bar iso^ => b.display(out) consume b
This was needed because the result of
display was assigned to an
let b: Bar iso = display(consume a)
If we used
Bar iso as the return type then the compiler expects us to be aliasing the object being returned. This alias is of type
iso!. The error message states that
iso! is not a subtype of
iso which is correct as there is no "
iso" rule. Thankfully the error message tells us that "this would be possible if the subcap were more ephemeral" which is the clue that we need the return type to be ephemeral.
I briefly mentioned in a previous section that reference capabilities are 'deep' and this is important when accessing fields of objects. It is also important when writing generic classes and methods and using collections.
The thing to remember is that a containing objects reference capability affects the reference capabilities of fields and methods when accessed via a user of the container.
let a: Array[Bar ref] iso = recover Array[Bar ref] end a.push(Bar) try let b: Bar ref = a(0)? end
Here is an
iso array of
Bar ref objects. The third line to retrieve an element of the array fails to compile, stating that
tag is not a subtype of
ref. Where does the
tag come from? Intuitively we can reason that we shouldn't be able to get a
ref alias of an item in an
iso array as that would give us two
ref aliases of an item in an
iso that could be shared across actors. This can give data races.
Viewpoint adaption encodes this in the type rules. We have receiver,
a of type
iso, attempting to call the
apply method of the array. This method is declared as:
fun apply(i: USize): this->A ?
this->A syntax is the viewpoint adaption. It states that the result of the call is the reference capability of
A as seen by
this. In our case,
this is an
A is a
ref. The viewpoint adaption for
tag and that's where the
tag in the error message comes from.
We could get an immutable alias to an item in the array if the array was
let a: Array[Bar ref] trn = recover Array[Bar ref] end a.push(Bar) try let b: Bar box = a(0)? end
The viewpoint adaption table shows that
box. To get a
ref item we'd need the array to be
let a: Array[Bar ref] ref = recover Array[Bar ref] end a.push(Bar) try let b: Bar ref = a(0)? end
In my examples I've used explicit reference capabilities in types to make it a bit clearer of what is happening. They can be left off in places to get reasonable defaults.
When declaring types the default capability is the capability defined in the
class ref Foo let a: Foo = ... class val Bar let b: Bar = ...
The type of
Foo ref and the type of
Bar val due to the annotation in the
class definition. By default classes are
ref if they aren't annotated.
The type of an object returned from a constructor is based on the type defined in the constructor:
class Foo new ref create() => ... new val create_as_val() => ... new iso create_as_iso() => ... let a: Foo val = Foo.create_as_val()
Foo class is the default type
ref, but there are constructors that explicitly return
iso objects. As shown previously you can use
recover to change the capability returned by a constructor in some instances.
let a: Foo val = recover Foo.create() end // Ok - ref to val let b: Foo ref = recover Foo.create_as_val() end // Not Ok - val to ref
As can be seen, converting
recover is problematic as shown in previous examples.
Reference capabilities are complex for people new to Pony. An intuitive grasp can be gained without needing to memorize tables of conversions. This intuitive understanding requires thinking about how objects are being used and whether such use might cause data races or prevent sharing across actors. Based on that understanding you pick the capabilities to match what you want to use. For those times that errors don't match your understanding use the viewpoint adaption and capability subtyping tables to work out where things are going wrong. Over time your intuitive understanding improves for these additional edge cases.