Bluish Coder

Performing Compatible Updates of Mozilla Central Git Forks

Earlier this year I posted about how I created my git mirror of the mozilla-central mercurial repository. I've been keeping a fork on github updated regularly since then that a number of people have started using.

Users of my mirror have wanted to be able to keep up to date with the mercurial mozilla-central repository themselves, or add updates from other branches of the Mozilla source. It takes a large amount of time to start a conversion from scratch but it's possible to start from the existing git mirror, perform the incremental updates from mercurial yourself, and still stay compatible with the incremental updates I push to my github mirror.

hg-git uses a git-mapfile in the .hg directory of the mercurial clone to keep track of the mapping between mercurial and git commits. By using the same git-mapfile that I use for my mirror you can start your own incremental update from the latest data in the git-mapfile instead of from the beginning. I keep an up-to-date git-mapfile in git-mapfile.bz2. It's compressed with bzip2 as the uncompressed version is quite large.

The steps to start your own update process are:

1. Clone the mercurial mozilla-central repository
2. Clone my git mirror as a bare repository in .hg/git
3. Place the git-mapfile in .hg
4. Do an hg bookmark -f -r default master to mark the commit to convert to
5. Perform hg gexport to update .hg/git with recent mercurial commits

The .hg/git directory should now be up-to-date with respect to the mercurial repository. And the additional git commits will have the same SHA id as any commits I push into my mirror when I perform my own update.

The steps to do an incremental update are the normal:

1. Pull from the mercurial mozilla-central repository
2. Run hg bookmark -f -r default master
3. Run hg gexport
4. Push or pull to/from .hg/git as needed

As a working example, the following shell commands on a Linux system should set up your own repository ready for incremental updating:

$ hg clone https://hg.mozilla.org/mozilla-central
$ cd mozilla-central/.hg
$ git clone --bare git://github.com/doublec/mozilla-central.git git
$ wget http://www.bluishcoder.co.nz/git-mapfile.bz2
$ bunzip2 git-mapfile.bz2
$ cd ..
$ hg bookmark -f -r default master
$ hg gexport
$ cd .hg/git
$ git push --all ~/some/other/repo.git

Incremental updates just repeat the last few commands above:

$ hg pull -u
$ hg bookmark -f -r default master
$ hg gexport
$ cd .hg/git
$ git push --all ~/some/other/repo.git

Note that I use hg gexport and push from the .hg/git repository using the git command instead of doing an hg push and relying on hg-git to do the conversion and push. In my original article I did the latter. The hg-git push method uses slower python based routines to do the push which can take a long time and large amounts of memory on big repositories like mozilla-central. By splitting this up into gexport and using the native git command I save a lot of time and memory.

Building Rust

Update 2011-11-08 - The steps to build Rust have changed since I wrote this post, there's now no need to build LLVM - it's included as a submodule in the Rust git repository and will automatically be cloned and built.

A while ago I wrote a quick look at Rust post, describing how to build it and run some simple examples. Since that post the bootstrap compiler has gone away and the rustc compiler, written in Rust, is the compiler to use.

The instructions for building Rust in the wiki are good but I'll briefly go through how I installed things on 64-bit Linux. The first step is to build the required version of LLVM from the LLVM source repository. I use a git mirror and install to a local directory so as not to clash with other programs that use older LLVM versions.

$ git clone git://github.com/earl/llvm-mirror.git
$ cd llvm-mirror
$ git reset --hard 9af578
$ CXX='g++ -m32' CC='gcc -m32' CFLAGS=-m32 CXXFLAGS=-m32 LDFLAGS=-m32 ./configure \
    --{build,host,target}=i686-unknown-linux-gnu \
    --enable-targets=x86,x86_64,cbe --enable-optimized --prefix=/home/chris/rust
$ make && make install

Note the use of prefix to ensure this custom LLVM build is a local install. I also do a git reset to go to the commit for SVN revision 142082 which is the current version that works with Rust according to the wiki. After installation add the install bin to the PATH and lib to LD_LIBRARY_PATH:

$ export PATH=~/rust/bin:$PATH
$ export LD_LIBRARY_PATH=~/rust/lib:$LD_LIBRARY_PATH

Next step, clone and build Rust:

$ git clone git://github.com/graydon/rust
$ mkdir build
$ cd build
$ ../rust/configure
$ make

The build process downloads an existing build for your platform to bootstrap from. It uses this to build a stage1 version of the compiler. This stage1 is used to build a stage2, and that is then used to build a stage3. All the built compilers should work exactly the same if things are working correctly. The compiler can be run within the build directory, or outside it if you put the stage3/bin directory in the path:

$ export PATH=~/path/to/build/stage3/bin:$PATH
$ rustc
error: No input filename given.

Test with a simple 'hello world' program:

$ cat hello.rs
use std;
import std::io::print;

fn main () {
  print ("hello\n");
}
$ rustc hello.rs
$ ./hello
hello

Overloading Functions in ATS

ATS allows overloading of functions where the function that is called is selected based on number of arguments and the type of the arguments. The following example shows overloading based on type to provide a generic print function:

symintr myprint
fun myprint_integer (a: int)    = printf("%d\n", @(a))
fun myprint_double  (a: double) = printf("%f\n", @(a))
fun myprint_string  (a: string) = printf("%s\n", @(a))

overload myprint with myprint_integer
overload myprint with myprint_double
overload myprint with myprint_string

implement main() = {
  val () = myprint ("hello")
  val () = myprint (10)
  val () = myprint (20.0)
}

The keyword symintr introduces a symbol that can be overloaded. The keyword overload will overload that symbol with an existing function. In this example we overload with three functions that take different types. The overload resolution is performed at compile time. The actual C code generated for the main function includes:

/* tmp4 = */ myprint_string_2 (ATSstrcst("hello")) ;
/* tmp5 = */ myprint_integer_0 (10) ;
/* tmp3 = */ myprint_double_1 (20.0) ;

The ATS standard prelude includes a print function that is overloaded in this manner for most of the standard types. One downside to the way overloading works is the overload resolution sometimes fails in template functions. The following code gives a compile error for example:

fun {a:t@ype} printme (x: a) = print(x)

implement main() = {
  val () = printme (10)
  val () = printme (20.0)
}

The error given is that the symbol print cannot be resolved. The ATS compiler attempts to resolve the overload by looking up the t@ype sort. There is no overload for this so the resolution fails. This can be worked around using a template function to call the overloaded function and partially specialize the implementation of the new template function. The following code demonstrates this:

extern fun {a:t@ype} gprint (x: a):void
implement gprint<int> (x) = print_int(x)
implement gprint<double> (x) = print_double(x)

fun {a:t@ype} printme (x: a):void = gprint<a>(x)

implement main() = {
  val () = printme (10)
  val () = printme (20.0)
}

The print symbol can be overloaded with the new gprint function to allow print to be called over t@ype sorts:

extern fun {a:t@ype} gprint (x: a):void
implement gprint<int> (x) = print_int(x)
implement gprint<double> (x) = print_double(x)

overload print with gprint

fun {a:t@ype} printme (x: a) = print(x)

implement main() = {
  val () = printme (10)
  val () = printme (20.0)
}

This example is contrived in that you could just specialize printme but in real world code this issue comes up occasionally. The most common example for me has been using = in a template function, comparing arguments that are template type parameters. = is an overloaded function and the overload lookup fails in the same manner as above. A workaround is to create an equals template function specialized over the types you plan to compare as above.

Temporal Media Fragment Support in Firefox

The W3C has a Media Fragments Working Group whose mission is to specify temporal and media fragments in the Web using URI's. The draft specification goes through in detail how these fragments work. I recently became a member of the working group and I've been working on adding support for the temporal dimension portion of the specification to Firefox.

In the most basic form you can specify a start time and an end time in the fragment part of a URI in an HTML video or audio element. For example:

<video src='/mediafrag/AudioAPI.webm#t=50,100'></video>

The '#t' portion of the URI identifies the fragment as being a temporal media fragment. In this example '50,100' means start the video at a current time of 50 seconds, and stop playing at 100 seconds. There are various other formats for the temporal media fragment defined in the specification. Examples can be seen in the UA Test Cases.

Development of this feature is being done in bug 648595. I've done test builds of the first iteration of the patch and they are available at my Firefox media fragment test builds page. The page has builds, an example, and a list of limitations which are currently:

• Temporal syntax only. This means no spatial or track dimensions.
• NPT time support only. No SMPTE time codes or Wall-clock time codes.
• When changing the media fragment portion of a URL the media is not immediately updated. You need to refresh the page to see the change. This is most noticeable when directly navigating to the video and adding or changing a fragment.
• The user interface for identifying the fragment in the standard controls is ugly and needs polish.
• The HTML standard includes an 'initialTime' attribute for obtaining the start time. There is no way to obtain an end time so I've exposed a 'mozFragmentEnd' attribute on the video DOM object.

Safe destruction in the presence of sharing in ATS

While using the libevent API from ATS I came across a scenario where it was important to call a function to release objects in a particular order. I wanted to have ATS enforce at compile time that the destruction occurs safely in the right order. The following example uses the built in libevent HTTP API for creating simple web servers. It uses the ATS libevent wrapper I wrote about previously.

fn cb {l1,l2:agz} (request: !evhttp_request l1, arg: !event_base l2): void = let
  val () = printf("here\n", @())
in
 ()
end

fn server () = let
  val _ = signal (SIGPIPE, SIG_IGN)
  val base = event_base_new ()
  val () = assert_errmsg (~base, "event_base_new failed")

  val http = evhttp_new (base)
  val () = assert_errmsg (~http, "evhttp_new failed")

  val r = evhttp_set_cb_with_base (http, "/", cb, base)
  val () = assert_errmsg (r = 0, "evhttp_set_cb_with_base failed")

  val r = evhttp_bind_socket (http, "0.0.0.0", uint16_of_int(8080))
  val () = assert_errmsg (r = 0, "evhttp_bind_socket failed")

  val r = event_base_dispatch (base)
  val () = assert_errmsg (r >= 0, "event_base_dispatch failed")

  val () = evhttp_free (http)
  val () = event_base_free (base)
in
  ()
end

implement main () = server ()

The call to http_new requires an event_base object. Internally, inside the C API, the returned evhttp object holds a pointer to this event_base. This results in the pointer being shared in two places and requires careful destruction to prevent using a destroyed object.

Later I call evhttp_free to destroy and release resources associated with the evhttp object, followed by event_base_free to do the same with the event_base object. I do it in this order to prevent a dangling reference to the event_base inside the evhttp object. The evhttp object is associated with that particular event_base and it's important not to pass an incorrect base to functions that use the http object. Ideally code like the following shouldn't compile:

val base = event_base_new ()
val base2 = event_base_new ()
val http = evhttp_new (base)

// Wrong event_base passed, fail compilation
val r = evhttp_set_cb_with_base (http, "/", cb, base2)

// Destruction out of order, fail compilation
val () = event_base_free (base)
val () = evhttp_free (http)

To model this in ATS I changed the ATS definition of evhttp to have an event_base associated with it at the type level and modified evhttp_new to return this relationship (agz is an address that cannot be NULL, agez is an address that can be NULL):

absviewtype evhttp (l:addr, l2:addr)
fun http_new {l1:agz} (base: !event_base l1): [l2:agez] evhttp (l2,l1) = "mac#evhttp_new"

With this change the type for evhttp depends on the value of the pointer to the event_base given as an argument. Now we can use this in the definition of evhttp_set_cb_with_base to ensure that the correct event_base object is used:

fun evhttp_set_cb_with_base {l1,l2:agz}
       (http: !evhttp (l1, l2),
        path: string,
        callback: {l3:agz} (!evhttp_request l3, !event_base l2) -> void,
        arg: !event_base l2): int = "mac#evhttp_set_cb"

This definition states that the event_base given as the arg paramater, and as the second argument to the callback, must be the same as that associated with the http object. This is done by using the same l2 dependent type argument with these types. This gives us the desired error checking in the first 'fail compilation' test above.

The next step is to prevent out of order destruction. This is done by passing the event_base object as a proof argument to the evhttp_free function:

fun evhttp_free {l1,l2:agz} (base: !event_base l1 | http: evhttp (l2, l1)): void = "mac#evhttp_free"

By adding this as proof argument I'm stating that the caller of evhttp_free must have a usable event_base object that was used to create this evhttp object to prove we can safely destroy it. The following is now a compile time error:

val () = event_base_free (base)
val () = evhttp_free (base | http)

This is due to event_base_free consuming the base linear object. The type of base following this call is no longer defined. The evhttp_free call is now an error due to using an undefined base. The following will work:

val () = evhttp_free (base | http)
val () = event_base_free (base)

This method of ensuring correct order of destruction and resource usage requires no changes to the libevent C code. Everything occurs during type checking. The generated C code looks exactly like normal libevent C usage with no runtime overhead for tracking the association between evhttp and event_base objects.