Bluish Coder

Programming Languages, Martials Arts and Computers. The Weblog of Chris Double.


2011-02-26

Reading data from a file in ATS

I'm back to using the ATS programming language after a bit of a break. My first thought upon getting back into the language is how easy it is to forget how to write in a language you don't use often. It's starting to come back to me though.

For my current task I needed to read string data from a file. This is a basic snippet of an approach:

%{^
#include "libc/CATS/fcntl.cats"
%}

staload "libc/SATS/fcntl.sats"

implement main() = let
  val (pf_fh | fh) = open_flag_exn("test.txt", O_RDONLY)
  var !p_buf with pf_buf = @[byte][4]()
  val n = read_exn(pf_fh | fh, !p_buf, 3)
  val () = bytes_strbuf_trans(pf_buf | p_buf, n)
  val () = print_strbuf(!p_buf)
  val () = close_exn (pf_fh | fh)
in
  pf_buf := bytes_v_of_strbuf_v (pf_buf)
end

This uses the fcntl libc routines for which ATS provides wrappers. The call to open_flag_exn returns a file handle, fh, and a proof, pf_fh. This proof needs to be passed to calls to read from the file (to ensure we're reading from an open file) and it must be consumed to ensure the file is closed.

The @[byte][4]() syntax defines a four element array, allocated on the stack, containing bytes. It returns a pointer to the array, p_buf, and a proof, pf_buf. read_exn reads three bytes from the file into this array. The size of the array is known by the type system and the definition of read_exn ensures that the number of bytes we read isn't greater than the size of the array at compile time. For example, the following would be a compile error:

val n = read_exn(pf_fh | fh, !p_buf, 5)

We have an array of bytes but I want to deal with this as a string. ATS has a number of string types, including:

  • string: A non linear string that can only be free'd by the garbage collector
  • strbuf: A flat array of bytes where the last byte is null
  • strptr: A linear string which must be manually free'd

I want to later pass the data along to C functions that expect a null terminated C string. To do this I will convert the array of bytes into a strbuf. The function bytes_strbuf_trans converts the array in-place, adding a null terminator. The length of string is passed as an argument. The terminator is added immediately following this. That's what this line does:

val () = bytes_strbuf_trans(pf_buf | p_buf, n)

p_buf is the pointer to our buffer. pf_buf is the proof that says this is a pointer to an array of bytes. This proof is changed to be a proof that p_buf is a strbuf by the bytes_strbuf_trans call. The definition of this function shows that (from the ATS prelude):

fun bytes_strbuf_trans
  {m,n:nat | n < m} {l:addr}
  (pf: !b0ytes m @ l >> strbuf (m, n1) @ l | p: ptr l, n: size_t n)
  :<> #[n1: nat | n1 <= n] void

The type system again ensure that the size does not exceed the size of the original array we allocated on the stack. It also ensure's there is room in the buffer for the null terminator. If I change the call to the following I will get a compile error:

val () = bytes_strbuf_trans(pf_buf | p_buf, 4)

The {m,n:nat | n<m} in the bytes_strbuf_trans definition constrains the size I pass in to be less than the size of the buffer.

Now that we have a proof that p_buf is a strbuf we can use print_strbuf to print the resulting three characters read from the file. Once we've finished with the strbuf we have to change the proof back to say it's an array of bytes otherwise the type checker thinks we've consumed the byte array (as we have no proof for it). This is what the following line does:

pf_buf := bytes_v_of_strbuf_v (pf_buf)

Although this is a simple snippet of code and a simple task the type checking in ATS did save me from some errors in my larger program that this snippet can be used to demonstrate:

  • The read call must not read more data than the size of the buffer allows.
  • The conversion to a C string in-place means we need to read one less than the size of the buffer.
  • The file must be closed after use.

All these issues resulted in compile time errors.

Tags: ats 

2011-02-10

Git Conversion of Mozilla Central

The Mozilla repository for Firefox development, mozilla-central, is a mercurial repository. I prefer using git for version control so I usually do my Firefox development using a git mirror of the mozilla-central repository.

To convert the repository into git I used hg-fast-export. This has worked well over the last couple of years. Unfortunately I've noticed that my git mirror and the official mercurial repository differ slightly. The git mirror contains files that have been deleted or moved in the official repository and lately the mirror has stopped building due to some commits not being correctly converted. As a result I stopped updating my mirror and used mercurial directly.

I still prefer developing using git though so I tried the hg-git mercurial plugin. This took about six days on my laptop to do the conversion and peaked at 14GB of memory. The result was a git repository with a working tree that, according to diff, is an exact copy of the original mercurial repositories working tree. Another nice feature of the hg-git is that commit id's in the resulting git repository are the same across multiple runs of hg-git. This would allow other users to push and pull from a git repository that they converted independently.

To perform the conversion I installed hg-git and ran the following series of commands:

$ mkdir git
$ cd git
$ git init --bare
$ cd ..
$ hg clone https://hg.mozilla.org/mozilla-central hg
$ cd hg
$ hg bookmark -r default master
$ hg push ../git

Six days and 14GB later, the resulting git repository ended up being about 7GB in size. I did a git gc --aggressive on it to get this down to 200MB. Incremental updates of new mercurial commits are done with:

$ cd hg
$ hg pull -u
$ hg bookmark -f -r default master
$ hg push ../git

I've pushed a mirror of this to github at http://github.com/doublec/mozilla-central. I'm using this for development and will see if it is able to continue importing without diverging from the original mozilla-central source.

More information about the use of git for Mozilla development can be found in the followup posts I did on the subject:

Tags: mozilla  git 

2011-01-15

WebM Decoder in Flash using Alchemy

Ralph Hauwert has been posting on twitter about work he's done on getting WebM decoding to work in Flash by compiling the libvpx source code using Adobe's Alchemy technology.

Alchemy is a research project that allows compilation of C and C++ libraries into code that runs on the ActionScript virtual machine used by Flash. Ralph originally tweeted:

Had a quickshot porting #WebM codec to Flash 10 using libvpx and alchemy. Surprisingly, got a preliminary version to work.

He followed up with a post saying the intial performance of a 1920x1080 VP8 video with no audio was decoding to YUV at about 1.5 frames per second. He provides an image of a decoded frame.

This is good stuff and it'll be worth following his twitter feed if you're interested in a pure flash implementation of a WebM decoder.

I've been interested in having decoders written in JavaScript, which I wrote about in my reading ogg files with JavaScript post. Maybe something similar to the Flash/Alchemy approach could be done with emscripten, an LLVM to JavaScript compiler, to get a WebM decoder in JavaScript.

Tags: mozilla  flash  webm 

2011-01-14

Wasp Lisp Respository Moves to Github

Scott Dunlop, The author of Wasp Lisp, has moved development from its bzr based launchpad location to WaspVM on github.

Hopefully this will encourage more interest in the implementation with people forking and working on it. For an overview of Wasp Lisp you can read a couple of my previous posts on it:

I used to have a private git repository containing an import of Wasp Lisp and various branches of things I was working on. With the move to github I've rebased my changes on top of Scott's github repository and put them in the following branches in my github fork:

  • real_numbers: This adds support for floating point numbers to Wasp Lisp. Some of the numerical operators are overloaded to accept floats and support for floats added to the serialisation code so they can be sent to other nodes.
  • opengl: This was a start at adding support for OpenGL graphics. I never finished it but it's there in case I (or someone else) decides to hack on it again.
  • node_retry: This changes MOSREF drones so that they try to connect again if the connection to the console node breaks, or it can't connect in the first place.

They've been a couple of other thing I had tinkered with and these have been upstreamed already. The branches above are things that are either not ready for upstreaming or not suitable for other reasons.

Another Wasp project I did a while back was create a skeleton reference documentation for the Wasp Lisp functions and primitives, with some basic build instructions. I put this in a WaspVM-Articles github repository. It requires ASCIIDoc to build. A pre-generated version is available in waspvm-articles.pdf.

The documentation was created by using the 'waspdoc' tool to get the import and exports of the Wasp Lisp modules. I added the primitives by grepping/searching the VM source code by hand. Hopefully this can form the base of some up to date reference material.

You can see MOSREF in action from the older Mosquito Lisp system in these videos:

MOSREF was ported to Wasp Lisp by Scott Dunlop and is included in the source repository.

Tags: waspvm 

2010-12-16

Handling POST requests with Ur/Web

Ur/Web makes it easy to write web applications that have HTML forms and handle the posted data. It gets a bit trickier if you have a POST handler that does not have an HTML form in the web application but is instead called by an external service.

The following is a function that handles a POST request field containing a string. It puts that string in a database:

table messages : { Message : string }

fun notifyMessage data =
  dml (INSERT INTO messages (Message) VALUES ({[data.Message]}));
  return <xml></xml>

This notifyMessage function won't type check unless we have a corresponding function that has an HTML form that submits to it. Without an HTML form in the application that uses notifyMessage we get a compile error like the following:

post.ur:4:46-4:62: Couldn't prove field name disjointness
   Con 1:  [#Message = Basis.string]
   Con 2:  <UNIF:T::{Type}>
Hnormed 1:  [#Message = Basis.string]
Hnormed 2:  <UNIF:T::{Type}>
You may be using a disallowed attribute with an HTML tag.

The error is pointing to the fact that it can't work out the type of data.Message. The fix is to ensure we have an HTML form in the application with an action that submits to notifyMessage. Since this particular POST handler is being used by an external caller we need to create a dummy page with a form:

fun dummyForm () =
  return <xml>
    <body>
      <form>
        <textbox{#Message}/>
        <submit action={notifyMessage}/>
      </form>
    </body>
  </xml>

This enables the Ur/Web type checker to know that notifyMessage has POST data containing a 'message' field. dummyForm needs to be added to the .urs file containing the exported function signatures:

val dummyForm : unit -> transaction page

I've created an example application to test this in github. The main function for the application displays instructions on how to POST using curl and displays the messages currently stored in the database:

fun main () =
  rows <- listMessages ();
  return <xml><head>
    <title>Post Example</title>
  </head>
  <body>
    <p>Send a curl command like the following to add data
       to the messages table and refresh the page:</p>
    <p>curl -F "message=Hello" http://127.0.0.1:8080/notifyMessage</p>
    {rows}
  </body>
</xml>

It uses a listMessages helper function to do a SQL query on the table and outputs an HTML table row for each result in the query:

fun listMessages () =
  rows <- queryX (SELECT * FROM messages)
         (fn row => <xml><tr><td>{[row.Messages.Message]}</td></tr></xml>);
  return <xml><table>{rows}</table></xml>

To build, run and send post data using curl

$ git clone git://github.com/doublec/urweb-post-example
$ cd urweb-post-example
$ make
$ ./post.exe
$ curl -F "message=hello" http://127.0.0.1:8080/notifyMessage

To display posted messages go to http://localhost:8080/main in a web browser.

The downside with this workaround is we end up with a dummyForm page that is publically explosed. This could be prevented from being served by putting the web application behind a reverse proxy which doesn't expose it. This issue and workaround is a result from a discussion on the mailing list about POST.

Another issue that popped up during that discussion was that Ur/Web can only handle POSTing of HTML form data. If the Content-Type of the data is something else (like application/json) then it can't be handled with the current release of Ur/Web without going through some form of intermediate proxy to convert the data to what Ur/Web expects.

Hopefully these can be resolved in a later Ur/Web release.

Update 2010-12-19: The issues above look to be solved now according to the Ur/Web bug tracker.

Tags: urweb 


This site is accessable over tor as hidden service 6vp5u25g4izec5c37wv52skvecikld6kysvsivnl6sdg6q7wy25lixad.onion, or Freenet using key:
USK@1ORdIvjL2H1bZblJcP8hu2LjjKtVB-rVzp8mLty~5N4,8hL85otZBbq0geDsSKkBK4sKESL2SrNVecFZz9NxGVQ,AQACAAE/bluishcoder/-61/


Tags

Archives
Links