Bluish Coder: pure

Vorbis Player implemented in Pure

2010-02-25T14:00:00+13:00

After writing the preforking echo server example in Pure I wanted to try something a bit more complex. I've written Ogg players in various languages before so I decided to do one in Pure to compare. Here's some of the ones I've written in the past:

oggplayer
plogg
Io Language Addons
Factor Ogg Player
An unreleased Haskell Vorbis player

I wrapped the libogg and libvorbis libraries using the Pure FFI. A program 'pure-gen' is included with Pure that will generate FFI wrappers given a C include file. I ran this against libogg and libvorbis. The generated Pure wrappers look like:

extern int ogg_page_version(ogg_page*);
extern int ogg_page_continued(ogg_page*);
extern int ogg_page_bos(ogg_page*);

As you can see Pure FFI definitions look very much like the C definitions.

I decided to model an Ogg file as a list of physical pages. This is esentially a list containing elements of the libogg ogg_page object. I have a Pure function that will take this list of pages and return a list of packets for each logical bitstream in the file. For vorbis playback I take this list of packets for the vorbis bitstream and produce a list of decoded PCM data.

Pure is a strict language by default but provides an '&' special form to perform lazy evaluation. I use this to make sure all the returned lists mentioned above are lazy. Playing the lazy list of decoded PCM data never loads the entire file in memory. The file is read, converted into packets and decoded as each element of the PCM data list is requested.

For audio playback I use OpenAL. I queue the PCM data to an OpenAL source up to a limit of 50 queued items, removing them as they get processed and adding new ones when room becomes available.

Here's what some of the API to process the Ogg files looks like in Pure. First load the player code. This will also load the Ogg, Vorbis and OpenAL wrappers:

$ pure -i player.pure

From the Pure REPL, play a vorbis file:

> play_vorbis (sources!0) "test.ogg";

Load an Ogg file getting a lazy list of all pages in the file:

> using namespace ogg;
> let p = file_pages "test.ogg";
> p;
ogg_page (#<pointer 0x9e61cc8>,[...]):#<thunk 0xb7259b38>

Get a list of all packets and the serial number of the first logical bitstream in the list of pages:

> let (stream1,serialno) = packets p;
> serialno;
1426627898
> stream1;
ogg_packet (#<pointer 0x9dd98f0>,[...]):#<thunk 0xb725bce0>

Get the next logical bitstream:

> let p2 = filter (\x->ogg_page_serialno x ~= serialno) p;
> p2;
#<thunk 0xb725dc30>
> let (stream2,serialno2) = packets p2;
> serialno2;
629367739
> stream2;
ogg_packet (#<pointer 0x9cfcf08>,[...]):#<thunk 0xb725c988>

Note that all these results are lazily evaluated so the entire file is not loaded into memory. If we look at the original list of pages you'll see what has been read so far (two pages):

> p;
ogg_page (...):ogg_page (...):#<thunk 0xb725dd98>

Get all the streams in the file:

> let s = all_streams p;

Get all the vorbis streams:

> let v = vorbis_streams s;

Get the first vorbis stream. I ignore the serial number here (this is what the _ does):

> let (v1, _) = v!0;

Decode the vorbis header packets and intialize a vorbis decoder for the first vorbis stream:

> let decoder = vorbis_header v1;

Get a lazy list of all the decoded PCM data from the vorbis stream:

> let pcm = pcm_data decoder;

The mechanics of getting the decoding floating point data from libvorbis into Pure and then back to the OpenAL C API is made easier due to it being possible to easily convert to and from raw C pointers and Pure matrix objects.

For example, the data returned from the vorbis call to decode the data is an array of pointers pointing to an array of floats. To convert this to a Pure matrix with dimensions (c,n) where 'c' is the number of channels and 'n' is the number of samples:

vorbis_synthesis_read dsp samples $$ matrix m
when
  f0 = get_pointer data;
  floats = [get_pointer (f0+(x*4))|x=0..(channels-1)];
  m = map (float_matrix samples) floats;
end;

This code uses a list comprehension to return a list of 'pointer to array of floats'. One pointer for each channel in the audio data. It then maps over this list producing a new list containing a matrix of float's. This is list of matrices each of which contains the audio data for a channel. The 'matrix m' call converts this to a single matrix of dimensions (c,n).

OpenAL expects the data as a pointer to a contiguous matrix of floats with dimensions (n,c) so when writing to OpenAL the matrix needs to be transposed:

play_pcm source rate pcm@(x:xs) =
  ....
  queue_sample source rate (transpose x) $$
  play source $$
  play_pcm source rate xs;

This transposed matrix is later converted from floats to 16 bit integers and 'packed' so it is contiguous in memory, and a pointer to the raw memory passed to OpenAL:

buffer16 s = pack (map clamp s);  
...
data = buffer16 ...pcmdata...;
alBufferData ... (cooked (short_pointer NULL data)) ((#data) * (sizeof sshort_t)) ...;

The 'cooked' call means the C memory has 'free' called on it when nothing in Pure is holding onto it anymore.

One downside with lazy lists is it can be easy to accidently read the entire file and decode everything before playing. This will cause issues with large Ogg files. I managed to avoid that and files play in constant memory. An iteratee style approach might be interesting to try as an alternative to lazy lists. I'd like to try decoding Theora files and doing a/v sync in Pure at some point as well.

I haven't used Pure much but it seemed to be fairly simple to put this together. I'd be interested in feedback on the approach (using lazy lists) and Pure style issues in the code.

The code for this is at http://github.com/doublec/pure-ogg-player.

Pure Preforking Echo Server Example

2010-02-20T23:50:00+13:00

A few months back there were some examples of preforking servers implemented in various languages showing how posix API's could be used in those languages. Some examples were:

I'm learning the Pure programming language and took a stab at doing a Pure implementation. Pure comes with a good FFI for calling C functions and the needed posix calls are provided in the 'posix' library that comes with it.

It doesn't have any socket routines however but there is a library by Mike Maul called pure-lang-extras that provides this. With pure-lang-extras installed the preforking echo server looks like this:

using ffi, system;
using posix, posix::socket;
using namespace posix, posix::socket;

let sockfd = socket AF_INET SOCK_STREAM 0;
let yes = {1};
setsockopt sockfd SOL_SOCKET SO_REUSEADDR (int_pointer yes) (sizeof sint_t) 0;
let ai = make_sockaddr AF_INET "*" 5000;
bind_socket sockfd ai;
listen sockfd 10;

fork_child f = if pid == 0 then f pid $$ exit 0 else pid when pid = fork end;

child pid  =  fprintf file "Child %d echo> " getpid $$
              fflush file $$
              fprintf file "%s\n" (fgets file) $$
              fclose file $$
              child pid 
              when
                client = accept_socket sockfd ai;
                file = fdopen client "r+";
              end;

let children = [fork_child child|x=0..2];
do (printf "Forked: %d\n") children;
do (\n->(waitpid n NULL 0)) children;

This can be run (assuming it's in a file called 'test.pure') with:

$ pure -x test.pure
Forked: 10433
Forked: 10434
Forked: 10435

It will fork three worker processes all waiting on the same socket provided by the main process. The main process waits until all three worker processes complete or are killed. You can test it out by using telnet to port 5000:

$ telnet localhost 5000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Child 10433 echo> test test test
test test test
Connection closed by foreign host.

If you do this a few times you'll see it cycle through the worker processes.

The example code mostly does posix calls and thin wrapper's around C functions so it doesn't show off much of the interesting parts of Pure. Pure has automatic currying (or partial evaluation) of functions by missing arguments from the function. For example, this code:

do (printf "Forked: %d\n") children;

printf is like the C printf function. It takes a format string followed by a tuple containing arguments for each format directive in that string. In the example above it has one of these, '%d'. The code (printf "Forked: %d\n") is missing the last argument expected to printf. Thanks to partial evaluation this evaluates to a function that takes one argument. do calls this function for each element in the children list, passing the list element to the function resulting in the printf call. It's equivalent to this more verbose code:

do (\n -> printf "Forked: %d\n" n) children;

Another Pure feature used is the ability to get raw pointers to matrices. The code {1} is a one row matrix containing one element, the interger '1'. A later line gets a raw pointer to this, converting it to a array of integers if it isn't already in that format: int_pointer yes.

To fork the child processes I use a list comprehension:

let children = [fork_child child|x=0..2];

0..2 is a short hand syntax to create a list of numbers from 0 to 2. The [...|....] is a list comprehension. The left hand side is called for each element of the right hand side and the result accumulated in a list. For example, at the Pure REPL:

> [x*2|x=0..5];
[0,2,4,6,8,10]

For more on Pure there is quite a bit of documentation:

Dealing with my archived Blogger posts

2010-02-14T16:54:00+13:00

I've stopped using Blogger for the weblog and have moved to using Jekyll.

I've set things up so I can edit the posts on my local push, push to my server using git and it publishes the new post.

Blogger provides a way to export all existing posts and comments as an XML file. I used this to manually import a few posts to test out Jekyll but then decided this was way to much work to convert everything. Instead I opted to create posts that link to my original Blogger ones so at least the archives and tags list in Jekyll will allow listing the posts and titles.

To import the existing archived blogger posts I wrote a program to read from the Blogger export file, extract the title, post tags/categories and existing URL and create Jekyll equivalents. It seems to have worked ok and I'll manually fix up any problems if I find them.

I wrote the importer in the Pure Programming Language to have a play with that language. I dabbled with it when it first came out but this was my first time using it in anger. It worked out pretty well. The core of the code to write out the posts looks like:

write_tag_header fd [] = ();
write_tag_header fd (x:xs) = fprintf fd "tags:\n" ();

write_tags fd (x:xs) = fprintf fd "  - %s\n" x $$ 
                       write_tags_fd xs;
write_tags fd [] = ();

create_post post = fprintf fd "---\n" () $$
                   fprintf fd "layout: post\n" () $$
                   fprintf fd "title: %s\n" title $$
                   write_tag_header fd tags $$
                   write_tags fd tags $$
                   fprintf fd "---\n" () $$
                   fprintf fd "Original Post [%s](%s)\n" (title,href) $$
                   fclose fd
                   when
                       filename = post_filename post;
                       fd = fopen filename "w";
                       title = article_title post;
                       href = article_url post;
                       tags = article_tags post;
                  end;

Basically I used the Pure XML library to get a list of posts (I use an XPATH expression to get the relevant parts of the post from the Blogger XML). For each post I then call create_post on it:

map create_post all_posts

I didn't go crazy and convert the actual HTML content into the markdown format I'm using fo the Jekyll posts and convert everything completely - maybe a task for another day.