Bluish Coder: ogg

Vorbis Player implemented in Pure

2010-02-25T14:00:00+13:00

After writing the preforking echo server example in Pure I wanted to try something a bit more complex. I've written Ogg players in various languages before so I decided to do one in Pure to compare. Here's some of the ones I've written in the past:

oggplayer
plogg
Io Language Addons
Factor Ogg Player
An unreleased Haskell Vorbis player

I wrapped the libogg and libvorbis libraries using the Pure FFI. A program 'pure-gen' is included with Pure that will generate FFI wrappers given a C include file. I ran this against libogg and libvorbis. The generated Pure wrappers look like:

extern int ogg_page_version(ogg_page*);
extern int ogg_page_continued(ogg_page*);
extern int ogg_page_bos(ogg_page*);

As you can see Pure FFI definitions look very much like the C definitions.

I decided to model an Ogg file as a list of physical pages. This is esentially a list containing elements of the libogg ogg_page object. I have a Pure function that will take this list of pages and return a list of packets for each logical bitstream in the file. For vorbis playback I take this list of packets for the vorbis bitstream and produce a list of decoded PCM data.

Pure is a strict language by default but provides an '&' special form to perform lazy evaluation. I use this to make sure all the returned lists mentioned above are lazy. Playing the lazy list of decoded PCM data never loads the entire file in memory. The file is read, converted into packets and decoded as each element of the PCM data list is requested.

For audio playback I use OpenAL. I queue the PCM data to an OpenAL source up to a limit of 50 queued items, removing them as they get processed and adding new ones when room becomes available.

Here's what some of the API to process the Ogg files looks like in Pure. First load the player code. This will also load the Ogg, Vorbis and OpenAL wrappers:

$ pure -i player.pure

From the Pure REPL, play a vorbis file:

> play_vorbis (sources!0) "test.ogg";

Load an Ogg file getting a lazy list of all pages in the file:

> using namespace ogg;
> let p = file_pages "test.ogg";
> p;
ogg_page (#<pointer 0x9e61cc8>,[...]):#<thunk 0xb7259b38>

Get a list of all packets and the serial number of the first logical bitstream in the list of pages:

> let (stream1,serialno) = packets p;
> serialno;
1426627898
> stream1;
ogg_packet (#<pointer 0x9dd98f0>,[...]):#<thunk 0xb725bce0>

Get the next logical bitstream:

> let p2 = filter (\x->ogg_page_serialno x ~= serialno) p;
> p2;
#<thunk 0xb725dc30>
> let (stream2,serialno2) = packets p2;
> serialno2;
629367739
> stream2;
ogg_packet (#<pointer 0x9cfcf08>,[...]):#<thunk 0xb725c988>

Note that all these results are lazily evaluated so the entire file is not loaded into memory. If we look at the original list of pages you'll see what has been read so far (two pages):

> p;
ogg_page (...):ogg_page (...):#<thunk 0xb725dd98>

Get all the streams in the file:

> let s = all_streams p;

Get all the vorbis streams:

> let v = vorbis_streams s;

Get the first vorbis stream. I ignore the serial number here (this is what the _ does):

> let (v1, _) = v!0;

Decode the vorbis header packets and intialize a vorbis decoder for the first vorbis stream:

> let decoder = vorbis_header v1;

Get a lazy list of all the decoded PCM data from the vorbis stream:

> let pcm = pcm_data decoder;

The mechanics of getting the decoding floating point data from libvorbis into Pure and then back to the OpenAL C API is made easier due to it being possible to easily convert to and from raw C pointers and Pure matrix objects.

For example, the data returned from the vorbis call to decode the data is an array of pointers pointing to an array of floats. To convert this to a Pure matrix with dimensions (c,n) where 'c' is the number of channels and 'n' is the number of samples:

vorbis_synthesis_read dsp samples $$ matrix m
when
  f0 = get_pointer data;
  floats = [get_pointer (f0+(x*4))|x=0..(channels-1)];
  m = map (float_matrix samples) floats;
end;

This code uses a list comprehension to return a list of 'pointer to array of floats'. One pointer for each channel in the audio data. It then maps over this list producing a new list containing a matrix of float's. This is list of matrices each of which contains the audio data for a channel. The 'matrix m' call converts this to a single matrix of dimensions (c,n).

OpenAL expects the data as a pointer to a contiguous matrix of floats with dimensions (n,c) so when writing to OpenAL the matrix needs to be transposed:

play_pcm source rate pcm@(x:xs) =
  ....
  queue_sample source rate (transpose x) $$
  play source $$
  play_pcm source rate xs;

This transposed matrix is later converted from floats to 16 bit integers and 'packed' so it is contiguous in memory, and a pointer to the raw memory passed to OpenAL:

buffer16 s = pack (map clamp s);  
...
data = buffer16 ...pcmdata...;
alBufferData ... (cooked (short_pointer NULL data)) ((#data) * (sizeof sshort_t)) ...;

The 'cooked' call means the C memory has 'free' called on it when nothing in Pure is holding onto it anymore.

One downside with lazy lists is it can be easy to accidently read the entire file and decode everything before playing. This will cause issues with large Ogg files. I managed to avoid that and files play in constant memory. An iteratee style approach might be interesting to try as an alternative to lazy lists. I'd like to try decoding Theora files and doing a/v sync in Pure at some point as well.

I haven't used Pure much but it seemed to be fairly simple to put this together. I'd be interested in feedback on the approach (using lazy lists) and Pure style issues in the code.

The code for this is at http://github.com/doublec/pure-ogg-player.

Playing Ogg files with audio and video in sync

2009-06-27T22:07:00+12:00

My last post in this series had Vorbis audio playing but with Theora video out of sync. This post will go through an approach to keeping the video in sync with the audio.

To get video in sync with the audio we need a timer incrementing from when we start playback. We can't use the system clock for this as it is not necessarily keeping the same time as the audio or video being played. The system clock can drift slightly and over time this audio and video to get out of sync.

The audio library I'm using, libsydneyaudio, has an API call that allows getting the playback position of the sound sample being played by the audio system. This is a value in bytes. Since we know the sample rate and number of channels of the audio stream we can compute a time value from this. Synchronisation becomes a matter of continuously feeding the audio to libsydneybackend, querying the current position, converting it to a time value, and displaying the frame for that time.

The time for a particular frame is returned by the call to th_decode_packetin. The last parameter is a pointer to hold the 'granulepos' of the decoded frame. The Theora spec explains that the granulepos can be used to compute the time that this frame should be displayed up to. That is, when this time is exceeded this frame should no longer be displayed. It also enables computing the location of the keyframe that this frame depends on - I'll cover what this means when I write about how to do seeking.

The libtheora API th_granule_time converts a 'granulepos' to an absolute time in seconds. So decoding a frame gives us 'granulepos'. We store this so we know when to stop displaying the frame. We track the audio position, convert it to a time. If it exceeds this value we decode the next frame and display that. Here's a breakdown of the steps: * Read the headers from the Ogg file. Stop when we hit the first data packet. * Read packets from the audio stream in the Ogg file. For each audio packet:

<ul>
<li>Decode the audio data and write it to the audio hardware.</li>
<li>Get the current playback position of the audio and convert it to an absolute time value.</li>
<li>Convert the last granulepos read (defaulting to zero if none have been read) to an absolute time value using the libtheora API.</li>
<li>If the audio time is greater than the video time:   
    <ol>
      <li>Read a packet from the Theora stream.</li>
      <li>Decode that packet and display it</li>
      <li>Store the granulepos from that decoded frame so we know when to display the next frame.</li>
    </ol>
</li>
</ul>

Notice that the structure of the program is different to the last few articles. We no longer read all packets from the stream, processing them as we get them. Instead we specifically process the audio packets and only handle the video when it's time to display them. Since we are driving our a/v sync off the audio clock we must continously feed the audio data. I think it tends to be a better user experience to have flawless audio with video frame skipping rather than skipping audio but smooth video. Worse is to have both skipping of course.

The example code for this article is in the 'part4_avsync' branch on github.

This example takes a slightly different approach to reading headers. I use ogg_stream_packetpeek to peek ahead in the bitstream for a packet and do the header processing on the peeked packet. If it is a header I then consume the packet. This is done so I don't consume the first data packet when reading the headers. I want the data packets to be consumed in a particular order (audio, followed by video when needed).

// Process all available header packets in the stream. When we hit
// the first data stream we don't decode it, instead we
// return. The caller can then choose to process whatever data
// streams it wants to deal with.
ogg_packet packet;
while (!headersDone &&
       (ret = ogg_stream_packetpeek(&stream->mState, &packet)) != 0) {
assert(ret == 1);

// A packet is available. If it is not a header packet we exit.
// If it is a header packet, process it as normal.
headersDone = headersDone || handle_theora_header(stream, &packet);
headersDone = headersDone || handle_vorbis_header(stream, &packet);
if (!headersDone) {
  // Consume the packet
  ret = ogg_stream_packetout(&stream->mState, &packet);
  assert(ret == 1);
}

To read packets for a particular stream I use a 'read_packet' function that operates on a stream passed as a parameter:

bool OggDecoder::read_packet(istream& is, 
                             ogg_sync_state* state, 
                             OggStream* stream, 
                             ogg_packet* packet) {
  int ret = 0;
  while ((ret = ogg_stream_packetout(&stream->mState, packet)) != 1) {
    ogg_page page;
    if (!read_page(is, state, &page))
      return false;

    int serial = ogg_page_serialno(&page);
    assert(mStreams.find(serial) != mStreams.end());
    OggStream* pageStream = mStreams[serial];

    // Drop data for streams we're not interested in.
    if (stream->mActive) {
      ret = ogg_stream_pagein(&pageStream->mState, &page);
      assert(ret == 0);
    }
  }
  return true;
}

If we need to read a new page (to be able to get more packets) we check the stream for the read page and if it is not for the stream we want we store the packet in the bitstream for that page so it can be retrieved later. I've added an 'active' flag to the streams so we can ignore streams that we aren't intersted in. We don't want to continuously buffer data for alternative audio tracks we aren't playing for example. The streams are marked inactive when the headers are finished reading.

The code that does the checking to see if it's time to display a frame is:

// At this point we've written some audio data to the sound
// system. Now we check to see if it's time to display a video
// frame.
//
// The granule position of a video frame represents the time
// that that frame should be displayed up to. So we get the
// current time, compare it to the last granule position read.
// If the time is greater than that it's time to display a new
// video frame.
//
// The time is obtained from the audio system - this represents
// the time of the audio data that the user is currently
// listening to. In this way the video frame should be synced up
// to the audio the user is hearing.
//
ogg_int64_t position = 0;
int ret = sa_stream_get_position(mAudio, SA_POSITION_WRITE_SOFTWARE, &position);
assert(ret == SA_SUCCESS);
float audio_time = 
  float(position) /
  float(audio->mVorbis.mInfo.rate) /
  float(audio->mVorbis.mInfo.channels) /
  sizeof(short);

float video_time = th_granule_time(video->mTheora.mCtx, mGranulepos);
if (audio_time > video_time) {
  // Decode one frame and display it. If no frame is available we
  // don't do anything.
  ogg_packet packet;
  if (read_packet(is, &state, video, &packet)) {
    handle_theora_data(video, &packet); 
    video_time = th_granule_time(video->mTheora.mCtx, mGranulepos);
  }
}

The code for decoding and display the Theora video is similar to the Theora decoding article. The main difference is we store the granulepos in mGranulepos so we know when to stop displaying the frame.

This version of 'plogg' should play Ogg files with a Theora and Vorbis track in sync. It does not play Theora files with no audio track - we can't synchronise to the audio clock if there is no audio. This can be worked around by falling back to delaying for the required framerate as the previous Theora example did.

The a/v sync is not perfect however. If the video is large and decoding keyframes takes a while then we can fall behind in displaying the video and go out of sync. This is because we only play one frame when we check the time. One approach to fixing this is to decode, but not display, all frames up until the audio time rather than just the next time.

The other issue is that the API call we are using to write to the audio hardware is blocking. This is using up valuable time that we could be using to decode a frame. When the write to the sound hardware returns we have very little time to decode a frame before glitches start appearing in the audio due to buffer underruns. Try playing a larger video and the audio and video will skip (depending on the speed of your hardware). This isn't a pleasant experience. Because of the blocking audio writes we can't skip more than one frame due to the frame decoding time taking too long causing audio skip.

The fixes for these aren't too complex and I'll go through it in my next article. The basic approach is to move to an asynchronous method of writing the audio, skip displaying frames when needed (to reduce the cost of the YUV decoding), skip decoding frames if possible (depending on location of keyframes we can do this), and to check how much audio data we have queued before decoding to always ensure we won't drop audio while decoding.

With these fixes in place I can play the 1080p Ogg version of Big Buck Bunny on a Macbook laptop (running Arch Linux) with no audio interruption and with a/v syncing correctly. There is a fair amount of frame skipping however but it's a lot more watchable than if you try playing it without these modifications in place. And better than watching with the video lagging further and further behind the longer you watch it. Further improvements can be made to reduce the frame skipping by utilising threads to take advantage of extra core's on the PC.

After the followup article on improving the a/v sync I'll look at covering seeking.

Decoding Vorbis files with libvorbis

2009-06-26T14:08:00+12:00

Decoding Vorbis streams require a very similar approach to that used when decoding Theora streams. The public interface to the libvorbis library is very similar to that used by libtheora. Unfortunately the libvorbis documentation doesn't contain an API reference that I could find so I'm following the approached used by the example programs.

Assuming we have already obtained an ogg_packet, the general steps to follow to decode and play Vorbis streams are:

Call vorbis_synthesis_headerin to see if the packet is a Vorbis header packet. This is passed a vorbis_info and vorbis_comment object to hold the information read from those header packets. The return value of this function is zero if the packet is a Vorbis header packet. Unfortunately it doesn't return a value to see that it's a Vorbis data packet. To check for this you need to check if the stream is a Vorbis stream (by knowing you've previously read Vorbis headers from it) and the return value is OV_ENOTVORBIS.
Once all the header packets are read create a vorbis_dsp_state and vorbis_block object. Initialize these with vorbis_synthesis_init and vorbis_block_init respectively. These objects hold the state of the Vorbis decoding. vorbis_synthesis_init is passed the vorbis_info object that was filled in during the reading of the header packets.
For each data packet:
1. Call vorbis_synthesis passing it the vorbis_block created above and the ogg_packet containing the packet data. If this succeeds (by returning zero), call vorbis_synthesis_blockin passing it the vorbis_dsp_state and vorbis_block objects. This call copies the data from the packet into the Vorbis objects ready for decoding.
2. Call vorbis_synthesis_pcmout to get an pointer to an array of floating point values for the sound samples. This will return the number of samples in the array. The array is indexed by channel number, followed by sample number. Once obtained this sound data can be sent to the sound hardware to play the audio.
3. Call vorbis_synthesis_read, passing it the vorbis_dsp_state object and the number of sound samples consumed. This allows you to consume less data than vorbis_synthesis_pcmout returned. This is useful if you can't write all the data to the sound hardware without blocking.

In the example code in the github repository I create a VorbisDecode object that holds the objects needed for decoding. This is similar to the TheoraDecode object mentioned in my Theora post:

class VorbisDecode {
  ...
  vorbis_info mInfo;
  vorbis_comment mComment;
  vorbis_dsp_state mDsp;
  vorbis_block mBlock;
  ...
      VorbisDecode()
  {
    vorbis_info_init(&mInfo);
    vorbis_comment_init(&mComment);    
  }
};

I added a TYPE_VORBIS value to the StreamType enum and the stream is set to this type when a Vorbis header is successfully decoded:

  int ret = vorbis_synthesis_headerin(&stream->mVorbis.mInfo,
                      &stream->mVorbis.mComment,
                      packet);
  if (stream->mType == TYPE_VORBIS && ret == OV_ENOTVORBIS) {
    // First data packet
    ret = vorbis_synthesis_init(&stream->mVorbis.mDsp, &stream->mVorbis.mInfo);
    assert(ret == 0);
    ret = vorbis_block_init(&stream->mVorbis.mDsp, &stream->mVorbis.mBlock);
    assert(ret == 0);
    stream->mHeadersRead = true;
    handle_vorbis_data(stream, packet);
  }
  else if (ret == 0) {
    stream->mType = TYPE_VORBIS;
  }
}

The example program uses libsydneyaudio for audio output. This requires sound samples to be written as signed short values. When I get the floating point data from Vorbis I convert this to signed short and send it to libsydneyaudio:

  int ret = 0;

  if (vorbis_synthesis(&stream->mVorbis.mBlock, packet) == 0) {
    ret = vorbis_synthesis_blockin(&stream->mVorbis.mDsp, &stream->mVorbis.mBlock);
    assert(ret == 0);
  }
  float** pcm = 0;
  int samples = 0;
  while ((samples = vorbis_synthesis_pcmout(&stream->mVorbis.mDsp, &pcm)) > 0) {
    if (!mAudio) {
      ret = sa_stream_create_pcm(&mAudio,
                 NULL,
                 SA_MODE_WRONLY,
                 SA_PCM_FORMAT_S16_NE,
                 stream->mVorbis.mInfo.rate,
                 stream->mVorbis.mInfo.channels);
      assert(ret == SA_SUCCESS);

      ret = sa_stream_open(mAudio);
      assert(ret == SA_SUCCESS);
    }

    if (mAudio) {
      short buffer[samples * stream->mVorbis.mInfo.channels];
      short* p = buffer;
      for (int i=0;i < samples; ++i) {
        for(int j=0; j < stream->mVorbis.mInfo.channels; ++j) {
          int v = static_cast<int>(floorf(0.5 + pcm[j][i]*32767.0));
          if (v > 32767) v = 32767;
          if (v <-32768) v = -32768;
          *p++ = v;
        }
      }

      ret = sa_stream_write(mAudio, buffer, sizeof(buffer));
      assert(ret == SA_SUCCESS);
    }

    ret = vorbis_synthesis_read(&stream->mVorbis.mDsp, samples);
    assert(ret == 0);
  }
}

A couple of minor changes were also made to the example program: 1. Continue processing pages and packets when the 'end of file' is reached. Otherwise a few packets that are buffered after we've reached the end of the file will be missed. 2. After reading a page don't just try to read one packet, call ogg_stream_packetout until it returns a result saying there are no more packets. This means we process all the packets from the page immediately and prevents a build up of buffered data.

The code for this example is in the 'part3_vorbis' branch of the github repository. This also includes the Theora code but does not do any a/v synchronisation. Files containing Theora streams will show the video data but it will not play smoothly and will not be synchronised with the audio. Fixing that is the topic of the next post in this series.

Decoding Theora files using libtheora

2009-06-25T20:33:00+12:00

My last post covered read Ogg files using libogg. The resulting program didn't do much but it covered the basic steps needed to get an ogg_packet which we need to decode the data in the stream. The thing step I want to cover is decoding Theora streams using libtheora.

In the previous post I stored a count of the number of packets in the OggStream object. For theora decoding we need a number of different objects to be stored. I encapsulate this in a TheoraDecode structure:

class TheoraDecode { 
  ...
  th_info mInfo;
  th_comment mComment;
  th_setup_info *mSetup;
  th_dec_ctx* mCtx;
  ...
};

th_info, th_comment and th_setup_info contain data read from the Theora headers. The Theora stream contains three headers packets. These are the info, comment and setup headers. There is one object for holding each of these as we read the headers. The th_dec_ctx object holds information that the decoder requires to keep track of the decoding process.

th_info and th_comment need to be initialized using th_info_init and th_comment_init. Notice that th_setup_info is a pointer. This needs to be free'd when we're finished with it using th_setup_free. The decoder context object also needs to be free'd. Use th_decode_free. A convenient place to do this is in the TheoraDecode constructor and destructor:

class TheoraDecode {
  ...
  TheoraDecode() :
    mSetup(0),
    mCtx(0)
  {
    th_info_init(&mInfo);
    th_comment_init(&mComment);
  }

  ~TheoraDecode() {
    th_setup_free(mSetup);
    th_decode_free(mCtx);
  }   
  ...

The TheoraDecode object is stored in the OggStream structure. The OggStream stucture also gets a field holding the type of the stream (Theora, Vorbis, Unknown, etc) and a boolean indicating whether the headers have been read:

class OggStream
{
  ...
  int mSerial;
  ogg_stream_state mState;
  StreamType mType;
  bool mHeadersRead;
  TheoraDecode mTheora;
  ...
};

Once we get the ogg_packet from an Ogg stream we need to find out if it is a Theora stream. The approach I'm using to do this is to attempt to extract a Theora header from it. If this succeeds, it's a Theora stream. th_decode_headerin will attempt to decode a header packet. A return value of '0' indicates that we got a Theora data packet (presumably the headers have been read already). This function gets passed the info, comment, and setup objects and it will populate them with data as it reads the headers:

ogg_packet* packet = ...got this previously...;
int ret = th_decode_headerin(&stream->mTheora.mInfo,
                             &stream->mTheora.mComment,
                             &stream->mTheora.mSetup,
                             packet);
if (ret == TH_ENOTFORMAT)
  return; // Not a theora header

if (ret > 0) {
  // This is a theora header packet
  stream->mType = TYPE_THEORA;
  return;
}

assert(ret == 0);
// This is not a header packet. It is the first 
// video data packet.
stream->mTheora.mCtx = 
  th_decode_alloc(&stream->mTheora.mInfo, 
                  stream->mTheora.mSetup);
assert(stream->mTheora.mCtx != NULL);
stream->mHeadersRead = true;
...decode data packet...

In this example code we attempt to decode the header. If it fails it bails out, possibly to try decoding the packet using libvorbis or some other means. If it succeeds the stream is marked as type TYPE_THEORA so we can handle it specially later.

If all headers packets are read and we got the first data packet then we call th_decode_alloc to get a decode context to decode the data.

Once the headers are all read, the next step is to decode each Theora data packet. To do this we first call th_decode_packetin. This adds the packet to the decoder. A return value of '0' means we can get a decoded frame as a result of adding the packet. A call to th_decode_ycbcr_out gets the decoded YUV data, stored in a th_ycbcr_buffer object. This is basically an array of the YUV data.

ogg_int64_t granulepos = -1;
int ret = th_decode_packetin(stream->mTheora.mCtx,
                             packet,
                             &granulepos);
assert(ret == 0);

th_ycbcr_buffer buffer;
ret = th_decode_ycbcr_out(stream->mTheora.mCtx, buffer);
assert(ret == 0);
...copy yuv data to SDL YUV overlay...
...display overlay...
...sleep for 1 frame...

The 'granulepos' returned by the th_decode_packetin call holds information regarding the presentation time of this frame, and what frame contains the keyframe that is needed for this frame if it is not a keyframe. I'll write more about this in a future post when I cover synchronising the audio and video. For now it's going to be ignored.

Once we have the YUV data I use SDL to create a surface, and a YUV overlay. This allows SDL to do the YUV to RGB conversion for me. I won't copy the code for this since it's not particularly relevant to using the libtheora API - you can see it in the github repository.

Once the YUV data is blit to the screen the final step is to sleep for the period of one frame so the video can playback at approximately the right framerate. The framerate of the video is stored in the th_info object that we got from the headers. It is represented as the fraction of two numbers:

float framerate = 
  float(stream->mTheora.mInfo.fps_numerator) / 
  float(stream->mTheora.mInfo.fps_denominator);
SDL_Delay((1.0/framerate)*1000);

With all that in place, running the program with an Ogg file containing a Theora stream should play the video at the right framerate. Adding Vorbis playback is almost as easy - the main difficulty is synchronising the audio and video. I'll cover these topics in a later post.

Reading Ogg files using libogg

2009-06-24T23:49:00+12:00

Reading data from an Ogg file is relatively simple. The file format is well documented in RFC 3533. I showed how to read the format using JavaScript in a previous post.

For C and C++ programs it's easier to use the xiph.org libraries. There are libraries for decoding specific formats (libvorbis, libtheora) and there is a library for reading data from Ogg files (libogg).

I'm prototyping some approaches to improve the performance of the Firefox Ogg video playback and while I'm at it I'll write some posts on using these libraries to decode/play Ogg files. Hopefully it'll prove useful to others using them and I can get some feedback on usage.

All the code for this is in the plogg git repository on github. The 'master' branch contains the work in progress player that I'll describe in a series of posts, and there are branches specific to the examples in each post.

The libogg documentation describes the API that I'll be using in this post. All that this example will do is read an Ogg file, read each stream in the file and count the number of packets for that stream. It prints the number of packets. It doesn't decode the data or do anything really useful. That'll come later.

You can think of an Ogg file as containing logical streams of data. Each stream has a serial number that is unique within the file to identify it. A file containing Vorbis and Theora data will have two streams. A Vorbis stream and a Theora stream.

Each stream is split up into packets. The packets contain the raw data for the stream. The process of decoding a stream involves getting a packet from it, decoding that data, doing something with it, and repeating.

That describes the logical format. The physical format of the Ogg file is split into pages of data. Each physical page contains some part of the data for one stream.

The process of reading and decoding an Ogg file is to read pages from the file, associating them with the streams they belong to. At some point we then go through the pages held in the stream and obtain the packets from it. This is the process the code in this example follows.

The first thing we need to do when reading an Ogg file is find the first page of data. We use a ogg_sync_state structure to keep track of search for the page data. This needs to be initialized with ogg_sync_init and later cleaned up with ogg_sync_clear:

ifstream file("foo.ogg", ios::in | ios::binary);
ogg_sync_state state;
int ret = ogg_sync_init(&state);
assert(ret==0);
...look for page...

Note that the libogg functions return an error code which should be checked, A result of '0' generally indicates success. We want to obtain a complete page of Ogg data. This is held in an ogg_page structure. The process of obtaining this structure is to do the following steps:

Call ogg_sync_pageout. This will take any data current stored in the ogg_sync_state object and store it in the ogg_page. It will return a result indicating when the entire pages data has been read and the ogg_page can be used. It needs to be called first to initialize buffers. It gets called repeatedly as we read data from the file.
Call ogg_sync_buffer to obtain an area of memory we can reading data from the file into. We pass the size of the buffer. This buffer is reused on each call and will be resized if needed if a larger buffer size is asked for later.
Read data from the file into the buffer obtained above.
Call ogg_sync_wrote to tell libogg how much data we copied into the buffer.
Resume from the first step, calling ogg_sync_buffer. This will copy the data from the buffer into the page, and return '1' if a full page of data is available.

Here's the code to do this:

ogg_page page;
while(ogg_sync_pageout(&state, &page) != 1) {
  char* buffer = ogg_sync_buffer(oy, 4096);
  assert(buffer);

  file.read(buffer, 4096);
  int bytes = stream.gcount();
  if (bytes == 0) {
    // End of file
    break;
  }

  int ret = ogg_sync_wrote(&state, bytes);
  assert(ret == 0);
}

We need to keep track of the logical streams within the file. These are identified by serial number and this number is obtained from the page. I create a C++ map to associate the serial number with an OggStream object which holds information I want associated with the stream. In later examples this will hold data needed for the Theora and Vorbis decoding process.

class OggStream
{
  ...
  int mSerial;
  ogg_stream_state mState;
  int mPacketCount;
  ...
};

typedef map<int, OggStream*> StreamMap;

Each stream has an ogg_stream_state object that is used to keep track of the data read that belongs to the stream. We're storing this in the OggStream object that we associated with the stream serial number. Once we've read a page as described above we need to tell libogg to add this page of data to the stream.

StreamMap streams;
ogg_page page = ...obtained previously...;
int serial = ogg_page_serialno(&page);
OggStream* stream = 0;
if (ogg_page_bos(&page) {
  stream = new OggStream(serial);
  int ret = ogg_stream_init(&stream->mState, serial);
  assert(ret == 0);
  streams[serial] = stream;
}
else
  stream = streams[serial];

int ret = ogg_stream_pagein(&stream->mState, &page);
assert(ret == 0);

This code uses ogg_page_serialno to get the serial number of the page we just read. If it is the beginning of the stream(ogg_page_bos) then we create a new OggStream object, initialize the stream's state with ogg_stream_init, and store it in out streams map. If it's not the beginning of the stream we just get our existing entry in the map. The final call to ogg_stream_pagein inserts the page of data into the streams state object. Once this is done we can start looking for completed packets of data and decode them.

To decode the data from a stream we need to retrieve a packet from it. The steps for doing this are:

Call ogg_stream_packetout. This will return a value indicating if a packet of data is available in the stream. If it is not then we need to read another page (following the same steps previously) and add it to the stream, calling ogg_stream_packetout again until it tells us a packet is available. The packet's data is stored in an ogg_packet object.
Do something with the packet data. This usually involves calling libvorbis or libtheora routines to decode the data. In this example we're just counting the packets.
Repeat until all packets in all streams are consumed.

The code for this is:

while (..read a page...) {
  ...put page in stream...  
  ogg_packet packet;
  int ret = ogg_stream_packetout(&stream->mState, &packet);    
  if (ret == 0) {
    // Need more data to be able to complete the packet
    continue;
  }
  else if (ret == -1) {
    // We are out of sync and there is a gap in the data.
    // We lost a page somewhere.
    break;
  }

  // A packet is available, this is what we pass to the vorbis or
  // theora libraries to decode.
  stream->mPacketCount++;
}

That's all there is to reading an Ogg file. There are more libogg functions to get data out of the stream, identify end of stream, and various other useful functions but this covers the basics. Try out the example program in the github repository for more information.

Note that the libogg functions don't require reading from a file. You can use these routines with any data you've obtained. From a socket, from memory, etc.

In the next post about reading Ogg files I'll go through using libtheora to decode the video data and display it.

Reading Ogg files with JavaScript

2009-06-05T15:09:00+12:00

On tinyvid.tv I do quite a bit of server side reading of Ogg files to get things like duration and bitrate information when serving information about the media. I wondered if it would be possible to do this sort of thing using JavaScript running in the browser.

The format of the Ogg container is defined in RFC 3533. The difficulty comes in reading binary data from JavaScript. The XMLHttpRequest object can be used to retrieve data via a URL from JavaScript in a page but processing the binary data in the Ogg file is problematic. The response returned by XMLHttpRequest assumes text or XML (in Firefox at least).

One way of handling binary data is described in this Mozilla Developer article. Trying this method out works in Firefox and I can download and read the data in the Ogg file.

Ideally I don't want to download the entire file. It might be a large video. I thought by handling the 'progress' event or ready state 3 (data received) I'd be able to look at the data currently retrieved. This does work but on each call to the 'responseText' attribute in these events Firefox copies its internal copy of the downloaded data into a JavaScript array. Doing this every time a portion of the file is downloaded results in major memory use and slow downs proving impractical for even small files.

I think the only reliable way to process the file in chunks is to use byte range requests and do multiple requests. Is there a more reliable way to do binary file reading via JavaScript using XMLHttpRequest? I'd like to be able to process the file in chunks using an Iteratee style approach.

I put up a rough quick demo of loading the first 100Kb of a video and displaying information from each Ogg packet. This probably works in Firefox only due to the workaround needed to read binary data. Click on the 'Go' button in the demo page. This will load transformers320.ogg and display the contents of the first Ogg physical page.

I decode the header packets for Theora and Vorbis. So the first page shown will show it is for a Theora stream with a given size and framerate. Clicking 'Next' will move on to the Next page. This is a Vorbis header with the rate and channel information. Clicking 'Next' again gets the comment header for the Theora stream. The demo reads the comments and displays them. The same for the Vorbis comment records. As you 'Next' through the file it displays the meaning of the granulepos for each page. It shows whether the Theora data is for a keyframe, what time position it is, etc.

Something like this could be used to read metadata from Ogg files, read subtitle information, show duration, etc. More interesting would be to implement a Theora and/or Vorbis decoder in JavaScript and see how it performs.

The main issues with doing this from JavaScript seem to be:

Handling binary data using XMLHttpRequest in a cross browser manner
Processing the file in chunks so the entire file does not need to be kept in memory
Files need to be hosted on the same domain as the page. tinyvid.tv adds the W3C Access Control headers so they can be accessed cross domain but it also hosts some files on Amazon S3 where these headers can't be added. As a result even tinyvid itself can't use XMLHttpRequest to read these files.

New

2007-12-08T01:21:00+13:00

I've uploaded new builds of Firefox with experimental support for the WHATWG <video> element. They are (compiled from this git commit):

The git repository has been updated with the code for this build.

Note that the code in these builds, and the git repository, is advanced from that in the patch attached to the video element bugzilla entry. That code can be accessed from git via the patch11 tag.

The usual disclaimer with the builds I provide applies.

Be aware that these are builds from a random point in the Mozilla CVS tree, with the Video patch applied. I don't guarantee they'll work for much more than demonstrating video support and it's very likely to contain bugs. That said, I run these builds often.

Two main additions with this build. The first is fixing the annoying bug whereby leaving a page that is playing video would leave the sound running in the background. This was due to the 'bfcache' keeping the page around in case the user hits the back or forward buttons. The video is effectively paused now when the page is in the bfcache.

The second is preliminary support for the <source> element, which can be used to provide a list of media resources, with mime types, and the browser selects out of the list the media resource to play based on what it supports. This initial implementation only supports a 'type' attribute set to 'video/ogg'. What this will let you do though, is add a source element for an MP4 encoding, and one for an Ogg encoding. When played on Firefox the Ogg version will play, and a browser that currently doesn't support Ogg but supports MP4 can choose to play that instead.

Issues I have partial fixes for but haven't yet made it into the main branch of the git repository, and not in these builds are:

Sound support on Linux is patchy. I'm using sydney audio's OSS support which has issues when multiple devices open /dev/dsp. This can occasionally affect synchronisation of the video.
There is no buffering for network loads. The video plays immediately so on a slow pipe or large video the playback will stutter. You can pause the video to let the download catch up but there is no progress indication yet. This can make performance look worse than it is in bandwidth constrained environments.
Seeking forward and back in the stream. There is some support, but it's a work in progress and quite broken.

You might notice that full page zoom works in this build. And it works with videos. You can zoom a page and any videos on the page are zoomed. You can test this with Ctrl+ (Apple-+ on the Mac).

Test videos are available at my test page, or Opera's page.

There are sites using the <video> right now.

The Metavid project has support for the <video> element. If you look at the videos in a <video> enabled browser, the player is the interface provided with the 'controls' attribute set. For example, Senate Proceeding 11-16-07. Nice job, Metavid, and thanks for using <video>!

The Wikimedia video's seems to have experimental support for it. Unfortunately the test to see if the video element is supported appears to be done last, so usually the Java applet, etc is found first. You can get around this by setting a cookie. When the Wikimedia video page open, enter this in the URL bar:

javascript:wgOggPlayer.setCookie('ogg_player', 'videoElement', 7*86400*1000, false, false, false, false)

Remove it by deleting the 'commons.wikimedia.org' cookie if you want to go back to the default ordering of selections.

You'll notice some issues, which is probably why it's currently right at the end of the chain. Clicking on the video picture and pressing play works fine. But clicking the play button on the main page listing all the videos does not correctly display the video. I'll look into what's causing this.

If you know of any other <video> supported sites, please let me know in the comments.

I'll be attending the W3C Video on the Web Workshop in San Jose on the 12th and 13th of December. I fly out from Auckland on the 9th and leave San Francisco on the 16th. If you're in the area and want to catch up, let me know.

Opera has a new Video enabled build

2007-11-09T00:03:00+13:00

Opera is making a call for video on the web, releasing an experimental build with video support modelled on the latest WHATWG specification.

Their post has some examples to try out and instructions on how to use the <video> element.

Their examples work quite well in the latest video enabled build of Firefox too. Thanks to help from Robert O'Callahan it now has support for the 'controls' attribute. There's no more need for JavaScript buttons to activate video. I've also implemented some of the events. This code is in the git repository.

Some very experimental binary builds if you want to try things out:

Windows
Mac OS X 10.4 (Intel)
Linux coming soon

Be aware that these are builds from a random point in the Mozilla CVS tree, with the Video patch applied. I don't guarantee they'll work for much more than demonstrating video support and it's very likely to contain bugs. That said, I run these builds often.

Try out the demo's that Opera have done, or the one's on my test page.

It's very cool to see video support using patent free formats running on more than one browser, with simple HTML that can be embedded by anyone. Thanks Opera!

Theora needs an MSVC compatible inline assembler expert

2007-10-03T13:06:00+13:00

The Theora project needs someone knowledgeable in the inline assembler syntax used in the Microsoft Visual C++ toolset.

Theora has some assembler portions written using the GNU assembler syntax. Some way of having these same optimizations working when building the library with the Microsoft toolset would be very useful.