Bluish Coder: video

Temporal Media Fragment Support in Firefox

2011-08-08T12:00:00+12:00

The W3C has a Media Fragments Working Group whose mission is to specify temporal and media fragments in the Web using URI's. The draft specification goes through in detail how these fragments work. I recently became a member of the working group and I've been working on adding support for the temporal dimension portion of the specification to Firefox.

In the most basic form you can specify a start time and an end time in the fragment part of a URI in an HTML video or audio element. For example:

<video src='/mediafrag/AudioAPI.webm#t=50,100'></video>

The '#t' portion of the URI identifies the fragment as being a temporal media fragment. In this example '50,100' means start the video at a current time of 50 seconds, and stop playing at 100 seconds. There are various other formats for the temporal media fragment defined in the specification. Examples can be seen in the UA Test Cases.

Development of this feature is being done in bug 648595. I've done test builds of the first iteration of the patch and they are available at my Firefox media fragment test builds page. The page has builds, an example, and a list of limitations which are currently:

Temporal syntax only. This means no spatial or track dimensions.
NPT time support only. No SMPTE time codes or Wall-clock time codes.
When changing the media fragment portion of a URL the media is not immediately updated. You need to refresh the page to see the change. This is most noticeable when directly navigating to the video and adding or changing a fragment.
The user interface for identifying the fragment in the standard controls is ugly and needs polish.
The HTML standard includes an 'initialTime' attribute for obtaining the start time. There is no way to obtain an end time so I've exposed a 'mozFragmentEnd' attribute on the video DOM object.

WDCNZ HTML Media Presentation

2011-08-08T09:30:00+12:00

Last month I went down to Wellington to give a joint talk at the WDCNZ conference. The topic was "HTML Media: Where we are and where we need to go". The talk was shared between Nigel Parker, Mobile and Developer Evangelist for Microsoft, and myself. The conference was excellent with some great speakers and talks.

Nigel and I covered using HTML video and audio elements and how they can be used today across multiple browsers and mobile devices. We also covered upcoming API's and directions in the web media area. Demo's were shown on Microsoft Internet Explorer 9, Windows Phone 7 (running the Mango update which has a browser that supports HTML media) and Firefox. Nigel has slides and a summary in his blog post about the talk. I think it took a bit for attendees to get over the shock over a Mozilla and Microsoft representative sharing the stage and working together! Nigel's post explains how this came about.

I spoke to a few Kiwi developers afterwards about using HTML video and there was a fair bit of interest in using it. The main obstacles seemed to be people unsure what codec to use and wanting support for adaptive streaming. The first is an education issue, getting people aware of what codecs to support for maximum coverage across browsers and how to encode to those formats. With regards to adaptive streaming there has been discussion between interested parties in various mailing lists and groups - it's definitely something that is wanted.

At the time of the talk Nigel and I weren't able to find any existing New Zealand based sites that use HTML video. Hopefully this will change in the future and Microsoft New Zealand are leading by example by using HTML video on their own site.

Firefox video scaling during YCbCr to RGB conversion

2010-11-24T15:00:00+13:00

When I changed to using the Chromium YCbCr color conversion code I didn't use the code that scales the video during the conversion as there was no infrastructure in place to do this at the time.

What this meant was we'd do the YCbCr conversion on the original image size and then later, during rendering, scale this using the browser's scaling routines. This was slow in the case of large videos being scaled using the 'width' and 'height' attributes on video elements.

YouTube's HTML 5 video user interface often scales down. By default it plays a good quality video scaled down to fit the standard YouTube user interface. In Firefox this results in a performance hit due to the slow scaling.

Bug 577843 was raised to address this issue. The fix was to implement scaling at YCbCr conversion time using the scaling code that already existed in Chromium's YCbCr code that we were using. This has now landed and will be in Firefox 4 resulting in better video playback when scaled.

As part of this work I also updated to the latest Chromium code. This was done in bug 583138. This gives more performant conversion routines and better quality. This has also landed recently and will be in Firefox 4.

For performance comparisons to see if there were improvements in playback I tested video playback using WebM, Theora and a commonly used open source flash-based video player playing an MP4 video. I compared dropped frame statistics using stats available from the flash player and the experimental playback statistics I wrote about earlier.

For a 1080p sized video, scaled down, the performance on my laptop improved by about 25% with these patches applied and was at least as good as the flash based player. For the Theora video performance was better - I suspect due to the lower CPU requirements to decode Theora video.

[update from feedback: Removed comment about YouTube's fullscreen not being hardware accelerated. Apparently Firefox trunk now accelerates it. Clarified that by 'flash player' I meant an open source video player written in Flash, not an open source Flash implementation]

Changes to Firefox Video Implementation

2010-09-15T06:25:00+12:00

I mentioned in my previous post about the Firefox video implementation that some changes were going to be made to make our implementation more spec compliant. These have now landed and will be in Firefox 4. The relevant bugs are:

Bug 571822 - Fire timeupdate less frequently than once per frame
Bug 584615 - Make media progress events be simple events

Prior to these landing we would try to fire a 'timeupdate' event every time a video frame changed. This allowed JavaScript applications to listen to this event and do things (like display subtitles) at fairly accurate time intervals. Unfortunately other browsers fire this event at different rates, usually with about 200ms between events. This made applications non-portable across browsers if they relied on per-frame events. As a result we've moved to firing 'timeupdate' no more frequently than 250ms. If you need to process time changes more often than this then it's best to use 'setTimeout' or 'setInterval' to query 'currentTime' at the rate you need. This will work across all browsers that support the media API.

The other change was to remove the 'progress' information from the 'progress' events. This was driven by a spec change. The 'progress' events used to contain additional information about the byte position and total number of bytes for the download. This additional information has been removed from the spec and we were the only browser that implemented that part. To prevent cross browser issues we've removed the information too. The alternative is to use the 'buffered' attribute on the media element and work with time data. 'buffered' support was also landed recently which is why we can now remove the non-compliant 'progress' event data.

Experimental Playback Statistics for HTML Video and Audio

2010-08-24T17:30:00+12:00

Now that HTML video is getting more usage there have been requests for statistics during playback so performance can be measured from JavaScript. This has come up a few times on the WHATWG mailing list.

I raised bug 580531 to add some additional data to media and video DOM elements to provide this information. The patch in that bug adds two attributes to the DOM HTMLVideoElement object, both prefixed by 'moz' to prevent name clashes as they are experimental:

interface nsIDOMHTMLVideoElement : nsIDOMHTMLMediaElement
{
   ...
   // A count of the number of frames that have been decoded and ready
   // to be displayed. This can be used for computing the decoding framerate
   readonly attribute unsigned long mozDecodedFrames;

   // A count of the number of frames that have been dropped for performance
   // reasons during playback.
   readonly attribute unsigned long mozDroppedFrames;
};

mozDecodedFrames is an incrementing count each time a frame is decoded and ready to be displayed. mozDroppedFrames is incremented each to a frame is dropped to keep playback going at the correct frame rate. These can be used to compute the current framerate that the video is being decoded at, and the framerate that it should be decoding at by combining the two counts. This will give client JavaScript code an indication if the hardware is able to play the video.

Another requested feature is information on the download rate. Currently the Firefox implementation of HTML video and audio uses a non-standard extension to 'progress' events. We provide information attached to the event that gives the current byte position in the media file as it downloads, and the total bytes available. This was actually required by the WHATWG spec at one point but it changed a while back and I don't think any other browser implements it.

This has been used in experimental pages to compute things like download rate and we use it in our own controls implementation to display a progress bar as we didn't implement the 'buffered' attribute. Support for 'buffered' has now landed so we want to remove the non-standard 'progress' information to be spec compliant.

To address the needs of those using the progress data for bandwidth calculation I've added two attributes to HTMLMediaElement:

interface nsIDOMHTMLMediaElement : nsIDOMHTMLElement
{
   ...
   // The approximate rate at which the media resource is being downloaded in
   // bytes per second. If the rate is not available (possibly due
   // to not having downloaded enough data to compute a consistent value)
   // this will be NaN.
   readonly attribute float mozDownloadRate;

   // The approximate rate at which the media resource is being decoded in
   // bytes per second. If the rate is not available this will be
   // NaN.
   readonly attribute float mozDecodeRate;
};

mozDownloadRate is an estimate, in bytes per second, of the current download rate of the media. If not enough information has been downloaded for a reliable estimate this will be NaN. mozDecodeRate provides the rate at which decoding is occurring. In the patch this is the length of the video divided by the duration and remains constant.

Whether this gets landed is yet to be determined but I think the information is useful and I'd be interested in any feedback on the data I've decided to include. There is some feedback in the bug from the patch review already and there's plenty of time between now and when the Firefox 4 rush is over to look over ideas of what could be included, removed or changed. I posted about the proposed additions in the WHATWG mailing list and there are some responses in that thread.

Building Firefox with WebM support

2010-05-20T21:20:00+12:00

All the patches to get Firefox built with WebM support have now been attached to their relevant bugs. These patches aren't ready for review yet but can be used to get a build that has the functionality of the special Firefox WebM builds.

To build use Mozilla Central as the base with the patches from the following bugs applied:

Bug 566241 - Video playback stops intermittently - This has now landed in mozilla-central.
Bug 566498 - Move generic video duration computing into Ogg decoder - This has now landed in mozilla-central.
Bug 566501 - Fix seeks needed only by Ogg backend - This has now landed in mozilla-central.
Bug 567056 - Rename some variables - This has now landed in mozilla-central.
Bug 566246 - nestegg, the webm container parser we use
Bug 566247 - libvpx, The VP8 SDK and associated build changes
Bug 566245 - nsWebMDecoder implementation

Hopefully these will be finished, reviewed and landed in short order.

VP8, WebM and Firefox

2010-05-20T04:40:00+12:00

For all the details on the recent announcing of VP8 video code support in Firefox, the WebM container and playing YouTube videos with Firefox, read Firefox, YouTube and WebM on hacks.mozilla.org.

HTML 5 Video Seeking and Redirects

2010-04-27T14:00:00+12:00

Bug 560806 has landed on trunk. Seeking and duration works for videos hosted on some servers where it didn't before.

Firefox's implementation of HTML 5 video allows seeking in unbuffered portions of the video as long as the server that hosts the video supports byte range requests.

Firefox needs to know in advance if it can seek the video so it can correctly display the 'progress' bar. It also needs this information so it can compute the duration (which requires seeking into the video to find the end time).

Most servers that support byte range requests send an Accept-Ranges header. If we see this on the initial request we know the server supports seeking.

Some servers support byte range requests but do not send this header. Amazon S3 is an example of a server that does this. To detect if these servers support seeking we make our inital request for the video include a 'Range' header that askes for the bytes from zero to the end of the file. Servers like S3 that support byte ranges will respond with a '206' response code meaning a partial request is returned. This is sent even though the byte range we asked for is actually for the entire file. Once we see the '206' we mark the resource as seekable.

A very few servers don't send an Accept-Ranges header and recognize our byte range request as actually being for the entire file and return a '200' response code. Unfortunately we don't recognize resources hosted on these servers as seekable. This means Ogg media on these servers don't have a duration since we can't seek to find it.

Another issue we came across, and exists in Firefox 3.5 and Firefox 3.6, is detecting seeking when the initial request for the media resource results in a '3XX' response code. This is a redirect to another URL. Current Firefox versions (and possibly other browsers) do not pass on the value of headers to the redirected URL. This means the 'Range' header is removed and we don't get a '206' response code on the redirected request. In servers like Amazon S3 that don't send 'Accept-Ranges' this means we don't support seeking.

This is why videos hosted on Amazon S3 and similar serving platforms that arrive there via a redirect don't get a duration, show progress or allow seeking. It's a fairly common practice to hide the fact that Amazon S3 is being used by having a redirect from a URL on the web site domain that redirects to the S3 location which results in this problem.

The following are two URL's that show the problem. The first is a direct link to the video. The second redirects to the final URL. In the first you can seek and see a duration of '1:00'. In the second you can't seek and no duration is shown:

This is Bug 560806 and has been fixed and is now on Firefox trunk and in the nightly builds. The fix was to explicitly pass the 'Range' header along to the final URL. This workaround exists in a few other areas in Firefox and Bug 311599 discusses a similar issue for plugins.

If your own videos are not seekable or showing a duration then you may be facing a similar issue and it might be worthwhile trying a nightly build and seeing if that fixes it. Other things to look at are the server configuration to make sure that byte range requests are supported and the Accept-Ranges header is being sent. If you use Amazon S3 you might want to try and get them to support Accept-Ranges. This forum post discusses the issue.

Changes to Firefox YCbCr Color Conversion Code

2010-03-12T13:30:00+13:00

I posted previously about colour conversion libraries. To replace the liboggplay conversion routines we decided to try the Chromium libraries.

Bug 551277 adds the Chromium code and integrates it with the new video layer painting code. The Chromium code uses a C fallback if MMX is not available but this is done at compile time. I modified it do runtime detection using Mozilla's SSE.h and moved the generic C routine from the Linux specific implementation to be used on all platforms that don't have MMX. Some other minor modifications were done (function name, namspacing).

I added an 'update.sh' script that can be used to update against the latest Chromium code. Given the path to the Chromium source it applies a patch for the changes I made.

Bug 551378 adds YCbCr 4:4:4 conversion (YV24). The Chromium code supported 4:2:0 and 4:2:2 (YV12 and YV16 respectively). Theora also allows 4:4:4 per spec. There aren't many 4:4:4 Theora videos out in the wild and the fix in this bug only adds support in the slower C implementation. I detect if the video is a 4:4:4 video and fallback to the C code.

At some point optimized MMX routines should be done for this. I've raised Bug 551844 for this. This would probably make a good first bug for anyone who wants something small and self contained to work on. It would be good to contribute this back to Chromium when done too.

Xiph has a test suite with example videos in the different formats.

HTML 5 Video Element for Internet Explorer

2010-02-22T13:54:00+13:00

Cristian Adam has been working on a plugin for Internet Explorer that implements the HTML 5 video element.

He has released a new version of the Xiph Ogg Codecs with a technical preview of the <video> element for IE. A screencast of it working is available at Cristian's website.