Bluish Coder: firefox

New Wave Audio Backend Landed

2011-03-31T16:00:00+13:00

Yesterday I landed bug 635649 on mozilla-central. This is a refactoring of the wave backend for the HTML audio element. The initial work on this bug was done by Matthew Gregan with me completing the patch.

Prior to this landing the wave backend implemented the HTML audio API itself, sharing very little implementation details with the other audio backends. This meant the backend had to implement the intricate details of what events get raised and when they get raised. When we wrote the WebM backend we refactored the code so we could share this sort of thing amongst backends. This bug refactored the wave backend to share this code.

The advantages of this are:

Less likelihood of 'wave backend only' bugs.
Less code to maintain.
The Audio Data API can now use wave files as this was implemented in the shared backend code.

Firefox video scaling during YCbCr to RGB conversion

2010-11-24T15:00:00+13:00

When I changed to using the Chromium YCbCr color conversion code I didn't use the code that scales the video during the conversion as there was no infrastructure in place to do this at the time.

What this meant was we'd do the YCbCr conversion on the original image size and then later, during rendering, scale this using the browser's scaling routines. This was slow in the case of large videos being scaled using the 'width' and 'height' attributes on video elements.

YouTube's HTML 5 video user interface often scales down. By default it plays a good quality video scaled down to fit the standard YouTube user interface. In Firefox this results in a performance hit due to the slow scaling.

Bug 577843 was raised to address this issue. The fix was to implement scaling at YCbCr conversion time using the scaling code that already existed in Chromium's YCbCr code that we were using. This has now landed and will be in Firefox 4 resulting in better video playback when scaled.

As part of this work I also updated to the latest Chromium code. This was done in bug 583138. This gives more performant conversion routines and better quality. This has also landed recently and will be in Firefox 4.

For performance comparisons to see if there were improvements in playback I tested video playback using WebM, Theora and a commonly used open source flash-based video player playing an MP4 video. I compared dropped frame statistics using stats available from the flash player and the experimental playback statistics I wrote about earlier.

For a 1080p sized video, scaled down, the performance on my laptop improved by about 25% with these patches applied and was at least as good as the flash based player. For the Theora video performance was better - I suspect due to the lower CPU requirements to decode Theora video.

[update from feedback: Removed comment about YouTube's fullscreen not being hardware accelerated. Apparently Firefox trunk now accelerates it. Clarified that by 'flash player' I meant an open source video player written in Flash, not an open source Flash implementation]

Changes to Firefox Video Implementation

2010-09-15T06:25:00+12:00

I mentioned in my previous post about the Firefox video implementation that some changes were going to be made to make our implementation more spec compliant. These have now landed and will be in Firefox 4. The relevant bugs are:

Bug 571822 - Fire timeupdate less frequently than once per frame
Bug 584615 - Make media progress events be simple events

Prior to these landing we would try to fire a 'timeupdate' event every time a video frame changed. This allowed JavaScript applications to listen to this event and do things (like display subtitles) at fairly accurate time intervals. Unfortunately other browsers fire this event at different rates, usually with about 200ms between events. This made applications non-portable across browsers if they relied on per-frame events. As a result we've moved to firing 'timeupdate' no more frequently than 250ms. If you need to process time changes more often than this then it's best to use 'setTimeout' or 'setInterval' to query 'currentTime' at the rate you need. This will work across all browsers that support the media API.

The other change was to remove the 'progress' information from the 'progress' events. This was driven by a spec change. The 'progress' events used to contain additional information about the byte position and total number of bytes for the download. This additional information has been removed from the spec and we were the only browser that implemented that part. To prevent cross browser issues we've removed the information too. The alternative is to use the 'buffered' attribute on the media element and work with time data. 'buffered' support was also landed recently which is why we can now remove the non-compliant 'progress' event data.

Experimental Playback Statistics for HTML Video and Audio

2010-08-24T17:30:00+12:00

Now that HTML video is getting more usage there have been requests for statistics during playback so performance can be measured from JavaScript. This has come up a few times on the WHATWG mailing list.

I raised bug 580531 to add some additional data to media and video DOM elements to provide this information. The patch in that bug adds two attributes to the DOM HTMLVideoElement object, both prefixed by 'moz' to prevent name clashes as they are experimental:

interface nsIDOMHTMLVideoElement : nsIDOMHTMLMediaElement
{
   ...
   // A count of the number of frames that have been decoded and ready
   // to be displayed. This can be used for computing the decoding framerate
   readonly attribute unsigned long mozDecodedFrames;

   // A count of the number of frames that have been dropped for performance
   // reasons during playback.
   readonly attribute unsigned long mozDroppedFrames;
};

mozDecodedFrames is an incrementing count each time a frame is decoded and ready to be displayed. mozDroppedFrames is incremented each to a frame is dropped to keep playback going at the correct frame rate. These can be used to compute the current framerate that the video is being decoded at, and the framerate that it should be decoding at by combining the two counts. This will give client JavaScript code an indication if the hardware is able to play the video.

Another requested feature is information on the download rate. Currently the Firefox implementation of HTML video and audio uses a non-standard extension to 'progress' events. We provide information attached to the event that gives the current byte position in the media file as it downloads, and the total bytes available. This was actually required by the WHATWG spec at one point but it changed a while back and I don't think any other browser implements it.

This has been used in experimental pages to compute things like download rate and we use it in our own controls implementation to display a progress bar as we didn't implement the 'buffered' attribute. Support for 'buffered' has now landed so we want to remove the non-standard 'progress' information to be spec compliant.

To address the needs of those using the progress data for bandwidth calculation I've added two attributes to HTMLMediaElement:

interface nsIDOMHTMLMediaElement : nsIDOMHTMLElement
{
   ...
   // The approximate rate at which the media resource is being downloaded in
   // bytes per second. If the rate is not available (possibly due
   // to not having downloaded enough data to compute a consistent value)
   // this will be NaN.
   readonly attribute float mozDownloadRate;

   // The approximate rate at which the media resource is being decoded in
   // bytes per second. If the rate is not available this will be
   // NaN.
   readonly attribute float mozDecodeRate;
};

mozDownloadRate is an estimate, in bytes per second, of the current download rate of the media. If not enough information has been downloaded for a reliable estimate this will be NaN. mozDecodeRate provides the rate at which decoding is occurring. In the patch this is the length of the video divided by the duration and remains constant.

Whether this gets landed is yet to be determined but I think the information is useful and I'd be interested in any feedback on the data I've decided to include. There is some feedback in the bug from the patch review already and there's plenty of time between now and when the Firefox 4 rush is over to look over ideas of what could be included, removed or changed. I posted about the proposed additions in the WHATWG mailing list and there are some responses in that thread.

Building Firefox with WebM support

2010-05-20T21:20:00+12:00

All the patches to get Firefox built with WebM support have now been attached to their relevant bugs. These patches aren't ready for review yet but can be used to get a build that has the functionality of the special Firefox WebM builds.

To build use Mozilla Central as the base with the patches from the following bugs applied:

Bug 566241 - Video playback stops intermittently - This has now landed in mozilla-central.
Bug 566498 - Move generic video duration computing into Ogg decoder - This has now landed in mozilla-central.
Bug 566501 - Fix seeks needed only by Ogg backend - This has now landed in mozilla-central.
Bug 567056 - Rename some variables - This has now landed in mozilla-central.
Bug 566246 - nestegg, the webm container parser we use
Bug 566247 - libvpx, The VP8 SDK and associated build changes
Bug 566245 - nsWebMDecoder implementation

Hopefully these will be finished, reviewed and landed in short order.

VP8, WebM and Firefox

2010-05-20T04:40:00+12:00

For all the details on the recent announcing of VP8 video code support in Firefox, the WebM container and playing YouTube videos with Firefox, read Firefox, YouTube and WebM on hacks.mozilla.org.