Bluish Coder

Programming Languages, Martials Arts and Computers. The Weblog of Chris Double.


2015-03-03

Firefox Media Source Extensions Update

This is an update on some recent work on the Media Source Extensions API in Firefox. There has been a lot of work done on MSE and the underlying media framework by Gecko developers and this update just covers some of the telemetry and exposed debug data that I've been involved with implementing.

Telemetry

Mozilla has a telemetry system to get data on how Firefox behaves in the real world. We've added some MSE video stats to telemetry to help identify usage patterns and possible issues.

Bug 1119947 added information on what state an MSE video is in when the video is unloaded. The intent of this is to find out if users are exiting videos due to slow buffering or seeking. The data is available on telemetry.mozilla.org under the VIDEO_MSE_UNLOAD_STATE category. This has five states:

0 = ended, 1 = paused, 2 = stalled, 3 = seeking, 4 = other

The data provides a count of the number of times a video was unloaded for each state. If a large number of users were exiting during the stalled state then we might have an issue with videos stalling too often. Looking at current stats on beta 37 we see about 3% unloading on stall with 14% on ended and 57% on other. The 'other' represents unloading during normal playback.

Bug 1127646 will add additional data to get:

  • Join Latency - time between video load and video playback for autoplay videos
  • Mean Time Between Rebuffering - play time between rebuffering hiccups

This will be useful for determining performance of MSE for sites like YouTube. The bug is going through the review/comment stage and when landed the data will be viewable at telemetry.mozilla.org.

about:media plugin

While developing the Media Source Extensions support in Firefox we found it useful to have a page displaying internal debug data about active MSE videos.

In particular it was good to be able to get a view of what buffered data the MSE JavaSript API had and what our internal Media Source C++ code stored. This helped track down issues involving switching buffers, memory size of resources and other similar things.

The internal data is displayed in an about:media page. Originally the page was hard coded in the browser but :gavin suggested moving it to an addon. The addon is now located at https://github.com/doublec/aboutmedia. That repository includes the aboutmedia.xpi which can be installed directly in Firefox. Once installed you can go to about:media to view data on any MSE videos.

To test this, visit a video that has MSE support in a nightly build with the about:config preferences media.mediasource.enabled and media.mediasource.mp4.enabled set to true. Let the video play for a short time then visit about:media in another tab. You should see something like:

https://www.youtube.com/watch?v=3V7wWemZ_cs
  mediasource:https://www.youtube.com/6b23ac42-19ff-4165-8c04-422970b3d0fb
    currentTime: 101.40625
    SourceBuffer 0
      start=0 end=14.93043
    SourceBuffer 1
      start=0 end=15

    Internal Data:
      Dumping data for reader 7f9d85ef1800:
        Dumping Audio Track Decoders: - mLastAudioTime: 7.732243
          Reader 1: 7f9d75cba800 ranges=[(10.007800, 14.930430)] active=false size=79880
          Reader 0: 7f9d85e88000 ranges=[(0.000000, 10.007800)] active=false size=160246
        Dumping Video Track Decoders - mLastVideoTime: 7.000000
          Reader 1: 7f9d75cbd800 ranges=[(10.000000, 15.000000)] active=false size=184613
          Reader 0: 7f9d85985000 ranges=[(0.000000, 10.000000)] active=false size=1281914

The first portion of the displayed data shows the JS API video of the data buffered:

currentTime: 101.40625
  SourceBuffer 0
    start=0 end=14.93043
  SourceBuffer 1
    start=0 end=15

This shows two SourceBuffer objects. One containing data from 0-14.9 seconds and the other 0-15 seconds. One of these will be video data and the other audio. The currentTime attribute of the video is 101.4 seconds. Since there is no buffered data for this range the video is likely buffering. I captured this data just after seeking while it was waiting for data from the seeked point.

The second portion of the displayed data shows information on the C++ objects implementing media source:

Dumping data for reader 7f9d85ef1800:
  Dumping Audio Track Decoders: - mLastAudioTime: 7.732243
    Reader 1: 7f9d75cba800 ranges=[(10.007800, 14.930430)] active=false size=79880
    Reader 0: 7f9d85e88000 ranges=[(0.000000, 10.007800)] active=false size=160246
  Dumping Video Track Decoders - mLastVideoTime: 7.000000
    Reader 1: 7f9d75cbd800 ranges=[(10.000000, 15.000000)] active=false size=184613
    Reader 0: 7f9d85985000 ranges=[(0.000000, 10.000000)] active=false size=1281914

A reader is an instance of the MediaSourceReader C++ class. That reader holds two SourceBufferDecoder C++ instances. One for audio and the other for video. Looking at the video decoder it has two readers associated with it. These readers are instances of a derived class of MediaDecoderReader which are tasked with the job of reading frames from a particular video format (WebM, MP4, etc).

The two readers each have buffered data ranging from 0-10 seconds and 10-15 seconds. Neither are 'active'. This means they are not currently the video stream used for playback. This will be because we just started a seek. You can view how buffer switching works by watching which of these become active as the video plays. The size is the amount of data in bytes that the reader is holding in memory. mLastVideoTime is the presentation time of the last processed video frame.

MSE videos will have data evicted as they are played. This size threshold for eviction defaults to 75MB and can be changed with the media.mediasource.eviction_threshold variable in about:config. When data is appended via the appendBuffer method on a SourceBuffer an eviction routine is run. If data greater than the threshold is held then we start removing portions of data held in the readers. This will be noticed in about:media by the start and end ranges being trimmed or readers being removed entirely.

This internal data is most useful for Firefox media developers. If you encounter stalls playing videos or unusual buffer switching behaviour then copy/pasting the data from about:media in a bug report can help with tracking the problem down. If you are developing an MSE player then the information may also be useful to find out why the Firefox implementation may not be behaving how you expect.

The source of the addon is on github and relies on a chrome only debug method, mozDebugReaderData on MediaSource. Patches to improve the data and functionality are welcome.

Status

Media Source Extensions is still in progress in Firefox and can be tested on Nightly, Aurora and Beta builds. The current plan is to enable support limited to YouTube only in Firefox 37 on Windows and Mac OS X for MP4 videos. Other platforms, video formats and wider site usage will be enabled in future versions as the implementation improves.

To track work on the API you can follow the MSE bug in Bugzilla.

Tags


This site is accessable over tor as hidden service 6vp5u25g4izec5c37wv52skvecikld6kysvsivnl6sdg6q7wy25lixad.onion, or Freenet using key:
USK@1ORdIvjL2H1bZblJcP8hu2LjjKtVB-rVzp8mLty~5N4,8hL85otZBbq0geDsSKkBK4sKESL2SrNVecFZz9NxGVQ,AQACAAE/bluishcoder/-61/


Tags

Archives
Links