2015-03-03
Firefox Media Source Extensions Update
This is an update on some recent work on the Media Source Extensions API in Firefox. There has been a lot of work done on MSE and the underlying media framework by Gecko developers and this update just covers some of the telemetry and exposed debug data that I've been involved with implementing.
Telemetry
Mozilla has a telemetry system to get data on how Firefox behaves in the real world. We've added some MSE video stats to telemetry to help identify usage patterns and possible issues.
Bug 1119947 added information on what state an MSE video is in when the video is unloaded. The intent of this is to find out if users are exiting videos due to slow buffering or seeking. The data is available on telemetry.mozilla.org under the VIDEO_MSE_UNLOAD_STATE
category. This has five states:
0 = ended, 1 = paused, 2 = stalled, 3 = seeking, 4 = other
The data provides a count of the number of times a video was unloaded for each state. If a large number of users were exiting during the stalled
state then we might have an issue with videos stalling too often. Looking at current stats on beta 37
we see about 3% unloading on stall with 14% on ended and 57% on other. The 'other' represents unloading during normal playback.
Bug 1127646 will add additional data to get:
- Join Latency - time between video load and video playback for autoplay videos
- Mean Time Between Rebuffering - play time between rebuffering hiccups
This will be useful for determining performance of MSE for sites like YouTube. The bug is going through the review/comment stage and when landed the data will be viewable at telemetry.mozilla.org.
about:media plugin
While developing the Media Source Extensions support in Firefox we found it useful to have a page displaying internal debug data about active MSE videos.
In particular it was good to be able to get a view of what buffered data the MSE JavaSript API had and what our internal Media Source C++ code stored. This helped track down issues involving switching buffers, memory size of resources and other similar things.
The internal data is displayed in an about:media
page. Originally the page was hard coded in the browser but :gavin suggested moving it to an addon. The addon is now located at https://github.com/doublec/aboutmedia. That repository includes the aboutmedia.xpi which can be installed directly in Firefox. Once installed you can go to about:media
to view data on any MSE videos.
To test this, visit a video that has MSE support in a nightly build with the about:config
preferences media.mediasource.enabled
and media.mediasource.mp4.enabled
set to true
. Let the video play for a short time then visit about:media
in another tab. You should see something like:
https://www.youtube.com/watch?v=3V7wWemZ_cs
mediasource:https://www.youtube.com/6b23ac42-19ff-4165-8c04-422970b3d0fb
currentTime: 101.40625
SourceBuffer 0
start=0 end=14.93043
SourceBuffer 1
start=0 end=15
Internal Data:
Dumping data for reader 7f9d85ef1800:
Dumping Audio Track Decoders: - mLastAudioTime: 7.732243
Reader 1: 7f9d75cba800 ranges=[(10.007800, 14.930430)] active=false size=79880
Reader 0: 7f9d85e88000 ranges=[(0.000000, 10.007800)] active=false size=160246
Dumping Video Track Decoders - mLastVideoTime: 7.000000
Reader 1: 7f9d75cbd800 ranges=[(10.000000, 15.000000)] active=false size=184613
Reader 0: 7f9d85985000 ranges=[(0.000000, 10.000000)] active=false size=1281914
The first portion of the displayed data shows the JS API video of the data buffered:
currentTime: 101.40625
SourceBuffer 0
start=0 end=14.93043
SourceBuffer 1
start=0 end=15
This shows two SourceBuffer objects. One containing data from 0-14.9 seconds and the other 0-15 seconds. One of these will be video data and the other audio. The currentTime attribute of the video is 101.4 seconds. Since there is no buffered data for this range the video is likely buffering. I captured this data just after seeking while it was waiting for data from the seeked point.
The second portion of the displayed data shows information on the C++ objects implementing media source:
Dumping data for reader 7f9d85ef1800:
Dumping Audio Track Decoders: - mLastAudioTime: 7.732243
Reader 1: 7f9d75cba800 ranges=[(10.007800, 14.930430)] active=false size=79880
Reader 0: 7f9d85e88000 ranges=[(0.000000, 10.007800)] active=false size=160246
Dumping Video Track Decoders - mLastVideoTime: 7.000000
Reader 1: 7f9d75cbd800 ranges=[(10.000000, 15.000000)] active=false size=184613
Reader 0: 7f9d85985000 ranges=[(0.000000, 10.000000)] active=false size=1281914
A reader
is an instance of the MediaSourceReader C++ class. That reader holds two SourceBufferDecoder C++ instances. One for audio and the other for video. Looking at the video decoder it has two readers associated with it. These readers are instances of a derived class of MediaDecoderReader which are tasked with the job of reading frames from a particular video format (WebM, MP4, etc).
The two readers each have buffered data ranging from 0-10 seconds and 10-15 seconds. Neither are 'active'. This means they are not currently the video stream used for playback. This will be because we just started a seek. You can view how buffer switching works by watching which of these become active
as the video plays. The size
is the amount of data in bytes that the reader is holding in memory. mLastVideoTime
is the presentation time of the last processed video frame.
MSE videos will have data evicted as they are played. This size threshold for eviction defaults to 75MB and can be changed with the media.mediasource.eviction_threshold
variable in about:config
. When data is appended via the appendBuffer
method on a SourceBuffer
an eviction routine is run. If data greater than the threshold is held then we start removing portions of data held in the readers. This will be noticed in about:media
by the start and end ranges being trimmed or readers being removed entirely.
This internal data is most useful for Firefox media developers. If you encounter stalls playing videos or unusual buffer switching behaviour then copy/pasting the data from about:media
in a bug report can help with tracking the problem down. If you are developing an MSE player then the information may also be useful to find out why the Firefox implementation may not be behaving how you expect.
The source of the addon is on github and relies on a chrome only debug method, mozDebugReaderData
on MediaSource. Patches to improve the data and functionality are welcome.
Status
Media Source Extensions is still in progress in Firefox and can be tested on Nightly, Aurora and Beta builds. The current plan is to enable support limited to YouTube only in Firefox 37 on Windows and Mac OS X for MP4 videos. Other platforms, video formats and wider site usage will be enabled in future versions as the implementation improves.
To track work on the API you can follow the MSE bug in Bugzilla.