This is an update on some recent work on the Media Source Extensions API in Firefox. There has been a lot of work done on MSE and the underlying media framework by Gecko developers and this update just covers some of the telemetry and exposed debug data that I've been involved with implementing.
Mozilla has a telemetry system to get data on how Firefox behaves in the real world. We've added some MSE video stats to telemetry to help identify usage patterns and possible issues.
Bug 1119947 added information on what state an MSE video is in when the video is unloaded. The intent of this is to find out if users are exiting videos due to slow buffering or seeking. The data is available on telemetry.mozilla.org under the
VIDEO_MSE_UNLOAD_STATE category. This has five states:
0 = ended, 1 = paused, 2 = stalled, 3 = seeking, 4 = other
The data provides a count of the number of times a video was unloaded for each state. If a large number of users were exiting during the
stalled state then we might have an issue with videos stalling too often. Looking at current stats on
beta 37 we see about 3% unloading on stall with 14% on ended and 57% on other. The 'other' represents unloading during normal playback.
Bug 1127646 will add additional data to get:
- Join Latency - time between video load and video playback for autoplay videos
- Mean Time Between Rebuffering - play time between rebuffering hiccups
This will be useful for determining performance of MSE for sites like YouTube. The bug is going through the review/comment stage and when landed the data will be viewable at telemetry.mozilla.org.
While developing the Media Source Extensions support in Firefox we found it useful to have a page displaying internal debug data about active MSE videos.
In particular it was good to be able to get a view of what buffered data the MSE JavaSript API had and what our internal Media Source C++ code stored. This helped track down issues involving switching buffers, memory size of resources and other similar things.
The internal data is displayed in an
about:media page. Originally the page was hard coded in the browser but :gavin suggested moving it to an addon. The addon is now located at https://github.com/doublec/aboutmedia. That repository includes the aboutmedia.xpi which can be installed directly in Firefox. Once installed you can go to
about:media to view data on any MSE videos.
To test this, visit a video that has MSE support in a nightly build with the
media.mediasource.mp4.enabled set to
true. Let the video play for a short time then visit
about:media in another tab. You should see something like:
https://www.youtube.com/watch?v=3V7wWemZ_cs mediasource:https://www.youtube.com/6b23ac42-19ff-4165-8c04-422970b3d0fb currentTime: 101.40625 SourceBuffer 0 start=0 end=14.93043 SourceBuffer 1 start=0 end=15 Internal Data: Dumping data for reader 7f9d85ef1800: Dumping Audio Track Decoders: - mLastAudioTime: 7.732243 Reader 1: 7f9d75cba800 ranges=[(10.007800, 14.930430)] active=false size=79880 Reader 0: 7f9d85e88000 ranges=[(0.000000, 10.007800)] active=false size=160246 Dumping Video Track Decoders - mLastVideoTime: 7.000000 Reader 1: 7f9d75cbd800 ranges=[(10.000000, 15.000000)] active=false size=184613 Reader 0: 7f9d85985000 ranges=[(0.000000, 10.000000)] active=false size=1281914
The first portion of the displayed data shows the JS API video of the data buffered:
currentTime: 101.40625 SourceBuffer 0 start=0 end=14.93043 SourceBuffer 1 start=0 end=15
This shows two SourceBuffer objects. One containing data from 0-14.9 seconds and the other 0-15 seconds. One of these will be video data and the other audio. The currentTime attribute of the video is 101.4 seconds. Since there is no buffered data for this range the video is likely buffering. I captured this data just after seeking while it was waiting for data from the seeked point.
The second portion of the displayed data shows information on the C++ objects implementing media source:
Dumping data for reader 7f9d85ef1800: Dumping Audio Track Decoders: - mLastAudioTime: 7.732243 Reader 1: 7f9d75cba800 ranges=[(10.007800, 14.930430)] active=false size=79880 Reader 0: 7f9d85e88000 ranges=[(0.000000, 10.007800)] active=false size=160246 Dumping Video Track Decoders - mLastVideoTime: 7.000000 Reader 1: 7f9d75cbd800 ranges=[(10.000000, 15.000000)] active=false size=184613 Reader 0: 7f9d85985000 ranges=[(0.000000, 10.000000)] active=false size=1281914
reader is an instance of the MediaSourceReader C++ class. That reader holds two SourceBufferDecoder C++ instances. One for audio and the other for video. Looking at the video decoder it has two readers associated with it. These readers are instances of a derived class of MediaDecoderReader which are tasked with the job of reading frames from a particular video format (WebM, MP4, etc).
The two readers each have buffered data ranging from 0-10 seconds and 10-15 seconds. Neither are 'active'. This means they are not currently the video stream used for playback. This will be because we just started a seek. You can view how buffer switching works by watching which of these become
active as the video plays. The
size is the amount of data in bytes that the reader is holding in memory.
mLastVideoTime is the presentation time of the last processed video frame.
MSE videos will have data evicted as they are played. This size threshold for eviction defaults to 75MB and can be changed with the
media.mediasource.eviction_threshold variable in
about:config. When data is appended via the
appendBuffer method on a
SourceBuffer an eviction routine is run. If data greater than the threshold is held then we start removing portions of data held in the readers. This will be noticed in
about:media by the start and end ranges being trimmed or readers being removed entirely.
This internal data is most useful for Firefox media developers. If you encounter stalls playing videos or unusual buffer switching behaviour then copy/pasting the data from
about:media in a bug report can help with tracking the problem down. If you are developing an MSE player then the information may also be useful to find out why the Firefox implementation may not be behaving how you expect.
Media Source Extensions is still in progress in Firefox and can be tested on Nightly, Aurora and Beta builds. The current plan is to enable support limited to YouTube only in Firefox 37 on Windows and Mac OS X for MP4 videos. Other platforms, video formats and wider site usage will be enabled in future versions as the implementation improves.
To track work on the API you can follow the MSE bug in Bugzilla.