Photo courtesy of Timecode Lab
With all the fuss about the UI changes in recently released iOS7, you'd be forgiven for missing an interesting little feature that promises to change the way we think about video play-back. On the iPhone 5s at least, you can capture video at 120 frames per second which makes for a smooth slow motion experience when played back at 60 fps.
I'm sure that there are much better applications than fashion shows, but it's indicative nonentheless.
What's it Good For?
So what interest is all this to makers of web based applications? Well the good news is, that we have a rough diamond in a little touted web audio and video method, known as playbackRate. PlaybackRate — as you can probably imagine — allows you to adjust the speed at which web video and audio is played at. This can be useful for a number of reasons - let's take the example of podcasts. Imagine you have a 50 minute commute, but your favourite podcast lasts an hour. Up the play-back rate slightly and you can probably squeeze it in - you may not even notice the difference in pace. Due to the algorithms used the pitch is maintained and only the rate is adjusted, so within certain reasonable limits it actually sounds pretty good!
As part of a the Hyperaudio project, I've been playing about with playbackRate to allow transcribers to adjust play-back speed when using our media transcription tools. In this post I'll explore support and write about the limits of its implementations in various browsers and how we can still use it for fun and profit.
Interesting aside: at the time of writing an equivalent of playbackRate doesn't appear to be implemented in current versions of Adobe's Flash Plugin, which means fallbacks for older browsers are unlikely. Perhaps more interestingly, services like YouTube who currently default to Flash will need to switch to HTML5 to support playbackRate.
How it Works
PlaybackRate holds a number which represents a multiple of the original media speed. For example 0.5 represents half speed and 2 represents double speed. Note, currently negative values are not allowed, this in theory would play things backwards, in practice this is difficult for browsers to handle and is not yet implemented. For those interested the reason it isn't implemented (at least in video) is that popular video codecs use a system of keyframes and use differences from those keyframes to calculate successive frames. I asked Mozilla's Chris Pearce — who works on implementing video in the browser — what it would take to play video backwards. His response:
"(...) in order to play backwards, basically the player needs to decode every frame since the previous keyframe in the timeline and the next keyframe, and run through that sequence in reverse order while playing, and repeat this process.
A single 1080p video frame takes up 1920x1080x4=8294400 bytes (about 8 megabytes). For 30fps video with keyframes every two seconds you're going to need to cache 480MB of video data. If this video data is stored in GPU memory, then you've swallowed most or all of the dedicated GPU memory on most hardware, so you'll cause performance problems."
So it looks like that option is out, at least for a while.
Results from our testing shows support looks something like this:
|Mobile Chrome (Android)||✖|
|Mobile Firefox 24+||✔|
|Mobile Safari 6+ (iOS)||✔|
As you can see there's mixed coverage but all modern browsers seem to coming up to speed.
There are a few caveats, so let's take a look at the issues and how we can work around them.
For all browsers that support playbackRate the audio will cut out at certain rates. It's probably wise then to limit your use of playbackRate to a certain range, currently this appears to be between 0.5 (half speed) and 4 (quadruple speed). Worryingly, the audio in Chrome (and Opera) may become corrupt if you move much beyond these limits and then back again. (WARNING turn down volume and remove earphones when testing).*
*This appears to be fixed in more recent versions.
Here's an example to try:
You may notice that Safari, although coping with audio slightly better than Chrome, copes less well with variable rate video playback. Firefox seems to work OK on all fronts.
defaultPlaybackRate and ratechange
In addition to playbackRate we also have defaultPlaybackRate which — unsurprisingly — lets us set the default play-back rate, which effectively means the rate at which that the media resets to, in the case that we change something like the source, or in some browsers (IE9/10) if an ended event is generated. We would usually set defaultPlaybackRate before playing the media while playbackRate allows us to change it during video play-back.
We also have a new event called ratechange which fires every time the playbackRate changes. Note however that while Firefox will fire this event when the media source is switched, (regardless of whether the rate changes) Chrome/Opera and Internet Explorer will not.
We have a different situation with iOS(7) where the playbackRate only appears to have any effect when the media is paused, it has no effect while the media is playing.
Internet Explorer 9+ appears to reset the playBackRate to the default when the media ends. IE9 appears to behave the same as IE10 with playback at different speeds working well.
In general we found that most browsers stop playing audio below a playbackRate of 0.5 and above 4. Video will continue to play silently however. So it's recommended for most applications that you limit the range to between 0.5 and 4, which in practice is good enough for most applications.
Another thing to note is that the duration value for any piece of media does not change in response to playbackRate changes, similarly currentTime. So if we think back to the Podcast commute case, we cannot adjust the rate to fit a specific time easily, without providing the user with a calculated duration (or time remaining).
Here's a more comprehensive example which includes defaultPlaybackRate and ratechange.
So as we can see playbackRate can be used on most browsers, but you should tread carefully for now. For your convenience we've abstracted the differences and support playbackRate in the latest version of jPlayer.
To sum up:
- Do not expect defaultplayBackRate to work consistently between browsers.
- Do restrict playbackRate values to between 0.5 and 4 (inclusively).
- Be aware that the ratechange event may fire when the source is changed, (regardless of whether the rate changes).
Note also that, media isn't easy to test. On areweplayingyet.com the test only checks if the playbackRate value is set but that's not enough to see if that value is used and hence to ascertain if playbackRate is actually implemented.
Thanks to my colleague Mark Panaghiston for much of the research conducted for this article.
This article has been written by Mark Boas