Video.js: frame-accurate subtitles

In the post-production and broadcast industry videos are processed frame by frame.
Subtitles follow the same rule : in and out timecodes are frame accurate (ex: 00:00:23:22 -> 23 seconds and 22 images).

Browsers process videos with millisecond timestamps, that's why subtitles have to be converted based on video framerate: for video at 25 images per second the previous timecode would look like 00:00:23.880 in milliseconds format (1000/25 * 22 = 880).

Frame accuracy is really important : subtitles need to disappear right before the next cut and appear right after the previous one.
Problem is that default HTML5 video's refresh rate is too low: subtitles appear and disappear too late creating a poor viewer's experience (and a non-broadcast compliant one).

Example below higlights the issue. Subtitles in the fixed video disappear right before the change of plan:

It might seem marginal, but the post production industry needs frame accuracy.

Due to browser limitations, timeupdate (event fired when the playing position of a video has changed) is fired every 150-250 milliseconds. It is not enough for frame-accuracy: 25fps means an update every 40ms.
We need to compute which subtitle has to be displayed every frame (instead of doing it every 4-5 frames by default). Video.js subtitles engine does the computation each time the text track's attribute activeCues getter is called.

text-track.js extract from Video.js source code:

Object.defineProperty(tt, 'activeCues', { get() { if (!this.loaded_) { return null; } // nothing to do if (this.cues.length === 0) { return activeCues; } const ct = this.tech_.currentTime(); const active = []; for (let i = 0, l = this.cues.length; i < l; i++) { const cue = this.cues[i]; if (cue.startTime <= ct && cue.endTime >= ct) { active.push(cue); } else if (cue.startTime === cue.endTime && cue.startTime <= ct && cue.startTime + 0.5 >= ct) { active.push(cue); } } changed = false; if (active.length !== this.activeCues_.length) { changed = true; } else { for (let i = 0; i < active.length; i++) { if (this.activeCues_.indexOf(active[i]) === -1) { changed = true; } } } this.activeCues_ = active; activeCues.setCues_(this.activeCues_); return activeCues; }, set() {} }); 

Then you need to call trigger('cuechange') on the text track to make sure the video display is up to date:

player.textTracks()[0].activeCues; // computes the current subtitle based on current time player.textTracks()[0].trigger('cuechange'); // updates the display 

requestAnimationFrame is optimized for animations and has a lot less delay than setInterval or setTimeout, so we are going to use it for our time sensitive loop (Frame rate control source here).

Here is the complete source code:

var fps = 25; var now; var then = Date.now(); var interval = 1000/fps; var delta; function reloadCues() { requestAnimationFrame(reloadCues); now = Date.now(); delta = now - then; if (delta > interval) { then = now - (delta % interval); if(videojs.players.player.textTracks().length == 1) { videojs.players.player.textTracks()[0].activeCues; videojs.players.player.textTracks()[0].trigger('cuechange') } } } reloadCues(); 

Demo project is available here.

Originally published on my personal blog.