Home Articles Projects Music About
Follow @stepchowfun

Netflix Party: synchronize Netflix video playback

December 28, 2015
Over the weekend I made Netflix Party, a Chrome extension to synchronize Netflix video playback on multiple computers. You know, for Netflix nights with that long-distance special someone. You can get it here from the Chrome Web Store.
screenshot
Figure 1: watching The Best Offer with Netflix Party.

The server

The basic architecture of Netflix Party is a client-server model. The clients connect to the server to synchronize their playback state. For the server, I wanted something low-latency that many clients could connect to at a time. Node.js fit the bill nicely.
The server turned out to be the easiest part of the project. It uses a simple in-memory store (a JavaScript object) to hold the sessions, which means that the sessions are lost if the server is ever restarted. For an app like Netflix Party, that isn’t a big deal, but maybe later I’ll switch to something like Memcached or Redis. Setting up the session store is just one line of code:
// in-memory store of all the sessions
// the keys are the session IDs (strings)
// the values have the form: {
//   id: '84dba68dcea2952c',             // 8 random octets
//   lastActivity: new Date(),           // used to find old sessions to vacuum
//   lastKnownTime: 123,                 // milliseconds from the start of the video
//   lastKnownTimeUpdatedAt: new Date(), // when we last received a time update
//   state: 'playing' | 'paused',        // whether the video is playing or paused
//   videoId: 123                        // Netflix id the video
// }
var sessions = {};
Sessions that have been idle for more than an hour are vacuumed.
The server also renders a simple homepage at netflixparty.com, using my minimalistic templating engine for Node.js.

The client

The browser extension turned out to be obnoxiously nontrivial to build, since Netflix doesn’t expose any JavaScript functions for manipulating video playback (or at least I couldn’t find them). That means, unfortunately, we have to resort to simulating user actions like clicking on buttons.

Playing and pausing

It didn’t take too long to figure out how to play and pause the video. We simply simulate a mouse click on the play/pause button:
var pause = function() {
  uiEventsHappening += 1;
  $('.player-play-pause.pause').click();
  return delay(1)().then(hideControls).then(function() {
    uiEventsHappening -= 1;
  });
};

var play = function() {
  uiEventsHappening += 1;
  $('.player-play-pause.play').click();
  return delay(1)().then(hideControls).then(function() {
    uiEventsHappening -= 1;
  });
};
The uiEventsHappening variable keeps track of how many simulated UI actions are in-flight. We don’t respond to user input unless uiEventsHappening === 0, to distinguish between simulated user actions and real user actions.

Hiding the playback controls

The Netflix player automatically shows the playback controls when the cursor is moved or any other user action occurs, and the controls hide automatically after a couple seconds. Unfortunately, naively firing a click event on the play/pause buttons causes the controls to stay visible forever. The fix is to simulate a subsequent click somewhere on the video:
var hideControls = function() {
  uiEventsHappening += 1;
  var player = $('#netflix-player');
  var mouseX = 100; // relative to the document
  var mouseY = 100; // relative to the document
  var eventOptions = {
    'bubbles': true,
    'button': 0,
    'screenX': mouseX - $(window).scrollLeft(),
    'screenY': mouseY - $(window).scrollTop(),
    'clientX': mouseX - $(window).scrollLeft(),
    'clientY': mouseY - $(window).scrollTop(),
    'offsetX': mouseX - player.offset().left,
    'offsetY': mouseY - player.offset().top,
    'pageX': mouseX,
    'pageY': mouseY,
    'currentTarget': player[0]
  };
  player[0].dispatchEvent(new MouseEvent('mousemove', eventOptions));
  return delay(1)().then(function() {
    uiEventsHappening -= 1;
  });
};
That magical spell was obtained by lots of trial and error. But that’s not even the hackiest part.

Seeking in the video

Controlling the playback position—the main point of Netflix Party—was by far the most difficult part. It took me two days to come up with this wretched incantation:
var showControls = function() {
  uiEventsHappening += 1;
  var scrubber = $('#scrubber-component');
  var eventOptions = {
    'bubbles': true,
    'button': 0,
    'currentTarget': scrubber[0]
  };
  scrubber[0].dispatchEvent(new MouseEvent('mousemove', eventOptions));
  return delay(10)().then(function() {
    uiEventsHappening -= 1;
  });
};

var seek = function(milliseconds) {
  uiEventsHappening += 1;
  var eventOptions, scrubber;
  return showControls().then(function() {
    // compute the parameters for the mouse events
    scrubber = $('#scrubber-component');
    var factor = milliseconds / getDuration();
    var mouseX = scrubber.offset().left + Math.round(scrubber.width() * factor); // relative to the document
    var mouseY = scrubber.offset().top + scrubber.height() / 2;                  // relative to the document
    eventOptions = {
      'bubbles': true,
      'button': 0,
      'screenX': mouseX - $(window).scrollLeft(),
      'screenY': mouseY - $(window).scrollTop(),
      'clientX': mouseX - $(window).scrollLeft(),
      'clientY': mouseY - $(window).scrollTop(),
      'offsetX': mouseX - scrubber.offset().left,
      'offsetY': mouseY - scrubber.offset().top,
      'pageX': mouseX,
      'pageY': mouseY,
      'currentTarget': scrubber[0]
    };

    // make the "trickplay preview" show up
    scrubber[0].dispatchEvent(new MouseEvent('mouseover', eventOptions));
  }).then(delay(10)).then(function() {
    // simulate a click on the scrubber
    scrubber[0].dispatchEvent(new MouseEvent('mousedown', eventOptions));
    scrubber[0].dispatchEvent(new MouseEvent('mouseup', eventOptions));
    scrubber[0].dispatchEvent(new MouseEvent('mouseout', eventOptions));
  }).then(delay(1)).then(hideControls).then(function() {
    uiEventsHappening -= 1;
  });
};
Here are the steps it takes:
  1. Jiggle the mouse so the playback controls appear. Wait 10ms for the UI to respond. 1ms is apparently not enough.
  2. Simulate the cursor hovering over the playback slider so the thumbnail preview appears, because apparently the playback slider doesn’t respond to input unless the preview is visible. Wait 10ms for the UI to respond.
  3. Click on the playback slider (called a “scrubber control” in the minified Netflix code) at the appropriate position to seek the video.
  4. Invoke the hideControls() magic from above.
The above snippet is almost certainly the worst code I have ever written.

Synchronization

First, we have this helper function that all Ajax requests go through:
var roundTripTimeRecent = [];
var roundTripTimeMedian = 0;
var localTimeMinusServerTimeRecent = [];
var localTimeMinusServerTimeMedian = 0;
var ajax = function(relativeUrl, method, data) {
  return new Promise(function(resolve, reject) {
    var startTime = (new Date()).getTime();
    $.ajax({
      url: 'https://www.netflixparty.com' + relativeUrl,
      method: method,
      data: method === 'POST' ? JSON.stringify(data) : data
    }).always(function(data, textStatus, jqXHR) {
      // calculate round trip time
      roundTripTimeRecent.push((new Date()).getTime() - startTime);
      if (roundTripTimeRecent.length > 10) {
        roundTripTimeRecent.splice(0, roundTripTimeRecent.length - 10);
      }
      var sorted = roundTripTimeRecent.concat().sort();
      roundTripTimeMedian = sorted[Math.floor(sorted.length / 2)];

      if (data.lastActivity !== undefined) {
        // calculate client-server time offset
        localTimeMinusServerTimeRecent.push((new Date()).getTime() - (new Date(data.lastActivity)).getTime());
        if (localTimeMinusServerTimeRecent.length > 10) {
          localTimeMinusServerTimeRecent.splice(0, localTimeMinusServerTimeRecent.length - 10);
        }
        sorted = localTimeMinusServerTimeRecent.concat().sort();
        localTimeMinusServerTimeMedian = sorted[Math.floor(sorted.length / 2)] - Math.round(roundTripTimeMedian / 2);
      }
    }).done(resolve).fail(reject);
  });
};
Note the instrumentation. The roundTripTimeMedian variable stores an estimate for the round-trip delay time to the server. It is the median of the measurements from the last 10 Ajax requests. I chose a simple median filter because it is robust to outliers. The round-trip time is used to compensate for network latency (we simply divide it by 2 and assume the one-way latencies are symmetrical).
The localTimeMinusServerTimeMedian variable estimates how the local time is different from the server time. Since it is measured, we don’t need to rely on accurate time zone information from the browser. Again, we take the median of the last 10 values to get a stable measurement. In the future, I might try a fancier clock synchronization protocol like NTP. But for now, the approximation is good enough.
Whenever a user plays, pauses, or seeks the video, the event is sent to the server with a timestamp (lastKnownTimeUpdatedAt) in server time. The server only knows about its own clock. Each client is responsible for converting its own times into the reference frame of the server.
Now, here’s the tricky part. Suppose the client receives some nominal playback position from the server. Seeking to that position, in general, takes nonzero time. Buffering can take several seconds. By the time you’ve jumped to that position, buffered, and started playing the video, the other clients are several seconds ahead!
My solution is to jump a little bit ahead of the nominal time (2 seconds in this case), and pause the video there while buffering in the background. A timer is set to wait there until the other clients have caught up, and then resume playback—and hopefully buffering is complete by then.
This has the added benefit of increasing the available temporal resolution. By just clicking on the scrubber, you get about 3-5 seconds of precision (a pixel covers about that much time depending on the window size and video length). But by jumping a little bit ahead and pausing for a moment to compensate, the resolution increases to a couple dozen milliseconds.

Conclusion

This was a fun (and, at times, stressful) weekend project, but now I can organize remote Netflix parties. Happy New Year!
 Hashpass: a stateless password manager for Chrome
Fun with promises in JavaScript 
© 2022 Stephan Boyer