Netflix is Chill with Breaking Their Services

Do it for the tests

TLDR;

Netflix has a LOT of services. Too many to count and too many to manage. It’s way too much to query from these services and put all these things like recommendations, favorite lists, and top viewed selections on the client to piece together.

Netflix’s solution? Add a middle tier server called MAP in between the frontend and backend that puts the API responses together and makes it cleaner for the client to read.

Now everything depends on MAP and Netflix does a ton of latency and intentional failure testing to make sure this middle layer is working smoothly.

WTF is MAP?

Because there’s way too many services at Netflix to recommend you an awesome show, they came up with MAP, Merchandising Application Platform, to simplify calling these services.

API responses go from server → MAP → client. Sometimes the request can just be from a cache if it’s already there.

Can’t they just read directly from the server?

Sure, they can but it’ll make the client really really complicated putting all the API response pieces together.

Sometimes there’s duplicated logic that can just be in a single place and that’s what the middle tier is for.

As an example, sometimes credits are shown after 95% of the show. Other times it’s after 90% of the show. This branched logic doesn’t need to sit in the client and can just be queried by a single endpoint on the MAP.

Requirements for Testing

With the MAP layer, this needs to be tested by it’s dependencies and it’s dependents.

Here’s a summary of the requirements:

1. Don’t let the user freak out if something breaks. Make the UI still presentable.

2. Don’t let your dependents down. Don’t let your dependencies down. Keep the MAP failures to itself

3. Don’t overload the dependencies. If there’s a lot of requests by MAP, make sure the load is controllable for the server.

How’d they broke the MAP

The overall strategy is to just break the connection with MAP to everything else and see what happens. How to break it is a different story. Trying to break prod without breaking prod is a challenge but here’s how they did it: