How Netflix Builds Code

Rahul Somasunderam

What does a team look like?

How does a Netflix team look like?

What is the full cycle?

What is supported?

What kind of teams support these tools?

What it really looks like

Guardrails, not gates

"TX 375 Road - Mountains 2" by formulanone is licensed with CC BY-SA 2.0.

The unpaved road

"Olympic Mountain Bike Cross Country Cycling Race - London 2012 MTB" by Sum_of_Marc is licensed with CC BY-NC-ND 2.0.

The paved road

"UCI Worlds in Valkenburg, September 2012 THE TEAM TIME TRIAL" by sjrowe53 is licensed with CC BY-NC-ND 2.0.

What we support

🤷 A few thousand users
🤷 A few hundred teams
~45k repositories
~45k job definitions
~600k builds per week

What are the tools like

What are the tools like

What are the tools like

What are the tools like

What are the tools like

The Java Path

The Java Path

allprojects {
  apply plugin: 'netflix.nebula'
}

Nebula is Netflix’s take on what gradle builds should be like. It is a custom distribution of Gradle that we’ve developed. Large parts of it are open source, but there are some bits and pieces that are internal to Netflix. The non-OSS bits mostly deal with getting to Netflix’s repositories for downloading and publishing artifacts.

It’s as simple as adding one line to your build.gradle. The Nebula plugin applies lots of open source Nebula plugins to your project to configure it correctly. But it also checks what other plugins are applied and configures them in a way that’s appropriate for building at Netflix. It also configures publishing correctly. Most teams apply netflix.nebula to their allProjects block and then add just a simpler plugin on each sub project - e.g. a Springboot plugin or a java library plugin.

The Node Path

Version Control

The CI Phase

The CI Phase

So we built something called Rocket CI which is a layer on top of Jenkins that makes it easy to do that at Netflix. It’s opinionated, but it’s easier to maintain.

You can curl an endpoint to setup a Jenkins job that does everything correctly and specify what kinds of builds you want - only the main branch, or pull requests, or tags, or a combination of these.

Most build configurations are simple enough that we expose a handful of properties through a yaml file that’s in the project’s repository. The build itself is a shell script that can be checked in.

If you have as many repositories as we do, and you push code as often as we do, webhooks become unreliable as a way to trigger builds. Even falling back to polling can result in service degradation.

So we built plugins for our Bitbucket, Jenkins, Artifactory and Spinnaker to rely on SNS and SQS.

Whenever a code event occurs, the plugin for Bitbucket emits an event that can be consumed by any service. Rocket consumes that event and triggers a build.

Whenever a build event occurs, Rocket emits that event on the SNS topic.

The CI Phase

The CI Phase

The CI Phase

More CI tasks

rocket

The CD Phase

The CD Phase

Operating

Operating

Astrid helps keep track of what’s running.
- During a bake, it will gather information on what debians go into each AMI.
- It can find out from metadata that we publish what goes into each debian.
- It know about each jar & npm package we’re running in addition to the debians.
- It can do the same thing for docker layers too.
Once a quarter, teams that publish libraries will get together and say what the minimum version of their library should be. Astrid will take this input and start sending nudges to anyone who consumes the library pointing to where they are using an older version. This allows teams to update their dependencies. If the version is too old, we can prevent new deploys from happening.
But it doesn’t always have to happen once a quarter. For example, if you depend on an OSS library for which a CVE (Common Vulnerabilities and Exposures) has been published for a specific version, we can block your deploys until you update to a safe version. This can happen at any time. Our build tooling also has a way to check for this, and can prevent your builds from happening.

Too many tools?

Newt

We have a tool for this called Newt.

You can think of Newt as Yeoman+Sdkman+NVM+RVM plus a lot more.

It has a templating system like Yeoman does. That helps you bootstrap your repository.

But it also knows how to talk to all these other services and configure them correctly.

Furthermore, it also wraps build tools. So you can configure your repository with information on what build tool needs to be selected. E.g. Node 12.16.2 and NPM 6.14.4. Then you can call newt build, and it will download the right tools and call the appropriate command for your build tool. This means that when you check out a new repository, you don’t need to try to figure out how to build it.

For most app-types, you can go from wanting to start a new app to having a hello-world up and running in production in about 15 minutes.

Thank you

@rahulsom