online posts 15
networking 101: how computers talk

lesson 2.3 · chapter 2 of 5: connecting hosts · 15 min

What's a data center?

objective

Say what a data center is and what’s inside it, follow a request from your phone to a server in a rack and back, and see why everything comes in twos.

1 explain

Where the internet lives

Servers need a home

When you watch a video or open a website, your message goes to a server: a computer whose job is to answer other computers. A video server sends videos; a web server sends web pages. Servers never sleep: someone, somewhere, always wants a video.

Think of a giant hotel, but for computers instead of people. Every room is full of servers, the lights never go off, and there’s staff on duty day and night. That’s a data center: a building (or a big room in one) full of servers, with everything they need to keep running all the time.

in one sentence A data center is a building full of servers, kept running day and night.

What’s inside

  • Racks: tall metal cupboards, about two metres high, with servers stacked inside like books on shelves. Rack space is measured in U (one unit, 4.45 cm); a normal rack holds 42U.
  • Power: lots of it, from two separate supplies, plus big batteries and diesel generators, so the servers keep running even if the electricity goes off.
  • Cooling: servers get hot, like a laptop on your knees, times thousands. Rows of racks face each other so cold air goes in at the front and hot air comes out of the back, where it’s pulled away.
  • Security: guards, cameras, locked cages and ID cards. You can’t just walk in.
  • The network: switches and routers (from lesson 2.1) joining every server to every other server, and to the internet.

in one sentence Inside: racks of servers, lots of power, lots of cooling, tight security, and a network joining it all.

Two of everything

What happens if a car gets a flat tyre? You use the spare. Data centers have a spare for everything: two power supplies, two cables, two switches, two internet connections. Having a spare ready like this is called redundancy. If one thing breaks, the other takes over and nobody notices.

in one sentence Redundancy means a spare for everything, so one broken part doesn’t stop anything.

Leaf and spine

Think of a school with long corridors of classrooms, and two main staircases. Every corridor joins both staircases. To get from any classroom to any other, you walk along your corridor, up or down a staircase, and along the other corridor: always the same short trip. And if one staircase is closed, you use the other.

Most data center networks are built just like that, and it’s called leaf-spine:

  • a leaf switch sits at the top of each rack, and that rack’s servers plug into it (people also call it a top-of-rack switch). The leaves are the corridors.
  • the spine switches sit in the middle. They’re the staircases.
  • Every leaf connects to every spine. So any server reaches any other by going leaf → spine → leaf.

in one sentence In leaf-spine, every leaf connects to every spine, so any two servers are always leaf → spine → leaf apart.

The door and the guard

Where the internet comes into the building, there’s a router (the post office from lesson 2.1) called the edge router, because it sits at the edge. Next to it is a firewall: a security guard for messages. It checks every message coming in and only lets through the kinds that are allowed, like requests for a video.

in one sentence The edge router brings the internet in, and the firewall checks every message at the door.

North-south and east-west

Draw the data center like a map, with the internet at the top:

  • messages going in and out, between the internet and the servers, travel up and down the map. They’re called north-south traffic.
  • messages between servers inside the data center, like a web server asking a database (a server that stores information, like a giant filing cabinet) for something, travel across the map. That’s east-west traffic.

And the cloud? It’s just someone else’s data center. When a company uses the cloud, it rents servers in a data center that a cloud company runs.

in one sentence North-south is traffic in and out of the data center; east-west is servers talking to each other inside it.

2 show

Inside a small data center

From the top: the internet comes in through an edge router and a firewall (a guard that only lets allowed messages in). Below them are two spine switches, three leaf switches, and a rack under each leaf: web servers, the servers that run the app (the program behind the website), and databases.

in one sentence Internet → edge router → firewall → spines → leaves → racks of servers.

3 follow along

Follow a request into the data center

You tap play on a video. Step through and watch which way each message goes.

What to notice:

  • Between any two racks, the trip is always leaf → spine → leaf. No server is far from any other.
  • The firewall sits where the internet comes in, so every north-south message gets checked.
  • When spine 1 broke, nothing else had to change. Engineers can fix it, or even unplug it for an upgrade, while everything keeps running.

Try it on your own computer (optional)

You can see the routers between you and a data center. This lists every router your message passes through on its way to a website (it takes up to a minute):

  • Windows, in Command Prompt: tracert networkdoodles.com
  • Mac, in Terminal: traceroute networkdoodles.com

Each numbered line is one router, with how long the round trip took in milliseconds. The first is your home router. The last few are usually inside, or at the door of, the data center that hosts the site. Lines with * * * are routers that chose not to answer: that’s normal.

in one sentence Requests come in through the router and firewall, go leaf-spine-leaf between servers, and still get through when one spine breaks.

4 challenge

Plan the cables

A data center is growing. It will have 4 spine switches and 10 leaf switches, wired leaf-spine.

  1. How many cables between the leaves and the spines?
  2. A server in the rack under leaf 2 talks to a server under leaf 9. How many switches does each message go through?
  3. Two spines break at the same time. Can every leaf still reach every other leaf?
hint

Every leaf connects to every spine. So each leaf has one cable per spine.

hint

Leaf to spine to leaf, whichever leaves they are.

show the solution
  1. 40 cables: each of the 10 leaves has one cable to each of the 4 spines, 10 × 4 = 40.
  2. Three switches: leaf 2, one spine, leaf 9. Always three, for any two leaves.
  3. Yes. Every leaf is still wired to the two spines that are left. The network has half as much room for traffic until they’re fixed, but nothing is cut off.

5 check: test yourself

Pick an answer for each question. You'll see straight away whether it's right, and why. Questions marked what happens next? are about the animations.

  1. What’s a data center?

  2. In a leaf-spine network, what does every leaf switch connect to?

  3. what happens next?In the animation, spine 1 breaks while you’re watching a video. What happens next?

  4. A web server asks a database in another rack for a file. What’s that kind of traffic called?