Kubernetes/AWS Networking for Dummies (like me)

This article is a bit of education for myself (and anyone else who wants to read it) on K8s and AWS and their different approaches to networking. If I’ve misunderstood anything, please let me know!

Julia Evans has just written a really excellent post on this subject, which you should definitely read!

So, how on earth does k8s networking work with AWS?

How Does a K8s Cluster Work?

A pod is a vital k8s concept, it is set of one or more containerised applications that always deploy and run together.

Who gets their own IP address?

Kubernetes DOES NOT work like this. For excellent reasons that I agree with they decided on a different model. K8s decided that in an orchestrated world, every POD should have a unique (currently IPv4) ip address like 192.0.2.0 and all the containers running “inside” the pod should have port numbers relative to the pod’s IP address like 192.0.2.0:123. They recommend allocating a block of IPs to each node in the cluster for assigning individual IPs to each pod on the node.

You can get an address range from which to allocate all your node IP blocks via an AWS VPC.

This is subtle but revolutionary. Each pod has its own ip address just like a VM and yet multiple pods run on a single AWS instance. Some fancy footwork will be required because this is not AWS’s normal view of the world.

What does k8s do for you?

Note for a master you just assign a boring old static IP address unless you are doing something exciting like running pods on masters (personally, I avoid that kind of excitement).

AWS throws up it’s networked hands — Use an Overlay

What’s the heck’s an overlay?

The overlay network uses etcd or equivalent k/v function to store mappings between the virtual IP addresses it’s using for the pods and the nodes’ own IP addresses. A daemon runs on each node and is responsible for checking information in etcd and routing packets to the correct pods (and ultimately containers). You must give the overlay network an address range to work with. You can choose how many IP addresses are available on each node for pods (for example, you can set this high if you want to run loads of pods on each node and you have plenty of IP addresses).

What’s the alternative?

Another alternative is to use a non-overlay solution like Calico (non-IPIP mode). That’s fast and lightweight compared to an overlay approach BUT only works within a single VPC subnet, so cannot handle a cluster that stretches over multiple AZs. To handle multi-AZ. you need to enable overlay, at least for packets that pass between those between those AZs (“microoverlay” is apparently the word for this in Calico).

What do we use?

Summary

Extras

Please hit the Recommend button below if you found this article interesting or helpful, so that others might be more likely to find it.

Check out MicroBadger to explore image metadata, and follow Microscaling Systems on Twitter.

References

SciFi author interested in tech, engineering, science, art, SF, economics, psychology, startups. Chaotic evil.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store