This blog was written by me after a long gap of close to 7 months. Many reasons including busy work schedule, some health issues in the middle and a little bit of laziness contributed to this. I hope to be a more active blogger going forward.

In this blog series, I will cover the following topics:

• GKE default IP address management(without IP aliasing)
• VPC native GKE clusters using IP aliasing
• Container native load balancing using Network Endpoint groups(NEG)

The first blog in this series will talk about GKE default IP address management.

Following are the Kubernetes abstractions that needs IP addresses:

• Node IP address – Assigned to individual nodes. The node ip address is assigned from the VPC subnet range.
• Pod IP address – Assigned to individual pods. All containers within a single pod share same IP address.
• Service IP address- Assigned to individual service

By default, “/14” address gets allocated for cluster IP range. Pod and service IP addresses comes out this pool. “/24” address that comes out of the cluster IP range gets assigned to each individual node and is used for pod IP allocation. “/20” address that comes out of the cluster IP range gets assigned for Kubernetes services. The user has a choice to select cluster IP range when creating the cluster.

To illustrate some of the above points, I have created a 3 node Kubernetes cluster with IP aliasing disabled. By default, VPC native clusters(ip aliasing enabled) is disabled and has to enabled manually. In the future GKE release, VPC native clusters will be the default mechanism.

Cluster output:

Following output shows the 3 node GKE cluster created using default IP options in the “default” subnet.

 $ kubectl get nodes -o wide
 NAME                                             STATUS    ROLES     AGE       VERSION          EXTERNAL-IP      OS-IMAGE                             KERNEL-VERSION   CONTAINER-RUNTIME
 gke-cluster-default-default-pool-0f6611b6-k68m   Ready     <none>    9d        v1.12.7-gke.10   35.225.186.179   Container-Optimized OS from Google   4.14.106+        docker://17.3.2
 gke-cluster-default-default-pool-0f6611b6-vrdc   Ready     <none>    9d        v1.12.7-gke.10   130.211.227.64   Container-Optimized OS from Google   4.14.106+        docker://17.3.2
 gke-cluster-default-default-pool-0f6611b6-xbql   Ready     <none>    9d        v1.12.7-gke.10   35.222.130.158   Container-Optimized OS from Google   4.14.106+        docker://17.3.2  

Cluster IP address:
Following output shows the allocated range of cluster IP address(10.60.0.0/14). Pod and service ip addresses are allocated out of this range. If we do not specify the cluster IP, GKE automatically assigns a “/14” address for cluster IP. We can manually specify cluster IP using “–cluster-ipv4-cidr” option when we create the cluster.

 $ gcloud container clusters describe cluster-default | grep Ip
 clusterIpv4Cidr: 10.60.0.0/14
 nodeIpv4CidrSize: 24
   podIpv4CidrSize: 24
 servicesIpv4Cidr: 10.63.240.0/20 

Node IP address:
Following output shows the node IP external and internal address, pod IP address allocated for each node. Node internal IP address(eg: 10.128.0.40) is assigned from the VPC subnet(10.128.0.0/20). Each node is allocated a /24 pod IP address range which allows maximum of 256 pods in a node. 10.60.2.0/24 is the pod ip address range allocated for node 1. To allow for pods going up and down, we need a buffer of pod IPs. Because of this reason, only 110 pod ip addresses(instead of 256) are allocated for each node.

 $ kubectl describe nodes | grep -i -e ip -e podcidr
   InternalIP:   10.128.0.40
   ExternalIP:   35.225.186.179
 PodCIDR:                     10.60.2.0/24
   InternalIP:   10.128.0.42
   ExternalIP:   130.211.227.64
 PodCIDR:                     10.60.0.0/24
   InternalIP:   10.128.0.41
   ExternalIP:   35.222.130.158
 PodCIDR:                     10.60.1.0/24 

Let’s deploy a sample application. The following commands will create 3 pods and create a “Nodeport” service on top.

kubectl run web --image=gcr.io/google-samples/hello-app:1.0 --replicas=3 --port=8080
kubectl expose deployment web --target-port=8080 --type=NodePort

Let’s look at the pod ip address allocated. 2 pods get allocated on single node, so it gets ip address out of “10.60.2.x/24” range and another pod gets IP out of “10.60.1.x/24” range.

$ kubectl get pods -o wide
NAME                   READY     STATUS    RESTARTS   AGE       IP          NODE
web-6d695d4565-4cg6t   1/1       Running   0          1m        10.60.1.3   gke-cluster-default-default-pool-0f6611b6-xbql
web-6d695d4565-6dj79   1/1       Running   0          1m        10.60.2.4   gke-cluster-default-default-pool-0f6611b6-k68m
web-6d695d4565-dhfqn   1/1       Running   0          1m        10.60.2.5   gke-cluster-default-default-pool-0f6611b6-k68m

Let’s look at service IP. Service IP(10.63.243.144) is allocated out of the range(10.63.240.0/20) which in turn comes out of the cluster IP range.

$ kubectl get services
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
kubernetes   ClusterIP   10.63.240.1     <none>        443/TCP          17m
web          NodePort    10.63.243.144   <none>        8080:32668/TCP   2m

For the pods to talk across each nodes, gke needs to add explicit routes to the routing table as shown below. Since we do not have IP aliasing enabled, GCP VPC is not aware of these pod and service ip addresses and that’s the reason gke explicitly adds these routes to the routing table.

gke-cluster-default-3db044-19175121-779c-11e9-b601-42010a800159  default    10.60.0.0/24    us-central1-b/instances/gke-cluster-default-default-pool-0f6611b6-vrdc  1000
gke-cluster-default-3db044-192c50f1-779c-11e9-b601-42010a800159  default    10.60.1.0/24    us-central1-b/instances/gke-cluster-default-default-pool-0f6611b6-xbql  1000
gke-cluster-default-3db044-193c1da9-779c-11e9-b601-42010a800159  default    10.60.2.0/24    us-central1-b/instances/gke-cluster-default-default-pool-0f6611b6-k68m  1000

In the next 2 blogs, I will cover VPC native clusters using IP aliasing and Container native load balancing.