Configure Multus with IPVLAN and Whereabouts

Standard

This document explains how to configure Pods in Google Kubernetes Engine (GKE) with multiple network interfaces by using the Multus CNI, the IPVLAN CNI plugin, and the Whereabouts IPAM plugin.

The IPVLAN CNI plugin provides Layer 2 connectivity for additional Pod interfaces, and the Whereabouts IPAM plugin dynamically assigns IP addresses to them.

This setup enables advanced networking configurations, such as separating control plane and data plane traffic for enhanced network isolation and segmentation.

This document is for Cloud architects and Networking specialists who design and architect the network for their organization. To learn more about common roles and example tasks that we reference in Google Cloud content, see Common GKE user roles and tasks.

Before reading this document, ensure that you are familiar with the following concepts:

Benefits of using Multus with IPVLAN

Configuring your Pods with multiple network interfaces by using this solution provides several key advantages. The primary use cases for configuring Multus with IPVLAN in Layer 2 mode are for network segmentation that requires Layer 2 adjacency:

Traffic isolation: isolate different types of traffic for enhanced security and performance. For example, you can separate sensitive management traffic from application data traffic.
Control and data plane separation: dedicate the primary network interface for control plane traffic while directing high-throughput data plane traffic through a secondary IPVLAN interface.
Layer 2 adjacency: fulfill requirements for applications that need direct Layer 2 connectivity between Pods on the same secondary network.

Limitations

Pods configured with Multus interfaces cannot simultaneously use GKE's built-in multi-networking capabilities. A Pod's network configuration must use either Multus or the cluster's built-in multi-networking.

How Multus works with IPVLAN and Whereabouts

Multus is a CNI meta-plugin that enables Pods to attach to multiple networks. Multus acts as a dispatcher, calling other CNI plugins to configure network interfaces based on NetworkAttachmentDefinition resources. You define each additional network by using a NetworkAttachmentDefinition, which specifies which CNI plugin (such as IPVLAN) and IPAM plugin (such as Whereabouts) to use for that network.

The following diagram illustrates the Multus architecture with IPVLAN and Whereabouts plugins.The Whereabouts plugin works with Multus and IPVLAN to handle IP address management (IPAM) for the Pods' additional network interfaces.

Diagram showing how Multus, IPVLAN, and Whereabouts work together in GKE. — Figure 1. Multus architecture with IPVLAN and Whereabouts plugins.

When using Multus with IPVLAN and Whereabouts on GKE, Pods typically have the following interface configuration:

Primary interface (eth0): GKE Dataplane V2 manages this interface, providing default cluster connectivity.
Additional interfaces (net1, etc.): Multus manages these interfaces. Multus invokes the IPVLAN CNI plugin in Layer 2 mode for each NetworkAttachmentDefinition that you specify in a Pod's annotations. This configuration provides Layer 2 connectivity to a secondary VPC network.
IP Address Management (IPAM): you configure the Whereabouts IPAM plugin within the NetworkAttachmentDefinition. The Whereabouts IPAM plugin dynamically assigns IP addresses to the additional IPVLAN interfaces from a predefined range.

Pod scheduling with multiple networks

When you create a Pod and specify a NetworkAttachmentDefinition in its annotations, the GKE scheduler places the Pod only on a node that can satisfy the network requirements. The scheduler identifies nodes within a node pool that have the necessary secondary network interface configured. This node identification process ensures that the scheduler schedules the Pod on a node that can connect to the additional network and receive an IP address from the specified range.

The following sections guide you through configuring Multus with IPVLAN and Whereabouts plugins on your GKE cluster.

Before you begin

Before you start, make sure that you have performed the following tasks:

Enable the Google Kubernetes Engine API.

Enable Google Kubernetes Engine API

If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running the gcloud components update command. Earlier gcloud CLI versions might not support running the commands in this document.
Note: For existing gcloud CLI installations, make sure to set the compute/region property. If you use primarily zonal clusters, set the compute/zone instead. By setting a default location, you can avoid errors in the gcloud CLI like the following: One of [--zone, --region] must be supplied: Please specify location. You might need to specify the location in certain commands if the location of your cluster differs from the default that you set.

Install the kubectl command-line tool.
Set up a GKE cluster running version 1.28 or later with Dataplane V2, IP Alias, and multi-networking enabled. To learn how, see Set up multi-network support for Pods. Enabling multi-networking also enables the Multi-IP-Subnet and Persistent-IP HA Policy features, which eliminate the need for manual inter-node connectivity setup.
Use a GKE-validated version of Multus CNI (such as v4.2.1) for compatibility.

Set up VPC

To set up the Virtual Private Cloud (VPC) to use with Multus, including creating a subnet for node networking and secondary ranges for Pod networking, complete the following steps:

Create a new VPC or use an existing one:
```
gcloud compute networks create VPC_NAME \ --subnet-mode=custom 
```
Replace VPC_NAME with the name of the VPC.
Create a new subnet within this VPC:
```
gcloud compute networks subnets create SUBNET_NAME \  --range=PRIMARY_RANGE \  --network=VPC_NAME \  --region=REGION \  --secondary-range=SECONDARY_RANGE_NAME=SECONDARY_RANGE_CIDR 
```
Replace the following:
- SUBNET_NAME: the name of the new subnet.
- PRIMARY_RANGE: the primary CIDR range for the subnet, such as 10.0.1.0/24. This command uses this range for node interfaces.
- VPC_NAME: the name of the VPC.
- REGION: the region for the subnet, such as us-central1.
- SECONDARY_RANGE_NAME: the name of the secondary IP address range for Pods in the subnet.
- SECONDARY_RANGE_CIDR: the secondary CIDR range for Pods, such as 172.16.1.0/24. Additional interfaces on Pods use this range.
This command creates a subnet with a primary CIDR range for an additional node interface and a secondary range for the additional Pod interfaces.

Create a GKE Standard cluster

Create a GKE Standard cluster with multi-networking enabled:

gcloud container clusters create CLUSTER_NAME \  --cluster-version=CLUSTER_VERSION \  --enable-dataplane-v2 \  --enable-ip-alias \  --enable-multi-networking

Replace the following:

CLUSTER_NAME: the name of the new cluster.
CLUSTER_VERSION: the version of your GKE cluster. You must use version 1.28 or later.

Enabling multi-networking lets you create node pools with multiple network interfaces, which Multus CNI requires.

Create a GKE Standard node pool

Create a GKE Standard node pool connected to additional VPC networks:

gcloud container node-pools create NODEPOOL_NAME \  --cluster CLUSTER_NAME \  --zone "ZONE" \  --additional-node-network network=VPC_NAME,subnetwork=SUBNET_NAME \  --additional-pod-network subnetwork=SUBNET_NAME,pod-ipv4-range=SECONDARY_RANGE_NAME,max-pods-per-node=8

Replace the following:

NODEPOOL_NAME: the name of the new node pool.
CLUSTER_NAME: the name of your cluster.
ZONE: the zone for the node pool, such as us-central1-c.
VPC_NAME: the name of the additional VPC.
SUBNET_NAME: the name of the subnet.
SECONDARY_RANGE_NAME: the name of the secondary IP address range for Pods in the subnet.

This command creates a node pool where nodes have an additional network interface in SUBNET_NAME, and Pods on these nodes can use IP addresses from SECONDARY_RANGE_NAME.

For more information about creating GKE clusters with multi-networking capabilities, see Set up multi-network support for pods.

Apply the Multus deployment

To enable multiple network interfaces for your Pods, install the Multus CNI plugin. Save the following manifest, which includes the required DaemonSet and Custom Resource Definition (CRD), as multus-manifest.yaml:

apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata:  name: ippools.whereabouts.cni.cncf.io spec:  group: whereabouts.cni.cncf.io  names:  kind: IPPool  listKind: IPPoolList  plural: ippools  singular: ippool  scope: Namespaced  versions:  - name: v1alpha1  schema:  openAPIV3Schema:  description: IPPool is the Schema for the ippools API  properties:  apiVersion:  description: 'APIVersion defines the versioned schema of this representation  of an object. Servers should convert recognized schemas to the latest  internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources'  type: string  kind:  description: 'Kind is a string value representing the REST resource this  object represents. Servers may infer this from the endpoint the client  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'  type: string  metadata:  type: object  spec:  description: IPPoolSpec defines the desired state of IPPool  properties:  allocations:  additionalProperties:  description: IPAllocation represents metadata about the pod/container  owner of a specific IP  properties:  id:  type: string  podref:  type: string  required:  - id  type: object  description: Allocations is the set of allocated IPs for the given  range. Its indices are a direct mapping to the IP with the same  index/offset for the pools range.  type: object  range:  description: Range is a RFC 4632/4291-style string that represents  an IP address and prefix length in CIDR notation  type: string  required:  - allocations  - range  type: object  type: object  served: true  storage: true status:  acceptedNames:  kind: ""  plural: ""  conditions: []  storedVersions: [] --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata:  annotations:  controller-gen.kubebuilder.io/version: v0.4.1  name: overlappingrangeipreservations.whereabouts.cni.cncf.io spec:  group: whereabouts.cni.cncf.io  names:  kind: OverlappingRangeIPReservation  listKind: OverlappingRangeIPReservationList  plural: overlappingrangeipreservations  singular: overlappingrangeipreservation  scope: Namespaced  versions:  - name: v1alpha1  schema:  openAPIV3Schema:  description: OverlappingRangeIPReservation is the Schema for the OverlappingRangeIPReservations  API  properties:  apiVersion:  description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'  type: string  kind:  description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'  type: string  metadata:  type: object  spec:  description: OverlappingRangeIPReservationSpec defines the desired state  of OverlappingRangeIPReservation  properties:  containerid:  type: string  podref:  type: string  required:  - containerid  type: object  required:  - spec  type: object  served: true  storage: true status:  acceptedNames:  kind: ""  plural: ""  conditions: []  storedVersions: [] --- kind: ConfigMap apiVersion: v1 metadata:  name: multus-cni-config  namespace: kube-system  labels:  app: gke-multinet data:  cni-conf.json: |  {  "name": "multus-cni-network",  "type": "multus",  "confDir": "/etc/cni/net.d",  "namespaceIsolation": true,  "logLevel": "verbose",  "logFile": "/var/log/multus.log",  "kubeconfig": "/var/lib/kubelet/kubeconfig",  "clusterNetwork": "gke-pod-network"  } --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata:  name: network-attachment-definitions.k8s.cni.cncf.io spec:  group: k8s.cni.cncf.io  scope: Namespaced  names:  plural: network-attachment-definitions  singular: network-attachment-definition  kind: NetworkAttachmentDefinition  shortNames:  - net-attach-def  versions:  - name: v1  served: true  storage: true  schema:  openAPIV3Schema:  description: 'NetworkAttachmentDefinition is a CRD schema specified by the Network Plumbing  Working Group to express the intent for attaching pods to one or more logical or physical  networks. More information available at: https://github.com/k8snetworkplumbingwg/multi-net-spec'  type: object  properties:  apiVersion:  description: 'APIVersion defines the versioned schema of this represen  tation of an object. Servers should convert recognized schemas to the  latest internal value, and may reject unrecognized values. More info:  https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'  type: string  kind:  description: 'Kind is a string value representing the REST resource this  object represents. Servers may infer this from the endpoint the client  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'  type: string  metadata:  type: object  spec:  description: 'NetworkAttachmentDefinition spec defines the desired state of a network attachment'  type: object  properties:  config:  description: 'NetworkAttachmentDefinition config is a JSON-formatted CNI configuration'  type: string --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata:  name: multus-role rules: - apiGroups: ["k8s.cni.cncf.io"]  resources:  - '*'  verbs:  - '*' --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata:  name: whereabouts rules: - apiGroups:  - whereabouts.cni.cncf.io  resources:  - ippools  - overlappingrangeipreservations  verbs:  - get  - list  - watch  - create  - update  - patch  - delete - apiGroups:  - coordination.k8s.io  resources:  - leases  verbs:  - create - apiGroups:  - coordination.k8s.io  resources:  - leases  resourceNames:  - whereabouts  verbs:  - '*' - apiGroups: [""]  resources:  - pods  verbs:  - list  - get --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata:  name: multus-role-binding subjects: - kind: Group  name: system:nodes roleRef:  kind: ClusterRole  name: multus-role  apiGroup: rbac.authorization.k8s.io --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata:  name: whereabouts-role-binding roleRef:  apiGroup: rbac.authorization.k8s.io  kind: ClusterRole  name: whereabouts subjects: - kind: ServiceAccount  name: whereabouts-sa  namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata:  name: whereabouts-sa  namespace: kube-system --- apiVersion: apps/v1 kind: DaemonSet metadata:  name: gke-multinet  namespace: kube-system  labels:  app: gke-multinet spec:  selector:  matchLabels:  app: gke-multinet  template:  metadata:  labels:  app: gke-multinet  spec:  priorityClassName: system-node-critical  hostNetwork: true  tolerations:  - operator: Exists  serviceAccountName: whereabouts-sa  containers:  - name: whereabouts-gc  command: [/ip-control-loop]  args:  - "--log-level=debug"  - "--enable-pod-watch=false"  - "--cron-schedule=* * * * *"  image: gcr.io/gke-release/whereabouts:v0.7.0-gke.3@sha256:2bb8450a99d86c73b262f5ccd8c433d3e3abf17d36ee5c3bf1056a1fe479e8c2  env:  - name: NODENAME  valueFrom:  fieldRef:  apiVersion: v1  fieldPath: spec.nodeName  - name: WHEREABOUTS_NAMESPACE  valueFrom:  fieldRef:  fieldPath: metadata.namespace  resources:  requests:  cpu: "100m"  memory: "50Mi"  limits:  cpu: "100m"  memory: "50Mi"  initContainers:  - name: install-multus-config  image: gcr.io/gke-release/multus-cni:v4.2.1-gke.6@sha256:25b48b8dbbf6c78a10452836f52dee456514783565b70633a168a39e6d322310  args:  - "--cni-conf-dir=/host/etc/cni/net.d"  - "--multus-conf-file=/tmp/multus-conf/00-multus.conf"  - "--multus-log-level=verbose"  - "--multus-kubeconfig-file-host=/var/lib/kubelet/kubeconfig"  - "--skip-multus-binary-copy=true"  - "--skip-config-watch=true"  resources:  requests:  cpu: "100m"  memory: "50Mi"  limits:  cpu: "100m"  memory: "50Mi"  securityContext:  privileged: true  volumeMounts:  - name: cni  mountPath: /host/etc/cni/net.d  - name: multus-cfg  mountPath: /tmp/multus-conf  - name: install-whereabouts  command: ["/bin/sh"]  args:  - -c  - >  SLEEP=false /install-cni.sh  image: gcr.io/gke-release/whereabouts:v0.7.0-gke.3@sha256:2bb8450a99d86c73b262f5ccd8c433d3e3abf17d36ee5c3bf1056a1fe479e8c2  env:  - name: NODENAME  valueFrom:  fieldRef:  apiVersion: v1  fieldPath: spec.nodeName  - name: WHEREABOUTS_NAMESPACE  valueFrom:  fieldRef:  fieldPath: metadata.namespace  resources:  requests:  cpu: "100m"  memory: "50Mi"  limits:  cpu: "100m"  memory: "50Mi"  securityContext:  privileged: true  volumeMounts:  - name: cni  mountPath: /host/etc/cni/net.d  - name: cnibin  mountPath: /host/opt/cni/bin  - name: install-binary  image: gcr.io/gke-release/multus-cni:v4.2.1-gke.6@sha256:25b48b8dbbf6c78a10452836f52dee456514783565b70633a168a39e6d322310  command: ["/gkecmd"]  args:  - "-operation=copy"  - "-cni-bin-dir=/host/opt/cni/bin"  resources:  requests:  cpu: "10m"  memory: "100Mi"  limits:  cpu: "10m"  memory: "100Mi"  securityContext:  privileged: true  volumeMounts:  - name: cnibin  mountPath: /host/opt/cni/bin  volumes:  - hostPath:  path: /var/lib/kubelet/kubeconfig  type: File  name: kubelet-credentials  - name: cni  hostPath:  path: /etc/cni/net.d  type: DirectoryOrCreate  - name: cnibin  hostPath:  path: /home/kubernetes/bin  type: DirectoryOrCreate  - name: multus-cfg  configMap:  name: multus-cni-config  items:  - key: cni-conf.json  path: 00-multus.conf  updateStrategy:  rollingUpdate:  maxUnavailable: 2  type: RollingUpdate

Then, apply the manifest to your cluster:

kubectl apply -f multus-manifest.yaml

Create a `NetworkAttachmentDefinition` manifest

To enable Pods to connect to additional networks, create a NetworkAttachmentDefinition manifest. This manifest defines how Pods connect to a network and specifies the IP address range that an IPAM plugin, such as Whereabouts, assigns. This range must be part of the subnet that connects your nodes' additional network interfaces.

Save this manifest as nad.yaml. This manifest uses IPVLAN and Whereabouts plugins.
```
apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata:  name: NAD_NAME spec:  config: '{  "cniVersion": "0.3.1",  "plugins": [  {  "type": "ipvlan",  "master": "eth1",  "mode": "l2",  "ipam": {  "type": "whereabouts",  "range": SECONDARY_RANGE_NAME  }  }  ]  }' 
```
The manifest includes the following fields:
- NAD_NAME: the name of your NetworkAttachmentDefinition.
- master: the name of the node's secondary network interface, which acts as the master interface for IPVLAN. On GKE, secondary network interfaces typically start with eth1 and are named sequentially. To confirm the interface name, connect to a node using SSH and run the ip addr command.
- range: he IP address range for Pod interfaces, which is the same as the secondary IPv4 range that you created for the Pods (SECONDARY_RANGE_NAME). For example, 172.16.1.0/24.
Apply the manifest to your cluster:
```
kubectl apply -f nad.yaml 
```

Attach Pods to additional networks

To attach a Pod to an additional network, add the k8s.v1.cni.cncf.io/networks annotation to the Pod manifest. For multiple networks, provide a comma-separated list of NetworkAttachmentDefinition names in the following format: <namespace>/<nad-name>.

The following example shows a Pod manifest that attaches to the NetworkAttachmentDefinition named NAD_NAME in the default namespace:

apiVersion: v1 kind: Pod metadata:  name: samplepod  annotations:  k8s.v1.cni.cncf.io/networks: default/NAD_NAME spec:  containers:  - name: sample-container  image: nginx

Replace NAD_NAME with the name of the NetworkAttachmentDefinition you created.

When you apply this manifest, Kubernetes creates the Pod with an additional network interface (net1) connected to the network that the NetworkAttachmentDefinition specifies.

Verify the Pod's additional IP address

To verify that the Pod receives an additional IP address after you attach the Pod to an additional network, inspect the network interfaces within the Pod:

To inspect the samplepod and verify the additional IP address, use the following command:
```
$kubectl describe pod PODNAME 
```
Replace the PODNAME with the name of your Pod, such as samplepod.

Examine the output. The eth0 interface has the Pod's primary IP address. The Whereabouts plugin assigns the additional IP address to another interface, such as net1.

The output is similar to the following:

k8s.v1.cni.cncf.io/network-status: [{ "name": "gke-pod-network", "interface": "eth0", "ips": [ "10.104.3.4" ], "mac": "ea:e2:f6:ce:18:b5", "default": true, "dns": {}, "gateway": [ "\u003cnil\u003e" ] },{ "name": "default/my-nad", "interface": "net1", "ips": [ "10.200.1.1" ], "mac": "42:01:64:c8:c8:07", "dns": {} }] k8s.v1.cni.cncf.io/networks: default/my-nad

In this example, 10.104.5.19 is the primary IP address on eth0, and 10.200.1.1 is the additional IP address on net1.

What's next

Learn more about Multus CNI.
Learn more about IPVLAN CNI.
Learn more about Whereabouts IPAM.