The Container Network Interface (CNI) is one of the great black boxes of Kubernetes. Most people who run clusters every day have never once looked inside it. They know the name of their CNI ("we run Calico," "we're on Cilium") the way you know the brand of the alternator in your car: as a label, not as a thing you actually understand.

It lives at the very bottom of the stack, beneath the kubelet, beneath your pods, quietly moving every single packet. And precisely because it never fails loudly on a good day, almost nobody learns what it does.

We won't run helm install cilium. We won't apply a single manifest. Instead, we'll wire up pod networking from scratch, feel exactly where it breaks the moment traffic tries to leave a physical machine, and fix it manually.

By the end, you'll understand it in your bones, not just in theory, why tools like Cilium exist and what they're really solving under the hood.

Who this is for:

• Developers, platform engineers, and SREs who use Kubernetes every day but quietly treat pod-to-pod networking as magic.

• Anyone who has ever watched a pod flip to Running and assumed the network "just works" and wants to know what's actually happening.

What you'll build with your own hands:

• Two isolated network namespaces wired together with a virtual cable (veth pair).

• A three-namespace virtual switch using a Linux bridge, the same trick legacy CNIs use on a single node.

• A deliberately broken two-node setup where a packet gets dropped on the floor, plus the manual fix that makes it work.

• A clear picture of the three jobs every CNI does, and why cloud providers force advanced CNIs like Cilium to use overlays and eBPF.

Note: Every command here needs a Linux host and root. Run this in a throwaway VM or lab environment, not on anything you care about. The whole point is to make a mess and learn from it.

Table of Contents

• Prerequisites

• The Illusion: Kubernetes Routes Zero Packets

• The Foundation: Virtual Ethernet (veth) Pairs

• How to Scale Locally with a Linux Bridge

• The Multi-Node Boundary Problem

• How to Fix It Manually with Direct Routing

• So What Is a CNI, Really?

• The Cloud Catch and Why Cilium Changes the Game

• Conclusion

Prerequisites

Before following along, you'll need:

• Two Linux VMs on the same network for the multi-node section so you can watch traffic cross a real machine boundary.

• The ip command from the iproute2 package (already installed on virtually every modern distro).

• A basic comfort with IP addresses, subnets, and the word "gateway." You don't need to be a network engineer.

• No Kubernetes. That's not a typo. We're going underneath Kubernetes on purpose.

The Illusion: Kubernetes Routes Zero Packets

Here's the uncomfortable truth most people never confront: Kubernetes can't route a single network packet.

Not one. Kubernetes is a orchestrator. It schedules pods, watches their health, and updates state in etcd. But when it comes to actually moving a packet from one container to another, it has zero built-in capability. None.

So how do your pods talk to each other? They rely completely on an external agent to wire up the virtual network plumbing on every node. That agent is the Container Network Interface (CNI). What the CNI does under the hood quietly, is what we would do ourselves to feel the pain and then the solution a CNI provides.

Here's the proof that it's load-bearing. Spin up a brand-new cluster with kubeadm and look at your nodes:

$ kubectl get nodes
NAME       STATUS     ROLES                  AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION                        CONTAINER-RUNTIME
no-cni     NotReady   control-plane,master   52s   v1.35.0   192.168.117.2   <none>        Ubuntu 24.04.3 LTS   7.0.11-orbstack-00360-gc9bc4d96ac70   containerd://2.1.6
worker-1   NotReady   <none>                 46s   v1.35.0   192.168.117.3   <none>        Ubuntu 24.04.3 LTS   7.0.11-orbstack-00360-gc9bc4d96ac70   containerd://2.1.6
worker-2   NotReady   <none>                 40s   v1.35.0   192.168.117.4   <none>        Ubuntu 24.04.3 LTS   7.0.11-orbstack-00360-gc9bc4d96ac70   containerd://2.1.6

NotReady. Every node. The control plane is healthy, etcd is up, the scheduler is alive, and the cluster still flatly refuses to be Ready. If you describe the node, it tells you precisely what's missing:

Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  Ready            False   Sat, 18 Jul 2026 10:25:51 +0200   Sat, 18 Jul 2026 10:25:20 +0200   KubeletNotReady              container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

Read that again: your cluster is not Ready until you install a CNI. Not "mostly ready." Not "ready except for networking." NotReady, full stop. Until an external plugin shows up and takes responsibility for the packets Kubernetes itself refuses to touch.

A cluster without a CNI is a telephone exchange with no lines plugged in: every operator is present and ready, but not a single call is able to connect.

So what do most people do at this exact moment? They copy one line from a getting-started page:

kubectl apply -f https://.../calico.yaml

They watch the nodes flip to Ready, and they move on. That's the entire relationship most engineers have with the single component that makes their cluster work. They wing it. It works, so they never ask what "it" is.

This matters because "the network just works" is a dangerous story to tell yourself. The moment something breaks (a pod can't reach a service, cross-node traffic vanishes, a cloud migration mysteriously blackholes packets), you're standing in front of a system you never actually understood. So let's understand it. From the bottom up.

The Foundation: Virtual Ethernet (veth) Pairs

To understand container networking, you first have to understand how Linux isolates it.

When a container (or a Kubernetes pod) is created, the kernel wraps it in an isolated Network Namespace (netns). Think of a fresh network namespace as an island with no bridges to the mainland. By default it's completely blind to the outside world: no interfaces, no IP addresses, and no routing tables. It can't talk to anything, and nothing can talk to it.

So how do we get off the island? With a kernel primitive called a Virtual Ethernet (veth) pair.

A veth pair is a virtual network cable. Whatever packet enters one end immediately pops out the other end, even if the two ends live in different namespaces. Plug one end into the island and the other end into the mainland, and suddenly you have a connection.

Let's wire two isolated namespaces, red and blue, directly together.

# Step 1: Create the isolated network namespaces
sudo ip netns add red
sudo ip netns add blue

# Step 2: Create the virtual ethernet cable (veth pair)
sudo ip link add veth-red type veth peer name veth-blue

# Step 3: Move each end of the cable into its namespace
sudo ip link set veth-red netns red
sudo ip link set veth-blue netns blue

# Step 4: Assign IP addresses and bring the interfaces UP
sudo ip netns exec red ip addr add 10.0.0.1/24 dev veth-red
sudo ip netns exec red ip link set veth-red up

sudo ip netns exec blue ip addr add 10.0.0.2/24 dev veth-blue
sudo ip netns exec blue ip link set veth-blue up

Now test the connection by pinging blue from inside red:

sudo ip netns exec red ping -c 2 10.0.0.2

In the end, it would look something like this:

• Isolation broken safely: The container transitions from an unreachable, isolated namespace (no IP, no routing) to an addressable endpoint (10.1.1.2) linked directly to the host network.

• Bi-directional traffic flow: Packets originating inside the container can reach external public IP networks, and incoming response packets from the internet can traverse back through the host's eth0 interface (192.168.1.10) directly into the container's veth pairs.

• Zero-latency in-kernel bridging: The veth pair (veth-island <--> veth-mainland) acts as a direct virtual pipe, allowing instant packet transit between distinct Linux network namespaces (netns) without requiring external physical hardware .

The verdict: the ping succeeds. You just manually wired two isolated environments together with nothing but a virtual cable.

But here's the problem with this approach: it scales horribly. It does not scale well because a veth pair is strictly point to point.

Following is the number of pairs need to be configured for the number of containers:

• 2 containers: 1 pair

• 3 containers: 3 pairs

• 4 containers: 6 cables

For ten, you'd need 45 cables to connect every pair.

• Explodes in cable count: The number of veth pairs grow quadratically as containers increase

• Complex to manage: Too many interfaces, routes and rules to configure and maintain.

This is the same reason data centers don't run a physical cable between every pair of servers. You need a switch.

How to Scale Locally with a Linux Bridge

When you need to connect more than two interfaces on a single host, you stop running cables between everything and plug everything into a central hub instead. In the Linux kernel, that hub is a Linux Bridge. It's a software Layer 2 virtual switch (you'll often see it named br0 or cni0).

A bridge does exactly what a physical switch does: it learns MAC addresses and forwards frames across every connected interface in the same broadcast domain.

The pattern changes slightly. Instead of connecting namespaces directly to each other, you attach one end of a veth pair to the namespace, and plug the other end into the host's bridge.

Let's tear down the old setup and build a three-namespace switch: red, blue, and green. They'll all share one broadcast domain.

# Clean up any previous configuration
sudo ip netns del red 2>/dev/null || true
sudo ip netns del blue 2>/dev/null || true
sudo ip netns del green 2>/dev/null || true
sudo ip link del br0 2>/dev/null || true

# Step 1: Create the host switch (bridge) and bring it up
sudo ip link add br0 type bridge
sudo ip link set br0 up

# Step 2: Wire namespace 1 (red) into the bridge
sudo ip netns add red
sudo ip link add veth-red type veth peer name veth-red-host
sudo ip link set veth-red netns red
sudo ip link set veth-red-host master br0
sudo ip link set veth-red-host up
sudo ip netns exec red ip addr add 10.0.0.1/24 dev veth-red
sudo ip netns exec red ip link set veth-red up

# Step 3: Wire namespace 2 (blue) into the bridge
sudo ip netns add blue
sudo ip link add veth-blue type veth peer name veth-blue-host
sudo ip link set veth-blue netns blue
sudo ip link set veth-blue-host master br0
sudo ip link set veth-blue-host up
sudo ip netns exec blue ip addr add 10.0.0.2/24 dev veth-blue
sudo ip netns exec blue ip link set veth-blue up

# Step 4: Wire namespace 3 (green) into the bridge
sudo ip netns add green
sudo ip link add veth-green type veth peer name veth-green-host
sudo ip link set veth-green netns green
sudo ip link set veth-green-host master br0
sudo ip link set veth-green-host up
sudo ip netns exec green ip addr add 10.0.0.3/24 dev veth-green
sudo ip netns exec green ip link set veth-green up

Because all three namespaces are connected to the shared br0 device, they can communicate freely with each other across the virtual network switch. But how do they actually "find" each other on the network? This is where ARP comes in.

ARP stands for Address Resolution Protocol. It's a fundamental part of local networking. When one computer (or namespace, in our case) wants to talk to another using an IP address, it needs to discover the other computer's hardware address (called a MAC address) to actually send packets on the network.

ARP is the system that allows this to happen — it sends out a broadcast asking "Who has IP address X? Please tell me your MAC address," and the right system answers back.

Thanks to ARP, all the namespaces plugged into br0 can learn each other's MAC addresses automatically and send packets directly within their shared network segment. Let's prove it by pinging every pair, both ways:

# red reaches blue and green
sudo ip netns exec red ping -c 1 10.0.0.2
sudo ip netns exec red ping -c 1 10.0.0.3

# blue reaches red and green
sudo ip netns exec blue ping -c 1 10.0.0.1
sudo ip netns exec blue ping -c 1 10.0.0.3

# green reaches red and blue
sudo ip netns exec green ping -c 1 10.0.0.1
sudo ip netns exec green ping -c 1 10.0.0.2

All six pings succeed. And notice there isn't a single routing rule involved anywhere. Every namespace lives in the same 10.0.0.0/24 subnet on the same Layer 2 switch, so the kernel resolves the whole mesh with plain ARP.

This is exactly how legacy single-node CNIs (the old kubenet) operate. On one machine, it's clean and simple.

But Kubernetes is a distributed system designed to scale across thousands of physical machines. So here's the question that breaks everything: what happens when our namespaces need to leave the host?

The Multi-Node Boundary Problem

Everything so far has lived on one machine. Kubernetes doesn't. So let's do the honest thing: stand up two real VMs and watch the single-node trick fall apart. Don't take my word for it: build this and watch the packet die.

Here's the setup:

• VM 1 (host IP 10.1.44.216): home to the red namespace, pod subnet 10.0.1.0/24.

• VM 2 (host IP 10.1.44.178): home to the blue namespace, pod subnet 10.0.2.0/24.

Two things to notice before we start. First, each node gets its own pod subnet (10.0.1.0/24 on VM 1, 10.0.2.0/24 on VM 2) because if both nodes handed out 10.0.0.x addresses, you'd get IP collisions the instant two pods landed on the same number.

Second, because the subnets now differ, each namespace needs a gateway to route through, and that gateway is its own host's bridge.

Note: this is the cleanup-multinode.sh script that should only be used in case you make any errors while setting up the cross node routes and veth pairs.

#!/usr/bin/env bash
#
# cleanup-multinode.sh
# Tears down the manual multi-node CNI lab (bridge + namespaces + veth
# pairs + cross-node static routes) from "Build a Mental Model for
# Kubernetes CNI by Doing It Manually."
#
# Safe to run on BOTH VMs. Every step is idempotent: anything that was
# never created on this host is skipped instead of erroring out, so
# re-running it is harmless.
#
# Usage:  sudo ./cleanup-multinode.sh
#
set -u

if [[ $EUID -ne 0 ]]; then
  echo "This script needs root. Run:  sudo $0" >&2
  exit 1
fi

echo "==> Deleting network namespaces (this also destroys their veth pairs)..."
ip netns del red  2>/dev/null && echo "    - removed netns 'red'"  || true
ip netns del blue 2>/dev/null && echo "    - removed netns 'blue'" || true

echo "==> Removing any orphaned host-side veth interfaces..."
ip link del veth-red-host  2>/dev/null && echo "    - removed veth-red-host"  || true
ip link del veth-blue-host 2>/dev/null && echo "    - removed veth-blue-host" || true

echo "==> Deleting the bridge..."
ip link del br0 2>/dev/null && echo "    - removed bridge 'br0'" || true

echo "==> Removing cross-node static routes..."
ip route del 10.0.1.0/24 2>/dev/null && echo "    - removed route to 10.0.1.0/24" || true
ip route del 10.0.2.0/24 2>/dev/null && echo "    - removed route to 10.0.2.0/24" || true

echo "==> Disabling IP forwarding (non-persistent; resets on reboot anyway)..."
sysctl -w net.ipv4.ip_forward=0 >/dev/null

# --- Optional: undo the 'Common Gotchas' tweaks, ONLY if you applied them ---
# On a throwaway lab VM, leaving FORWARD at ACCEPT or rp_filter at 0 is
# usually harmless, so these are opt-in. Uncomment whatever you changed.
# sysctl -w net.ipv4.conf.all.rp_filter=1 >/dev/null
# iptables -P FORWARD DROP

echo
echo "==> Teardown complete. Verifying nothing is left behind:"
echo "--- namespaces (expect: no red/blue) ---"
out=$(ip netns list);                          echo "${out:-  (none)}"
echo "--- bridges (expect: no br0) ---"
out=$(ip -br link show type bridge 2>/dev/null); echo "${out:-  (none)}"
echo "--- lab routes (expect: none) ---"
out=$(ip route | grep -E '10\.0\.[12]\.0/24'); echo "${out:-  (none)}"

On VM 1 (10.1.44.216), build the bridge, wire up red, and turn the host into a router:

# Make the host a router so it can transit packets that aren't its own
sudo sysctl -w net.ipv4.ip_forward=1

# Build the bridge and give it a gateway IP for VM 1's pod subnet
sudo ip link add br0 type bridge
sudo ip addr add 10.0.1.254/24 dev br0
sudo ip link set br0 up

# Wire the red namespace into the bridge
sudo ip netns add red
sudo ip link add veth-red type veth peer name veth-red-host
sudo ip link set veth-red netns red
sudo ip link set veth-red-host master br0
sudo ip link set veth-red-host up
sudo ip netns exec red ip addr add 10.0.1.1/24 dev veth-red
sudo ip netns exec red ip link set veth-red up

# Point the namespace's default route at its bridge gateway
sudo ip netns exec red ip route add default via 10.0.1.254

On VM 2 (10.1.44.178), do the mirror image for blue:

sudo sysctl -w net.ipv4.ip_forward=1

sudo ip link add br0 type bridge
sudo ip addr add 10.0.2.254/24 dev br0
sudo ip link set br0 up

sudo ip netns add blue
sudo ip link add veth-blue type veth peer name veth-blue-host
sudo ip link set veth-blue netns blue
sudo ip link set veth-blue-host master br0
sudo ip link set veth-blue-host up
sudo ip netns exec blue ip addr add 10.0.2.1/24 dev veth-blue
sudo ip netns exec blue ip link set veth-blue up

sudo ip netns exec blue ip route add default via 10.0.2.254

Both hosts are routers now. Both namespaces are wired up. Ping blue on VM 2 from red on VM 1:

# On VM 1
sudo ip netns exec red ping -c 3 10.0.2.1

PING 10.0.2.1 (10.0.2.1) 56(84) bytes of data.

--- 10.0.2.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2043ms

100% packet loss. The packet is dropped on the floor, exactly as promised, but now you've seen it with your own eyes.

Here's the part worth proving to yourself: the packet really does leave VM 1. It just never arrives at VM 2. Run tcpdump on both boxes and ping again:

# Detect your physical NIC once (enp1s0, ens3, eth0, ...)
NIC=$(ip route get 1.1.1.1 | grep -oP 'dev \K\S+')

# On VM 1: the echo requests march out the door
sudo tcpdump -ni "$NIC" icmp
IP 10.0.1.1 > 10.0.2.1: ICMP echo request, id 5, seq 1, length 64
IP 10.0.1.1 > 10.0.2.1: ICMP echo request, id 5, seq 2, length 64

# On VM 2: dead silence. Nothing ever shows up.
sudo tcpdump -ni "$NIC" icmp
(no output)

So where does it die? Follow the life and death of that packet:

To keep it even simpler, the flow is as follows:

Red Pod (10.0.1.1)
      │
      ▼
eth0
      │
      ▼
veth-red
      │
      ▼
br0 (10.0.1.254)
      │
      ▼
VM 1 Routing Table
(No route for 10.0.2.0/24)
      │
      ▼
Default Route (0.0.0.0/0)
      │
      ▼
Physical NIC (eth0)
      │
      ▼
LAN Gateway (192.168.1.1)
      │
      ▼
❌ No route for 10.0.2.0/24
(Packet Dropped)
      │
      ▼
VM 2 Never Receives the Packet

That's the multi-node boundary problem in one sentence: your per-node scripts are completely blind to the rest of the cluster's topology. VM 1 built its island, VM 2 built its island, and neither has any idea the other exists.

How to Fix It Manually with Direct Routing

The fix is almost insultingly small. VM 1 doesn't need a smarter network. It needs a map. We just have to tell each host one fact it's missing: "the other node's pod subnet lives behind the other node's physical IP." That's a single static route per side. Nothing gets rebuilt: the bridges, namespaces, and forwarding you set up a moment ago all stay exactly as they are.

On VM 1 (10.1.44.216), teach it where VM 2's pods live:

# VM 2's pods (10.0.2.0/24) are reachable via VM 2's physical IP
sudo ip route add 10.0.2.0/24 via 10.1.44.178

On VM 2 (10.1.44.178), teach it the way back:

# VM 1's pods (10.0.1.0/24) are reachable via VM 1's physical IP
sudo ip route add 10.0.1.0/24 via 10.1.44.216

That's it. Two lines. Re-run the exact same ping from red on VM 1:

sudo ip netns exec red ping -c 3 10.0.2.1

PING 10.0.2.1 (10.0.2.1) 56(84) bytes of data.
64 bytes from 10.0.2.1: icmp_seq=1 ttl=62 time=0.412 ms
64 bytes from 10.0.2.1: icmp_seq=2 ttl=62 time=0.388 ms
64 bytes from 10.0.2.1: icmp_seq=3 ttl=62 time=0.401 ms

--- 10.0.2.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss

It works. (See ttl=62? Your packet started at 64 and lost one hop on each host it was forwarded through, proof it crossed two routers to get there.) The packet now completes the full journey:

The simpler flow looks like:

1. Pod (10.0.1.1)
        │
        ▼
2. veth-red
        │
        ▼
3. VM 1 br0 (10.0.1.254)
        │
        ▼
4. VM 1 Routing Table
   ✅ Static route:
   10.0.2.0/24 → 10.1.44.178
        │
        ▼
5. VM 1 Physical NIC
        │
        ▼
6. Direct Link
   VM 1 → VM 2
        │
        ▼
7. VM 2 Physical NIC
        │
        ▼
8. VM 2 Routing Table
   ✅ 10.0.2.0/24 is directly connected
        │
        ▼
9. VM 2 br0
        │
        ▼
10. Blue Pod (10.0.2.1)
        │
        ▼
✅ Reply follows the same path back

That one line changed everything. Instead of dumping the packet at your LAN's clueless gateway, VM 1 now hands it directly to VM 2, which knows exactly which local namespace owns 10.0.2.1. The reply follows the mirror route home. You just hand-built cross-node pod networking.

Now sit with how painful that was. Two nodes took a stack of careful commands and a hand-written route on each side.

Imagine a thousand nodes, pods being created and destroyed every second, each one needing a fresh IP and a route on every other node in the cluster. Doing that by hand isn't just tedious. It's impossible.

So What Is a CNI, Really?

Everything you just did by hand (the namespaces, the veth pairs, the bridges, the IP assignment, the routes) is exactly what a Container Network Interface automates dynamically, at scale, the instant a pod is scheduled.

When you apply a pod manifest, the CNI plugin intercepts the lifecycle event and performs three core jobs:

1. Namespace and interface provisioning: It creates the network namespace, generates the veth pair, and attaches it to the bridge (or its own datapath), cleanly, every time, with no fat-fingered typos.

2. IP Address Management (IPAM): It hands out unique, non-colliding subnets per node and leases an individual IP to every single container in the cluster. That "unique Pod CIDR per node" rule you set up manually? IPAM enforces it automatically.

3. Cluster-wide route distribution: It programs the routing so every node knows how to reach pods on every other node: the static routes you wrote by hand, generated and pushed everywhere, kept in sync as nodes and pods come and go.

That's the mental model. A CNI is the thing that does your dozen-command lab a thousand times a second and never makes a mistake.

The Cloud Catch and Why Cilium Changes the Game

Here's the part that surprises people. The manual direct-routing approach we just built works flawlessly in a bare-metal lab. In a modern public cloud (AWS, GCP, Azure), it breaks completely.

Why? Cloud providers don't let arbitrary IP addresses roam across their network fabric. Your 10.0.1.0/24 pod subnet means nothing to the VPC. Unless those IPs are explicitly registered through heavyweight cloud-controller API calls, the underlying network sees pod traffic as illegitimate and drops it: the exact failure from the boundary-problem section, except now the cloud itself is the thing saying "no."

This is where advanced CNIs like Cilium stop playing by the old rules. Instead of leaning on fragile Linux bridges and hand-written host routes, Cilium reaches for two much stronger mechanisms.

• Overlay networks (VXLAN / Geneve). Cilium takes your raw pod packet and encapsulates it, wrapping it inside an ordinary UDP packet addressed from Node 1's physical IP to Node 2's physical IP. To the cloud provider, it looks like completely normal node-to-node host traffic, so it sails straight through every VPC restriction. Your pod's real addresses are hidden inside the envelope.

• eBPF kernel programmability. Traditional CNIs push every packet through the full Linux bridge path and hundreds of sequential iptables rules: slow, and slower as your cluster grows. Cilium replaces that entire pipeline by loading compiled eBPF programs directly into the kernel at the network-interface level. Packets get short-circuited from the pod's veth straight toward the physical NIC, giving you near line-rate performance and deep security visibility for free.

Here's the whole progression in one table:

Look at that last column, then look at the last row. That's the entire value proposition of a CNI in two cells.

Conclusion

You didn't read about Kubernetes networking. You built it, broke it, and fixed it. Here's the mental model you now carry:

1. Kubernetes routes zero packets. It fully delegates the network to a CNI, and that CNI is doing real, physical plumbing on every node.

2. A veth pair is a virtual cable, and it's the atom of container networking: great for two endpoints, useless at scale.

3. A Linux bridge is a virtual switch that connects many namespaces on one host with nothing but Layer 2 and ARP. That's a single-node CNI in a nutshell.

4. The node boundary is where naïve networking dies. Different subnets and an unaware physical network mean cross-node packets get dropped until you teach every host how to route.

5. Static routes plus IP forwarding fix it manually, and doing that by hand for two nodes shows you instantly why nobody does it for a thousand.

6. A CNI automates three jobs: interface provisioning, IPAM, and cluster-wide route distribution.

7. The cloud breaks direct routing, which is precisely why Cilium leans on VXLAN/Geneve overlays and eBPF instead of bridges and iptables.

The next time a pod flips to Running and the network "just works," you'll know the truth: nothing just works. A CNI just did (silently, and at a scale you now truly respect) everything you just did by hand.

From here, the natural next step is to tear down these scripts, deploy Cilium into a real cluster, and watch eBPF orchestrate this entire topology automatically. Having felt the manual pain first, you'll actually appreciate the elegance.

If this helped you build a clearer picture of Kubernetes networking, come say hi:

• LinkedIn: linkedin.com/in/shubhamkatara

• YouTube: @kubesimplify

Who this is for:

• Platform engineers, SREs, and security-focused cloud architects who operate mixed on‑prem and cloud Kubernetes estates.

• Teams that need scalable, auditable access from on‑prem workloads to GCP resources (especially GPU instances) while minimizing operational overhead and blast radius.

What you’ll get from this guide:

• The motivation and economics behind a hybrid approach (why GPUs often push workloads to the cloud).

• Common pitfalls with service account keys and how “accidental air gaps” occur in real environments.

• A practical, end‑to‑end pattern that uses Workload Identity Federation to give on‑prem pods short‑lived, auditable access to GCP without embedding keys.

What’s included:

• Conceptual explanations, security tradeoffs, and operational best practices.

• Concrete examples and Kubernetes/Terraform artifacts (linked in the GitHub repo at the end of this article) so you can reproduce the setup in your environment.

Read on for the theory, then follow the hands‑on sections to provision GCP resources, configure federation, enforce policies with CEL and Kyverno, and validate secure, scalable GPU access from your on‑prem Kubernetes clusters.

Note: Kubernetes and Terraform artifacts are linked in the GitHub repo at the end of this article.

Table of Contents

• Prerequisites

• Why Hybrid Cloud Matters

• The Economics of Hybrid: GPUs Changed Everything

• Why Service Account Keys Fail at Scale

• How the Accidental Air Gap Happens

• How Workload Identity Federation Bridges the Gap

• How Kubernetes Identity Works

• How to prepare Google Cloud Platform resources

• How to Use CEL for Fine-Grained Access Control

• How to Inject Credentials Automatically with Kyverno

• How to Grant IAM Permissions to Federated Identities

• How to Verify the Setup

• How to Connect On-Prem Apps to Cloud GPUs

• How to Scale GPU Access with CEL Conditions

• The Security Properties Compared

• The Complete Infrastructure as Code Layout

• How to Run a Proof of Concept with vCluster

• Common Issues and How to Solve Them

• Conclusion

Prerequisites

Before following along, you'll need:

• A Kubernetes cluster that is not GKE (on-premises, bare-metal, or a virtual cluster)

• A Google Cloud project with the following APIs enabled: IAM, Security Token Service (STS), and Workload Identity

• Terraform installed and configured

• Kyverno installed in your cluster

• Python 3 with google-cloud-secret-manager and google-cloud-aiplatform libraries (for the verification steps. Code available in the github repository.)

• kubectl access to your cluster

Why Hybrid Cloud Matters

If everything goes right, a hybrid cloud platform lets your on-premises and cloud workloads talk to each other as if they were part of the same network.

There are many practical reasons to run a hybrid cloud setup:

• Offloading analytics to BigQuery: You keep your analytics apps on-prem for data sovereignty, but pipe large datasets into BigQuery for world-class processing power — without buying extra servers.

• Creating a unified network with Cloud Interconnect: Using Cloud Interconnect or Cloud VPN, your on-premises datacenter becomes an extension of the Google Cloud Platform (GCP) Virtual Private Cloud (VPC). Your on-prem invoice apps can talk to cloud-based user services with low latency and no public internet exposure.

• Cost-effective scalability via Cloud Storage: You can use cloud storage as a backend for local apps, storing logs, backups, and historical data while paying only for what you use.

• Event-driven syncing with Pub/Sub: When something happens on-prem, a message through Cloud Pub/Sub lets cloud services react instantly — no manual polling required.

The Economics of Hybrid: GPUs Changed Everything

Before diving into the technical problem, it's worth understanding why hybrid clouds matter more than ever.

Your organization, like most enterprises, has made significant investments in on-premises datacenters. Servers are bought. Racks are filled. Network infrastructure is paid for. The marginal cost of running one more workload is essentially zero.

Then came the AI wave.

Suddenly every team needs Graphics Processing Units (GPUs). Not one or two — dozens of A100s for training, fleets of inference endpoints, vector databases that need to sit close to the models. GPUs are scarce. Lead times for on-prem GPU hardware stretch into months. Cloud providers have them available in minutes.

The architecture that actually makes economic sense looks like this:

• The on-prem datacenter handles the bulk of compute — web servers, business logic, databases, batch processing. This is commodity compute you've already paid for.

• The cloud handles what's scarce — GPU-accelerated inference, model training, AI/ML endpoints. You pay per request, scale on demand, and don't wait six months for hardware.

The cloud isn't a full migration destination — it's an extension for capabilities you can't easily build on-prem.

But those on-prem workloads need to authenticate to cloud services. Every API call from the datacenter to a Vertex AI endpoint, every request to a GPU-powered inference service, every write to Cloud Storage for model artifacts — all of it needs credentials. That's the problem this article solves.

Why Service Account Keys Fail at Scale

Here's a scenario that plays out in thousands of enterprises daily.

A development team needs their on-prem application to write to Google Cloud Storage. The "obvious" solution? Generate a GCP service account key, base64 encode it, store it in a Kubernetes Secret, and mount it in the pod:

apiVersion: v1
kind: Secret
metadata:
  name: gcp-credentials
type: Opaque
data:
  key.json: eyJ0eXBlIjoic2VydmljZV9hY2NvdW50IiwicHJvamVjdF9pZCI6…

This works. It also introduces serious problems:

• Never expires. That key is valid until someone remembers to rotate it (they won't) or it gets compromised (it will).

• Can be exfiltrated trivially. Anyone with read access to that namespace can run kubectl get secret -o yaml and walk away with permanent GCP access.

• Has no audit trail for the actual workload. GCP sees "service-account-xyz accessed this bucket" — not "pod frontend-abc-123 in namespace production."

• Scales terribly. 50 teams × 3 environments × 4 GCP projects = 600 keys to track, rotate, and hope haven't been committed to git.

Security teams know this. That's why many organizations have done the only sensible thing: they have disabled service account key generation entirely.

How the Accidental Air Gap Happens

When you disable key generation, you haven't solved the hybrid cloud platform problem — you've just made it someone else's problem. That someone is usually a platform team staring at a Jira ticket that says "cannot access GCP from on-prem, P1, blocking release."

The result? Your "hybrid cloud platform" isn't hybrid at all. It's two disconnected systems.

Teams resort to building intermediary services, API gateways that proxy requests, or finding creative ways to get keys anyway. None of this is a platform. It's duct tape.

How Workload Identity Federation Bridges the Gap

Every Kubernetes cluster already issues cryptographically signed identity tokens to every pod. And Google Cloud has a service specifically designed to trust those tokens.

This is Workload Identity Federation — and combined with OpenID Connect (OIDC), it's the missing piece that makes hybrid platforms actually work.

The service is quite well named because of the word Federation. it means GCP doesn't store your identity — it agrees to trust identities issued by another system, as long as they can be cryptographically verified. This all works with a very well orchestrated set of steps in the following order:

1. Pod presents its Kubernetes-issued JWT to GCP's STS endpoint.

2. STS verifies the signature against your cluster's public JWKS.

3. STS checks the JWT's claims against the Workload Identity Pool's rules (audience, issuer, CEL conditions).

4. STS returns a short-lived Google access token (typically 1 hour) that the pod uses for API calls.

It is also worth mentioning that Workload Identity Federation is not Kubernetes specific. It works with AWS IAM, Azure AD, GitHub Actions OIDC, and any OIDC-compliant identity provider.

How Kubernetes Identity Works

Every pod with a ServiceAccount gets a JSON Web Token (JWT) automatically mounted at /run/secrets/kubernetes.io/serviceaccount/token. This isn't just an opaque blob — it's a signed assertion of identity:

{
  "iss": "https://kubernetes.default.svc.cluster.local",
  "sub": "system:serviceaccount:production:backend-api",
  "aud": ["https://iam.googleapis.com/..."],
  "kubernetes.io": {
    "namespace": "production",
    "serviceaccount": {
      "name": "backend-api"
    }
  },
  "exp": 1735689600
}

In a JWT, claims are just the key-value pairs inside the token's payload — each one is a claim the issuer is making about the subject. Think of them as facts the token is asserting, signed cryptographically so the verifier can trust them.

The critical insight: this token is created by a set of JSON Web Key Set (JWKS) and is verifiable by anyone who has your cluster's public keys, exposed via the JSON Web Key Set (JWKS) endpoint:

kubectl get --raw /openid/v1/jwks

Google Cloud's Security Token Service (STS) can validate these tokens. No keys are exchanged. No secrets are stored. Just cryptographic proof of identity.

How to Prepare Google Cloud Platform resources

The Workload Identity Pool is a trust boundary — a declaration that says "I accept identities from external sources." The OIDC Provider configures how to validate those identities.

resource "google_iam_workload_identity_pool" "pool" {
  workload_identity_pool_id = "hybrid-platform-pool"
  project                   = "my-project"
}

resource "google_iam_workload_identity_pool_provider" "k8s_provider" {
  project                            = "my-project"
  workload_identity_pool_id          = google_iam_workload_identity_pool.pool.workload_identity_pool_id
  workload_identity_pool_provider_id = "on-prem-cluster"

  attribute_mapping = {
    "google.subject"      = "assertion.sub"
    "attribute.namespace" = "assertion['kubernetes.io']['namespace']"
  }

  attribute_condition = "attribute.namespace in [\"production\", \"staging\"]"

  oidc {
    issuer_uri = "https://kubernetes.default.svc.cluster.local"
    jwks_json  = file("jwks.json")  # Your cluster's public keys
  }
}

Two things to note here:

1. attribute_mapping extracts claims from the Kubernetes JWT and makes them available as GCP attributes. By using `assertion['kubernetes.io']['namespace']`, the namespace is pulled out so you can use it for access control.

2. attribute_condition is where security policy lives. More on this in the next section.

How to Use CEL for Fine-Grained Access Control

The attribute_condition field uses Common Expression Language (CEL). This single line of policy can replace dozens of Identity and Access Management (IAM) bindings:

attribute.namespace in ["production", "staging"]

With this condition, a pod in the kube-system namespace cannot authenticate to GCP at all — the token exchange is rejected before IAM is even consulted.

You can get more sophisticated:

// Only production namespace, and only specific service accounts
attribute.namespace == "production" &&
  attribute.service_account in ["payment-processor", "order-service"]

// Allow staging, but only during business hours
attribute.namespace == "staging" &&
  request.time.getHours("America/New_York") >= 9 &&
  request.time.getHours("America/New_York") < 17

This is defense in depth. Even if someone creates a rogue ServiceAccount or has kubectl access, they cannot authenticate to GCP unless the CEL condition passes. The security boundary is enforced by Google's infrastructure, not by hoping developers follow policy.

How to Inject Credentials Automatically with Kyverno

Having a working identity federation is only half the battle. Your customers and developers shouldn't need to understand OIDC, STS, or credential configuration files. They should deploy their app and have it work.

Before we get to the automation, it's worth pausing on what a credential configuration file actually is — because the name is a little misleading.

A credential configuration file (sometimes called an "external account config" or "ADC config") is a small JSON document that tells Google's client libraries how to obtain a credential at runtime. It is not itself a credential. You'll see the actual file later in this article — it contains no secrets. Just metadata: the Workload Identity Pool audience, the STS token-exchange endpoint, the source token type, and the path on the pod's filesystem where the real (short-lived) Kubernetes ServiceAccount token lives.

Compare that to a traditional service account key:

Both files end up referenced by GOOGLE_APPLICATION_CREDENTIALS and look interchangeable from the application's point of view — but only one of them is dangerous to lose. The credential config file is safe to ship in a ConfigMap precisely because there's nothing to steal.

Having this file in the ConfigMap is half the solution. It actually needs to end up in the workload pods that need access to GCP services. This is where Kyverno comes in. A single ClusterPolicy automatically injects everything a pod needs:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: workload-identity-federation
spec:
  rules:
    - name: inject-gcp-credentials
      match:
        any:
          - resources:
              kinds:
                - Deployment
              selector:
                matchLabels:
                  workload-identity-federation: "enabled"
      mutate:
        patchStrategicMerge:
          spec:
            template:
              spec:
                volumes:
                  - name: workload-identity-credential-configuration
                    configMap:
                      name: workload-identity-federation-config
                containers:
                  - (name): "*"
                    volumeMounts:
                      - name: workload-identity-credential-configuration
                        mountPath: /etc/workload-identity
                        readOnly: true
                    env:
                      - name: GOOGLE_APPLICATION_CREDENTIALS
                        value: "/etc/workload-identity/credential-configuration.json"

The above cluster policy does three things:

1. Mounts the configmap inside the containers in the deployment at /etc/workload-identity.

2. Injects an environment variable called GOOGLE_APPLICATION_CREDENTIALS that points to the absolute path of the credential config file.

From a developer's perspective, this is their entire integration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    workload-identity-federation: "enabled" # That's it.
spec:
  # ... normal deployment spec

The credential configuration file (created by Terraform as a ConfigMap) tells Google's client libraries how to exchange tokens:

{
  "type": "external_account",
  "audience": "//iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_ID/providers/PROVIDER_ID",
  "subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
  "token_url": "https://sts.googleapis.com/v1/token",
  "credential_source": {
    "file": "/run/secrets/kubernetes.io/serviceaccount/token"
  }
}

This JSON file is a credential configuration for Google's Workload Identity Federation. It instructs Google Cloud client libraries to obtain cloud access tokens by exchanging a Kubernetes ServiceAccount token (located at /run/secrets/kubernetes.io/serviceaccount/token) for a Google Cloud access token, using an external identity provider configured via a Workload Identity Pool. This allows workloads running outside of GCP, such as on-premises Kubernetes clusters, to authenticate to Google Cloud services without needing to manage long-lived service account keys.

Every Google Cloud SDK and client library understands this format. Python, Go, Java, and Node.js all just work.

How to Grant IAM Permissions to Federated Identities

The service account token that has been trusted by the STS service, also known as a federated identity, need permissions to access resources. You bind IAM roles to the identity pool attributes:

resource "google_project_iam_member" "secret_access" {
  for_each = toset(["production", "staging"])
  project  = "my-project"
  role     = "roles/secretmanager.secretAccessor"
  member   = "principalSet://iam.googleapis.com/projects/\({PROJECT_NUMBER}/locations/global/workloadIdentityPools/\){POOL_ID}/attribute.namespace/${each.value}"
}

This grants Secret Manager access to all pods authenticated from the production or staging namespaces. The principalSet syntax allows matching on attributes. You can also restrict to specific service accounts:

member = "principal://iam.googleapis.com/.../subject/system:serviceaccount:production:payment-processor"

How to Verify the Setup

You can verify the setup with a simple Python script that lists secrets from Secret Manager. This runs inside a pod on your on-premises cluster:

# list_secrets.py - running on-prem, accessing GCP Secret Manager
from google.cloud import secretmanager

def list_secrets(project_id: str):
    """
    List all secrets in a GCP project.

    No credentials are passed explicitly. The google-cloud-secret-manager
    library automatically:
    1. Reads GOOGLE_APPLICATION_CREDENTIALS env var (set by Kyverno)
    2. Loads the credential configuration JSON
    3. Reads the K8s ServiceAccount token from /run/secrets/...
    4. Exchanges it for a GCP access token via STS
    5. Uses that token to call the Secret Manager API
    """
    client = secretmanager.SecretManagerServiceClient()
    parent = f"projects/{project_id}"

    print(f"Secrets in {project_id}:")
    print("-" * 40)

    for secret in client.list_secrets(request={"parent": parent}):
        secret_name = secret.name.split("/")[-1]
        print(f"  - {secret_name}")

    print("-" * 40)
    print("Authentication: Workload Identity Federation")
    print("Credentials: None stored, token exchanged at runtime")

if __name__ == "__main__":
    list_secrets("my-project-id")

Run this inside your labeled pod:

$ kubectl exec -it my-app-xyz -- python list_secrets.py

Secrets in my-project-id:
----------------------------------------
  - database-password
  - api-key-stripe
  - oauth-client-secret
  - ml-model-api-key
----------------------------------------
Authentication: Workload Identity Federation
Credentials: None stored, token exchanged at runtime

No service account key. No secret mounted. Just a Kubernetes ServiceAccount token exchanged for GCP credentials at runtime.

This same pattern works for any GCP service — Secret Manager, Cloud Storage, BigQuery, Pub/Sub, and Vertex AI.

How to Connect On-Prem Apps to Cloud GPUs

Consider a typical flow: an on-prem order processing service needs to call a Vertex AI endpoint for fraud detection. The model runs on GPUs in Google Cloud (you can spin up A100s in minutes, not months). The application logic stays on-prem (you've already paid for that compute).

With the IAM bindings in place, any pod in the allowed namespaces can call Vertex AI:

# fraud_detector.py - running on-prem, calling cloud GPUs
from google.cloud import aiplatform

def check_fraud(transaction: dict) -> float:
    """
    Call a Vertex AI endpoint for fraud detection.

    The model runs on A100 GPUs in Google Cloud.
    This code runs on-prem in the datacenter.

    Authentication is automatic:
    1. Kyverno injected GOOGLE_APPLICATION_CREDENTIALS
    2. The aiplatform SDK reads the credential config
    3. K8s SA token is exchanged for GCP token via STS
    4. Request is authenticated to Vertex AI
    """
    endpoint = aiplatform.Endpoint(
        endpoint_name="projects/my-project/locations/us-central1/endpoints/fraud-model"
    )
    prediction = endpoint.predict(instances=[transaction])
    return prediction.predictions[0]["fraud_score"]


def generate_embeddings(texts: list[str]) -> list[list[float]]:
    """
    Generate text embeddings using a cloud-hosted model.

    Embedding models are GPU-intensive. Running them on-prem
    would require dedicated hardware. In the cloud, you pay per request.
    """
    from vertexai.language_models import TextEmbeddingModel

    model = TextEmbeddingModel.from_pretrained("text-embedding-004")
    embeddings = model.get_embeddings(texts)
    return [e.values for e in embeddings]

The developer doesn't think about authentication at all. They add the label to their deployment, and their on-prem pod can call:

• Vertex AI endpoints for ML inference on cloud GPUs

• Cloud Storage for model artifacts and training data

• BigQuery for feature stores and analytics

• Pub/Sub for event streaming between environments

• Secret Manager for API keys and configuration

This is the hybrid platform working as intended.

How to Scale GPU Access with CEL Conditions

CEL conditions become especially powerful when you want to restrict GPU access to specific namespaces. For example, to allow only ML-related namespaces to access Vertex AI:

attribute.namespace in ["ml-inference", "ml-training", "data-science"] &&
  attribute.service_account.startsWith("ml-")

You can also grant different access levels per namespace:

# ML inference namespace gets prediction access
resource "google_project_iam_member" "ml_inference" {
  project = "my-project"
  role    = "roles/aiplatform.user"
  member  = "principalSet://iam.googleapis.com/.../attribute.namespace/ml-inference"
}

# Data science namespace gets full Vertex AI access (for experimentation)
resource "google_project_iam_member" "data_science" {
  project = "my-project"
  role    = "roles/aiplatform.admin"
  member  = "principalSet://iam.googleapis.com/.../attribute.namespace/data-science"
}

The on-prem application teams don't need to know or care about GCP IAM. They deploy to the right namespace, add a label, and the platform handles the rest.

The Security Properties Compared

Here's a side-by-side comparison of the two authentication approaches:

The short-lived nature of tokens deserves emphasis. Even in a worst-case scenario where a token is somehow exfiltrated, it expires. Kubernetes ServiceAccount tokens have a configurable lifetime, and the GCP access tokens issued by STS are valid for one hour. A service account key, by contrast, remains valid until someone explicitly rotates it — often years.

The Complete Infrastructure as Code Layout

The entire solution is codified in Terraform, managing both GCP and Kubernetes resources:

workload-identity-federation/
├── providers.tf      # Google + Kubernetes providers
├── locals.tf         # Configuration (namespaces, project ID, etc.)
├── gcp.tf            # Identity pool, provider, IAM bindings
└── kubernetes.tf     # ConfigMap with credential configuration

A single terraform apply:

1. Creates the Workload Identity Pool in GCP

2. Configures the OIDC provider with your cluster's JWKS

3. Sets up IAM bindings for allowed namespaces

4. Creates ConfigMaps in each namespace with the credential configuration

Combined with the Kyverno policy, you get a fully automated pipeline:

New namespace added to allowed list
        │
        ▼
Terraform creates ConfigMap in that namespace
        │
        ▼
Developer deploys with label
        │
        ▼
Kyverno injects credentials automatically
        │
        ▼
Pod authenticates to GCP via OIDC
        │
        ▼
Application accesses GCP services

No tickets. No key requests. No secrets to manage.

How to Run a Proof of Concept with vCluster

To validate this works outside GKE, you can set up a demonstration using vCluster — a virtual Kubernetes cluster that runs inside another Kubernetes cluster. This proves the solution works for any cluster. You can setup vCluster in Docker using vind

# vcluster.yaml
experimental:
  docker:
    nodes:
      - name: worker-1
      - name: worker-2
deploy:
  cni:
    flannel:
      enabled: true
controlPlane:
  distro:
    k8s:
      version: "v1.35.0"

[root@localhost #] vcluster create hybrid --driver docker -f vcluster.yaml
[root@localhost #] kubectl get nodes
hybrid-control-plane   Ready    control-plane   14d   v1.34.0   192.168.107.2   <none>        Debian GNU/Linux 12 (bookworm)   7.0.5-orbstack-00330-ge3df4e19b0a0-dirty   containerd://2.1.3
hybrid-worker          Ready    <none>          14d   v1.34.0   192.168.107.3   <none>        Debian GNU/Linux 12 (bookworm)   7.0.5-orbstack-00330-ge3df4e19b0a0-dirty   containerd://2.1.3
hybrid-worker2         Ready    <none>          14d   v1.34.0   192.168.107.4   <none>        Debian GNU/Linux 12 (bookworm)   7.0.5-orbstack-00330-ge3df4e19b0a0-dirty   containerd://2.1.3

Inside the vCluster, deploy a simple test deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gcp-test
  labels:
    workload-identity-federation: "enabled"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gcp-test
  template:
    metadata:
      labels:
        app: gcp-test
    spec:
      containers:
        - name: test
          image: google/cloud-sdk:slim
          command: ["sleep", "infinity"]

Exec into the pod and verify:

$ kubectl exec -it gcp-test-xxx -- bash

# Inside the pod:
\( gcloud auth login --cred-file=\)GOOGLE_APPLICATION_CREDENTIALS
Authenticated with external account credentials for: [principal://iam.googleapis.com/...]

$ gcloud secrets list --project=my-project
NAME                 CREATED
database-password    2024-01-15T10:30:00Z
api-key              2024-01-14T09:15:00Z

No keys. No secrets mounted. Just identity federation working as designed.

Common Issues and How to Solve Them

How to Handle JWKS Retrieval for Air-Gapped Clusters

If your cluster's OIDC discovery endpoint isn't publicly reachable (most on-prem clusters aren't), you need to manually export the JWKS and upload it to GCP:

kubectl get --raw /openid/v1/jwks > jwks.json

This file must be updated if the cluster's signing keys rotate. Set up a periodic job that checks for key changes and updates the Terraform configuration.

How to Fix Issuer URL Mismatches

The iss claim in the Kubernetes token must exactly match the issuer URL configured in the OIDC provider. For clusters using internal DNS:

issuer_uri = "https://kubernetes.default.svc.cluster.local"

This URL doesn't need to be reachable from GCP — the JWKS file provides the validation keys. But it must match what's in the token exactly.

How to Debug Token Exchange Failures

When authentication fails, the error messages can be cryptic. Common causes and fixes:

Conclusion

After implementing this setup, on-premises workloads authenticate to Google Cloud exactly like GKE workloads do — without a single long-lived credential. The security team is happy (no keys to audit), developers are happy (just add a label), and the platform team is happy (no more credential management tickets).

Here's what you accomplished in this tutorial:

1. /Understood why service account keys fail at scale and the security risks they introduce

2. Created a Workload Identity Pool and OIDC provider in GCP to trust your cluster's token issuer

3. Used CEL conditions to enforce fine-grained, namespace-level access policies

4. Automated credential injection into pods using a Kyverno ClusterPolicy

5. Bound IAM roles to federated identity attributes — no long-lived keys anywhere

6. Verified the setup by calling GCP APIs (Secret Manager, Vertex AI) from an on-prem pod

7. Proved the solution works on any Kubernetes cluster using vCluster

The technologies used here aren't new. OIDC has been in Kubernetes since version 1.20. Workload Identity Federation has been in GCP for years. Kyverno and Terraform are mature tools. What this tutorial puts together is an end-to-end solution that developers can adopt with minimal effort.

If your organization has disabled service account keys (or should), this is the path forward. Your on-prem and cloud clusters can finally be what they were always meant to be: secure extensions of each other.

The complete implementation is available as a Terraform module with Kyverno policies: github.com/shkatara/hybrid-platform-gcp-workload-identity-federation

If this helps, you can follow me on https://www.linkedin.com/in/shubhamkatara/, https://www.youtube.com/@kubesimplify, https://www.linkedin.com/company/kubesimplify/ and