Situation

I Was Removing hostNetwork: true From An Internal API Service Running On GKE, Because It Capped Each Node To One Pod And Forced The Cluster To Scale Out Wider Than It Needed To. Right After The Change, Every Connection From These Pods To An On-Prem Service Started Getting Blocked By The Firewall. The Rule Clearly Allowed The Node Subnet, So I Suspected A Stale Rule, But Three Re-Checks Later The Real Problem Showed Up: The Source IP On The Wire Wasn’t The Node IP, It Was The Pod’s Own IP. I’d Always Assumed Anything Leaving A Node Gets SNAT’d To The Node IP, And That Doesn’t Hold On GKE’s VPC-Native Mode, Where The Pod IP Is A Real, Routable Secondary Range Inside The VPC And RFC1918 Traffic Isn’t SNAT’d. The Fix Was Deploying ip-masq-agent And Explicitly Excluding The On-Prem CIDR From The “Don’t SNAT” List. It Left A Bigger Question Behind: How Many Of My K8s Networking Assumptions Only Hold On One Specific Platform? I’d Hit The Pod IP Trap, But I Didn’t Know How ClusterIP And LoadBalancer IP Work On On-Prem, GCP, And AWS, So This Post Is What I Found Chasing That Down.

Result First:

Trigger Removed hostNetwork: true On GKE; On-Prem Firewall Started Blocking Traffic That Used To Pass
Root Cause GKE VPC-Native Mode: Pod IP Is A Real, Routable VPC Secondary Range, Not An Overlay Address, So No SNAT By Default
Fix Deployed ip-masq-agent, Explicitly Excluded The On-Prem CIDR From The Non-SNAT List
Bigger Question Which K8s Networking Assumptions Are Actually Platform-Specific? Compared Pod IP And Service IP Behavior Across On-Prem, GCP, And AWS
Outcome Full Pod IP + Service IP Comparison Table Across Three Platforms, Real AWS Target-Group Testing Included

Part One: Pod IP, Where The Three Platforms First Diverge

Traditional K8s (Overlay Mode, E.g. RKE2 + Flannel)

Pod IP: 10.100.x.x  ← Virtual Overlay, Unroutable Outside The VPC
                       ↓ Packet Leaves The Node
                  iptables MASQUERADE (Automatic, Built In)
                       ↓
External View: Node IP

Outside The Cluster, The Pod IP Means Nothing. SNAT Is Mandatory, No Exceptions, No Configuration Needed.

GCP: VPC-Native (Default Mode)

Node Subnet:    10.0.0.0/16     ← Primary Range
Pod CIDR:       10.4.0.0/14     ← Secondary Range, Pod IPs Live Here
Service CIDR:   10.8.0.0/20     ← Secondary Range

The Pod IP Is A VPC Secondary Range: A Real, Routable Address Across The Entire VPC. GKE Treats This Traffic As Not Needing SNAT, Unless You Explicitly Tell It Otherwise Via ip-masq-agent.

AWS: VPC CNI (Default Mode)

EKS Pod IPs Come Directly From The Node’s Subnet, Specifically An ENI Secondary IP, Not An Independent Secondary Range; Pods Share The Exact Same VPC Subnet As Their Node. SNAT Logic Is Built Into The aws-node DaemonSet:

AWS_VPC_K8S_CNI_EXTERNALSNAT=false  # Default: SNAT To Node IP Only When Leaving The VPC

Traffic Staying Inside The VPC (Including To On-Prem, Since Pod IP And Node IP Share The Same VPC CIDR) Doesn’t Get SNAT’d. Most EKS Users Never Need An ip-masq-agent-Style Fix, Because On-Prem Firewall Rules Are Already Written Against The VPC CIDR. Pod IPs Fall Inside It Naturally.

Pod IP: Side By Side

Platform Pod IP Source What On-Prem Sees By Default Needs An Extra SNAT Patch?
On-Prem Overlay (Flannel/Calico) Virtual Overlay Node IP (SNAT Is Inherent) No
GCP VPC-Native (Default) Dedicated Secondary Range Pod IP ← The Trap Yes (ip-masq-agent)
AWS VPC CNI (Default) Same Subnet As Node (ENI Secondary IP) Pod IP (But Usually Already Inside The On-Prem Allowlist) Usually No

The Core Insight: Both GCP And AWS Give Pods “Real VPC IPs”, But The Odds Of Getting Bitten Are Different. GCP Puts Pods In A Separate, Dedicated Secondary Range, Which Is Easy For An On-Prem Firewall Rule To Miss. AWS Puts Pods In The Same Subnet As The Node, Which Existing Rules Usually Already Cover. Same “VPC-Native” Label, Different Range-Carving Strategy, Different Place The Trap Hides.

Part Two: Service IP, Three Different Stories

Pod IP Is Only Half The Picture. K8s Services Have Two Layers, ClusterIP (Internal) And LoadBalancer/External IP (External), And These Two Layers Behave Differently Across The Three Platforms, Independent Of Whether Pod IP Is VPC-Native Or Not.

ClusterIP: Identical Everywhere, Purely Virtual

Whether You’re On-Prem, On GKE, Or On EKS, ClusterIP Is Never Allocated From Any VPC Range, Primary Or Secondary. It Exists Only As iptables/IPVS Rules, With kube-proxy DNAT’ing To The Actual Pod IP.

I Confirmed This Incidentally While Setting Up MetalLB On-Prem. Before Installing It You Have To Check kube-proxy’s Mode (iptables Or IPVS, Since IPVS Mode Needs strictARP Enabled Or It Conflicts With MetalLB’s ARP Broadcasting):

kubectl logs -n kube-system kube-proxy-<node-name> | grep "proxy mode\|Proxier"
# Output: Using iptables Proxier

Whether kube-proxy Is VPC-Native Or Which Platform You’re On Doesn’t Touch How ClusterIP Works. It’s A Core K8s Mechanism, Not Something The Cloud Vendor’s Network Model Decides. The Real Fork In The Road Is The Next Layer Up.

LoadBalancer / External Traffic: Where The Three Platforms Actually Split

On-Prem: MetalLB, Fundamentally An Extension Of A Node’s IP

Bare-Metal Environments Have No Cloud LB API, So MetalLB Simulates One In Two Modes:

  • L2 Mode (ARP): One Node Announces “This VIP Is Mine” Via ARP Broadcast:
External Request → 192.168.50.100:80 (VIP)
              ↓
  MetalLB Speaker (DaemonSet, One Per Node)
  → Announces VIP Ownership Via ARP Broadcast
  → Routes Traffic Into The Matching Service

The VIP Is Fundamentally Still A Node’s IP Answering On The Wire, It Can Just Fail Over To Another Node. It’s Unrelated To Pod IP Or Service CIDR, A Separate Mechanism.

GCP: Internal ALB Uses A Third, Dedicated Range, The Proxy-Only Subnet

I Only Understood This While Deploying GKE Gateway API: The IP An Internal Application Load Balancer (gke-l7-rilb) Gets Is Neither The Pod’s Secondary Range Nor The Node’s Primary Range. It’s A Dedicated Proxy-Only Subnet (purpose=REGIONAL_MANAGED_PROXY), Carved Out Specifically For The Envoy Proxy Layer:

Existing Proxy-Only Subnet: prod-proxy-only-subnet (10.0.240.0/23)
Internal Gateway Endpoint: 10.0.240.10 / 10.0.240.11 (gke-l7-rilb)

Traffic Path:
Inside The Shared VPC → Internal ALB (From The Proxy-Only Subnet)
              → GKE HTTPRoute → ClusterIP Service → Pod

Which Means A Single GKE VPC-Native Cluster Has Three Distinct IP Sources Simultaneously: Node (Primary Range), Pod (Secondary Range), LB Endpoint (Proxy-Only Subnet, Yet Another Independent Carve-Out). None Of The Three Overlap, Which Is Also Why GCP’s Network Planning Docs Casually Ask You To Reserve Four Or Five Separate Ranges. Every Layer Gets Its Own.

AWS: Target Group’s instance vs. ip Mode Is The Real Fork

I Didn’t Have A Clean Production Case For This Layer, So I Spun Up A Throwaway EKS Cluster Just To Test It. I Verified It Directly Instead Of Trusting The Docs Alone. Using eksctl (Default VPC CNI Addon), Installed The AWS Load Balancer Controller, Then Built Two Services Off The Same Deployment, Differing By A Single Annotation:

# ip mode
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip

# instance mode
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance

The Two Pods That Actually Came Up:

nginx-test-xxx-mx2nn   192.168.28.157   (Node A, AZ a)
nginx-test-xxx-wx2ds   192.168.61.82    (Node B, AZ b)

Each Service Triggered Its Own NLB. Once Both Were Ready, I Pulled The Real Registered Targets Via aws elbv2 describe-target-health:

ip Mode Target Group instance Mode Target Group
Registered Targets 192.168.28.157:80, 192.168.61.82:80 i-0a016cfa17859c450:31383, i-096a2d52c05e70700:31383
Maps To The Two Pods’ Real IPs, One-To-One Two EC2 Instance IDs + A Randomly-Assigned NodePort (31383, From The nodePort Field In kubectl get svc)
Health Check Status Healthy Healthy

The ip Mode Target Is Literally The Pod IP, Identical To What kubectl get pods -o wide Prints. The NLB Bypasses NodePort And kube-proxy, Sending Packets Straight To The Pod. The instance Mode Target Is An EC2 Instance + NodePort. Traffic Still Has To Land On A Node First, Then Get Redirected By kube-proxy’s iptables Rules Before It Reaches The Pod. Same Path As Traditional On-Prem Routing, Just With A Cloud LB Sitting In Front.

ip Mode Only Works Because Of VPC CNI: The Pod IP Is Itself A Real, VPC-Routable IP That The LB Can Target Directly. This Is A Different Design Philosophy From GCP’s Proxy-Only Subnet: AWS Lets The LB See The Pod Directly, GCP Routes The LB Through An Intermediate Range Before It Ever Reaches ClusterIP.

Side-By-Side Comparison

Layer On-Prem GCP (GKE) AWS (EKS)
Pod IP Source Virtual Overlay Address VPC Secondary Range Same Subnet, ENI Secondary IP
Pod IP Needs SNAT Leaving The Cluster? Always (Inherent To Overlay) Needs Explicit Config (ip-masq-agent) Usually Not (Built Into The VPC)
ClusterIP, Fundamentally Virtual (kube-proxy DNAT) Virtual (kube-proxy DNAT) Virtual (kube-proxy DNAT)
LoadBalancer IP Source Node IP Extension (MetalLB ARP/BGP) Dedicated Proxy-Only Subnet Target Group (instance Or ip Mode)
LB-To-Pod Path LB → Service → kube-proxy → Pod LB → HTTPRoute → ClusterIP → Pod ip Mode: LB Hits Pod Directly; instance Mode: LB → NodePort → kube-proxy → Pod
Which CIDR Should On-Prem Firewall Allow? Node CIDR Pod Secondary Range (If Routing Direct Via VPC-Native) The Shared Pod/Node Subnet CIDR

Notes

  • The Trap Isn’t “VPC-Native Vs. Overlay,” It’s Where Each Platform Carves Out The Pod Range. GCP And AWS Both Give Pods Real VPC IPs, But GCP Puts Them In A Dedicated Secondary Range (Easy For A Firewall Rule To Miss) While AWS Shares The Node’s Own Subnet (Usually Already Covered).
  • ClusterIP Behavior Doesn’t Vary By Platform. It’s A kube-proxy Mechanism, Not A Cloud Vendor Networking Decision. Don’t Waste Time Cross-Checking It Between Environments; Spend That Time On The LoadBalancer Layer Instead.
  • GKE’s Proxy-Only Subnet Is A Third, Independent Range. Not Pod, Not Node. A GKE VPC-Native Network Plan Needs To Reserve It Explicitly, Both For External And Internal Gateways.
  • AWS Target-Type ip Vs. instance Isn’t Just A Performance Knob. It Changes Whether The LB Talks To The Pod Directly Or Goes Through NodePort + kube-proxy. Worth Checking Deliberately, Not Leaving On The Default.

The Rule: Never Assume Which IP Is On The Wire

The One Assumption That Broke Everything For Me Was Treating “Traffic Leaving A Node Gets SNAT’d To The Node IP” As Universal K8s Behavior. It’s Not. It’s Overlay-Networking Behavior, And Only Overlay Networking Guarantees It. The Moment A Platform Gives Pods Real, Routable VPC IPs (GCP And AWS, By Default), That Guarantee Is Gone, And Whether You Get Bitten Depends On Whether The Platform’s Pod Range Happens To Fall Inside Your Existing Firewall Rules. Same Story One Layer Up: Whether The LoadBalancer Talks To The Pod Directly Or Bounces Through NodePort Isn’t A Given Either, It’s A Per-Platform, Sometimes Per-Annotation Decision. The Only Safe Default Is To Check, Every Time, On Every Platform.

Real-World Application

Scenario Checklist
Before Any New K8s Service Needs To Reach On-Prem (DB, API, Internal System) Check Whether The Platform’s Pod IP Is VPC-Native (GCP/AWS Default Is Yes). If So, Confirm The On-Prem Firewall Covers The Pod Range, Not Just The Node CIDR
Choosing A Service Type (ClusterIP / NodePort / LoadBalancer) Remember ClusterIP Is Always Virtual And Behaves Identically Across Platforms. The Platform-Specific Research Belongs At The LoadBalancer Layer
Want An AWS LB To Hit Pods Directly, Skipping The NodePort Hop Confirm The Target Group Is Set To ip Mode, Not The Default instance Mode
Planning A New GKE VPC-Native Cluster’s Network Layout Remember The Proxy-Only Subnet Is A Third Range, Independent Of Pod/Node, Both External And Internal LBs Consume It. Reserve Enough Space For It Upfront
Debugging “On-Prem Firewall Is Blocking This Connection” Always Start By Confirming The Actual Source IP On The Wire (conntrack -E Or The On-Prem Firewall Log). Don’t Assume It’s The Node IP. The Answer Differs By Platform

Reference: