Situation

Needed The PRD Domain’s Certificate To Auto-Renew, Which Meant Switching GatewayClass From Regional (gke-l7-regional-external-managed) To Global (gke-l7-global-external-managed) — Certificate Manager’s Google-Managed Auto-Renewing Certs Only Support Global. That Night I Hit Three Traps Back To Back: Certificate Manager’s Annotation Syntax Is Completely Different, GCPBackendPolicy Can Only Bind One Service And Two Of Mine Fought Over It, And Global Security Policy’s Default Rule Is Allow, Not Deny. Five Days Later, Adding An Internal Gateway, The Second Trap Bit Me Again — Verbatim. That’s When I Knew It Wasn’t Bad Luck, It Was A Rule.

Result First:

Gateway API’s Three-Layer Model, Quick Primer

If You Haven’t Touched Gateway API Yet: It Splits Ingress’s “One Resource Rules Everything” Mess Into Three Layers — GatewayClass (Vendor Declares Who Implements It, Here That’s GKE), Gateway (Infra Team Owns The Entry Point, Protocol, TLS), HTTPRoute (Developers Own The Routing Rules). Every Trap In This Post Happened At The GatewayClass Layer — Same GKE Gateway Controller, But Regional And Global GatewayClass Map To Completely Different Underlying GCP Resources (Regional External LB vs. Global External LB). Syntax, Limits, Behavior — None Of It Carries Over. Switching Between Them Isn’t Tweaking A Parameter, It’s Swapping Out The Whole Underlying Mechanism.

Trap One: Certificate Manager Mounts Differently On Regional vs Global

Copied The Regional Syntax First, Got This Immediately:

Error GWCER105: invalid TLS option key "networking.gke.io/certificate-map"

The Two GatewayClasses Don’t Share A Certificate-Mounting Syntax At All:

It’s Not A Typo’d Key. It’s Two Entirely Different Mounting Mechanisms — One Lives At The Listener Level Under tls.options, The Other Lives At The Gateway Metadata Level As An Annotation. Different Locations Entirely. Copy The Regional Syntax Onto Global And It’s Guaranteed To Blow Up.

Trap Two: GCPBackendPolicy Only Binds One Service — And Fails Silently

This Is The Nastiest Of The Three Because It Doesn’t Error Out At kubectl apply Or helm upgrade Time:

kubectl get gcpbackendpolicy <name> -n <namespace> -o jsonpath='{.status}'
# conflicted with GCPBackendPolicy "xxx" of higher precedence, hence not applied

The Reason: A K8s Service — Regardless Of Whether It Backs A Regional Or Global Backend Service — Can Only Be Bound By One GCPBackendPolicy. If Both A Regional And A Global Gateway’s HTTPRoute Point At The Same Service (My Case: Each Side Needed A Different Cloud Armor Policy Attached), Whichever Got Created First Wins, And The Second One Just Silently Fails — No Error At Apply Time, You Have To Go Check status.conditions To Even See It. My Fix At The Time Was Going With “Only One Gateway Serves This Service At A Time” — Fully Tearing Down The Regional Gateway’s Resources So Global Could Claim The GCPBackendPolicy Binding. The Other Option Is Giving Each Gateway Its Own Dedicated Service (Same Pod Selector, Two ClusterIP Services).

Five Days Later, The Repeat: Adding An Internal Gateway (For Private Connectivity), I Hit The Exact Same Trap — PRD’s Main Service Had A Global Policy Attached, Which Couldn’t Bind To The Regional Backend Service The Internal Gateway Generated, Same SetSecurityPolicy ... does not exist Error. This Time I Just Applied What I’d Already Learned: Give The Internal Gateway Its Own Dedicated Service, Don’t Reuse The Main One. Actually Getting Bit A Second Time Was A Good Thing — It Proved “One Service, One GCPBackendPolicy” Wasn’t A One-Off Quirk Of This Particular Migration. It’s A Checklist Item For Every New Gateway You Ever Add.

Trap Three: Global Security Policy Defaults To Allow, Not Deny

gcloud compute security-policies describe <policy-name> --format=yaml
# priority 2147483647 "default rule" defaults to action: allow — your allowlist does nothing

Regional Policy (google_compute_region_security_policy) Doesn’t Have This Problem, But The Moment You Create A Global Policy (google_compute_security_policy), GCP Auto-Generates A Priority 2147483647 allow Default Rule — And That Priority Collides With Whatever Deny-All Rule You Meant To Create By Hand (Cannot have rules with the same priorities). The Right Fix Isn’t Deleting And Recreating — It’s terraform import-ing The Auto-Generated Rule Into State, Then Using Terraform To Flip The Action To Deny:

terraform import google_compute_security_policy_rule.<resource_name> \
  "projects/<project>/global/securityPolicies/<policy-name>/priority/2147483647"
terraform plan   # Should Show An In-Place Update: action: allow -> deny(403)
terraform apply

Notes

• GWCER105 Isn’t A Typo’d Key, It’s Two Entirely Different Cert-Mounting Mechanisms. Regional Uses Listener-Level tls.options, Global Uses A Gateway-Metadata-Level Annotation — Different Locations Entirely. Copy One Onto The Other, It Breaks.
• GCPBackendPolicy Conflicts Are The Nastiest Of The Three Because They Don’t Error At Apply Time. You Only See “Conflicted With … Of Higher Precedence, Hence Not Applied” By Checking status.conditions — First Created Wins, Second One Fails Silently.
• Import The Global Security Policy’s Auto-Generated Allow Rule Into Terraform State, Don’t Delete And Recreate It. The Priority Is A Fixed Value (2147483647) That Collides With Any Hand-Built Deny-All Rule.
• Global LB Provisioning Takes Noticeably Longer Than Regional. Regional Usually Programs In 1-3 Minutes, Google’s Docs Say Global Can Take 10-15 (Config Has To Propagate To Global GFE Nodes), And Even After Programmed: True, TLS Taking Full Effect At The Edge Can Take A Few More Minutes — If curl Throws SSL_ERROR_SYSCALL But openssl s_client Tests Fine, It’s Usually Just Still Propagating. Wait It Out.
• This Same Migration Also Hit A Helm Field Manager Conflict, Which I Pushed Through With --server-side --force-conflicts And Ended Up Wiping A Namespace As A Result — That’s A Completely Different Mechanism (Helm 3’s Release Tracking/Delete Logic), Covered In A Separate Post, Not Repeated Here.

The Rule: Check Service Sharing Before Adding Any New Gateway

The Real Value Of This Post Isn’t The Three Traps Individually — It’s That Trap Two Got Confirmed Twice. Hit It Once, You’d Chalk It Up To “Just This Migration’s Bad Luck.” Hit It Again Five Days Later Adding An Internal Gateway, And Now It’s Confirmed: This Is A Checklist Item For Every New Gateway You Ever Add, Not A One-Time Fluke. External Or Internal, Regional Or Global — Before Adding A New Gateway, Check Whether It’ll Share A Service With An Existing One. If It Does, Pick One Of Trap Two’s Two Fixes (“Only One Gateway Serves This Service At A Time” Or “Give Each Gateway Its Own Service”) Upfront, Don’t Wait For The Silent Failure To Send You Digging.

Real-World Application

Reference:

• Secure a Gateway | GKE
• GatewayClass capabilities | GKE
• GCPBackendPolicy | GKE Gateway API
• Real Incident: Internal Service, PRD Regional→Global Migration; Internal Gateway Repeat Confirmation