Situation

Was Adding An HTTPS Listener To The PRD GKE Gateway. helm upgrade Got Stuck On A Field Manager Conflict, So I Threw --server-side --force-conflicts At It To Push Through — And It Wiped The Entire prd-litellm Namespace (Deployment / Pod / Service / HPA / Secret, All Gone). Recovery Worked, No Data Lost, But The Mechanism Is Worth Understanding Properly, Or I’ll Walk Into It Again.

Result First:

Turns Out Helm Upgrade Isn’t As Simple As “Apply The New Stuff.” Every Single Time, It Runs A Full Set Difference: Resources Recorded In The Last Release Minus Resources In This Render = What Gets Deleted. Break That Assumption Once (Some Resource That Should’ve Been Cleaned Out Of The Chart Long Ago Wasn’t) And Upgrade Turns Into A Delete Button — This Time It Was My Own Landmine Going Off.

How It Broke (Root Cause)

How A Release Gets Tracked

Every Helm 3 install/upgrade Stores The Full Rendered Manifest As A Secret (Default Backend), Living In The Release’s Namespace, Named sh.helm.release.v1.<release-name>.v<revision>. That Secret Is Helm’s Single Source Of Truth For “What Resources Does This Release Own.”

# See What Helm Thinks It Created "Last Time"
helm get manifest <release> -n <namespace>

The Three-Way Comparison On Upgrade

helm upgrade Runs A Three-Way Comparison:

① Last Release's Recorded Manifest (What's In The Secret)
② Current Live Cluster State
③ This Time's Freshly Rendered Chart Manifest

The Diff Between ① And ③ Decides What Gets CREATE / PATCH / DELETE:

The Mechanism Itself Is Completely Reasonable — It’s What Lets “Removing A Resource From The Chart” Be Handled Automatically Without A Manual kubectl delete. The Problem: If ① Contains A Resource That Got Rendered By Accident On Some Old Revision And Was Never Supposed To Exist, It Just Sits In ① Until Some Future Upgrade’s ③ Finally Doesn’t Have It — And That’s When The Delete Fires. Could Be Months Later, In Someone Else’s Hands Entirely, With No Way To Trace “Why Is This Getting Deleted Now.”

Incident Timeline

Before 6/17   templates/namespace.yaml Had No {{- if }} Guard,
              An Early Revision Deployed With createNamespace: true,
              The Namespace Object Got Recorded Into That Release's ① (Manifest Record)
              ↓
Later         Switched To createNamespace: false + {{- if .Values.createNamespace }} Guard,
              But From Then On Every Rendered ③ No Longer Included The Namespace
              (Regular Helm Upgrades Never Actually Triggered This Delete Path)
              ↓
2026-06-29    Field Manager Conflict Forced The Switch To
              --server-side --force-conflicts
              ↓
              ① (Namespace In Record) − ③ (No Namespace In Render)
              = Namespace Marked For Deletion
              ↓
              kubectl Sends DELETE On The Namespace
              ↓
              K8s Behavior: Deleting A Namespace Deletes "Everything" Inside It
              (Deployment, Pod, Service, HPA, Secret — Even Helm's Own
              Release Record Secret, All Taken Down With It)

One Command, The Whole Environment Gone — And What Got Blown Up Wasn’t Even Related To This Change (I Just Wanted To Add An HTTPS Listener). It Was A Completely Unrelated Piece Of Historical Baggage That Had Been Dormant For Months, Finally Detonated.

Client-Side vs --server-side

Key Misconception To Clear Up: It’s Not That --server-side Is More Dangerous. It’s That This Particular Upgrade — Which Happened To Use Server-Side — Was The First Time The Long-Dormant “Namespace Missing From ①” Discrepancy Actually Got Settled. This Landmine Was Going To Go Off Eventually, It Just Happened To Be Server-Side That Stepped On It.

Notes

• kubectl diff -f - Is Blind To Deletions. It Only Tells You What Changed In The Content You Fed It — It Can’t Tell You What Disappeared From The List. This Was The Second Trap I Fell Into During The Postmortem: I Assumed kubectl diff Would Act As A Safety Net, Turns Out It’s Completely Useless For This.
• --server-side Wasn’t The Culprit, Just The Trigger. The ①−③ Delete Logic Runs In Both Modes — The Only Difference This Time Was The Field Manager Conflict Forcing The Switch, Which Was The First Time This Path Actually Got Exercised.
• What Got Deleted Had Nothing To Do With What I Was Changing. I Just Wanted To Add An HTTPS Listener To The Gateway — The Namespace That Got Deleted Was Baggage Left Over From A Chart Change Months Earlier.
• Namespace / PVC / Secret Are The Most Dangerous Because Deletion Cascades. Delete A Namespace, Everything Inside Goes With It, Including Helm’s Own Release Record Secret.

How To Prevent It

The Core Tool: Diff The Resource “List,” Not The “Content”

The Right Way Is To List Out The Resources (Kind + Name) In ① And ③ Separately, Then Diff The Sets:

# ① Resource List From The Last Release's Recorded Manifest
helm get manifest <release> -n <namespace> | grep -E "^kind:|^  name:" > /tmp/before.txt

# ③ Resource List From This Time's Rendered Chart
helm template <release> ./chart -f values-<env>.yaml -n <namespace> \
  | grep -E "^kind:|^  name:" > /tmp/after.txt

# Look At The Diff — Lines That Only Show Up In "before" Are What's About To Be Deleted
diff /tmp/before.txt /tmp/after.txt

If Any Unexpected “About To Be Deleted” Item Shows Up — Especially High-Blast-Radius Resources Like Namespace, PVC, Secret — Stop And Investigate Before Applying.

Scope The Blast Radius, Don’t Flip The Whole Release To Server-Side

A Field Manager Conflict Usually Happens On A Specific Field Of A Specific Resource, Not The Whole Release:

# Scenario One: A Resource Is Missing Its Helm Ownership Annotation (What Actually Happened Here — Safe Fix)
kubectl annotate <kind> <name> -n <namespace> \
  meta.helm.sh/release-name=<release> \
  meta.helm.sh/release-namespace=<namespace>

# Scenario Two: A Single Field Got Claimed By Another Controller
kubectl patch <kind> <name> -n <namespace> --type=json \
  -p '[{"op":"replace","path":"/spec/xxx","value":"yyy"}]'

Both Are Surgical Fixes — Neither Triggers Helm’s Full ①−③ Diff Across The Whole Release.

Real-World Application

Reference:

• Helm Upgrade — Using Helm
• Helm — Server-Side Apply
• Kubernetes — Server-Side Apply
• Real Incident: litellm-gke PRD Environment, 2026-06-29