Upgrading an Istio Service Mesh: What We Learned About Rollbacks, Auth, and Capacity

There is a moment in platform engineering when someone says:

We need to patch the service mesh.

And you think: "Okay. We'll update the version and roll it out."

Then you discover two things.

The service mesh sits in the path of almost every request.

And the gateway has authentication-related configuration that has to keep working through the change.

Suddenly, "update the version" sounds a lot less casual.

That was my experience upgrading a self-managed Istio-based service mesh on GKE.

Why We Had to Do It

We received a security notice about a high-severity vulnerability affecting Envoy, the proxy the mesh is built on.

The fix wasn't a simple patch on our existing release line. It required moving to a newer release.

At a high level, a request looked like this:

1. 01User request
2. 02Ingress gatewayan Envoy proxy, with authentication-related configuration
3. 03Envoy sidecar
4. 04Application

Envoy wasn't some optional component running off to the side.

It handled the traffic to our applications.

And the gateway's authentication-related proxy configuration needed to be validated during the upgrade.

If the upgrade broke it, two bad outcomes were possible:

• Users locked outnobody can sign in
• Auth misbehavesnot working as expected

Neither was something I wanted to discover from a user report.

First Surprise: The Repository Wasn't the Whole Truth

I started where many of us start: the infrastructure repository.

It told me which mesh version had been configured when the cluster was created.

But the installation ran through a one-time provisioning step. Changing the version in Git wouldn't run it again.

So I checked the cluster itself.

• The repositorywhat was configured at creation
• The live clusterwhat is actually running today

The running mesh was significantly older than the repository made obvious.

And the live cluster contained configuration that wasn't fully represented in the infrastructure repository.

That changed my first question from:

"What does the configuration say we deployed?"

to:

"What is actually running, and what would we lose if we had to rebuild it?"

Infrastructure-as-code is valuable.

But it describes intent.

Before a production change, the live system has a story of its own.

The Part I Was Most Worried About Wasn't the Install

Installing a new control plane wasn't my biggest concern.

Authentication was.

Low-level proxy configuration gives you flexibility — and it also means an upgrade can change behavior in ways a successful deployment won't reveal. Istio's own documentation warns that EnvoyFilter resources, for example, can break across versions.

So before changing anything, I captured a baseline of the gateway's working proxy configuration.

I wanted to know which authentication-related pieces were present before the upgrade — not try to reconstruct that afterward.

But even a baseline has a limit:

Configuration exists ≠ authentication works

The real test would be the user journey: does an unauthenticated request redirect to login, and can someone actually complete a login in the browser?

How We Made the Upgrade Reversible

Instead of replacing the old control plane in place, I installed the new revision alongside it.

Current revision

Existing workloads

Rollback path

New revision

Migrated workloads

The future

Then I moved workloads to the new revision gradually, starting in development:

1. 01Install the new revision alongside the old one
2. 02Point namespaces at itistio.io/rev label
3. 03Restart workloadssidecars get the new proxy
4. 04Migrate the gateways
5. 05Verify
6. 06Bake, then remove the old revision

We could inspect each stage before moving on, while the old revision stayed available as a rollback path.

The version gap was larger than Istio's documented canary upgrade path, which covers jumps of up to two minor versions. So I tried the direct jump in dev first.

Not because I assumed it was safe — but because dev was the place to find out.

If the jump failed, we could stop and take a more incremental path before touching production.

That distinction matters:

• “It worked in my plan”not evidence
• “It worked in an attended, reversible test”with the risky behavior verified

Then the Environment Joined the Conversation

The upgrade wasn't just an Istio problem.

The environment introduced its own operational constraints: private cluster access, restricted outbound connectivity, and permissions required by the newer installation process that the old one never needed.

That last one taught me the most.

I had checked my own access.

But the installer wasn't running as me.

The identity executing the installation mattered more than my own interactive access.

Once I understood that, the fix wasn't "retry the command." It was to give the identity actually running the installer the permission it needed — through the people allowed to grant it.

A tool can be the same tool in name, and still have different requirements a few versions later.

A Rollout Timeout That Wasn't the End of the Rollout

After installing the new revision, workloads needed new pods with updated sidecars.

Restarting too many deployments at once created a surge of replacement pods:

1. 01Many deployments restarted at once
2. 02A surge of replacement pods
3. 03Not enough spare capacity, all at once
4. 04Pods waiting to be scheduled
5. 05ProgressDeadlineExceeded
6. 06Capacity frees up — rollouts converge

ProgressDeadlineExceeded looks alarming.

But it doesn't mean the Deployment controller gave up. Kubernetes only reports the condition — it doesn't roll anything back. The old pods kept serving while the replacements waited for capacity, and once capacity was available, the rollouts converged.

We didn't see an outage from that stall.

The lesson was clear:

A mass restart is a capacity event.

For the next environment: batch the restarts, and wait for each group before starting the next.

The Verification That Mattered

Once the workloads and gateways were on the new revision, I checked the version of the running proxies, and compared the gateway configuration with the baseline.

Then I tested the authentication flow itself:

1. 01Unauthenticated request
2. 02Redirect to login
3. 03Login page responds
4. 04Browser login succeeds

That last step was the decision gate.

A new version number and the expected configuration were useful signals.

Neither proved that users could still sign in.

We kept the old control plane during a bake period. Removing it immediately would have made the cluster tidier. Keeping it gave us a rollback option while the new version saw real traffic.

What I Learned

Check the running system, not only the repository. A configuration file can't tell you everything that has happened in a cluster since it was created.

Find the fragile behavior before planning the rollout. The important question wasn't "will the new control plane start?" It was "will authentication still work?"

Know which identity executes the command. Human access, jump-host access and automation access are different things.

Treat capacity as part of the change plan. Rolling updates are safer than taking everything down — but replacement pods still need somewhere to run.

Verify outcomes, not just objects. A healthy pod and applied configuration are not substitutes for a working login.

And perhaps the biggest one:

Make the next upgrade easier than this one.

A mesh that drifts far behind isn't just a version problem. It's an operational-process problem. Regular upgrades and better automation — or a managed control plane — keep the next security notice from turning into a big jump.

Final Thought

I started with what sounded like a routine request:

"Upgrade the service mesh to address a vulnerability."

What I actually worked through was:

• Configuration drift
• Restricted network access
• Execution identity & permissions
• Kubernetes scheduling
• Low-level proxy configuration
• Authentication testing
• Rollback strategy

The upgrade taught me something I keep seeing in platform engineering:

The component you change is rarely the whole change.

The real work is understanding everything connected to it — and proving those connections still work afterward.