Key takeaways

  • On a GKE Dataplane V2 cluster, deleting a Kubernetes NetworkPolicy did not restore egress for already-running pods. Their Cilium eBPF endpoint kept enforcing a stale default-deny rule until the pod restarted.
  • The tell was a clean split: a netshoot pod created after the deletion connected fine, the original pod did not, and a restart fixed it instantly.
  • Root cause: per-endpoint eBPF policy maps were not recomputed for long-lived endpoints after the policy was removed. New endpoints picked up correct state from scratch.
  • This is a dataplane-level behaviour, not an application bug. Cilium now backs a majority of production clusters (adoption grew 47% year over year, per the Cilium Annual Report 2025, CNCF, Dec 2025), so the blast radius is wide.

A platform team deleted every NetworkPolicy in a namespace, expected traffic to open, and watched a running pod stay blocked anyway. No policy left to enforce, yet the packets kept dropping. This is the postmortem for that incident: a stale eBPF state bug on GKE Dataplane V2, reproduced end to end, with the exact commands that isolate it.

The environment was a production GKE cluster operated by a major European automotive manufacturer’s platform team. The affected workload was an internal PKI/ACME issuer used for cert-manager integration testing. We’ve kept the details generic on purpose. The failure mode is what matters, and it can hit any Dataplane V2 cluster.

What happened on the cluster?

The workload lost its database connection and never got it back. An internal PKI/ACME issuer started throwing MongoSocketOpenException timeouts against its MongoDB backend on port 27017. The platform team deleted all NetworkPolicies in the namespace to rule out policy enforcement, and the errors continued unchanged.

That is the part that felt wrong. When you remove every policy object, the API server has nothing left to enforce. The expected result is open egress. Instead, the running pod kept timing out on a TCP connection to mongodb:27017, as if a default-deny rule were still live.

The symptom chain was short and specific:

  1. The issuer pod lost MongoDB connectivity and reported connection timeouts.
  2. The team deleted every NetworkPolicy in the namespace, expecting full open traffic.
  3. Connectivity did not return. The same pod kept failing.

Gotcha: A deleted NetworkPolicy disappears from kubectl get netpol instantly. That tells you the API object is gone. It tells you nothing about whether the dataplane has actually recomputed the affected endpoints. Those are two different clocks.

Cilium is now the most-deployed CNI in production Kubernetes, with adoption up 47% year over year and a presence in more than 60% of surveyed deployments (over 75% once managed dataplanes like GKE Dataplane V2 are included), according to the Cilium Annual Report 2025 (CNCF, Dec 2025). Dataplane behaviour is not a niche concern.

Why did a fresh pod work when the old one stayed blocked?

The isolating test was a second pod. A netshoot debug container, launched after the policy deletion, reached MongoDB immediately. The original issuer pod, created before the deletion, still could not. Same namespace, same labels, same destination. The only difference was lifecycle. That split points at state, not routing.

This is the diagnostic fork that changes everything. If the network path itself were broken, a new pod would fail too. If DNS or the database were down, both pods would fail. Only the old pod failing, while a fresh one succeeds, narrows the cause to something attached to the existing endpoint that a new endpoint never inherits.

We confirmed it with the bluntest possible test. Restarting the issuer pod restored connectivity at once, with no other change. No config edit, no policy re-apply, no node action. Delete the pod, let the scheduler recreate it, traffic flows.

Three observations, one conclusion:

  • New pod created after deletion: works.
  • Old pod created before deletion: still blocked.
  • Old pod restarted: works.

In our experience, this exact triad, new works, old fails, restart fixes, is the fingerprint of stale dataplane state. It shows up with conntrack entries, with service backend maps, and here with eBPF policy maps. When a restart is the only thing that helps, stop suspecting the policy and start suspecting the map that enforces it.

What was the root cause?

The root cause was a stale per-endpoint eBPF policy map on the long-lived endpoint. Cilium, which backs GKE Dataplane V2, keeps a policy map per endpoint. Deleting a NetworkPolicy should trigger the control plane to recompute and flush the affected maps. For the existing endpoint, that recomputation did not land, so the deny rule stayed active in the map.

Here is the mechanism, step by step. Cilium’s control plane translates high-level NetworkPolicy objects into concrete allow and deny entries inside each endpoint’s BPF policy map. Enforcement happens in the kernel, against that map, not against the API object. So the map is the source of truth at packet time. When policy changes, the map has to be recomputed, or the kernel keeps enforcing the last state it was handed.

For the running pod, the map was never recomputed after the deletion. It kept a default-deny egress entry that no longer had any backing policy. A newly created endpoint builds its map from the current desired state, which by then had no restriction, so it came up open. The restart worked because it destroyed the stale endpoint and created a fresh one.

The important reframing: on an eBPF dataplane, kubectl delete netpol is a request to converge, not a guarantee that convergence happened on every endpoint. Most guides treat policy deletion as instant and universal. It is neither when the control-plane-to-map reconciliation misses a long-lived endpoint — the same class of cache-staleness failure that quietly bites Kubernetes controllers when a watch or a work queue falls behind desired state.

How do you reproduce the stale eBPF state?

The failure is reproducible on an affected Dataplane V2 cluster in seven steps, and the whole loop takes a few minutes. The key is to compare the same, already-running pod against a fresh pod after you delete the policy. That comparison is what turns an ambiguous “it’s still blocked” into a clear stale-state signal.

Run this against a non-production namespace first. You need one target pod, one egress-restricting policy, and a second pod to use as your control.

  1. Create a namespace and a pod with a known label, for example app=probe.
  2. Apply a default-deny egress NetworkPolicy targeting that pod.
  3. From the pod, confirm egress is blocked. This is the expected baseline.
  4. Delete the NetworkPolicy.
  5. From the same, already-running pod, retry egress. Observe it is still blocked.
  6. Create a new pod in the same namespace with the same labels. Confirm its egress is not blocked.
  7. Restart the original pod. Confirm its egress now works too.
# 2. default-deny egress on the target
kubectl -n probe-ns apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-egress
spec:
  podSelector:
    matchLabels:
      app: probe
  policyTypes:
    - Egress
EOF

# 4. remove it
kubectl -n probe-ns delete networkpolicy deny-egress

# 5. same running pod, still blocked
kubectl -n probe-ns exec deploy/probe -- \
  timeout 5 curl -sS https://example.com || echo "STILL BLOCKED"

# 6. fresh pod, works immediately
kubectl -n probe-ns run probe-new --image=nicolaka/netshoot \
  --labels app=probe --restart=Never -- \
  timeout 5 curl -sS https://example.com

If step 5 stays blocked while step 6 succeeds, you are looking at the stale-map bug, not a live policy.

Which commands expose stale BPF policy maps?

Three Cilium commands turn the hypothesis into evidence, and they run from inside the Cilium agent pod on the node hosting the stuck endpoint. Together they show the endpoint’s computed policy, the raw map contents the kernel enforces, and the live drops. That evidence is exactly what a Google Cloud support ticket needs.

Find the agent pod on the affected node, exec into it, then work from the endpoint ID.

# locate the cilium agent on the node running the stuck pod
kubectl -n kube-system get pods -l k8s-app=cilium -o wide

# inside the agent pod:

# a. the endpoint's computed policy state
cilium endpoint get <endpoint-id>

# b. the raw eBPF policy map the kernel actually enforces
#    the stale deny entry shows here after the policy is gone
cilium bpf policy get <endpoint-id>

# c. live drops attributable to the stale rule
cilium monitor --type drop

Read them in order. cilium endpoint get shows what the control plane thinks the endpoint should enforce. cilium bpf policy get shows what the kernel is enforcing: after deletion, a lingering deny entry here is the smoking gun. cilium monitor --type drop captures the drops in real time so you can tie a specific timed-out connection to the stale map.

Gotcha: Compare cilium bpf policy get between the stuck pod and the fresh pod. If the running endpoint carries a deny entry the new endpoint lacks, you have proof the divergence is in the map, not in any surviving policy object. Capture both dumps to a timestamped file before you restart anything, because the restart destroys your evidence.

Hubble helps too. Flow logs from the affected pod show the egress attempts terminating as policy drops, which corroborates the map dump with an application-level view.

What other Dataplane V2 limitations should you plan for?

Stale policy state is one item on a longer list of Dataplane V2 constraints worth designing around. Google Cloud’s own GKE documentation flags several sharp edges that surprise teams migrating from a legacy CNI. Knowing them upfront prevents the “we deployed it and it silently did nothing” class of incident that is far harder to debug after the fact.

The documented limitations that bite most often:

  • No L7 CiliumNetworkPolicy. Dataplane V2 supports L3/L4 policy only. L7 rules in a CiliumNetworkPolicy are not enforced (Google Cloud GKE docs, 2026).
  • endPort silently no-ops. A NetworkPolicy using the endPort range field is accepted by the API but not enforced on Dataplane V2, so a port range you think is open or closed may not be.
  • CRD scaling ceiling. CiliumNetworkPolicy (the namespaced CRD) does not scale past 5,000 nodes. Above that, only CiliumClusterwideNetworkPolicy is supported.
  • Agent CPU under churn. The anetd agent can burn 2 to 3 vCPUs per node under high TCP connection churn, which shows up as node-level CPU pressure that is easy to misattribute.
  • Fragmented ICMP dropped. Fragmented ICMP packets are dropped, which can quietly break path-MTU discovery and some diagnostics.

There is also a community-reported bug worth watching: network policies can stop applying on preemptible nodes (cilium/cilium#17446, GitHub). If your cluster mixes spot and on-demand capacity, that is another lifecycle-linked policy gap to test for. The lesson generalises to the rest of your GCP network design: treat unusual GKE and Private Service Connect topologies as things to validate under load, not to assume work.

GKE Dataplane V2 enforces L3/L4 network policy only, silently ignores the NetworkPolicy endPort field, and limits the namespaced CiliumNetworkPolicy CRD to clusters of 5,000 nodes or fewer, per Google Cloud’s GKE documentation (2026). Treat unsupported policy features as no-ops, not as errors you’ll be warned about.

The direction of travel is positive. Cilium v1.19, released in 2026, hardens policy defaults and improves large-cluster visibility (InfoQ, Feb 2026), which targets exactly the class of silent-enforcement and scale problems above.

How do you mitigate and detect this in production?

Mitigation splits into two moves: force convergence for the immediate incident, and add detection so you catch divergence before users do. The immediate fix is the one we already proved. Restarting the affected pod rebuilds its endpoint and its eBPF map from current state. It is crude, but it is reliable and fast.

For the incident in front of you:

  • Restart the affected workload after any policy change that “didn’t take”. A rollout restart rebuilds every endpoint’s map from the current desired state.
  • Capture evidence first. Dump cilium endpoint get, cilium bpf policy get, and a short cilium monitor --type drop to a timestamped file before the restart, so you can open a support ticket with real data.

For detection and prevention over time:

  • Treat policy changes as needing verification, not just application. After a delete or edit, confirm the affected endpoints actually reflect it. Do not assume kubectl delete reached the kernel on every node.
  • Alert on the fingerprint. A workload reporting connection timeouts to a destination that has no surviving policy is a strong signal. Pair Hubble drop flows with the absence of a matching policy object.
  • Prefer redeploys over in-place policy edits for long-lived, connection-sensitive workloads until you are on a Cilium version that resolves the recompute gap.

We’ve found that the cheapest durable guardrail is a runbook line: “if a policy change didn’t apply, restart the pod and file the eBPF dump.” It converts a multi-hour head-scratch into a two-minute recovery, and it preserves the evidence the platform vendor needs to fix the root cause.

The takeaway

A deleted NetworkPolicy is a request for the dataplane to converge, not proof that it did. On GKE Dataplane V2, a long-lived pod kept enforcing a default-deny egress rule from an eBPF map that was never recomputed after the policy vanished. The fingerprint was unmistakable once we saw it: new pod works, old pod blocked, restart fixes it.

Carry three habits out of this. First, when a policy change “doesn’t apply”, suspect the map before the policy, and prove it with a fresh-pod split test. Second, capture cilium bpf policy get and drop logs before you restart, because the restart erases your evidence. Third, design around Dataplane V2’s documented limits so unsupported features fail loudly in review, not silently in production.

The dataplane is where enforcement actually lives. Watch it there, the same way you would watch IP address management across a multi-tenant Kubernetes fleet: at the layer that decides the outcome, not the layer that merely describes it.

Straight answers

Frequently asked questions

Does deleting a NetworkPolicy always restore traffic on GKE Dataplane V2?

Not reliably for already-running pods. On an affected cluster, the endpoint's eBPF policy map can keep enforcing a deleted rule until the pod restarts. New pods created after the deletion build their maps from current state and connect fine, which is the clearest sign the problem is stale state.

How do I confirm the block is stale eBPF state and not a live policy?

Run the split test. Create a fresh pod with the same labels in the same namespace after deleting the policy. If the new pod connects but the original stays blocked, and a restart fixes the original, the cause is a stale endpoint map. Confirm with cilium bpf policy get on both endpoints.

Why does restarting the pod fix it?

A restart destroys the old Cilium endpoint and creates a new one. The new endpoint computes its eBPF policy map from the current desired state, which no longer contains the deleted rule, so it comes up unrestricted. The stale map that enforced the phantom deny is discarded with the old endpoint.

Is this a Cilium bug or a GKE bug?

It is a dataplane-level behaviour in the Cilium-backed GKE Dataplane V2 stack, not an application or controller fault. Cilium adoption grew 47% year over year (Cilium Annual Report 2025, CNCF, Dec 2025), so the pattern affects a large installed base. Cilium v1.19 (InfoQ, Feb 2026) hardens policy handling.