Running one Kubernetes cluster is a solved problem. Running eighty of them is not. Each one needs the same twenty-two addons, at slightly different versions. So this is where most platform teams start to hurt. According to Portainer’s Kubernetes Fleet Management 2026 Guide, 48% of specialists expect their cluster count to grow more than 50% within a year. Another 28% expect 20-50% growth. At that pace, addon delivery stops being a scripting chore. Instead, it becomes a governance problem. This article walks through one concrete pattern. You group clusters into a fleet hierarchy, then promote addons through it stage by stage with Sveltos. The lessons come from a major European automotive manufacturer’s internal Kubernetes platform.

Key takeaways

  • A fleet hierarchy (value-stream, criticality, scope, environment) lets you target addon rollouts by query instead of cluster-by-cluster.
  • Sveltos ClusterProfile label selectors plus dependsOn chains give you ordered, staged promotion from dev to production tiers.
  • Per-cluster annotations act as an auditable kill switch for a single addon without touching fleet-wide rollout labels.
  • GKE rejects any maintenance window shorter than 4 contiguous hours, so derive one window and project it into every subsystem.
  • 93% of enterprise platform teams report major cloud-cost challenges (Portainer 2026), which staged rollout helps contain.

Why does fleet-scale addon management break down?

Fleet-scale addon delivery breaks down because both obvious strategies fail. Portainer’s 2026 guide reports 42% GitOps adoption and 35% multi-cloud adoption. Yet most teams still push addons one of two ways: all-at-once, or cluster-by-cluster. All-at-once bets every cluster on a single chart. In contrast, cluster-by-cluster is slow, manual, and drifts.

The middle path is progressive rollout. But progressive rollout needs structure. Say you want to ship Kyverno version N+1 to your least critical dev clusters first. You cannot, unless those clusters already carry labels for criticality and environment. Without that taxonomy, every promotion decision is a human reading a spreadsheet. In our experience, skipping that structure ends badly. In our fleet, we discovered the cost the day a policy engine upgrade broke admission on a production cluster. That cluster should never have been in the first wave. As a result, we stopped promoting anything by hand and built the hierarchy that follows.

Portainer’s guide frames why this matters: with 48% of specialists expecting cluster counts to grow more than 50% within a year, and only 42% on GitOps and 35% multi-cloud, manual per-cluster addon delivery becomes unsustainable for platform teams (Portainer, 2026). Structure like this is the foundation of any internal platform; we cover the broader control-plane design in our guide to building Kubernetes as a service.

What is a fleet-based cluster hierarchy?

A fleet-based cluster hierarchy is a nested grouping of clusters expressed as labels. Any subset can then be targeted as a single query. Instead of flat tags, you nest levels: value stream, then criticality, then cluster type, then scope, then environment. Each cluster belongs to exactly one value-stream fleet. And that fleet nests child fleets underneath.

The hierarchy levels

In practice, the levels read top to bottom. Value stream maps to a business unit. Criticality is Standard, Critical, or Strategic. Cluster type distinguishes standard from autopilot nodes. Scope separates Workload, Control Plane, and Edge. Finally, environment is Dev, Int, or Ope. A single cluster carries all of these as labels on its SveltosCluster object, plus one label per enabled addon.

This layering complements GKE Enterprise fleet primitives rather than replacing them. Google gives you Scopes, Team Scopes, Fleet Namespaces, and Rollout Sequencing for parent-to-child update ordering. On top of that, the platform team layers its own value-stream and criticality taxonomy. Why bother? Because business criticality is not something GKE’s grouping model knows about. And for enterprise fleets spanning European and North American regions, that business taxonomy decides rollout order, not geography alone.

Registering a cluster into the fleet

Before any labeling matters, a cluster has to join the fleet. First, once its infrastructure is ready, a controller locates or creates a kubeconfig secret. Then it applies bootstrap RBAC directly into the workload cluster. That RBAC stays minimal: a projectsveltos namespace, a service account, and a cluster role scoped to the kinds addons touch. Finally, the controller creates the SveltosCluster object in the management cluster. Teardown runs this sequence in reverse. Get the order wrong and you corrupt fleet state. Specifically, that is one of the most common failures at scale, and we will come back to it.

Under the hood, Sveltos targets clusters using label-based ClusterProfile selectors and supports ordered deployment with dependency chains and Secure Pull Mode, and is now integrated into the CNCF k0rdent project maintained by Mirantis (projectsveltos.io, 2026). Those primitives are what make a nested fleet hierarchy queryable and safe to roll out against.

How does staged promotion move addons across fleet tiers?

Staged promotion moves an addon version through fleet tiers in waves. It widens the blast radius only as confidence grows. You express each wave as a Sveltos ClusterProfile whose selector matches one tier. Then you chain profiles with dependsOn, so a later tier never deploys until its predecessor is healthy.

Cluster API v1.12, released January 2026, added in-place updates and chained multi-version upgrades that complement this addon-level staging (Cluster API Book, 2026).

ClusterProfile with dependencies

The mechanism is two profiles and one dependency edge. The dev-tier profile selects clusters labeled for the least critical environment. The staging profile depends on it. So Sveltos deploys the dependency first, then only proceeds when it reports healthy.

apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  name: kyverno-dev
spec:
  clusterSelector:
    matchExpressions:
      - key: platform.example.io/criticality
        operator: In
        values: ["Standard"]
      - key: platform.example.io/environment
        operator: In
        values: ["Dev"]
  helmCharts:
    - repositoryURL: https://kyverno.github.io/kyverno
      chartName: kyverno/kyverno
      chartVersion: "3.2.6"
      releaseName: kyverno
      releaseNamespace: kyverno
---
apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  name: kyverno-staging
spec:
  dependsOn:
    - kyverno-dev
  clusterSelector:
    matchExpressions:
      - key: platform.example.io/criticality
        operator: In
        values: ["Standard"]
      - key: platform.example.io/environment
        operator: In
        values: ["Int"]
  helmCharts:
    - repositoryURL: https://kyverno.github.io/kyverno
      chartName: kyverno/kyverno
      chartVersion: "3.2.6"
      releaseName: kyverno
      releaseNamespace: kyverno

To promote, you bump the chart version on the dev profile. You watch it land, then bump staging, then production. Each tier is a separate git commit. As a result, you get a clean audit trail and a natural rollback point. For example, say you run 40 clusters across three tiers. A bad chart now hits only your handful of dev clusters, not all 40 at once. This is the hub-and-spoke, agent-based GitOps pattern that ITNEXT documented as an emerging approach with Sveltos in April 2026.

How do you toggle a single addon on one cluster?

You toggle a single addon per cluster with an annotation, not by editing fleet labels. The pattern is addons.example.io/enable-<addon-name>: "false" on the cluster resource. A webhook validates it. Every default addon supports it. And the annotation only ever disables, so it works as a fast, auditable kill switch for one misbehaving addon on one cluster.

The activation annotation

The distinction matters. Fleet labels decide which tier gets which addon version. The annotation decides whether one specific cluster gets a given addon at all. Two different axes. So never confuse them.

apiVersion: platform.example.io/v1alpha1
kind: StandardCluster
metadata:
  name: vs1-workload-ope-03
  annotations:
    # Disable a single addon on this cluster only
    addons.example.io/enable-otel-collector: "false"
    # Pause all Sveltos reconciliation for manual debugging
    sveltos.example.io/pause: "true"
spec:
  # ...

A webhook rejects any annotation that names an unknown addon or carries a non-boolean value. The platform also keeps this control operator-only on purpose. The public API cannot set these annotations. And the server explicitly rejects them, even though the request shape cannot carry them today. Why bother? It is belt-and-braces against a future change that accidentally turns an operator control into an end-user one. The sveltos.example.io/pause annotation is a separate escape hatch. It stops reconciliation entirely, so an engineer can debug directly. Then you unpause to resume GitOps.

Gotcha: Do not use the pause annotation as a long-lived config mechanism. A paused cluster silently drifts from the fleet baseline, and the longer it stays paused, the more surprising the reconciliation becomes when you unpause it. Treat pause as a debugging session, measured in minutes, not days.

How do you coordinate maintenance windows across layers?

You coordinate maintenance windows by deriving one logical window from a single source of truth. Then you project it into every subsystem. A cluster runs GKE’s own maintenance policy, Sveltos’s ActiveWindow, and an autoscaler’s rebalance schedule. Each uses a different format. Configure them separately and they drift. So derive once, project three times.

The GKE 4-hour minimum gotcha

This is the failure that teaches the lesson. In our fleet, our early default used a two-hour window, 02:00 to 04:00 Paris time. We expected no trouble. However, GKE rejected it outright in production.

Error 400: Error validating maintenance policy: maintenance policy would go longer
than 32d without 48h maintenance availability of >= 4h contiguous duration

GKE requires every recurring maintenance window to be at least four contiguous hours. That holds regardless of cumulative monthly hours. So the fix moved the default to 02:00 to 06:00 Paris time. More important, it added API-server validation that rejects any window shorter than four hours at request time. That converts a slow runtime failure loop into an immediate, readable error.

Gotcha: Validate against the strictest downstream consumer at the API boundary, not just the format level. An RFC3339 timestamp can be perfectly well-formed and still describe a window GKE will refuse. The duration check has to live where the request enters, not where it eventually fails.

There is a second trap here. It took us three attempts to get this one right. Say you enforce a Sveltos ActiveWindow on a brand-new cluster, and creation happens outside the window. Sveltos then refuses to deploy addons, and the cluster never reaches ready. Our first fix was too fragile, and the second still raced on a second reconcile. Therefore the fix that stuck has two parts. First, only set the window once every addon reports provisioned. Second, make the window opt-in per cluster rather than on by default. Most clusters do not need the constraint. Meanwhile, a silent default reintroduces the deadlock for anyone who does not know to disable it.

How do you surface addon status across the fleet?

You surface addon status by reading the per-cluster deployment records Sveltos already writes. For every profile-and-cluster pair, Sveltos creates a ClusterSummary. It holds the deployment method and per-resource status. Reading those directly needs cluster access. So the platform exposes a typed API instead. It lists addons, resolves their type, and reconstructs the dependency graph for a fleet-health view.

Getting that read model right depends on the same cache-freshness discipline that keeps Kubernetes controllers reliable: the status you serve is only as trustworthy as the last reconcile you observed.

Reading ClusterSummary correctly

Two details save you from confusing output. First, the display name for a Helm addon should come from the chart name. Do not use the ClusterProfile name, which is often a generated hash like falco-install-4fcf3b5d. Second, the dependency graph must key on the profile name, not the display name, because dependsOn edges reference the stable identifier. Get either wrong, and your status view shows the right data under the wrong labels.

So this read model turns “I hope the rollout worked” into something concrete: “the staging tier shows all addons provisioned, promote to production.” In addition, for teams operating across Europe and North America, a single fleet-health recap removes the need to reason about each regional cluster by hand.

What breaks in production, and how do you avoid it?

The two failures that cost the most time are name collisions and unsafe teardown. Neither shows up in a demo. Both show up at scale. And with 93% of enterprise platform teams reporting major cloud-cost challenges (Portainer 2026), the last thing you want is orphaned addon state quietly burning resources after a botched deletion.

The name-length collision

Sveltos builds a ClusterSummary name by appending the cluster name to the profile name. So if your event-trigger template also embeds the cluster name, you duplicate it. The combined string can then exceed Kubernetes’s 63-character label limit. We discovered this in production the day a longer cluster name pushed a previously fine profile over the limit. The error is blunt: metadata.labels: Invalid value ... must be no more than 63 characters. The fix is simple once you know it. Specifically, never put the cluster name in your profile-name template, because Sveltos already adds it once.

Safe SveltosCluster teardown

Deleting a SveltosCluster outright, or wiping all its labels, can corrupt Sveltos state and strand addons. We found this out on a real decommission, where the label-strip approach is what saved us from orphaned addons. The safe sequence has four steps. First, strip only the labels that drive profile matching. Then wait for every ClusterSummary for that cluster to disappear, as Sveltos withdraws each addon. In practice, that takes up to about five minutes. If any finalizer is still stuck after the timeout, force-remove it. Only then delete the SveltosCluster. Removing labels first, rather than deleting the object, lets the selectors stop matching naturally. As a result, Sveltos runs its own graceful withdrawal path.

Sveltos supports ordered deploys with dependency chains and Secure Pull Mode, and is integrated into CNCF’s k0rdent project (projectsveltos.io, 2026). Those same dependency edges that stage a rollout also govern teardown order, so removing selector labels lets Sveltos withdraw addons gracefully instead of orphaning them.

Conclusion

Fleet-scale addon management is really a labeling and ordering problem wearing a Kubernetes costume. Once clusters carry a nested hierarchy of value stream, criticality, scope, and environment, everything changes. “Promote this addon” becomes a query you widen deliberately, not a spreadsheet you maintain by hand. Sveltos ClusterProfile selectors and dependsOn chains give you the ordering. Per-cluster annotations give you the kill switch. And a single derived maintenance window keeps GKE, Sveltos, and the autoscaler in agreement. With 48% of teams expecting clusters to grow past 50% in a year (Portainer, 2026), the teams that build this structure early stay in control as the fleet multiplies. For the wider practice this sits inside, see our guide to platform engineering at scale. So start with the hierarchy. Then let staged promotion do the rest.

Straight answers

Frequently asked questions

Do I need Cluster API to use Sveltos for fleet addons?

No. Sveltos targets any registered cluster through label-based selectors, whether it came from Cluster API, GKE, or manual registration. Cluster API v1.12, released January 2026, adds in-place and chained upgrades that pair well with addon staging (Cluster API Book, 2026). But Sveltos itself only needs a SveltosCluster object carrying the right labels.

How is staged promotion different from a canary deployment?

Canary deployments split traffic within one cluster to test a new app version. Staged promotion instead moves an addon across cluster tiers, dev to staging to production. It uses dependsOn chains, so a later tier waits for an earlier one to report healthy. One tests app code on live users; the other tests platform addons on progressively more critical clusters.

What happens if an addon fails midway through promotion?

Because each tier is a separate ClusterProfile gated by dependsOn, a failure in an early tier stops later tiers from deploying. The healthy tiers keep their working version. So you fix the chart, commit, and the pipeline resumes from the failed tier forward. A bad version never reaches production clusters automatically.

Can I mix GKE and other providers in one fleet?

Yes, and enterprise fleets across Europe and North America routinely do. Sveltos abstracts the cluster behind its SveltosCluster label set, so a ClusterProfile selector matches clusters regardless of provider. The maintenance-window projection is provider-specific, but the addon-promotion logic stays identical across GKE, other managed offerings, and self-managed clusters.