Kubernetes v1.37: Scale to Zero Is On by Default, So Read the Upgrade Notes First

Kubernetes v1.37, named Garhwal after a Himalayan region of Uttarakhand, India, shipped on August 26 with 67 enhancements: 16 graduating to stable, 23 to beta, 27 new alphas and one deprecation or removal. Reading the release announcement next to the v1.37 changelog, what stands out is not one headline feature but how many things moved from opt-in to on by default. That shifts the cost and security baseline of a cluster the moment you upgrade.

HPA scale to zero is finally on#

HorizontalPodAutoscaler scale to zero, first introduced in v1.16, graduates to beta in v1.37 and is enabled by default. Set spec.minReplicas: 0 and the HPA can take a workload down to zero pods when idle, then restore it when demand returns.

The constraint matters: it only works with object or external metrics. CPU and memory are not supported, because those metrics depend on active pods. So this is for queue consumers, batch jobs and GPU workloads, where "is there work waiting?" can be answered from outside the pods.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: invoice-worker
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: invoice-worker
  minReplicas: 0
  maxReplicas: 10
  metrics:
    - type: External
      external:
        metric:
          name: queue_messages_ready
        target:
          type: AverageValue
          averageValue: "20"

While the HPA holds a workload at zero, it records a ScaledToZero condition set to True; after scaling back up it becomes False with reason NotScaledToZero. That is how the controller tells its own scale-down apart from someone manually setting replicas to 0. Any alert or runbook that treats "0 replicas" as "broken" should read that condition now.

On the node: rootless kubelet and memory QoS#

The KubeletInUserNamespace feature gate is promoted to beta. The documentation describes it as running node components such as kubelet, CRI, OCI and CNI without root privileges inside a user namespace, also known as rootless mode, and lists cgroup v2 as a prerequisite.

Memory QoS also reaches beta, with the MemoryQoS gate enabled by default. It maps requests and limits onto cgroup v2 controls like memory.min, memory.low and memory.high. memoryThrottlingFactor now defaults to nil and memory.high is not set unless explicitly configured, which the release post says is meant to avoid surprise throttling on upgrade.

The common thread is cgroup v2. Since v1.35 the kubelet fails to initialize on cgroup v1 nodes unless you set failCgroupV1: false, and the release post says memory QoS only works on cgroup v2. If that override is still in your node config, plan its removal.

A quieter control plane, fewer manual scripts#

The metrics.k8s.io API graduates to stable after nearly nine years in beta. It was promoted to v1 without changes, v1beta1 stays usable during the transition, and kubectl top gains v1 support in this release.

Storage Version Migration goes stable and on by default (storagemigration.k8s.io/v1). Instead of kubectl get and kubectl replace scripts or the out-of-tree migrator, you create a StorageVersionMigration object and a built-in controller rewrites stored resources. It also covers rewriting data after an encryption-at-rest change, and CRD authors can trigger it as part of a CRD upgrade.

For large clusters, EtcdRangeStream ships as beta and on by default in kube-apiserver. It needs etcd 3.7+ and falls back to the old behaviour automatically on older etcd. Combined with concurrent watch decode, also now on by default, the project reports about 55% less cache initialization time in benchmarks over 150k pods. The tradeoff: up to 10 conversions can now run concurrently against a conversion webhook during cache init.

Read the upgrade notes first#

From the urgent upgrade notes and the deprecations, this is what I would put in front of a team before anyone touches production:

• SELinuxMount is GA and on by default. On SELinux-enabled clusters, pods with different SELinux labels sharing a volume on the same node can fail to start when the CSI driver opts in. seLinuxChangePolicy: Recursive on a pod keeps the old behaviour.
• The scheduling.k8s.io v1alpha2 API is dropped; remove those objects before upgrading.
• kubelet eventRecordQPS: 0 now means unlimited, and the kubelet refuses to start if deprecated cAdvisor flags are set.
• Static pods can no longer reference Secrets or ConfigMaps.
• kube-proxy ipvs mode logs a deprecation warning, is expected to be off by default by v1.40 and removed by v1.43. kube-dns is deprecated in favour of CoreDNS.
• While its watch cache warms up, kube-apiserver can reject requests with HTTP 429. Custom controllers should respect Retry-After and back off.

What I'd do on Monday#

Some of what I ship runs on a plain VPS with systemd and nginx, not Kubernetes, so this is the order I would hand a platform team:

1. Diff kubelet configs against the urgent notes and the list of removed cAdvisor flags.
2. Confirm every node runs cgroup v2 and list any failCgroupV1: false overrides.
3. Check the kube-proxy mode and schedule the move off ipvs.
4. Pilot scale to zero on one queue consumer where a cold start is acceptable, not on anything user-facing.
5. Try rootless kubelet on a dev or CI cluster before any production node pool.

Re-Shell generates Docker and Kubernetes scaffolds, and the temptation will be to make minReplicas: 0 a template default. I wouldn't. Scale to zero is a per-workload decision about latency and cost, and a beta cycle is the right time to learn where it hurts.