Temporal Worker Controller

The Temporal Worker Controller provides automation to enable rainbow deployments of your Workers by simplifying the tracking of which versions still have active Workflows, managing the lifecycle of versioned Worker deployments, and calling Temporal APIs to update the routing config of Temporal Worker Deployments. The Temporal Worker Controller makes it simple and safe to deploy Temporal Workers on Kubernetes.

Why adopt the Worker Controller?

The traditional approach to revising Temporal Workflows is to add branches using the Versioning APIs. Over time these checks can become a source of technical debt, as safely removing them from a codebase is a careful process that often involves querying all running Workflows.

Worker Versioning is a Temporal feature that allows you to pin Workflows to individual versions of your Workers, which are called Worker Deployment Versions. Using pinning, you will not need to add branching to your Workflows to avoid non-determinism errors. This allows you to bypass the other Versioning APIs.

The Worker Controller gives you direct, programmatic control over your Worker deployments, and integrates with the Temporal CLI. You do not need to use the Worker Controller to use Worker Versioning, but when used together, Worker Versioning and the Worker Controller can provide more graceful deployments and upgrades, and less need to manually tune your Workers.

Note that in Temporal, Worker Deployment is sometimes referred to as Deployment, but since the Worker Controller makes significant references to Kubernetes Deployment resource, within this page we will stick to these terms:

• Worker Deployment: A Worker Deployment is a logical service that groups similar Workers together for unified management. Each Deployment has a name (such as your service name) and supports versioning through a series of Worker Deployment Versions.
• Worker Deployment Version: A Worker Deployment Version represents an iteration of a Worker Deployment. Each Deployment Version consists of Workers that share the same code build and environment. When a Worker starts polling for Workflow and Activity Tasks, it reports its Deployment Version to the Temporal Server.
• Deployment: A Kubernetes Deployment resource. A Deployment is "versioned" if it is running versioned Temporal workers/pollers.

Features

• Registration of new Temporal Worker Deployment Versions
• Creation of versioned Deployment resources (that manage the Pods that run your Temporal pollers)
• Deletion of resources associated with drained Worker Deployment Versions
• Manual, AllAtOnce, and Progressive rollouts of new versions
• Ability to specify a "gate" Workflow that must succeed on the new version before routing real traffic to that version
• Autoscaling of versioned Deployments

Refer to the Temporal Worker Controller repo for usage details.

Configuring Worker Lifecycles

To use the Temporal Worker Controller, tag your Workers following the guidance for using Worker Versioning.

Here is an example of a progressive rollout strategy gated on the success of the HelloWorld Workflow:

rollout:
  strategy: Progressive
  steps:
    - rampPercentage: 1
      pauseDuration: 30s
    - rampPercentage: 10
      pauseDuration: 1m
  gate:
    workflowType: "HelloWorld"

As you ship new deployment versions, the Worker Controller automatically detects them and gradually makes that version the new Current Version of the Worker deployment it is a part of. As older pinned Workflows finish executing and deprecated deployment versions become drained, the Worker Controller also frees up resources by sunsetting the Deployment resources polling those versions.

Running the Temporal Worker Controller

You can install the Temporal Worker Controller using our Helm chart:

RELEASE=temporal-worker-controller
NAMESPACE=temporal-system
VERSION=1.0.0

helm install $RELEASE oci://docker.io/temporalio/helm-charts/temporal-worker-controller \
  --version $VERSION \
  --namespace $NAMESPACE \
  --create-namespace
  
helm install temporal-worker-controller ./helm/temporal-worker-controller \
  --namespace $NAMESPACE \
  --create-namespace 

Refer to GitHub for other Worker Controller deployment templates.

Per-version autoscaling

You can perform autoscaling by attaching Kubernetes resources like HPAs, PodDisruptionBudgets, or other custom scalers to each versioned Deployment via the WorkerResourceTemplate.

The Worker Controller creates one Kubernetes Deployment per Worker version or Build ID. If you attach an HPA directly to a single Deployment, for example, it will break as the versions roll over. For example, the old HPA may still target the old Deployment, the new Deployment has no HPA, and you have to manage clean up.

WorkerResourceTemplate solves this by treating the Kubernetes resource as a template. The controller renders one instance per Worker version with running Workers, injects the correct versioned Deployment name, and cleans up automatically when the versioned Deployment is deleted.

tip

This is also the recommended mechanism for metric-based or backlog-based autoscaling: attach a standard HPA with custom metrics to your Workers and the controller keeps one per running Worker version, each pointing to the correct Deployment.

How it works

1. You create a WorkerResourceTemplate that references a TemporalWorkerDeployment and contains the resource spec in spec.template.
2. The validating webhook checks that you have permission to manage that resource type yourself and that the resource kind is on the allowed list.
3. On each reconcile loop, the controller renders one copy of spec.template per Worker version with a running Deployment, injects fields, and applies it via Server-Side Apply.
4. Each copy is owned by the corresponding versioned Deployment, so it's garbage-collected automatically when that Deployment is deleted.
5. WorkerResourceTemplate.status.versions is updated with the applied or failed status for each version.

Auto-injection

The controller auto-injects two fields when you set them to null in spec.template. Setting them to null is the explicit signal that you want injection. If you omit the field entirely, nothing is injected. If you set a non-null value, the webhook rejects the WorkerResourceTemplate because the controller owns these fields.

Field	Injected value
spec.scaleTargetRef (any resource with this field)	{apiVersion: apps/v1, kind: Deployment, name: <versioned-deployment-name>}
spec.selector.matchLabels (any resource with this field)	{temporal.io/build-id: <buildID>, temporal.io/deployment-name: <twdName>}

The scaleTargetRef injection applies to any resource type that has a scaleTargetRef field, not just HPAs. Other autoscaler custom resource definitions (CRDs) use the same field and benefit from the same injection.

Resource naming

Each per-Build-ID copy is given a unique, DNS-safe name derived from the (twdName, wrtName, buildID) triple. Names are capped at 47 characters to be safe for all Kubernetes resource types, including Deployment which has pod-naming constraints that effectively limit Deployment names to ~47 characters. The name always ends with an 8-character hash of the full triple, so uniqueness is guaranteed even when the human-readable prefix is truncated.

Use kubectl get <kind> after a reconcile to see the created resources and their names.

Allowed resource kinds and RBAC

The workerResourceTemplate.allowedResources value in Helm serves two purposes: it defines which resource kinds the webhook will accept and it drives the RBAC rules granted to the controller's ClusterRole. Only kinds listed here can be embedded in a WorkerResourceTemplate.

The default allows HPAs:

workerResourceTemplate:
  allowedResources:
    - kinds: ["HorizontalPodAutoscaler"]
      apiGroups: ["autoscaling"]
      resources: ["horizontalpodautoscalers"]

To also allow PodDisruptionBudgets, add an entry:

workerResourceTemplate:
  allowedResources:
    - kinds: ["HorizontalPodAutoscaler"]
      apiGroups: ["autoscaling"]
      resources: ["horizontalpodautoscalers"]
    - kinds: ["PodDisruptionBudget"]
      apiGroups: ["policy"]
      resources: ["poddisruptionbudgets"]

Each entry has three fields:

• kinds — kind names the webhook accepts (case-insensitive)
• apiGroups — API groups used to generate the controller's RBAC rules
• resources — resource names used to generate the controller's RBAC rules

What the webhook checks

When you create or update a WorkerResourceTemplate, the webhook performs SubjectAccessReviews to verify:

1. You (the requesting user) can create and update the embedded resource type in that Namespace.
2. The controller's service account can create and update the embedded resource type in that Namespace.

If either check fails, the request is rejected. This prevents privilege escalation so you can't use WorkerResourceTemplate to create resources you don't already have permission to create yourself.

Users who create WorkerResourceTemplates need RBAC permission to manage the embedded resource type directly. For example, to let a team create WorkerResourceTemplates that embed HPAs, they need the standard autoscaling permissions in their Namespace.

Webhook TLS

The WorkerResourceTemplate validating webhook requires TLS. The Helm chart uses cert-manager to provision the certificate (certmanager.enabled: true is the default).

If cert-manager is not already installed in your cluster, you can either install it separately (cert-manager installation docs) or let the Helm chart install it as a subchart by setting certmanager.install: true.

Example: HPA per worker version

apiVersion: temporal.io/v1alpha1
kind: WorkerResourceTemplate
metadata:
  name: my-worker-hpa
  namespace: my-namespace
spec:
  # Reference the TemporalWorkerDeployment to attach to.
  temporalWorkerDeploymentRef:
    name: my-worker

  # The resource template. The controller creates one copy per worker version
  # with a running Deployment.
  template:
    apiVersion: autoscaling/v2
    kind: HorizontalPodAutoscaler
    spec:
      # {} tells the controller to auto-inject the versioned Deployment reference.
      # Do not set this to a real value — the webhook will reject it.
      scaleTargetRef: {}
      minReplicas: 2
      maxReplicas: 10
      metrics:
        - type: Resource
          resource:
            name: cpu
            target:
              type: Utilization
              averageUtilization: 70

See the temporal-worker-controller wrt-hpa.yaml for an example pre-configured for the helloworld demo.

Example: PodDisruptionBudget per worker version

apiVersion: temporal.io/v1alpha1
kind: WorkerResourceTemplate
metadata:
  name: my-worker-pdb
  namespace: my-namespace
spec:
  temporalWorkerDeploymentRef:
    name: my-worker
  template:
    apiVersion: policy/v1
    kind: PodDisruptionBudget
    spec:
      minAvailable: 1
      # {} tells the controller to auto-inject {temporal.io/build-id, temporal.io/deployment-name}.
      selector:
        matchLabels: {}

Checking status

# See all WorkerResourceTemplates and which TWD they reference
kubectl get WorkerResourceTemplate -n my-namespace

# See per-Build-ID apply status
kubectl get WorkerResourceTemplate my-worker-hpa -n my-namespace \
  -o jsonpath='{.status.versions}' | jq .

# See the created HPAs
kubectl get hpa -n my-namespace