Runner Management & Provisioning

This guide walks through the full operational lifecycle of an Orkestia runner group — from picking a kind and provisioning compute, through warm-pool capacity tuning and health monitoring, to clean decommissioning. For the conceptual overview, start with Runners concepts. For the catalog of every backend_type / purpose / integration_type, see Runner groups.

The defining property to keep in mind throughout: the runner compute always lives in your cloud account. Orkestia is the control plane — it provisions, registers, scales, and reaps runners — but it never hosts the compute and never sees your job contents. The runner binary phones home to GitHub directly; Orkestia mints short-lived registration tokens and observes. This is the Zero Code Custody posture applied to CI: your builds run on infrastructure you own and pay for.

Mental model: two object families

Everything in Runner Management reduces to two first-class entities:

The control loop you operate is: provision environment → register runners with GitHub → launch executions → serve jobs → scale → drift-repair → deregister. Each arrow is a runner.* workflow on the Orkestia engine; you rarely call them by hand (the Runners UI and SDK do), but understanding the chain is what makes monitoring and recovery legible.

Prerequisites

Before you can provision a group you need:

Step 1 — Choose a kind

The create wizard’s Runner type step is not “pick a cloud.” It is backend_type, filtered by the connection you selected. Purpose (github_actions / gitlab_runner / agent / generic) and integration (github / gitlab / none) are a separate choice — Purposes & integrations.

A group binds to one kind and one connection. There is no cross-cloud pool — a job cannot transparently spill from AWS to GCP within a single group. Multi-cloud orgs run multiple groups from the one control plane. Required config keys per kind come from data.runner.list-provider-config-specs.

Step 2 — Create a runner group

The fastest path is the Runners command center UI (the dedicated console at runners.orkestia.dev), which walks you through GitHub org → cloud target → scaling policy → network placement and submits the provisioning DAG for you. The same provisioning is available programmatically — useful for IaC-driven group creation.

A provisioning request describes the GitHub binding, the cloud target, the scaling policy, and the labels GitHub will route jobs by:

{
  "workflow": "runner.environment-provision-kubernetes",
  "input": {
    "organization_id": "org_01HXZ...",
    "github_connection_id": "cn_gh_main",
    "github_org": "myorg",
    "target": {
      "provider": "kubernetes",
      "env": "in-cluster",
      "connection_id": "cn_k8s_prod",
      "namespace": "ci-runners",
      "network_profile_id": "np_01HXZ..."
    },
    "scale": { "min": 0, "max": 10, "idle_timeout_s": 300 },
    "labels": ["self-hosted", "orkestia", "prod-builds"]
  }
}

What happens after you submit:

1. The group enters provisioning — the provider's environment DAG runs (runner.group-creation / runner.environment-provision-*), standing up the ECS service / EC2 group / namespace inside your cloud.
2. Orkestia mints a short-lived registration token via the GitHub App.
3. Runner executions launch and register themselves with GitHub, labelled per the group config.
4. The group goes active and starts picking up jobs that match its labels.

Registration is pull, not push: Orkestia never pushes runners onto GitHub. It hands each runner binary a short-lived token and the runner calls GitHub itself — so no long-lived secret sits on disk.

Step 3 — Configure warm pools & capacity

Capacity is governed by the group's scaling policy. The two knobs that matter most:

Set min: 0 for spiky, cost-sensitive pipelines (every runner is on-demand, accepting a cold-start delay). Set min to your steady-state concurrency for latency-sensitive CI where developers wait on builds.

How scaling actually fires

Scaling is driven by a reconciliation loop, not by webhooks. A per-group controller continuously converges the live pool toward your min/max — the same reconcile model Orkestia uses for drift & self-healing. GitHub workflow_job webhooks are a best-effort nudge that triggers an immediate reconcile, never the source of truth:

• workflow_job.queued → nudge: reconcile may scale up (bounded by max)
• workflow_job.completed / idle timeout → nudge: reconcile may scale down (bounded by min)

Because the loop — not the webhook — is authoritative, a missed or delayed workflow_job event never strands a group: the next reconcile pass converges it anyway. All long-running provisioning, scale, and drain steps run asynchronously over the platform's Kafka bus (workflow.transition), so a scale event never blocks an API call.

Step 4 — Register the build/runner app

Two registrations connect a group to your CI:

1. The GitHub App (done once per org, at connection time) authorizes Orkestia to mint registration tokens and subscribe to workflow_job / workflow_run events. This is set up when you create the GitHub connection (see Cloud connections).
2. The job labels. Orkestia registers each runner with the labels you set on the group (e.g. self-hosted, orkestia, prod-builds). To route a job to the group, target those labels in your workflow YAML:

jobs:
  build:
    runs-on: [self-hosted, orkestia, prod-builds]
    steps:
      - uses: actions/checkout@v4
      - run: make build

When a matching job is queued, GitHub emits workflow_job.queued, Orkestia (via the dispatch/scaling workflows) decides whether a new runner is needed, and an execution picks the job up. No runner endpoint is exposed to you — the binding is entirely through GitHub labels.

Step 5 — Monitor health

Runner activity is surfaced as workflow.transition events on the same Kafka bus as the rest of the platform — there is no runners-specific log silo. The signals you watch:

Health checks are periodic liveness probes. When a runner fails one, Orkestia deregisters it from GitHub and reaps the compute, so a dead runner never sits in the pool starving the queue.

These signals are first-class in Lumen observability — pickup latency, utilization (% of time runners are busy), scale-event counts, and failure rate by environment all roll up there. See the observability guide for building runner dashboards and alerts.

Common failure modes & what to do

Step 6 — Decommission a group

When you no longer need a group, archive it. Teardown drains in-flight jobs, revokes the GitHub registrations, and reaps the cloud resources — leaving no orphaned compute and no zombie runners on GitHub.

Because teardown is serialized per group by the advisory lock, you can safely archive one group while others keep serving. If teardown is interrupted mid-flight, the self-healing reconciler detects the partial state and completes the cleanup on the next sweep.

Limits & constraints

• GitHub Actions is the GA job source. GitLab is a first-class integration_type in the create wizard (purpose=gitlab_runner); Bitbucket and Jenkins are not in scope. See Purposes.
• One GitHub org per group — cross-org runner sharing isn't modeled; multi-org users create multiple groups.
• Compute stays in your cloud — a broken cloud connection means a broken group, not an Orkestia fallback. You pay the provider directly.
• No cross-cloud pool — a group targets exactly one environment.
• Scaling is bounded by group policy — Orkestia never exceeds max, even under queue pressure.
• Not a millisecond-burst replacement — self-hosted runners scale in seconds-to-minutes, not the instant elasticity of GitHub-hosted ephemeral runners.

Where to go next