Self-Hosted CI Runner Autoscaler with Warm Pools

The problem with fixed build capacity

Teams that run their own CI runners (for private networks, special hardware, large caches or cost reasons) usually size them once. Either machines sit idle overnight and at weekends, or developers wait many minutes for a runner at 10 a.m. when everyone pushes at once. Both cost money; the second also costs attention, because a developer waiting on a build switches context and loses the thread.

Software houses and product teams feel this as soon as they outgrow hosted runner minutes or need runners inside their own network. The fix sounds simple (scale with the queue) but the details are where teams get hurt: cold starts, runners killed mid-job, and bursts that arrive faster than machines boot.

What already exists and the gap

• Actions Runner Controller (ARC) is GitHub's Kubernetes operator that scales self-hosted runners through runner scale sets and a listener watching the job queue.
• terraform-aws-github-runner creates ephemeral GitHub runners on EC2 spot instances using webhooks and Lambda, can scale to zero, and supports an idle pool on a cron schedule.
• GitLab Runner autoscaling uses a fleeting plugin to create and remove cloud instances for jobs.

These are mature, but they expose knobs (minimum runners, idle schedules, scale-down delay) without telling you what values to use for your team's traffic. The gap this project fills is the policy and the evidence: a controller small enough to understand, plus a simulator that maps the cost versus wait-time curve for a given workload so a team can choose settings deliberately.

What you build

• Queue monitor: subscribes to job events for one CI system and keeps an accurate view of pending, running and finished jobs.
• Scaling controller: a reconcile loop that compares desired runners (from the queue and the policy) with actual runners and acts.
• Warm pool: a small number of ready runners, sized by time of day, so common jobs start immediately.
• Metrics and dashboard: wait time, runner utilisation and estimated cost per job.
• Simulator: replays recorded or synthetic job arrival traces against different policies without spending money.

The technically interesting part

The trade-off at the centre is cold start against idle cost. Terminating aggressively saves money and makes the next job wait for a machine to boot and pull images. Keeping a warm pool costs money and removes the wait. There is no single right answer, only a curve, and mapping that curve is the project.

The second hard part is correctness of the control loop. Webhooks arrive late, twice or not at all; a runner may register but never pick up a job; scale-down may race with a new assignment. Use the reconcile pattern Kubernetes controllers use (compare desired and observed state on every tick, rather than trusting individual events) and show tests for each race you found.

How to evaluate it

Running a real cluster for weeks under varied load is not feasible in a semester, so evaluate in two steps.

• Simulation: drive the simulator with three patterns: steady arrival, a weekday morning spike, and a burst when a large pull request triggers fifty jobs at once. Compare fixed capacity, scale-from-zero and your policy on total cost and p95 wait. Policies that look good on smooth load often fail on the burst, which is the case developers actually complain about.
• Validation: run the real controller on a small kind or k3s cluster with a scripted workload and check that its measured wait times match the simulator within a stated margin.
• Safety: zero jobs killed mid-run across all tests, and correct recovery after the controller is restarted.

Suggested plan

1. Weeks 1 to 2: study ARC and one CI API, record or generate job traces, define metrics.
2. Weeks 3 to 5: queue monitor and reconcile-based controller on a local cluster.
3. Weeks 6 to 8: warm pool, safe scale-down, metrics and dashboard.
4. Weeks 9 to 11: simulator, policy comparison across workloads.
5. Weeks 12 to 14: validation against the real cluster, write-up and demo.

Path to a product

The realistic path is open source plus consulting. Publish the simulator as a tool that takes a team's exported job history and recommends warm-pool and scale-down settings for ARC or the Terraform module; that is useful even to teams who never run your controller. A software house that runs its own runners is a good first user. It is also a strong portfolio centrepiece for platform and infrastructure engineering roles, because it shows control loops, cost thinking and measured results.

Scope warning

One CI system and one environment (a local cluster or one cloud provider). Do not build a CI system, a build cache or a multi-cloud scheduler. The scaling policy and its evaluation are the contribution; keep the rest thin.

Learn what you need on Sefism

• DevOps and cloud roadmap : containers, Kubernetes and CI/CD in the order you need them.
• DevOps and cloud cheat sheet : quick reference for Kubernetes objects and pipeline concepts.
• System design cheat sheet : queues, autoscaling and back-pressure in one place.
• GitHub Engineering Blog : how GitHub runs Actions and its own infrastructure at scale.
• Trivy : scan the runner images your controller launches.

Get help with this idea

This project succeeds when the evaluation is honest and the scope stays narrow. Tauseef can help you pick the CI system, design the simulator and choose workload patterns that convince examiners. If you are new here, create a free account first.

• Discuss this idea with a mentor
• Send a query to Tauseef
• Book a 1:1 session