Github Actions Self Hosted Runners on Kubernetes

3 min readMay 14, 2021

How to setup and monitor them.

Overview

GitHub actions have become very popular for doing CI/CD workflows for applications and infrastructure. Initially the workflows only ran on GitHub infrastructure. Self hosted runners has been introduced to run workflows inside your infrastructure. The advantages are:

Don’t have to post your precious credentials to GitHub as required using runners on GitHub infrastructure.
Don’t have to pay to use the GitHub infrastructure to run runners. Its free to run your own self-hosted runners.

Note: Only use Self-Hosted Runners with GitHub private repositories.

Setup

The most popular runner seems to be actions-runner-controller. This can be run in Kubernetes using their public helm chart (Deploy using Flux V2 https://github.com/neillturner/flux-examples/tree/main/actions-runner-controller).

The controller can be configured to run multiple different runners for different purposes.
Action runner controller can only be configured for one GitHub organization but multiple GitHub organizations can be supported by setting up multiple actions-runner-controllers in one Kubernetes cluster using the watch namespace configuration option.

Runner Images

I have not been able to get the supplied runner images to work:

and generally they don’t meet your specific requirements so it is necessary to build your own runner docker image. See:

Monitoring

Its important to be able to monitor the runners. Be alerted if they are down and be informed of the GitHub workflows run on the runners. Prometheus has become commonly used for monitoring systems. When monitoring GitHub there are existing GitHub exporters however these monitor the commits, pull requests etc on GitHub repos rather than actions on runners. Currently the information GitHub supplies about GitHub actions is limited but hopeful this will improve as GitHub do further development.

There are 2 ways to get metrics about self hosted runners:

The Actions runner controller has a metrics endpoint.
github-actions-exporter can be used to regular read the GitHub api for information about the actions and runners and expose a metrics endpoint.

Actions runner controller metrics endpoint

The easiest way to get at the metrics endpoint is to install with version 0.12.0 or greater of the action-runner-controller public helm chart and set:

    metrics:
      serviceMonitor: false
      port: 8080
      proxy:
        enabled: false

if you have the full Prometheus stack installed in your Kubernetes cluster you can specify serviceMonitor: true and Prometheus will automatically scrape the endpoint otherwise you need also configure in Prometheus:

extraScrapeConfigs: |
  job_name: ‘github-actions-exporter’
  static_configs:
    targets:
      github-actions-exporter.github-actions-exporter:8080

Once this is configured various metrics are returned under the names:

runner-controller, runnerdeployment-controller, horizontalrunnerautoscaler-controller, runnerreplicaset-controller.

Github Actions Exporter

Deploy from the helm chart available from Spendesk/github-actions-exporter. Deploy using Flux V2 see https://github.com/neillturner/flux-examples/tree/main/github-actions-exporter

Configuration can be either for:

One of more Github Organizations.
A list of Github Repositories.

When configuring with an organization metrics are returned that can be used to check if the runner is up:

github_runner_organization_status{id=”499",name=”company-org-runner--79999999k”,organization=”company”,os=”linux”} 1

When configured with a list of repositories metrics are returned that show the GitHub workflows run on the runner:

github_job{event=”pull_request”,head_branch=”config-change”,head_sha=”85c8f0d534999999999999d644b”,id=”816383374",node_id=”MDZZZZZZZZZZZZZZZZZMzc0",repo=”bigcorp/bigcorp-infrastructure”,run_number=”99",status=”completed”,workflow=”Development”,workflow_id=”5199999"} 1