Our CI pipeline for building and scanning the Docker image for our self-hosted runner was taking 8 minutes and 28 seconds. The image-build step accounted for 92% of the total time, forcing us to wait on every pull request.
We reduced this time to around 3 minutes through a four-phase improvement process. This article documents what we changed in each phase and why we tackled them in that specific order.
Context: What Are We Building?
We build a custom Docker image to run GitHub Actions self-hosted runners in containers. Our CI pipeline (lint.yml) consists of the following jobs:
- lint — ShellCheck / yamllint / xmllint / secret-scan
- dockerfile-lint — hadolint
- image-build-and-scan — Docker Build → Trivy Scan → Push to ghcr.io
- smoke-scale-startup — Verify that the container can actually start and register with GitHub
The bottleneck was job #3, image-build-and-scan, which was performing a full build of the 430MB+ image every time.
Phase 1: Warm Cache + BATS Parallelization
Goal: Start with the low-hanging fruit: improving the cache hit rate and parallelizing tests.
The Caching Problem
We were using the GHA cache (type=gha), but because we only had cache-to in the push step, pull request builds couldn’t write to the cache. This resulted in a cache miss even on subsequent pushes to the same PR.
# Before: Only the push step had cache-to
- name: Push image
if: github.ref == 'refs/heads/main'
uses: docker/build-push-action@v6
with:
cache-from: type=gha,scope=runner-image
cache-to: type=gha,scope=runner-image,mode=max # ← Only here
We moved cache-to to the build step, allowing PR builds to write to the cache as well.
# After: cache-to in the build step
- name: Build image
uses: docker/build-push-action@v6
with:
cache-from: type=gha,scope=runner-image
cache-to: type=gha,scope=runner-image,mode=max
Concurrency Control
We also had an issue where old builds would keep running after consecutive pushes to the same branch.
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
This cancels older, in-progress jobs on a force push to a pull request, while protecting the main branch and scheduled runs.
Parallel BATS Test Execution
The BATS unit tests (46 tests) that run after ShellCheck were executing serially, taking 103 seconds. Since each test uses an independent BATS_TEST_TMPDIR, they could be safely parallelized.
- name: Run unit tests (bats)
run: bats --jobs 4 tests/
103s → 33s (about 3x faster). The --jobs 4 flag requires GNU parallel, so we added it to our apt-get installation.
Phase 1 Results
| Item | Before | After |
|---|---|---|
| BATS tests | 103s | 33s |
| Cache hit rate | Always missed on PRs | Hit on 2nd push onwards |
Phase 2: GHA Cache → Registry Cache
Goal: Improve the cache restoration speed itself by changing the cache backend.
Why Registry Cache?
The GHA cache is based on Azure Blob Storage and restores layers sequentially. In contrast, a registry like ghcr.io (an OCI registry) can pull layers in parallel. Benchmarks reported in a docker/buildx issue show a 20-40% speed difference.
# After: Switched to registry cache
- name: Build image
uses: docker/build-push-action@v6
with:
cache-from: type=registry,ref=ghcr.io/${{ github.repository_owner }}/runner:buildcache
cache-to: >-
${{ github.ref == format('refs/heads/{0}', github.event.repository.default_branch)
&& format('type=registry,ref=ghcr.io/{0}/runner:buildcache,mode=max,compression=zstd,compression-level=3',
github.repository_owner)
|| '' }}
There are three key points:
- Write to
cache-toonly on the default branch. PRs will only read from the cache. - Use zstd compression (
compression-level=3) to reduce blob transfer size by about 20%. - Permission changes: The
actions: writepermission (for GHA cache) is no longer needed. Thepackages: writepermission now covers both registry cache and image pushes.
Authentication Token Cleanup
Using the registry cache requires authentication with ghcr.io. Unlike GitHub-hosted runners, self-hosted runners don’t automatically clean up credentials after a job finishes, so we must explicitly remove them.
- name: Cleanup docker config
if: always()
run: rm -rf "${RUNNER_TEMP}/docker-config" || true
The if: always() condition ensures this step runs even if the job is canceled or times out.
Phase 3: Dockerfile Layer Splitting + paths-ignore
Goal: With the cache backend optimized, the next step is to improve the granularity of cache invalidation.
The Monolithic RUN Problem
Our Dockerfile had a single, massive RUN instruction that created a layer of about 430MB. Just changing the corepack version would trigger a rebuild of the entire layer, including the apt-get install and Node.js installation steps.
Splitting into 3 Layers
# Layer 1: base apt packages (220MB, rarely changes)
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
--mount=type=cache,target=/var/lib/apt/lists,sharing=locked \
apt-get update \
&& apt-get upgrade -y \
&& apt-get install -y --no-install-recommends \
curl git jq ca-certificates ...
# Layer 2: Node.js (210MB, only changes on major version updates)
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
--mount=type=cache,target=/var/lib/apt/lists,sharing=locked \
curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \
&& apt-get install -y --no-install-recommends nodejs
# Layer 3: corepack (1.3MB, updated quarterly)
RUN npm install -g corepack@0.35.0 \
&& corepack enable
By using BuildKit’s --mount=type=cache to share /var/cache/apt, we only need to download apt packages once, even though the commands are split. This has zero impact on the final image size because cache mounts are not committed to the layer.
We also pinned the corepack version to 0.35.0 instead of latest during this phase. This helps reduce supply chain risk and stabilizes the cache.
Skipping Documentation Changes with paths-ignore
on:
push:
branches: [main]
paths-ignore:
- 'docs/**'
- '**/*.md'
- 'LICENSE'
pull_request:
paths-ignore:
- 'docs/**'
- '**/*.md'
- 'LICENSE'
Running CI for PRs that only change documentation is a waste of time. We use paths-ignore for this, assuming you don’t have required status checks configured in your branch protection rules. (If you do, you’d need to switch to an action like dorny/paths-filter).
Phase 4: Skip Scheduled Jobs + OCI Directory Output
Goal: Eliminate unnecessary build runs and ensure Trivy compatibility.
Optimizing the Schedule Trigger
The purpose of our scheduled run was only to perform a smoke test. However, it was running the full pipeline every time, including linting and image building, which was unnecessary.
lint:
if: github.event_name != 'schedule'
image-build-and-scan:
if: github.event_name != 'schedule'
Now, only the smoke-scale-startup job runs on a schedule, which frees up our self-hosted runners.
Trivy v0.70 Compatibility: tar → OCI Directory
Starting with Trivy v0.70, directly inputting an OCI layout directory is the recommended approach. Using a tar archive introduces overhead from extraction.
# Before: tar archive
outputs: type=docker,dest=${{ runner.temp }}/runner-image.tar
# After: OCI directory (tar=false)
outputs: type=oci,dest=${{ runner.temp }}/runner-image-oci,tar=false
trivy image \
--input ${{ runner.temp }}/runner-image-oci \
--severity HIGH,CRITICAL \
--ignore-unfixed \
--pkg-types os \
--exit-code 1 \
--format table
This eliminates the tar extraction step, reducing I/O costs.
Summary of Results
| Phase | Action | Main Effect |
|---|---|---|
| 1 | Warm cache + BATS parallelization | Tests 103s→33s, improved cache hit rate |
| 2 | Registry cache | 20-40% faster cache restoration |
| 3 | Dockerfile 3-way split + paths-ignore | Enabled partial rebuilds, skipped docs PRs |
| 4 | Schedule skip + OCI directory | Eliminated unnecessary builds, ensured Trivy compatibility |
Total: 8m 28s → ~3 minutes (64% reduction)
On the Order of Improvements
You might wonder, “Why not just split the Dockerfile layers from the beginning?” There’s a reason we proceeded in the order of Phase 1 → 2 → 3 → 4.
- Phase 1 (Cache + Parallelization): These were small, low-risk changes. We started with improvements that were guaranteed to have an impact.
- Phase 2 (Registry Cache): Changing the cache backend has a wide-ranging impact, but it was safer to proceed after understanding the cache behavior from Phase 1.
- Phase 3 (Dockerfile Splitting): We tackled cache granularity improvements only after the registry cache from Phase 2 was stable. If we had done this in reverse, it would have been difficult to isolate problems between the cache backend and the layer splitting.
- Phase 4 (Schedule Skip + OCI): After addressing the main issues in Phases 1-3, we moved on to peripheral optimizations.
When improving CI, it’s better to proceed one step at a time while measuring the results, rather than trying to do everything at once. This approach makes it clear which changes are effective and makes rollbacks easier if a problem occurs.
