Background

Our team keeps frontend (web, admin) and backend in separate repositories and builds them with GitLab CI.

We used to run GitLab Runner by setting up runners on servers we provisioned on AWS EC2.

The backend was a monorepo of multiple services, and since the source changed frequently, the build cache rarely stuck. Even with five Runners running, builds were still slow, and Kaniko's caching behavior made the cache practically useless.

At that point we decided it was time to improve things, so we requested a new on-premise server, got one allocated, and set it up from scratch.

This post walks through what we changed: from switching Kaniko to BuildKit, to reworking the Runner setup, the caching strategy, and eventually the disk layout.

Migrating to an On-Premise Server

From Kaniko to moby/buildkit

We originally adopted Kaniko to avoid the security issues of docker-in-docker. The problem turned out to be how it handles cache lookups.

Kaniko checks layer caches in order, and if even one layer misses, it skips checking the cache for every layer after that, even if a cache exists for them.

Since the source changed on nearly every commit, an early layer would miss often, making the caches further down useless. We weren't getting much value out of the registry cache.

Rather than work around this limitation, we replaced the tool itself with moby/buildkit.

Reducing Runners and Raising Concurrency

We had been running five registered Runners. But a single GitLab Runner can already process multiple jobs at once through the concurrent option.

So there was no real reason to keep five separate ones. We kept a single Runner and set concurrency to 8.

Performance itself didn't change much, but we went from managing registration and configuration in five places down to one.

Why We Moved the Cache Locally

We used to push the Kaniko layer cache to a registry and pull it back down. We replaced that with BuildKit's local layer cache and reorganized where things were stored. pnpm and Next.js caches were moved to local storage on the Runner server the same way.

The network round trip of pushing and pulling the cache to the registry disappeared, and the wait time dropped by that much.

Adding a Prepare Step

Steps like pnpm install, which should be cacheable, were running in parallel across multiple jobs at the same time. That timing overlap meant the cache wasn't reliably hit.

We introduced a separate prepare step that downloads and caches dependencies first, so the jobs that run afterward can just reuse that cache instead of racing each other.

Finding an HDD I/O Bottleneck

Even after all of this, CI results didn't improve as much as we expected. So we looked at server metrics and found %util on the HDD (sdb) sitting near 100%.

Cache and build traffic was still passing through the HDD, and write await was spiking into the tens of milliseconds.

We moved Docker's data-root, the BuildKit cache, the GitLab cache, and the /builds workspace to an NVMe SSD (nvme0n1).

Afterward, write await on the NVMe stayed around 1-2ms in the same window.

Moving the Root (/) Disk from HDD to SSD

Even after that, we kept watching the Runner and noticed it was still slowing down.

Even with CI-related workloads already moved to NVMe, iowait still spiked to around 10%, and load average, which reflects the number of processes stuck waiting, climbed as high as 7.

We traced the cause to the OS root (/), which was still sitting on the HDD (sdb). Small but frequent writes were all passing through that HDD.

In other words, there were fsync-triggered writes like the ext4 journal, journald, and dockerd/containerd state files, and write await in that window reached 30-200ms, with %util hitting 100%.

We added the SSD as an LVM PV and moved the root LV with pvmove.

Because of LVM, we finished without a reboot or any Runner downtime. We didn't need to touch the bootloader either.

Afterward, write await on root dropped to 0.7ms, and load average settled at 0.5. /boot sees almost no writes, so we left it on the HDD.

Results

Here's how build times changed for three repositories, comparing quality builds and build-and-deploy jobs before and after.

Wrap-up

Our goal was simple: the old GitLab Runner setup was so slow it was dragging down the team's productivity, and we wanted to fix that as fast as we could. Getting there felt genuinely good.

Looking back, though, no single change fixed this on its own.

We had to swap the build tool, clean up the Runner setup, move where the cache lives, and eventually rework the disk layout before the bottlenecks cleared one by one.

The root disk sitting on an HDD was especially easy to miss until we looked closely at the metrics, and it turned out to be one of the most fundamental causes.

If CI feels slow, it's worth checking more than just the build tool or cache settings — check the disk I/O on the machine running your Runner too.