Hardening Linux Against OverlayFS: Blue Team Defense Guide
Overview & Defensive Context: Linux OverlayFS SUID Exploitation (CVE-2023-0386)#
Harden Linux against OverlayFS CVE-2023-0386: restrict unprivileged user namespaces, mount /tmp nosuid, deploy eBPF probes, and patch kernels.
In our morning architectural analysis, we deconstructed the attack mechanics of CVE-2023-0386 (CVSS 7.8, CWE-282), a critical local privilege escalation flaw in the Linux kernel's OverlayFS subsystem (fs/overlayfs/copy_up.c). We analyzed how unprivileged attackers chain Filesystems in Userspace (FUSE) and unprivileged user namespaces (CLONE_NEWUSER) to bypass the kernel's MS_NOSUID security boundary. By simulating a binary owned by root (UID 0) with the SetUID bit set (04755) on a FUSE mount and triggering a copy-up into an unconstrained host filesystem directory (such as /tmp), the kernel writes an executable SUID-root binary directly onto physical storage.
For systems engineers, platform architects, and blue teams, defending enterprise Linux infrastructure and container environments against CVE-2023-0386 reveals severe architectural blind spots in standard access control stacks:
- The Inefficacy of Traditional Discretionary Access Control (DAC): The attack does not require modifying
/etc/passwd,/etc/shadow, or existing system binaries. Standard file integrity monitoring (FIM) tools watching system paths miss the creation of the transient FUSE mount and subsequent copy-up into scratch directories. - The Namespace Confinement Illusion: Unprivileged user namespaces (
unshare -U) are often perceived as harmless sandboxes because root inside a user namespace does not map toGLOBAL_ROOT_UIDon the host. However, becauseovl_copy_up_one()failed to checkkuid_has_mapping(), the kernel acted as a confused deputy, bridging capabilities between the unprivileged namespace and the host filesystem. - FUSE Security Boundary Failure: The kernel guarantees that binaries executed from FUSE mounts cannot escalate privileges by automatically applying the
MS_NOSUIDmount flag. However, OverlayFS copy-up transfers files out of the FUSE mount and into theupperdirfilesystem, stripping away theMS_NOSUIDenforcement if the upper directory is mounted without it. - Container Breakout and Multi-Tenant Exposure: While container runtimes use OverlayFS to stack container image layers, multi-tenant Kubernetes clusters or shared development servers that allow unprivileged user namespaces or access to
/dev/fuseallow local users to elevate to host root instantaneously.
Standard antivirus, EDR file hooks, and filesystem DAC permissions are completely bypassed by CVE-2023-0386. Securing enterprise Linux systems requires kernel patch validation, restricting unprivileged user namespaces via sysctl, enforcing strict nosuid mount options across scratch storage, and deploying runtime eBPF/auditd detection.
⚔️ Exploitation Mechanics: For root-cause memory corruption analysis and attack flow, see our deep-dive: Linux OverlayFS: Deconstructing CVE-2023-0386 SUID Privilege Escalation.
Architecture Hardening: Multi-Layered Defense-in-Depth#
Mitigating OverlayFS SUID smuggling and user namespace abuse demands a four-ring defensive architecture: Kernel Patch Verification, Restricting Unprivileged User Namespaces, Mount Security Controls (nosuid/noexec), and Linux Security Module (AppArmor/SELinux) Confinement.
Defense Architecture Flowchart#
flowchart TD
classDef client fill:#1e293b,stroke:#ef4444,stroke-width:2px,color:#f8fafc
classDef gate fill:#0f172a,stroke:#3b82f6,stroke-width:2px,color:#93c5fd
classDef kernel fill:#064e3b,stroke:#10b981,stroke-width:2px,color:#6ee7b7
classDef audit fill:#4c1d95,stroke:#8b5cf6,stroke-width:2px,color:#ddd6fe
classDef drop fill:#450a0a,stroke:#dc2626,stroke-width:2px,color:#fca5a5
classDef pass fill:#065f46,stroke:#34d399,stroke-width:2px,color:#a7f3d0
UnprivUser[Unprivileged User Initiates Exploit]:::client --> SysctlCheck{Ring 1: Sysctl UserNS & FUSE Gate}:::gate
SysctlCheck -- unprivileged_userns_clone = 0 --> DropUserNS[Action: Deny unshare with EPERM]:::drop
SysctlCheck -- UserNS Allowed & FUSE Accessible --> FUSECreate[Attacker Instantiates FUSE SUID File]:::client
FUSECreate --> MountGate{Ring 2: OverlayFS Mount & Namespace Gate}:::gate
MountGate -- AppArmor / SELinux Denies OverlayFS in UserNS --> DropMount[Action: Block Mount via LSM Policy]:::drop
MountGate -- Mount Permitted: lower=FUSE, upper=/tmp --> CopyUpTrigger[Process Touches File in Merged Directory]:::client
CopyUpTrigger --> KernelPatchCheck{Ring 3: ovl_copy_up_one Kernel Check}:::kernel
KernelPatchCheck -- Unpatched: Missing kuid_has_mapping Check --> VulnWrite[Vulnerability Path: Kernel Writes SUID-root to /tmp]:::drop
KernelPatchCheck -- Patched Kernel: kuid_has_mapping Returns False --> CleanDrop[Action: Abort Copy-Up with -EOVERFLOW]:::pass
VulnWrite --> ExecGate{Ring 4: Host Mount Flags Evaluation}:::gate
ExecGate -- /tmp Mounted with nosuid or noexec --> DropExec[Action: EACCES / SUID Bit Ignored]:::pass
ExecGate -- /tmp Missing nosuid: Binary Executes as Root --> Compromised[Host Compromised: uid=0 gid=0]:::drop
DropUserNS -.-> TelemetryPipeline[Ring 5: eBPF & Auditd Telemetry Stream]:::audit
CleanDrop -.-> TelemetryPipeline
DropExec -.-> TelemetryPipeline
TelemetryPipeline --> SIEMAlert[SIEM Telemetry Alert Triggered]:::passLayer 1: Kernel Patch Verification and Distribution Updates#
The root cause of CVE-2023-0386 was patched in upstream Linux via commit 4f114674094a (fs/overlayfs/copy_up.c). The fix introduces explicit verification of user identifier mapping before proceeding with file replication:
// Upstream fix in fs/overlayfs/copy_up.c
if (!kuid_has_mapping(current_user_ns(), ctx.stat.uid) ||
!kgid_has_mapping(current_user_ns(), ctx.stat.gid))
return -EOVERFLOW;
Verify that your running kernel version is >= 6.2, or contains the backported vendor fix on LTS releases:
# Check running kernel release
uname -r
# Verify distribution patch status
# Ubuntu / Debian:
apt-get update && apt-get --only-upgrade install linux-image-generic
# RHEL / Rocky / AlmaLinux:
dnf check-update kernel && dnf upgrade -y kernel
Layer 2: Restricting Unprivileged User Namespaces (sysctl)#
Unprivileged user namespaces (CLONE_NEWUSER) represent one of the most prolific privilege escalation surfaces in the Linux kernel. Unless required for rootless container engines (such as rootless Podman), unprivileged user namespace creation should be disabled across enterprise servers:
# Immediately disable unprivileged user namespaces at runtime
sudo sysctl -w kernel.unprivileged_userns_clone=0
# Limit total user namespaces allocation
sudo sysctl -w user.max_user_namespaces=0
# Persist configuration across reboots in /etc/sysctl.d/99-security.conf
cat << 'EOF' | sudo tee /etc/sysctl.d/99-security.conf
# Mitigate unprivileged namespace exploitation (CVE-2023-0386, CVE-2024-1086)
kernel.unprivileged_userns_clone = 0
user.max_user_namespaces = 0
EOF
# Reload sysctl settings
sudo sysctl --system
On Debian and Ubuntu systems, kernel.unprivileged_userns_clone is the canonical sysctl parameter. On modern systemd-based distributions (RHEL 9, Ubuntu 24.04), setting user.max_user_namespaces = 0 provides an additional layer of kernel-level enforcement.
Layer 3: Hardening Filesystem Mount Options (nosuid, noexec, nodev)#
Even if an attacker successfully triggers the copy-up vulnerability, privilege escalation is completely thwarted if the destination filesystem (upperdir) enforces the nosuid mount option. The Linux Virtual File System (VFS) ensures that binaries executed from a nosuid filesystem run with the caller's actual UID and GID, stripping the SUID effect.
Audit /etc/fstab and ensure that all world-writable scratch filesystems (/tmp, /var/tmp, /dev/shm) enforce strict isolation flags:
# Inspect current mount parameters for scratch storage
mount | grep -E "(/tmp|/var/tmp|/dev/shm)"
# Recommended /etc/fstab entries for production servers
# Ensure nosuid, noexec, nodev are explicitly enforced
cat << 'EOF' | sudo tee -a /etc/fstab
tmpfs /tmp tmpfs defaults,nosuid,noexec,nodev,size=4G 0 0
tmpfs /var/tmp tmpfs defaults,nosuid,noexec,nodev,size=2G 0 0
tmpfs /dev/shm tmpfs defaults,nosuid,noexec,nodev 0 0
EOF
# Remount active filesystems immediately with hardened parameters
sudo mount -o remount,nosuid,noexec,nodev /tmp
sudo mount -o remount,nosuid,noexec,nodev /var/tmp
sudo mount -o remount,nosuid,noexec,nodev /dev/shm
Layer 4: AppArmor and SELinux Mandatory Access Control Confinement#
Deploy Linux Security Modules (LSM) to enforce confinement on unprivileged user namespaces and FUSE device access:
- Restrict
/dev/fuseDevice Node Permissions: Prevent arbitrary unprivileged users from mounting FUSE filesystems:
# Restrict FUSE device node to members of the fuse group or root
sudo chmod 0600 /dev/fuse
sudo chown root:root /dev/fuse
- Deploy AppArmor User Namespace Restrictions (Ubuntu 23.10+ / 24.04 LTS): Modern Ubuntu releases support granular AppArmor mediation for unprivileged user namespaces:
# Verify AppArmor user namespace restriction status
sudo sysctl kernel.apparmor_restrict_unprivileged_userns
# Enforce AppArmor restriction on unprivileged user namespaces
sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=1
- SELinux Policy for RHEL / Fedora:
Verify that SELinux enforces the
deny_user_namespaceboolean on sensitive server roles:
# Inspect SELinux boolean
getsebool deny_user_namespace
# Enable deny_user_namespace to block unprivileged namespace creation
sudo setsebool -P deny_user_namespace 1
Production Detection Queries: Sigma Rule, Auditd & eBPF Telemetry#
Detecting CVE-2023-0386 requires monitoring three critical telemetry points: execution of SUID binaries out of temporary directories, unprivileged user namespace instantiation, and anomalous OverlayFS mount syscalls.
1. Production Sigma Rule: SUID Privilege Escalation from Scratch Directories#
The following production-ready Sigma rule monitors process creation events via Linux auditd, Sysmon for Linux, or endpoint agents. It alerts when a process is spawned from /tmp, /var/tmp, or /dev/shm where the effective user ID (euid: 0) diverges from the real user ID (uid != 0):
title: SUID-Root Execution in Temporary Directory (CVE-2023-0386)
id: c24f781a-9d22-48a1-b841-7a6c9e012386
status: production
description: |
Detects the execution of binaries residing in world-writable directories
(/tmp, /var/tmp, /dev/shm) where the process elevates privileges to root (euid=0)
from an unprivileged caller (uid!=0), characteristic of OverlayFS SUID copy-up exploits.
references:
- https://securitylabs.datadoghq.com/articles/overlayfs-cve-2023-0386/
- https://nvd.nist.gov/vuln/detail/CVE-2023-0386
tags:
- attack.privilege_escalation
- attack.persistence
- attack.t1068
- attack.t1548.001
logsource:
product: linux
service: auditd
detection:
selection_exec:
type: 'EXECVE'
a0|startswith:
- '/tmp/'
- '/var/tmp/'
- '/dev/shm/'
selection_elevation:
euid: '0'
filter_legit_root:
uid: '0'
condition: selection_exec and selection_elevation and not filter_legit_root
fields:
- uid
- euid
- comm
- exe
- a0
- cwd
- ppid
falsepositives:
- Highly anomalous in hardened environments. No standard production service executes SUID binaries from /tmp.
level: critical
2. Linux Auditd Syscall Monitoring Configuration#
Deploy comprehensive auditd rules to generate forensic audit records whenever unprivileged processes attempt namespace clones or directory attribute tampering:
# Place in /etc/audit/rules.d/99-overlayfs-exploit.rules
# 1. Monitor unshare and clone syscalls requesting CLONE_NEWUSER (0x10000000)
-a always,exit -F arch=b64 -S unshare -F a0&0x10000000 -k userns_abuse
-a always,exit -F arch=b64 -S clone -F a0&0x10000000 -k userns_abuse
# 2. Monitor OverlayFS filesystem mount invocations
-a always,exit -F arch=b64 -S mount -F a2=overlay -k overlayfs_mount
# 3. Monitor file permission and SUID bit modifications in /tmp and /var/tmp
-a always,exit -F arch=b64 -S chmod,fchmod,fchmodat -F dir=/tmp -k suid_temp_creation
-a always,exit -F arch=b64 -S chmod,fchmod,fchmodat -F dir=/var/tmp -k suid_temp_creation
# 4. Monitor extended attribute writes (xattr smuggling)
-a always,exit -F arch=b64 -S setxattr,lsetxattr,fsetxattr -F dir=/tmp -k xattr_tamper
3. Falco Runtime Security Rule for Kubernetes & Linux Nodes#
For organizations operating containerized workloads or running cloud instances with Falco, deploy the following detection rule:
# Falco Rule: Detect SUID Execution from Scratch Paths
- rule: SUID Execution in Writable Directory
desc: Detects execution of a binary with SUID bit set in /tmp, /var/tmp, or /dev/shm
condition: >
spawned_process and
user.uid != 0 and
user.euid == 0 and
(proc.pcmdline startswith "/tmp/" or
proc.pcmdline startswith "/var/tmp/" or
proc.pcmdline startswith "/dev/shm/")
output: >
Critical: SUID binary executed from writable path (user=%user.name user_uid=%user.uid
euid=%user.euid command=%proc.cmdline path=%proc.exepath container=%container.id)
priority: CRITICAL
tags: [host, container, privilege_escalation, cve-2023-0386]
4. Production eBPF Kernel Probe (bpftrace) for OverlayFS Copy-Up#
Deploy this lightweight bpftrace script on staging or critical nodes to trace instances where ovl_copy_up_one replicates files with root UID:
// trace_copy_up.bt - Trace OverlayFS SUID Copy-Up Operations
kprobe:ovl_copy_up_one
{
$dentry = (struct dentry *)arg1;
$name = (char *)$dentry->d_name.name;
printf("[eBPF ALERT] ovl_copy_up_one called by PID %d (%s) on file: %s\n",
pid, comm, str($name));
}
kprobe:vfs_getattr
/comm == "fuse_server"/
{
printf("[eBPF AUDIT] FUSE vfs_getattr invoked by PID %d (%s)\n", pid, comm);
}
Enterprise Mitigation Matrix: Workarounds, Hotfixes & Architecture#
When hardening enterprise Linux infrastructure against CVE-2023-0386, security engineering teams must weigh performance overhead against operational disruption:
| Remediation Strategy | Implementation Effort | Blast Radius & Operational Risk | Detection & Prevention Efficacy | Performance Overhead | Architectural Longevity |
|---|---|---|---|---|---|
| Workaround: Restrict sysctl unprivileged_userns_clone=0 | 15 - 30 Minutes (Deploy sysctl configuration across fleet via Ansible/Puppet) |
Low - Moderate May impact rootless container engines (Podman, Docker rootless). |
High (95%) Prevents the unshare(CLONE_NEWUSER) step required to mount OverlayFS. |
Zero Native kernel sysctl evaluation. |
Tactical Barrier Immediate mitigation while scheduling kernel updates. |
| Workaround: Enforce nosuid,noexec on /tmp and /var/tmp | 1 Hour (Update /etc/fstab and remount active tmpfs/ext4 mounts) |
Low Scripts running directly out of /tmp will be blocked by noexec. |
High (90%) Strips SUID execution semantics if the file is smuggled to disk. |
Zero Standard VFS mount flag enforcement. |
Baseline Best Practice Essential CIS Benchmark compliance requirement. |
| Operational Fix: Kernel Upgrade to >= 6.2 (or Vendor Backport) | 2 - 4 Hours (Deploy patched kernel package and execute rolling node reboots) |
Low Standard maintenance reboot; upstream patch is clean and surgical. |
Absolute for CVE-2023-0386 (100%) Enforces kuid_has_mapping(), terminating unauthorized copy-up. |
Zero Native kernel path. |
Mandatory Standard Permanently resolves the kernel copy-up logic flaw. |
| Architecture Fix: LSM Confinement (AppArmor / SELinux) & Read-Only Root | 2 - 4 Weeks (Deploy AppArmor userns profile, dm-verity, and IMA appraisal) |
Moderate - High Requires testing application profiles and pipeline automation. |
Comprehensive (100%) Neutralizes all namespace-based SUID smuggling and memory exploits. |
< 1% LSM hook evaluation overhead. |
Permanent State Hardened zero-trust Linux host architecture. |
If your environment requires rootless containers (e.g., developers utilizing rootless Podman), do not globally set user.max_user_namespaces = 0. Instead, apply Layer 3 (nosuid on /tmp), update the kernel to a patched release, and use AppArmor profiles to restrict unconfined binaries from mounting arbitrary FUSE filesystems.
Incident Response & Verification Playbook: Live Triage & Eradication#
If a CVE-2023-0386 exploitation attempt or unauthorized SUID binary creation is detected, execute this four-phase incident response playbook immediately:
flowchart TD
classDef alert fill:#450a0a,stroke:#dc2626,stroke-width:2px,color:#fca5a5
classDef check fill:#1e293b,stroke:#f59e0b,stroke-width:2px,color:#fef3c7
classDef triage fill:#0f172a,stroke:#3b82f6,stroke-width:2px,color:#93c5fd
classDef clean fill:#064e3b,stroke:#10b981,stroke-width:2px,color:#6ee7b7
Triage[Phase 1: Forensic Scan for Rogue SUID Binaries]:::triage --> ScanScratch{Check /tmp, /var/tmp, /dev/shm for SUID Files}:::check
ScanScratch -- SUID Files Identified in Scratch Paths --> IsolateNode[Phase 2: Immediate Host Isolation & Process Kill]:::alert
ScanScratch -- No SUID Files in Scratch Paths --> CheckAuditd{Check auditd for userns_abuse Events}:::check
CheckAuditd -- Unshare Logged without SUID Creation --> KillFUSE[Terminate Rogue FUSE Processes]:::check
CheckAuditd -- Clean Audit Logs --> CloseIncident[Close Incident as False Positive]:::clean
IsolateNode --> KillAllExploit[Terminate Attacker PIDs and FUSE Daemons]:::alert
KillAllExploit --> PurgeScratch[Phase 3: Unmount FUSE & Secure Erase SUID Binaries]:::alert
PurgeScratch --> ApplySysctl[Apply sysctl unprivileged_userns_clone=0]:::clean
ApplySysctl --> RemountNosuid[Remount /tmp with nosuid,noexec,nodev]:::clean
RemountNosuid --> Phase4[Phase 4: Post-Remediation Verification & Kernel Update]:::cleanPhase 1: Live Triage & Forensic File Audit#
- Scan Scratch Paths for Rogue SUID Binaries:
Immediately inspect
/tmp,/var/tmp,/dev/shm, and user home directories for files with the SUID bit set (04000):
# Find all SUID binaries in world-writable and scratch paths
find /tmp /var/tmp /dev/shm /home -perm -4000 -type f -exec ls -la {} + 2>/dev/null
- Inspect Active Mount Points for Unmapped OverlayFS Mounts:
Check
/proc/mountsand/proc/self/mountinfofor suspicious OverlayFS mounts pointing to FUSE directories:
# Inspect active overlay mounts
grep overlay /proc/mounts
# Look for FUSE mounts owned by non-root users
grep fuse /proc/mounts
- Query Auditd Logs for Exploitation Artifacts:
Search
auditdlogs for the execution of unshare or SUID execution from temporary folders:
# Search for user namespace creation events
ausearch -k userns_abuse --start recent
# Search for SUID executions from temporary directories
ausearch -k suid_temp_creation --start recent
Phase 2: Immediate Host Isolation & Containment#
- Isolate the Affected Host: Sever network connectivity to prevent lateral movement:
# Drop all inbound and outbound network traffic except administrative SSH
sudo iptables -P INPUT DROP
sudo iptables -P FORWARD DROP
sudo iptables -P OUTPUT DROP
sudo iptables -A INPUT -p tcp --dport 22 -j ACCEPT
sudo iptables -A OUTPUT -p tcp --sport 22 -j ACCEPT
- Terminate Malicious Processes & Unprivileged Sessions: Identify and kill the attacker's shell session and any lingering FUSE background daemons:
# Terminate processes with open file descriptors in /tmp
sudo fuser -k -9 -v /tmp
# Terminate all active FUSE daemons
sudo killall -9 fuse_server 2>/dev/null
Phase 3: Eradication, File Restoration & Hardening#
- Securely Erase Smuggled SUID Binaries: Remove any discovered malicious binaries and force filesystem synchronization:
# Remove malicious files
sudo rm -rf /tmp/cve_work /tmp/rootshell /tmp/upper /tmp/work /tmp/merged
# Force page cache sync and flush
sudo sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
- Enforce Immediate Emergency Mitigations: Apply sysctl and mount restrictions without requiring an immediate reboot:
# Disable unprivileged user namespaces immediately
sudo sysctl -w kernel.unprivileged_userns_clone=0
sudo sysctl -w user.max_user_namespaces=0
# Enforce nosuid on all scratch mounts
sudo mount -o remount,nosuid,noexec,nodev /tmp
sudo mount -o remount,nosuid,noexec,nodev /var/tmp
sudo mount -o remount,nosuid,noexec,nodev /dev/shm
Phase 4: Post-Remediation Verification & Health Checklist#
Before restoring the host to production:
- Verify that
uname -rindicates kernel version>= 6.2or an approved vendor security errata release. - Verify that
sysctl kernel.unprivileged_userns_clonereturns0(oruser.max_user_namespaces = 0). - Confirm that
mount | grep /tmpshowsnosuid,noexec,nodevoptions are active. - Test that running
unshare -Urm /bin/bashas an unprivileged user fails withEPERM. - Verify that the Sigma detection rule
c24f781a-9d22-48a1-b841-7a6c9e012386is active in your SIEM. - Confirm that
auditdrules in/etc/audit/rules.d/99-overlayfs-exploit.rulesare loaded and generating logs.
Executive Takeaway Table#
| Attack Vector / Phase | Forensic Detection Signal | Primary Defensive Mitigation |
|---|---|---|
| FUSE SUID Emulation | Non-root process running with access to /dev/fuse; file reported with st_mode = 0104755. |
Restrict access to /dev/fuse via chmod 0600 /dev/fuse; confine via AppArmor/SELinux. |
Namespace Isolation (unshare) |
Syscall unshare(CLONE_NEWUSER) or clone(CLONE_NEWUSER) logged under auditd. |
Disable unprivileged user namespaces via sysctl -w kernel.unprivileged_userns_clone = 0. |
| OverlayFS Layering | Mount syscall with type=overlay where lowerdir references FUSE mount path. |
Enforce AppArmor/SELinux policy restricting unprivileged namespace mount capabilities. |
Kernel copy_up Smuggling |
ovl_copy_up_one() writes file with st_uid = 0 and SUID bit to host physical filesystem. |
Upgrade Linux kernel to version >= 6.2 (enforcing upstream patch kuid_has_mapping()). |
| Host SUID Payload Execution | EXECVE syscall on binary in /tmp where uid != 0 and euid == 0. |
Mount /tmp, /var/tmp, and /dev/shm with nosuid,noexec,nodev mount options. |
Authoritative References#
- National Vulnerability Database (NVD): CVE-2023-0386 Detail: Linux Kernel OverlayFS Local Privilege Escalation — NIST NVD
- Datadog Security Labs: The OverlayFS Vulnerability CVE-2023-0386: Architecture, Exploitation, and Detection — Datadog Security Research
- Linux Kernel Git Repository: Commit 4f114674094a: ovl: fail copy up if cannot map lower st_uid/st_gid by Miklos Szeredi — Kernel.org Git
- Cybersecurity and Infrastructure Security Agency (CISA): Known Exploited Vulnerabilities Catalog (CVE-2023-0386) — CISA KEV
- Center for Internet Security (CIS): CIS Linux Distribution Benchmark (Filesystem Configuration and Mount Options) — CIS Security
Comments
Post a Comment