Blog

443 dead FUSE mounts wedged kubelet: a production postmortem

· the plori team

Incident diagnosed and remediated 2026-07-23. Published 2026-07-24.

TL;DR. Pods on one of our agent nodes had been leaking FUSE mounts for about four weeks. A dead FUSE mount answers every access with ENOTCONN ("transport endpoint is not connected"), and kubelet's volume teardown cannot get past that error, so it retries forever. By the time we looked, 443 dead mounts were pinning 447 orphaned pod directories and 464 leaked cgroups, and the node's k3s-agent process was burning 0.83 cores at idle just walking the wreckage. Cold pod boots went from about a second to between 20 and 105 seconds, tripping our 120-second boot timeout. Lazy-detaching every dead mount from the host mount namespace dropped that CPU from 0.85 to 0.18 cores within minutes. A 60-second janitor in our FUSE CSI node driver now reaps dead mounts automatically.

Background: why our nodes churn FUSE mounts

plori is a cloud AI agent. Each agent gets its own computer: its disk is a per-agent FUSE mount over JuiceFS, and the agent sleeps when idle, which stops billing. This post is an engineering note from running that.

The architecture means pods come and go constantly. A warm pool pre-boots pods so agents wake fast; idle pods get reaped; deploys replace them. Each executor pod carries a FUSE mount served by orlop, our open-source storage plane, and there are two ways that mount can outlive its pod:

  • Late-attach pods (the warm pool): a privileged sidecar mounts the agent disk onto an emptyDir after the pod starts, and the mount record propagates into the host mount table. Nothing except that sidecar knows the mount exists. It is not a CSI volume, so no CSI teardown will ever be called for it.
  • CSI-mode pods: the FUSE client process is exec'd by our CSI node plugin. A plugin restart kills those child processes before NodeUnpublishVolume ever runs for them.

In both cases, when the pod (or plugin) dies, the userspace side of the FUSE mount dies with it, but the kernel's mount record survives on the host under /var/lib/kubelet/pods. Every access to that path from then on returns ENOTCONN. No agent data is at risk at that point: the data lives in the storage plane, and a short lease TTL reclaims the storage-side lease. The corpse is purely a host-local liability. We knew individual mounts could leak (it is in our runbook as a known limitation). What we had not understood was what hundreds of them would do to kubelet.

What happened

On 2026-07-23 our alerting channel reported warm pod boot failures: context deadline exceeded, boot_timeout_s=120. Digging into that day's boot telemetry, every cold pool or warm pod boot on the node was taking 20 to 105 seconds, with zero active agent runs on it.

The obvious suspects cleared quickly. Scheduling latency was zero (scheduled_ms=0 on every boot) and the pods request very little (50m CPU, 64Mi), so this was not scheduler pressure or bin-packing. All the time was in container startup. Something on the node itself was slow.

One honest note before the fun part: there was no user impact. The pod that tripped the 120-second timeout was a pool-reconciler replacement pod, not a user agent; user-facing agent claims during the incident window were warm-pool hits at 249ms and 644ms. Our stuck-pod reaper cleaned up the failed pod and the retry succeeded. The alert fired precisely because the platform was quietly eating the latency on our side of the pool.

What we found on the node

top in the host PID namespace showed k3s-agent (the k3s process that bundles kubelet and container runtime supervision) averaging 0.83 cores with nothing running. That is most of a small agent node's CPU budget, spent at idle.

/proc/mounts explained why. It contained roughly 427 FUSE mount entries with source orlop, all belonging to executor pods that were long dead, plus 16 more leaked through the CSI path: 443 corpses in total. Downstream of them:

  • 447 orphaned pod directories under /var/lib/kubelet/pods. The oldest was dated 2026-06-28, so the pile had been growing for about four weeks, surviving reboots.
  • 464 leaked cgroups under kubepods.slice (447 of them pod-level).
  • A journal full of two signatures, abridged here:
UnmountVolume.TearDown failed ... : transport endpoint is not connected
Housekeeping took longer than expected ...

The first appeared about 423 times per two minutes across the leaked volumes. The second (cAdvisor's container housekeeping) appeared 123 times in ten minutes. The journal was taking on roughly 4,341 lines per ten minutes on an idle node.

Why does kubelet fail to unmount a dead FUSE mount?

First, the term, because it is what you will type into a search engine at 2 a.m.: "transport endpoint is not connected" (ENOTCONN) on a FUSE path means the userspace filesystem process behind the mountpoint is gone while the kernel's mount record remains. The kernel has nobody to forward operations to, so every stat, open, or readdir on the mountpoint fails with ENOTCONN. The mount cannot recover. The only useful operation left is unmounting it.

kubelet's volume teardown (UnmountVolume.TearDown) wants to unmount and then clean the pod's volume directories. Its filesystem calls on the dead mountpoint fail with ENOTCONN, teardown returns an error, and kubelet re-queues the operation. Per volume. Every ~2 minutes. Forever. It never falls through to "the endpoint is dead, just detach it". This is long-standing upstream behavior, not something unique to our stack: kubernetes#96361 and kubernetes#129550 both describe FUSE mounts, particularly in emptyDir volumes, that kubelet can never clean up.

Because teardown never completes, nothing downstream of it ever runs:

  1. The pod directory is never removed (447 of them).
  2. The pod's cgroups are never removed (464).
  3. cAdvisor housekeeping and kubelet's volume reconciler walk all of that garbage continuously, plus the retry loop itself fires hundreds of times per minute.

Each dead mount is individually harmless, a few log lines and a directory. The failure is the accumulation: the node crossed from "some noise in the journal" to "cannot start a container inside two minutes" gradually, over weeks, with no single triggering event. That made it invisible until a timeout fired.

The remediation

The fix for the backlog was to lazily detach every dead mount from the host mount namespace, enumerating from /proc/mounts. The simplified shape (each caveat is one of the traps below):

# inside the HOST mount namespace (nsenter -t 1 -m; see trap 1)
awk '$1 == "orlop" { print $2 }' /proc/mounts   # enumerate; do not glob (trap 2)
umount -l "$mountpoint"                          # repeat until gone (trap 3)

We detached all 443. Kubelet drained its own backlog with no restart needed. Within about five minutes:

Metric Before ~5 minutes after
Dead orlop FUSE mounts 443 0
Orphaned dirs under /var/lib/kubelet/pods 447 21
Cgroups under kubepods.slice 464 23
UnmountVolume.TearDown errors (per 2 min) 423 0
k3s-agent CPU (cores) 0.85 0.18
Load average 1.5 0.33

The remaining ~21 directories and ~23 cgroups belong to live pods. On the CPU numbers: 0.83 cores was the average we measured during diagnosis; 0.85 to 0.18 is the spot reading immediately before and after the detach.

The three traps that cost the most wall-clock time

The diagnosis took far longer than it should have, because three separate behaviors each lie to you.

Why doesn't umount work from kubectl debug node?

kubectl debug node/... --profile=sysadmin followed by chroot /host feels like a root shell on the node. For mount operations it is not. The debug pod receives a recursive bind copy of the host's mount table inside its own mount namespace. umount there succeeds, returns zero, and changes nothing on the host. It is a silent no-op, and we spent real time believing mounts were gone that were not.

Rule: mutating the host mount table requires entering the host mount namespace, nsenter -t 1 -m, not chroot /host. The sysadmin debug profile has host PID, so PID 1 is reachable.

Why can't ls or shell globs find a dead FUSE mount?

Glob expansion lstats candidate paths. On a dead FUSE mountpoint, lstat returns ENOTCONN, and the shell treats the path as nonexistent. A command like umount /var/lib/kubelet/pods/*/volumes/*/agent-disk quietly skips exactly the mounts you are trying to kill, while matching any healthy ones. find behaves no better.

Rule: enumerate dead FUSE mounts from /proc/mounts or /proc/self/mountinfo, never from directory listings or globs.

Why is the path still broken after a successful umount?

Mounts stack. The same path can carry two or more mount records, for example when a replacement pod mounted over a predecessor's corpse. One umount pops the top record and reveals the one beneath it, which is just as dead.

Rule: unmount in a loop until the path no longer appears in /proc/mounts.

The durable fix: a janitor in the CSI node driver

A one-off cleanup would leave the leak in place, and the next four quiet weeks would rebuild the pile. So the durable fix is a janitor inside fuse-csi-node, the CSI node driver DaemonSet we already run on agent nodes (shipped as PR #786, about 175 lines with comments).

Every 60 seconds, plus once at startup so a plugin restart cleans up the mounts its own death orphaned, the janitor:

  1. reads /proc/self/mountinfo and collects mounts whose source is orlop under /var/lib/kubelet/pods/;
  2. stats each mountpoint;
  3. lazy-detaches (MNT_DETACH) any that answer ENOTCONN.

The DaemonSet mounts /var/lib/kubelet/pods with Bidirectional propagation, so it sees the host's mounts in its own mountinfo and its detaches propagate back to the host. The same propagation property that lets these mounts leak into the host mount table is what lets an ordinary container clean them up.

The safety argument is the part worth stealing. The janitor reaps a mount only when stat on it returns ENOTCONN. A dead endpoint serves nothing and holds no unflushed state, so a lazy detach is safe by construction. A live mount stats fine and is never touched. A hung-but-alive mount blocks the stat; probes therefore run in a goroutine per path, with an in-flight set so repeated sweeps do not pile goroutines onto the same hung path, and hung mounts are deliberately left alone (a lease-takeover mechanism owns that case, and a janitor guessing about hung-but-alive filesystems is how you cause the outage you were preventing).

Why not fix kubelet instead? Upstream has known about the class of problem for years, the issues are open, and the fix is genuinely hard to do generally (kubelet cannot know that an arbitrary wedged filesystem is safe to force-detach). We can know that, for our own mounts, because we know their semantics. A small janitor we own, scoped to our mount source, was shippable in a day. The pattern generalizes to any FUSE filesystem that appears under /var/lib/kubelet/pods with a recognizable source name.

The second bug hiding in the same alert

While reading the same alert stream we found an unrelated bug with the identical signature. Deleting an agent immediately after creating it (something our e2e suite does constantly) raced the pre-warm boot: the delete path only tore down registered sessions, so a launch still in flight kept booting a pod whose disk access had already been revoked. That pod could never become ready, burned the full 120 seconds, and raised the same timeout alert. The fix (PR #788) registers a cancel function per in-flight launch, and delete now cancels it; a canceled boot logs at INFO instead of alerting, while genuine timeouts still do.

Worth stating because it is the postmortem cliché that keeps being true: one symptom, two causes. If we had deduplicated alerts by signature we would have shipped the janitor and kept paging ourselves.

What to check when this happens to you

  1. Journal for the two signatures: UnmountVolume.TearDown failed ... transport endpoint is not connected and cAdvisor's Housekeeping took longer than expected.
  2. grep fuse /proc/mounts on the node, and compare mount entries against actually-running pods.
  3. Count directories under /var/lib/kubelet/pods versus scheduled pods on the node.
  4. Count cgroups under kubepods.slice the same way.
  5. Remediate with nsenter -t 1 -m, enumerate from /proc/mounts, umount -l in a loop.
  6. Then automate it, or the pile comes back. Our janitor is small and the approach transfers.

FAQ

What does "transport endpoint is not connected" mean on a FUSE mount? The userspace FUSE server process behind the mountpoint has exited (crashed, was killed, or its pod died) while the kernel mount record remains. Every filesystem operation on the path returns ENOTCONN. The mount cannot recover; the only remaining useful operation is unmounting it. Your data is whatever the backing store holds; the dead mount itself holds nothing.

Why do Kubernetes pods get stuck terminating with FUSE mounts in emptyDir volumes? kubelet's volume teardown performs filesystem operations on the volume path. On a dead FUSE mount those fail with ENOTCONN, so teardown errors and retries indefinitely, and the pod's directories and cgroups are never released. See kubernetes/kubernetes issues #96361 and #129550.

Is umount -l (lazy detach) safe on a dead FUSE mount? If a stat on the mountpoint returns ENOTCONN, yes: the server side is gone, there is nothing left to flush, and lazy detach removes the mount record even if something still holds the path open. Do not lazy-detach mounts that merely hang; a hung mount may have a live server with unflushed state.

How do I find dead FUSE mounts on a node? Read /proc/mounts (or /proc/self/mountinfo) and stat each FUSE mountpoint; ENOTCONN marks a corpse. Do not rely on ls, find, or shell globs, which skip ENOTCONN paths as if they did not exist.