Why we put a server between our AI agents and JuiceFS

TL;DR. In JuiceFS Community Edition, the client is the filesystem: there is no server process, the metadata engines are database drivers, and every enforcement point (--subdir, directory quotas, POSIX ACLs) executes inside the client process. That's a coherent design for its trust model, where you control every mount host. Our mount hosts run untrusted AI agent code, so we put a trusted server (orlop, open source) between agents and JuiceFS: agents hold no JuiceFS credentials of any kind, they speak mTLS to orlop-server, which holds the only JuiceFS mount and confines each connection to that agent's path prefix. JuiceFS's own answers for untrusted clients (S3 Gateway IAM, Enterprise Edition access tokens) have the same shape: a trusted process in front handing out scoped access.

Why this matters to us

plori gives every AI agent its own computer, and the disk in that computer is a FUSE mount backed by JuiceFS. We run JuiceFS in production and it's working well for us (well enough that we moved its metadata engine to Redis rather than replace it).

But our machines run untrusted code, which puts us in the middle of a question that has come up in JuiceFS threads for years: can you mount it on machines you don't trust? An HN commenter comparing it with SeaweedFS put it precisely back in 2023: "the machine where you mount the filesystem talks to the metadata and data backends directly; this means you can't mount a filesystem on an untrusted machine". Correct, and not a criticism; that deployment just isn't what CE is for. This post is about where the trust boundary sits and what we run instead.

Is --subdir a security boundary?

No, and it doesn't claim to be. JuiceFS CE has no server process in the data path: the metadata engine is a database (Redis, MySQL, Postgres, TiKV, and others) that the client connects to directly, and the client executes the metadata transactions itself, from inode allocation to permission checks to quota accounting. So every enforcement mechanism, --subdir included, runs inside the client process.

--subdir is a client-side chroot. The client resolves the subdirectory to an inode and uses it as the root for everything that follows; the SDK and the S3 gateway add a path-prefix check on top. A client you control can be pointed at a subdirectory and will stay there. A client someone else controls picks whatever root it likes. It's a namespacing convenience, honest about what it is.

What can a client with the metadata URL do?

Everything. Suppose we had mounted JuiceFS directly inside agent pods. Every agent would then effectively hold the metadata URL (something like redis://meta.example:6379/1), because a CE mount cannot exist without it.

What does that URL get you? The filesystem's Format JSON is stored in the metadata engine, and it carries the object storage AccessKey and SecretKey. They're encrypted at rest, but the cipher key is derived from the volume UUID, and the UUID sits in plaintext in the same JSON. That protects against casual reads, not against someone with access to the metadata engine, and Juicedata doesn't claim otherwise: the official CLI will hand you the credentials with juicefs dump --keep-secret-key. (The default omits them for exactly this reason, which is fair-minded design.)

So in that hypothetical deployment, every agent would effectively hold the object storage credentials for every other agent's data. Directory quotas and POSIX ACLs wouldn't change that: they're library code linked into the client, a modified client simply doesn't call them, and there's no server to call them instead.

To be clear about what this is and isn't: this is JuiceFS working as designed, for a deployment where every mount host is trusted. Our premise is different, because our mount hosts run code we didn't write. That's a trust-model difference, not a configuration gap.

Doesn't the Kubernetes CSI driver fix this?

It changes who mounts, not who's trusted. The CSI driver moves the JuiceFS client into dedicated mount pods, so application pods never see the credentials directly. But the mount pods hold the metadata URL and the object storage secrets, and the driver deliberately shares one mount pod between PVs that differ only by subPath (the subpath is excluded from the pod-reuse hash).

That's a sensible resource optimization. It also tells you the credential granularity: one mount pod, one whole-filesystem credential, many tenants' PVs. Path confinement comes from kubelet bind mounts on the node. Per-tenant subPath PVs are namespacing, not isolation.

Had we run agents this way (we don't, this is the second rejected hypothetical), a container escape on any agent node would reach a mount pod holding whole-filesystem credentials, and through it every tenant's data.

Why not one filesystem per agent?

Juicedata's own multi-tenancy writeup for AI inference recommends per-tenant filesystems, so this is the fair alternative to price out.

Each filesystem needs its own metadata database, and each mounted filesystem is its own mount pod, defaulting to 1 CPU / 1Gi requests. At those defaults an 8-vCPU node fits about 7 mount pods before you've scheduled a single agent. The requests are tunable (we tune them), but no tuning removes the operational cost of one metadata engine and one mount lifecycle per agent.

We hit the resource math early on: on a small node, the default mount pod requests squeezed CoreDNS out and cluster DNS blipped. The fix is setting the StorageClass's mount CPU request.

Quotas deserve a paragraph too. The CSI driver's quota setup floors at 1GiB and truncates fractional values (1.5GiB silently becomes 1GiB; the core CLI itself accepts --capacity 128M). And if the juicefs binary in the mount image predates quota support, the driver logs, skips the quota, and reports the volume provisioned. The one failure mode that hides is a PV that silently has no quota, which is why our application-layer accounting backstops every quota anyway.

What JuiceFS actually offers for untrusted clients

JuiceFS is not naive about any of this. The S3 Gateway (v1.2+) has real server-side IAM: per-user credentials and policies, managed with the standard MinIO admin tooling. The gateway process holds the filesystem credentials; untrusted clients get scoped S3 credentials and never see the metadata URL. If your untrusted clients can speak S3, this is a genuine answer.

Ours can't. Agents run shells, git, SQLite, editors; they need rename, append, mmap, and file locks, so S3 semantics rule the gateway out for us.

Then there's the commercial product. JuiceFS Enterprise Edition is reported to support access tokens that restrict a client to a subdirectory, an IP range, or read-only access, enforced by its closed-source metadata service. We have not operated EE; this is from Juicedata's published materials, not first-hand testing.

Both answers have the same shape: a trusted process holds the whole-filesystem credentials and hands scoped access to untrusted clients. That's the architecture JuiceFS itself reaches for when clients can't be trusted, and it's what we built too, speaking POSIX instead of S3.

What we built: a server in front

orlop is our open-source storage plane. orlop-server is the only process that mounts JuiceFS and the only holder of its credentials. Agents run a small FUSE client that speaks mTLS to orlop-server; agents hold no JuiceFS credentials of any kind.

Each agent enrolls and receives a short-lived client certificate (1 hour TTL) carrying SPIFFE SANs for its tenant and agent IDs. There's one shared CA, and the SAN is the authorization input (no per-agent CAs). A certificate without an agent SAN is rejected at the door. The server reads the agent SAN and confines that connection to the agent's path prefix. The checks run in a process the agent cannot modify; the isolation boundary is on our side of the TLS connection.

A side benefit of per-agent tenancy: re-homing an agent to another owner is a metadata operation instead of a data copy.

None of this is an invention. SPIFFE identity is standard practice; the part worth copying is one shared CA with the SAN as the path scope and the certificate TTL as the lease.

Notes from running it: the official mount image's juicefs binary is dynamically linked and exits 127 silently on a distroless base, so we download and verify a truly static binary at build time. Write-through means every fsync is a synchronous object PUT, so cold starts want --writeback. For what the per-operation metadata hop costs and how we shrank it, see the Redis migration post.

Limitations

• We have not operated JuiceFS EE; that comparison is from Juicedata's published materials.
• Our design costs us a custom FUSE client and a server hop on every file operation. The Redis post covers what that hop costs and how we shrank it.
• orlop's POSIX surface has known, deliberate gaps, documented in the pjdfstest report in the orlop repo.
• This describes JuiceFS CE and its CSI driver as of July 2026; both projects move.
• We make no security-guarantee claims here. We're describing where enforcement lives, not promising it can't fail.

FAQ

Is JuiceFS --subdir a security boundary? No, and JuiceFS doesn't claim it is. It's a client-side chroot: the client process assigns a different root inode and filters paths by string prefix. Against a hostile client it's a namespacing convenience; confinement has to live in a process the client can't modify.

Can I mount JuiceFS Community Edition on machines I don't trust? Not safely, by design. A CE mount needs the metadata URL, and metadata access implies the object storage credentials for the whole filesystem. CE's trust model assumes mount hosts are trusted; for untrusted hosts, put a trusted process in front (JuiceFS's S3 Gateway, EE access tokens, or your own server).

How does plori isolate each agent's disk on top of JuiceFS? Agents never hold JuiceFS credentials. Each agent's FUSE client speaks mTLS to orlop-server using a short-lived certificate whose SPIFFE SAN names the agent; the server holds the only JuiceFS mount and confines each connection to that agent's path prefix. orlop is open source.

Doesn't JuiceFS Enterprise Edition or the S3 Gateway already solve this? For some workloads, yes. The S3 Gateway adds server-side IAM if your clients can speak S3; EE is reported to enforce per-token subdirectory restrictions in its metadata service. We needed POSIX semantics (rename, append, mmap, locks) and per-agent credentials with an API-driven lifecycle, so we built a POSIX-speaking server in front instead.