How we store AI agent files: local disks and restic revisions

TL;DR. Each plori agent works on a plain local disk on the server that runs it. During a run, the platform saves a read-only version of that disk to object storage every minute, and again at the end. It accepts an agent's result only if no other result landed first. A crash can lose the work done since the last finished save.

plori keeps each AI agent's files on a local disk next to its microVM and saves read-only versions to object storage. We call this storage Workspaces. Why we stopped using JuiceFS explains why we changed. This post explains the design that replaced it.

Where does an AI agent's working copy live?

Each run gets its own disk image on the local disk of the server that runs its microVM. The image holds an ext4 filesystem. The Firecracker microVM sees it as a block device, separate from its root filesystem. The agent reads and writes an ordinary local filesystem. Writes do not go through FUSE or a network filesystem.

How do agents on the same server get their own copies?

Agents on the same server get reflink copies of one image. A reflink copy shares disk blocks with the original until one side writes, so making one copies no file data. The server keeps its images on an XFS filesystem, which supports reflinks.

When the server holds a verified image of the revision, each agent's copy starts as a reflink of it. A manager and its workers each get a private, writable disk without a byte copy. That shared image comes from a verified restore, never from another agent's private disk. The scheduler prefers a server that already holds the revision, but it can use another server.

How do we save a running agent's disk?

The platform freezes the agent's filesystem for a moment, copies the disk image with a reflink and lets the agent continue. The copy is consistent at the filesystem level. For an application, it looks like a crash at that moment.

Next, a separate helper VM reads that copy. Restic is an open-source backup tool that stores deduplicated, encrypted snapshots of a directory tree. The helper mounts the copy read-only and uses restic to upload it to the Workspace's own repository in object storage. Restic runs only in trusted platform code, never in the agent's VM.

The server's own kernel never mounts a filesystem that an agent wrote. The agent can write any bytes to its disk, and kernel filesystem code is not hardened against crafted disk images.

Our storage service then checks the uploaded snapshot, its tree and its file counts. Only a verified snapshot becomes a ready revision. A save with no changes reuses the existing revision.

Why is saving separate from accepting a result?

A save only stores a version, and accepting it is a separate step that changes which version is current. Postgres holds one pointer per Workspace to its current revision.

The platform accepts with a compare-and-swap. This is an update that succeeds only if the pointer still holds the value the caller expected. If another result moved the pointer first, the update changes nothing and the accept fails. Before the update, the platform checks again that the run may still publish. A save from a run that lost that right stays recoverable, but it cannot become current.

The platform accepts the manager agent's own saves automatically. It compares a worker's result with the current files, and the manager reviews it before it can become current.

How much work can a crash lose?

A crash can lose the work done since the last finished save. During a run, the platform tries to save a changed copy every 60 seconds. It saves again when the run ends and before a planned stop.

The work at risk is the save interval, plus the time the save takes, plus any failed attempt. Recovery rebuilds a lost copy from its last ready revision. It does not restore running processes or open files.

What happens when a run starts on another server?

On a server that has no copy, the platform can start the run before all file contents arrive. It creates every file and folder first. Then it fetches a file's content from the revision the first time the agent reads it. The platform uses this path only where it is faster than a full restore. A server can also build the file list before the run.

Case	Start p50 / max	Save p50 / max	n
Same server, 100 MB tree, no changes	229 ms / 268 ms	2.40 s / 2.73 s	24
Other server, 100 MB tree	541 ms / 654 ms	2.60 s / 3.32 s	11
56,000 small files, nothing on the server	1.85 s / 3.13 s	7.66 s / 8.57 s	10
56,000 small files, file list built ahead	830 ms / 891 ms	7.73 s / 9.60 s	10 starts, 9 saves

These are staging measurements on one shared 16-core development host that also ran builds and tests. Start runs from the accepted request until the agent can run its first tool, without model time. Save runs from the end of the run until the remote save is ready. The rows come from different rounds and builds with small samples. They are not a matched comparison, and we have none with JuiceFS.

Fetching content later still costs time. In one staging round, reading a whole 56,000-file tree took 9 min 58 s, because the agent's VM fetched each file on its first read.

What did we give up?

We gave up continuous replication, so a crash can lose the work since the last ready revision. A working copy lives on one server's disk, so unsaved changes must fit there. We also build restic from our own fork instead of using the upstream binary.

First saves of large new trees can be slow. Our target is 10 s from the end of a run to a remote save. In the three staging rounds behind our release decision, 14 of 18 first saves of a fresh 56,000-file tree met it. The slowest took 14.05 s. We planned to check this again in production.

What should you test if you build agent storage like this?

Test the slow and risky cases first:

1. The first save of a large, freshly written tree, not only later saves.
2. Your recovery point under server loss, including save time and failed attempts.
3. Which kernel reads a disk that untrusted code wrote.
4. A whole-tree read after a cold start on another server.
5. Two results that race to become current.