Benchmarks

What does saving an agent’s files cost on v349?

After a 64 MiB run, the saved revision was ready in 4,076 ms at p50, with a maximum of 4,431 ms. Each size has 5 observations. Saving the Workspace after a run is the persistence cost measured here.

Production release v349 · 2026-10-14, 12:16–12:42 UTC

How long until the saved revision is ready?

The save starts after the run ends. These samples place it outside the recorded run and compute intervals.

Run end to revision ready measures the wait after the run. Save phase total measures the save operation itself. They have different timing boundaries.

Each size has n = 5. The table shows p50 and max in milliseconds. We omit p90 and p99 at this sample size.

The payload contains random bytes from os.urandom, flushed with fsync. Saved-revision manifests confirm the byte counts and distinct payload hashes for each round. Added bytes include repository metadata. We did not download and independently hash the payload.

The phase components include parallel and child operations. Do not sum them to reconstruct the total. A missing component is unobserved, not zero.

Workspace saves by payload size, p50 and maximum milliseconds
Payload MiBPlannRun end to ready: p50 / max msSave total: p50 / max msAdded bytes: p50 / max
1free51,769 / 1,8511,865 / 1,9891,053,760 / 1,076,168
64free54,076 / 4,4314,299 / 5,54267,117,199 / 67,139,414
256power58,434 / 9,0918,749 / 9,233268,452,240 / 268,475,255
1,024power523,042 / 24,88223,198 / 25,0191,073,793,111 / 1,073,817,276

Which runs repeated the save script?

At 64 MiB, round 3 executed the script twice. At 1,024 MiB, rounds 4 and 5 executed it twice. Each repeat overwrote the same payload path. The measurements retain the final saved file.

The split below gives sample counts and the full timing range for each execution count. The main table includes all final-file saves. These groups are too small for a percentile comparison.

The 1,024 MiB throughput run in round 1 appended an exit-status echo. It reported exit=0 and did not modify the timed CSV. No scored round was excluded.

Save observations split by script execution count
Payload MiBExecutionsnRun end to ready: min–max msSave total: min–max ms
64143,660–4,1873,773–5,542
64214,431–4,4315,079–5,079
1,0241321,685–24,88221,791–25,019
1,0242221,582–23,04221,690–23,198

What do the in-VM disk timings measure?

Both /workspace and /tmp use node-local ext4 inside the VM. /workspace uses Firecracker Writeback caching. /tmp uses Unsafe rootfs caching: its guest fsync does not flush the host.

O_DIRECT bypasses only the guest cache. It does not establish a cold host read. These are cache-mode and dd process timings, not object-storage throughput. The save measurement above measures persistence work.

Each cell shows median [min, max], with n = 5 per size and path. Values are rounded to one decimal place. Throughput uses decimal MB/s. Small-file operations use 500 files of 4,096 bytes.

The 64 MiB /workspace writes formed two groups: rounds 1–3 measured 180.7 / 185.5 / 184.7 MB/s. Rounds 4–5 measured 30.5 / 59.8 MB/s. The cause is not established.

A canary run overlapped round 5 in the recorded timestamps. Round 4 preceded that canary. We did not calibrate cross-host clock skew, so timing overlap alone does not establish a cause.

The subsequent buffered read does not prove cache residency. The 1 MiB case is sensitive to process overhead. The throughput payload is zero-filled, while the save payload is random.

In-VM disk operations, median and full range per size and path
CasePath / cacheWrite MB/sGuest-direct read MB/sSubsequent buffered read MB/sCreate files/sWalk msDelete ms
1 MiB / free/workspace Writeback102.2 [75.0, 131.1]637.7 [566.9, 681.2]972.3 [802.4, 1,055.8]1,029.9 [1,015.7, 1,032.7]0.9 [0.8, 0.9]5.9 [4.2, 7.5]
1 MiB / free/tmp Unsafe159.9 [148.4, 554.5]702.2 [537.9, 727.3]1,043.8 [989.5, 1,077.1]1,212.4 [1,189.8, 1,283.2]0.9 [0.8, 0.9]3.5 [3.3, 4.4]
64 MiB / free/workspace Writeback180.7 [30.5, 185.5]1,538.7 [1,371.2, 1,964.1]20,999.2 [5,245.9, 21,582.1]1,038.1 [172.3, 1,051.0]0.8 [0.8, 1.2]5.2 [3.9, 28.8]
64 MiB / free/tmp Unsafe1,451.2 [92.0, 1,795.5]1,496.8 [720.7, 1,762.4]20,133.0 [12,361.8, 22,277.7]773.1 [672.2, 1,252.7]1.0 [0.8, 1.3]3.5 [3.3, 6.5]
256 MiB / power/workspace Writeback184.7 [163.9, 189.3]1,782.7 [1,552.4, 1,907.0]29,663.2 [27,249.3, 31,025.0]1,025.5 [1,017.7, 1,042.3]0.8 [0.8, 0.9]4.0 [3.7, 6.1]
256 MiB / power/tmp Unsafe1,805.4 [1,127.0, 2,254.9]1,876.3 [1,519.2, 2,033.8]30,377.5 [28,072.7, 30,720.3]1,309.7 [892.1, 1,337.2]0.9 [0.8, 1.0]3.3 [3.3, 5.4]
1,024 MiB / power/workspace Writeback180.2 [102.3, 263.7]1,141.4 [1,042.9, 1,259.3]1,423.7 [847.1, 1,800.4]1,032.5 [947.8, 1,051.6]0.9 [0.8, 1.0]4.2 [3.8, 7.5]
1,024 MiB / power/tmp Unsafe347.0 [273.4, 537.1]894.6 [639.4, 1,675.9]1,288.9 [1,101.8, 1,574.8]1,304.1 [1,181.6, 1,305.9]0.9 [0.8, 1.1]4.0 [3.7, 5.3]

What does this measurement leave open?

The driver used one host on the public internet, through Cloudflare, with one active disk-benchmark run at a time. All 40/40 pool claims were hits. Other production traffic could affect the timings.

Fresh VMs do not prove cold host caches. Plans and disk capacities differ across sizes. The 256 MiB and 1,024 MiB cases used power plan accounts. No restore after a cold copy was measured.

The finite sample does not establish population tail latency or a save-failure rate. No result is pooled across sizes, plans, or cache modes.

How can you inspect and reproduce the results?

The downloads contain each save observation, the guest measurements, and the report data. Sample labels and round numbers are test labels. Infrastructure and account identifiers are omitted.

The measurement used scripts/bench/disk-save.sh and scripts/bench/disk-sample.sh. scripts/bench/disk-analyze.py produces the aggregate data. scripts/bench/disk-report.py produces the evidence report.

Run this command from the repository root to regenerate the page JSON and both download locations from the committed results. The repository is private. Public downloads are sufficient to inspect the measurements, but do not include the live harness or private infrastructure.

python3 scripts/bench/disk-publish-v349.py

Data license: CC BY 4.0. Use the dated downloads to cite this measurement.

data.csv records size, plan, round, added bytes, save timings, and script executions. guest.csv records each guest metric and its unit. Its historical tier names map persistent to /workspace and ephemeral to /tmp. data.json contains the report aggregates. Null percentiles are not reported at these sample counts.

The previous page measured FUSE storage against an ephemeral container filesystem. Its 2026-07-25 CSV and JSON remain available as a separate measurement.

Related: idle compute billing, agent wake latency, and what a task costs.