Benchmarks
Do idle Workspace agents incur compute charges?
Six native Workspace agents each remained idle for at least 30 minutes. Their idle windows incurred $0 in charges. Across 366 checks, no VM was present for those agents and their balances did not change.
Production release v349 · 2026-10-14, 06:21–08:04 UTC
What did the runs cost?
The test used three anonymous trials and three free accounts. Each sample completed a fixed-reply run, a run that executed sleep 65, and a wake run after the idle window.
Each sample paid for 1 / 2 / 1 started compute minutes. The run with a 65-second shell wait incurred 2 billed minutes. This v349 measurement on 14 October 2026 used $0.01 per started minute, with no monthly compute allowance. Compute costs $0.01 per minute, billed by the second.
Across 18 runs, 24 billed compute minutes cost $0.24. Model charges were $0.235, for a combined $0.475. These are usage charges before promotional credits.
Compute charges use two decimal places. Model and combined totals are rounded to three decimals, with trailing zeros omitted. Per-sample model charges use four decimals. Zero charges appear as $0.
Model charges vary with the resolved model. Their differences do not measure idle billing. The table separates model and compute charges, in USD. Short / long / wake timings are measured compute seconds, rounded to two decimal places.
| Sample | Account | Compute seconds: short / long / wake | Billed minutes | Long-run end to VM exit (s) | Idle (s) | Compute USD | Model USD | Total USD |
|---|---|---|---|---|---|---|---|---|
| s05 | anonymous | 2.13 / 68.39 / 3.56 | 1 / 2 / 1 | 1.419 | 1,802.02 | $0.04 | $0.0020 | $0.042 |
| s06 | free | 1.76 / 68.64 / 4.55 | 1 / 2 / 1 | 1.978 | 1,802.08 | $0.04 | $0.1124 | $0.152 |
| s07 | anonymous | 2.34 / 68.55 / 3.56 | 1 / 2 / 1 | 1.598 | 1,802.02 | $0.04 | $0.0020 | $0.042 |
| s08 | free | 3.93 / 71.14 / 6.35 | 1 / 2 / 1 | 1.489 | 1,802.00 | $0.04 | $0.0047 | $0.045 |
| s09 | anonymous | 1.99 / 68.48 / 3.66 | 1 / 2 / 1 | 1.308 | 1,802.02 | $0.04 | $0.0020 | $0.042 |
| s10 | free | 2.72 / 73.71 / 4.82 | 1 / 2 / 1 | 1.312 | 1,802.01 | $0.04 | $0.1124 | $0.152 |
What changed during the idle windows?
Every sample had 61 checks at a 30-second cadence, including both endpoints: 366 checks in total. Balance, usage-event count, and ledger count remained unchanged in every idle window.
The checks found 0 agent VMs, 0 new usage events, and 0 new ledger entries. Idle charges were $0. API balances matched database balances.
Observed idle windows lasted 1,802.00–1,802.08 s. Polling proves VM absence at those sampled instants. VM exit logs and closed compute intervals provide separate evidence that the runs stopped.
Every run had a settled compute row and a closed compute interval with driver stop proof. Each charge matched started-minute rounding and its corresponding ledger debit.
When did the VMs exit?
Across 18 runs, VM exit after run end was p50 1.739 s, p90 2.249 s, and max 3.007 s.
The exit time comes from the embedded Firecracker timestamp, not the log ingestion timestamp. Percentiles use linear interpolation between closest ranks. The p90 describes this sample. We do not estimate p99.
Cross-host clock skew was not calibrated. The raw run data retains negative driver-stop deltas, but we do not use them as a latency claim.
Which agents and conditions does this cover?
These were native Workspace agents, the storage type used by new agents. After each run, the VM exits. No VM stays assigned to the agent throughout the idle window.
Agents not yet adopted into Workspaces follow the older idle-stop lifecycle. This test does not measure them.
The measurement covers 6 samples, 1 observed node, and 30-minute idle windows. The three runs from each account are dependent observations. There is no population-tail estimate.
The driver used one host on the public internet, through Cloudflare. At most one idle-billing run was active at a time. Concurrent startup measurements and canary traffic shared the production fleet and could affect timing.
No scored sample was excluded. This test does not establish purchased-storage charges or idle behavior across the full fleet.
How can you inspect and reproduce the results?
The downloads contain the sample summaries, individual runs, usage events, idle observations, and ledger entries. Sample labels identify test cases. They do not disclose account or infrastructure identifiers.
The measurement used scripts/bench/sleep-live.py and scripts/bench/sleep-score.js. scripts/bench/sleep-analyze.py validates the idle observations and billing reconciliation. scripts/bench/sleep-report.py produces the evidence report.
Run this command from the repository root to regenerate the page JSON and both download locations from the committed results. The repository is private. Public downloads support independent inspection, but do not include the live harness or private infrastructure.
python3 scripts/bench/sleep-publish-v349.pyData license: CC BY 4.0. Use the dated downloads to cite this measurement.
- 2026-10-14 data.csv · Latest data.csv
- 2026-10-14 runs.csv · Latest runs.csv
- 2026-10-14 events.csv · Latest events.csv
- 2026-10-14 observations.csv · Latest observations.csv
- 2026-10-14 ledger.csv · Latest ledger.csv
- 2026-10-14 data.json · Latest data.json
Field dictionary
- sample / account_kind / plan
- Test case label, account type, and plan. These labels are not account identifiers.
- phase / measured_compute_seconds / billed_compute_minutes
- Run task, measured compute interval, and charged whole minutes. run_seconds is the elapsed run duration, which has different boundaries.
- compute_usd / model_usd / total_usd
- Compute, model, and combined usage charges in USD, before promotional credits.
- idle_start_utc / idle_end_utc / idle_seconds
- First and last idle-check timestamps and elapsed seconds between them.
- idle_usage_events / idle_ledger_entries / idle_usage_usd / idle_ledger_delta_usd
- New events, new ledger entries, usage charges, and signed ledger change during the idle window.
- idle_vm_checks / vm_present_checks / vms_present
- Number of checks, checks that found an agent VM, and VM count at each observation.
- post_run_to_absence_seconds / vm_exit_after_end_seconds
- Embedded VM exit timestamp minus run end. The summary field refers to the long run. It is not the first polling observation.
- driver_stopped_after_end_seconds
- Driver stop timestamp minus run end, with negative values retained and no clock-skew correction.
- check / event / elapsed_seconds / usage_events / ledger_entries / balance_usd
- Idle-check sequence, marker, timer seconds, cumulative counts, and observed balance.
- created_at / kind / name / tokens / amount_usd / delta_usd
- Event timestamp, usage category, metered name, token count, usage charge, and signed ledger amount. Negative ledger amounts are debits.
The 2026-09-08 dataset measured the retired pod and allowance lifecycle. The earlier archived CSV also remains downloadable. These are separate historical measurements.
Related: Workspace save time, agent wake latency, what a task costs, and pricing.