What does a task cost on plori? Two real tasks and parallel workers on v349
On production release v349, we ran two real tasks ten times each: run a pinned repository's test suite, and research a topic from several sources into Markdown notes. All 20 runs passed. 17 of the 20 tasks cost $0.25 or less. The model that served each run explains most of the spread in cost and time. We also measured N workers that each edit one Workspace, from dispatch until every change is accepted.
Prices below were measured on v349 on 14 October 2026 under started-minute compute billing.
Production release v349 · 2026-10-14 · tests 05:49:04–06:10:25 UTC · research 06:10:30–07:00:23 UTC · workers 14:19:13–15:01:27 UTC
What a task cost.
§01| Task (n = 10 each) | passed | run wall p50 / p90 / max | save p50 / max | cost p50 / p90 / max | total |
|---|---|---|---|---|---|
| Run a repository's test suite | 10 / 10 | 50.8 s / 91.8 s / 109.1 s | 2.40 s / 2.83 s | $0.0274 / $0.0404 / $0.1031 | $0.3083 |
| Research from several sources, saved as notes | 10 / 10 | 103.8 s / 464.3 s / 731.0 s | 1.65 s / 1.81 s | $0.0547 / $0.6230 / $1.7367 | $3.1186 |
Of the 20 tasks, 8 cost less than $0.03, 9 cost between $0.03 and $0.25, and 3 cost more than $0.25. So 17 of 20 tasks cost $0.25 or less, model, run time and tools included. 6 of the 10 test-suite runs cost $0.028 or less, and 5 of the 10 research runs cost $0.040 or less.
With 10 runs, the p50 is the midpoint between the 5th and the 6th cheapest run. For research these two runs cost $0.0395 and $0.0699, so the p50 is $0.0547. All 10 costs, from low to high:
- tests: $0.0140, $0.0142, $0.0145, $0.0165, $0.0270, $0.0277, $0.0288, $0.0290, $0.0335, $0.1031
- research: $0.0362, $0.0369, $0.0379, $0.0391, $0.0395, $0.0699, $0.1892, $0.4339, $0.4992, $1.7367
The test-suite run wall p50 of 50.8 s describes no run. The run times fall into groups by model. The four anthropic/claude-haiku-5.5 runs took 26.7 s to 31.5 s. The five openai/gpt-6-luna runs took 70.1 s to 109.1 s. The one anthropic/claude-sonnet-5.5 run took 20.0 s. The p50 falls in the gap between the haiku and the luna runs.
Run wall time is the server run time, from run start to run end. The save is the post-run save of the agent's files, from run end until the saved revision is ready. It comes after the run and is not part of the run time. Cost is the change in the account balance for the sample, which equalled the run's charges in all 20 samples. For what each plan includes, see pricing and how runs are billed.
The two tasks.
§02- Run a repository's test suite. Clone github.com/pallets/itsdangerous at tag 2.2.0 (commit 096c8d42545d3b68ea21a4f890fb2b2d8979c0bd). Install it and its test needs, run pytest, and write the counts to results.txt. Pass: the saved file reads passed=297 failed=0 errors=0 skipped=0.
- Research from several sources. Daylight saving time rules in the European Union, the United States and Mexico, and the laws passed or proposed since 2018. At least five sources, one Markdown file per region plus a summary, every source cited with its full URL. Pass: the saved notes exist, contain at least 5 distinct URLs, and name all three regions. This check tests the structure of the notes. It does not check that the cited facts are correct.
Every run passed on the first message, in one run, with no approval or question card.
Why the cost varies: the model draw.
§03Each run's executor model is drawn from a configured pool: gpt-6-luna 40%, claude-haiku-5.5 40%, claude-sonnet-5.5 10%, gpt-6.1-sol 10%. That draw explains most of the spread in cost and time. With n under 5, the table lists every value instead of a p50.
| Task | Model | n | cost | run wall |
|---|---|---|---|---|
| tests | anthropic/claude-haiku-5.5 | 4 | $0.0140, $0.0142, $0.0145, $0.0165 | 26.7 s, 27.9 s, 29.4 s, 31.5 s |
| tests | anthropic/claude-sonnet-5.5 | 1 | $0.1031 | 20.0 s |
| tests | openai/gpt-6-luna | 5 | p50 $0.0288 (min $0.0270, max $0.0335) | p50 86.4 s (min 70.1 s, max 109.1 s) |
| research | anthropic/claude-haiku-5.5 | 6 | p50 $0.0385 (min $0.0362, max $0.0699) | p50 94.4 s (min 82.6 s, max 105.8 s) |
| research | openai/gpt-6-luna | 3 | $0.1892, $0.4339, $0.4992 | 408.5 s, 434.6 s, 731.0 s |
| research | openai/gpt-6.1-sol | 1 | $1.7367 | 213.5 s |
In the research task, the model also decided how often the agent searched. Per run, gpt-6-luna made 23 to 52 metered web searches, claude-haiku-5.5 made 3 to 7, and the gpt-6.1-sol run made 8.
Where the money goes.
§04| Task, 10 samples | model | run time | tools | total |
|---|---|---|---|---|
| tests | $0.1583 | $0.1500 | $0.0000 | $0.3083 |
| research | $2.0986 | $0.4400 | $0.5800 | $3.1186 |
- In this v349 measurement on 14 October 2026, run time was charged per started minute ($0.01), so every short run paid at least $0.01. Run time costs $0.01 per minute, billed by the second.
- The research tools cost is web search only. Web fetches and the platform's own checks cost $0.
- One research run made one advisor call on claude-opus-5.5, which cost $0.1700. It is part of the model cost above.
- Prompt cache hit rate (cached prompt tokens over prompt tokens, executor rounds): tests 89.7%, research 81.8%. The single gpt-6.1-sol research run had 19.0%.
- VM claims: all 10 test-suite runs took the warm-pool VM. Of the research runs, 9 took the warm-pool VM and 1 booted a new VM.
Web search: billed cost and list price.
§05- The first 100 web_search calls each month are free, then $0.01 per call. A search that returns no results is not charged.
- All 10 research runs used one account, so they shared its 100 free calls. They made 158 metered searches. The free calls ran out during the 8th run, and 58 searches were billed: $0.5800.
- No single run made 100 searches (the most was 52), so each run on a fresh account would have paid $0 for search.
- At list price, the 158 metered searches cost $1.5800.
- The run events show 185 web search calls, 27 more than the metered count. Those calls have no usage record, which is consistent with the rule that a search with no results is not charged. We did not check this call by call.
N workers in one Workspace.
§06Each repeat creates a Workspace, saves a base file, and makes N copies of it. Then it starts N workers at the same moment. Worker k writes one file, out/worker-k.md (a heading and three bullets), and changes nothing else. When every worker has saved, the changes are accepted one at a time, each against the revision the previous accept produced. N = 1, 2 and 4 ran on free accounts and N = 8 on a power-plan account. Each N ran 3 repeats, one repeat at a time.
Every phase ends when the slowest worker reaches it. "Started" means the worker VM sent its first event. "Saved" means the post-run save is ready. "Accepted" means the last change is part of the Workspace. With 3 repeats per N, the table lists each repeat and gives no percentiles. Times are in seconds.
| N, plan, repeat | dispatch → all started | all started → all finished | all finished → all saved | all saved → all accepted | dispatch → all accepted | cost |
|---|---|---|---|---|---|---|
| N = 1, free, repeat 1 | 2.2 | 6.6 | 1.7 | 10.1 | 20.6 | $0.0130 |
| N = 1, free, repeat 2 | 1.8 | 4.4 | 1.7 | 8.6 | 16.5 | $0.0108 |
| N = 1, free, repeat 3 | 1.7 | 9.8 | 1.6 | 12.4 | 25.6 | $0.0119 |
| N = 2, free, repeat 1 | 5.0 | 5.4 | 2.2 | 19.6 | 32.1 | $0.0227 |
| N = 2, free, repeat 2 | 4.8 | 4.4 | 2.3 | 21.4 | 32.9 | $0.0224 |
| N = 2, free, repeat 3 | 4.5 | 4.5 | 1.8 | 17.1 | 27.8 | $0.0756 |
| N = 4, free, repeat 1 | 8.7 | 18.8 | 1.7 | 38.7 | 68.0 | $0.0483 |
| N = 4, free, repeat 2 | 7.0 | 38.0 | 1.6 | 36.7 | 83.3 | $0.0476 |
| N = 4, free, repeat 3 | 7.8 | 11.8 | 1.9 | 39.7 | 61.2 | $0.0464 |
| N = 8, power plan, repeat 1 | 16.0 | 26.6 | 2.9 | 80.1 | 125.7 | $0.0970 |
| N = 8, power plan, repeat 2 | 8.3 | 7.9 | 2.4 | 82.4 | 101.0 | $0.1233 |
The accepts are serial by design. Each change took 4.8 s to 10.7 s from submit to accepted, with a p50 of 9.9 s over all 45 changes. So the "all saved → all accepted" phase grows by about 10 s for each worker. All 45 changes were accepted on the first attempt, with no conflicts. In 12 of 12 repeats every worker's file was in the final Workspace.
| N, plan | VM claims: pool / new VM | prompt cache hit rate | total cost, 3 repeats |
|---|---|---|---|
| N = 1, free | 3 / 0 | 69.6% | $0.0357 |
| N = 2, free | 3 / 3 | 81.4% | $0.1207 |
| N = 4, free | 6 / 6 | 70.8% | $0.1423 |
| N = 8, power plan | 8 / 16 | 79.2% | $0.3840 |
- In this v349 measurement on 14 October 2026, run time was most of the cost: each worker run paid at least one started minute ($0.01). Copies, the task group, the changes and the accepts added no charge.
- The warm pool holds one ready VM, but in some repeats more than one worker was recorded as a pool hit. Under load some of those claims took up to 11.6 s. We did not trace why. Claims that booted a new VM took 2.6 to 10.8 s.
- The fleet started with one VM host. During N = 8 repeat 1, the autoscaler logged a scale-up at 14:30:31 UTC and created a second VM host at 14:30:34. The autoscaler logs no "ready" line. The first VM was placed on the new host at about 14:33:10, so it was ready within 2 min 36 s of creation. Repeat 1's eight workers had all been placed on the first host before the new host existed. The new host served repeats 2 and 3.
N = 8, repeat 3: three saves waited about 15 minutes.
§07| N, plan, repeat | dispatch → all started | all started → all finished | all finished → all saved | all saved → all accepted | dispatch → all accepted | cost |
|---|---|---|---|---|---|---|
| N = 8, power plan, repeat 3 | 37.7 | 27.0 | 908.2 | 82.9 | 1055.8 | $0.1637 |
All eight workers of this repeat ran on the new VM host. Four of the eight post-run saves could not start their save helper VM. The host's memory reserve was full: 6,723 MiB reserved plus 1,088 MiB for the helper exceeded the 6,910 MiB budget. One save was retried at once and committed. The other three stayed in the running state with no retry for about 15 minutes, then committed in 2.5 to 2.7 s. Each of those three saves took 935 to 939 s from run end to ready.
In the same minutes the autoscaler reported the host as empty and tried to remove it. Each drain stopped because three saves were still running. The three saves retried 2 s after one of those drains, and the host was removed 4 s later. The log order suggests that the drain released the stalled saves. It does not prove it. We are investigating this case. All eight changes were still accepted.
Not measured: a scheduled workflow task.
§08A third task, "build and activate a scheduled workflow that posts a Hacker News front-page digest", could not be completed on v349. We stopped it after two trial runs. First, the workflow's chat notice is a fixed status line, so it cannot carry a digest. Second, with the digest kept in the run output, the agent wrote a step without its required authentication field. The workflow engine rejected the step before it ran, but the error the agent received did not list the validation issue, and the agent's own validation check returned valid. The agent ran 11 tests, of which 10 were rejected, and activation was refused. That trial cost $0.225. These are known gaps that we are fixing.
Methodology.
§09- Production release v349, driven from one host on the public internet through the public REST API. The fleet was shared with real users, our hourly test probe and, during the task runs, another benchmark.
- Tasks: one new registered free account per task, marked as test traffic. Each sample is a new agent with its own Workspace on that account, so the 10 samples of a task share the account's monthly allowances. Samples ran one at a time.
- Workers: one account per N, three free and one on the power plan.
- Each test account received a $20 welcome credit. What a new account receives is on the pricing page.
- Task times come from the server's run record. Costs come from the account balance and the usage records. Worker dispatch is a client timestamp, and every later phase is a server timestamp. In the N ≤ 4 repeats the first worker's run was admitted 0.23 to 0.77 s after dispatch, so any clock difference is under a second.
- Percentiles use linear interpolation between closest ranks. p50 needs n ≥ 5 and p90 n ≥ 10. Smaller groups list their values.
- No sample or repeat was excluded.
Limitations.
§10- n = 10 per task with a four-model draw. The per-model rows have n = 1 to 6, and the pooled percentiles describe this draw, not a fixed model.
- Two tasks. This page does not measure longer tasks, tasks that need several messages, or other models.
- The worker edit is tiny on purpose, so the phases measure platform overhead (VM claim, save, compare, accept) more than model work. The accepts were driven through the public API about once every 0.5 s. An agent that accepts the changes itself adds its own model time.
- 3 repeats per N on a shared production fleet that started with one VM host. The N = 8 repeats ran on different fleets (one host, two hosts, mostly the new host).
- All numbers are from 2026-10-14 only.
- No cross-vendor comparison: this page measures plori against itself, under a stated method, and claims nothing about other platforms.
Reproducibility and data.
§11The scripts for this measurement are in scripts/bench/ of plori's private source repository:
- tasks-run.py and workers-run.py: the runs.
- tasks-analyze.py and workers-analyze.py: the statistics.
- tasks-lib.py: shared code.
- tasks-publish-v349.py: writes the files below and this page's data from the analyzer output.
Raw data, published under CC BY 4.0, without VM host names:
- tasks-tests.csv: test-suite task, one row per sample
- tasks-research.csv: research task, one row per sample
- workers-repeats.csv: N workers, one row per repeat
- workers-workers.csv: N workers, one row per worker
- workers-accepts.csv: N workers, one row per changeset accept
The same files stay at /benchmarks/task-cost/2026-10-14/. Every table on this page can be computed again from them. Reuse the data with attribution to plori.
plori engineering. The page changes only when new data is collected.
Related: agent wake latency, Workspace save time and idle compute billing.