What a Kubernetes warm pool cannot prewarm for a stateful AI agent

TL;DR. Prewarm the runtime, not the tenant. A Kubernetes warm pool can pay for image pulls, container startup, runtime initialization, and even blank storage before a request arrives. It cannot safely guess which existing agent disk, tenant identity, or final credentials the next request will need. Those belong behind an atomic, claim-once transition. If that late binding fails, the correct fallback is the normal cold path, not a partly claimed sandbox.

A warm pool sounds simple when the workload is stateless: start more identical replicas than you currently need, then route traffic to one. A stateful AI agent changes the question. The next request may need a particular workspace, a particular security principal, and authority limited to one account. Starting the process early helps, but binding any of those things early can give the wrong user the right computer.

We hit this boundary while reducing agent wake latency. The useful design was not "make the whole agent warm." It was to split activation into a tenant-agnostic phase that can run ahead of demand and a tenant-bound phase that must wait for a claim.

Prewarming is four different operations

Treating "the sandbox" as one object hides the security boundary. There are at least four classes of state:

State	Safe to create before a claim?	What the pool actually buys
Runtime and image	Yes	Image pull, container start, agent runtime initialization
Blank resources	Usually	Empty volumes, network allocation, generic capacity
Existing persistent state	No, not generically	The specific agent workspace must be selected and attached
Tenant identity and final credentials	No	Authority must be derived from the claimant and bound to one sandbox

The second row matters because "storage cannot be prewarmed" is too broad. Kubernetes can provision blank storage ahead of time. The current Agent Sandbox API even supports storage customization through claims, and recent releases added volume claim templates to that path. A blank volume is still tenant-agnostic. A returning agent's existing disk is not.

The distinction is provenance, not medium. An empty persistent volume can sit in a pool without knowing its future owner. A volume that already contains an agent's repository, artifacts, and history carries identity even if Kubernetes represents both resources as PVCs.

Prewarm everything that has no tenant meaning

The safe half of a warm pool should be deliberately boring. Pull the image. Start the container. Initialize language runtimes. Load immutable libraries. Establish generic network plumbing. Create an empty workspace if the product can assign fresh storage to a new agent.

This is where most cold-start time is removable. None of these operations needs to know who will make the next request, and every result can be discarded if it becomes stale. That gives the pool a clean replacement rule: a sandbox with the wrong image version or failed health check is deleted, not repaired while a claimant waits.

Kubernetes SIG Apps now expresses this pattern directly through Agent Sandbox: a SandboxWarmPool maintains ready sandboxes and a SandboxClaim acquires one. As of July 2026, the project is moving quickly and its current releases use v1beta1 APIs. The abstraction is useful, but it does not remove the need to define what "ready" means for your application.

For a stateful agent, ready should mean "capable of being bound," not "already impersonating someone." A health check can prove the runtime accepts a claim. It should not require a real user's disk or credentials to do so.

The claim is an isolation boundary

At claim time, the platform finally knows the requesting tenant and the target agent. It can select the existing workspace, attach it, establish the sandbox's final identity, and deliver narrowly scoped authority. These operations belong in one logical transition:

tenant-agnostic prewarm -> atomic claim -> tenant-bound attach

The transition must be claim-once. After a sandbox accepts one tenant, it never returns to the generic pool. Deleting it is cheaper than proving that every process, file, socket, cache entry, and in-memory secret has been scrubbed. More importantly, destroy-on-release makes isolation review finite. One sandbox has zero tenants before claim and exactly one afterward.

This is not merely a scheduling optimization. Two control-plane workers may race for the same ready sandbox. Labels, caches, and list results can all be stale. The sandbox itself therefore needs an authoritative one-way state transition that accepts one claimant and rejects the rest. Relabeling or removing it from the pool is still useful for discovery, but discovery metadata should not be the final arbiter.

Identity deserves the same late-binding treatment. A pool can contain a trusted mechanism that obtains credentials, but not the next tenant's final credential. The distinction is easy to miss: prewarming the broker is safe; pre-assigning the principal is not. The Agent Sandbox 2026 roadmap lists dynamic sandbox or pod identity association at claim time as planned work, which is a good signal that this remains an application and platform boundary, not a solved side effect of having a WarmPool CRD.

Existing state is attached, not prewarmed

A returning agent should see the same files after its previous compute disappeared. That continuity comes from storage outliving the sandbox, not from keeping one warm replica per agent. Our persistent-disk benchmark tests that distinction directly: the disk survives replacement of the machine underneath it.

At activation, a pooled runtime attaches the selected agent disk. This keeps the expensive generic startup work off the request path while leaving the user-specific mount where it belongs. It also makes the remaining latency honest. A warm pool can remove image and process startup, but it cannot make tenant-specific authentication and attachment take zero time.

This model also prevents a misleading capacity calculation. A pool of ten blank sandboxes is ten units of ready compute. It is not ten warmed copies of every user's state. If your product requires memory snapshots or per-agent local caches, those are a different tier with different invalidation and isolation costs. Do not hide them inside the word "warm."

The cold path is the correctness oracle

Late binding adds new failure modes: the chosen sandbox may die, lose a race, reject the claim, or fail while attaching state. The warm path is an optimization, so none of those failures should make an activation fail when the ordinary cold path would have worked.

The rule we use is:

1. Claim a ready, tenant-agnostic sandbox.
2. Bind identity and attach state.
3. Publish it as available only after both are ready.
4. If any step fails, discard that sandbox and start through the cold path.

Do not return a claimed sandbox to the pool after an ambiguous failure. The control plane may not know whether tenant data reached it. Also do not report success after only the claim acknowledgement. Availability means the agent can access the correct workspace under the correct authority.

This fallback has an operational benefit beyond reliability. It gives every warm-pool change a simple invariant: disabling the pool may increase latency, but it must not change which tenant gets which state or whether a valid activation succeeds.

Limitation

This design does not eliminate all cold starts. Pool depletion still exposes the normal boot path, and tenant-specific disk attachment and identity issuance remain on the claim path. It also assumes a disposable runtime whose durable truth lives elsewhere. If the only valid state is an in-memory process image, a generic claim-once pool is the wrong abstraction; snapshot restore or dedicated suspended sandboxes may fit better, with a larger security and invalidation surface.

FAQ

Can a Kubernetes warm pool prewarm persistent volumes?

It can pre-provision blank persistent volumes. That is different from attaching an existing tenant's agent disk. The former is generic capacity; the latter contains tenant-bound state and should be selected at claim time.

Why not scrub a sandbox and return it to the pool?

Because complete scrubbing is hard to prove across processes, memory, filesystem state, caches, sockets, and credentials. Claim-once plus destroy-on-release turns a broad sanitization problem into a much smaller lifecycle rule.

Should credentials ever exist in a warm sandbox?

A generic sandbox may contain a trusted credential acquisition mechanism. It should not contain the final credential or principal for an unknown future tenant. Bind that authority only after the claim identifies the tenant.

What should happen when late binding fails?

Discard the ambiguous sandbox and use the same cold activation path that works without a pool. A warm pool is allowed to improve latency, not to weaken isolation or availability.