Blog

How to secure a cloud AI agent that can run arbitrary code

· the plori team

The short answer: treat every process an agent can influence as attacker-controlled. Keep platform credentials and identity in a trusted broker. Give the sandbox only the smallest capabilities it needs for the current run, and enforce those limits outside the model. An environment-variable allowlist is useful hygiene. It is not a substitute for authentication or network policy.

We arrived at that answer after one of our agents printed a surprisingly good map of our Kubernetes cluster. No model-provider key or platform credential was exposed. The output still included internal service names, cluster IPs, an internal model endpoint, a Redis address, a debug MCP address, and run identifiers. A curious user had been handed the first page of a penetration tester's notebook for free.

After fixing it, we wrote down the review we should have run earlier. This guide is for hosted coding agents and general-purpose agents that can run shell commands, install packages, start servers, and make network requests. It is not a formal security standard. The exact controls will differ for containers, microVMs, and dedicated VMs, but the trust boundaries hold.

Start with a harder threat model

A code-running agent is an untrusted workload, even when the model and the user are both well behaved.

The model reads material the platform did not write: repositories, web pages, emails, issue comments, package metadata, and tool output. Any of that material can contain an instruction that changes what the model tries next. Prompt filtering and approval dialogs can reduce risk, but the sandbox still has to be safe when the model executes an unfortunate command.

We use this working assumption:

If a value or endpoint is available to an agent-controlled process, assume the agent can read it, copy it, transform it, write it to disk, and send it anywhere the network permits.

That assumption saves a lot of confused design discussion. It also separates assets that are too often lumped together as secrets:

Asset What the sandbox should receive Where enforcement belongs
Platform model-provider keys Nothing Trusted model broker
Agent and run identity A short-lived, resource-bound capability The service that consumes it
User-provided credentials Only when the task needs them, with the exposure stated plainly Secret broker and task policy
Internal service topology Only endpoints required by the runtime Network policy plus service authentication
Kubernetes or node identity Nothing unless the workload genuinely calls that API Pod spec, RBAC, and metadata controls
User files The storage scope assigned to that tenant Server-side storage boundary

Topology is not a credential. Leaking it is still useful reconnaissance. The right response is to stop handing out the map and to make sure every door on the map remains locked.

What actually leaked

Our executor process needs private configuration to do its job. Agent commands run below that executor. We had tried to separate the two by copying the executor's environment and deleting a short list of known credentials before launching a shell.

That is a denylist, so it aged badly. New internal variables appeared over time. None was individually alarming enough to trigger a review, but together they described the control plane, the data plane, and the current run.

Kubernetes added a second source of disclosure. By default, kubelet can inject service-link variables for active Services into every new Pod. A service named redis-primary can become variables containing its name, cluster IP, port, and protocol. Those values were never declared in our deployment manifest, which made them easy to overlook in code review.

The audit found a nastier adjacent bug in a plugin path. In Go, an os/exec command with a nil Env field inherits the complete parent environment. Our stdio MCP launcher built an environment slice from optional configuration. When there was no configuration, the slice remained nil. The code looked like it was passing an empty environment; the operating behavior was the opposite.

All of these failures came from ambient inheritance. We had not made the child process boundary explicit.

1. Put every agent subprocess behind one environment policy

The environment boundary should have one owner and one implementation. Do not keep a slightly different scrubber in the shell tool, terminal service, background runner, and plugin launcher. They will drift.

Our replacement is a small internal package with two policies:

  1. Agent commands receive a conservative compatibility set such as HOME, PATH, locale, terminal settings, temporary-directory settings, the outbound proxy, and a few documented runtime values.
  2. Untrusted stdio plugins receive an even smaller set plus their explicit configuration.

The code is deliberately dull:

func SandboxEnv(parent []string, user map[string]string) []string {
    env := make([]string, 0)
    for _, item := range parent {
        if allowed(variableName(item)) {
            env = append(env, item)
        }
    }
    return appendValidatedUserValues(env, user)
}

The loop matters less than its contracts:

  • Start from an allowlist. A new executor variable is private until someone deliberately adds it to the sandbox contract.
  • Return a non-nil empty slice when nothing is allowed. In Go, nil means inherit.
  • Reserve a platform namespace and reject user-supplied names in that namespace, case-insensitively.
  • Sort explicit values before appending them. Deterministic output makes tests and incident review easier.
  • Decide override behavior once. If user values may override PATH or proxy settings, make that a documented choice rather than an accident of append order.

Then inventory every place that starts a process. We found ordinary shell commands, context-aware execution, background jobs, a Python bridge, interactive PTYs, and stdio MCP servers. Build systems, package managers, user services, language kernels, git hooks, and debug consoles deserve the same search. A secure shell helper does not help if the terminal still calls exec directly.

For Kubernetes workloads, set:

spec:
  enableServiceLinks: false

Do this for every untrusted workload type, including workflow runners and user-started services. It removes the automatic service directory from the environment. It does not disable cluster DNS, block a guessed service name, or prevent a connection.

2. Remove platform secrets before the untrusted process exists

Calling unsetenv after startup is weaker than it looks on Linux. The proc filesystem exposes the initial environment of a process, subject to ptrace access checks. Removing a variable from the process's current environment does not rewrite that initial image.

If a trusted supervisor must receive bootstrap secrets through environment variables, one practical pattern is:

  1. Read the secrets before any untrusted child can start.
  2. Hand them across an anonymous file descriptor or another narrow in-memory channel.
  3. Re-exec the supervisor with a scrubbed environment.
  4. Mark the supervisor non-dumpable.
  5. Drop Linux capabilities from the untrusted container.

We already used a scrubbed re-exec and set PR_SET_DUMPABLE to zero. The review was still worth doing because a child-process allowlist alone would not have protected the original process image in /proc.

Treat non-dumpable state as defense in depth. A cleaner design puts the trusted supervisor and the arbitrary-code runtime in separate containers, process namespaces, or VMs, then mounts secrets only into the trusted side. Do not share a secret volume with the sandbox and assume file permissions will save a same-UID process.

Best case, delete this problem: never deliver the durable platform secret to the agent Pod. Keep it in the control plane and expose a narrow authenticated operation instead.

3. Broker privileged actions with short-lived capabilities

An agent often needs to call a model, read its assigned storage, report progress, or spend from a user's balance. None of those jobs requires a durable provider key inside the sandbox.

The broker can be small. At activation time, mint a short-lived token bound to:

  • one tenant;
  • one agent or run;
  • an explicit operation or audience;
  • a tight expiry;
  • a revocation or generation number when immediate invalidation matters.

The receiving service must derive identity from the verified token. Do not accept an agent ID in the request body and merely check that the caller presented some valid token. Also apply rate limits, cost limits, idempotency, and request-size limits at this boundary.

Authentication must cover every route, including compatibility routes and error paths. In our review, we added a regression test that the guessed OpenAI-compatible /v1/chat/completions path returns 404. We also verified that model routes with a missing or invalid token return 401 without calling the upstream provider. A forgotten route is often more dangerous than the intended one.

Webhooks and internal callbacks should fail closed too. A missing verification key should disable the route or make every request fail. It should not quietly switch the endpoint into unauthenticated mode.

Human approval is useful for irreversible actions, but it is not an authorization system. The service still needs to verify the capability after the user clicks approve. Approval records should name the exact operation and resource, not grant an open-ended shell window.

4. Make network reachability match the capability model

Removing service names from the environment reduces accidental disclosure. It does not make an internal network private from a process that can resolve DNS, scan addresses, or open raw sockets.

Start untrusted namespaces with default-deny ingress and egress. Add explicit allowances for:

  • DNS, if the workload needs it;
  • the model or action broker;
  • the agent's storage gateway;
  • an audited outbound path for general internet access.

An HTTP_PROXY variable is a compatibility setting, not enforcement. Arbitrary code can ignore it and connect directly unless the CNI, node firewall, VM boundary, or transparent proxy blocks that path.

Keep databases, Redis, cluster APIs, node agents, observability collectors, and other tenants off the allowlist. Block cloud instance-metadata endpoints unless the workload has a reviewed need for them. If metadata hands the Pod a node credential, one compromised sandbox can become a cloud-account incident.

The same session exposed another egress trap that ordinary port scans miss: a reverse tunnel made a localhost service public. Any sandbox with broad internet access may be able to do this and bypass the cluster's inbound firewall. Removing tunnel binaries from the image is not a boundary because an agent can download or reimplement a client. If this matters to your threat model, enforce outbound destinations and protocols at the network layer, and test that a sandbox cannot create an unapproved public listener.

Network policy is not a reason to weaken service authentication. Policies are usually coarse, can be misconfigured through labels, and may not express an HTTP route or tenant. Use them to reduce reachability. Let the destination service enforce identity and scope.

5. Harden the workload below the application layer

The Pod or VM remains the blast-radius boundary when application code fails. A reasonable Kubernetes baseline for arbitrary-code workloads includes:

spec:
  automountServiceAccountToken: false
  enableServiceLinks: false
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: agent
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]
        runAsNonRoot: true

Avoid host networking, host PID, host IPC, privileged containers, Docker or containerd sockets, broad hostPath mounts, and device access. Make the image filesystem read-only where your runtime permits it, then provide a deliberate writable workspace and temporary directory.

Containers share a kernel. If your risk or compliance requirements assume a hostile tenant trying to escape the kernel boundary, use a sandboxed runtime, microVM, or dedicated VM and validate it independently. A hardened Pod is not automatically the right answer for every customer.

Storage needs the same skepticism. A client-side subdirectory flag is not a tenant boundary when the client is under agent control. The server that accepts storage operations must enforce the tenant and path. We made that mistake earlier with JuiceFS and wrote up why we put a server in front of it.

6. Treat logs and model context as data exits

Secrets do not only leave through sockets. Command output can be copied into the model context, event stream, tracing span, error report, support bundle, or browser console.

Redact known platform credentials before output enters those systems. Keep structured logs to identifiers and outcomes when the raw payload is unnecessary. Restrict who can read traces, and give retained execution output a clear lifetime.

Exact-value redaction is useful, but it is not containment. An agent can add whitespace, encode the value, split it across messages, or write it to a file. The real control is to withhold the credential or replace it with a narrow capability. Redaction limits damage from accidents after that.

User-provided secrets need honest product language. If a user gives a shell-capable agent an API key, the agent can use and reveal that key within the network and tool boundaries you provide. A secret entry UI can prevent accidental display in chat. It cannot make a credential simultaneously usable by arbitrary code and unreadable to that code.

7. Revoke tenant state at the lifecycle boundary

Warm pools make this problem easier to get wrong. Prewarm runtimes and immutable tools. Bind disk, identity, tokens, and user configuration only when a specific agent claims the environment. We cover that split in what a Kubernetes warm pool cannot prewarm.

At the end of a run:

  • expire or revoke its capabilities;
  • kill the whole process tree through a cgroup or equivalent boundary;
  • stop background jobs and detached sessions;
  • close inherited file descriptors;
  • remove per-run temporary files;
  • clear user-specific network and storage mounts before reuse.

Killing the direct child is not enough. A process can detach, create a new session, or leave a descendant holding a token or file descriptor. Short token lifetime limits the damage when cleanup misses one.

If environments are reused across tenants, test the transition as an adversarial event. Look for stale processes, shell history, package-manager credentials, temporary sockets, language caches, mounted files, and kernel resources. For higher-risk workloads, destroying the VM is easier to reason about than proving a perfect cleanup.

Test the negative space

Happy-path tests prove the agent can work. Security tests should prove what it cannot inherit, reach, or reuse.

Boundary Regression test
Process environment Start every command type and assert forbidden prefixes, service-link names, and canary secrets are absent
Empty environment Launch a plugin with no configured variables and assert the parent canary is absent
Process image Try to read the supervisor's /proc environment and memory from the untrusted UID
Pod defaults Build each PodSpec and assert service links, service-account tokens, privilege escalation, capabilities, and seccomp settings
Internal APIs Probe intended, guessed, and legacy routes with no token, a malformed token, the wrong audience, and an expired token
Broker behavior Assert rejected requests never reach the model provider or downstream side effect
Network Attempt DNS discovery, direct IP connections, metadata access, cross-tenant traffic, and a reverse tunnel
Lifecycle Leave a detached process and a canary file, recycle the environment, and prove the next tenant sees neither
Observability Put canary secrets in command output and errors, then inspect events, logs, traces, and support exports

Keep the canaries fake and unique. Run the same suite against the deployment manifest, not only against helper functions. In our case, unit tests around the environment builder would not have caught kubelet adding service variables to the final Pod.

A small static check can also help. Enumerate process-launch call sites and require each one to set the approved environment explicitly. The goal is not to ban exec. It is to make a new execution path fail review until its trust boundary is visible.

What we shipped

We moved the shared agent environment boundary into one package and routed shells, background jobs, the Python bridge, PTYs, and stdio MCP servers through it. We made the stdio plugin environment non-nil even when empty. We disabled Kubernetes service links for agent, workflow, and user-service Pods. We also pinned the probed compatibility route to 404 and kept invalid model tokens from reaching the upstream provider.

The patch has a narrow scope. Internal DNS remains discoverable, and the agent can still reach the broker and storage gateway it needs, so those services must authenticate it. User-provided credentials remain readable by code that needs to use them. An outbound proxy variable remains a compatibility setting. Those are separate controls, and saying so is part of the fix.

This review was not a formal penetration test. Teams with strong tenant isolation, regulated data, or hostile-code requirements should add an independent assessment of the container runtime, kernel, storage plane, network plugin, and operational access.

A compact review worksheet

Before shipping a cloud agent, sit down with the runtime, control-plane, storage, and security owners and answer these questions with code or a test:

  1. Which exact values cross from the trusted supervisor into each subprocess?
  2. Can an empty environment accidentally mean inherit?
  3. Can the sandbox read the supervisor through /proc, shared files, sockets, or file descriptors?
  4. Which durable credentials exist anywhere inside the workload?
  5. Does every reachable service authenticate the agent and bind it to one tenant and resource?
  6. What can the sandbox reach by DNS, direct IP, metadata address, and public egress?
  7. Can it publish a local port through a reverse tunnel?
  8. Who enforces storage tenancy when the client is malicious?
  9. What survives process exit, run completion, sleep, and environment reuse?
  10. Where can raw command output and tool arguments be retained or viewed?
  11. Which tests fail if a new Pod type or process launcher skips the boundary?
  12. What part of the design still depends on the model choosing to behave?

The last answer should be limited to product behavior, not access control. Model behavior and approval UX will keep moving. A durable cloud-agent security boundary is the part that stays correct when the agent makes the worst plausible choice.