A staging token reached production: lessons from an attack on our AI agent platform

On October 29, we saw a production request carrying a validly signed token for a workflow that did not exist in our production database. Production rejected the request. The signature should never have verified there in the first place.

The token came from our staging environment. Staging and production shared a signing secret, and an attacker had found a way to obtain a short-lived token issued to a staging workflow. The attacker did not need to learn the signing secret. They only needed a token signed by it.

This is an account of what we observed from October 28 through October 31, 2026, and what other teams can check in their own systems. The attacker was still active, and our investigation was continuing, at the end of that window. Our findings are limited to the evidence we reviewed.

The scan moved between boundaries

The activity began with requests to administrative and debug routes, attempts to read across accounts, and probes of login and OAuth flows. Later requests came through a mix of hosting infrastructure and rotating proxy addresses. Blocking an initial address did not end the activity.

We also saw attempts against agent execution environments. The attacker tried to reach private services and collect credentials. The administrative and cross-account access attempts we identified were denied. We found no evidence of a sandbox escape in the staging investigation.

That result is narrower than saying every layer worked. A workflow token from staging did reach production's signature check, and it verified successfully.

One signing key connected two environments

The attacker created workflows in the publicly reachable staging environment. A workflow could read the short-lived token it received for model access. The attacker obtained one of those tokens and sent it to production.

Production and staging used the same signing secret. The token did not carry a verified environment distinction. Production therefore accepted its signature. A later check rejected the request because the referenced workflow did not exist in production. The request did not result in a production model call.

We rotated the production signing secret and separated the internal secrets used by staging and production. We also removed public access to the staging API. A later replay using a staging token failed signature verification. We added a banned-owner check to model calls. A signed token can otherwise remain valid after the account that obtained it is banned.

The earlier clue was an identifier in production requests. We initially thought it had been invented because we could not find it in production. About three hours later, we found it in the staging database. The token replay led us to the shared signing secret.

What to check:

• Inventory signing keys and service credentials across production, staging, CI and developer environments. Check that the values differ, including copies in deployment and automation systems.
• Treat any bearer token readable by user-controlled code as a token that can leave that environment. Limit its lifetime, scope and the resources it can reach.
• Validate the intended issuer and audience where the token format supports them. Keep the production and staging signing keys separate as an independent control. RFC 8725 describes issuer and audience validation for JWTs. OWASP SAMM recommends keeping production secrets out of test environments.
• Test a token issued in each environment against every other environment. Check both signature validation and the authorization that follows it.

A valid signature establishes that a trusted key signed the token. It does not, by itself, establish that the token belongs to the environment, account or resource receiving the request.

A ban has to reach running work

Banning an account and rejecting its API key are useful first steps. They are not enough if other credentials remain valid, a scheduled workflow can still run, or a previously opened connection keeps access.

Our review found that some checks needed to extend beyond login and ordinary API requests. We added a banned-owner check for model calls and later shipped checks covering workflow starts and additional relay calls. These changes address the paths we identified. They do not establish that every possible background path has been covered.

What to check: Make an inventory of every way work can continue after a ban: active sessions, short-lived tokens, scheduled jobs, webhooks and open connections. Test each path after revocation. Decide what happens to work already in progress, and record whether a refusal occurred before or after an external side effect.

A public endpoint can cause damage without granting access

After sign-up was disabled, the attacker could still trigger login-code emails to addresses without accounts. Verification did not create an account, but the email had already been sent. The attacker also submitted anonymous feedback that generated operator notifications.

We changed the code-issuing path so that it does not create or send a code to an unknown address while email sign-up is disabled. We restricted feedback to registered accounts and added a per-account limit.

What to check: List public endpoints by their effects, not only by the data they return. Sending an email, delivering a webhook, starting a computation or writing a database row can all cost money or affect another person. Put authorization and limits before the effect. Test the entire flow, including requests that receive a harmless-looking response.

OWASP's API Security guidance recommends operation-specific limits and spending controls. Rate limits reduce the volume of abuse. They do not replace authorization or make a shared signing key safe.

Treat the model as part of the workload

During the investigation, a separate user asked an agent to run a cryptocurrency miner using trial credit. One model refused repeated requests. Another complied. The miner ran for about ten minutes before the account was banned. This was a separate abuse incident, not evidence that the scanner escaped a sandbox.

The distinction matters for systems that route between models. A refusal by one model does not guarantee that another model will refuse the same request. Nor should model refusal be the boundary protecting credentials or internal networks.

Each of our agent sessions runs in its own microVM. We also added a separate check for mining commands. Those controls serve different purposes: isolation limits what a command can reach, while an abuse check decides whether the platform should run it. A check can make mistakes or become unavailable, so its failure behavior needs an explicit decision.

What to check: Test from an actual workload, not only from the host configuration. Check access to private networks and metadata services, what credentials a command can read, and what happens when a safety check times out. Re-test after changes to networking, sandbox images or model routing.

Make investigation possible before an incident

Our first production request logs did not record enough information to associate every request with a caller. That made it harder to connect individual requests to the activity we were investigating.

The absence of an alert is not proof that a request never happened. Preserve the logs needed to reconstruct authentication, authorization and side effects. Check that exports contain the expected records before the source retention window expires. Keep secrets and customer content out of routine diagnostic output.

Pick an account and a token in a test environment, then make a request. Can you trace who issued the token, which environment issued it, and which resource the request named? Can you see the authorization decision and any side effects? Repeat after banning the account. If those answers require guesswork, improve the records before you need them in an incident.