Does context compaction lose the user's request? We tested it
· by Plori Engineering
TL;DR. In 108 trials, the user's current request was in the part of an agent conversation that compaction replaces with a summary. Of the 672 exact values in those requests, 0 were dropped and 0 were changed. One summary rewrote two dates in another format, which an exact-string check counts as missing. A value that a tool printed once was lost in 1 of 72 trials. Limits: synthetic conversations, three low-cost models, one summary format, one to three compactions.
An LLM agent sends its whole conversation to the model at each step. In a long task, the conversation becomes too large for the model's context window. Compaction replaces part of the conversation with a shorter summary that the model writes, so the agent can continue. Plori runs AI agents in cloud Workspaces. Its agent keeps the first messages and the most recent messages word for word, and replaces the middle with a summary.
What does a recent paper say about compaction?
The paper You don't need long context reports that, with frequent compaction, "frontier language models sacrifice little to no task accuracy when maintaining short contexts". It also states: "Compaction is a lossy context compression method." It finds that the compactions that keep the most accuracy contain "a verbatim or near-verbatim copy of the input task".
After each compaction, the paper's harness "starts a fresh conversation containing the original task and the compacted state". So the model gets the task text again in full. The paper measures task accuracy on benchmarks. It does not measure whether one exact value of a request is still present after a compaction.
Why did we test our own agent?
Our agent does not give the request again in full after a compaction. The first request of a conversation stays in the part kept word for word. A request made in a later turn is in the middle once the agent has worked on it long enough. Then only the summary's wording of it remains. So we measured whether the exact values of the current request are still in the conversation after compaction.
How did we test it?
On 2026-11-11 we tested three low-cost models from three providers: model A, model B and model C. Each trial builds a synthetic conversation: an earlier finished task, the request under test, then tool calls and their output as filler. The part that gets summarized holds about 20,000 or about 80,000 estimated tokens. Our compaction code runs in every trial, with its trigger lowered, and the model under test writes the summary. Then we search the conversation for each value of the request as an exact string.
The short request has 2 to 6 exact values, such as file paths and the threshold 0.37.
The long request is 1,633 bytes of prose with 14 exact values, such as two dates, three
account ids to leave out, a rounding rule and an e-mail address. In one case the user
changes a value in a later message. In another, the user replaces the whole request. In
some trials the agent compacts the conversation three times, and each pass summarizes the
earlier summary. We used no customer conversation.
Were the request's values still in the conversation after compaction?
Yes. In the 108 trials, 0 of 672 values were dropped and 0 were changed. 670 were still
present as the exact string. The other 2 are in one trial. The request gave the dates
2026-08-17 and 2026-10-25, and the summary wrote "Aug 17–Oct 25, 2026 inclusive". The
dates are correct. The exact-string check counts them as missing. The rewrite happened in
the first of three passes and stayed unchanged in passes two and three.
| Case | Trials | Values present |
|---|---|---|
| Short request, one pass, including a changed value and a replaced request | 72 | all values in all 72 |
| Long request, one pass | 18 | 14 of 14 in all 18 |
| Long request, three passes | 9 | 14 of 14 in 8, 12 of 14 in 1 (the date rewrite) |
| Short request, three passes | 9 | 6 of 6 in all 9 |
| Control: request kept word for word | 72 | all values in all 72 |
The 72 control trials are separate from the 108.
We expected a summary to drop details such as the account ids to leave out, the column order and the commit-message prefix. All 27 long-request trials kept them.
In 107 of the 108 trials, the summary has a section headed "Active goal" that states the current request. One of them, from the replaced-request case, opens like this (synthetic data):
## Active goal
Produce `reports/active-team-2025.csv` containing the **accounts on plan `team-2025`**, exported via `scripts/export_accounts.py`.
- User's exact instruction: "export the accounts on plan team-2025 to reports/active-team-2025.csv using scripts/export_accounts.py"
Can compaction lose a value that a tool printed once?
Yes. We also tested values that the user never typed and that a tool printed once, such as a database port printed by a start script. The values were in the part that gets summarized. They were lost in 1 of 72 trials, with model B at the size of about 80,000 estimated tokens. That summary said the database had not been started, and the model's next action was to start it again. The other 23 trials of the same case kept the values.
Does adding the request word for word help?
We built a variant that adds the current turn's user messages word for word after the summary. This is the practice the paper describes. In 102 trials with the variant, 54 of them with the request in the summarized part, all values were present. The unchanged version lost no request values, so the variant had nothing to fix in this test. We did not ship it. This is not a result against the paper: it measured task accuracy on benchmarks, and we measured exact strings in synthetic conversations.
What does the model do after compaction?
In a second stage, each model got the compacted conversation, and we classified its next action. In no trial did a model use a replaced value, follow the replaced request, repeat finished work or ask the user to repeat the request. In 37 of 162 trials the model took the correct action with the correct values, and in 7 it ran a tool again to get a value. This stage is weak evidence. In 58 of the 162 trials the model was still inspecting files after two rounds, and we did not classify it.
How can you test this in your own agent?
- Put the values you measure only in the part that gets summarized.
- Add control trials in which the request is kept word for word, and report them separately.
- Search for each value as an exact string, then read every miss by hand. A format change is not a loss.
- Test a value that appears once in tool output separately from the user's request.
- Test repeated compaction. In our test, a change made in the first pass stayed in the later passes.
What are the limits of this test?
- Synthetic conversations only.
- One summary format, ours. A summary that does not state the current request can give a different result.
- One to three compaction passes, and requests up to 1.6 KB.
- Three low-cost models. We did not measure other models.
- The exact-string check counts a correct value in another format as missing. It cannot see a value that is present but attached to the wrong thing.
- "About 80,000 estimated tokens" is 88,000 to 167,000 real tokens, depending on the model's tokenizer.