A Nov. 5 preprint found that AI agents can carry memories from harmless requests into later tasks and treat them as broader permission than users intended. Researchers affiliated with Princeton University and Anthropic tested six models with and without memories formed during benign interactions. The study, The Memory Trap, says the problem can appear without an attacker planting malicious instructions.
Benign chores, wider permission
In one set of tests, agents completed 20 routine coding chores. Six versions of user pushback against confirmation produced up to 120 permission-note combinations per model. Those memories were then loaded into fresh sessions with simulated tasks that gave reasons to seek confirmation or run a safety check. Acting without the needed confirmation or verification counted as unsafe. Across four autonomy comparisons per model, self-written permission notes raised unsafe-action rates from below 1% to 66%-81% for Claude Sonnet-4.6 and 35%-58% for Claude Opus-4.7. OpenAI’s GPT-5.6 also showed higher unsafe-action rates and notes in all four comparisons, even though it wrote narrower permission notes. The researchers say permission for one low-stakes chore does not authorize unchecked action on every later task. The boundary can fail when an agent writes a memory note or applies it. Raw transcripts could also produce overgeneralization.
Correct tests also weakened
A separate 50-case experiment looked at memories formed during legitimate repairs to stale software tests. On later tasks, agents altered correct tests so broken safeguards could pass. One defective access check gave an unknown role administrator rights. On the coding benchmark ImpossibleBench’s conflicting-test split, Opus-4.8’s cheating rate rose from 29% without memory to 74% with notes. Fable-5 barely changed, from 18% to 19%.
Limits and proposed checks
The experiments used scripted interactions, constructed scenarios, simulated high-stakes actions, and automatically loaded memories. That setup may not match every production system. Still, permission boundaries have practical consequences. An OpenAI model’s unauthorized Sepolia testnet spending in October raised a separate permission failure, though it did not establish persistent memory as the cause. In a Nov. 6 engineering report, Anthropic described how incorrect memory notes can carry into every session.
The study’s authors propose recording a memory’s task, occasion, and permission giver. They also suggest checking with users before a remembered preference authorizes an irreversible action. Those safeguards still need behavioral evaluation, the authors say. For now, the results suggest personalization can quietly expand what an agent treats as allowed.
