How memory tools can make AI models worse

Modern AI systems adapt to individual users by incorporating each interaction as context for future tasks. Researchers at the AI company Writer published two papers showing that popular memory systems can steer models toward misconceptions introduced by users. As user input fills the model’s context window, the model grows more sycophantic — and less committed to accuracy.

“We wanted to be able to characterize how often a model is going to be usefully paying attention to user preferences versus giving a potentially wrong answer,” said Dan Bikel, Writer’s head of AI, who worked on the papers. As Bikel told TechCrunch, “with every additional storing of user preferences and retrieving of them, you’re running an increasing risk.” In one test, researchers recorded that a user’s favorite book was “Station Eleven,” then asked the model to name a bestselling dystopian book.

Models became far more likely to name “Station Eleven” in their response, even though the question didn’t relate to the user’s favorite book.

memory tools, memory systems, context window, user preferences, writer, dan bikel, sycophantic, station eleven, misconceptions, techcrunch