Tools: State, Memory, and Context: What AI Actually “Remembers”

Traditional Systems: Where State Is Clear ## What the Model Does Not Have ## Three Concepts That Must Stay Separate ## Context ## Memory ## The Most Common Design Failure ## A Cleaner Mental Model ## A Simple Rule ## Where This Leads Next Have you noticed this before? These aren’t model quirks. They’re signals that state is unclear in the system design. This post builds a clean mental model for how state, memory, and context work in AI-enabled systems—and how to design without accidental coupling. State in traditional applications is usually: Stored in databases Permanent or semi-permanent information that the system relies on. Example: Customer records, order history, or inventory levels in an Oracle or PostgreSQL database. Cached intentionally Temporary copies of data to improve performance or reduce repeated computations. Example: Session data stored in Redis or frequently accessed product catalog data cached in memory. Passed explicitly through APIs Data sent between services or components as part of a request. Example: A service call that includes a user ID and account type to fetch specific account information. Auditable and recoverable Every state change can be tracked and traced for debugging or compliance. Example: Versioned financial transactions, order status changes, or audit logs that can reconstruct system behavior at any point. Most importantly, this state is: When something breaks, you can trace what changed and where, which makes debugging reliable. AI challenges this—not by removing state, but by hiding where developers expect it to live. Let’s start with a hard boundary. Models do not have memory. Each call is independent. If something feels “remembered,” your system supplied it again. Many AI bugs come from mixing these up. Think of context like function arguments, not variables. Memory is external state you manage. If memory feels implicit, the design is already fragile. State is system-level truth. AI can influence state, but the actual state must be stored and controlled outside the model. Many systems accidentally treat the model as stateful. If restarting your service changes outcomes, state is leaking. A more stable framing: The model is a stateless function. Your system decides: Once this boundary is clear, complexity drops sharply. If something matters tomorrow, it cannot live only in today’s prompt. Store it. Version it. Own it. AI can assist—but it should never quietly carry state for you. Context and Data Flow: Feeding AI the Right Information That’s where the next post goes. Templates let you quickly answer FAQs or store snippets for re-use. Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment's permalink. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse - The model forgets something it clearly knew earlier - The same input gives a different answer later - Adding a bit of “context” suddenly fixes the behavior - Stored in databases Permanent or semi-permanent information that the system relies on. Example: Customer records, order history, or inventory levels in an Oracle or PostgreSQL database. - Cached intentionally Temporary copies of data to improve performance or reduce repeated computations. Example: Session data stored in Redis or frequently accessed product catalog data cached in memory. - Passed explicitly through APIs Data sent between services or components as part of a request. Example: A service call that includes a user ID and account type to fetch specific account information. - Auditable and recoverable Every state change can be tracked and traced for debugging or compliance. Example: Versioned financial transactions, order status changes, or audit logs that can reconstruct system behavior at any point. - Explicit – you know exactly where it lives - Addressable – you can query, update, or delete it - Deterministic – the same inputs produce predictable results - Does not remember previous requests - Does not retain conversation history - Does not accumulate state over time - Prompt text - Instructions - Retrieved documents - Conversation history you resend - Exists only for one request - Token-limited - Consumed, not stored - Stored chat history - User preferences - Retrieved embeddings - Cached tool outputs - Lives outside the model - Must be fetched intentionally - Must be injected back into context - Workflow progress - Decisions made - User-visible outcomes - Persist beyond a single request - Be inspectable - Be owned by the application - Relying on conversation flow instead of stored data - Assuming consistency without re-supplying context - Letting decisions live only in generated text - Non-reproducible behavior - Debugging without ground truth - Silent behavior drift - Models compute - Systems remember - Applications decide - What context to assemble - What outputs to persist - What becomes authoritative state