Agent Memory: Making AI Remember Everything

Every LLM session starts fresh. Your agent doesn’t remember the meeting you told it about last Tuesday. It doesn’t remember that your boss prefers bullet points. It doesn’t remember that you already tried the approach it’s about to suggest — and it failed.

The solution isn’t a better model. It’s a better memory system.

The Problem with “Remembering Everything”

The naive approach: dump everything into a massive MEMORY.md and load it every session. This breaks quickly:

• Context windows fill up, forcing truncation of recent history
• The agent attends to everything equally instead of what matters
• Irrelevant old context causes noise in responses
• Costs balloon as every call loads megabytes of history

The right approach is layered memory — different stores for different types of information, loaded selectively.

Three-Layer Memory Model

┌─────────────────────────────────────────────────────┐
│  Layer 3: Curated Summaries (always in context)     │
│  MEMORY.md, entity summaries                        │
│  ~2,000-5,000 tokens  |  Rewritten weekly           │
├─────────────────────────────────────────────────────┤
│  Layer 2: Atomic Facts (queryable on demand)        │
│  memory/entities/*/items.json                       │
│  Structured JSON  |  Never deleted                  │
├─────────────────────────────────────────────────────┤
│  Layer 1: Raw Logs (archive + retrieval)            │
│  memory/YYYY-MM-DD.md                               │
│  Complete record  |  Loaded selectively             │
└─────────────────────────────────────────────────────┘

Layer	Files	When Loaded	Purpose
Raw logs	memory/YYYY-MM-DD.md	Session start (today + yesterday)	Complete record of events
Atomic facts	memory/entities/*/items.json	On-demand when querying	Structured, queryable facts
Summaries	MEMORY.md, entity summary.md	Always (main session only)	Fast context for common questions

This is how humans actually remember things. You don’t replay every conversation you’ve ever had — you have a working mental model (Layer 3), can recall specific facts when relevant (Layer 2), and can dig into detailed memories when needed (Layer 1).

Layer 1: Daily Log Files

Raw session logs. Everything significant that happened, preserved.

• Directorymemory/

  • 2026-03-25.md
  • 2026-03-26.md
  • 2026-03-27.md ← Today

Format:

# 2026-03-27 - Daily Notes

## Session 1 (9:14 AM)

- JD asked about competitor pricing for ProductCo launch
- Researched 4 competitors: Acme ($49/mo), Beta ($79/mo), Gamma ($39/mo), Delta ($99/mo)
- JD decided to price at $59/mo — above Gamma, below Beta
- **Decision:** $59/mo launch price, revisit at 100 customers

## Session 2 (2:30 PM)

- Drafted outreach email to 12 beta users
- JD approved draft with minor edits (softened the urgency in para 2)
- Emails sent at 3:45 PM
- **Follow-up needed:** Check open rates in 48 hours

## Lessons
- JD prefers direct pricing (no "starting at" language)
- Beta users were acquired from Twitter thread, not paid ads

Rules for good daily logs:

• Capture decisions, not just actions (“decided X because Y” beats “looked up prices”)
• Mark follow-up needed items explicitly
• Note preferences when you learn them (“JD prefers…”)
• Keep it scannable — bullets over paragraphs
• Skip routine noise (routine file reads, failed first attempts)

Tip

The agent should write to daily logs during the session, not just at the end. When a significant decision is made or preference learned, log it immediately. Don’t rely on reconstructing the session from memory afterward.

Loading strategy for sessions:

# In SOUL.md or agent instructions:
At session start:
1. Read memory/{today}.md if it exists
2. Read memory/{yesterday}.md if it exists
3. In main sessions only: also read MEMORY.md

Loading today + yesterday gives context without overwhelming the window. Older logs are available via search.

Layer 2: Entity Memory

Structured, queryable facts about people, companies, and projects. Unlike daily logs (narrative), entity memory is atomic — one fact per entry, tagged for filtering.

• Directorymemory/

  • Directoryentities/

    • Directorypeople/

      • Directorysarah-chen/

        • summary.md
        • items.json
    • Directorycompanies/

      • Directoryacme-corp/

        • summary.md
        • items.json
    • Directoryprojects/

      • Directoryproduct-launch/

        • summary.md
        • items.json

items.json schema:

[
  {
    "id": "sarah-001",
    "date": "2026-03-15",
    "category": "preference",
    "fact": "Sarah prefers async communication. Dislikes being called without notice.",
    "source": "2026-03-15 session",
    "confidence": "high",
    "superseded": false
  },
  {
    "id": "sarah-002",
    "date": "2026-03-20",
    "category": "context",
    "fact": "Sarah is VP of Engineering at Acme Corp. Reports to CEO Mike Torres.",
    "source": "LinkedIn + 2026-03-20 session",
    "confidence": "high",
    "superseded": false
  },
  {
    "id": "sarah-003",
    "date": "2026-02-01",
    "category": "context",
    "fact": "Sarah was Director of Engineering at Acme Corp.",
    "source": "2026-02-01 session",
    "confidence": "high",
    "superseded": true,
    "superseded_by": "sarah-002",
    "superseded_date": "2026-03-20"
  }
]

Key principles:

• Never delete facts — mark superseded: true instead. History matters.
• One fact per entry — makes search and filtering reliable
• Track confidence — distinguish “JD told me” from “I inferred”
• Cite sources — links fact to the session or external source that produced it

summary.md — fast retrieval:

# Sarah Chen

**Role:** VP of Engineering at Acme Corp (as of March 2026)
**Contact:** sarah@acme.com, LinkedIn: linkedin.com/in/sarah-chen
**Working relationship:** JD's main technical contact at Acme

## Key facts
- Prefers async; no surprise calls
- Direct communicator — skip small talk in emails
- Currently leading a major infrastructure migration (Q2 2026)
- Budget authority up to $50K without CFO approval

## Recent context
- March 2026: Promoted from Director → VP
- Evaluating JD's product for Q3 pilot

## Notes
- Met at SaaStr 2025, introduced by Mike Torres

Summaries are 3-7 sentences covering the most relevant current state. They’re rewritten from items.json periodically — they’re a view of the facts, not the source of truth.

Layer 3: Curated MEMORY.md

The highest-level layer. Loaded in main sessions only. Contains the distilled essence of everything the agent should always have in mind.

What belongs in MEMORY.md:

# Long-Term Memory

## About JD (my user)
- JD Davenport, MBA candidate at BYU (graduating 2027)
- Building ProductCo — AI agent platform for SMBs
- Wife: Sam. Kids: Ava and John. Provo, Utah.
- Budget-conscious; always ask about ROI tradeoffs
- Communication style: direct, no filler, facts first

## Active Projects
- **ProductCo launch** — Target: May 2026. Status: Beta testing.
- **BYU MBA** — Year 2, thesis: AI adoption in SMBs

## Key Decisions (Don't Re-Litigate)
- Pricing: $59/mo flat (decided 2026-03-27, revisit at 100 customers)
- Stack: Next.js + Supabase + Vercel (locked in, migration cost too high)
- Go-to-market: Bottom-up through Twitter community, not enterprise sales

## Lessons Learned
- JD wants bottom-line-up-front on everything. Lead with the answer.
- Don't suggest things that require >$100/mo new spend without flagging it
- JD is in the GTD system; use "capture → clarify → organize" language

## Last Updated: 2026-03-27

Rules for MEMORY.md:

• Keep it under 5,000 tokens — it loads every main session
• Include only information that affects future behavior
• Cull it regularly — old context wastes tokens
• Never load in group chats or shared contexts — contains personal data

Caution

MEMORY.md is sensitive. It contains personal context about your user — their family, finances, preferences, private projects. Never pass it to sub-agents or load it in group conversations where others can see the agent’s context.

The Bookend Workflow

The bookend workflow ensures memory is maintained consistently:

Session Start (Read)

1. Read state/current.md — "What was I working on?"
2. Read memory/{today}.md and memory/{yesterday}.md
3. [Main session only] Read MEMORY.md
4. Check for open loops: "follow-up needed" items from recent logs

During Session (Write as You Go)

- When a significant decision is made → log it to memory/{today}.md
- When you learn a new preference → add to memory/{today}.md AND
  create/update entity memory item
- When a project status changes → update memory/entities/projects/*/items.json
- When context is complex → update state/current.md for "future me"

Session End (Synthesize)

1. Update state/current.md with final status and next steps
2. If significant new information → update entity summaries
3. If major event/decision → add to MEMORY.md
4. Commit memory files to git (if using version control)

state/current.md — the session bridge:

# Current State — Updated 2026-03-27 3:45 PM

## What I'm Working On
- ProductCo landing page redesign (started 2026-03-26)
- Draft complete at ~/projects/productco/src/pages/landing.tsx
- Waiting on: JD to approve hero copy before deploying

## Open Loops
- [ ] Email open rates from beta outreach (check 2026-03-29)
- [ ] Sarah Chen follow-up on Q3 pilot (ping 2026-04-01)

## Next Session Focus
Start by asking JD about landing page approval.
If approved, spawn coding agent to deploy.

This file survives session compaction. When the context window fills up and older messages are dropped, state/current.md preserves what matters.

Memory Search

For older memories not in the current context window, use memory_search:

# Search across all memory files
memory_search "Sarah Chen pricing discussion"
# Returns: memory/2026-02-15.md:34 — "Sarah asked about volume discounts"

# Search entity memory
memory_search "acme-corp budget"
# Returns: memory/entities/companies/acme-corp/items.json:12 — "$50K budget authority"

The retrieval strategy:

1. Check summaries first — MEMORY.md and entity summaries are already in context
2. Semantic search — memory_search finds relevant snippets across all files
3. Direct read — If search points to specific file, read it with memory_get
4. Transcript fallback — Check session JSONL for very recent conversations

Don’t read the entire daily log directory to find one fact. Search first, read specifically.

Setting Up the Memory System

1. Create the directory structure

   mkdir -p ~/agents/myagent/memory/entities/{people,companies,projects}
   mkdir -p ~/agents/myagent/state
   touch ~/agents/myagent/MEMORY.md
   touch ~/agents/myagent/state/current.md

2. Add session start instructions to SOUL.md

   ## Every Session
   
   Before doing anything else:
   1. Read `state/current.md` — what was I working on?
   2. Read `memory/YYYY-MM-DD.md` (today + yesterday)
   3. If this is a main session with JD: also read `MEMORY.md`
   
   Never skip this. It's how I have continuity.

3. Define the write protocol

   ## Memory Write Rules
   
   Log to memory/{today}.md when:
   - A decision is made ("decided to price at $59/mo because...")
   - A preference is learned ("JD prefers async communication")
   - A task is completed with notable result
   - Something needs follow-up
   
   Update entity memory when:
   - A fact about a person/company/project changes
   - New important context is learned about someone
   
   Update MEMORY.md weekly or after major events.

4. Write your first MEMORY.md

   Start with the most important facts about your user. 20-30 bullets covering: who they are, current projects, key decisions, communication preferences.

5. Commit memory to git (optional but recommended)

   cd ~/agents/myagent
   git init
   git add .
   git commit -m "Initial memory setup"
   # Push to private repo for backup

Common Mistakes

Writing everything to MEMORY.md

MEMORY.md loads every session. Keep it under 5K tokens. Put detailed facts in entity memory instead.

Loading MEMORY.md in sub-agents

Sub-agents don’t need your full life context. They need task context. MEMORY.md in a sub-agent prompt wastes tokens and leaks personal data.

Skipping the session start reads

An agent that doesn’t read its daily log is an agent that forgets what it did this morning. Make it non-negotiable.

Deleting superseded facts

Mark facts as superseded: true, never delete them. History matters — knowing what changed and when is valuable context.

The Payoff

With three-layer memory, your agent:

• Opens every session aware of current context, open loops, and next steps
• Remembers preferences learned months ago without being reminded
• Handles “remember when…” questions by searching logs
• Never re-litigates decisions that were already made
• Notices patterns across sessions (“you’ve asked about this 3 times now…”)

Memory is what turns a capable tool into a trusted collaborator.

---

About the author: JD Davenport builds AI agent systems at OpenClaw. Follow on LinkedIn for updates on building AI agents for business.

Enjoyed this? Get weekly agent architecture insights.

Subscribe and get a free Agent Architecture Cheatsheet.

Subscribe