Plaud: Voice-Note Second-Brain Ingest

Tier 3 · Real Build 7 min read

Note

Before this: The CRM: Every Person Becomes an Entity. The Plaud pipeline is one of the CRM’s primary data sources — but the precision problems that emerged were mostly in the CRM path, not the Obsidian path.

JD carries a Plaud recorder. It captures voice notes, meetings, and ambient audio. The Plaud pipeline (plaud-second-brain-v1) turns that audio into structured Obsidian notes, CRM entries, and action items — automatically, without requiring JD to type anything after recording.

The pipeline works. It also surfaced concrete problems with precision that are worth understanding before you build anything similar.

---

What the pipeline does

1. Download and transcribe. plaud-pipeline.sh runs on a cron (every 3 hours at :15) and pulls new recordings from Plaud’s API. Audio goes to Whisper (ggml-small.en.bin via whisper-cli -bs 5 — the base.en model garbles JD’s voice notes; the small model is needed for accuracy). The upstream guard skips files over 4 hours or 200 MB.

2. Classify and parse. The transcript is classified by topic: which domain, what type of content (meeting, brain dump, action items, reference). The classifier also extracts CRM candidates — people mentioned by name — and action items.

3. Route. Obsidian gets a structured note in 01-Inbox/. The CRM receives entries for any people mentioned. Action items go into the domain task queue. A calendar event matcher (plaud_slot_matcher.py) pairs recordings with calendar events by a 15-minute time-window, slicing the transcript proportionally to the event duration.

4. Digest. An arrival-report digest summarizes new recordings and routes a brief to Telegram. A 72-hour aging alert fires if a recording has been sitting unprocessed.

---

The precision problems

The June 2026 audit (CHANGELOG 2026-06-08) found three specific precision failures that had gone unnoticed:

Domain misclassification. The classifier was routing some notes to “other” instead of the correct domain. A recording from an AI Foundry meeting was filed under a generic catch-all rather than the ai-foundry domain.

Invented CRM entries. The CRM candidate extractor was producing entries for strings that were not people. The examples: “Executive Assistant” and “Sweet Cherry” — one an abstract role, one apparently an audio fragment — were being treated as person-names and written to the CRM as new contacts.

Hardcoded domain in update_crm. plaud_handlers.py line 96 had domain=life-ops hardcoded for all CRM updates. Every person extracted from a Plaud recording was being filed under life-ops regardless of which domain the recording came from. The specific example: Felix (a contact) was mis-filed.

The underlying root cause across all three: the classifier prompt was not specific enough, and the CRM extraction had no filter for non-person strings.

---

The behavioral change

The audit produced a redesigned action model. Before: the pipeline acted silently on all extractions. After: the pipeline proposes actions in a single message and waits for approval before acting.

From the CHANGELOG: “JD’s refined auto-listener spec: propose-all-in-one-message → ask → act on approval (NOT silent auto-act).”

This is a real constraint that conflicts with the appeal of a fully automatic pipeline. The fully automatic version is faster — you record something, it gets processed, done. But the fully automatic version also writes bad CRM entries without telling you, and the damage compounds over time as the CRM fills with invented contacts.

The propose-then-ask model is noisier in one direction (JD has to tap approve) and much cleaner in the other (only verified information enters the long-term store).

Tip

The pattern generalizes. For any pipeline that writes to a long-term store (CRM, Obsidian, task queue), a propose-then-ask gate is worth the friction. The cost of one approval tap is low. The cost of auditing a contaminated long-term store is high.

---

The calendar slot matcher

plaud_slot_matcher.py pairs recordings with calendar events by time-window overlap (15-minute fuzz), then slices the transcript proportionally to the overlap duration. It reads from gcal-merged.yaml with a gcal API fallback.

The motivation: a voice note from a 2:00 PM meeting and a voice note from a 4:00 PM brain dump should go into different Obsidian notes with different context tags, even if they were both transcribed in the same pipeline run. The slot matcher makes that automatic.

One gap surfaced in backfill testing: the 2026-05-22 AI Foundry meeting at 2 PM was genuinely unrecorded. The matcher surfaced this as a GAP callout — it found a calendar event with no matching recording — rather than silently ignoring it. Honest gaps are better than silent ones.

---

What went wrong with the backfill

The pipeline was silent for two days at one point because the cron job was lost (CHANGELOG 2026-06-03 13:12). When the pipeline was restored, a backfill run processed 99 recordings and produced 88 Obsidian notes. The previous counts were 0/0 — all those recordings had accumulated while the cron was missing.

The root fix was mkdir -p in plaud-pipeline.sh (a missing directory was causing silent failures) plus a process fix: any cron that handles time-sensitive data needs a staleness alert so a 2-day silence is caught in hours, not weeks. The 72-hour aging alert was added specifically to prevent this.

---

Current state

The pipeline routes correctly to Obsidian. The precision improvements (domain classifier, CRM filter, domain assignment fix) were proposed to JD in the June 2026 audit and the build work was scoped. Whether the specific fixes shipped by the time you read this is something to check in the CHANGELOG — the audit document is at ~/clawd/domains/plaud/state/ if you are running the system yourself.

---

Next: Group H covers quality, safety, and operations. The QA Agent and Self-Test Loops starts with how the system tests itself before shipping.

Enjoyed this? Get weekly agent architecture insights.

Subscribe and get a free Agent Architecture Cheatsheet.

Subscribe