Skip to content
Agent Tree Knowledge Hub
🎓 Find your path Subscribe

Watchers: Condition-Until-It-Fires

Tier 2 · Building 6 min read

Before this, read:

A watcher answers one question: has this condition become true yet? It checks on a cadence, fires exactly once when the answer is yes, and then stops. No repeated alerts for the same trigger. No manual cleanup when the condition resolves.

The watchers primitive lives at ~/agent-system/agents/ai_os/watchers.py. The registry is ~/agent-system/state/watchers.json. A sweep cron runs every 15 minutes to check all active watchers.

Setting up a watcher

Section titled “Setting up a watcher”
Terminal window
python3.12 -m agents.ai_os.watchers add \
  --title "Supplier reply" \
  --kind gmail_search \
  --query "from:vendor@example.com subject:quote" \
  --cadence 30 \
  --escalate "Vendor replied with a quote — review thread and respond" \
  --expires 14d

All fields:

  • --title — a human-readable label, used in Telegram alerts and the registry
  • --kind — one of gmail_search, file_exists, file_mtime, shell, date_reached
  • --cadence — how often to check, in minutes
  • --escalate — the message sent to JD’s Telegram when the watcher fires
  • --expires — when the watcher expires even if it never fires (default: 30d)

The four check kinds

Section titled “The four check kinds”
Section titled “gmail_search”

True when a Gmail search query returns at least one result.

Terminal window
python3.12 -m agents.ai_os.watchers add \
  --title "Canvas grade posted" \
  --kind gmail_search \
  --query "from:notifications@instructure.com subject:grade" \
  --cadence 60 \
  --escalate "Canvas posted a new grade — check it"

The direction-check pattern. The first watcher deployed for a vendor reply false-triggered on JD’s own outbound email because the query wasn’t pinned to the sender. Always include from:<expected sender> in a gmail_search query when you’re waiting for a specific person’s reply. Without it, any email matching the subject — including your own — will fire the watcher.

file_exists

Section titled “file_exists”

True when a file appears at a given path.

Terminal window
python3.12 -m agents.ai_os.watchers add \
  --title "Export ready" \
  --kind file_exists \
  --path "/tmp/export-2026-06-10.csv" \
  --cadence 5 \
  --escalate "Export file is ready — import it"

Useful for coordinating between two scripts: the first writes a file when it finishes; the watcher fires and triggers the second step.

file_mtime

Section titled “file_mtime”

True when a file’s modification time changes since the last check.

Terminal window
python3.12 -m agents.ai_os.watchers add \
  --title "Config drifted" \
  --kind file_mtime \
  --path "/etc/some-config.yaml" \
  --cadence 60 \
  --escalate "Config file was modified — review the changes"

Distinct from file_exists: this watches for a change to a file that already exists, not for a new file to appear.

shell

Section titled “shell”

True when a shell command exits 0.

Terminal window
python3.12 -m agents.ai_os.watchers add \
  --title "Service healthy" \
  --kind shell \
  --command "curl -sf https://myservice.example.com/health" \
  --cadence 10 \
  --escalate "Service came back healthy — resume the rollout"

Shell watchers are whitelisted to prevent arbitrary code execution. The whitelist is configurable in the watcher config.

date_reached

Section titled “date_reached”

True on a specific date. The “clock as a sensor.”

Terminal window
python3.12 -m agents.ai_os.watchers add \
  --title "LinkedIn token expiry" \
  --kind date_reached \
  --date 2026-06-10 \
  --escalate "LinkedIn OAuth token expires today — run: bash ~/agent-system/scripts/reauth-linkedin.sh"

This kind shipped 2026-06-08. First use: a watcher that fired three days before a LinkedIn token would have expired silently, giving JD the exact reauth command in the Telegram alert.

Mandatory expiry

Section titled “Mandatory expiry”

Every watcher has an expiry. After expiry, the watcher is removed from the registry even if it never fired. The default is 30 days.

This is not optional. An immortal watcher is a cron with worse semantics — a job that keeps running but with a condition-check you can’t see in crontab -l. The expiry forces you to re-examine the obligation. If the condition didn’t fire in 30 days, either the condition was never right, or you need to extend the watcher deliberately.

Setting a longer expiry is fine when the obligation genuinely needs it:

Terminal window
--expires 90d   # quarterly cycle
--expires 365d  # annual renewal

Open-loop pairing

Section titled “Open-loop pairing”

For open-loop entries that are machine-checkable, pair them with a watcher:

# In the open-loops ledger
id: OL-042
title: "Waiting on supplier reply about bulk pricing"
owner: jd
linked_watcher: watcher-id-1234

The ledger tracks the human promise; the watcher does the checking. When the watcher fires, the open-loop closes automatically.

Not every open loop is machine-checkable. “Decide which framework to use” isn’t a condition a watcher can verify — that’s a human judgment. But “reply landed from supplier” is a gmail_search; “export file appeared” is a file_exists. The discipline is identifying which shape applies.

Viewing active watchers

Section titled “Viewing active watchers”
Terminal window
# List all active watchers
python3.12 -m agents.ai_os.watchers list


# Check watcher status and time-to-expiry
python3.12 -m agents.ai_os.watchers status

The registry at ~/agent-system/state/watchers.json is human-readable. Each entry has its title, kind, cadence, expiry, and a log of the last check result.

Next: What happens when the Mac Mini reboots and all those cron jobs miss their runs. The catchup pattern and stale-cron alerting. Cron catchup & resilience.