Capture

Capture is the only layer that touches the outside world. It produces one JSON file per observation into ~/.persome/capture-buffer/; nothing above it ever talks to macOS directly.

In source="daemon" mode, live capture requires Accessibility for the native mac-ax-helper and, when event-driven capture is enabled, the separate mac-ax-watcher executable. The terminal and Python daemon do not read AX on their behalf. Screen Recording is additionally required when screenshot storage or effective OCR needs pixels. In a standard interactive macOS install, persome onboard explains and requests each required principal separately, verifies architecture-native OCR, starts the final daemon owner, checks local health, and proves one fresh capture inside that daemon before returning success. The authenticated /permissions endpoint runs the actual helper/watcher trust checks plus the Runtime’s Screen Recording preflight. With HTTP auto-start disabled, the same generation publishes its permission, worker, phase, and capture receipts to owner-only .runtime-state.json.

In source="ingest" mode a trusted local producer owns macOS AX and pixel permissions and sends bearer-authenticated frames to /captures/ingest; the daemon starts no OS watcher. Onboarding proves that ingest runner is ready instead of manufacturing a daemon AX capture. This mode requires an enabled HTTP transport because the authenticated endpoint is its only input channel.

Owner-selected mobile observations join the same S1 path through the paired companion bridge and loopback-only /mobile/events/ingest. The request’s Idempotency-Key must equal event_id. A durable (device.id, event_id) receipt and deterministic capture ID make bridge retries and process-loss recovery idempotent; identity reuse with different content is rejected. Mobile captures preserve the timezone-aware owner-reported captured_at, a separate Runtime received_at, and explicit paired-bridge provenance. Distinct event identities are never collapsed by screen-content deduplication.

Two signal sources

The following two sources apply to daemon capture mode.

mac-ax-watcher (primary, event-driven). A vendored Swift binary that subscribes to AX notifications across all running apps: window focus, value changes (typing), title changes, app activation. It emits one JSON object per event on stdout. The Python side reads that stream line-by-line in capture/watcher.py → capture/event_dispatcher.py.

Heartbeat timer (fallback). Every heartbeat_minutes (default 10), the scheduler fires a capture even if no event arrived — so long idle periods leave a trail. Set heartbeat_minutes = 0 to disable entirely (watcher-only); values >0 are clamped to a 60-second floor.

Both funnel into capture_once in capture/scheduler.py, which runs:

ax_capture.capture_frontmost(focused_window_only=True) — one-shot invocation of mac-ax-helper for the current window, pruned to ax_depth layers.
s1_parser.enrich() — extracts focused_element, visible_text, and url from the AX tree (see S1 fields below).
cmux_source.maybe_inject() — when the frontmost bundle is cmux, appends the real terminal text read over cmux’s local socket RPC (see cmux signal source below); a successful injection skips step 4’s OCR fallback for this window.
OCR fallback — when the AX render produced no usable content and enable_ocr_fallback is on, submit a focused-window screenshot to an isolated local worker.
screenshot.grab() — unless include_screenshot = false.
window_meta.active_window() — app name, title, bundle_id via NSRunningApplication.
Write {iso8601_safe}.json to the buffer.

The filename is ISO-8601 with : → - and + → p / - → m for the TZ offset. Example: 2026-04-21T17-07-32p08-00.json.

The same capture scheduler also invokes SessionManager.on_event (wired as a pre_capture_hook in daemon.py), so the session cutter sees every written, non-duplicate observation without a separate subscription path.

Local OCR fallback

The installer runs persome onboard, whose OCR step checks the architecture-native Runtime and required assets, requests Screen Recording after an explicit explanation, persists enabled intent, then starts the daemon and waits for the daemon-owned isolated worker to initialize. [capture].ocr_policy distinguishes auto (fresh/unconfigured), enabled, and disabled (explicit opt-out). Ordinary onboarding and every update preserve an explicit policy and selected tier; persome onboard --tier ... and persome ocr setup explicitly enable OCR, while persome ocr disable records disabled intent. The standalone setup repair command keeps its own explicit worker check. The focused screenshot is used locally and is never placed in an LLM prompt. The OCR path is:

focused screenshot bytes
  -> local OCR worker subprocess
  -> text + geometry
  -> app-aware structuring when available
  -> captures FTS backfill
  -> timeline/modeling fallback when AX text is empty

The subprocess is the native-crash boundary: Apple Silicon runs bundled PP-OCRv6 there; Intel invokes an on-device Apple Vision helper with the same text, box, and confidence contract. A native fault fails the OCR call without killing the daemon. PERSOME_DISABLE_OCR=1 prevents either backend from running. PERSOME_OCR_IN_PROCESS=1 exists only for debugging the Paddle path and removes its worker isolation.

The AX helper, watcher, and Intel Vision OCR helper are compiled into immutable <PERSOME_ROOT>/native/<source-digest>/ directories keyed by a format version, architecture, and Swift source bytes. A same-version reinstall returns the exact existing executables. For the AX helper and watcher, this preserves their macOS TCC identity; changed AX source uses a new digest path and therefore requires a deliberate new Accessibility grant. The Vision helper does not request AX access. Rollback resolves the old wheel source and old binaries again.

persome onboard            # permissions + OCR + daemon + health + fresh capture
persome ocr setup          # enable, request permission, verify worker
persome ocr status         # quick config/runtime/model/TCC state
persome ocr status --check # also start and verify the worker engine
persome ocr disable        # explicit opt-out; restart to apply

Debounce / dedup / gap

Four time-based knobs throttle the event firehose (capture/event_dispatcher.py):

Knob	Default	What it does
`debounce_seconds`	3.0	`AXValueChanged` events within this window collapse — only the last triggers a capture. Prevents one-capture-per-keystroke during typing.
`dedup_interval_seconds`	1.0	Repeated non-focus events on one surface are dropped. `AXApplicationActivated` waits for this short trailing window and is replaced by a more-specific `AXFocusedWindowChanged`; Focus first is immediate and its later Activation is dropped. Only the latest surface transition may retain a pending Activation. Any newer cross-surface focus/click/text signal invalidates an older pending Activation even if rate limiting rejects the newer capture, because an older trigger cannot safely label content sampled after the foreground surface changed. Repeated same-type Focus notifications continue to the content gate because they may be different real windows.
`min_capture_gap_seconds`	2.0	Hard floor between consecutive non-focus captures. Focus transitions bypass it.
`same_window_dedup_seconds`	5.0	Non-focus-change events on the same surface (`bundle_id`, falling back to `app_name`) collapse within this window. Dynamic titles are not surface identities; focus changes always bypass it.

Tune these if you see capture.log flooded; the defaults produce a few hundred captures per work-day, comfortably under the buffer retention.

Content dedup (durable head + bounded recent window)

On top of the time-based knobs, the live capture runner compares each built capture against recently committed capture-content receipts. Its versioned fingerprint includes the stable surface (bundle_id, falling back to app_name), window title, focused role/value, visible text, URL, and any explicit mobile device/event identity. Title remains part of the exact context identity because two windows or channels can expose the same empty-state text; short dynamic-title bursts are handled by the title-independent surface gates above. Timestamp, trigger metadata, screenshots, and raw AX geometry are excluded.

Watcher events enter a bounded queue before AX and window_meta are sampled. Before commit, the worker therefore rebinds the trigger’s app, bundle, and title to the surface it actually observed. If the stable surface or an explicit non-empty window title changed while the event waited, the trigger becomes QueuedSurfaceRefresh, retains only its sanitized source event type, and drops stale click details; the session hook receives that same reconciled trigger. This preserves current content without attributing an action or app switch to the wrong window under backlog.

The latest successfully written fingerprint remains a duplicate with no time limit while its backing raw capture exists, including across daemon restart. An older non-head fingerprint is suppressed only within same_window_dedup_seconds (five seconds by default), catching burst noise such as A → B → A without erasing a later genuine revisit; the live horizon uses a monotonic clock. The bounded receipt history is advanced only after capture persistence succeeds. Raw-retention deletion or hard-cap eviction removes the matching receipt in the same SQLite transaction as its capture projection and reloads the live runner’s head, so an identical current frame can be observed again after its old evidence ages out. A no-runner direct write does not invent a new head: it durably marks and clears prior receipts, and a runner reload consumes any crash-stranded marker fail-open. persome capture-once holds the Runtime lifetime lock and refuses to race a running daemon. A duplicate is not written and the session manager’s pre_capture_hook is not fired. Mobile observations still use their explicit (device.id, event_id) as the authoritative exactly-once identity; a forced mobile write merely becomes the runner’s new content head.

This catches the case the time knobs can’t: a screen that doesn’t change (lock screen overnight, a paused video, an idle IDE) keeps generating AX events with the same content indefinitely. Without content-dedup those would both fill the buffer and keep the current session from ever idling out. Timestamps, triggers, and screenshots are excluded from the fingerprint so only meaningful changes count.

cmux signal source

cmux (com.cmuxterm.app) renders terminals on the GPU; its AX tree carries only window chrome (workspace tab titles, buttons — spike #558 measured ~30 chars median of content per app subtree). Instead of OCR, capture/cmux_source.py talks to cmux’s local unix-socket RPC (~/Library/Application Support/cmux/cmux-<uid>.sock, newline-delimited JSON — the same protocol the official cmux read-screen CLI uses):

system.tree — visible windows → selected workspace → panes → selected surfaces.
surface.read_text with surface_id (the UUID; the surface/surface_ref param spellings are ignored by the server and silently fall back to the focused surface) for each visible terminal surface. Browser/filepreview surfaces and unselected tabs are skipped.
The texts are appended to the capture’s visible_text under ### [cmux terminal] <workspace · surface> section headers, and cmux_text_injected: true is set on the capture.

Downstream stages (timeline, focus_excerpt, captures_fts) consume visible_text as-is — there is no cmux-specific path beyond this injection. Discipline: the whole socket conversation shares one sub-second deadline; per-surface budget 6 k chars (tail-kept), total 12 k; a single bad surface (tree/type drift, e.g. “Surface is not a terminal”) is skipped without aborting the rest; any other failure degrades silently to the AX-only capture with a rate-limited warning. Successful injection skips the OCR fallback for that window. Gate: [capture] cmux_source_enabled (default on).

Privacy note: terminal text can contain secrets echoed on screen. There is no general visible_text redaction layer in the capture pipeline today; helper-level [REDACTED] handling only covers AX password inputs. Terminal text therefore carries the same risk profile as other screen full text.

AX depth — the #1 footgun

AX Trees for native Cocoa apps are shallow (5–15 layers). Electron apps (Claude Desktop, VS Code, Slack, Notion) nest user content 20–60 layers deep under chrome.

Default ax_depth = 100 was chosen after diagnosing silent capture misses: a 90-second Claude Desktop conversation about an interview at 18:00 was producing captures where “18:00” appeared at character 5639 of the tree — past any reasonable prune limit. At depth 8, the tree contained only window chrome and sidebar headers; at depth 100, the full conversation was there.

If you’re running on limited hardware and only care about native apps, lowering to 30 is safe. Don’t go below 20.

Diagnostic:

./resources/mac-ax-helper --app-name Claude --depth 30 --raw | wc -c
# vs.
./resources/mac-ax-helper --app-name Claude --depth 100 --raw | wc -c

A 10×+ ratio means there’s content past depth 30 you’d miss.

What’s in a capture file

{
  "timestamp": "2026-04-21T09:07:32.123456+00:00",
  "schema_version": 2,
  "trigger": { "event_type": "window_focus_changed", "app": "Claude", ... },
  "window_meta": {
    "app_name": "Claude",
    "bundle_id": "com.anthropic.claudefordesktop",
    "title": "New conversation — Claude"
  },
  "focused_element": {
    "role": "AXTextArea",
    "title": "Message composer",
    "value": "I have an interview at 18:00",
    "is_editable": true,
    "value_length": 30
  },
  "visible_text": "### New conversation — Claude\n...",
  "url": null,
  "ax_tree": { ... pruned tree with roles, titles, values ... },
  "ax_metadata": { ... },
  "screenshot": {
    "image_base64": "iVBORw0KGgoAAAANS...",
    "mime_type": "image/jpeg",
    "width": 1920,
    "height": 1200
  }
}

trigger is {"event_type": "heartbeat"} for timer captures and {"event_type": "manual"} for capture-once. Screenshot is omitted entirely when include_screenshot = false.

Secure fields (password inputs) are replaced with "[REDACTED]" at the helper level — the Python side never sees them.

S1 fields

Ported from Einsia-Partner’s s1_collector. These are what downstream LLM stages consume — the raw ax_tree is kept only for future vision-model support and debugging.

focused_element — {role, title, value, is_editable, value_length} for the currently focused AX element. This is the user’s cursor context: what they’re typing into, which sidebar row is selected, etc.
visible_text — a length-capped markdown rendering of the AX tree (up to ~10 k chars). What the user is currently reading on screen.
url — regex-extracted from visible_text when present; null otherwise.

Editable values are placeholder-cleaned before they enter S1. The native AX helper’s focused-element projection first uses AXPlaceholderValue, then a bounded fallback for Chromium’s exact .placeholder descendant shape. The raw tree retains that evidence; the Python S1 projection applies the same structural check to the tree, older helpers, watcher triggers, and trusted-ingest producers. An exact placeholder descendant is removed only inside its owning editable subtree and only when its text is locally paired with that control. Matching text is then cleared from the parent projection. Ordinary page text and broad CSS classes that merely contain the word placeholder remain intact. The same copy-on-write S1 boundary removes embedded NUL code points from AX strings before focused-value and visible-text limits are applied. This repairs iTerm2-style NUL padding around CJK text without rewriting the diagnostic raw ax_tree; focused_element.value, its length metadata, and AX-derived visible_text are projected from the normalized copy. Historical timeline, MCP, and classifier chat-drill reads plus rebuild-captures-index apply the same sanitizer, so replaying an old buffer cannot turn input hints into authored text; index rebuilds also preserve DB-only OCR backfills. When an AX-empty surface falls back to screenshot OCR, the same frontmost/focused-window evidence removes exact OCR lines/fields only for controls whose placeholder can currently be visible (the control is empty or exposes that same hint as its value). It never performs substring replacement. If identical OCR units outnumber the proven placeholder controls, the value is ambiguous and every occurrence is preserved. Explicit --raw native captures retain the diagnostic AX structure.

Persisted screenshots are not passed to timeline, reducer, memory-delta, or schema prompts. They support optional local provenance drill-down and debugging. When encrypt_screenshots=true, PERSOME_SCREENSHOT_KEY seals them with AES-256-GCM. install.sh generates this machine-local key automatically and preserves it across reinstalls. If the key is absent or malformed, persistence fails closed by omitting pixels while retaining AX text and metadata. Set include_screenshot=false when persistent pixels are not required; OCR can still take an ephemeral focused screenshot when enabled.

Buffer hygiene — tiered retention

Captures are pruned by the timeline tick, not the writer. After each timeline scan, capture_scheduler.cleanup_buffer applies three age-based passes (oldest-safe-first), gated on “this file has already been absorbed by a closed timeline block”:

Pass	Condition	Action
Delete	mtime older than `buffer_retention_hours` (default 168 = 7 days)	Whole JSON removed
Strip screenshot	mtime older than `screenshot_retention_hours` (default 24)	Rewrite JSON without `screenshot` field; sets `screenshot_stripped: true`. The AX tree, `visible_text`, `focused_element`, and `url` stay
Evict by size	Total buffer > `buffer_max_mb` (default 2000, i.e. 2 GB; `0` disables)	Delete oldest files until under the cap

The separate buffer_max_mb limit is a hard disk-safety boundary. It evicts oldest captures even when the reducer watermark has stalled; this can sacrifice an unabsorbed trailing frame, but prevents an ingest or reducer failure from growing the buffer without bound. Ingest timestamps more than five minutes in the future are replaced with the server clock so they cannot evade ordering and retention. Accepted timestamps are normalized to UTC with fixed-width microseconds, preventing same-second ID collisions. Upgrade paths can still contain older local-offset filenames, so search, timeline, and retention compare their parsed instants rather than raw strings; daylight-saving fall-back cannot reverse processing order. Atomic-write remnants from a crashed process have no recovery contract and are removed after a five-minute race-safety grace period.

Why tiered: the screenshot base64 is ~77% of each capture’s bytes and is not needed to build the durable model. Stripping it at 24h drops each stale capture to ~20% of its original size while preserving AX/OCR evidence for local search.

To wipe manually:

persome clean captures

This removes buffered JSON, the captures/captures_fts projection, and the durable content-dedup receipts. The same tables are securely scrubbed from retained snapshots and integrity-quarantine database copies so a later recovery cannot restore the deleted capture state. Timeline and personal-model state are separate deletion scopes. Both exact and --merge forms of persome rebuild-captures-index also clear the receipt history. Neither raw buffer replay nor an older snapshot can reconstruct original commit order, so the next observation passes fail-open instead of trusting a stale head.

Search index — `captures_fts`

Every successful capture write is also indexed into an FTS5 virtual table (captures_fts, backed by a captures content table — see src/persome/store/fts.py). This is what powers the MCP search_captures and current_context tools, which let LLM clients reach the raw screen content directly without having to scan JSON files on disk.

Lifecycle.

Event	Effect on index
`_write_capture` (write-through)	Upsert one row into `captures` (`INSERT OR REPLACE` on the file stem). Triggers keep `captures_fts` in sync.
`cleanup_buffer` time-based delete	FTS and matching content-receipt deletion are attempted in one transaction before each JSON deletion; filesystem erasure remains authoritative if SQLite is temporarily unavailable, and final reconciliation removes stale searchable/receipt rows after recovery. The live receipt cache fails open until reconciliation is trustworthy.
`cleanup_buffer` size-based eviction	Same, including hard-cap eviction of unabsorbed files when necessary and continued eviction after one unlink failure.
Screenshot strip	Untouched. Strip only removes the base64 image; the indexed text (`visible_text`, `focused_value`, `window_title`, `app_name`, `url`) is unchanged.
`persome clean captures`	Securely delete capture files, indexed rows, and content-dedup receipts from the live store, retained snapshots, and integrity-quarantine copies.
`persome rebuild-captures-index`	Offline maintenance: run `persome stop` first. The command then holds the exclusive database gate while it atomically clears stale rows and indexes every surviving `~/.persome/capture-buffer/*.json` in one rollback-safe transaction. Use `--merge` after snapshot recovery to preserve older rows whose JSON aged out, then run `persome start`.

Indexed columns. Only the searchable text is in FTS: app_name, window_title, focused_value, visible_text, url. Filterable metadata (timestamp, bundle_id, focused_role) lives on the captures table for WHERE-clause filtering. Screenshots are deliberately not duplicated — the JSON file on disk stays the authoritative copy of the raw image bytes.

Tokenizer. unicode61 remove_diacritics 2 — case-insensitive, accent-folded, Unicode-aware. Same setup as the compressed-memory entries index.

If captures_fts falls out of sync (e.g. capture worker crashed mid-write, or the daemon was killed during cleanup), the index is recoverable in one shot. Rebuild first clears stale rows, then indexes every surviving JSON file:

persome rebuild-captures-index

Index health and the capture heartbeat

A runtime that is silent by design must let its owner tell intentional silence from a broken pipeline. The capture path feeds an in-process heartbeat (src/persome/index_health.py): every buffer write, every write-through FTS insert outcome, and every bounded-queue drop is recorded. The daemon’s index-health task (default every index_health.tick_seconds = 300s; runs in --capture-only too) then:

verifies the main B-tree (PRAGMA quick_check) plus both FTS5 inverted indexes (captures_fts is reconciled against its external-content captures table);
measures the buffer-vs-index backlog (on-disk captures that search cannot see yet);
classifies the pipeline as active / paused / idle / broken (broken = index_health.failure_streak_threshold consecutive FTS insert failures);
publishes the report to the private ~/.persome/.index-health.json sidecar.

The report surfaces on persome status (Index Health / Capture Pipeline / Index Backlog rows), on GET /status (index_health), coarsely on the unauthenticated GET /health (index, capture_pipeline states only), and as a warning banner in the /model viewer. When the evidence chain is unavailable, downstream readers degrade explicitly: search_captures returns an in-band index_health note while degraded, and raises an actionable error instead of a raw SQLite traceback when the index is corrupt. The self-check only observes — repair stays with startup integrity recovery and persome rebuild-captures-index.

Pause

persome pause

Drops a ~/.persome/.paused sentinel. The watcher keeps streaming but capture_once short-circuits on sentinel presence. resume removes the sentinel.

Smoke test

persome capture-once

This is a low-level developer diagnostic. It creates a provider and scheduler in the calling CLI, writes one immediate capture, and can confirm the one-shot helper returns useful AX content. It does not run through the active daemon’s capture runner, prove the event watcher, bind a Runtime generation or lifecycle owner, validate a privacy/mode receipt, or wait for the daemon-owned OCR worker. It refuses to run while the Runtime PID or lifetime lock says the daemon is running or starting; use persome stop before isolating a helper problem. Use persome onboard for onboarding, update, and release proof.