Chapter 9: The Evidence Trail — Persistence, Failure, and Verification

Source lock. This chapter describes pi-subagents 0.18.0 at 3f9d35cd078d18a141eb5a6d8f4fc5010d756280 against Pi 0.84.2 at 914cf1472e715297caa30db4b9535d534a9eb718. All source links are commit-pinned.

An agent run can leave several artifacts with similar-looking JSON, yet no single component owns “persistence.” Pi owns resumable conversation history. pi-subagents owns an optional streaming transcript, settings, memory instructions, schedules, and worktree cleanup. This chapter’s observable goal is to produce an evidence-backed diagnosis by naming each artifact, its owner, the event that writes it, and the proof boundary behind the claim.

1. Evidence has levels

Evidence strength is claim-specific. Use the strongest level that actually crosses the boundary in question.

Do not rank only by filenames. schedule-e2e.test.ts uses real timers and the real on-disk store, but a mocked AgentManager. The error-status and worktree suites boot real Pi and the real extension on a faux provider. Faux is not automatically weak: an empty provider-error turn is easier to prove with a scripted response than with a live model that cannot be ordered to fail in that exact shape.

2. Persistence matrix: six records, six owners

View larger

There is also a smaller parent-session trace: on top-level completion, the extension calls pi.appendEntry("subagents:record", …) with status, result, and error. That is useful for history reconstruction, but it is not the child’s resumable session and does not replace any row above.

Recall

Which artifact must exist to reopen an evicted agent’s conversation: the .output file or the Pi child session file?

3. Follow the write event, not the filename

Pi’s AgentSession receives agent-core events. On message_end, it calls SessionManager.appendMessage; the manager maintains an append-only tree whose entries carry id and parentId. SessionManager.create is persistent, open loads an existing conversation, and inMemory retains the same logical API without a file. pi-subagents chooses among those constructors; it does not implement Pi’s session codec.

The .output writer instead subscribes as an observer. Its counter tracks the portion of session.messages already projected. Compaction makes this coupling visible: Pi can replace the message array with a shorter summary. The writer flushes before compaction and re-anchors after a successful compaction_end; a microtask accounts for Pi trimming an overflow error after emitting that event. The real-Pi faux test proves manual-compaction array replacement and continued streaming at the pinned dependency; a fake-session unit test simulates the narrower post-event overflow trim. Both are compatibility evidence, not a public guarantee that upstream event order will never change.

Settings show a third write model. Malformed JSON produces a warning and contributes {}; invalid individual fields are silently omitted by the sanitizer. Project values override global values field by field. saveSettings then writes the supplied project object directly. Therefore “the toggle changed now” and “the toggle will survive restart” are separate outcomes, represented by the persisted flag and toast.

Memory is different again: the extension reads an index and injects instructions. A writable memory block eagerly creates the directory and tells the agent to use files; a read-only block only exposes existing MEMORY.md. Symlink checks and safe-name validation constrain path handling, but memory truth still depends on what the model writes.

Predict

Project settings contain outputTranscript: false and rememberAgents: true. Agent auditor.md declares output_transcript: true, persist_session: false, memory: project, and has read, write, and edit. One Agent call registers it with schedule; a later timer fire runs it with isolation: worktree. What can each of those two moments write?

4. Terminal failure is a classification pipeline

A resolved promise is not synonymous with success. Pi may resolve a turn whose final assistant message has stopReason: "error". For a fresh spawn, pi-subagents inspects the final assistant message created during the current invocation, bounded by the message count captured before the prompt. Settlement priority is: an existing stopped state, hard abort, final-turn failure, soft-limit steered, then clean completion. Foreground resume has the narrower semantics described in Chapter 5.

Text and status are deliberately orthogonal. An errored turn may retain partial text, which the UI labels as partial output rather than laundering it into a clean answer. On resume, the starting message index prevents a failed empty turn from returning the previous successful answer. Startup failures are different: the tool execution throws so Pi marks the tool call failed; returning a diagnostic would falsely resemble a subagent result.

Scheduler status is a projection of this record state. After a spawned record settles, completed and steered become schedule success; error, aborted, and stopped become schedule error. That mapping does not make scheduling exactly once: a process can stop between a side effect and the store update, and overlapping interval fires bypass the normal concurrency queue.

5. Source-trace capstone: from symptom to owner

Suppose the parent model sees:

Agent failed: invalid request: provider rejected the prompt

Partial output before the failure:
EARLIER-PARTIAL-TEXT

Trace it backwards:

1. Presentation owners — index.ts and status-note.ts: the Agent tool chooses the error branch and prints Agent failed:; partialOutputSuffix supplies the explicitly labeled salvage text.
2. Lifecycle owner — AgentManager: a resolved RunResult.failure becomes status error; responseText remains on the record. A hard abort has higher precedence, while an external stopped state is not overwritten.
3. Classification owner — agent-runner.ts: finalTurnError reads only the final assistant message in the current invocation. Earlier text is recoverable context, not evidence that the final turn succeeded.
4. Protocol owner — Pi AgentSession: the assistant message and its stop reason arrive through real session events and are appended to Pi session history on message_end.
5. Independent observer — .output: if enabled, streaming may also contain the messages, but it neither decides terminal status nor makes the conversation resumable.
6. Proof: the faux full-stack regression boots a real Pi loader, real extension, real runner, and real child session, then scripts the otherwise hard-to-induce empty error. It proves this pipeline without claiming live-provider verification.

Practice

A bug report says, “The transcript says the agent answered, but /agents says error.” Write a three-part investigation plan.

6. Brittle upstream bridges

Three seams deserve explicit compatibility alarms:

• Pi changed createAgentSession from modelRegistry to modelRuntime, while ExtensionContext still exposes the registry façade. pi-subagents reads (ctx.modelRegistry as …).runtime and passes both fields. This modelRegistry.runtime bridge is intentionally type-cast and brittle; a mock-wiring test proves propagation, not that every supported Pi build exposes the field.
• Transcript compaction recovery depends on Pi’s message-array replacement and overflow-trim event ordering. A real-Pi faux test guards array replacement under manual compaction; the overflow trim after compaction_end is simulated by a fake-session unit test, so that narrower ordering claim has mock-wiring evidence only.
• Child cleanup reaches session.extensionRunner to emit session_shutdown before dispose(), with a timeout and graceful degradation for missing surfaces. This closes real lifecycle leaks but couples to upstream session internals.

Treat these as monitored dependency seams. When Pi changes, rerun the real-Pi suites and inspect upstream source before “simplifying” the adapters.

7. Explicit proof gaps

• Live provider: not run here; authentication, network backpressure, provider-specific stream errors, and billing paths remain unverified.
• Pi session durability: source and Pi tests establish append behavior, but this chapter does not prove power-loss durability, filesystem fsync, concurrent writers to one session file, or recovery from a torn final line.
• .output: compaction and resume behavior are tested; disk-full, process-kill between events, concurrent writers, and best-effort write-loss recovery are not.
• Settings: merge, sanitization, and save failure are tested on real temporary files; concurrent UI/process writes and crash-atomic replacement are not provided.
• Memory: paths, truncation, and symlink rejection are tested; no test proves that a model writes useful, current, non-conflicting memory.
• Scheduler: tests cover stale PID-lock recovery, sequential lock release, temp-file replacement, real timers, and mocked-manager status projection. They do not exercise simultaneous writers or crash atomicity. There is no real-Pi scheduler e2e here, nor proof of missed-run replay, exactly-once firing, long-duration clock behavior, sleep/wake handling, or process-independent execution.
• Worktree: a real-git/real-Pi/faux e2e proves the successful branch path. The outer cleanup catch can force-remove a corrupted worktree and return { hasChanges: false }, losing work without surfacing the failure. Reliability beyond the success path must not be claimed.

8. End-to-end capstone: prove the complete delegation path

Without reopening earlier answers, trace one top-level foreground Agent call and then contrast the background delivery branch. Your artifact must be a table with one row per edge:

tool registration -> parameter validation -> AgentRecord / queue decision
-> runAgent() -> createAgentSession() -> prompt() -> record settlement
-> foreground tool result OR background follow-up delivery

For every edge, record:

1. the component that owns the transition;
2. one commit-pinned source or test that crosses it;
3. the observable state before and after it; and
4. one proof gap or guarantee that the evidence does not establish.

Finish with two boundary statements: why the child is an in-process Pi AgentSession, not an OS sandbox; and why background delivery is not a second execution engine. If an edge cannot be supported at the pinned versions, label it unverified instead of filling it from memory.

You can now

• trace an Agent invocation from Pi’s extension contract through a child AgentSession and back to its parent;
• predict record, queue, control, settlement, and delivery behavior for foreground, background, and nested runs;
• resolve the effective model, prompt, context, tools, extensions, persistence, and isolation policy of a child;
• identify the owning component and strongest available evidence for each behavior; and
• state the security and reliability boundary without turning an implementation control into a sandbox guarantee.

Source index

pi-subagents, commit 3f9d35cd078d18a141eb5a6d8f4fc5010d756280

• Agent construction: memory injection, terminal classification, session choice and the runtime bridge, and invocation-bounded output
• Manager ownership: record settlement and worktree result attachment and session-file tombstones
• Transcript projection and compaction re-anchor: src/output-file.ts lines 40–144
• Partial-output labeling: src/status-note.ts lines 82–90
• Settings merge/write and persistence reporting: src/settings.ts lines 376–489
• Memory paths and prompt blocks: src/memory.ts lines 20–178
• Schedule store and timer dispatcher: src/schedule-store.ts lines 17–152, src/schedule.ts lines 49–297
• Host wiring: session-bound scheduler, switch/shutdown, transcript gate, stream attachment, and user-visible failure
• Worktree success and destructive failure path: src/worktree.ts lines 60–210
• Evidence fixtures: real-Pi faux error status, real-Pi compaction, fake-session overflow ordering, real-Pi/real-git worktree, and real timers with mocked manager

Pi, commit 914cf1472e715297caa30db4b9535d534a9eb718

• Pi session manager: append-only tree and deferred first flush and create/open/in-memory constructors
• Agent event persistence on message_end: agent-session.ts lines 620–665