Defensive Patterns & Postmortem

Defensive Patterns & Postmortem

In One Sentence

A long-running Agent that calls tools autonomously must defend three boundaries: report results faithfully (no swallowed errors, no fake success), clean up resources deterministically (no leaked handles/temp files/listeners), and minimize credential exposure (no keys in logs/results); when a bug still escapes, a postmortem focuses not on the one-line fix but on why every safety net failed and what new guard makes the same problem fail loudly next time.

Three Defensive Classes

1. Result Reporting: Failure Must Be Visible

  • Never swallow errors: a tool error must set IsError, never return an empty value faking success;
  • Discriminable errors: an error chain (pkg/llm/errorchain.go) separates retryable from deterministic failures;
  • panic backstop: external callbacks may panic; waterfall.Chain.RunSafe turns panic into error at the boundary;
  • Explicit truncation: oversized results are truncated and labeled by post-execute, never silently dropped.

2. Cleanup: Register Returns Its Inverse

1dispose := bus.On(handler)
2defer dispose()                 // listener removed on teardown
3
4proc, kill, err := subprocess.Start(...)
5defer kill()                    // child process reaped at Turn end
  • Subprocesses/terminal sessions close deterministically at Turn/Step end, cancellation propagated via ctx;
  • Temp artifacts land in the spill directory (pkg/spill) and are cleaned afterwards;
  • dispose/cleanup are idempotent, safe across multiple cleanup paths.

3. Credentials: Minimal Exposure

  • Reference, not plaintext: capabilities hold a CredentialRef, resolving via credentials.Store.Resolve only at use; keys never flow in cleartext in the config tree, prompts, or event log;
  • Dump redaction: paths marked by settings.MarkSecret are redacted in Describe; credentials.Store.Describe returns metadata only, never values;
  • Per-request fetch: pkg/credentials manages lifecycle (Set/Unset) and authorization flows.

Three Runtime Guardrails

Aborted Before Dispatch

If the request is cancelled before entering the tool body, the pipeline returns TOOL_ABORTED_BEFORE_DISPATCH and never runs the tool.

Cooperative Timeout

A tool declares TimeoutMs; tools.WrapTimeout arms the deadline and maps its own expiry to TOOL_TIMEOUT. A parent ctx cancelling first reads as an ordinary cancel. It is cooperative — Go cannot kill a goroutine, so the tool must observe ctx.Done().

Repeat Reminder

tools.RepeatState counts consecutive identical calls. CanonicalizeArgs deep-sorts keys so property order does not matter. At thresholds (default 3/5/8) it emits a gentle-then-detailed reminder, advisory only, never a veto. A user interjection resets the chain.

Tool Pairing

A compaction cut must never sit between a tool_call and its tool_result. compaction.BalancedCuts treats tool calls as +1 and results as -1; only zero-balance cuts are legal. NearestBalancedFrom moves a desired cut to the nearest balanced position. A single over-long result is handled by compaction.PruneText, which keeps the head and tail and replaces the middle with a marker (rune-safe slicing).

Postmortem: Four Questions

QuestionWhat to answer
What brokea short paragraph graspable in thirty seconds
Mechanismplain root cause, no blame
Why every safety net failedgaps in tests/tools/conventions, not a one-off typo
What guard was addedtests/rules/assertions so it fails loudly next time

Not every bug deserves a postmortem — only when it is simultaneously: (1) hidden — the mechanism is non-obvious; (2) systemic — it escaped through a gap in tests/tools/conventions; (3) costly to rediscover — it consumed real debugging time and would again.

A Case

Upstream postmortem 0001: a plugin added an extra export default apply; the loader got a bare function and dropped its inject, so the editor crashed on connect because it could not obtain the agents service. Lesson: the count of green unit tests does not prove wiring is correct — 178 passing tests never covered the “load a real plugin” path. The added guard is an integration assertion over the loading/wiring chain so a missing dependency fails at startup, not on first call.

Fix-to-Guard Loop

Defensive patterns push errors left to write/startup/compile time; when something still escapes, a postmortem upgrades the one-off fix into a durable guard — a failing test, an enforced rule, a fail-closed assertion. The safety net grows denser after every incident.

Source Map

ConceptGo implementationUpstream TypeScript
panic backstoppkg/waterfall/waterfall.go — RunSafedefensive patterns
Process cleanuppkg/subprocess, pkg/terminalcleanup conventions
Temp artifactspkg/spill(dsh-go counterpart)
Credential storepkg/credentials/credentials.go — Storecredentials
Secret redactionpkg/settings/settings.go — MarkSecret(dsh-go addition)
Aborted before dispatchpkg/tools/aborted.go — AbortedBeforeDispatchResultsession/session-checkpoint-policy
Cooperative timeoutpkg/tools/timeout.go — WrapTimeoutguard/timeout-policy
Repeat reminderpkg/tools/repeat.go — RepeatStateguard/repeat-tool-reminder
Tool pairingpkg/compaction/pairing.go — BalancedCutscompaction/tool-pairing
Head-tail prunepkg/compaction/pruner.go — PruneTextcompaction/compaction-tool-result-pruner
Error chainpkg/llm/errorchain.go(dsh-go addition)

Next Steps