Follow the failure to its source

Trace It

no-slop-bug-hunter's contract opens with "the bug is reproduced — if it can't be reproduced, it hasn't been found." This skill is what you run when you can't satisfy that line. It fixes nothing. Its entire job is to convert a vague symptom into evidence: a captured trace showing the bad state, and a one-sentence located cause you can hand downstream. It sits upstream of no-slop-bug-hunter and bug-zero the way hotspot-scout sits upstream of optimize-proof: the finder half, kept strictly apart from the fixer half.

Core Insight: The gate is that the instrumentation reproduces the hypothesis: a probe you installed captures the bad state (the wrong value, the impossible interleaving, the call that never came) before any product code changes. Until you hold that capture, you have a theory, and fixes built on theories are guesses that sometimes compile. The discipline cuts both ways: no fixing before the capture, and no probe left behind after. Instrumentation is scaffolding, and scaffolding left up becomes the "fix" nobody can explain in six months.

The contract

Before installing any probe, the following must be written down:

1. The symptom, as an observation. "3 users/week get a blank invoice PDF." Not "PDF generation is broken"; that's already a theory wearing the symptom's clothes.
2. One falsifiable hypothesis. A sentence a probe could confirm or deny: "the render starts before the line-items query resolves." If no probe could deny it, it isn't a hypothesis, it's a vibe.
3. The minimal probe set that decides it. Which values, at which points, with which correlation id. If the probe list reads "add logging to the module," start over: you're instrumenting the codebase, not the hypothesis.
4. A fix freeze. No product-code change other than instrumentation until the gate passes. The urge to "just fix the likely thing while we wait" is the urge this skill exists to kill.

Hard rules while probing

• Instrument the hypothesis, not the codebase. A handful of probes per pass. Carpet-logging produces a haystack and calls it observability.
• Structured, correlated, or it didn't happen. Every probe emits fields (not prose) and carries the request/job/session id end to end. An uncorrelated log line from a concurrent system is gossip. Recipes in references/PROBES.md.
• Probes must be prod-safe by construction: sampled, rate-limited, no secrets or PII in fields, no behavior change beyond negligible cost. A probe that changes timing enough to hide the bug is evidence in itself (see anti-patterns).
• One hypothesis per probe set. Probes for a dead hypothesis come OUT before the next goes in. Accumulated zombie probes are how carpet-logging happens one reasonable step at a time.
• Logging is never the fix (no-slop rule, inherited verbatim). The capture is the end of THIS skill, not the end of the bug.

The pass structure (capped)

Per pass:

1. HYPOTHESIZE: one falsifiable sentence, ranked from the symptom's shape (per-user → state/data; intermittent under load → race; time-of-day → clock, cache expiry, cron; one host → environment).
2. PROBE: install the minimal set; deploy to where the symptom lives.
3. TRAP: arm it and make the symptom come to you: wait on real traffic, replay captured requests, or amplify (loop, parallelism, clock skew: the same amplification matrix flake-zero uses; see PROBES.md Part 3).
4. READ: did the capture confirm, deny, or miss? - Confirmed → GATE PASSED. Write the located cause in one sentence with the capture attached. Hand to no-slop-bug-hunter (or the bug-zero pipeline). Then decommission (below). - Denied → the hypothesis is dead, which is progress. Pull its probes, next hypothesis. - Missed (symptom fired, probes saw nothing) → the probes are in the wrong place; the miss narrows where the cause ISN'T. Re-site, same hypothesis, one more trap.

Cap: three dead hypotheses without a capture → stop. The next move is not a fourth guess. Widen the trap (more traffic, longer window, heavier amplification), take a fundamentally different probe class (a crash-state capture instead of logs, a differential trace of a good vs bad request), or run problem-breakthrough on the framing. Escalate to the operator if the widening has real cost (prod sampling rate, retention).

Decommission (the second gate: the pass isn't over at the capture)

Every probe installed gets an explicit verdict, recorded next to the located cause:

• REMOVED: the default. Scaffolding comes down.
• PROMOTED: deliberately kept as permanent observability: named, leveled, documented, added at the right place (which is often not where the debug probe sat), and justified in one line ("this invariant broke once in prod; this line is the alarm"). Promotion is a decision, never inertia.

A probe with no verdict is a leak. git diff against the pre-pass checkpoint must show only promoted probes when this skill exits.

Stopping condition

Done = a captured trace showing the bad state and the capture repeats (re-arm the trap once; a capture you can't get twice is an anecdote) and the cause fits in one sentence a fixer can act on. Not "we added good logging." That's the consolation prize, and it's claimed via promotion, explicitly, or not at all.

Anti-patterns that look like progress

• Fix-on-theory: shipping the "likely fix" with the probes still armed "to confirm." If it was confirmable, confirm first; if the fix works you've learned nothing and left probes behind.
• Carpet logging: instrumenting the whole module because the hypothesis is weak. Weak hypothesis → sharpen the hypothesis, not the coverage.
• The vanishing act as victory: the probe changed timing and the bug stopped. That's not fixed, that's a race with a flashlight in its eyes: hand the evidence to deadlock-finder-and-fixer (if installed; otherwise switch to a timing-free probe class such as a crash/state capture).
• Leaving debug probes in as the outcome: "at least we'll see it next time" without the promotion decision. Next time is exactly when the uncorrelated, unleveled probe will mislead someone.
• Uncorrelated probes: five log lines that can't be joined to one request prove nothing in any system with concurrency, i.e. every system.
• Probing without the written symptom/hypothesis: you'll read whatever the logs show as confirmation of whatever you already believed.

THE EXACT PROMPT

/trace-it
Target:  <project path>
Symptom: <the observation, verbatim — what happens, to whom, how often>
Lives:   <where the symptom occurs: prod / staging / CI / one host>
Budget:  <hypothesis cap per run; default 3>

References

• PROBES.md: probe recipes (structured logs, spans, metrics, crash-state capture), correlation-id threading, prod-safe sampling, trap amplifiers, and the promote-vs-remove checklist