← all sprint docs

hello@ JOB FAMILY — FULL AUDIT FINDINGS · 31 July 2026

Why this exists. Aga's pre-flight rule before arming the failed-purchase lane: "look for any inconsistencies, anything that could break or any bugs for all the jobs, document them all — and then we execute." This is that register.

Scope: all 21 retention/email scripts + the launchd wrapper/guardian + the live plists + live state files, read against AUTOMATION_JOB_REGISTER.md. Read-only audit by an independent agent; every finding below was re-verified by hand before any fix. Status key: ✅ FIXED (this session) · 🔴 OPEN · ⚪ ACCEPTED (known, not worth fixing yet).

0. The headline

30 findings. 8 fixed today (every one that could email the wrong person, plus the two that could take the whole rail down). The remainder are documented, ranked, and none of them block the failed-purchase lane.

The single most valuable find had nothing to do with the new lane: nothing was watching replies to dunning emails. Every dunning and failed-purchase note promises "Reply STOP and I won't email you about it again" — and no scanner watched those subjects, so a STOP never reached suppress.csv. Within one minute of the fix going in, it surfaced a real reply that had sat unread since 28 Jul: Simon replying "I've already cancelled the subscription" to a dunning note. Nobody had ever seen it.

1. FIXED THIS SESSION

# Sev What was wrong Fix
C1 CRITICAL STOP replies to dunning/failed-purchase were never captured. check_replies walks only R1 threads; reply_fulfil.RULES covered only the 986/976 arcs. A member could reply STOP and still get T2 three days later — auto-sent, no human in the loop. The failed-purchase lane's entire conversion mechanic ("hit reply and I'll send you a fresh link") also had no pipeline. reply_fulfil.py now watches all 4 payment subjects (imported, never retyped). STOP → suppress.csv automatically. Live replies are surfaced as 🔴 NEEDS AGA and counted into r1_status.json — deliberately no auto-draft, because an auto-drafted "here's your link" with no link is worse than nothing.
C2 CRITICAL upgrade_tag.py:286 read row['annual_saving'] — a key that never existed. KeyError on every live tagging, swallowed by a blanket except. AC fields + upgrade-eligible tag were written first, so 990 Monthly→Annual Upgrade Engine would start emailing while the log row was never written → Stage 2/3 could never fire, conversion detection never ran, member re-processed daily. Dry runs never reached the line. row['annual_saving_129']. Latent (gated behind the UPGRADE_LIVE sentinel) — it would have broken the day you armed the upgrade engine.
C3 (part) CRITICAL Non-atomic state writes + write-once-at-end. A crash mid-run lost the whole run's state → next run re-drafted the same notes → two identical drafts, both auto-sendable. failed_purchase_lane: atomic save_state() (tmp+rename) after every draft. Dunning/autosend pair: see 🔴 O1.
M7 MEDIUM→HIGH The shared Gmail token was written non-atomically. Four scheduled lanes + the guardian all refresh through one token.json; two interleaved writes corrupt it and every hello@ lane goes dark until manual re-auth. r1_lib._creds() now writes tmp+os.replace.
H1 HIGH The new lane's plist (written earlier this session) had --auto inside an XML comment — illegal XML. plutil passes but plistlib throws: job never loaded, and harden_launchd.py crashed outright, so no job could be wrapped or enrolled. Self-inflicted, caught by the audit. Comment reworded (no -- sequences) + a warning note in the plist so it can't recur. harden_launchd.py --audit verified working again.
H2 HIGH ever_paid() used unpaginated customers limit:10 + charges limit:100. A serial retrier (one had 21 attempts; Payment-Element checkouts can mint a customer per attempt) could push the successful charge off page 1 → "your payment didn't go through" emailed to a PAYING member. Fully paginated via a new _paged() helper, and fails SAFE: any Stripe error now returns True (treat as paid → don't email).
H3 HIGH "One note per person, ever" was actually one note per charge — state keyed by charge id, so a retrier who failed again next week got a second "one-time" note. Now also skips any email already in state.
M11 MEDIUM Charge filter was status != "succeeded", which swept up pending charges → a payment still processing could be told it failed. status == "failed" only.

Guarded against regression: an assert at import time fails loudly if the failed-purchase subject ever collides with a dunning subject — the invariant that keeps its drafts human-gated.

2. 🔴 OPEN — ranked, none blocking

31 Jul (later): O1 is FIXED (struck through below, kept for the record). O6 remains the gate on arming 976 Cancelled Win-Back — paginate the customer lookup first, or an active paying member can be tagged wb-eligible and offered a win-back discount.

# Sev Finding Why it can wait
~~O1~~ ✅ FIXED 31 Jul ~~The dunning ↔ autosend pair still has C3's double-send risk~~ — CLOSED. Aga ruled "fix it" 31 Jul. New tools/retention/state_io.py: atomic_write_json() (temp file in the same dir → fsync → os.replace, so a crash leaves the old file or the new one, never a truncated one) + lane_lock() (shared, non-blocking flock). Both payment_recovery_lane.py and send_payment_drafts.py now take the SAME lock around their whole run and exit 0 if the other holds it — a skipped pass is free, an overlapping pass is the double-send. Both also save state per item immediately after a touch/send, so a crash costs at most one member instead of the run's stamps. read_json() survives an empty/truncated file without the delete-the-file "fix" that re-drips T1. Verified: atomic round-trip + no temp files left · second process correctly excluded · both real scripts skip when the lock is held · live dry run of the lane leaves all 31 records byte-identical. —
O2 HIGH send_payment_drafts lists at most 100 drafts, unpaginated; Gmail returns newest-first, so the oldest (ripest to send) fall off first. Approved dunning email silently never sends and nothing reports it. Only bites above 100 simultaneous drafts. Follow nextPageToken.
O3 MEDIUM r1_queue_emails() fails open (except: return set()): if the subscriptions call fails while invoices succeed, the "R1 owns them" skip silently evaporates → two lanes email the same person in one week. Should raise and abort the run instead.
O4 MEDIUM Autosend does no send-time spacing or re-check: after an outage, T2 and T3 for one member can both be >14h and fire in the same pass; it never re-reads suppress.csv or re-checks whether the invoice got paid. One send per recipient per pass + re-check suppress/invoice.
O5 MEDIUM Guardian treats a long-running job as MISSED — compares only last_success, never the last_attempt stamp. com.tma.n973to758 runs ~95 min → burns both kicks and raises a false alert. Cosmetic today (kickstart on a running service is a no-op), but it cries wolf.
O6 MEDIUM saves()/check_saves cap customer lookups at 10 records → a member whose active sub sits on an 11th record is misread as "expired unsaved" → wb_tag marks them wb-eligible → when 976 Cancelled Win-Back activates, an active paying member gets a win-back discount. Latent until 976 is armed. Paginate before then.
O7 MEDIUM upgrade_tag._lifetime_paid caps at 100 invoices → undercounts "you've paid $X" precisely for the longest-tenure members the offer targets. Latent behind UPGRADE_LIVE.
O8 MEDIUM migrate_973_to_758.api() retries a POST after an ambiguous timeout → duplicate AC records → contact runs 758 Nurture Sequence 2 twice in parallel. Job self-completes ~2 Aug; check already_in_dst before retrying if it recurs.
O9 MEDIUM Crash-mid-run can duplicate reply drafts (check_replies/reply_fulfil save state only at end). Human-gated (Aga reads before sending). Same fix pattern as C3.
O10 MEDIUM Two unguarded cross-lane pairs: cart-abandon vs failed-purchase (same person, same day, two emails) once the blur-write ships; and send_upgrade_stage doesn't skip current R1 cancel-queue members. Both latent. Add the state-file checks when either ships.
O11 MEDIUM R1 can never re-touch a repeat canceller — already_sent() is a lifetime set, so someone saved in July is invisible if they cancel again next year. Design gap that grows quietly. Key on (email, subscription) or expire on re-entry.
O12 LOW ×10 draft_body reads only text/plain (HTML-only edited draft → empty body → needs_aga sees subject alone) · sent stamp hits the first matching record when a person has two open invoices · missing internalDate → infinite age → immediate send · tz off-by-one on manual evening runs · watcher send-count scraped from stdout · EMAIL_FIXUPS typo remap not applied in the new lane · reply_fulfil searches in:anywhere (spam included) · retention_cockpit_sync unescaped regex replacement · wb_tag never retries NOT_IN_AC · wrapper keeps a stale last_error. Hygiene. Documented for whoever touches these next.

3. Register corrections applied

The audit checked AUTOMATION_JOB_REGISTER.md against reality and found these drifted:

4. The safety question, answered explicitly

Can send_payment_drafts --auto ever send a failed-purchase draft?

No. Its filter requires BOTH: subject exactly equal to one of the three dunning subjects (imported from payment_recovery_lane.SUBJECTS — list membership, not prefix match) AND the recipient present in payment_recovery_state.json. The failed-purchase subject diverges at character 5 and its recipients live in a different state file. R1 and reply_fulfil drafts all carry Re: prefixes and cannot match either. Unrecognised drafts are skipped and counted "left untouched".

The invariant previously hung on string constants in two files with nothing guarding it — now covered by an import-time assert.

5. Pagination inventory (the failure class that keeps recurring)

A limit=100 scan with no pagination silently truncated a Stripe census on 30 Jul (undercount 4×) and would have emailed a paying member here. Full sweep:

Standing rule going forward: never use a bare limit=N list for a safety check or a count.

Audit + fixes: 31 Jul 2026. Companion: AUTOMATION_JOB_REGISTER.md (canonical job register) · cockpit ⚙️ Daily Jobs tab (live health).