Silent Failure Audit — 2026-09-21

Automated read-only audit for swallowed errors, unlogged fallbacks that degrade quality, and discarded error context. No source code was modified.

Summary


Files changed in last 7 days	676 (551 code files)
Code files scanned (changed, non-test, still present)	376
Additional sampled unchanged files (`routes/`, `utils/`)	10
Total files scanned	386
Candidate swallowed-catch sites detected (automated)	295 TS/JS + 59 Python
Candidates manually verified by reading source	41
Confirmed findings	1 critical · 4 high · 7 medium · 4 low

Sampled unchanged files

routes/push.ts, routes/admin-uploads.ts, routes/lti.ts, routes/auth.ts, routes/admin.ts, routes/gpt.ts, routes/workshops.ts, utils/csv-parser.ts, utils/billing-messages.ts, utils/app-logger.ts. These were clean for swallowed catches (2 benign hits) but are the heaviest console.* users in the codebase (see M7).

What the audit did not find (worth recording)

The AI provider cascade logs its failovers well. smart-cascade-converter.ts names the failing provider, the model, and the reason on every escalation (lines 1667, 1681, 1685, 1692, 1763). The “cascade falls back without saying which provider failed” risk does not apply here.
utils/pii-scrubber.ts fails open to 'off' but logs at error level with an explicit “PII/PHI protection may not be active” message — this is the correct pattern for a deliberate fail-open (see L3).
utils/pdf-utils.ts:80 (permitsExtraction = true on unreadable flags) is a documented deliberate choice tied to #3230, not a defect.

CRITICAL

C1 — `invalid-nesting` WCAG rule reports “pass” when HTML parsing fails

File: workers/api/src/services/wcag-validator.ts:1724 Severity: Critical

    } catch (_err) {
      // If parsing fails, skip this check
    }
    const nestingPassed = nestingViolations.size === 0;   // 1727
    ...
    evaluatedRules.push({
      id: 'invalid-nesting',
      result: nestingPassed ? 'pass' : 'fail',            // 1742

If parseDom(html) or the tree walk throws, nestingViolations stays empty, so nestingPassed is true and the rule is emitted to evaluatedRules as a passing WCAG 4.1.1 check that never actually ran. Nothing is logged.

This is a false conformance claim, the same class of defect as #3231 (VPAT multimedia “N/A” claimed without checking). The evaluated-rules array feeds customer-facing ACR/report output.

The fix already exists 200 lines away. The sibling nested-interactive rule in the same file does it correctly:

    } catch (_err) {
      // Parsing failed — record as warning rather than a false pass
      parseFailed = true;                                  // 1935
    }
    result: parseFailed ? 'warning' : nestedInteractiveFound ? 'fail' : 'pass',

Recommended fix: add a parseFailed flag to the invalid-nesting block, emit result: 'warning' when set, and log the parse error via getAppLogger(). Then grep the rest of wcag-validator.ts for any other rule whose pass is derived from an empty violations set inside a try.

HIGH

H1 — Entitlement `dbError` fails open in 8 places, silently, on paid AI work

Severity: High

consumeEntitlement / checkCredits return a dbError flag for a transient backend failure. The codebase handles it two opposite ways:

Fail-closed (correct):

routes/convert.ts:1139 — returns BILLING_UNAVAILABLE 503
routes/forms.ts:287, routes/forms-pdf.ts:159, routes/forms-docx.ts:115, routes/forms-detect.ts:184, routes/flyer.ts:456 — same 503

Fail-open (silent):

File:line	Effect
`workers/api/src/routes/remediate-v2.ts:144`	`if (!charge.ok && !charge.dbError)` → proceeds unbilled
`workers/api/src/routes/remediate-v2.ts:265`	same
`workers/api/src/routes/remediate-v2.ts:656`	same (alt-text suggestions)
`workers/api/src/routes/remediate-v2.ts:909`	`if (charge.dbError) return;`
`workers/api/src/routes/remediate-pptx.ts:121`	proceeds unbilled
`workers/api/src/routes/remediate-docx.ts:122`	proceeds unbilled
`workers/api/src/mcp/tools/url-remediate.ts:240, 253, 420`	`if (charge.dbError) return;`
`workers/api/src/services/site-audit.ts:122`	`if (res.dbError) return;`

Failure scenario: Supabase has a 5-minute blip. Every request to /api/remediate-v2/*, /remediate-pptx, /remediate-docx, the MCP url-remediate tools and site-audit proceeds with full paid Bedrock work, charges nothing, and writes no log line. There is no signal in Grafana/Loki that billing was bypassed, and no way to reconcile afterwards.

Recommended fix: pick one policy per route and make it explicit. At minimum, every dbError branch must getAppLogger()?.error('entitlement_db_error', {...}) so a bypass is reconcilable. Prefer fail-closed 503 (matching convert/forms/flyer) for user-initiated paid work; where fail-open is a deliberate product decision, record a cost_ledger row flagged unbilled_db_error per the cost-tracking standard.

H2 — `updateFileProgress` swallows every error, including the job-failure marker

File: workers/api/src/scheduler/chunk-scheduler.ts:1778-1798 (catch at 1796) Severity: High

      await putFileMeta(this.config.storage, userId, fileId, meta);
    } catch (_err) {
      // Non-critical — don't fail the chunk on KV update errors
    }

The comment is true for progress ticks (lines 477, 511, 603) but not for line 1523, which is the path that marks the user-visible file as failed:

    } catch (err) {
      log('error', `Assembly failed for job ${parentJob.id}: ${message}`);
      await this.supabase.from('large_conversion_jobs').update({ status: 'failed', ... })
      await this.updateFileProgress(parentJob.userId, parentJob.fileId, 0, undefined, message);  // 1523

Failure scenario: assembly fails; KV/R2 is briefly unavailable when getFileMeta/putFileMeta runs. The large_conversion_jobs row says failed, but the file the customer polls stays processing forever, with zero log output — the swallow has no log() call at all. The dashboard spins; nobody is alerted; the only trace is the assembly error line, which looks handled.

Recommended fix: log inside the catch (log('error', ...) with userId/fileId), and when the error argument is set, treat a write failure as significant — retry with the existing withRetry helper used elsewhere in this class, and alert on exhaustion.

H3 — 55 Supabase writes discard `error`; the job-failure subset strands jobs

Severity: High

supabase-js does not throw — it returns { error }. These call sites neither destructure nor check it. The dangerous subset is the writes whose only job is to record a terminal state:

File:line	Write
`workers/api/src/routes/audit.ts:389`	`audit_runs → status:'failed', error:'INSUFFICIENT_CREDITS'`
`workers/api/src/routes/audit.ts:401`	`audit_runs → status:'failed', ENQUEUE_FAILED`
`workers/api/src/routes/audit.ts:941`	`audit_runs → status:'complete'` (+ grade, score)
`workers/api/src/routes/forms-detect.ts:251, 273, 561`	`form_detect_jobs → status:'failed'`
`workers/api/src/routes/forms-docx.ts:167, 187`	`docx_form_conversions → status:'failed'`
`workers/api/src/routes/forms-pdf.ts:223, 244, 379, 400`	`*_conversions → status:'failed'`
`workers/api/src/routes/org-chart/extract.ts:168, 227, 273, 530`	`org_charts → extraction_status:'failed'`
`workers/api/src/scheduler/chunk-scheduler.ts:1518`	`large_conversion_jobs → status:'failed'`
`workers/course-map-api/src/routes/onboarding.ts:162`	`onboarding_jobs → status:'failed'`

Failure scenario: the failure-marking UPDATE is rejected (RLS, constraint, connection reset). The route returns its error response to the caller, but the row stays pending/processing. A user who reloads sees a job that never finishes; audit.ts:941 is worse — the audit completed and the grade is lost with no error anywhere.

The remaining 55−22 are analytics/telemetry writes (org_chart_share_views, crm_issue_events, view-count bumps) where a silent drop is lower impact but still unobservable.

Recommended fix: destructure const { error } = await supabase… on every write and log it. This is the same defect class the 4-agent review fixed in #3199/#3204 (C1–C3 were all const {data} = await supabase dropping error into a fail-open); it has regrown in the routes above. A lint rule (no-unchecked-supabase-write) would stop the regrowth — currently there is none.

H4 — VPAT parser silently downgrades from AI parse to regex parse

File: apps/vpat/src/lib/parsers.ts:297 and :622 Severity: High

  if (authToken) {
    try {
      ...
      if (text.length > 100) {
        onProgress?.({ phase: 'analyzing' });
        return parseViaAi({ text }, authToken, opts);
      }
    } catch {
      // Fall through to regex parsing
    }
  }
  // Fallback: regex/DOM-based parsing
  return parseHtmlTables(text);

The try wraps the entire parseViaAi call, so a network error, a 500 from vpat-parse, an auth failure, or a Bedrock error is indistinguishable from “document too short for AI”. The user is shown phase: 'analyzing', then silently receives the weaker regex parse of their VPAT — with no log, no toast, and no marker on the result saying which parser produced it.

Failure scenario: workers/vpat-parse is down. Every upload still “succeeds”; conformance rows are parsed by regex and quietly differ from what the AI path would have produced. Since this output feeds ACR/VPAT conformance statements, the degradation changes the claims made about a customer’s product.

Recommended fix: narrow the try to the DOMParser call only, catch parseViaAi failures separately, report them through useNotifyError (the pattern from #2633), and stamp the parse mode (ai | regex) on the result so a degraded parse is visible in the UI and in the stored record.

MEDIUM

M1 — Gateway job failure marker is a floating promise

File: workers/api/src/routes/gateway.ts:105 (also :570)

  ).catch((err) => {
    console.error(`[gateway] Conversion ${conversionId} failed:`, err);
    updateJob(storage, conversionId, {          // 105 — async, not awaited
      status: 'failed',
      error: err instanceof Error ? err.message : 'Unknown error',
    });
  });

updateJob is async (…): Promise<void> and does two KV round-trips. Every other call site awaits it (335, 354, 395, 404, 431, 442, 608, 666, 680, 749). Here the handler resolves before the write lands: a KV rejection becomes an unhandled rejection and the gateway job is never marked failed, so /api/gateway/status/:id reports fetching until the 1-hour TTL expires. :570 (onPageComplete) has the same shape for progress updates. Fix: await updateJob(...) (the callback is already async-capable) or .catch() the inner promise explicitly.

M2 — `image-alt` silently degrades from DOM to regex

File: workers/api/src/services/wcag-validator.ts:520

DOM parse failure falls through to html.matchAll(/<img[^>]*>/gi), unlogged. The regex path cannot see alt on multi-line tags reliably, and the rule is still reported with impact: 'critical' pass/fail. Same family as C1 but less severe (it degrades rather than inventing a pass). Fix: log the parse failure and mark the rule’s provenance.

M3 — Daily alt-text spend cap silently treated as 0 when KV is unavailable

File: workers/api/src/routes/remediate-v2.ts:629

  } catch {
    // KV unavailable → treat as 0 and let other guards handle it.
    dayCount = 0;
  }

The “other guards” are the consumeEntitlement call 20 lines down — which itself fails open on dbError (H1). A KV outage plus a Supabase blip removes both guards on paid Bedrock alt-text work, with no log from either. Fix: log the KV failure; fail closed on the cap when the counter is unreadable.

M4 — Visual-layout score failure indistinguishable from “not scored”

File: workers/api/src/services/smart-cascade-converter.ts:2314

                try {
                  const vl = await deps.scoreVisualLayout(pageBuffer, iterResult.html, browser);
                  iterVl = vl.score >= 0 ? vl.score : undefined;
                } catch (_err) { /* visual layout scoring failed — ignore */ }

iterVl stays undefined, which the acceptance check below treats as “no visual threshold to meet” — so a scorer crash silently accepts output that may have failed the visual-layout gate. Unlogged, so escalation-rate telemetry is skewed without anyone noticing. Fix: log and distinguish “scored below threshold” from “could not score” in the accept decision.

M5 — Tenant-domain CORS check: DB failure looks like a malformed origin

File: workers/api/src/index.ts:187-193

  try {
    const hostname = new URL(origin!).hostname;
    const verified = await isDomainVerified(hostname, c.env);
    if (verified) return origin;
  } catch (_err) {
    // Invalid URL
  }
  return null;

The await isDomainVerified(...) is inside the same try as new URL(). A Supabase/KV failure in verification silently denies CORS for every verified tenant domain, and the comment (“Invalid URL”) sends a debugger down the wrong path. Same shape in index-aws.ts:439/455 and server.ts:672/693/702, but those wrap only new URL() and are fine. Fix: move isDomainVerified out of the URL-parse try and log its failures.

M6 — Upload store failure returns a reason with no server-side log

File: workers/api/src/services/forms-upload-scan.ts:72

  } catch {
    return { ok: false, reason: 'store_error' };
  }

The user does get an error response via formUploadStoreErrorResponse, but the underlying R2/S3 exception is discarded — there is no way to tell a credential problem from a bucket outage from a size rejection. This sits on the malware-scan gate path (#3262), where the known .4 gotcha is exactly a credential mismatch (R2 creds against a native AWS S3 staging bucket) — the class of failure this catch makes invisible. Fix: log the error before returning.

M7 — 2,150 `console.*` calls instead of the mandated structured logger

Scope: workers/api/src, workers/batch/src — 290 files

File	`console.*` calls
`routes/convert.ts`	141
`services/smart-cascade-converter.ts`	102
`routes/stripe.ts`	85
`routes/admin.ts`	76
`routes/flyer.ts`	71
`services/chunk-assembler.ts`	61

The global standard mandates @anglinai/logger (log.error('msg', err, meta)), with Promtail → Loki → Grafana alerting on error/fatal. console.error output is not JSON-structured, carries no requestId/userId, and does not fire the EmailAlertTransport or the Grafana “Fatal error” alert rule. Every finding above that “is logged” via console.error is therefore still invisible to alerting. routes/admin.ts:134 additionally console.logs a full profile object on every admin check.

Fix: this is too large for one PR. Ratchet it: add a lint rule banning console.* in workers/**/src/** with the current offenders grandfathered via an allowlist file, then burn the allowlist down starting with the files that carry billing and conversion-failure paths (stripe.ts, convert.ts, flyer.ts).

LOW

L1 — `_has_alt_text` swallows all XML errors

File: services/pptx-remediate/modules/alt_text.py:201

    except Exception:
        pass
    return False

Direction is safe (a parse failure reads as “no alt text”, so alt text gets added), but a broad except Exception in a predicate hides real XML bugs and inflates the “images remediated” count. Fix: narrow to the expected exception types and log at warning.

L2 — Conversion history read failure yields a silently empty dashboard

File: apps/md/src/hooks/useConversions.ts:146

IndexedDB read failure → items stays empty with no notice, so a user with saved conversions sees an empty dashboard and may assume the data is gone. Documented as best-effort. Fix: surface a dismissible “couldn’t load your history” toast.

L3 — PII filter fails open to `'off'` (logged correctly — recorded for visibility)

File: workers/api/src/utils/pii-scrubber.ts:125, :173

Behaviour is a fail-open: a lookup failure disables PII/PHI scrubbing for a user who explicitly configured it, and a Comprehend failure passes a chunk through unredacted. Both log at error level with an explicit message, which is the right pattern and will reach the alert transport once M7 is resolved. No code change recommended; listed so the fail-open is on the record.

L4 — Orphaned ACR object when cache invalidation delete fails

File: workers/api/src/routes/files.ts:1328, :1538

      await storage.objects.delete(fileMeta.acrReportR2Key);
    } catch (_err) { /* ignore if already gone */ }

The metadata key is cleared regardless, so correctness is fine, but a failing delete leaks R2 objects with no counter. Fix: log at debug/warn so the leak is measurable.

Resource usage

GitHub Issues filed: #3393 (C1), #3394 (H1), #3395 (H2), #3396 (H3), #3397 (H4).


Start time (UTC)	2026-09-21T07:17:55Z
End time (UTC)	2026-09-21T07:27:43Z
Duration	9m 48s
Tool invocations	36 (all Bash; includes 3 scratchpad analysis scripts and 5 `gh issue create`)
Files read or analysed	386 scanned automatically; 41 read directly for verification
Source files modified	0

Method

git log --since="7 days ago" --name-only → 676 paths, filtered to 376 existing non-test code files.
Two automated passes (scratchpad/an.py, an2.py): brace-balanced extraction of every catch body in TS/JS and indentation-based extraction of every except body in Python, flagging bodies with no logging call and no re-throw. (The first pass mis-counted } catch (e) { lines — corrected in an2.py; the 867 → 295 drop is that fix, not a change in scope.)
Cross-referenced each candidate’s try block against critical operations (Supabase, Bedrock/Anthropic, fetch, R2/S3 put, file writes).
Separate pass for Supabase writes whose { error } is never destructured.
Manual read of 41 candidates to separate deliberate, documented degradation from genuine silent failure. Only verified findings are listed above.