org.inovus.notifications Private
↗ Repo

# High availability

Where resilience comes from today, what we already get “for free” on Cloudflare Workers, and future options if we need more. This is planning context — not a commitment to build every idea below.

# Picture of the system

  Apps / npm client
         │
         ▼
  Cloudflare Worker  ──many isolates at the edge──┐
  (auth, routes, rate limits)                      │
         │                                         │
         ▼                                         │
  Supabase Postgres (PostgREST)  ◄─────────────────┘
  (durable inbox data)
  • Worker = compute + auth. Stateless for production data.
  • Supabase = source of truth for notifications.
  • Cognito = identity (tokens). Outage there affects login/publish credentials, not rows already stored.

HA for “can users see their inbox?” is mostly database + Worker platform. HA for “can someone publish?” also needs Cognito (or API keys) and the Worker reachable.

# What we already get (no extra work)

# Many Workers, one database — by design

A single deployed script (e.g. inovus-notifications) is not one long-lived server. Cloudflare runs many isolates across the network. They all talk to the same Supabase project with the service role.

That is safe because:

Property Why it matters
No production state in Worker memory Restart / new isolate doesn’t lose inbox data (DB_DRIVER=supabase)
Every store method is user-scoped Handlers don’t “own” rows outside userGuid
Dedupe is enforced in Postgres Two isolates publishing the same (user, source, dedupeKey) at once → one row (conflict treated as duplicate)
Deny-all RLS + service role only DB isn’t opened to browsers; Worker is the trusted path

So: parallel Workers → one Supabase is normal and intended. You do not need a second Worker script just to get horizontal scale of compute.

See also Storage backends (memory vs Supabase) and DESIGN.md.

# Cloudflare platform resilience

Roughly what the platform gives you without us building a cluster:

  • Traffic spreads across locations; a bad isolate doesn’t take down the whole product
  • Deploys roll forward; you keep a known Worker version + config in the dashboard
  • Custom domain (notifications.totumcloud.com) terminates TLS at the edge

Exact SLAs are Cloudflare’s / your plan’s — treat this as architecture, not a contract quote.

# Client-side resilience (npm package)

@inovus-medical/notifications already retries on 429 / 5xx / network (publish only when dedupeKey is set). That softens brief blips; it does not replace HA for a long outage. See Publishing — dedupeKey.

# What this does not cover today

If this fails… Effect
Supabase (region / project) API can’t read/write inbox (/health storage check fails)
Cognito (issuer / JWKS) New logins / JWT publish fail; existing API-key publishers may still work
Wrong Worker config (secrets wiped, bad AUTH_ISSUER) Self-inflicted outage — ops, not multi-instance
Single custom domain / DNS Users can’t reach the API even if Workers elsewhere are fine

Worker fan-out ≠ database redundancy. Scaling isolates does not keep serving inbox data if Postgres is down.

# Retention cron (single-script note)

The Worker’s scheduled job purges rows older than RETENTION_DAYS. With one production script, one cron schedule is enough.

If you later run multiple named Workers that each enable the same cron against the same DB, purges may run more than once (usually harmless, wasteful). Prefer one script owning retention, or disable cron on standbys.

# Future options (not built yet)

These are possible directions if product/ops ask for more HA. Order roughly by “how much we’d have to invent.”

# 1. Stronger database HA (highest leverage)

Inbox durability lives in Supabase/Postgres.

Ideas:

  • Supabase plan features (HA / replicas) as they apply to our project
  • Backups + tested restore
  • Clear RPO/RTO with the platform team
  • Longer term: read replica for heavy list traffic (would need adapter/query design)

Without this, extra Workers don’t help a DB outage.

# 2. Explicit Worker failover (optional)

Only useful if we want our own standby path beyond Cloudflare’s default routing:

Approach Idea Watch-outs
Second Worker + DNS failover Standby script, same Supabase secrets, flip DNS/custom domain Cron ownership; keep config in sync; rate-limit namespaces
Blue/green Workers Two versions; switch route when promoting Same as above; smoke /health before switch
Multi-account / multi-region Workers Rare; usually overkill Still one DB unless DB is multi-region too

Apps using a fixed baseUrl need DNS or client config to follow failover.

# 3. Multi-region data (hard)

True active-active inboxes across regions means multi-region Postgres or a different store, conflict rules for dedupe, and careful auth. Not planned as a near-term Worker tweak.

# 4. Soften dependency on Cognito for publish

API keys already let backends publish without a live user login. Future: more publishers on keys, or a documented degraded mode if the IdP is unhealthy (ops policy, not a new HA product by itself).

# 5. Observability and recovery drills

Often better ROI than a second Worker:

  • Alert on /health failing (including “Storage unreachable”)
  • Dashboard for Worker errors / Supabase status
  • Runbook: restore DB, roll back Worker, rotate secrets

# 6. Push / realtime later

Polling + ETag is the HA-friendly path today (any isolate can serve). WebSockets / Durable Objects would add sticky infrastructure to reason about — see DESIGN trade-offs. Not required for HA of the inbox store.

# Practical stance for now

  1. Rely on Cloudflare’s multi-isolate Workers + one production Supabase + race-safe dedupe.
  2. Treat Supabase (and backups) as the real durability story.
  3. Don’t spin up a second Worker “for redundancy” unless there is a concrete failover requirement (DNS, region, blue/green).
  4. Do keep deploy/config hygiene (keep_vars, secrets, smoke /health) — config mistakes cause more outages than isolate count.
  5. Revisit this page when we re-enable CI deploys or change hosting.