Customers are not getting emails, SMS or push

Notification delivery is what a customer sees after a withdrawal decision, a KYC decision, a password reset or a one-time code. A silent mailer does not look like an outage — it looks like every decision you make turning into a support ticket, and it can run for days before anyone connects the two.

The console that answers this is System → Communication Tools → Notification Service (/admin/system/notification). The active tab is in the URL as ?tab=, so a link to one is shareable.

The seven tabs, and the four that matter in an incident

Overview polls every 30 seconds; the Queue tab refreshes every 5.

Every endpoint behind the six notification tabs gates on the single key access.notification.settings. The PWA tab is the exception, and it is not really part of this service: it reads and writes /api/admin/system/pwa (view.settings and edit.settings), uploads screenshots through /api/admin/system/pwa/screenshot and app icons through /api/admin/system/settings/logo (both edit.settings). A role that can work this console can therefore be unable to open that one tab, and vice versa.

Reading the queue

The Queue tab shows five counts, from GET .../queue/stats:

Beside them: a failureRate percentage, and a verdict that reads degraded once failures exceed 10% of completions.

Three shapes are worth recognising on sight:

• waiting climbing while active stays at zero. Nothing is consuming the queue. That is a process problem, not a mail problem — the backend or cron process that registers the queue processor is down, or Redis is unreachable. Do not touch SMTP settings for this one.
• failed climbing alongside completed. The transport is rejecting sends. Read a queue item's provider and attemptsMade, then reproduce it from the Test tab on that channel.
• A large delayed count with almost nothing failed. Retries backing off, or the rate limiter doing its job. It resolves itself.

The items list does not show failures

Up to 200 (50 by default), each with the user, the notification id, the title, the type, the channel, the template, the provider, attemptsMade, when it was queued and its age. Its status is only ever pending or processing — active jobs first, then waiting, then delayed.

Completed and failed jobs are deliberately excluded. Bull retains the last 100 completed and 500 failed, which would bury the handful still in flight. So a non-zero failed count on the stats card has no corresponding row anywhere in this console; to identify which notifications failed you need the backend log (pm2 logs backend), where each failure is written with its provider and notification id.

The clean action

Takes olderThan in milliseconds, defaulting to 24 hours, and returns how many were removed with the first ten ids.

It removes completed and failed jobs older than the grace period, and nothing else. It does not cancel anything waiting, does not retry anything failed, and cannot lose a message that has not been delivered. It is safe to run at any time — but it is housekeeping, not a fix: clearing a failure count does not clear its cause, and once cleared you have lost the evidence. Read the log first, clean second.

There is no admin control to pause or resume the queue. (paused appears in the stats endpoint's documented response schema; the handler does not return it.)

The health components

The response has an overall status plus a components object:

The overall status is healthy when Redis is connected and degraded when it is not. It is not a statement about SMTP: a perfectly connected Redis with a dead mail server still reads healthy. That is what makes "send a test first" the right first move.

A channel that never registered sends nothing, forever

Channels are registered once, during startup:

• IN_APP and EMAIL are always registered.
• SMS registers only if the resolved provider has credentials and the channel's own validation passes.
• PUSH registers only if FCM or VAPID is configured and the channel validates.

A channel that failed to register is simply absent from the list, and every send to it fails. Changing an environment variable re-registers nothing: the service reads the environment at boot, so a mail, SMS or push change needs a backend restart before any of these screens can tell you the truth about it.

Boot also writes three settings rows — emailChannelStatus, smsChannelStatus and pushChannelStatus — so the rest of the platform can see which channels are live without asking this endpoint.

The test senders

Four buttons, four endpoints, and every one of them sends a real message. Each takes a userId, defaulting to your own account.

Email and SMS accept an override recipient and both refuse to relay: the address must match the calling administrator's own account email, and the phone must match their own account phone after normalisation. An unguarded test endpoint is an open relay.

Each response names the channels delivered and the channels failed, with the channel's own error string. That string is the diagnosis — read it rather than the red or green.

There is a fifth, older test worth knowing about:

It always goes to the calling administrator's own account email and it goes through the queue rather than inline — so it also proves the queue is draining, which the Test tab's email button does not.

SMS: a successful send is not evidence of delivery

The SMS comparison screen is System → Communication Tools → SMS Providers (/admin/system/notification/sms).

The routing rule is fixed and not symmetric:

• One-time codes go to SMS_OTP_PROVIDER — twilio or msg91, defaulting to Twilio. That covers sign-in codes, phone verification, withdrawal codes, P2P codes and password-change codes.
• Everything else goes to Twilio, always. There is no switch.

Use this before writing a key into .env and restarting. It builds a throwaway provider from the values you supply, checks them against the vendor and discards it — nothing persisted, no SMS sent, and any field you omit falls back to the configured environment so you can test just the one value you are changing.

Templates: the two silent stoppers

System → Communication Tools → Notification Templates (/admin/system/notification/template). The screen itself — both the menu entry and the page — is gated on access.notification.template; view.notification.template and edit.notification.template are the keys the endpoints behind it enforce. A role granted only view and edit never sees the menu entry and cannot open the page, so grant all three together.

Each template carries a subject, an emailBody, an smsBody and a pushBody, plus three independent booleans — email, sms, push — saying which channels it is used for.

The wrapper is the other one, and it fails in the opposite direction.

Every outgoing email is a template body substituted into one shared wrapper — backend/email/templates/generalTemplate.html — which supplies the header, the logo, the footer and the unsubscribe link through the placeholders %SITE_URL%, %SITE_NAME%, %LOGO_URL%, %HEADER%, %MESSAGE%, %SUBJECT%, %FOOTER%, %YEAR% and %UNSUBSCRIBE_URL%.

If that file is missing or unreadable, prepareEmailTemplate throws 500 General email template not found and every template on the platform stops at once. The endpoint is read-only — the console shows you the wrapper, it does not edit it — so this is a file-on-disk problem, usually a bad deploy or an overwritten backend/email/ directory.

The environment decides the transport, and this console never writes it

Nothing about delivery is saved from this console: there is no PUT behind the Settings tab and no write behind the SMS screen. (The PWA tab does save — but what it writes is the installable-app manifest, its icons and its screenshots, none of which touch a transport.) Everything below is read from the process environment at boot.

Two behaviours that produce "nothing was even attempted":

• MAIL_DISABLED=true drops every notification email with a debug log line and returns a synthetic job id, so nothing appears in the queue at all.
• Reserved test domains are skipped. A recipient ending .invalid, .test, .example or .localhost is dropped before queueing. It only skips when every recipient is undeliverable, so a mixed list still goes out. If your staging accounts use @example.test addresses, this is why they receive nothing.

The Email Service entry on the system health card is a presence check on these variables — it never talks to a mail server. up there means "the right strings exist in the environment", nothing more.

Full variable reference, including the naming families that configure nothing, is in Environment variables.

Fault to first screen

Related

• The notification service — the full console reference, the four mail transports and the SMS/push provider setup.
• Environment variables — every mail, SMS and push variable.
• Troubleshooting — "Emails are not being sent", from the infrastructure side.