Case study

Our n8n Alerts Failed Silently for Weeks: How We Found It and Fixed It

The most dangerous automation failure is the one that looks like silence. Our notification workflow stopped delivering and nothing told us. Here is what happened in our own setup, and the checks we added so it cannot hide again.

Published 6 November 2026 · 5 min read · Our own system, numbers from its execution history
Disclosure: this describes our own automation. We do not sell an automation service today, and the fix below is one example, not a promise about yours. No keys, addresses or customer data appear here.

What went wrong

We run n8n for scheduled jobs and for relaying alerts to a chat channel. On the n8n 2.x series, expressions that read environment variables (the usual place for tokens and webhook secrets) were blocked. Workflows that depended on them did not crash loudly. They ran, failed at the first node that needed a secret, and ended with an error nobody was watching.

  • The alert relay failed on 28 of its 30 recorded runs. Every run since 23 October that tried to send a message had failed.
  • Several scheduled company workflows had been failing since about 10 October for the same reason.
  • No alert fired about the broken alerting. That is the point of the title.

How we found it

Not through a dashboard, but by reading the execution history directly and asking one question per workflow: what did its most recent run do? A workflow that last succeeded days ago, or whose last run errored, is not healthy, however green the process list looks.

What we changed

  1. Secrets moved out of expressions into n8n's own credential store. The Telegram token and the webhook secret are now credentials; no secret is read from the environment inside a workflow.
  2. The webhook requires a header secret. We tested both sides: a call without the secret is refused (HTTP 403), a call with it is accepted (HTTP 200).
  3. Retry on fail. The send node now retries three times, three seconds apart, because one of the recorded errors was a plain connection timeout.
  4. A health check that lets the latest execution decide. It reads each workflow's last run and reports FAILING, STALE, OK or UNKNOWN. A workflow that has never run, or only runs when called by hand, is reported UNKNOWN, never OK.
  5. A daily watch. It compares today's checks with yesterday's and alerts only when something that was fine turns bad, so the alert channel stays quiet unless it matters.
What is still not proven. The second relay (production notifications) shows its old error until the next real event reaches it, so we cannot yet claim it works end to end. We have tested the webhook, not a full production event. We will not call it fixed until a real run succeeds.

Lessons you can reuse

  • After any automation platform upgrade, run every workflow once by hand and read its result.
  • Alert on the absence of success, not only on errors: a job that did not run leaves no error.
  • Treat UNKNOWN as its own state. It is the honest answer for anything you have not seen succeed.
  • Test the refusal path of a webhook as well as the success path.