What went wrong
We run n8n for scheduled jobs and for relaying alerts to a chat channel. On the n8n 2.x series, expressions that read environment variables (the usual place for tokens and webhook secrets) were blocked. Workflows that depended on them did not crash loudly. They ran, failed at the first node that needed a secret, and ended with an error nobody was watching.
- The alert relay failed on 28 of its 30 recorded runs. Every run since 23 October that tried to send a message had failed.
- Several scheduled company workflows had been failing since about 10 October for the same reason.
- No alert fired about the broken alerting. That is the point of the title.
How we found it
Not through a dashboard, but by reading the execution history directly and asking one question per workflow: what did its most recent run do? A workflow that last succeeded days ago, or whose last run errored, is not healthy, however green the process list looks.
What we changed
- Secrets moved out of expressions into n8n's own credential store. The Telegram token and the webhook secret are now credentials; no secret is read from the environment inside a workflow.
- The webhook requires a header secret. We tested both sides: a call without the secret is refused (HTTP 403), a call with it is accepted (HTTP 200).
- Retry on fail. The send node now retries three times, three seconds apart, because one of the recorded errors was a plain connection timeout.
- A health check that lets the latest execution decide. It reads each workflow's last run and reports FAILING, STALE, OK or UNKNOWN. A workflow that has never run, or only runs when called by hand, is reported UNKNOWN, never OK.
- A daily watch. It compares today's checks with yesterday's and alerts only when something that was fine turns bad, so the alert channel stays quiet unless it matters.
Lessons you can reuse
- After any automation platform upgrade, run every workflow once by hand and read its result.
- Alert on the absence of success, not only on errors: a job that did not run leaves no error.
- Treat UNKNOWN as its own state. It is the honest answer for anything you have not seen succeed.
- Test the refusal path of a webhook as well as the success path.