Docker ManagerAlerts
Problems Docker Manager spots by itself, finished backups, restores, prunes and updates, and how you see and dismiss them.
Docker Manager raises an alert when something needs you, such as a failing disk, a full disk or a server that went offline. It also records a notification when a backup, restore, prune or update finishes. You see both in the app, and your notification channels can send them to you.
What raises an alert
| Alert | When | Ends when |
|---|---|---|
| Disk Health | A disk is failing (Critical), or shows damaged sectors, wear, low spare or heat, can't be read or is missing (Warning). Also when the server's disks can't be scanned or read, or their health is out of date (Warning). Never for a sleeping disk or one without health data, such as a virtual disk. | The disk is healthy again, or the server's disks are checked again. |
| RAID | An array has failed or is inactive (Critical), or is degraded or rebuilding (Warning). A degraded ZFS pool is a Warning; a faulted, unavailable, suspended or removed one is Critical. Also when the RAID state can't be read or is out of date (Warning). | The array or pool is healthy again, or its state can be read again. |
| Temperature | The hottest sensor of a server stays at or above the warning level (Warning) or the critical level (Critical) for 5 minutes. | It is more than 3 °C below the warning level. |
| Disk Space | A server's charted disk (Docker's data disk, the stacks disk, or a disk you mounted under /disks) stays at or above the warning or critical level for 5 minutes. | It is more than 3 points below the warning level. |
| Memory | A server's memory use stays at or above the warning or critical level for 5 minutes. On a server with ZFS, the part of ZFS's cache that ZFS can give back counts as free, and the alert says ZFS cache counted as freeable. | It is more than 3 points below the warning level. |
| Environment Offline | A server has been offline for 5 minutes (Critical). | The server is back online. |
| Failed Job | A scheduled job, or one started with an API token, failed (Critical), partly failed or was interrupted (Warning). Jobs you start yourself show only in your Notices. A backup's alert names the environment and the repository, and lists the items that failed. Also, for every restore: the deploy of a stack restored with Deploy … Afterwards couldn't be started (Warning; a deploy that starts and then fails is reported like any other deploy job), or a failed restore could not put the original data back, so its containers stay stopped (Critical). | The next run of the same job succeeds. A restored stack's alert ends when the stack is deployed; a restore that could not put the original data back stays until you dismiss it. |
| Updates Available | An update check finds newer images (Info). | No update is left. |
| Backups | Backups are on but no repository is the Primary one, for example after removing the Primary without a Secondary (Critical). Docker Manager was restored from a backup, so backups wait for Resume Backups (Critical). A scheduled backup was missed or skipped because the previous one still ran (Warning), or was refused or couldn't start (Critical). An environment, or Docker Manager State, has no complete backup in a repository for two scheduled runs (Critical, sent again as it gets older; data left out because it stores that repository doesn't make a backup incomplete). A different repository is at a repository's location (Critical). The retention after a backup couldn't start (Warning). Backups were removed from the list because two checks in a row didn't find them in their repository, for example after a bucket lifecycle rule deleted them (Warning, sent again when more are removed); when another repository holds the same backups (a copy, or the same storage at another address), the alert says its retention may have removed them; when a retention there stopped before it reported what it removed, because its Docker Agent went offline or the environment was archived, the alert says that retention may have removed them. A check didn't find most backups of an environment in their repository, so none were removed from the list; they show Not Found at Last Check (Warning). | In the same order: a repository is the Primary one again or backups are turned off; you press Resume Backups; the next scheduled backup starts or backups are turned off; a complete backup is made, a schedule change means it's no longer overdue (the message says no complete backup was made), or backups stop covering it; the repository is removed, or the other repository isn't seen for 7 days; the retention in that repository succeeds, the repository is removed, or 7 days pass without it being raised again; 7 days after the last removal, or when the repository is removed; a later check no longer misses most of them (backups still not found after two weeks are removed from the list), or the repository is removed. |
| New User | Someone used an invite link and has an account now (Info). Only the owner sees it; it links to the new account. | After 7 days, or when the account is deleted. |
These alerts end without a message: a disk or array no longer reported for a day, a failed job that doesn't run again for 7 days (never a restore that could not put the original data back), a new user's alert, the alerts of an archived environment (except those about its backups' repository: backups no longer found there, most backups not found there, a different repository there, or a failed verification or retention there, which end as they otherwise would), and a backup alert about something the backups no longer cover or a different repository not seen for 7 days. The alert about a retention that couldn't start also ends without a message when its repository is removed or 7 days pass without it being raised again (old backups stay until retention runs), and the alert about backups no longer found 7 days after the last such removal or when its repository is removed. While a server's disks or RAID state can't be read, their alerts stay. When a disk shows up under another name, for example after a restart, its alert moves with it.
Change the thresholds
- Open Settings → Notifications.
- Under Alert Thresholds, set the Warning and Critical levels
of Temperature (°C) (80 and 90 by default), Disk Space (% Used)
(85 and 95) and Memory (% Used) (90 and 95).
0turns a level off. - Press Save Thresholds.
To use other levels on one server, press Add Override under Environment Overrides, choose the Environment and fill in only the levels you want to change; empty ones use the defaults. Only the owner can change thresholds; on the Alerts tab, the owner's temperature, disk space and memory alerts link here with Change Thresholds.
See alerts and notifications
- Notifications in the side menu has two tabs. Notifications lists every finished backup, restore, prune and update by day, with how it went (Success, Warning or Failure) and its numbers, such as what a prune reclaimed or a backup saved. Alerts lists the alerts: choose Active, Dismissed or Resolved, a kind or an environment.
- The Notices bell at the top shows active alerts, and what resolved or finished in the last 7 days, as the In App channel is set (Choose what the bell shows), plus the jobs you start yourself. It counts what you haven't dismissed. View All Notifications and View All Alerts open the two tabs.
- Click an alert or notification to open what it is about: the server's System tab, the server, the job, Updates, Backups or the new user.
- On the dashboard, an environment's card shows how many of its disks and RAID arrays need attention as a small mark beside its status; click it to open Alerts for them. An environment's page shows a notice with View Disks or View RAID Arrays.
You see an alert only if you may see what it is about: the server, the job, the stack or container, or the backup settings. You see a notification when you may see its job.
Dismiss an alert
Press Dismiss next to the alert on the Alerts tab, or × next to it in Notices. The alert then leaves Notices, the dashboard and the environment's notice for everyone, and stays under Dismissed until the problem is gone.
Dismiss All on the Alerts tab dismisses the listed alerts you may dismiss, after you confirm. In Notices it clears everything, and asks first when that dismisses alerts for everyone.
If the problem gets worse (a higher severity, or a new problem such as another failed disk in the array), the alert opens again and is sent again.
Get them as messages
Add a notification channel and choose under What to Send which alerts and outcomes it gets. That page also says which row a failed job uses and how messages look.
Good to know:
- Dismissing for everyone needs the “Dismiss Alerts” permission (the Operator and Admin presets have it). Without it, dismissing in Notices hides the alert only in your browser.
- Change how long a server may be offline with
DOCKER_MANAGER_ALERT_OFFLINE_GRACE(see Configuration). After a restart of Docker Manager, servers get that full time to reconnect. - Resolved alerts and notifications stay for 90 days.