Types of alerts
Cabot supports arbitrary alerting channels via its plugin architecture. Out of the box, it supports:
You can see others on PyPI, including Pushover, Telegram, Pagerduty, Opsgenie and Slack.
Rather than focusing on channels, we classify alert types as follows:
Notifications- things that let you know what’s going on but don’t interrupt your work. This includes email and non-alerting Hipchat messages (i.e. ones that do not @reference anyone).
Alerts- more intrusive but not guaranteed to wake you up at 3 am. This includes SMS and alerting Hipchat messages.
Telephone alerts- the most intrusive. Use sparingly unless you want to be woken up at 3am because stage is reporting a spike in CPU utilization.
Telephone alerts only on
Because telephone alerts are so intrusive, Cabot will only send telephone alerts when all the following conditions are met:
- Alerts are enabled for the service
- The service status is
Critical(at least one failing check has
- Telephone alerts are enabled for the service
- There is at least one user configured as either currently on duty (via calendar sync) or as the fallback duty officer (set on the user’s user profile)
Telephone alerts ignore the
Users to notify list associated with a Service and will only call the duty officer or fallback duty officer.
Configuring alert channels
All alerting channels (except email, which can run locally) depend on external services so need to be configured appropriately - see Configuration for instructions on how to set up account keys.
Hipchat alerts require the user’s Hipchat alias be configured in their user profile. See Users for instructions for adding Hipchat aliases.
Telephone and SMS alerts
Telephone and SMS alerts need the users full mobile phone number entered in their user profile. See Users for instructions for adding Hipchat aliases.
- Enter your full telephone number including international dialing prefix
- Remove any leading zeroes that are included when dialling locally
- Remove initial
Most alerts give information about which checks are failing:
Telephone alerts give minimal information because it’s hard to parse detail about what is failing and why over the phone, especially from a robocall. We recommend that you always use telephone alerts in conjunction with another alerting channel (Hipchat or SMS).