Fork me on GitHub

Alerting

Types of alerts

Cabot supports arbitrary alerting channels via its plugin architecture. Out of the box, it supports:

  • Email
  • Hipchat
  • Slack
  • SMS
  • Telephone

Other alerting channels are supported through plugins. See a list

Rather than focusing on channels, we classify alert types as follows:

  • Notifications - things that let you know what’s going on but don’t interrupt your work. This includes email and non-alerting Hipchat messages (i.e. ones that do not @reference anyone).
  • Alerts - more intrusive but not guaranteed to wake you up at 3 am. This includes SMS and alerting Hipchat messages.
  • Telephone alerts - the most intrusive. Use sparingly unless you want to be woken up at 3am because stage is reporting a spike in CPU utilization.

Telephone alerts only on Critical status

Because telephone alerts are so intrusive, Cabot will only send telephone alerts when all the following conditions are met:

  • Alerts are enabled for the service
  • The service status is Critical (at least one failing check has Critical importance)
  • Telephone alerts are enabled for the service
  • There is at least one user configured as either currently on duty (via calendar sync) or as the fallback duty officer (set on the user’s user profile)

Telephone alerts ignore the Users to notify list associated with a Service and will only call the duty officer or fallback duty officer.

Configuring alert channels

Global config

All alerting channels (except email, which can run locally) depend on external services so need to be configured appropriately - see Configuration for instructions on how to set up account keys.

Hipchat alerts

Hipchat alerts require the user’s Hipchat alias be configured in their user profile. See Users for instructions for adding Hipchat aliases.

Telephone and SMS alerts

Telephone and SMS alerts need the users full mobile phone number entered in their user profile. See Users for instructions for adding Hipchat aliases.

Remember:

  • Enter your full telephone number including international dialing prefix
  • Remove any leading zeroes that are included when dialling locally
  • Remove initial +

Alert content

Most alerts give information about which checks are failing:

Sample hipchat alerts

Telephone alerts give minimal information because it’s hard to parse detail about what is failing and why over the phone, especially from a robocall. We recommend that you always use telephone alerts in conjunction with another alerting channel (Hipchat or SMS).