Alerts Outage
Incident Report for IOpipe
Postmortem

Alerts went down for a few hours. I’m sorry that happened.

What happened?

On Wednesday April 3 at 11:31am EST we attempted to update a piece of our product that monitors for events and delivers alerts notifications. However, that project hadn’t been updated in a while and the release had some faulty environment variables, causing a loss of service for all alerts.

Around 2:15pm EST we resolved the issues with alerts and are taking steps to ensure such issues with deploying parts of our service don’t cause such a disruption.

Posted Apr 03, 2019 - 22:40 UTC

Resolved
We've identified the issue, and rolled out a fix. Alerts are now functioning normally.
Posted Apr 03, 2019 - 18:08 UTC
Investigating
Alerts have been down since 8:40A PDT. We are currently investigating this issue.
Posted Apr 03, 2019 - 17:11 UTC
This incident affected: Alerts.