Downtime Processes
Last updated: September 21, 2021
This page outlines steps for client-facing teams to take during an outage or widespread incident, whether it’s with FareHarbor, FareHarbor Sites, or a third party like Stripe or Zendesk.
Steps to take
Detailed steps and information on how to identify issues can be found here.
- Check the #ambulance channel in Slack. This is where all updates on downtime are communicated and reported.
- Make sure that @ambulancecrew has been tagged in #ambulance.
- Once the issue has been identified, respond to or reach out to clients as needed. Review approved language below for speaking with clients.
- Wait for a member of @ambulancecrew to update the channel with an official update once the issue is resolved. If things look resolved on your end, make sure to confirm with @ambulancecrew before letting clients know.
- Call back clients to let them know the issue has been resolved and thank them for their patience.
Approved language
Let clients know “we are aware of the issue and are working to restore FareHarbor to full functionality.”
The following message can be used in email or phone communications with clients:
The FareHarbor Product Team is working diligently to bring performance back to normal. This was an unplanned interruption, and we are working as fast as possible to ensure everything returns to normal as soon as possible. Please refer to status.fareharbor.com for the most up to date information. If you’d like, I can follow up as soon as things are back up and running.
Tone and message are very important. During business hours, look to your team leads or the #ambulance channel for messaging. We want to be succinct and confident in our response. We also want to be empathetic for the inconvenience.
You can elaborate on the issue if information is available, for example if AWS servers (hosting for FareHarbor) are down. See status.fareharbor.com and #ambulance for more info.