Knowledge Base/What's New in Rypple/Announcements

Outage on Feb 2, 2012

Vinoj Rajendran
posted this on Feb 02 09:08

Earlier this morning Rypple experienced an unscheduled outage. We're very sorry for the inconvenience and are taking every possible step to prevent this issue from reoccurring in the future. We take any disruption to service very seriously and wanted to share what happened and some of the steps we are taking. 

Issue overview:
  • Our firewall required an update which we determined with our hosting provider would be performed off hours.
  • At 2AM EST the firewall update was started by our hosting provider. The update failed and the change was rolled back. 
  • As a result of the rollback, a particular load balancer began causing packet loss and resulted in a degradation to the Rypple service from 2AM EST to 6AM EST. The Rypple staff was alerted and resolved the issue by rerouting traffic.
  • At 9AM EST while investigating the root cause of the 2AM degradation our hosting provider caused a network outage between our application and database tier. This made the Rypple service unavailable until approximately 10 AM EST. The Rypple staff was alerted and resolved the issue by rerouting traffic.
Action Items:
  • We have fully removed the problematic load balancer appliance from our environment.
  • We are adding a fully redundant firewall pair.
  • We are working with our infrastructure provider to ensure a more rigorous change management process.

Thanks for your understanding. 

Rypple Support