Post-Mortem on Tuesday Evenings Network Incident

By August 17, 2016CloudSingularity, Datacentre

We would like to share some details on the incident that occurred yesterday evening at our Vancouver Datacentre YVR1.

At approximately 21:00 PST on Monday, August 15th there was a brief 4 minute spike in traffic that caused some packet loss.

On Tuesday, August 16th between 16:30 and 18:00 PST we detected intermittent packet loss until around 18:00 where there was a complete loss in traffic.

Traffic GraphWe have concluded both incidents were related to a Distributed Denial of Service (DDoS) attack. Tuesday’s attack was a larger scale DDoS attack at over 3Gbps/300Kpps. The attack saturated our port which resulted in dropped traffic. We worked with our upstream providers to mitigate the attack by null-routing the IP being attacked. At around 18:24 after the null-route, all traffic resumed to normal.

We understand how critical it is to our customers that the network is up 100% of the time. This was unfortunately something out of our control, as is the nature of DDoS attacks. We strive to minimize disruptions and outages regardless of the source of the issue; and promise to remain transparent on these kind of attacks and incidents of any nature. Finally, as we continuously make improvements to the network we will eventually be able to absorb these type of attacks without incident.

Leave a Reply