Post-Mortem on Tuesdays BGP Hijack

By September 29, 2020 October 7th, 2020 CloudSingularity, Datacentre

What happened?

Today at 10:46 AM PDT our monitoring system detected a hijack of two of our six IPv4 prefixes. This would have affected some traffic to/from networks who are not yet RPKI filtering invalids. (hint: for those who have no idea what we are talking about, some great resources are listed at the bottom of this page)

Shortly after the automated alert, we started our investigation and determined that Telstra Wholesale Australia (AS1221) had hijacked the two prefixes. Only customers who have IPs assigned from those two prefixes (as in above screenshot) would have been affected.

In the cropped BGP map below, you can see some internet traffic being re-routed toward the hijacked routes to AS1221, instead of us on the far right.

AS1221 Hijacked BGP Map

Normal BGP Map

BGPStream also picked up the hijacks and the interactive BGP map below.

BGPlay Visualization

View the visualization yourself: and

How it was resolved

Our team tried numerous methods to reach out to AS1221 (Telstra Wholesale Australia), and AS4637 (Telstra Global) which seems to be the parent network. We had difficulties reaching the NOC by phone, since it was still early morning in Australia/Hong Kong. We did manage to reach someone by phone in their “IP Faults” department, but unfortunately they were unaware how to deal with the issue or who to put us in touch with. We managed to find an email for the NOC contact and forwarded our email to them. By approximately 1:41 PM PDT,  Telstra NOC replied confirming they fixed the (what we assume) mis-configuration on their side.

Screenshot of attempts to communicate with Telstra

From our own internal investigation it seems like AS1221 issue originated in their Sydney POP using their own looking glass tool.

How could have this been prevented?

The current standards in place is to implement RPKI signing and filtering . We did this ourselves back in January 2016 by adding Route Origin Authorizations (ROA) to prefixes we announce. And most recently in April 2020, we started validating and filtering invalid routes. Unfortunately until all network operators around the world start doing the same, they are vulnerable to accepting and re-announcing the hijacked or invalid routes to others.

Without rewriting some already great articles, we recommend visiting some of the below resources and articles for more information on BGP hijacking and RPKI:

Despite this being caused by something outside our control, we would like to apologize for the disruption. We hope more network operators start adopting better routing security through standards such as RPKI, and ultimately make the internet a safer place, routing wise ;).

Update September 30 – New information

Via an iTnews Australia article and social media posts, it seems 102 networks around the world were affected including Quad9 DNS and Packet’s Sydney facility that hosts Google’s local region.

Per a tweet, Telstra reported the cause to be:

“due to a technical error overnight, a number of internet prefixes were incorrectly advertised as Telstra’s. This meant some internet traffic may have been routed to Telstra incorrectly…”

Update October 3 – Official statement from Telstra

Telstra has posted an official statement on their webite.

Update October 7

Added new BGPplay diagram and links

One Comment

Leave a Reply