At around 17:00 today we were made aware of an all service network issue across both our primary datacentres. This issue seems to have now been resolved (as of 17:17), however it appears to have affected a large portion of the internet as a whole with many services down.
According to Down Detector as well as reports from Twitter, the issue also affected Facebook, O2, BT, EE, Vodafone, Sky, Twitter, Google, Experian as well as other internet based services.
Please find a log below of the events;
17:02 – Our NOC in Neath began receiving alarms on our PRTG Network Monitoring software. SMS were sent to our Technical Director, Customer Manager, Support Team and Company Director
17:02 – Our NOC in Swansea began receiving the same alarms. Network failover was initiated, but unsuccessful. Connectivity between the sites remained available, but access to servers was not available outside our network, not connectivity to other service providers.
17:05 – We redirected our support number to our backup mobile phones.
17:07 – We continued to investigate the cause of the issue (at the time believing it was just an issue with our network), with our network partners Virgin, Exponential-E and BT. We believed it may have been an issue with BGP.
17:16 – While investigating, connectivity was partially restored.
17:17 – Connectivity was fully restored, with PRTG reporting all networks as normal.
We are still awaiting confirmation from Exponential-E, Virgin and BT as to what the cause of the issue was.
Unfortunately, as this issue affected multiple networks and the issue was outside of our network infrastructure, we cannot see a way this could have been avoided.
Our failover and backup plans were implemented and would have been successful had this have been an issue with our network.
Please see map attached of where the issues were primarily reported, according to https://downdetector.co.uk/
We apologise for any inconvenience, and should you have any questions, please do not hesitate to contact my team on 03334445558 or email firstname.lastname@example.org