17/07/13 - Loss of Service on Hosted PBX

Originally from ticket #1863.

Hi,

We are receiving multiple reports of loss of service in relation to our hosted PBX platform at present.

We believe this problem is related to an issue with hardware in our data centre and there are engineers working to resolve this issue at present.

We apologise for any inconvenience caused and will provide an update as soon as the issue is resolved.

UPDATE:

We have been informed that the problem is due to a major outage on a piece of hardware in the data centre that manages SIP registration. Engineers are circumventing this problem as we speak and normal service is expected to resume very shortly.

UPDATE 12:50:

We have an update in relation to this problem. Originally the feedback we received from the data centre indicated a problem with connectivity to one of our registration servers, however this has been proven not to be the case and the problem has been traced upstream to the BT IP Exchange in London.

We have now been informed there is a major service failure within the BT IP Exchange effecting numerous VoIP service providers nationally, not just ourselves. BT Core Network Service Engineers are now working on the problem and we hope to have an update very shortly.

UPDATE 13:30:

We are now seeing a return to service and users are reporting normal operation again. Our platform is also starting to receive data/calls as normal from the BT IP Exchange, so we are cautiously assuming that BT have resolved the problem. Although we haven't had official confirmation, this issue appears to be resolved. Once we have a concrete answer from the BT IPX, we will post it here. Our apologies for any inconvenience caused.

UPDATE 13:55:

Tim Blackburn in SIP Voice Operations at BT TSO, has confirmed with us that this issue was caused within BT's network and is currently being documented as incident IMT 30580/13. Many providers were effected. We will have a further update as soon as possible. In the mean time, we are pleased to announce normal service has resumed for all our customers.

RESOLUTION UPDATE 18/07/13:

The area within BT's network that was causing congestion was identified and rectified by our 3rd line technical support team at 13:14 yesterday. Calls that were caught up in that routing element were neither re-routing, nor clearing, hence causing congestion and the subsequent 503 SIP response to some calls using the SBCs at KST-CP01 and BND-CP04. BT have been carrying out subsequent checks in the interim period to ensure stability is maintained.