Inbound and outbound calls have been working for some time but we are still monitoring the service.
BLF has not been brought back yet while we investigate.

The datacentre link was restored shortly before 05:00. Registrations have since returned to normal, and services at the affected site have been brought back online and monitored to confirm stability.

Wednesday 23rd September 2026 17:51
Upstream Network Outage: A third-party provider (Cogent) suffered an outage, breaking connectivity between data centers (THN and ENF).

Wednesday 23rd September 2026 18:20
All ENF servers stopped and calls were diverted to THN, which enabled calls to be made, but BLF's were disabled

Database Split Dependency: Call handling and Busy Lamp Fields (BLF) rely on sqlsrv2 (ENF), while remaining systems rely on sqlsrv (THN). Because database connections were intentionally split across sites to balance load, the network outage severed cross-site DB communication and disrupted calls and BLF updates.

SIP Server Bug: A bug in the SIP server caused registration drops during the extended outage, and failed to recover automatically once connectivity was restored.

Thursday 24th September 2026 04:00

Cogent resolved the primary network link issue, and the Data links were restored at ENF

Thursday 24th September 2026 06:15

BLF's re-enabled

SIP Service Recovery: Manually rebooted the SIP server and executed a service failover, restoring registration levels to normal

Immediate / Short-Term Actions

Consolidate Databases (Target: Nov 15): Following the Oct 6 web portal release, migrate call handling and BLF back to a single primary database on new hardware.
Enable Inter-DC Link: Complete the ILF–ENF and ENF-THN interconnect to provide resilient cross-site database failover.

Medium-Term Improvements

Database Proxy (Contingency): Implement and test a DB proxy to automatically fail over call handling and BLF traffic if the primary database becomes unavailable.
Asterisk Infrastructure: We will add more Asterisk servers to each DB to handle the traffic increase when running on a single DC in case of an outage.