Updata would like to apologise to all users for the on going reliability issues we are experiencing with the core network Firewalls. We have been working closely with Cisco, the supplier of the devices, for a number of months to try to ascertain the root cause of the failures we have been seeing. Due to the very intermittent nature of these failures and our desire to restore full service to you it has been a lengthy process.
Last month we believed, based on the Cisco diagnosis, that the root cause had been identified as an inherent hardware problem in a batch of Firewall modules that included the 4 devices deployed in Hertfordshire. We then undertook a series of engineering changes to replace all 4 units which concluded on the evening of Monday 16th and we were confident that the issue was resolved.
Unfortunately we have had another occurrence of a Firewall issue this morning and had to failover the service to the secondary datacentre but when we do so we experience throughput restrictions which results in reduced performance for you as users. We decided to leave the system in a failover state to allow Cisco engineers to fully diagnose the failing device while it was in a failed state to allow them to properly identify the root cause. Subsequent to the diagnostic period we attempted a reboot of the device which normally restores the service but in this case the device went into a hardware failure state and is in the process of being replaced. There is currently a Priority 1 TAC case open with Cisco and we have an assurance of 24/7 attention to this.
We are also working to ensure that when we fail over to a single datacentre the service operates at full capacity with no affect to you. This requires additional infrastructure which we are currently scoping and hope to have in place early in the new year.
Please be assured that we are doing everything possible to have a permanent resolution for this issue and we would once again like to apologise for the unacceptable interruption to service that you have been experiencing.