The Road to Recovery - How Databarracks Assisted WTB with Massive Data Loss
WTB Group Limited is the UK's leading supplier of civil engineering and building materials to the infrastructure, environmental and industrial markets. With 75 years of experience and having grown to produce an annual turnover in excess of £320 million, WTB is definitely a force in their industry. WTB employs cutting edge technology in their warehouses and utilizes integrated stock management control systems to coordinate the logistics involved in offering a product range of over 18,000 lines. Depots where customers can choose products in a more traditional shopping atmosphere are available in over 50 locations around the UK and Ireland, but a very large contributor to WTB's reputation as the pinnacle in customer service can be attributed to the online enquiry management system they employ. The Enquiry List Service allows customers to select products they are interested in and provides communication via email with their local WTB which supplies price and availability information on the items in question.
Email communication is of the utmost importance to WTB as a means of maintaining the high standard of Customer Service and flexibility they offer. Much communication involving critical details is done between employees and their customers as well as with suppliers. In March of 2007, WTB decided to use Databarracks to provide fully managed online backup services in order to safeguard nearly 3 Terabytes of their valuable emails, databases, and files on 21 servers. Databarracks employs Asigra software to safeguard WTB's data. The functions of the software that are currently utilised for the account are the local storage, archiving, and backup lifecycle management capabilities.
"Blue Screens of Death"
On Wednesday, 23 January 2008, Disaster struck at WTB. A SAN with one Terabyte of data and eight virtual servers experienced massive problems. The virtual management logs revealed that at approximately 6:00am, a drive failed; then when it attempted to recover itself, the SAN suffered corruption across the volume group. Initially, IBM was contacted to come in and attempt to repair the damage, however, it was determined that there was no solution at hand that would not result in severe and permanent data loss. Tim Brice, IT Systems Manager for WTB recalls, "The situation continued to worsen throughout the morning, with blue screens of death appearing on several virtual servers hosted on the SAN and key databases losing data and becoming corrupted. In the end we made the decision to rebuild the SAN volume group, rebuild the virtual machines and restore data and configurations from backup."
At approximately 12 noon, restoration efforts and the rebuilding of machines began. That afternoon, Databarracks was contacted for help. Within four hours, two of Databarracks most senior engineers were on site with data necessary for recovery which they had collected from the Data Centre en-route. The engineers stayed through the night, doing everything in their power to assist with the recovery.
"I was amazed at the service we received, as the engineers were wonderful! They acted professionally and were in good spirits all night, which helped to raise our morale. You don't expect people to come out immediately and stay working until morning, but they did! It felt like Databarracks were part of our IT team. They were with us every step of the way as a safety net and really helped us get through this difficult time," reveals Laura Smith, the Helpdesk manager who was involved in the recovery efforts. Databarracks' devoted staff achieved recovery of sufficient systems to enable WTB to be up and running the following day.
Deep Impact
Overall, the disaster hindered the normal flow of business quite significantly for more than half of WTB's 1,000 internal users. One SAN failure resulted in impact across servers designated for their Exchange and SQL in addition to other data being lost. As WTB performs vital business transactions via electronic messages, sending and receiving an average of 25,000 emails every day in addition to the millions of archived historical communications they maintain, the loss of Exchange functionality was no small problem. The corruption impacted the largest of three Exchange Servers which hosts 400 users, containing 50% of the estate of around 600GB. Included in the loss were mailboxes used for electronic ordering as well as those belonging to senior level management. In addition to Exchange, half of WTB's user profiles were wiped out, resulting in hundreds of employees being unable to log on to the company system.
The SQL server affected was home to five major databases, each of which provided crucial operational value to WTB. Overnight, key financial reports are generated by business intelligence software, and automatically emailed to senior management, business partners, and financial institutions. The database required to assist with the production of the reports was compromised, resulting in unavailability of essential business analysis and performance information. The Blackberry Database required for facilitating email and intranet access via mobile devices was lost, resulting in no access to information for remote users. Databases necessary to run systems monitoring and fleet management applications became corrupt and unusable, impacting most notably the fleet management system which is responsible for orchestrating the logistics and maintenance for WTB's 250 trucks. Remote VPN access to company computers was lost as was access to the databases required for the IT Service Desk software, which is used by the helpdesk to support users and also used by systems administrators to manage the IT systems infrastructure.
"Because half of office based users and all remote users were without Email, we obviously missed out on important correspondence with business partners including potential customers. In addition, business operations were severely impacted by having fifty percent of users unable to access the systems and several key applications unavailable. Luckily, we were able to run with much higher availability the following day. The resulting delays and costs would have been much worse if Databarracks had not acted so quickly" summarised Tim.
In the days following the disaster at WTB, efforts had to be made to recover additional historical data and further reinstate working order. Databarracks had the same two engineers, working in shifts 24 hours a day, retrieving data from the data centre and restoring it to WTB's systems through the weekend.
A New Peace of Mind
Since January, WTB's confidence in its choice of online backup service provider has been reaffirmed once more. In early May 2008, a couple of workers cabling computer sockets in the WTB computer room shorted a mains circuit resulting in the main electrical feed into the computer room being tripped and the Uninterruptible Power Supply automatically shutting down to protect itself. All of the server, storage, and network systems crashed and some data had to be restored through remote connection to ensure systems were operational once the electrical fault had been fixed.
Tim says he chose Databarracks based on the fact that it offered "the right product at the right price," and continues "I used to have nightmares when backing up to tape due to reliability, now I can just forget about backups in the knowledge that backup data is safe and sound in a managed data centre environment. The company has benefitted from the solution, and I have too! I have time to focus on other areas of the IT systems and projects since the daily chore of backup managing has been removed. I feel comfortable since the disaster. I know I can rely on Databarracks solution 100%. It has been proven and it works."