Desaster Recovery Service – A Customer Reference Story

The starting point is that South Africa is experiencing what is termed „load shedding“, causing havoc throughout the country especially in IT.
http://loadshedding.eskom.co.za/loadshedding/description

Our Company is Duys Engineering. We manufacture vehicle components (rear steps, side steps, nudge bars, roll over protections systems). We also work within the metal industry framework, (welding, potshells.)

The Group maintains extensive facilities throughout South Africa and has found considerable success by focusing its engineering capacity on key relationships with industries – most notably the aluminium, automotive, sugar, mining and logistics sectors. One of Duys’s core values is their dedication to lasting partnerships with clients. www.duys.co.za

The problem occured when our HP Proliant DL380 G5 server with a Raid Controller which uses battery caching exploded when experienced load shedding which resulted in our UPS batteries failing and our backup generator running out of fuel. The server ran esx 4.1 update 3, 8 X 146 gig SAS drives striped in Raid 5, 7 Virtual Machines (Citrix, Mhlongonline, SQL, Server 2008R2, Wsus, File Server, Time and Attendance).

Upon booting the server we received a vsd mount error. Half of our datastore was accessible , the other half was not because we had used extents to grow the datastore formatted with VMFS 3. We looked at the VMware support forums and all answers pointed to the fact that our data was lost and there was no way of restoring from such a problem. We kept coming across a name Mr. Uli Hankeln from MightyCare, on the forums who had recovered critical systems. I contacted him and asked me to open a ticket with MightyCare and there was hope again.

The first thing that MightyCare did for our company was to ensure that we were able to provide services to our network. Throught a series of utilities the Raid was determined to be damaged. 3 out of 7 virtual machines were readable and able to copy, the rest were not able to copy. He ran a utility called ddrescue to dump raw data to a vmfs store that we created. He thereafter recreated the vmdk files from the flat files and also recreated the vmx files. That was the easy part. One particular machine was corrupt beyond repair, which needed a carving utility called scalpel, which literally involved recalculating the hex values of the files needed to restore. At this point our concern was about granular restoration of the database, which we were able to restore about 90 %. We used that information together with the backups we had to piece the system together. There I was on a remote town in South Africa, being rescued from a huge mess caused by our National electric infrastructure. I had spend about three nights in the office, sleeping over looking for a solution and it was in Germany.

I was satisfied with the support from MightyCare, I had been turned away by some of the leading data recovery centers in South Africa, because of the complexity of the situation and the fact that we didn’t have thousands of Rands accept some of the quotations. MightyCare was able to accomodate us with our non existent budget. I was given follow up and updated on every process and also given honest information about the possible outcome. Once satisfied with the outcome I wiped the old server, replaced faulty hardware and restored the data successfully.

„Mightycare restores our Vmware infrastructure after successive power outages“

Nhlanhla Mazibuko
IT Manager | Duys Engineering Group (Pty) Ltd