The following saga is the story of the recovery of a single Outlook PST (Personal Information Store) file. It was accomplished using a simple hex editor some diagnostics, and the courage of David when he slew the Philistine Goliath. Well, maybe not so dramatic, but it was pretty cool.
As fate would have it I received a three drive RAID 5 that had some serious problems. Two of the drives had gone down and the array was now degraded to the point of not mounting. I have been in this business for quite some time and I can tell you that I can count on one hand how many times two drives in a RAID 5 have gone down at EXACTLY the same time. It is just a very rare occurrence. However, I wish I had a nickel for every time two drives go down in a RAID 5 and one of the drives has been offline for an extended period of time. In other words, one drive went down and either was ignored, or the RAID was configured to NOT send a warning, or the RAID card was defective and did not send the proper warning,. In any event, I think you get the picture. Whatever the reason, one drive is down longer than the other drive and creates a ‘stale data ‘state . I believe this is what happened to this array.
In order to bring the array back online, the client forced the two drives back online, and then tried to mount the array. The array goes from a degraded state to an online state. The clouds part, the birds sing and all is right with the world. Almost. Upon booting the array into the file system ‘chkdsk’ was automatically run. Microsoft’s way of aligning the file system so that it to can be mounted. However, ‘chkdsk’ shows no mercy. Bad MFT record here, a bad INDX record there, a few misplaced INODES here and voila! No more files. You have a nice clean file system, with no data. Not good. In this case however, it was not that bad, the data was still basically intact, it had just a few anomalies. I love that word anomaly. I use it every time I am clueless as to what is going on. In addition to chkdsk being run one of the drives was still bad in the array, so the client swapped it out and began a rebuild. After all this, the file system still mounted and ran for about another week. Some users were complaining that files were missing or corrupt. Whole folders were missing, so on and so forth, but basically, everything was cool.
Then something wonderful happened. Another drive went out of the array, and now nothing would mount. No array, no file system, no data, just quiet, kinda like a graveyard. That’s when I got a call. After hearing this story I almost bypassed the recovery because I KNEW it was going to be a headache, however, my technicians curiosity got the better of me and the client sent the array. After some massaging, and tickling I got a great deal of the data back, however there was still this one pesky PST file that their entire business hinged on, and if I could get that back , well, the ebb and flow of the Atlantic tides, the drift of the continents, the very position of the sun along its ecliptic would align and all would be right with the world.
Next time I will describe the painstaking way in which I created the de-stripes used to recover the array, as well as a little help from some diagnostics in DTI Data’s Recover It All product. Until next time.
[...] time I gave a brief description of a RAID that we received here at the shop. I also shared some of the things the client had done to try and get their RAID back online. I [...]