RAID Data Recovery Case Study – SCO Unix – 1

Data Recovery: A partnership for success

With close to 30 years of software design, programming, and implementation I have seen thousands of clients where their data has been, lost, drowned, burned, erased, deleted, and maliciously destroyed. In all of these cases where the data was successfully recovered there has existed a common thread. That being the client had an integral part of the recovery. Let me repeat that. The client and the technician had a quasi-symbiotic relationship, each depending upon the other for the greater good.

This being said, rather than throw a bunch of theories, hypothesis, and philosophies out for you to digest, let’s look at a real case study in data recovery. I received a call from a doctor who had an old RAID 5 with four drives. The operating system was SCO UnixWare 7, and the file system was VxFS. The client related that the RAID had been sent out to another company but the file system it was too corrupt to be recovered. The company indicated they had worked on the RAID for over a week, but could not get the file system to align. They sent the array back to my client with their apologies, but not so apologetic that they didn’t keep the diagnostic fee which was substantial.

Since DTI does not charge a diagnostic fee our initial interrogation of a prospective client is extensive. We will not take any damaged media in that we feel would take too many man hours to recover, or parts would be so expensive that it would not be fiscally sound to attempt recovery. Pre recovery interrogation is actually a very important part of the data recovery process. One must understand that you are speaking to a client that is usually in an agitated state, bordering on hysteria.

They may be losing thousands of dollars every hour, and they have no real sense of how to get their business back on track. You are basically a life preserver in a sea of tribulation. As an interrogator you always keep in mind what is best for the client first and what is best for your company second. Ultimately, if you keep this philosophy the client will be happier, and will be much more cooperative when assisting you in recovering their data.

In the case of the doctor I already knew that someone had tried to recover the RAID and gave up. I also knew who the company was that had attempted the recovery and they were usually very thorough. I asked the doctor a few more questions as to how the RAID went offline, what had he done to try and recover it, what did he believe the original recovery company did, what state was the media in, etcetera, etcetera. DTI specializes in recovering data when others have failed. If you have sent your data somewhere else and your files were not recovered DTI can help.

After all the physical questions about the media I turned my questioning to the data to be recovered and what data was most important to the doctor. The doctor explained that he a third party piece of software that kept all of his patient data. This included billing, patient histories, procedures, and a great deal more. It was important, above all else that this database, and all related data be recovered and brought back online within the application. After a few more questions about his business, and this database I agreed to take the RAID in and attempt a recovery.

What made me decide to take this RAID was that I am very familiar with databases. I have designed several for very large corporations and even programmed some low level B-Tree handlers. I also had an idea what the data would look like, and how it would be laid out on the drive. The doctor was also very knowledgeable about this application and could help me with the recovery. As you can see, the recovery was not just about RAID, hard drives, operating systems, or file systems. It was about this doctor’s practice and how could I understand his patient base in order to locate his data and ultimately bring him back online.

The RAID arrived and the in-house software I wrote and used to view the array found exactly what the other company found. A corrupted file tree and inode structure in the beginning of the file system. There would have to be extensive eyes on disk data using a hex editor and calculator to try and piece together the superblock and inode structure. Even if that was accomplished there was no guarantee that the file name or the inode would be intact to recover the file system. My recovery strategy turned from trying to find the file names, to locating the raw data. In order to do this I had to understand the type of data, and its on-disk structure.

I searched the Internet looking for the file extender as a search parameter. That was of no use, the data was of a proprietary nature. I went to the website of the company looking for clues as to how the data was stored, there were some but not enough. I then took out my trusty hex editor WinHex and started cruising the destripe. After several hours I was rewarded with a block of sectors that looked like a database. I called the doctor and he and I discussed what the data looked like. He said that he knew which file that was, and it sure sounded like his data. At this point although I found the data block, there was still no way for me to break up the data into files.

There were over 30 flat files in this database, some were related to others and each had at least one index tied to it. I asked the doctor if he had any sample data he could send me to look at. He immediately sent me several files and said that these are the file types that must be recovered. So with a WinHex I looked at each file to determine if there were any unique markings that would distinguish the beginning of one file, and the end of another. After several more hours I found the key to these files. I called the doctor again, we discussed my findings. He then asked me how I would break the files up. I told him I would write a program specifically for his data that would filter out the file blocks and then save those blocks into several files. He asked for a time frame I gave him one and began coding.

The software scanned the entire array looking for beginning and ending signatures, once found it created a file and placed it in a folder. The software found over 300 file segments and the scan took 8 hours. Once again I contacted the doctor and told him this. This was the single most critical portion of this client’s recovery. I told the doctor that he was going to have to help me with looking at the data. He knew what was in the files, he knew what data belonged with what, and he was the only one that could piece the data together. He asked how he could do that. I said I would teach him how to use WinHex, show him how he could piece the files together, and then take those files and import them into his application.

At first he was a bit apprehensive, but I tried to make him feel at ease by saying I would be there to help him every step of the way. I also indicated that nobody knew his data like he did. He was a recovery expert when it came to his data and I was only a tool that he was using to get his data recovered. He agreed to help in his recovery and we became partners. I gave him my personal cell number so that he could reach me whenever he needed, night or day.

As a side note, he did call me once while I was on the golf course. I missed a putt, but hey, he was my client. Over the next few days we sent emails and spoke on the phone extensively. We became friends of a sort and worked together to get his database online. It was long, tedious, and at times fun to see this project grow. All in all I wrote three custom pieces of software, sent the doctor DVDs of his data, and learned a great deal about his application. At the end of it all we were successful. My client got all of his data back, his practice was brought back online, and in a very real sense I made a friend.

If this doctor, this friend, ever runs across a similar situation with either his business, or someone else, who do you think will come to mind first? Hopefully, it will be me, the guy who spent countless hours in the trenches with him, and together we brought his practice back online.

Data recovery is not only about dollars and cents. It is not just cold labs and mindless pieces of software. It’s about taking to heart somebody’s troubles. It’s about helping first and business second. With this philosophy, every recovery is a partnership, and a potential friendship waiting to bloom. It has proven successful for me in over thirty years working with clients.

Call Toll Free: 1-866-438-6932 or fill out an online quote form if you need data recovery.

Comments

  1. Davedata99 says:

    Wow. alot more goes into data recovery than I thought. Coded custom sotware after the recovery to access the primary data. Impressive, the force is strong with you.

  2. This is the actual email from the client in the above story, word for word:

    Hello Dick,
    Well, it’s been about 10 days since we last spoke. Last weekend, I finally had enough of the financial files recovered so we could start posting our billings for the previous 4 weeks. This past Friday, we did our first insurance billing. I’m finally sleeping at night again.
    I just wanted to thank you for all you did, and all you DIDN’T do. You didn’t bullshit me, you didn’t delay me, and you didn’t keep me from helping the process along. I’m not sure how many knowledgeable, professional, no-nonsense people there are in the world, but I do know there are precious few. You are one of them, and I appreciate your help. Winhex is a great program, and by the way, I’ve discovered it’s relatively easy and fast to use Access to eliminate duplicate records from a file. You can also use Access to compare two files and eliminate duplicates in both.
    I’ll send your drive back as you requested. I’d like to hold onto it a few weeks just in case I find I’m missing a file I forgot to retrieve. Let me know if that’s not OK.
    Again, thank you.

  3. What is Unix? says:

    What is Unix says
    I see that you mention that the data format was proprietary, so it was difficult to decypher. Would you recommend open-source software for that reason?

  4. In answer to your question, “Would I recommend open source”, then answer is a resounding, and emphatic YES!!!.

    When a qualified technician can use the design and format documentaion to build tools to recover what with great certainity will soon be lost, the tool becomes much more comprehensive, and will in all cases have a higher success rate than a tool developed with little or no knowledge of the design.

    As an example, Exchange recovery is at best hit and miss, and at its worst using stone knives and bear skins to solve a modern problem. The file format of the ‘priv’ file is not documented anywhere, and that “SECRET” has cost many an admin their job, and cost many companies millions of dollars. If the format were made open source, then wonderful tools could be developed that would recover a much higher percentage of Exchange crashes.

    This is of course, only my opinion. However, an opinion steeped in 25 years of computer programming and data recovery expertise.

  5. Krishna prasad.R says:

    Hai, Our scounix server went off due to VRM failure.It went off many times without proper shut down and also in the time of booting.One of the engineer here pulled out the raid configured hard disk drive and cleaned it using the vaccum cleaner when the server was off.Please guide me wether it will cause any problem to operating system when the VRM is replaced and server is ready to boot.Please reply me as soon as possible thru my mail and i am waiting for it.

Trackbacks

  1. [...] the best track records for customer service. Proof in the pudding is Dick Correa’s post about SCO Unix RAID Data Recovery ”Data Recovery A Partnership For Success” where he lays out the process of interaction [...]

  2. [...] in the ointment’ when it comes to using standard UFS data recovery tools. Read my article on SCO Unix RAID Data Recovery for more insight on the [...]

  3. [...] explains about how the customer is the most important part of the recovery process in his article Data Recovery A Partnership For Success where he explains how nobody knows more about their data than the [...]