In the business of data recovery viewing, seeing, looking, putting an eye on, or however you want to describe it is the single most valuable tool for a technician.  There is nothing more powerful than a technician with a hex editor and a calculator trying to piece together a file system.  As John Keats said, “A thing of beauty, is a Joy forever!”.  In line with this philosophy DTI Data will offer a set of free tools to help the technician view a file system, a hard drive, or a single file in order to help them and their client solve a problem.  This set of tools will be called View It Now with the proper specific tag related to the software.  For instance, the first tool will be View It Now SNAP Appliance 4.x. This tool will allow the technician to tree a SNAP 4.x file system.  Each tool will have a set functions that are unique to the file system, for instance the SNAP 4.x viewer will allow the end user to check the cylinder group alignment to make sure that the file system is not corrupted. We have also finished the NTFS version of View It Now! Download it here.

In addition to the reading and treeing the file system there will be tools such as surface scanners, block maps, raw file scanner, forensic tools, and many others.  We are hoping to give the technician a leg up on recovering a clients data so that their possibility of success is higher, and the chance of making an already bad situation worse far less likely.

The documentation for these tools will be published  in these blogs so you can refer to them at any time.  There will be a general set of documentation that will work for all tools.  Items such as configuring a river, mounting a file as a stream, scanning for a volume and many others.  There will also be functions that are specific to the tool and will be highlighted as such.  It is our hope that with the documentation as well as an open forum to make suggestions or ask questions the usefulness of the tool will increase as we update the software to reflect your input.

Hopefully these tools will be an asset to any technicians arsenal of software.  DTI Data looks forward to serving you.

If your disk is clicking or not seen by the BIOS, you will need hard drive recovery.

With all this being said this kind of design can easily lead to the file system being corrupted.  In other words  if a sector goes bad, or gets overwritten, or the data becomes corrupt, the entire file system may become unusable. As an example, the FAT file system uses a centrally located File Allocation Table which maps every cluster on the hard drive to its appropriate file.  With NTFS there is a Master File Table that is also centrally located that is even more susceptible to corruption and destruction.

In addition there are file system components such as an OS boot record that if corrupted or destroyed will give you the message “File System Not Found. Do you wish to format?”  The Master Boot Record, INDX Records, File Entry Records etc. etc. These components when destroyed or corrupted will cripple your file system and ultimately make hard drive recovery impossible.

With that in mind there are two options that I have added to the NTFS flavor of the View It Now series of software to help view the file system even when some of the components are corrupt or missing completely.  They are as follows.

Under the heading Volume Tools the option NTFS FS Functions is viewed under the drop down menu.  Mousing over that function the following is revealed.

Build Virtual Volume

Clicking on the above function will display a dialog box that reveals three values for you to enter.

1. Sectors Per Cluster:  Valid values for this field are 1, 2, 4, 8, 16, 32, 64, and 128.

2. Total Sectors:  Valid value is from 1 to the total size of the river.

3. Volume Offset: Valid Value is from 0 to total sectors of the river.

Once you have entered the appropriate values for your partition just click on mount and the Volume will be displayed in the lower left hand list box of the main dialog box.  This volume is now usuable for the following function.

Last Resort File System Build

This particular function does not depend on much of the file system being intact in order to tree the river.  This fucntion takes a defined volume and scans it searching for valid MFT records.  Once an MFT record has passed my filtering process it is added to a list of other records found. When the scan has finished or been interrupted by the end user a tree is formed and displayed just like any other river.  This will then give you the chance to view the file names and see if in fact your file is there.

In addition, after the scan is completed and the records treed you can Export File List To Text File.  That file can then be imported into a spread sheet to do searches and sorts on all the fields.

Download View It Now NTFS File System Viewer

If the hard drive is clicking or not seen by the BIOS, you will need hard drive recovery.

Well, as time went on drives got cheaper and the way to recover data from a corrupt file system changed.  The developers started writing software that moved data from the damaged file system, to a new hard drive.  This was easier on the developer and there was a very small chance that the client using the hard drive recovery software would destroy the data on the host drive.  This has worked very well… until now.

The end user of today stores more data on their hard drives than you can shake a gigabyte at.  They will store pictures, music, video, email files, games, and databases of all sorts.  All of these things use enormous amounts of disk space, so, if the drive develops a few bad sectors then trying to transport 300 GB worth of pictures and movies becomes a big deal and can actually exacerbate the problem on a damaged hard drive.   Even if it is not bad sectors, but the file system becomes corrupt, trying to move all that data to another drive just becomes too cumbersome.

In addition to a damaged drive, the operating systems have become so complex that configuring the thing takes four weeks.  Along with that you have loaded and configured your virsus scanner, malware scanner, port scanner, firewall and a thousand other things to make sure that no one breaks into your computer and swipes your bank information which is also stored on your hard disk. To wipe and reload an operating system, reconfigure all your protection, reload all your software and download all the latest releases could take weeks.  Who wants to do that?

All of this being said what is the answer to a file system that cannot be accessed.  DTI has a simple solution.  We went old school.  In this world of complex operating systems, file systems, software, virus scanners, on and on and on DTI offers a better solution.  Let one of our highly trained technicians remote into your personal computer and diagnose your file system problem.  Instead of having some piece of hard drive recovery software guess at what is wrong, let a trained tech in there to ferret out the real problem and possibly get your file system back up and running in less than ten minutes.  In addition to that many times the technician will stop a problem from getting worse and saving you the agonizing possibility of losing all of your data.

Old school for DTI Data has a new wrinkle and can save you money, time, and more importantly your data.  Give us a call and let us recover your data. To schedule a remote data recovery using Recover It Now call Toll Free 1-866-438-6932 ext. 236 or direct 1-727-345-9665 ext. 236.

If your hard drive is clicking or unable to boot, then you need physical hard drive recovery. Call 866-438-6932 to speak with a technician.

Verify Cylinder Groups

The SNAP OS file system is very similar to many Linux/Unix type file systems.  The file system is spread across the entire drive with many little sub-groups that maintain a microcosm of the entire file system.  Some file systems call these groups ‘Allocation Groups‘, others call them ‘Block Groups‘, and this file system uses the label ‘Cylinder Groups‘.

The reason that groups are used is to allow the file system to manage the hard drive in such a way as to keep wide seeks to a minimum.  For instance, if you have a database that is spread across an entire drive due to fragmentation you get the heads moving up and down across the drive which causes unnecessary heat, wear, and time to read or write data.  If the data is kept in a smaller area then the drive will have shorter seeks, hopefully contiguous reads, and a more stable file system.

All of this being said each cylinder group has its own set of inodes, a data set area, a block bitmap and other system data.  The report takes a look at some of the system information to try and determine if the cylinder groups are in alignment.  To execute the report use the following steps.

• 1. Make sure that a volume is mounted. Refer to View It Now documentation to see how to mount a volume.
• 2. Move the mouse to the ‘Volume Tools‘ item on the menu bar and click. You will be presented with a drop-down menu.
• 3. From the drop-down menu select the item labeled ‘Verify Cylinder Groups‘ and click on it. The testing will begin.
• 4. To monitor the progress of the test look in the ‘Search Information‘ group box. The ‘Block‘ label will indicate which cylinder group is being tested, and how many total cylinder groups are in this particular volume.
• 5. Once the test is finished a dialog box will appear indication how many cylinder groups did not pass the test.

The comments below are for users to ask questions and request product updates.

View It Now Quick Start Guide

DOWNLOAD VIEW IT NOW SNAP OS HERE

   The file entry record has several components that are used to extract the related stored data.  One of these components is the starting cluster number of where the data might be found.  Logic is then used to with the  File Allocation Table (FAT) map to chain the clusters together.  Lets say for instance that the starting storage cluster for file “foobar” is 1000h.  Lets also say that the size of a cluster is 64k.  Finally, lets assume, for illustration purposes that the file is 640k.  With these facts in hand we can say that foobar will occupy ten clusters. If the file had been saved only once and not edited we can make a pretty educated guess that the file is not fragmented. So the following statement may be true “Foobar is stored in clusters 1000h – 1009h”.
The FAT would look like this

    The top table shows the actual cluster number 1000h -1009h. The bottom table shows the value in each one of the FAT cells. The value in the FAT cell points to the next cluster, which in turn points to the next cluster so on and so forth. Finally the final FAT cell has the value 0FFFh which indicates the end of the cluster chain. This method is called a “one way linked list” as it links the clusters forward only, and not reverse.
    Now, this is a FAT representation of a file that is not fragmented. The following set of tables for the same file, but this time the file is fragmented.

    Now, as you can see from the above table the cluster chain looks basically the same until we get to cell 1004h. Instead of the cell containing the value 1005 which would of course point to the next cell, it has the value 100B. This means to look at cell 100B to see if we are at the end of our chain, or if we in fact continue the chain. In our example we see that the chain does continue until it reaches the end of the linked-list in cell number 100F. This is an example of a fragmented file as it is divided into two fragments. The first fragment is from cell 1000 to 1004, and the second fragment is from cell 100B to cell 100F.
    With all of this being said one can see how important it is to keep the FAT cluster map intact. When a drive is formatted for FAT the entire FAT table is destroyed and zeroed out. In the above fragmented file we can see that with the FAT destroyed there is no way to easily recover the data. The simple logic of a FAT recovery is to get the starting cluster from the File Entry record as well as the file size and then read each cluster contiguously until the file size is satisfied. In other words to recover foobarwe would read cluster 1000h since that is the starting cluster and continue for ten clusters since the size of each cluster is 64k and the file size is 640k. Doing this will lead us to cluster 1009h. Looking at our two examples we can see that in the unfragmented file we would get all of the data, however, in the fragmented file we would only get the first five clusters and the rest would be zeroes or garbage.
    I have made several references to the starting cluster number which is found in the File Entry record and then used with the FAT to get all the data for a particular file. In my next installment I will explain the mechanincs of getting the starting cluster from the File Entry record. Until next time…

Related Articles:

• Part 1 Recovering FAT 32 With File Entry Data
• Part 2 Recovering FAT 32 With File System Markers
• Part 3 FAT32 Recover File Entry Table On-Disk Layout Using a C Structure
• Part 4Using FAT32 File Entry Record For Recovering Folders Using Software Logic
• Freeware Data Recovery
• Hard Drive Recovery
  

  The first two elements of the file entry record are the file name which is eight bytes, and the file extension which is 3 bytes.  For the beginning of every folder the first file entry record has the file name “.          “, that is, a period, followed by ten spaces.  In other words the first eleven bytes of the beginning of a sector that stores the beginning of a folder are static and are always the same.  Now this fact is important in as much as we can look for this particular attribute in a sector and when we find it there is a very good possibility that we are looking at the beginning of a folder.  With the being said it is always better to try and refine a filter in order to make sure that you have in fact found the beginning of a folder.  Oddly enough, the second file entry record had the file name “..         “, that is, TWO periods followed by nine spaces. 

   Each file entry record is a static thirty-two bytes long.  That being said we can now assert this.  If bytes zero through seven of a sector are a period followed by ten spaces and bytes thirty-two through forty-two of the same sector is two periods followed by nine spaces then there is a high probability that we have found the beginning of a folder.

   The pseudo logic for this might look like:

     The chances of a sector slipping by that may not be a folder entry is very slim.  This is good in as much as it can be difficult to define a filter that will give you the most optimal results without flooding you with a set of worthless data.

     In my next installment I will explain what this ‘.’ and ‘..’ mean.  If there are some old DOS command line users out there I am sure you are well aware of what I am talking about.  Until next time…

Related Articles:

• Part 1 Recovering FAT 32 With File Entry Data
• Part 2 Recovering FAT 32 With File System Markers
• Part 3 FAT32 Recover File Entry Table On-Disk Layout Using a C Structure
• Freeware Data Recovery
• Hard Drive Recovery
  

Since a small program will be offered at the end of this sequence of articles the on-disk explanation will use ‘C’ notation and a defined record data type using structures. The following is the structure I use in my software to denote a file entry record.

#pragma pack(1)
typedef struct
{
UCHAR FE_Name[8];
UCHAR FE_Extension[3];
UCHAR FE_Attribute;
UCHAR FE_NTReserved;
UCHAR FE_TimeCreatedTenth;
USHORT FE_TimeCreated;
USHORT FE_DateCreated;
USHORT FE_DateLastAccess;
USHORT FE_StartClusterHI;
USHORT FE_TimeLastWrite;
USHORT FE_DateLastWrite;
USHORT FE_StartClusterLO;
UINT FE_FileSize;
}FILE_ENTRY, *PFILE_ENTRY;
#define SZ_FILE_ENTRY sizeof(FILE_ENTRY)
#define FILE_ENTRY_NULL (PFILE_ENTRY)0

This file entry record format has been in use since the inception of DOS FAT File System. There may have been some fields that were at one time referred to as reserved but the size and basic design of the record has always remained the same. The following is a description of the fields used to recover the original folder heierarchy.

1. FE_Name[8] — This is from the old 8.3 notation and the first eight characters of the file name. In later articles I will cover the long file name logic and how we can use that to enhance our search routines. It is important to note that if the file name is less than eight characters the rest of the name is space filled (0×20)

2. FE_Extension — The second part of the 8.3 notation that has the same rule set as the FE_Name field. When displayed the data in this field is after the period ‘,’.

3. FE_StartClusterHI — Stored here is the binary high short value of the starting cluster of the file, and or folder.

4. FE_StartClusterLO — Stored here is the binary low short value of the starting cluster of the file, and or folder

5. FE_FileSize — Lastly the file size. If this file entry record defines a folder then this value is zero (0×00). In addition you can see that the file size is an unsigned int, so the largest file size that can be stored is 4,294,967,295 bytes, or 4GB.

Now the time of file creation and last update are important as well but the five fields that I have outlined are necessary to recover the folders.

In my next installment I will outline the attributes of the file entry record that can define a folder entry.

Related Articles:

• Part 1 Recovering FAT 32 With File Entry Data
• Part 2 Recovering FAT 32 With File System Markers
• Freeware Data Recovery
• Hard Drive Recovery
  

In the FAT file system the index records for each file and folder are not stored in one static area. As an example of an alternate technique, if you were to format a drive using Windows XP then, of course, the default file system would be NTFS. One of the characteristics of NTFS is that it uses a Master File Table to store all the information about each file and folder. The MFT is stored almost exclusively in the same place every time a drive is formatted. Normally the MFT will start at cluster 786432 (LBA 6291456 assuming a 4K cluster) and will extend contiguously for several thousand records. In other words, the entire index for your file system is stored in an area approximately 150MB to 200MB in size. If this area were zeroed out it would destroy all of the information as to where your files are stored and in most cases hamstring the end-users ability to recover their data, especially if the data is fragmented. One may think that 200 MB of data is a lot of data to zero out, but I can assure you with Windows XP optimized for disk I/O and hard drives using a blazing fast DMA the destruction of the MFT would almost be transparent. You would never know it happened until it was too late.

That being said, conversely the FAT file system stores their folder and file information in clusters using a file entry record. As the file system matures the clusters that are used move farther down the drive since data is now occupying the clusters closer to the beginning of the drive. The FAT chaining system is used for folders that have more files than can be stored in a single cluster. It is easy to see that the folder information can be scattered across the drive. Although this plays havoc with hard drives and access speeds it makes it difficult to destroy the file system. This cluster scattering was not by design and to this day is considered a drawback to the file system, however, it does make data recovery much easier when major file system components are lost.

So now that we know that file entry records are used to index the file system for FAT the problem arises as to how best to identify a file entry record from the billions of other bytes on the hard drive. In my next installment I will outline a file entry record and reveal the attributes that allow us to filter a folder storing file names from all of the other data.

Related Resources:

Recovering FAT 32 with File Entry Records – the first part of this series.

External Hard Drive Recovery

Hard Drive Recovery

  In its simplest explanation a hard drive works much like the old phonograph record players.  The record would be placed on a spindle and a needle that is on the end of a tone arm would be placed on the record.  The needle reads the recorded music from the record and transfers that data to the amplifying device.  Now that being said how is a hard drive like a phonograph record? 

  A hard drive has a platter or set of platters that are similar to the vinyl record.  These platters are mounted on a spindle. There is also a ‘tone arm’ only on a hard drive it is called an actuator arm.  At the end of the actuator arm is a set of heads which are comparable to the phonograph needle.  This is a basic description of a hard drive, however, there is one huge difference in functionality.

  On a phonograph record the needle sits ON the record.  With a hard drive the needle floats on a cushion of air over the platters.  Since this is the technology used then it is important that the heads remain at a constant distance from the platters.  When a head reads a hard drive it doesn’t send a single beam to the hard drive to read it.  The signal looks like a cone and the farther the head gets from the drive the wider the cone is.  Conversely, the closer the to the platter the more compressed the cone is. 

  With this being said, each track on a hard drive is basically equidistant from each other.  So, if the head is the correct distance from the platter then the head will read only that track.  However, if the head is too far away it will read multiple tracks, this is called over-scan.  If the heads are too close only part of the track will be read, this is called under-scan.  As an example imagine a paint sprayer and you are spraying a picket fence.  Each slat on the picket fence is a different color.  Now, if you hold the paint gun (head) too far away from the fence (platter) it will spray the neighboring slats (tracks).  On the other hand, if the paint gun is too close you only spray part of the slat.

  With these facts here is why platter swelling is bad.  A platter is not completely flat.  Microscopically there are many flaws in a platter, however, because the heads float on a cushion of air the imperfections do not affect the physics of the read.  If, however, the platter swells then the imperfections become accentuated and the cushion of air can no longer compensate.  In many cases the platter imperfection can become so large that it exceeds the distance of the air cushion and actually touches the head.  This is called a head crash and can scratch the platter, damage the head and wreak all kinds havoc with a hard drive.

  If your computer is in a room that exceeds eighty degrees then that could cause platter swelling.  If your tower is inside of a desk and the air flow is restricted that will cause platter swelling.  If there is dust in your tower then that restricts airflow and can cause platter swelling. 

  So keep your tower dust free, keep your computer room cool, and keep the tower in an area with good air flow.

  Well, now that I got that off of my  technical chest hopefully next time I will cover the the mechanics of finding a stale drive in your RAID five.

  Until next time…
For more info on hard drive recovery

In these next few installments I will reveal what actually happens when using the Windows XP fdisk to partition a disk. What data components get destroyed, how those components can be replaced, and what is really important to the operating system when trying to mount a file system.

The tool we will use is ‘Free Partition Recovery’ with a few enhancements for this particular problem. This tool is extremely powerful and can make matters worse if not used in a safe and proper manner. However, I will take you step by step on how to do the recovery of your fdisked drive.

Next installment I will discuss exactly what happens to a file system when ‘fdisk’ is used and the stop gaps Microsoft has put in to help us in the recovery of their file system. Hopefully with this knowledge we will be enlightened and begging to chant the mantra “Microsoft… Microsoft… Microsoft…”…

Until next time…

The next step in our “Search for the Stale Drive” is to use the RAID Diagnostic Toolkit to scan all the drives. From the menu bar “Configuration” option select “Populate Stream List”. This function will scan the current system you are working on for all hard drives including ones tied to a USB device. These devices will be listed in the “Stream List”. If you are working from images then from the same “Configuration” option on the menu bar select “Add File To Stream List”. A standard Microsoft Windows file selection dialog box will appear. You may then use that to add your images to the “Stream List”.

Once all the streams have been placed in the “Stream List” list box, double click on each one that is part of your RAID five. As you double click each stream object will be placed into the “RAID Configuration” list box. Please be sure to place the streams in order in the “RAID Configuration” list box as it will help in determining which drive is stale.

I can offer this clue. In a stacked set of drives that have been residing in a cage several things happen. The outside drives remain the coolest for two reasons. First, they have unrestricted air flow across at least one side of the drive, and two, dust has a more difficult time collecting and obstructing air flow. This fact alone is one of the main reasons why the interior drives seem to fail before the outside drives do as the inside drives have restricted air flow.

Another thing that happens is the interior drives have heat coming not only generated by them, but they have heat from the drives that are next to them. This can exacerbate an already poor air flow problem and cause an early fault for the hardware.

As a brief aside the reason why heat is bad for hard drive is multi fold but there are two main points. First, each hard drive has a motor and that motor has some type of lubricant. If in fact the drive gets hot the lubricant can lose its viscosity and the spindle can begin to slow and cause a head crash. Secondly, heat will swell components of a hard drive, in this particular case I am referring to the platters.

Next time I will discuss how platter swelling can cause a head crash, and what to look for once the parity scan is executed.

Until next time…

The main objective of the parity check is to make sure that:

1. We do not have a stale drive in the array

2. We do not have a drive in the array that does not belong

3. All RAID data offsets are correct.

Lets take each item from one to three and explore their impact. Item one basically means that there is a drive in the array that has not been functioning for a certain period of time. Normally an alarm goes off, an email may be sent, there is some sort of notification that a drive has dropped out of the array and now the RAID is running in a degraded state. When the technician who is administering the array does not get a warning it is usually because there has been some type of hardware malfunction that, although the drive is out of the array, the RAID BIOS does not sound the alarm. A second reason is that the alarm stops working. The little speaker on the RAID card that sends this terrible shrill through the server room is malfunctioning and nobody hears it. Another reason might be that the original RAID administrator may have shut off all alarm notification flags during configuration and never turned them back on. There are a lot of other reasons but the fact of the matter is that a RAID administrator may have a RAID that has been degraded for a year and not even be aware of it.

Item two is rare, however, it happens enough to where you need to be concerned if you are trying to recover your RAID. This item also is not very common in the SNAP line of servers as it is in DELL. There are times when a RAID is configured as ‘X’ drives, and one hot swap. The RAID admin who is now working for the company you are trying to recover the data for sends the RAID he tells you it has four drives when it is really three drives and one hot swap. He may not know the original configuration. He may not know how to get into the RAID BIOS to look to see how it was configured. There could be a hundred and one reasons as to why you get a hot swap drive sent to you along with the rest of the array. The point is, be aware that it can happen.

As a side note, DELL has configure many of their RAID models to have two mirrored drives for the OS, and 3 to X drives as a RAID 5. I have received all the drives from a client with them ’swearing’ that all of these drives are in the array. Once I have analyzed the parity, and look at the drives through a hex editor I come to the realization that I have two RAIDS on my hands, not one. Once again, be aware that the client may not know their exact configuration.

Finally item three. Sometimes, not often, actually this was the first time with a SNAP server, the RAID data offsets are staggered. In my next installment I will explain what happened with this particular job, and why it happened. Until next time.

Click here to Download the RAID Diagnostic Toolkit. Be sure to read the instructions on the page as well as follow the links to the instructions with screenshots. You may also visit our page: RAID Configuration and Parity Check for more information. Find out more about RAID Data Recovery.

First thing I do is to make images of all four drives.  These were four identical Seagate Barracuda ST380011A hard drives, so I made sure I had at least 320GB of space on one of my partitions on my server and, using WinHex dumped the images.  Once I had done this I put the clients original drives in their bin hopefully not to use them again.

Next step is to use WinHex and eyeball the beginning of the RAID data.  With SNAP OS this is a simple matter of looking for the first cylinder group on the first drive then subtracting forty eight sectors from that.  The assumption is that the block size is 8192 bytes, or sixteen sectors.  If we were to look sixteen sectors before the first cylinder group you would see the file system superblock.  If we skip back another 16 sectors you see another super block.  Finally, another sixteen sectors and there should be a null sector.  Sometimes I see data in there but that is usually because the drive has somehow been corrupted.

So, once again, to find the beginning of the RAID data segment you find the first cylinder group and subtract forty eight sectors from that.  The sector offset derived from that formula is the beginning of the RAID data segment of each drive.  They will be the same on all four drives or at least I thought that until this particular recovery.

Next step will be to check the drive parity which, in this case, was unusual. This step will be in the next blog titled “Analyzing RAID parity“.

For more info on RAID Data Recovery or SNAP Data Recovery

First we must populate the RAID with streams. There are basically two types of streams that we will use, the first is a physical data stream or ‘hard drive’. The second is an image data stream or ‘file’. Figure A depicts populating the ’stream list’ with physical streams. As you can see the ‘Populate Stream List’ menu item is highlighted. Clicking on this will poll all hard drives on the local machine and display them as shown in Figure B.

Figure A

Figure B

The best way to test an array is to make images of the hard drives and then use the images for testing. From the ‘Configuration’ menu option click on “Add File Stream To List”. A standard Windows file selection dialog box will appear. Go to the proper folder and choose the image that you would like to add to your stream list. Click on the file, and then open and the file will be added to your stream list. You are now free to add this item into your RAID Configuration list.

In order to add an item from the stream list into the RAID Configuration simply double-click on the stream list item and it will be added into the RAID Configuration list of items as depicted in Figure C.

Figure C

Next, in order to start the parity test click on the menu item “Diagnostics”. Doing so will reveal the menu item “Raid Five Parity Check”. Click on that menu item and the diagnostic will begin. This function will check the RAID five on a stripe by stripe basis and validate the parity using XOR mathematics.

In the lower left hand corner of the software is a small status/information window that offers real time data of the parity scan. this window contains five items which describe the state of the diagnostic.

Type: The configured RAID/River type

Ident: Identifier give to the RAID/River type

Block: The block, currenty being scanned by the software

Time: Time remaining till the scan has completed.

Errors: The total blocks that a parity error has been found.

Two of the five items are most pertinent for this particular function. They are the “Errors” item and the “Block” item. If the “Error” item is ten to fifteen percent of the array then the array stripe is probably corrupt and you may have a stale drive in the array. For all practical purposes however, there should be less that or a total of three or four total errors for the entire array. A healthy array will have no errors and if even only one appears that could mean either the hardware is starting to fail, or worse, the firmware and or its accompanying memory me be buggy. Either scenario could spell disaster for your array and should be looked at immediately. View Figure D as an example.

Figure D

Finally, if you wish to interrupt the diagnostic just click on the “Configuration” menu item, and then the “Interrupt Processing” item and all processing will stop.

That’s it! Of course you must always bear in mind that even if the RAID does not pass the parity test there may still be data to recover. Alternatively if it does pass, this does not necessarily mean that the RAID is good for a rebuild. There will be other functions added to the software that will help you better determine if a rebuild is advisable.

Dick Correa

In addition to the infomation stored in the Master File Table it has been my experience that if a previous copy of the Master File Table had been saved off into a file onto a remote site I could have easily imported that file and used it to recover the data. In other words, it is rarely the occasion that an entire file system gets totally wiped out. It is usually some small piece of information either corrupted or omitted from the Master File Table that causes the problem. Even a restore disk used on a hard drive that totally destroys all remnants of a file system cannot keep a backup copy of the Master File Table from recovering some data.

How, you may ask can this be? Well grasshopper, read on and see. Imagine a book. A reference book preferably. Now, let us define the attributes of a reference book. Lets see, there is a forward where the author may offer a few remarks so we know how intelligent he is. There is a table of contents that give you a general idea of what is in the book and where it is located. There is the body of the book, the actual information. Last but not least, an index. A detailed description, with page numbers that tell you exactly where the data is that you are looking for. For illustration purposes we can say that the index of the book is the Master File Table, and the body of the book is the data on your hard drive. If the index of the book is ripped out of the back, how would it be possible to find the information you are looking for? I suppose you could wade through the entire book and possibly, after several hours of searching, find the answers you are looking for. I have done that with some of my older books where the back, and the front of the book have disappeared. A book may have 200, 300, 400, maybe even 500 pages to look through, and if the information is important enough it is worth the look. However, wouldn’t it have been easier if I would have just photo copied the index and placed that in a nice safe place. Then, when the book gets old, and I lose the index, I have this nice copy that I have kept to help me find my information.

Leafing through a 500 page book may be time consuming but it is feasible, however, apply that same logic of the index and the book to a hard drive. Who wants to scan through 234,000,000 sectors looking for data. If the data is fragmented then the data is probably lost. Wouldn’t have been nice to have a copy of the Master File Table to use and find all of your old tax returns, or doctoral thesis, or the only pictures of your grandsons birth? I would say, “Yeah!! It would’ve been nice!”.

Please don’t get the wrong idea. This is not the same as entire backup, on another set of media. There are holes to this system. First, if the drive actually goes bad, then it will be difficult if not impossible to get the data back. Secondly, any thing that writes to the data portion of the drive will make the Master File Table useless. However, it takes a long time to destroy a 250 GB hard drives data area. Lastly, I have not been able to find a piece of software that just dumps the Master File Table to a remote site. Looks like someone should write one?

Secondly, an email will be sent, advising you that the four years worth of data that you thought was being backed up, but you discovered two days ago wasn’t,  is now in peril of being lost into the never never land of lost bits and socks that inexplicably disappear from the dryer.  Yes, your job, your home, your marriage, all will be lost unless you heed the email warning and immediately shut down the server, to the chagrin of 427 end users who are reading about American Idol. HAH! Welcome to the party pal! (Quote circa 1976:  Bruce Willis: Die Hard)

Keep in mind that although there are many RAID cards, as well as on-board RAID interfaces that perform these functions, your particular RAID firmware, as well as its current configuration may not.

The reason that a RAID 5 can have a drive go down and still run is the mathematics of XORing (eXclusive ORing) the data.  This method for keeping the data relatively safe in a RAID 5 is called parity. It is a manipulation of bits in each byte of data.  For the unwashed a byte is eight bits. 
       
In order to under stand XORing it is imperative that you understand the XOR truth table.  Figure 1 is the truth table for eXclusive ORing.

                                                                                           Figure 1
        Figure 2 is an example of XORing and how it relates to a four drive RAID 5 and the parity.
       

                                                                                           Figure 2

The data is arranged thusly:
‘R’ is the ASCII letter
52h is the ASCII hexadecimal value of ‘R’
0101 0010 is the ASCII binary representation of the letter.
As you can see each letter has is set up the same way.

For illustration purposes the following can be assumed.  Each line is considered a single byte of a stripe as conveyed by Figure 3.  If we take the ‘R’, and the ‘F’, and the ‘T’, and XOR them together, we get the value in D4, where each bit, of each byte is individually XORed across the stripe. 

                                                                                         Figure 3

Using Figure 4 as a base, and the truth tables we can see the following:

Figure 4

 Now, lets say we lose D2 (drive two) in the array.  The following is how the RAID card firmware handles it.

Figure 5

We have built drive 2 on the fly.  We do not need to know the data since we can use the XOR truth table and the remaining three drives data to calculate the value of drive two. In the above example the process was illustrated for one byte across one stripe on a four drive array.  All of these calculations are done in an instant on a stripe by stripe basis.  The full stripe is recalculated for every write, and if a drive is out of the array for every read of the down drive.  With all these calculations you would think it would slow down the processing. To a degree, it does, however, bus I/O is infinitely slower than any XOR math a CPU may have to perform.  A way to emphasize this point is imagine you are standing on a bridge.  Below you is a river.  Each byte of data is a boat that passes under the bridge.  The boat travels from the hard drive, down the river, to memory, and to the CPU.  As one boat passes, you wait for the next boat.  The next boat will not pass for one hundred years.  The CPU is in a perpetual wait state.  It is always waiting for data to process.  So, if you want to speed up your PC, by high speed I/O smart boards that can RAID, on a high speed bus. 

To be continued…

Learn more about RAID Data Recovery

We have been asked to do many similar jobs where the archive of a set of data has been compromised.  Many lawyers have databases of all of their scanned briefs as well as all documentation pertaining to a particular case. If that information is lost and the case reopened for appeal it could be devastating to not be able to review the documentation in a timely manner. I mention this because it took me over a month to complete this task, and although interesting, was very tedious.

What made this recovery interesting was that the drives were in two physical devices.  The first device was a four drive SNAP array that was used as the head. The other device was a twelve drive SNAP server that was broken up into two RAID fives.  The challenge for this recovery was that no one knew which drives were in which array, no one knew the drive order of any array, the configuration given to me by the SNAP server was in error, no one knew the stripe size of the array, and finally, the data recovery company who had the array before me, marked the drives out of order. In other words, I was handed 16 drives and told to figure out a triple spanned RAID five.

So here are the steps I took to solve this data recovery problem for my client.

Step one, I had to find out which drives went with each other.  I would have hoped that each RAID was equal in size.  In other words, I hoped the RAIDs would all be four drives for the head in one array, and eight drives each for the other two RAIDS, but this was not the case. In order to find which drives went with which array I had to know several things. 

First, I had to know the SNAP layout for arrays.  Each drive in a SNAP array is basically broken up into two parts, the operating system, and the data area.  In order to find the size of each you must look at the master boot record (MBR) of each of the drives.  The MBR houses the partition table which is a listing of the active partitions.

SNAP partitions are divided into three basic areas, an operating system partition, a swap partition and a data partition.  SNAP Appliance designed their device so that if one of the drives went down the firmware would roll to the next drive to load the operating system, network interface, and RAID handler.  The important piece of information is what the standard offset to the data area is. The data area of each drive is used for the RAID 5.  I have found the data area sector offset for the Guardian OS series to be LBA sector 2216970.  This information may change from version to version, but all the Guardian operating systems I have worked with have been the same.

Now that we know the data area offset we can take the next step, which is to determine which drive sets comprise the three RAID sets.

To be continued……..

RAID Data Recovery

MOV SI,07C1B  ;set SI (source index) to 7C1B
MOV DI,0061B ;set DI (destination index) to 061B
PUSH AX            ;PUSH AX for segment for RETF to 0
PUSH DI             ;PUSH DI for offset for RETF = 061B
MOV CX,01E5   ;1E5 bytes to copy
REP MOVSB      ;Copy the MBR
RETF                   ;PULL 0000:061B off stack and JMP to MBR

Once we have the bootstrap code relocated we begin to scan the partition table for a bootable partition.The following figure is one record entry in a partition table. There can be a total of four entries in a partition table.

Partition Record Layout

Once a bootable partition is found, the boot code pointed to by the relative sectors field of the partition record is read into memory location 0000:7C00 and then an RETF is executed and the boot code for the operating system takes over. This type of boot system is called two phase because the MBR does not load the operating system, it only loads a loader.In other words, the MBR is a preloader for the OS boot loader.

During the boot process and the parsing of the partition records many things can go wrong.If there is not a boot flag set then the system may just hang with a blinking cursor in the upper left hand corner. You may get the message “Invalid partition table”, if the boot flag is set to something other than zero.You may also get the message “Missing operating system” in the AA55 magic number is not present in the OS boot loader. These errors can be fixed by hand. The following is a step by step approach to fixing the boot flag, as well as the OS magic number.

Repairing Boot Flag with WinHex

Each technician has his own set of tools that he relies on.In my case many of my tools are custom written since I am a software engineer. However, there are some tools that are just so good they cannot be improved upon.WinHex from X-Ways Software is such a tool.It has two attributes that are essential to a technician, one, it is extremely flexible, and two, it is very reliable. So, that being said fire up your copy of WinHex, and if you don’t have a copy, the download one, and let the boot flag repairing begin.First of all, as depicted in Figure A, the boot flag is located at offset 446, or 1BEh.In this particular case the boot flag is set to 80h, which is bit seven set. This is a normal partition record, in a single partitioned hard drive.

Figure A

In Figure B we can see what a multi partitioned drive looks like:

Finally, in Figure C we can see what bad boot flag value looks like. The bad boot flag value in Figure C is the value 65h. In order to fix this problem merely change the value to either 80h if you want it bootable, or 00h if you just want it mounted.

Figure C

As you can see, that hard part is knowing where the boot flags are, and then knowing which values you can place in them in order to mount the volume. I have given them both to you. Learn more about data recovery.

Over the years many things in the world of computers has changed. We have gone from command line, to GEM, to a windowed GUI type of operating system. A stick of memory used to be one hundred dollars for 1 MB, now its seventy five dollars for 1024 MB. A forty megabyte hard drive was $1000.00. Now you can get 750 gigabytes worth of hard drive for two hundred and fifty dollars. We have gone from 8 bit (XT), 16 bit (286) 32 bit (386), and finally 64 bit (EMT 64) central processing units.

With all of this obviously monumental progress, one of the most important functions of the computer has never changed. The boot sequence. Oh yes, it may have been enhanced, there may have been little items added here and there, but the focal point of the boot up process has never changed. Let’s take a quick look at the steps in booting a PC.

When you switch on your PC immediately the BIOS takes over. In its process it does what is called a POST (Power On Self Test). The POST is a set of diagnostics that will test the hardware of your computer. Examples would be memory, bus, CPU, ports, PCI bridge, and the like. Through the use of checksum, and data echoing the POST can tell if something is amiss in your hardware. If the POST finds something wrong, and considers it a fatal error, the boot process will be halted, and a series of beeps may be given. These beeps, depending upon the BIOS developer, will guide you in diagnosing your fatal error. The beeps are used because many times an error in the hardware will make it so the video cannot be used. The POST is much more comprehensive than is being presented here; however, this set of articles is about data recovery and not how to diagnose your POST process.

Master Boot Record

The BIOS has found a HDD that is in the list of bootable devices. The BIOS will then load the first sector of that HDD into memory. Just as a point reference a sector is defined as 512 bytes of data. So, once again, the sector is loaded into memory. You may ask yourself, “Self, where exactly in memory does the BIOS load the MBR?”. An excellent question! From nearly the beginning of the PC industry. From the dawn of the BIOS, it has been scribed by the ancients that the MBR will be loaded into memory location 0000:7C00. (Drum Roll, cut to Yul Brenner laughing maniacally shouting “So let it be written, so let it be done!”. He was a good Ramses but Charlton Heston was a great Moses!)

In addition, there are some BIOS’ that will perform a small test on the MBR to see if it is valid. The test is to make sure there is a 0×55, and a 0xAA in bytes 510, and 511 respectively. If those bytes are not present, some BIOS will stop the boot process for that HDD and continue onto an alternative device. If all devices fail, then an appropriate error message will be displayed.

Boot Failure: System Halted is the choice of Award BIOS. If , however, the MBR is loaded and passes all of the BIOS testing INT 0×19 is called and a jump to memory location 0000:7C00 is performed. There, control of the entire PC is passed to the MBR. Awesome!

The knowledge that I will impart over the next several weeks is not for the faint of heart. Although I will use plain language, as well as diagrams where needed, the application of the information is meant for technicians and software engineers. However, everyone I am sure will come away with a better understanding of the NTFS file system.

The following are the topics that will be covered on a week by week basis. Hopefully, I will be able to maintain the weekly schedule, however if life and my work get in the way, some weeks I may have to skip until the following week.

Week 1:

Boot Up Sequence: This will include how the BIOS determines a boot device. Once a boot device is determined how the BIOS hands over the boot sequence to the Master Boot Record. The layout of the Master Boot Record and how the boot code determines the boot partition. What can go wrong during the boot sequence and some ways you can fix those problems.

Week 2:

Continuing the Boot Sequence: This will cover how the Master Boot Record hands over the booting of the operating system to the OS Boot Record. This will include the layout of the BIOS Parameter Block, and how its data elements relate to data storage. A brief explanation of the NTLDR. I will also cover the problems that can arise during the OS boot and how you can possibly repair the problem.

Week 3:

Data Storage Part I: Understanding how data is stored is critical if you want to have even a remote chance of file recovery. Discussion on clusters, why they are used, how they are allocated. How the operating system stores the data on the physical media. Logical and physical sector addressing arithmetic will be explained. Fragmentation, the enemy of data recovery will also be explored.

Week 4:

Data Storage Part II: Once we have covered how the data is stored, we will need to then understand how NTFS handles keeping track of file and folder placement. The Master File Table will be discussed in great detail. Where it is placed, many of the components of the record will be discussed and how they relate to what we see translated into the file explorer.

Week 5:

Data Storage Part III: This week will be totally dedicated to run lists. This component is the key to breaking down how the clusters are stored. The method that Microsoft uses to track clusters is very complicated so I want to give this subject a full week.

Week 6:

Data Storage Part IV: Now that we have all of the theory of the MFT, this week we will cover how to recover data using a damaged MFT.

Week 7:

Hard drive theory: This will be a brief overview of hard drive design from a data recovery specialist’s point of view. Sector mapping, system area design. Permanent Defect tables. Growing Defect tables. Bad sectors, and how that relates to performance. How the operating system reacts to a bad sector. S.M.A.R.T early warning technology.

Week 8:

JPEG File recovery: The JFIF file format and how that relates to raw data recovery. Data mining, file carving and techniques used to extract data from a totally destroyed file system.

Week 9:

MP3 File recovery: The MPEG III file format will be covered and how that relates to file recovery. How data mining and file carving techniques may be used to recover the file. The ID3 data tag format and how that can be used to recover a more complete MP3 file.

Week 10:

Scenario: Hard drive has been fdisked, how do I recover?

Week 11:

Scenario: Hard drive has been formatted, how do I recover?

Week 12:

Scenario: Files have been deleted, how do I recover?

Week 13:

Scenario: I used the restore function from the manufacturer, how do I recover?

Week 14:

Scenario: Multiple partitioned drive made into single partition, how do I recover second partition?

Week 15:

Scenario: USB hard drive cannot be addressed, how do I recover?

Week 16:

Scenario: I lost all of my Canon CR2 Raw format pictures, how do I Recover from the flash chip?

Week 17:

Scenario: Hard Drive has reached maximum capacity and file system as well as the operating system have become inoperable. How do I recover?

Week 18:

Scenario: Malware virus attack on the Master Boot Record, or the operating system boot record. How do I get my operating system online?

Week 19:

Scenario: Drive has been formatted, and the operating reloaded. How do I recover data from the drive?

Week 20:

Scenario: Deleted email from my Outlook Express mail handler. How do I recover the deleted emails?

Week 21:

Scenario: Outlook PST file has exceeded the two gigabyte limit. How do I recover my email file without damaging all of my data?

Week 22:

Scenario: Reloaded Windows over the top of an existing Windows setup and lost my access Documents and Settings folder. How do I gain access to the folder?

Week 23:

Scenario: Hard drive has exhibited symptoms of possible bad sectors. How do I safely recover my data without compromising the physical attributes of the hard drive?

Week 24:

The future of data storage and what is needed in order to safeguard the data on your system. I am going to cover a great deal of material in the next six months. As I reveal each secret hopefully that information will help you recover and safe guard your data. If you have any questions please feel free to call or drop me an email.

Dick Correa
dickc AT dtidata.com
(727)345-9665 Ext 203

Recovering a single file from a SNAP OS Part 3

The inode is the final link in the chain of data storage.  It holds the map of the blocks where all of the data of each file and directory is stored. Let us dissect the inode and find its most important elements.

The SNAP OS Inode

Figure 1 is a raw hex representation of an inode. There are several data elements within the inode that define the date the file or directory was created, the last time it was updated, and the size of the file. For our purposes however, we are only concerned with one area of data elements and those are the direct and indirect block definitions. 

Fig 1

The direct block definitions are defined in the shaded green area, and there are a maximum of twelve direct data blocks. The term direct means that each one of the four byte numbers in the shaded green area point to an actual data storage block. In other words, if we take the first value of 0×14A80E (1353742 decimal) and go view that data block, we will find the first values for our file 2003STEP.PDF. In figure 2 we can see the first few bytes of data from block 0×14A80E.

Fig 2

There are only twelve direct data blocks so if your file exceeds 96 k, then the file system will use a method defined as indirect blocks. There are three data elements of these blocks, they are:

• Indirect Block: Points to a block that has a list of data blocks.
• Double Indirect Block: List of blocks that point to an Indirect block.
• Triple Indirect Block: List of blocks the point to Double Indirect blocks.

From the above explanation you can see how deciphering a very large file can be extremely complicated. Once understood, this method works well and is very fast. Along with those facts, it is also very easy to program using recursion and a set of flags to let the recursive function know what is being processed. 

Figure three is a listing of the 2003STEP.PDF direct blocks from its only indirect block.

Fig 3

Well, that’s it!  By using the formulas and techniques I have outlined in my last three articles you can easily retrieve any file. I hope this helps those of you that have lost data due to hardware and or software failures on your SNAP Server.

If you have any questions, or if I can be of any help, please feel free to call me, or drop me an email.

(727)345-9665 Ext 203

dickc AT dtidata.com

SNAP Server Data Recovery of a Single File

Here are all the articles about SNAP Server Recovery of a single file:

1. SNAP Data Recovery - the first post about the SNAP OS.
2. SNAP Server Data Recovery of a Single File – A detailed post about recovering a lost file on the SNAP OS.
3. SNAP Server Data Recovery Using The Super Block - the next article about SNAP file recovery.

Our main page for SNAP Server Data Recovery.

Recovering a single file from a SNAP OS Part 2

Last week we discussed how to find the file name on a SNAP OS file system. Using a sector editor we searched the hard disk drive for the file name. Once we found the file name I broke down the on disk data structure format for a directory/file entry.  Among the many elements of the structure the most important in determining where the data for the file is stored is the inode number. In this weeks installment we will discover how to use the on disk data structure called the super block. This is the key to the entire file system and is essential if we are to find the data related to this file name.

In order to find the super block, we must first understand how the SNAP OS positions itself on a drive. In windows the basic element of storage is a cluster. Standard cluster size for an NTFS drive is 4096 bytes, or eight (8) sectors. For SNAP OS the basic element of storage is a block. Standard block size for SNAP OS is 8192 bytes or sixteen 16 sectors. Using the block as a basis for storage, we then have groupings of blocks. These groupings of blocks are called cylinder groups. The on disk layout of a cylinder group is as follows.

Cylinder Group On Disk Layout

All blocks are relative to the beginning of the cylinder group.

• Super Block: Block 0: This block houses a copy of the on-disk structure of the super block
• Cylinder Group: Block 1:  On-disk structure of the cylinder group
• Inodes: Block 2 – (2 + n)  Inode StorageEach inode is 128 bytes, therefore 4 inodes per sector, or 64 inodes per block.
• Data Blocks: Blocks (2 + n)  – (end of cylinder group)  All remaining blocks to the end of the cylinder group are data blocks.

Applying real world numbers

In order to help illustrate how all of this works together, let’s take the 2003STEP.PDF example and apply it to our on disk definitions.

First of all, we must find the super block. The best way to do that, is to find the first cylinder group using the magic number I spoke of in my article “SNAP RAID Recovery Using SNAP OS”. The magic number for the cylinder group is 0×550209.  So, using WinHex as my sector editor, I plug that value into the “Hex Search” field and run the search.  In this case the Cylinder Group is stored at sector 48.  Now, we know that the Cylinder Group data structure is stored in relative block 1, and we also know that a copy of the super block is stored in relative block 0. The size of the blocks is 8K, so, we can count back 16 sectors, or 8K and find a copy of the super block.  

So, a copy of the super block is stored at sector 32.  There are some data elements within the super block that will help us identify the exact placement of the inode we are looking for. These elements are as follows: (Remember the numbers are for this real life situation only, your numbers may differ because of disk size, formatting flags etc.)

Super Block Offset:               2

Cylinder Block Offset:          3

Inode Block Offset                4

Data Block Offset                  16

The above numbers are relative to the beginning of the volume. Therefore we can find the beginning of the volume by using the Super Block offset.  The Super Block is stored on block two, or translated to sectors, sector 32. If we subtract 2 blocks, to find block zero, which is the beginning of the volume we will find the beginning of the drive.  This is important since many of the SNAP OS volumes I work on are RAID-ed. There is a great deal of extraneous data when dealing with RAIDs, however, using this formula, we can easily find the beginning of the drive on a destriped RAID set.

Secondly, and more importantly, we can determine the total inodes per cylinder group.  As defined before, we know that there are 64 inodes per block.   In our real world example we can see that the inode block starts at relative block 4, and the data block starts at relative block 16.  If we subtract 4 from 16 we know that there are 12 blocks of storage per cylinder group.   We know that there are 64 inodes per block, times12 blocks, or 768 inodes per cylinder group.  There is a data element in the super block that tells us the inodes per cylinder group.  If we take our previous calculation, and it matches the super block data element, then we know that our file system is aligned.  In this case they both match.

Now, if we know that we have 768 inodes per cylinder group, and the current inode we are looking for is 1015297, we can divide the inode we are looking for, by total inodes per cylinder group to find the cylinder group which house our inode. That value 1322.   We then do the mod of the same values to tell us which inode within the cylinder group is the one we are looking for.  That value is 1.  So, we can say in cylinder group, 1322, inode 1, we have the inode we are looking for.

Lastly, how do we find Cylinder Group 1322?  The size of the Cylinder Group is the size of the data group plus 64 sectors.  So, in my case, the data group was 1024 blocks, or 16,384 sectors.  You add 64 sectors to that and you have each cylinder at 16,448 sectors.  One note, every 16 cylinder groups is an adjustment of 1024 sectors.   So the 16th cylinder group is only 15,424 sectors.

That’s, it!  Now that we have a method for finding the inode, we can actually start pulling data off.  I will cover direct disk blocks and the formula for pulling data off of the drive in my next installment.

As we know SNAP Appliance used a proprietary Unix File System (UFS) handler in order to run there Network Attached Storage (NAS) product. This particular OS ran a Berkley Software Distribution (BSD) flavor of UFS. Although there are many similarities to the original file system, there are also enough changes to make file recovery extremely difficult. In the following paragraphs, and articles we will explore the arithmetic and methodology of recovering a single file from a UFS volume.

In order to recover the file we must use the on-disk data structures that give the OS its road map to the file name, inode, and finally data block placement. Normally I would start with the coarsest data structure, but in order to facilitate an understanding of the file hierarchy I will start from the smallest granularity. That data structure being the directory entry.

To illustrate the data elements and their use I chose a PDF file for recovery. The file name is “2003STEP.PDF”. Using you favorite sector editor, do a scan and search for the file name that you want to recover. The tool I use is a wonderful product called “WinHex”. Figure A is a capture of my sector search.

Figure A

There are many elements in a directory entry structure; however there are actually five key elements.

1. File Name: This is the actual name of the directory/file. In Figure A this is defined as “2003STEP.PDF”
2. File Name length: This data element is self explanatory as it defines the length of the name. In Figure A the name length is defined as 0×0C, or 12 in decimal.
3. File Type: In this case we are only concerned about two types. First 0×04 which is a directory entry and 0×08, which is a standard file name entry. In Figure A this file entry is regular file name.
4. Record length: This defines the entire length of this particular directory/file name record. In variable length records there is always a record length. In Figure A you can see the record length is 0×00000028 in hex, or 40 in decimal.
5. Inode number: The name, and name length are important, however the inode number holds the key to the data block placement. In Figure A this is defined as 0×000F7E01, or 1015297 in decimal.

Next installment I will describe the ‘cylinder group’ data structure and how we can use that to find our inode element.

SNAP Appliance, now owned by Adaptec was one of the pioneers of the Network Attached Storage (NAS) technologies. Through the use of  the Berkeley Software Distribution (BSD) and the UNIX File System (UFS), SNAP developed a reliable and easy method for using a mass storage device through a shared network.  In order to do this SNAP used an abbreviated version of the file system in conjunction with a set of hard coded variables that allowed for a fast boot up, easier recovery facilities within the spectrum of the operating system, and a ROM based web interface that was closely tied to several of the standard UNIX/Linux/BSD recovery tools. However, that being said, when it came to catastrophic recovery this particular OS/FS marriage made it virtually impossible for any third party standard file system handler, or tool, to recover lost, or deleted data. The following is an outline of one of the basic data structures, the Super Block, and how it differs from the standard UFS file system. These differences are the ‘fly in the ointment’ when it comes to using standard UFS data recovery tools. Read my article on SCO Unix RAID Data Recovery for more insight on the UFS. 

On-disk file system data structures are the key to data recovery. The knowledge of how a file system resides on the disk is the only way to recover from catastrophic data loss. Using on-disk data structures and their relationship with each other will help a recovery expert piece together lost data on a file system that will not mount. In essence, the data recovery technician creates a virtual file system using key data elements from the on-disk structure. These data elements go through a mathematical and geometrical scrutiny. This evaluation of the data must be strict enough to allow for corrupt data parsing, but flexible enough to build the file system from a partial data structure.   In other words, a sort of ‘artificial intelligence’ is used to compare, evaluate, and assign data values to key data elements through the use of file system structure placement.  A basic element of the file system in this particular case is the Super Block.

The Super Block is a broad spectrum definition of the entire file system.  Although not defining file placement, and block usage, the Super Block is the crux of on-disk data element placement that will lead the data recovery technician to file name, inode definition, and ultimately data block placement.   Data fields that reveal such values as total inodes, total data blocks, total cylinder groups, can be used to define a cohesive file system and in many cases rebuild a corrupted data structure.  The Super Block defines coarse data that can be used to calculate cylinder group definitions that inevitably lead to directory definitions, and a methodology to build a file tree.

The Super Block is defined across the disk in each cylinder group.  This fact alone can aide the trained data recovery technician in the alignment of the file system. Once aligned, it is a simple matter of back tracing directory name, inode definition, and data block in order to build a file tree. As an example the Super Block designates the primary inode block.   When parsing the first cylinder group inode 0, and 1 are undefined and the 128 byte data elements are zeroed.  However, inode 2 is defined, and can be traced to the root of the directory structure. Using recursion, one can easily define a full tree by using this single data element.

SNAP UFS File System Data Recovery

There are many more data elements that are an integral part of the SNAP UFS, however, the one basic element that is needed in order for third party UFS handlers to function is missing. Each on-disk data structure maintains an element that is unique to its particular type.  This element is defined as a ‘MAGIC NUMBER’. This magic number, however derived, is a tell tale element that can be used by the technician to find certain data structures. For whatever reason, SNAP decided to ignore the magic number and it is not stored on-disk. This may be an indication that the SNAP file system designers did not want to carry extra data elements that were superfluous to the functionality and definition of the file system. It is a good strategy for saving precious space in a ROM perhaps, but is not a sound strategy if one is trying to piece together a file system and has no idea where to start. I am not trying to second guess the SNAP Appliance designers, it is merely a fact of the on disk structure and must be dealt with.

If a software engineer wishes to design, code and implement a SNAP Appliance UFS recovery handler then the magic number must be taken into consideration. There are several other data elements of the super block structure that must have certain values. These values can be boundary tested, and used to find other data elements that have a more traditional on-disk data structure. In other words, if the super block cylinder group element points to a particular sector on the disk, that sector can be loaded and masked with a cylinder group on-disk structure. The structure can then be boundary tested and if the testing proves positive then the original super block placement may be correct. Of course, several other elements must be tested, but if the tests return in a positive manner, it is very likely that you may have found your super block without the use of a magic number.

In the final analysis it is up to the data recovery technician to evaluate each SNAP Appliance, and the possibility of recovery. However, with calculator in hand, and hex editor on screen, a well versed data recovery technician can find the super block, and in that, use that key to unlock his clients lot data.

If you have any questions pertaining to this article, or others I have wrote; please contact me at:

Dick Correa
DTI Data
Senior Systems Analyst / Senior Software Designer
dickc AT dtidata.com