NTFS MFTs (both!) gone - Recovery possible?

[geohei]

Hi.

The disk:
- External using USB bay
- Size 300 GB
- 1 NTFS partition (no hidden partition)
- Encrypted with TC back in 2008
- Password and keyfile are present - partition mounts correctly
- I did a dd image to experiment

The problem:
After mounting. the partition, I see checked the 2 MFTs
0x0000000030 : 0x00000C000000000049F52E0200000000
MFT : 0x00C0000000 : first 16 sectors are corrupt
MFT Mirror : 0x22ef549000 : first 8 sectors are corrupt
After the corrupted sectors, I see FILE0 entries (correct MFT data) at the 0x1000 boundaries.

Trying to access the mounted partition via Windows Explorer, I get:
"The disk structure is corrupted and unreadable."

Windows Event Log confirms the trouble:
The Master File Table (MFT) contains a corrupted file record. The file reference number is 0x1000000000000. The name of the file is "<unable to determine file name>".

How it probably happened:
After initially putting the disk (after years) into the USB bay (TC mount as r/w), I was able to verify all (!) stored files since I have an md5sum file of the entire disk. No, I didn't read/copy the data, just read/verified the data (#*%$§& !!!). But then, I probably (not sure ?!) didn't TC dismount properly, or (?!) I ejected the disk prior TC dismount. A subsequent mount showed the NTFS errors above.

What I then did:
Since could not rely on MFT data, I wrote some C code to check for the cluster starts (0x1000 boundaries). I only have 3 filetypes. .iso, .gz and .tgz on the disk. Scanning the entire disk, I managed to recover >50%. Data integrity could be guaranteed due md5 checksums. >50% is already very good, but I need all (important data)! I don't give up until all possibilities are exhausted.

What I believe:
I think that only the MFT is corrupted. Some files however show bad md5s, which is confirmed for .tgz files by "tar -tzf <file> > /dev/null". This could be due fragmentation since my manual restoration binary assumes contiguous sectors.

Questions:
1. What I really don't get is, how the beginning of both MFTs could become corrupted, while the data payload seems intact. Any ideas?
2. Is there a possibility to scan the entire disk in order to "rebuild/repair" the MFT? If there is no tool, I can code it myself, but first I need to know by principle whether it's feasible or not.
3. Some of the files seem to be non-contiguous. Is the MFT the only place on the partition where details about file fragments are stored?

Notes:
- If I say "corrupt", I don't know if the data I see is trash or encrypted.
- If I say "TrueCrypt" (TC), it might be VeraCrypt (CV) - not relevant for this problem.

TestDisk didn't find any data.

I'm pretty stuck. Any help would be greatly appreciated.

[recuperation]

Hi.

The disk:
- External using USB bay
- Size 300 GB
- 1 NTFS partition (no hidden partition)
- Encrypted with TC back in 2008
- Password and keyfile are present - partition mounts correctly
- I did a dd image to experiment

The problem:
After mounting. the partition, I see checked the 2 MFTs
0x0000000030 : 0x00000C000000000049F52E0200000000
MFT : 0x00C0000000 : first 16 sectors are corrupt
MFT Mirror : 0x22ef549000 : first 8 sectors are corrupt


After the corrupted sectors, I see FILE0 entries (correct MFT data) at the 0x1000 boundaries.

Trying to access the mounted partition via Windows Explorer, I get:
"The disk structure is corrupted and unreadable."

Windows Event Log confirms the trouble:
The Master File Table (MFT) contains a corrupted file record. The file reference number is 0x1000000000000. The name of the file is "<unable to determine file name>".

Sometimes running chkdsk solves the problem but as the chkdsk action is not reversible I would try that on a copy.

How it probably happened:
After initially putting the disk (after years) into the USB bay (TC mount as r/w), I was able to verify all (!) stored files since I have an md5sum file of the entire disk. No, I didn't read/copy the data, just read/verified the data (#*%$§& !!!). But then, I probably (not sure ?!) didn't TC dismount properly, or (?!) I ejected the disk prior TC dismount. A subsequent mount showed the NTFS errors above.

What I then did:
Since could not rely on MFT data, I wrote some C code to check for the cluster starts (0x1000 boundaries). I only have 3 filetypes. .iso, .gz and .tgz on the disk. Scanning the entire disk, I managed to recover >50%. Data integrity could be guaranteed due md5 checksums. >50% is already very good, but I need all (important data)! I don't give up until all possibilities are exhausted.

What I believe:
I think that only the MFT is corrupted. Some files however show bad md5s, which is confirmed for .tgz files by "tar -tzf <file> > /dev/null". This could be due fragmentation since my manual restoration binary assumes contiguous sectors.

Questions:
1. What I really don't get is, how the beginning of both MFTs could become corrupted, while the data payload seems intact. Any ideas?

There is no MFT duplicate. There is only a duplicate of some MFT entries.

2. Is there a possibility to scan the entire disk in order to "rebuild/repair" the MFT? If there is no tool, I can code it myself, but first I need to know by principle whether it's feasible or not.

Every serious recovery program has to do that but they only recover. Testdisk is the only exception but the repair handles stuff where the chance of doing something wrong is low. When trying to "rebuild" or "repair" you would need space that is not in use by something else. If a file is fragmented I doubt you can rebuild that. You'd be puzzling with an enourmous number of pieces.

3. Some of the files seem to be non-contiguous. Is the MFT the only place on the partition where details about file fragments are stored?

There could be some information in journals. I remember a test in my preferred computer journal when they discovered one recovery program beating all the rest.

Notes:
- If I say "corrupt", I don't know if the data I see is trash or encrypted.

That will be rather trash. The operating system has no way of determing if there is some encrypted data.
If it does not fit the expected structure it becomes trash.

TestDisk didn't find any data.

Runing photorec on the decrypted volume should reproduce your results. In your case it would be smart to only check the families that contain the file types you used.

[geohei]

Hi.

Thanks for the answers.

chkdsk failed after seconds! No way to get any info/data get from the drive using that tool. This sounds logical to me since it takes its info from one of the MFTs, which are both gone.

However meanwhile, I got the data back. All data, that is 300 GB! Except 5 files <1500 Bytes, which were stored inside the MFT. But I only managed this due to the fact, that 90% of the files were written contiguously. Only 10% was fragmented (any only 2 fragments). Also, all files were backup archives (.gz and .tgz). So with magic numbers, I could find the start of the files. I also managed to find the end (.. 00 .. .. ..). All this required some self made (C) coding tool to get all data back mostly automatic. Since I had md5 checksums of the entire drive, I could verify 100% integrity.

I didn't touch journals, but this might have been another option. Thanks!

[recuperation]

So you basically reprogrammed a short version of Photorec?
Did you use your list of file names and md5 checksums to assign file names to the data found?

[geohei]

I don't really know how Photorec works, but I believe it relies on the MFT (not sure, but I think I tested Photorec as well - without successful recoveries).

What I did - I checked the beginning of each cluster (4KB size) and looked for the the magic bytes for .gz, .tgz and .iso files (the only file types stored on that disk). Then I looked for the "00 .. .. .." (4th last byte of the file), followed by "00" to the end of the sector (!= cluster). I extracted the the content and calculated the md5, which I could verify with the previously saved md5s which were off-disk (stored on a different device).

For the fragments, I found some file starts without end (before the next start), and some ends without start. There were only a dozen of these. I shuffled them one by one and found like this the matching parts. Unfortunately I didn't know file sizes, which would have made shuffling obsolete.

I still don't understand which manipulation could lead to the complete corrupted MFTs, while all data remained valid. Mystery!

[recuperation]

Defragmentation during restart maybe.