Flash drive re-formatted, Testdisk sees nothing but Photorec does:am I missing something?

[Andremidis]

Greetings to all, and specially to the dev., for putting up these tools.
[edit] AND SORRY, I misclicked and posted the message too soon, completing it now.

First of all, I'll confess being the kind of "Jon Snow tech dude": knows nuthing, asks for help, get things done... often getting into trouble on the way.

So, I did find a similar post ( viewtopic.php?t=12425 ) but I'm persistent, and I suspect that I'm not doing things in the right order, simply because I have a poor understanding of File Systems and Boot sector (long list follows...). After getting back to "testdisk step by step" I got stuck on chapter 3: "running TestDisk executable". Maybe someone can help me?

For the specs: I'm running Linux Mint 21.3 Cinnamon. My problem revolves around a 2To SSD USB drive that used to be my data storage (mainly audio, movies, photos and some random stuff like software or flash games I was too sentimental to delete). There were two primary partitions on it, and I THINK (really not sure) one was NTFS and one ext4 (I used to dual boot).

I wanted to plug it into my internet "box" to access the files from my TV and use it as its hard drive when recording stuff, or torrenting (if you're from France, it's Illiad's Freebox Pop). When I first plugged it to the "player" unit, the box warned me that the drive needed formatting, and that would make it unusable by any other device and that it could not be undone blah blah. So I moved all the files to one of the partitions, and after selecting the empty one I gave the box the green light. Alas, it formatted the whole drive, and all my files were gone...
I then unplugged the drive, and made things worse by re-formatting again (with my laptop) into 2 partitions as close as I could remember to the original ones (but this time both are NTFS). I even copied a few files from my laptop to see if it still worked, but deep inside I was grieving for my memories. Now I decided to try to salvage them, and this brought me to Testdisk/Photorec.

I'be already had a run with Testdisk, a simple "next" process with quick search, then with deeper search (took forever!). All that it found were the 2 recently built partitions, with the files I've put in them. Then I Photorecced, and wow! LOTS of files found (and copied to another external drive). As expected, files are named in numbers (except for a few, mostly software component I guess, like graphics, audios and .xml or .html that may come from the flash games). "Funny" thing, some movies got chopped into small chunks, and maybe some audio got separated from videos. I got about 15 "recup_dir.N" folders, 500 to 700 files in each, so you can guess I would very much prefer if it was possible to restore the original directories...

Maybe I should have "restored file system"? or "change type"? or... maybe I needed o mount the drive first?? Or it's simply hopeless. But how can Photorec find so much and Testdisk so *nothing*?
By the way, PhotoRec and TestDisk were able to access the external drives on the go, what's lucky because I never managed to run "sudo testdisk-6.13/testdisk_static" (command not found).

Hereunder are screenshots from the process. (they come from: Mint's Disks utility, Photorec and Testdisk, of which "contents" are the files I've put in for testing purposes) The first one, in this very message, is TestDisk devices choice menu, firts step I took.I'm also adding the text from the log file.

The source (formatted) SSD is /sdd while /sdc is the other drive I've told PhotoRec to restore files to. The "MS partitions" found by TestDisk are the ones I made after my box formatted the whole thing.

Thank you for your time, and work, and sorry for the long post on such a trivial case.

Log file content:

Sat Sep 14 22:59:14 2024
Command line: TestDisk

TestDisk 7.1, Data Recovery Utility, July 2019
Christophe GRENIER <grenier@cgsecurity.org>
https://www.cgsecurity.org
OS: Linux, kernel 5.15.0-121-generic (#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024) x86_64
Compiler: GCC 11.2
ext2fs lib: 1.46.5, ntfs lib: libntfs-3g, reiserfs lib: none, ewf lib: none, curses lib: ncurses 6.3
/dev/sda: LBA, HPA, LBA48, DCO support
/dev/sda: size 234441648 sectors
/dev/sda: user_max 234441648 sectors
/dev/sda: native_max 234441648 sectors
/dev/sda: dco 234441648 sectors
/dev/sdb: LBA, HPA, LBA48 support
/dev/sdb: size 468862128 sectors
/dev/sdb: user_max 468862128 sectors
/dev/sdb: native_max 468862128 sectors
Warning: can't get size for Disk /dev/mapper/control - 0 B - 0 sectors, sector size=512
Warning: can't get size for Disk /dev/loop4 - 0 B - 0 sectors, sector size=512
Warning: can't get size for Disk /dev/loop5 - 0 B - 0 sectors, sector size=512
Warning: can't get size for Disk /dev/loop6 - 0 B - 0 sectors, sector size=512
Warning: can't get size for Disk /dev/loop7 - 0 B - 0 sectors, sector size=512
Hard disk list
Disk /dev/sda - 120 GB / 111 GiB - CHS 14593 255 63, sector size=512 - KINGSTON SM2280S3120G, S/N:50026B7262002CC1, FW:S8FM06.A
Disk /dev/sdb - 240 GB / 223 GiB - CHS 29185 255 63, sector size=512 - KINGSTON SUV400S37240G, S/N:50026B77640BE3FD, FW:0C3FD6SD
Disk /dev/sdc - 500 GB / 465 GiB - CHS 60801 255 63, sector size=512 - ST350041 8AS, FW:0041
Disk /dev/sdd - 2000 GB / 1863 GiB - CHS 243201 255 63, sector size=512 - TOSHIBA External USB 3.0, FW:5438
Disk /dev/loop0 - 109 MB / 104 MiB - 213384 sectors (RO), sector size=512
Disk /dev/loop1 - 391 MB / 373 MiB - 764904 sectors (RO), sector size=512
Disk /dev/loop2 - 109 MB / 103 MiB - 212976 sectors (RO), sector size=512
Disk /dev/loop3 - 325 MB / 310 MiB - 636528 sectors (RO), sector size=512

Partition table type (auto): EFI GPT
Disk /dev/sdd - 2000 GB / 1863 GiB - TOSHIBA External USB 3.0
Partition table type: EFI GPT

Analyse Disk /dev/sdd - 2000 GB / 1863 GiB - CHS 243201 255 63
hdr_size=92
hdr_lba_self=1
hdr_lba_alt=3907029163 (expected 3907029163)
hdr_lba_start=34
hdr_lba_end=3907029130
hdr_lba_table=2
hdr_entries=128
hdr_entsz=128
Current partition structure:
1 P MS Data 2048 1953128447 1953126400 [BlackyW]
2 P MS Data 1953128448 3907028991 1953900544 [BlackyL]

search_part()
Disk /dev/sdd - 2000 GB / 1863 GiB - CHS 243201 255 63
NTFS at 0/32/33
filesystem size 1953126400
sectors_per_cluster 8
mft_lcn 4
mftmirr_lcn 122070399
clusters_per_mft_record -10
clusters_per_index_record 1
NTFS part_offset=1048576, part_size=1000000716800, sector_size=512
NTFS partition cannot be added (part_offset<part_size).
NTFS at 0/32/33
filesystem size 1953126400
sectors_per_cluster 8
mft_lcn 4
mftmirr_lcn 122070399
clusters_per_mft_record -10
clusters_per_index_record 1
MS Data 2048 1953128447 1953126400 [BlackyW]
NTFS, blocksize=4096, 1000 GB / 931 GiB
FAT32 at 121576/158/55
FAT1 : 64-238527
FAT2 : 238528-476991
start_rootdir : 476992 root cluster : 2
Data : 476992-1953900479
sectors : 1953900522
cluster_size : 64
no_of_cluster : 30522242 (2 - 30522243)
fat_length 238464 calculated 238456

FAT32 at 121576/158/55
FAT: cluster=2(0x2), pos=1953605440
MS Data 1953128448 3907028969 1953900522 [BlackyL]
FAT32, blocksize=32768, 1000 GB / 931 GiB

Results
P MS Data 2048 1953128447 1953126400 [BlackyW]
NTFS, blocksize=4096, 1000 GB / 931 GiB
P MS Data 1953128448 3907028969 1953900522 [BlackyL]
FAT32, blocksize=32768, 1000 GB / 931 GiB

Hint for advanced users: dmsetup may be used if you prefer to avoid rewriting the partition table for the moment:
echo "0 1953126400 linear /dev/sdd 2048" | dmsetup create test0
echo "0 1953900522 linear /dev/sdd 1953128448" | dmsetup create test1
ntfs_device_testdisk_io_ioctl() unimplemented


dir_partition inode=5
P MS Data 2048 1953128447 1953126400 [BlackyW]
NTFS, blocksize=4096, 1000 GB / 931 GiB
Directory /
5 dr-xr-xr-x 0 0 0 26-May-2024 23:27 .
5 dr-xr-xr-x 0 0 0 26-May-2024 23:27 ..
67 -r--r--r-- 0 0 31474656 23-Feb-2023 00:33 vlc-record-2023-02-23-00h33m17s-VID_20230218_150416.mp4-.mp4
66 -r--r--r-- 0 0 25750653 23-Feb-2023 00:34 vlc-record-2023-02-23-00h34m07s-VID_20230218_150416.mp4-.mp4
65 -r--r--r-- 0 0 11170923 23-Feb-2023 00:35 vlc-record-2023-02-23-00h34m54s-vlc-record-2023-02-23-00h33m17s-VID_20230218_150416.mp4-.mp4-.mp4
64 -r--r--r-- 0 0 18323807 23-Feb-2023 00:38 vlc-record-2023-02-23-00h38m04s-VID_20230218_150416.mp4-.mp4


dir_partition inode=0
P MS Data 1953128448 3907028969 1953900522 [BlackyL]
FAT32, blocksize=32768, 1000 GB / 931 GiB
FAT: cluster=2(0x2), pos=1953605440
Directory /
3 -rwxr-xr-x 0 0 2487579498 19-Apr-2020 13:36 The Phantom of The Opera - FULL STAGE SHOW The Shows Must G.webm
ntfs_device_testdisk_io_ioctl() unimplemented


dir_partition inode=5
P MS Data 2048 1953128447 1953126400 [BlackyW]
NTFS, blocksize=4096, 1000 GB / 931 GiB
Directory /
5 dr-xr-xr-x 0 0 0 26-May-2024 23:27 .
5 dr-xr-xr-x 0 0 0 26-May-2024 23:27 ..
67 -r--r--r-- 0 0 31474656 23-Feb-2023 00:33 vlc-record-2023-02-23-00h33m17s-VID_20230218_150416.mp4-.mp4
66 -r--r--r-- 0 0 25750653 23-Feb-2023 00:34 vlc-record-2023-02-23-00h34m07s-VID_20230218_150416.mp4-.mp4
65 -r--r--r-- 0 0 11170923 23-Feb-2023 00:35 vlc-record-2023-02-23-00h34m54s-vlc-record-2023-02-23-00h33m17s-VID_20230218_150416.mp4-.mp4-.mp4
64 -r--r--r-- 0 0 18323807 23-Feb-2023 00:38 vlc-record-2023-02-23-00h38m04s-VID_20230218_150416.mp4-.mp4
Directory /
5 dr-xr-xr-x 0 0 0 26-May-2024 23:27 .
5 dr-xr-xr-x 0 0 0 26-May-2024 23:27 ..
67 -r--r--r-- 0 0 31474656 23-Feb-2023 00:33 vlc-record-2023-02-23-00h33m17s-VID_20230218_150416.mp4-.mp4
66 -r--r--r-- 0 0 25750653 23-Feb-2023 00:34 vlc-record-2023-02-23-00h34m07s-VID_20230218_150416.mp4-.mp4
65 -r--r--r-- 0 0 11170923 23-Feb-2023 00:35 vlc-record-2023-02-23-00h34m54s-vlc-record-2023-02-23-00h33m17s-VID_20230218_150416.mp4-.mp4-.mp4
64 -r--r--r-- 0 0 18323807 23-Feb-2023 00:38 vlc-record-2023-02-23-00h38m04s-VID_20230218_150416.mp4-.mp4
Directory /
5 dr-xr-xr-x 0 0 0 26-May-2024 23:27 .
5 dr-xr-xr-x 0 0 0 26-May-2024 23:27 ..
67 -r--r--r-- 0 0 31474656 23-Feb-2023 00:33 vlc-record-2023-02-23-00h33m17s-VID_20230218_150416.mp4-.mp4
66 -r--r--r-- 0 0 25750653 23-Feb-2023 00:34 vlc-record-2023-02-23-00h34m07s-VID_20230218_150416.mp4-.mp4
65 -r--r--r-- 0 0 11170923 23-Feb-2023 00:35 vlc-record-2023-02-23-00h34m54s-vlc-record-2023-02-23-00h33m17s-VID_20230218_150416.mp4-.mp4-.mp4
64 -r--r--r-- 0 0 18323807 23-Feb-2023 00:38 vlc-record-2023-02-23-00h38m04s-VID_20230218_150416.mp4-.mp4

interface_write()
1 P MS Data 2048 1953128447 1953126400 [BlackyW]
2 P MS Data 1953128448 3907028969 1953900522 [BlackyL]
simulate write!

Interface Advanced
hdr_size=92
hdr_lba_self=1
hdr_lba_alt=3907029163 (expected 3907029163)
hdr_lba_start=34
hdr_lba_end=3907029130
hdr_lba_table=2
hdr_entries=128
hdr_entsz=128
1 P MS Data 2048 1953128447 1953126400 [BlackyW]
NTFS, blocksize=4096, 1000 GB / 931 GiB
2 P MS Data 1953128448 3907028991 1953900544 [BlackyL]
FAT32, blocksize=32768, 1000 GB / 931 GiB
New options :
Dump : No
Align partition: Yes
Expert mode : No

TestDisk exited normally.

[recuperation]

Please upload your pictures to this site using the attachment tab on the bottom of the page and post your logfile.

Please remove all abbreviations from your text.

Thank you!

[Andremidis]

Thank you for your attention, and sorry for the abbreviations.
Here are the first 3 attachments: Log file, Mint's Disks window displaying the faulty ssd drive (unmounted), and TestDisk first partitions detection. (more to follow)

[Andremidis]

Following stages of TestDisk process: Partition choice menu, Quick Search stage, and results of said quick search. (next I'll post screenshots of PhotoRec use)

[Andremidis]

PhotoRec process screenshots (sorry, I see thet I can't really chose the order in which the attachments are displayed in my posts, but hopefully the file naming and the screenshots themselves are enough to understand what is what)

[recuperation]

Greetings to all, and specially to the dev., for putting up these tools.
[edit] AND SORRY, I misclicked and posted the message too soon, completing it now.

First of all, I'll confess being the kind of "Jon Snow tech dude": knows nuthing, asks for help, get things done... often getting into trouble on the way.

So, I did find a similar post ( viewtopic.php?t=12425 ) but I'm persistent, and I suspect that I'm not doing things in the right order, simply because I have a poor understanding of File Systems and Boot sector (long list follows...). After getting back to "testdisk step by step" I got stuck on chapter 3: "running TestDisk executable". Maybe someone can help me?

You are using an outdated version of the TestDisk package. Use version 7.2 or 7.2-WIP. Please refer to current documentation and try to be more specific in asking your question.

For the specs: I'm running Linux Mint 21.3 Cinnamon. My problem revolves around a 2To SSD USB drive that used to be my data storage (mainly audio, movies, photos and some random stuff like software or flash games I was too sentimental to delete). There were two primary partitions on it, and I THINK (really not sure) one was NTFS and one ext4 (I used to dual boot).

I wanted to plug it into my internet "box" to access the files from my TV and use it as its hard drive when recording stuff, or torrenting (if you're from France, it's Illiad's Freebox Pop). When I first plugged it to the "player" unit, the box warned me that the drive needed formatting, and that would make it unusable by any other device and that it could not be undone blah blah. So I moved all the files to one of the partitions, and after selecting the empty one I gave the box the green light. Alas, it formatted the whole drive, and all my files were gone...
I then unplugged the drive, and made things worse by re-formatting again (with my laptop) into 2 partitions as close as I could remember to the original ones (but this time both are NTFS). I even copied a few files from my laptop to see if it still worked, but deep inside I was grieving for my memories. Now I decided to try to salvage them, and this brought me to Testdisk/Photorec.

I'be already had a run with Testdisk, a simple "next" process with quick search, then with deeper search (took forever!). All that it found were the 2 recently built partitions, with the files I've put in them. Then I Photorecced, and wow! LOTS of files found (and copied to another external drive). As expected, files are named in numbers (except for a few, mostly software component I guess, like graphics, audios and .xml or .html that may come from the flash games). "Funny" thing, some movies got chopped into small chunks, and maybe some audio got separated from videos. I got about 15 "recup_dir.N" folders, 500 to 700 files in each, so you can guess I would very much prefer if it was possible to restore the original directories...

Maybe I should have "restored file system"? or "change type"? or... maybe I needed o mount the drive first??

Although you are mentioning "TestDisk step by step" you probably have not read
https://www.cgsecurity.org/wiki/TestDisk_Step_By_Step
I don't know what "o mount" should mean. I only know "mount" and "unmount".
Do not apply functions of Testdisk randomly on a logically broken disk unless you know what you do, having created a clone before.

Or it's simply hopeless.

Your own formatting efforts diminished your chances of recovery even more, the same applies to you copying stuff on your newly formatted disk.
Instead you should have cloned your disk after the fatal format operation by your box.
You destroyed lots of metadata, obviously effectively disabling TestDisk. You are now left with PhotoRec or any other third party software.

But how can Photorec find so much and Testdisk so *nothing*?

Imagine a library. If you destroy the index and all signs on the shelfs you would need to look at each book to find the book you were looking for.
But in reality it is even worse. "Books", files on a computer are not necessarily stored together but in parts. Metadata provides the information where each part (page or a couple of pages of the book) is located.
PhotoRec is looking at all pages (sectors on a HDD/SDD) trying to find book covers (fingerprints of file) assuming that the content of your books (files) is not ripped apart but rather existing in a steadily continuating way at one location.

This is why PhotoRec is the last line of defense. It works particularly well in case of storage media that is written continuously without files being deleted and their empty spaces being refilled.

By the way, PhotoRec and TestDisk were able to access the external drives on the go, what's lucky because I never managed to run "sudo testdisk-6.13/testdisk_static" (command not found).

The command is referring to an old version of TestDisk. Please describe where you found that command in the manual or the online documentation!
You basically have to run the statically compiled version with admin rights.
"testdisk-6.13" is an unpacked folder of TestDisk containing the "static" version which can run in a standalone manner, not needing external libraries.

But this procedure is not necessary with Linuxmint (that I use occasionally). If you key in "TestDisk"? or "testdisk" in a command shell Linuxmint tells you that this software is not installed and proposes to install it for you which works if you have an internet connection.
When dealing with an USB pen drives the TestDisk installation gets lost upon restart.

Hereunder are screenshots from the process. (they come from: Mint's Disks utility, Photorec and Testdisk, of which "contents" are the files I've put in for testing purposes) The first one, in this very message, is TestDisk devices choice menu, firts step I took.I'm also adding the text from the log file.

The source (formatted) SSD is /sdd while /sdc is the other drive I've told PhotoRec to restore files to. The "MS partitions" found by TestDisk are the ones I made after my box formatted the whole thing.

Thank you for your time, and work, and sorry for the long post on such a trivial case.

Your case is not trivial and chances have been lowered by your former interventions. You are left with what PhotoRec extracted for you but third party software might do better. You have to try that out if you want to know.

[Andremidis]

Well, thank you very much for this comprehensive and extensive answer.
I guessed that I had dug my own grave when I re-formatted, but after the Freebox formatting the disk could not be read nor mounted, at least by the means within my reach (and I was a bit shocked).
Now to clarify a few points you mentioned:

I don't know what "o mount" should mean. I only know "mount" and "unmount".

That is a typo, it was meant to read: "maybe I needed TO mount the drive first??"
(since my last upgrade of Mint the function "disable touchpad while typing" doesn't work anymore, so it's a real pain, my entry point keeps jumping around the text as I type on the keyboard)

outdated version of the TestDisk package. Use version 7.2 or 7.2-WIP

I added TestDisk from Mint's software manager. I will try again with the newest version, for the sake of trying.

Although you are mentioning "TestDisk step by step" you probably have not read
https://www.cgsecurity.org/wiki/TestDisk_Step_By_Step

This is exactly the page I was referring to. Adn it is precisely on that page (third bullet down on the chapter "Running TestDisk executable" that the command

Code: Select all

sudo testdisk-6.13/testdisk_static

is mentioned "in extenso". My bad, I had not understood that the bullets were all relative to the line "To recover a lost partition or repair the filesystem from a hard disk, USB key, Smart Card, etc., you need enough rights to access a physical device" AND that it meant "running TestDisk from [a hard disk, USB key, Smart Card, etc...]

Do not apply functions of Testdisk randomly on a logically broken disk unless you know what you do, having created a clone before.

I consider that I did not apply anything randomly. On the first pass I followed a simpler tutorial found elsewhere, that greatly praised the comprehensiveness of TestDisk, and advised to trust the File System detection/identifications. Sadly I did not - and still don't! - have any 2To volume to clone my disk to (it's not full, but knowing that data is not continuous I guess I'd need an equal space to clone it). After the first TestDisk try (when I just kept forwarding what seemed a normal procedure of identification and search), and after PhotoRec found those 6k files, I went back to the "step-by-step" page, and realized that maybe I should have applied some other functionality prior to the search, and that brought me to this board, because on the chapter "Running TestDisk executable" there was this intriguing suggestion,

To repair a filesystem not listed by TestDisk, run testdisk device

, but I wouldn't do something that I do not understand enough.

Contrary to the "carrot" member, I am donating to the project, because this is what I believe "informatique" is for: going forth, searching for solutions, helping and getting help. You have all my respect for your developer policy

Thanks again
(I'm assuming you wouldn't hint on what other software to try, otherwise it would already be in the boards)

[recuperation]

Please provide me with the date and time of your donation and your time zone in a private message.

Thank you!

[recuperation]

Andremidis,

thank you for getting back to me!

I was asking you this question because I often read phrases like:

"I want to make a contribution!"
"I will make a contribution!"
"If you help me I might consider donating"

You said

"Contrary to the "carrot" member, I am donating to the project, because this is what I believe "informatique" is for: going forth, searching for solutions, helping and getting help. You have all my respect for your developer policy
"

I encourage donations - I do not receive a share and don't want to.
I am happy advocating and encouraging donations.

With you referring to the "carrot" member I simply would have preferred reading "I have donated" instead of writing "I am donating", because the donation process might have been interrupted, who knows?

In other words - if you want to mention MADE/PERFORMED donations that's fine, I don't mind you mentionning it, but talking about donations that have not been wired is making me angry.

The best thing to get a response from me is a clear case description, explaining and documenting what you did what you had expected and where you failed. In other cases I don't mind answering general questions regarding the TestDisk package, but referring to which documents you read and which keywords you used shows me that you made an effort yourself.

Best regards
recuperation

[cgrenier]

A little post to thank you for your donation

When running TestDisk, try TestDisk, Advanced, select the "BackyL" partition, List. If you see your files (with a non-empty size), you should be able to copy them.
You can also try "Scrounge NTFS" to recover files from your NTFS partition but it doesn't always work after reformating, so PhotoRec may remain the "best" solution.
You will recover more files when using PhotoRec on each partition (the first partition, next the second one) instead of the whole disk (ext4 uses indirect blocks, double indirect blocks that need to be handled)

TestDisk package for Debian, Ubuntu and Mint is still the old 7.1 but version 7.2 has landed in Debian testing September 9, so it will eventually appear in your Linux distribution.

To everyone reading the message, feel free to provide updated text for https://www.cgsecurity.org/testdisk_doc/ It's not always easy to provide information that can be understood by both native-English speaker and non-English readers, by people with varied technical skills.