Request of exctraction of range sector and optimization to scan device and recover files

[Dioxide]

Hi, I'm new on this forum; sadly I have got a problem (I think it's related to motherboard age considering that laptop have 16 years) and while I was copying files from my new notebook to old notebook on a usb pendrive of 512 GB with realvnc, the notebook (or the realvnc) crashes, I don't know how but usb pendrive cannot be read and windows suggest me to repair with check disk. This was not enough, so before to use testdisk I have exctracted files with Tenorshare 4DDiG. Now some files are exctract correctly, other not. So the first question is:
1. it's possible add a feature both to photorec that qphotorec that, reading all that is possible on NTFS or exfat file table, can list all the files with path and near to filename show the sector ranges that contains the file and exctract that to a file txt ? In this case, if a file is stored in contiguos sectors (so one sector range), I can be sure that isn't damaged.
The second question is the following: if I have made a previous backup of some files, it's possible make a table with sha512 code of all sectors of pendrive and, give in input to testdisk files that I'm sure that integrity check is true, split the file in part of size of file table cluster, made a check of contiguos sector (verified by contiguos sha512 codes) with sha512 codes calculated from file? In such case, with canvas recovery, it's useful to exclude sector from scan that are knowed contains data that are already recovered and reduce the probability of recover damaged files (moreover exclude sector that contain file more smaller of their size, exclude sectors that can be read from ntfs/exfat table).

Hope that I have explained clearly my idea

[recuperation]

Hi, I'm new on this forum; sadly I have got a problem (I think it's related to motherboard age considering that laptop have 16 years) and while I was copying files from my new notebook to old notebook on a usb pendrive of 512 GB with realvnc, the notebook (or the realvnc) crashes, I don't know how but usb pendrive cannot be read and windows suggest me to repair with check disk. This was not enough, so before to use testdisk I have exctracted files with Tenorshare 4DDiG. Now some files are exctract correctly, other not. So the first question is:
1. it's possible add a feature both to photorec that qphotorec that, reading all that is possible on NTFS or exfat file table, can list all the files with path and near to filename show the sector ranges that contains the file and exctract that to a file txt ? In this case, if a file is stored in contiguos sectors (so one sector range), I can be sure that isn't damaged.

PhotoRec is the last line of defense. It is to be used when metadata is lost or dammaged. Therefore metadata like file name, folder paths are not available. PhotoRec is useless in your scenario. If metadata exists you can use TestDisk to browse the partition and select files and folders to be recovered.

The second question is the following: if I have made a previous backup of some files, it's possible make a table with sha512 code of all sectors of pendrive and, give in input to testdisk files that I'm sure that integrity check is true, split the file in part of size of file table cluster, made a check of contiguos sector (verified by contiguos sha512 codes) with sha512 codes calculated from file? In such case, with canvas recovery, it's useful to exclude sector from scan that are knowed contains data that are already recovered and reduce the probability of recover damaged files (moreover exclude sector that contain file more smaller of their size, exclude sectors that can be read from ntfs/exfat table).

Hope that I have explained clearly my idea

Unfortunately no. Typically if I don't understand a question upon the first time I read it, chances are big that rereading the text a couple of times does not provide more insight. This applies to your second question. Think about formatting your text in sections and avoid run-on sentences.

I have a vague idea what you mean here:

it's useful to exclude sector from scan that are knowed contains data that are already recovered

There is no double assignment of clusters to recovered files.

PhotoRec does not only recover files. It provides you with file location information. Please read:

https://www.cgsecurity.org/testdisk_doc/photorec.html

Photorec generates a file named "report.xml". That contains a wealth of information you are probably looking for.
In subsection 11.12 there is a description how to read it.

[Dioxide]

PhotoRec is the last line of defense. It is to be used when metadata is lost or dammaged. Therefore metadata like file name, folder paths are not available. PhotoRec is useless in your scenario. If metadata exists you can use TestDisk to browse the partition and select files and folders to be recovered.

in my case a part of metadata exist, another part is lost and are viewed strange characters. So testdisk doesn't allow to export report such photorec? If testdisk can read all the chain of segments of a file, I can be sure that file recovered from 4ddig (but also from photorec) isn't damaged (because some files opened from 4ddig but also from photorec are damaged). Moreover

Unfortunately no. Typically if I don't understand a question upon the first time I read it, chances are big that rereading the text a couple of times does not provide more insight. This applies to your second question. Think about formatting your text in sections and avoid run-on sentences.

ok, so I make an example: if I copy and past 3 file from 500 MB on the usb empty, delete second file and copy a file of 1000 MB that start to write from first megabyte where was stored second file, the first 500 MB on the middle are written and the last 500 MB are written after the end of third file. If I have a backup of the third file, make a matching on the pendrive I know the third file in what sectors is written, so I can exclude that sectors and recover the file of 1000 MB.


Another example: I have deleted a file of 3.25 GB that keep maven cache, I have copied another file of 950 MB on that sectors and the file recovered is a file of 3.25 gb with the content of the file of 950 MB (this file I have made backup). So analyzing sectors and making a list of sha512 checksums of each sectors, and split the file(of the size of clusters of pendrive) that I have already made a backup and make a list of sha512 checksums, comparing sectors checksums (almost 5 should be contiguos), I can exclude the sectors that I already know that I have recovered the content, so canvas and bruteforce combinations can be made on a reduced number of clusters.

why photorec doesn't access to metadata to try of retrieve fie path such as test disk?

PhotoRec does not only recover files. It provides you with file location information. Please read:

https://www.cgsecurity.org/testdisk_doc/photorec.html

Photorec generates a file named "report.xml". That contains a wealth of information you are probably looking for.
In subsection 11.12 there is a description how to read it.

I have read section 11.12, in file object tag are written all the segments that compound a file, or only the first?

[recuperation]

PhotoRec is the last line of defense. It is to be used when metadata is lost or dammaged. Therefore metadata like file name, folder paths are not available. PhotoRec is useless in your scenario. If metadata exists you can use TestDisk to browse the partition and select files and folders to be recovered.

in my case a part of metadata exist, another part is lost and are viewed strange characters.

In this case TestDisk will recover only the files whose metadata lies in the readable part of the total metadata. If you want to recover a half broken folder I can't tell you what that gives once TestDisk reads the broken directory entries.
These are cases where you should try other recovery software.

So testdisk doesn't allow to export report such photorec?

If testdisk can read all the chain of segments of a file, I can be sure that file recovered from 4ddig (but also from photorec) isn't damaged (because some files opened from 4ddig but also from photorec are damaged). Moreover

If metadata is OK, there is no reason to question any other software.

Unfortunately no. Typically if I don't understand a question upon the first time I read it, chances are big that rereading the text a couple of times does not provide more insight. This applies to your second question. Think about formatting your text in sections and avoid run-on sentences.

ok, so I make an example: if I copy and past 3 file from 500 MB on the usb empty, delete second file and copy a file of 1000 MB that start to write from first megabyte where was stored second file, the first 500 MB on the middle are written and the last 500 MB are written after the end of third file. If I have a backup of the third file, make a matching on the pendrive I know the third file in what sectors is written, so I can exclude that sectors and recover the file of 1000 MB.

At least I understand that example. There is no guarantee that the filesystem places your file with a size of 1000 MB partly into the 500 MB hole created by the deletion of the second file and the other 500 MB into the free space following after the end of the third file unfortunately. These are your assumptions, you have to find out.

Another example: I have deleted a file of 3.25 GB that keep maven cache, I have copied another file of 950 MB on that sectors and the file recovered is a file of 3.25 gb with the content of the file of 950 MB (this file I have made backup).

Not every file type has an end marker. This is the reason why PhotoRec might recover overly lengthy files because it stops recovering once it finds a new fingerprint. This is the general rule. I guess Christophe Grenier has taken end markers into account somehow.
Your example makes perfectly sense to me. PhotoRec behaves as intended.

So analyzing sectors and making a list of sha512 checksums of each sectors, and split the file(of the size of clusters of pendrive) that I have already made a backup and make a list of sha512 checksums, comparing sectors checksums (almost 5 should be contiguos), I can exclude the sectors that I already know that I have recovered the content, so canvas and bruteforce combinations can be made on a reduced number of clusters.

I don't understand you. "Canvas" ???
If partition space is allocated on cluster level (which is a multiple of type 2^n of the sector size there is no need to create checksums on sectors.

why photorec doesn't access to metadata to try of retrieve fie path such as test disk?

Because PhotoRec is the last line of defense to be used when metadata is lost. This is the key assumption. Furthermore the fingerprinting method is generic and does not require knowing the metdata structure of any file system. The only other assumption is that there is a tendancy that data will be saved to storage in a continuous linear way if possible.

PhotoRec does not only recover files. It provides you with file location information. Please read:

https://www.cgsecurity.org/testdisk_doc/photorec.html

Photorec generates a file named "report.xml". That contains a wealth of information you are probably looking for.
In subsection 11.12 there is a description how to read it.

I have read section 11.12, in file object tag are written all the segments that compound a file, or only the first?

See the explanation above. I guess you will only see the first part of a fragmented file. There might be a better result when files contain end markers and file length information in a internal! file header. File fragmentation is the natural enemy of all fingerprint readers.

I think I am guessing now what you want:

Based on a given set of healthy files and their known location on a broken file system you want PhotoRec to exclude dealing with the clusters that belong to those healthy files.

[Dioxide]

PhotoRec is the last line of defense. It is to be used when metadata is lost or dammaged. Therefore metadata like file name, folder paths are not available. PhotoRec is useless in your scenario. If metadata exists you can use TestDisk to browse the partition and select files and folders to be recovered.

in my case a part of metadata exist, another part is lost and are viewed strange characters.

In this case TestDisk will recover only the files whose metadata lies in the readable part of the total metadata. If you want to recover a half broken folder I can't tell you what that gives once TestDisk reads the broken directory entries.
These are cases where you should try other recovery software.

ok, so You confirm me that for example if testDisk find and list 10 files of a directory and I extract all 10 files, I can be sure that all the 10 files are intact such it was when are stored and I can exclude the risk that testdisk extract only a part of original file size breaking final segments or skipping intermediate segments. Right?

So testdisk doesn't allow to export report such photorec?

If testdisk can read all the chain of segments of a file, I can be sure that file recovered from 4ddig (but also from photorec) isn't damaged (because some files opened from 4ddig but also from photorec are damaged). Moreover

If metadata is OK, there is no reason to question any other software.

ok

Unfortunately no. Typically if I don't understand a question upon the first time I read it, chances are big that rereading the text a couple of times does not provide more insight. This applies to your second question. Think about formatting your text in sections and avoid run-on sentences.

ok, so I make an example: if I copy and past 3 file from 500 MB on the usb empty, delete second file and copy a file of 1000 MB that start to write from first megabyte where was stored second file, the first 500 MB on the middle are written and the last 500 MB are written after the end of third file. If I have a backup of the third file, make a matching on the pendrive I know the third file in what sectors is written, so I can exclude that sectors and recover the file of 1000 MB.

At least I understand that example. There is no guarantee that the filesystem places your file with a size of 1000 MB partly into the 500 MB hole created by the deletion of the second file and the other 500 MB into the free space following after the end of the third file unfortunately. These are your assumptions, you have to find out.

Yes I know, but if my scenario happens, You can be sure that third file end before the start of second part of fourth files or continue after the end of the fourth files.

Another example: I have deleted a file of 3.25 GB that keep maven cache, I have copied another file of 950 MB on that sectors and the file recovered is a file of 3.25 gb with the content of the file of 950 MB (this file I have made backup).

Not every file type has an end marker. This is the reason why PhotoRec might recover overly lengthy files because it stops recovering once it finds a new fingerprint. This is the general rule. I guess Christophe Grenier has taken end markers into account somehow.
Your example makes perfectly sense to me. PhotoRec behaves as intended.

ok, and for this reason is useful my idea about sha512 of each sector: with comparison of sha512 code between the code calculated from sector and and the code calculated from split of file backupped, knowing that file backupped is intact, I know in which sector of usb pendrive file stop. So subsequent sector can contain fragments of previous file that photorec, doesn't knowing their presence, have ignored and not used to reassemle previous file.

So analyzing sectors and making a list of sha512 checksums of each sectors, and split the file(of the size of clusters of pendrive) that I have already made a backup and make a list of sha512 checksums, comparing sectors checksums (almost 5 should be contiguos), I can exclude the sectors that I already know that I have recovered the content, so canvas and bruteforce combinations can be made on a reduced number of clusters.

I don't understand you. "Canvas" ???
If partition space is allocated on cluster level (which is a multiple of type 2^n of the sector size there is no need to create checksums on sectors.

ok, I'm not skilled about the fondamentals of file system, but idea is exclude from scan of photorec sector that I know belong to files already recovered to reduce the sectors to scan and avoid false "assumption" of photorec

why photorec doesn't access to metadata to try of retrieve fie path such as test disk?

Because PhotoRec is the last line of defense to be used when metadata is lost. This is the key assumption. Furthermore the fingerprinting method is generic and does not require knowing the metdata structure of any file system. The only other assumption is that there is a tendancy that data will be saved to storage in a continuous linear way if possible.

ok

PhotoRec does not only recover files. It provides you with file location information. Please read:

https://www.cgsecurity.org/testdisk_doc/photorec.html

Photorec generates a file named "report.xml". That contains a wealth of information you are probably looking for.
In subsection 11.12 there is a description how to read it.

I have read section 11.12, in file object tag are written all the segments that compound a file, or only the first?

See the explanation above. I guess you will only see the first part of a fragmented file. There might be a better result when files contain end markers and file length information in a internal! file header. File fragmentation is the natural enemy of all fingerprint readers.

I think I am guessing now what you want:

Based on a given set of healthy files and their known location on a broken file system you want PhotoRec to exclude dealing with the clusters that belong to those healthy files.

yes, my idea is:
1. extract all files with testdisk (that You can confirm me that I can be sure about intact);
2. store the sectors intervals of these files in a file map;
3. give all the previous healthy files that I have made a backup and find where they are stored excluding from scan the clusters in the file map 2;
4. add the sector intervals of this healthy files on file of step 2;
5. scan photorec the following 2 type of files: 1. header that contain the lenght; 2 header that is knowed end marker of file smaller than cluster size of file system excluding from scan the clusters in the file map updated on point 4;
6. add the sectors of this files on file updated at step 4;
7. make the scan of photorec excluding all the sectors of the file updated, and extract all the unknowed sectors and flag these with end-marker excluding from scan the clusters in the file map updated on point 6 , in that case the content can be keeped only on previous sectors, and last hope is try bruteforce combination and give it to parser of related file extension to check if are compliant.
With file rar without password it's hard but can be done, with other files not.

p.s.: some file rar or zip was protected with password, do You know if it's possible scan raw data until an approximated cluster taht could keep file system to extract string near to "pass", " psw", "ps", "password"?