Feature request: limit file size or skip file Topic is solved

[FransM]

I started recovery on a disk that had ransomware on it. The disk has a btrfs filesystem on it and apparently the ransomware created a large zip file (maybe a bogus one).

When running qphotorec on this disk it very quickly complained about a file that could not be stored on the rescue drive (an exfat formatted 5TB drive), and it prompted for another drive.

I suspected this was due to the zip drive so I excluded .zip and recovered quite some files (thank you Christophe and all who made this program possible!).

It would be nice if, on the message that the file could not be stored I would be given some more info (e.g. on why not, file size and type and maybe other data that could be recovered) and that I was given the option to skip the file.
Alternately it would be nice if one e.g. could tune the files to be restored. (E.g. do not restore files > 4 GB or maybe if you are trying to restore photo's do not resture files < 1 MB)

[FransM]

I was trying to recover whatever possible from an 8 TB harddisk that was locked by ransomware.
I used qphotorec but unfortunately recovery stalled at 23 %. The program was just hanging and not reading data any more.
This is the status
. It was like this for the last 12 hrs.(image will be autoremoved after a week).

Eventually I killed qphotorec and wanted to resume. However qphotorec does not seem to have a way to say you want to resume.
This was with the version from git head from yesterday. I do have a .ses file.
I then tried photorec from the command line but that seemed to start way before where I left off. That is: the first file it recovered was fully identical to the one it already recovered (including the file name)

No idea what is going on here, I could use some help/advise.
I'm planning to restart from scratch but now using photorec instead of qphotorec.

(oh and thanks for the nice tool, I already recovered quite some useful data)

edit, not sure if the image will show up properly but the link is https://ibb.co/kB1J62X

[recuperation]

I started recovery on a disk that had ransomware on it. The disk has a btrfs filesystem on it and apparently the ransomware created a large zip file (maybe a bogus one).

When running qphotorec on this disk it very quickly complained about a file that could not be stored on the rescue drive (an exfat formatted 5TB drive), and it prompted for another drive.

And your source disk was 24TB?

I suspected this was due to the zip drive so I excluded .zip and recovered quite some files (thank you Christophe and all who made this program possible!).

It would be nice if, on the message that the file could not be stored I would be given some more info (e.g. on why not,

Unfortunately you did not post a screenshot of that moment when Qphotorec asked for another drive.

file size and type and maybe other data that could be recovered) and that I was given the option to skip the file.

A request after each file maybe?

Alternately it would be nice if one e.g. could tune the files to be restored. (E.g. do not restore files > 4 GB or maybe if you are trying to restore photo's do not resture files < 1 MB)

What is "resture"?

I am sorry to say that I don't see a chance to promote your ideas which are all supposed to circumvent the problem that are resulting from a lack of free recovery space.

You can interrupt Photorec, delete files that you don't want or move them to another drive. You can combine disks to represent one big single volume.

[recuperation]

I was trying to recover whatever possible from an 8 TB harddisk that was locked by ransomware.

I merged your second topic with the first one.

Now we know why it did not work out well: You are trying to recover a 8TB disk to a 5TB disk.
Unfortunately you did not mention the size of the source and the target in one sentence. Astounding!

I used qphotorec but unfortunately recovery stalled at 23 %. The program was just hanging and not reading data any more.
This is the status
.

Please upload your picture to this site.

It was like this for the last 12 hrs.(image will be autoremoved after a week).

You are not supposed to destroy case information so that others cannot learn from your case.

Eventually I killed qphotorec and wanted to resume. However qphotorec does not seem to have a way to say you want to resume.
This was with the version from git head from yesterday. I do have a .ses file.
I then tried photorec from the command line but that seemed to start way before where I left off. That is: the first file it recovered was fully identical to the one it already recovered (including the file name)

No idea what is going on here, I could use some help/advise.
I'm planning to restart from scratch but now using photorec instead of qphotorec.

(oh and thanks for the nice tool, I already recovered quite some useful data)

edit, not sure if the image will show up properly but the link is https://ibb.co/kB1J62X

Please upload your picture to this site.

[FransM]

I started recovery on a disk that had ransomware on it. The disk has a btrfs filesystem on it and apparently the ransomware created a large zip file (maybe a bogus one).

When running qphotorec on this disk it very quickly complained about a file that could not be stored on the rescue drive (an exfat formatted 5TB drive), and it prompted for another drive.

And your source disk was 24TB?

Source disk was 8 TB; actually it is one of the two disks of a RAID-1 volume. It comes from a nas that is infected with ransomware.
It seems the ransomware created a large zip file on the disk (not fully sure; I just got the disk with the question "can you still recover files from it".

I suspected this was due to the zip drive so I excluded .zip and recovered quite some files (thank you Christophe and all who made this program possible!).

It would be nice if, on the message that the file could not be stored I would be given some more info (e.g. on why not,

Unfortunately you did not post a screenshot of that moment when Qphotorec asked for another drive.

Sorry, I didn't make one. It was my very first attemt at using photorec

file size and type and maybe other data that could be recovered) and that I was given the option to skip the file.

A request after each file maybe?

Alternately it would be nice if one e.g. could tune the files to be restored. (E.g. do not restore files > 4 GB or maybe if you are trying to restore photo's do not resture files < 1 MB)

What is "resture"?

I am sorry to say that I don't see a chance to promote your ideas which are all supposed to circumvent the problem that are resulting from a lack of free recovery space.

You can interrupt Photorec, delete files that you don't want or move them to another drive. You can combine disks to represent one big single volume.

"resture" is a typo, I meant "restore" (or if you prefer "recover".

In my case there was substantial free space, source disk being 8 TB, rescue disk being empty and 5TB.
Could be because the source disk is btrfs inside a raid-1 volume.
It could also be that the ransomware created a very large file.
and lastly this could be because the recovery disk was ntfs, not btrfs.

I'll see if I can reproduce it, but it would be nice if the prompt for space would tell what file it wanted to restore and how big it is.
And then an option to skip recovering that file and to move on could be handy.

For the size option I can imagine the following use case:
I want to recover all photo's from a disk, but not thumbnails so I only want files > 1 MB
Or maybe I want to exclude large files (because I e.g. am nit interested in restoring large video files).

I know you can delete them and continue, but having this as an option would speed up the recovery as the data does not need to be written.
(just reading all the data of this 8TB disk takes at least 12.5 hrs, writing it to a hd could take another 12.5 hours, so if some large files could be skipped upfront it would be a time-saver).
But I agree that it is possible to do so with the program in its current form.

[FransM]

I was trying to recover whatever possible from an 8 TB harddisk that was locked by ransomware.

I merged your second topic with the first one.

Now we know why it did not work out well: You are trying to recover a 8TB disk to a 5TB disk.
Unfortunately you did not mention the size of the source and the target in one sentence. Astounding!

I don't think this is the case. The source disk was not full so I expected to have enough space.
Also when gphotorec hangs it is at 23% of the source disk. So close to 2TB has been processed. That fitted quite well on the rescue disk. Actually the rescue disk is only filled for 1.3 TB.

I used qphotorec but unfortunately recovery stalled at 23 %. The program was just hanging and not reading data any more.
This is the status
.

Please upload your picture to this site.

It was like this for the last 12 hrs.(image will be autoremoved after a week).

You are not supposed to destroy case information so that others cannot learn from your case.

Eventually I killed qphotorec and wanted to resume. However qphotorec does not seem to have a way to say you want to resume.
This was with the version from git head from yesterday. I do have a .ses file.
I then tried photorec from the command line but that seemed to start way before where I left off. That is: the first file it recovered was fully identical to the one it already recovered (including the file name)

No idea what is going on here, I could use some help/advise.
I'm planning to restart from scratch but now using photorec instead of qphotorec.

(oh and thanks for the nice tool, I already recovered quite some useful data)

edit, not sure if the image will show up properly but the link is https://ibb.co/kB1J62X

Please upload your picture to this site.

Attached is the screenshot.
I referred to an external place because either as a new member I was not allowed to post attachments, or because I overlooked how to do it.

I have also added my .sses file (had to add .txt to it as .ses is not allowed for attachments)

(oh and /mnt is where I mounted the usb disk I am recovering to)

[cgrenier]

When qphotorec was hanging, was it eating 100% of the CPU ? Was there I/O errors (See "dmesg" output, /var/log/messages, journalctl output...) ?
Please tell us if photorec has been able to process more than 24% of the disk.

[FransM]

When it was hanging it indeed used up 100% of the CPU.
dmesg gives nothing around the time of failure.
kern.log is same info as dmesg
syslog has messages about metadata extraction, not sure if that is related. These messages seem to stop around the time the last data was received.
This is a snippet

Dec 19 03:17:21 frans-desktop dbus-daemon[1391]: [session uid=1000 pid=1391] Activating via systemd: service name='org.freedesktop.Tracker1.Miner.Extract' unit='tracker-extract.service' requested by ':1.1
' (uid=1000 pid=1378 comm="/usr/libexec/tracker-miner-fs " label="unconfined")
Dec 19 03:17:21 frans-desktop systemd[1370]: Starting Tracker metadata extractor...
Dec 19 03:17:21 frans-desktop tracker-extract[65662]: Set scheduler policy to SCHED_IDLE
Dec 19 03:17:21 frans-desktop tracker-extract[65662]: Setting priority nice level to 19
Dec 19 03:17:21 frans-desktop dbus-daemon[1391]: [session uid=1000 pid=1391] Successfully activated service 'org.freedesktop.Tracker1.Miner.Extract'
Dec 19 03:17:21 frans-desktop systemd[1370]: Started Tracker metadata extractor.
Dec 19 03:17:31 frans-desktop systemd[1370]: tracker-extract.service: Succeeded.
Dec 19 03:17:51 frans-desktop tracker-store[65655]: OK
Dec 19 03:17:51 frans-desktop systemd[1370]: tracker-store.service: Succeeded

journalctl -a gives the same message.

I was unable to decently continue; no idea why that did not work. So this morning I retried from scratch but slightly different.
I recompiled photorec with -g and -ggdb
started it under gdb with /log and /debug
This is now at about 18%. The speed seems to be around 2%/hour so it will hit the 23% later tonight. Will report back by then or tomorrow.
And if it hangs hopefully gdb or the log will give some info.

[FransM]

EDIT: the issue seems to be solved. See post #11. I kept this here for reference/additional information.

Ok, this hang is repeatable.It just halted again at about the the same spot.
See attachment.
Screen is not updated any more (neither sectors read nor the elapsed time and expected duration.
top says that photorec uses 100 or near 100% cpu. Performance monitor indicates that one core is running at 100 %

dmesg shows no info.
journalctl -a has no relevant info either.

recovery started Thu Dec 21 08:27:46 2023 according to the log.
First recovered file has timestamp Dec 21, 08:48
Elapsed time is 10 hr 35 min 17 sec.
The timestamp of photorec.ses is
-rw-r--r-- 1 root root 1864524 dec 21 19:23 ../photorec.ses
That is indeed 10 hr 35 min after 8:48

Attached is a zip with edited log and ses files. Unedited the zip was > 512 KB being the size limit for attachments so I cut out most of the content, leaving beginning and end
(I still have the full files if needed)

Two observations:
the log file endis truncated (due to block buffering?) with
/media/frans/WIM-RECOVERY/recup_dir.1642/f3638352280.txt 3647805560-3647805567
/media/frans/WIM-RECOVERY/recup_dir.1642/f3638353240.txt 3647806520-36478


The ses file ends with
3647740384-3647805559
3647805568-3647806519
3647806528-3647806583
3647806592-3647809551
3647809560-3647809623
3647809632-3647809751
3647809760-3647833495
3647833504-3647865431
3647865440-3647865591
3647865600-3652722527
3658436976-15627846239

The first two entries here seem to match with the last two lines of the log
Note also the last number. It is the end of the disk, suppose this is expected.
After the last number of the .ses file there is still a bunch of zeroes.

After doing this I interrupted photorec with ^C and fell into gdb.
This gave me the following:
36 ../sysdeps/unix/sysv/linux/lseek64.c: No such file or directory.
(gdb) where
#0 __lseek64 (fd=40, offset=4096, whence=0) at ../sysdeps/unix/sysv/linux/lseek64.c:36
#1 0x00007ffff7b0ad7f in _IO_new_file_seekoff (fp=0x555555dc1e20, offset=0, dir=0, mode=<optimized out>) at libioP.h:948
#2 0x00007ffff7b087cd in __fseeko (fp=fp@entry=0x555555dc1e20, offset=offset@entry=0, whence=whence@entry=0) at fseeko.c:40
#3 0x0000555555570527 in my_fseek (stream=0x555555dc1e20, offset=offset@entry=0, whence=whence@entry=0) at filegen.c:1077
#4 0x000055555558d29a in check_riff_list (fr=fr@entry=0x7fffffffaf50, depth=depth@entry=2, start=start@entry=2147266380, end=end@entry=3830196731) at file_riff.c:130
#5 0x000055555558d325 in check_riff_list (fr=fr@entry=0x7fffffffaf50, depth=depth@entry=1, start=start@entry=2147266368, end=end@entry=3830207121) at file_riff.c:156
#6 0x000055555558d3f4 in file_check_avi (fr=0x7fffffffaf50) at file_riff.c:223
#7 0x0000555555567156 in file_finish_aux (file_recovery=0x7fffffffaf50, params=0x7fffffffe2f0, paranoid=1) at photorec.c:602
#8 0x00005555555679c6 in file_finish2 (file_recovery=file_recovery@entry=0x7fffffffaf50, params=params@entry=0x7fffffffe2f0, paranoid=<optimized out>,
list_search_space=list_search_space@entry=0x5555555f5da0 <list_search_space>) at photorec.c:722
#9 0x00005555555c30cf in photorec_aux (params=params@entry=0x7fffffffe2f0, options=options@entry=0x7fffffffe2d0, list_search_space=list_search_space@entry=0x5555555f5da0 <list_search_space>)
at psearchn.c:244
#10 0x00005555555c0fb9 in photorec (params=params@entry=0x7fffffffe2f0, options=options@entry=0x7fffffffe2d0, list_search_space=list_search_space@entry=0x5555555f5da0 <list_search_space>)
at phrecn.c:363
#11 0x00005555555c2611 in menu_photorec (params=params@entry=0x7fffffffe2f0, options=options@entry=0x7fffffffe2d0, list_search_space=list_search_space@entry=0x5555555f5da0 <list_search_space>)
at ppartseln.c:260
#12 0x00005555555bdc74 in photorec_disk_selection_ncurses (list_search_space=0x5555555f5da0 <list_search_space>, list_disk=0x55555570ce90, options=0x7fffffffe2d0, params=0x7fffffffe2f0)
at pdiskseln.c:225
#13 do_curses_photorec (params=0x7fffffffe2f0, options=0x7fffffffe2d0, list_disk=0x55555570ce90) at pdiskseln.c:322
#14 0x0000555555565fb9 in main (argc=3, argv=<optimized out>) at phmain.c:452
(gdb) step
__fseeko (fp=fp@entry=0x555555dc1e20, offset=offset@entry=0, whence=whence@entry=0) at fseeko.c:39
39 fseeko.c: No such file or directory.
(gdb) step
_IO_acquire_lock_fct (p=<synthetic pointer>) at libioP.h:884
884 libioP.h: No such file or directory.
(gdb) step
885 in libioP.h
(gdb) step
check_riff_list (fr=fr@entry=0x7fffffffaf50, depth=depth@entry=2, start=start@entry=2147266380, end=end@entry=3830196731) at file_riff.c:135
135 if (fread(buf, sizeof(buf), 1, fr->handle)!=1)
(gdb) where
#0 check_riff_list (fr=fr@entry=0x7fffffffaf50, depth=depth@entry=2, start=start@entry=2147266380, end=end@entry=3830196731) at file_riff.c:135
#1 0x000055555558d325 in check_riff_list (fr=fr@entry=0x7fffffffaf50, depth=depth@entry=1, start=start@entry=2147266368, end=end@entry=3830207121) at file_riff.c:156
#2 0x000055555558d3f4 in file_check_avi (fr=0x7fffffffaf50) at file_riff.c:223
#3 0x0000555555567156 in file_finish_aux (file_recovery=0x7fffffffaf50, params=0x7fffffffe2f0, paranoid=1) at photorec.c:602
#4 0x00005555555679c6 in file_finish2 (file_recovery=file_recovery@entry=0x7fffffffaf50, params=params@entry=0x7fffffffe2f0, paranoid=<optimized out>,
list_search_space=list_search_space@entry=0x5555555f5da0 <list_search_space>) at photorec.c:722
#5 0x00005555555c30cf in photorec_aux (params=params@entry=0x7fffffffe2f0, options=options@entry=0x7fffffffe2d0, list_search_space=list_search_space@entry=0x5555555f5da0 <list_search_space>)
at psearchn.c:244
#6 0x00005555555c0fb9 in photorec (params=params@entry=0x7fffffffe2f0, options=options@entry=0x7fffffffe2d0, list_search_space=list_search_space@entry=0x5555555f5da0 <list_search_space>)
at phrecn.c:363
#7 0x00005555555c2611 in menu_photorec (params=params@entry=0x7fffffffe2f0, options=options@entry=0x7fffffffe2d0, list_search_space=list_search_space@entry=0x5555555f5da0 <list_search_space>)
at ppartseln.c:260
#8 0x00005555555bdc74 in photorec_disk_selection_ncurses (list_search_space=0x5555555f5da0 <list_search_space>, list_disk=0x55555570ce90, options=0x7fffffffe2d0, params=0x7fffffffe2f0)
at pdiskseln.c:225
#9 do_curses_photorec (params=0x7fffffffe2f0, options=0x7fffffffe2d0, list_disk=0x55555570ce90) at pdiskseln.c:322
#10 0x0000555555565fb9 in main (argc=3, argv=<optimized out>) at phmain.c:452


I continued with signal 0 and reinterrupted a bit later. gdb again stopped in the lseek.

If there is more info needed let me know a.s.a.p. as I still have the job into gdb.
And if there is an idea how to resume let me know (I didn't quit photores yet in order to attempt continuation)

[FransM]

EDIT: the issue seems to be solved. See post #11. I kept this here for reference/additional information.

Ok, after peeking into the source I figured that the fail could be at the read at line 135 of file_rff.c so I went back to gdb and used next to find out a bit more of the flow

Program received signal SIGINT, Interrupt.
__lseek64 (fd=40, offset=4096, whence=0) at ../sysdeps/unix/sysv/linux/lseek64.c:36
36 ../sysdeps/unix/sysv/linux/lseek64.c: No such file or directory.
(gdb) n
__fseeko (fp=fp@entry=0x555555dc1e20, offset=offset@entry=0, whence=whence@entry=0) at fseeko.c:39
39 fseeko.c: No such file or directory.
(gdb) n
check_riff_list (fr=fr@entry=0x7fffffffaf50, depth=depth@entry=2, start=start@entry=2147266380, end=end@entry=3830196731) at file_riff.c:135
135 if (fread(buf, sizeof(buf), 1, fr->handle)!=1)
(gdb) n
144 next_fs=file_size + (uint64_t)8 + le32(list_header->dwSize);
(gdb) n
145 if(next_fs > end)
(gdb) n
151 if(memcmp(&list_header->dwList, "LIST", 4) == 0)
(gdb) n
167 file_size = (file_size&1);
(gdb) n
125 for(file_size=start; file_size < end;)
(gdb) n
130 if(my_fseek(fr->handle, file_size, SEEK_SET)<0)
(gdb) n
135 if (fread(buf, sizeof(buf), 1, fr->handle)!=1)
(gdb) n
144 next_fs=file_size + (uint64_t)8 + le32(list_header->dwSize);

So the fread at line 135 in file_riff.c does not fail.

While debugging the following question came to mind: What happens if photorec encounters a bad/unreadable block on the disk (not that I am seeing read errors in the log)