Page 1 of 1

Catastrophic data loss event

Posted: 10 Oct 2019, 19:36
by jdb2
I recently set up an Intel NUC8i7BEH using ZFS root on Linux Mint 19.2 Cinnamon 64-bit, to replace the ancient 10+ year old Toshiba laptop that I had been using. After I had copied all my data over everything was fine on my new setup for months. One fateful day around this September 16th or so, at about 11 AM in the morning, I made a mistake : I had a drive with a huge amount of very important data on it and I was trying to figure out why the rsync backup script that I had written wasn't working correctly when backing up the data to another ZFS backup drive. I saw that the problem was with the wine directory, so I just blindly, without thinking, issued a "sudo rm -rf wine". Unfortunately, it turns out that the "wine/dosdevices/z:" directory was a link to my root directory! >:( After I had realized what had happened I panicked and did all the wrong things. I issued an ALT-SYSRQ-RSEIUNB. I then booted off of a recovery USB drive and rolled back my ZFS datasets to around 10 AM that morning and restored my system. I then went to check, over the course of three or four mounts and umounts ( oops! ), if the ext4 formatted external drive that I was trying to backup and which had been mounted in my "media" directory was intact. Well, it wasn't :cry:
I'd say about 50% of the files had been deleted. I ended up using ext4magic to attempt a restore, but, I never got any more data restored that wasn't already in the partial backup. Note to self : if something like this happens again, *immediately* unplug drive, *don't* mount it again and then save the journal. Anyways, today, I recovered a huge amount of files from a dd image of the drive using photorec. The drive contained many old Windows and Linux system backups, disk images, journal notes, ideas, source code and my other irreplaceable items. What I really want to recover are all the ASCII text files which consist of my source code, journal entries, ideas etc. I'm hitting myself for not prefixing the contents of these types of files with some type of signature. As it is now, I have recovered about 100K+ ASCII text files, with seemingly random names. I first plan on removing any duplicates. But, I was wondering if there's any tool out there that can parse the text files and determine if each file is some configuration cruft that I don't need or is actually something that I want and is important. My guess is that without strong human-level AI, I'm going to have to do it manually :/

Since the data loss event, I'm hesitant to use "sudo rm -rf" anymore. Also, the man page for rm says :

Code: Select all

‘--preserve-root’
     Fail upon any attempt to remove the root directory, ‘/’, when used
     with the ‘--recursive’ option.  This is the default behavior.
Shouldn't this have prevented rm from bulldozing through my filesystems?

Any suggestions or help would be appreciated.

Thanks,

jdb2

Re: Catastrophic data loss event

Posted: 11 Oct 2019, 05:55
by cgrenier
A symbolic link isn't followed when deleting files.

Code: Select all

$ mkdir test1 test2
$ ln -s ../test1 test2/test1
$ ls -d test1 test2/test1 
test1  test2/test1
$ rm -rf test2
$ ls -d test1
test1
.

zfs_undelete may give good results but it may be too late if you have reversed your partition state to a previous snapshot.

findup from fslint can be use to remove duplicates.
There is no magic tool to sort the files, grep is often the "best" way to find source code related to a project. It's very time consuming.
Good luck

Re: Catastrophic data loss event

Posted: 11 Oct 2019, 17:25
by jdb2
cgrenier wrote: 11 Oct 2019, 05:55 A symbolic link isn't followed when deleting files.

Code: Select all

$ mkdir test1 test2
$ ln -s ../test1 test2/test1
$ ls -d test1 test2/test1 
test1  test2/test1
$ rm -rf test2
$ ls -d test1
test1
.
Thank you for your reply! :D In my case though, I had a "/media/mint/1c207119-3d71-4969-863a-6c077b73e35f/" directory
that had been auto-mounted from the ext4 formatted external drive with the important data. There was a
"wine/.wine/dosdevices/z:" file that was some type of recursive directory loop through my root directory. I don't
know why "rm -rf" followed the links -- maybe it was some type of hard link? I don't see how that's possible though.
zfs_undelete may give good results but it may be too late if you have reversed your partition state to a previous snapshot.
I actually didn't have any permanent data loss on any of my ZFS formatted drives. After I accidentally issued the "rm -rf" I
rebooted off a recovery USB drive and rolled back all my ZFS datasets to a known good snapshot which was about 10 AM
the morning of the disaster. Unfortunately, I couldn't roll back the ext4 formatted external drive. It's very ironic that my
attempt to back-up this drive resulted in massive data loss. For all my ZFS formatted drives I have ZFS auto snapshots
turned on and managed by zfs-auto-snapshot via cron jobs.
findup from fslint can be use to remove duplicates.
Thanks for the tip :) I had already read about fslint from the "After PhotoRec" section in the FAQ, and it seems that I have a massive
amount of duplicate files, mostly logs, which I don't need.
There is no magic tool to sort the files, grep is often the "best" way to find source code related to a project.
Well, I can't believe that no one has written some tool that parses text files and looks for source code or English grammar and words.
That would indeed be very helpful. :)
It's very time consuming.
Are you intimating that you've had to go through this same hell before? :P
Good luck
Thanks :)

Regards,

jdb2

Re: Catastrophic data loss event

Posted: 11 Oct 2019, 19:33
by jdb2
I didn't realize that I was talking to the author of TestDisk and PhotoRec :oops:

Thanks so much for your great effort over the years in creating these data recovery tools! :D

Without them, I'd have no chance of recovering my data :)

Thanks and regards,

jdb2