mdadm raid5 recovery

Using TestDisk to repair the filesystem
Forum rules
When asking for technical support:
- Search for posts on the same topic before posting a new question.
- Give clear, specific information in the title of your post.
- Include as many details as you can, MOST POSTS WILL GET ONLY ONE OR TWO ANSWERS.
- Post a follow up with a "Thank you" or "This worked!"
- When you learn something, use that knowledge to HELP ANOTHER USER LATER.
Before posting, please read https://www.cgsecurity.org/testdisk.pdf
Locked
Message
Author
frawst
Posts: 1
Joined: 03 Mar 2021, 01:19

mdadm raid5 recovery

#1 Post by frawst »

TL,DR; I might've messed up huge, I need someone's help who knows mdadm better than I. I can provide any sort of data, but this crazy long story will get you up to par, if you care to read it. I'm at a loss and worried.

Okay, I'm not even sure where to start with this post. the short of it is that i'm not sure if i'm screwed, or what to do to even check. I fear that I may have made one too many mistakes in my raid in the attempt to recover it. I've never posted for such a thing, so i'm just going to start from the top.. After days of research, I've learned that naïve me did not create the best situation. I'm aware of it now, and have a better game-plan going forward. the only concern now is the hope of recovering my data to get out of this.

I have an mdadm Raid-5 that consists of 9 8Tb disks with an LVM / XFS filesystem. The other day I was moving data from one directory to another in this volume, and out of nowhere it throws an I/O error, and the XFS filesystem shut down, this is the first time i've ever seen this. I couldn't figure out how to re-enable it, so i rebooted, everything mounted, and I immediately proceeded to get the data off and onto other drives. It does it again, but this time on reboot 3 of the disks go missing. No raid.

After going and physically verifying everything is connected and powered, i reboot the machine. and this time the disks seem to not be spinning at all, and don't show up in the bios. The next step is where I learned something the hard way. I took those 3 disks to my desktop, and hooked them up using the WD EasyStore controller they were paired with on purchase. Apparently these guys change the drive layout in some way, as they were then labeled as "easystore" drives, even after connecting them directly via sata. The easystore controller spun them all up to show that SMART was clean (all drives are, for that matter).

Now we're in trouble. The mdadm superblocks are gone / missing from these disks, and I now have no idea of knowing what order they should be in to re-assemble the raid (too many missing to automatically assemble). After a bunch of research, verification and noting down what I know, I went to a clean new vm for testing.

in this vm I made a brand new raid 5, formatted it, and put some data on it. I then stopped it, and zero'd the superblock of two of them. tried to assemble, and it didn't work. (expected). I then did the mdadm --create command again, calling the disks in the wrong order (and marked one missing). it assembled, the lvm showed, but wouldn't mount. I then went back and did the same thing, but in the right order, it worked. files and all.

At this point I knew the order of 7 out of 9 disks in the array, I took this knowledge and tried it on the real thing.

For the two unknowns, I tried it one way "/dev/sda, /dev/sdb"

It re-wrote all the superblock data (I have what was before notated to the best of that bad situation).

it then created the raid of all 9 disks and immediately started syncing, but no LVM to be found. I immediately stopped the raid for worries of improper syncing, and tried it again with the two unknown disks in reverse order "/dev/sdb, /dev/sda". same result, resyncing raid. no lvm.

I'm now worried that I've lost everything, and I'm at a loss. I have no idea where to go from here, aside from laying in darkness for the next year (again!).

If anyone has any sufficient knowledge with mdadm, I desperately need your help.

Locked