recovering Linux Raid disk from Synology - HOWTO?

[ipanini]

Hi folks,

First of all a big thank you for TestDisk!
I would like to ask a little help on how to proceed.

What happened:
My 8 bay Synology was standing on a shelf in our garage, but unfortunately a pending electrical cable to be installed, inadvertently opened one of the drive bays.
Result: At first sight all data gone.
I'm not sure, but I think my Disk Station hat two volumes. What I'm sure of is that all volumes were JBOD.
Primary mistake: off course no decent backup

TestDisk to the rescue:
- I have connected all 8 hard drives to a PC that is running Ubuntu 20.04 from a usb drive. (slow)
- Gparted can see the disk, but sees it as unknown filesystm
- I then installed TestDisk and ran it
- After running for approx two days. TestDisk apparently has found the necessary ingredients and proposes to go to the next step
And this is where I'd like some help..

(I've navigated to the first line in the attached screenshot, filesys. data, then I could press P to view data, and it seems to me that my files are intact.)
I assume that the Partition Table has been damaged.
I think Synology is using mdadm, at least mdadm can be used to interrogate the disks

TestDisk shows:
Keys A: Add partition, L: Load backup and T: change type

Am I correct in assuming that:
a) I need to navigate using up / down arrow to each partition
b) on each partition > press Key A (to add partition)
Is this the way to reconstruct the partition table?
Or are their other steps to be taken?

Thanks for helping out!


Apparently I'm not allowed to post or upload a screenshot?
So I'm mimicking the screenshot

----------------------
TestDisk 7.1, Data Recovery Utility, July 2019
Christophe GRENIER <grenier@cgsecurity.org>

----------------------
Disk /dev/sdb - 8001 GB / 7452 GiB - CHS 972801 255 63
Partition Start End Size in sectors
D Linux filesys. data 2048 4982399 4980352 [1.42.6-24922]
D Linux Raid 2048 4982527 4980480 [md0]
D Linux Swap 4982528 9176687 4194160
D Linux Raid 4982528 9176831 4194304 [md1]



Structure: Ok. Use Up/Down Arrow keys to select partition
Use Left / Right Arrow keys to CHANGE partition characteristics:
P = Primary D= Deleted
Keys A: add partition, L load backup, T: change type,
Enter: to continue
md 0.90.0 B.Endian Raid 1: devices 0(8,17) 1(8,1) 2(8,33) 3(8,49) 4(8,68) 5(8,113) 6(8,97)* 7(8,81), 2550 MB / 2431 MiB
------------------------

[recuperation]

Hi folks,

First of all a big thank you for TestDisk!
I would like to ask a little help on how to proceed.

What happened:
My 8 bay Synology was standing on a shelf in our garage,

Just wondering if your "8 bay Synology" feels comfortable with the thermo windows (max and min temperature) in your garage.

but unfortunately a pending electrical cable to be installed, inadvertently opened one of the drive bays.
Result: At first sight all data gone.
I'm not sure, but I think my Disk Station hat two volumes. What I'm sure of is that all volumes were JBOD.

I don't know how your secret "Synology" operates. In addition to that I neither have Linux mdadm and lvm experience! But it looks like you are having 8 disks with 1 TB each installed and maybe some controller presents them as one disk having 8 TB.

Primary mistake: off course no decent backup

TestDisk to the rescue:
- I have connected all 8 hard drives to a PC that is running Ubuntu 20.04 from a usb drive. (slow)
- Gparted can see the disk, but sees it as unknown filesystm
- I then installed TestDisk and ran it

"ran it" is a pretty bad description. There is nothing more vague than that.

- After running for approx two days.

Doing what? Analysis? Quick search? Deeper search?

TestDisk apparently has found the necessary ingredients and proposes to go to the next step
And this is where I'd like some help..

(I've navigated to the first line in the attached screenshot, filesys. data, then I could press P to view data, and it seems to me that my files are intact.)
I assume that the Partition Table has been damaged.
I think Synology is using mdadm, at least mdadm can be used to interrogate the disks

TestDisk shows:
Keys A: Add partition, L: Load backup and T: change type

Am I correct in assuming that:
a) I need to navigate using up / down arrow to each partition
b) on each partition > press Key A (to add partition)
Is this the way to reconstruct the partition table?
Or are their other steps to be taken?

Please read:

https://www.cgsecurity.org/wiki/TestDisk_Step_By_Step
https://www.cgsecurity.org/wiki/TestDisk
There is a manual as well.

Your "screenshot emulation" shows partitions whose content you were able to verify.
There is no need to add additional partitions. You just have to enable the existing ones that you verified by setting them to primary.

Partitions may not share space. Based on the search below you may only enable the first or (exclusive or! just one!) second partition combined with the third or (exclusive or! just one!) fourth.

Thanks for helping out!


Apparently I'm not allowed to post or upload a screenshot?

Have you tried the attachments tab on the bottom of the page?

So I'm mimicking the screenshot

----------------------
TestDisk 7.1, Data Recovery Utility, July 2019

This is an outdated version. Use version 7.2 or 7.2-WIP.

Christophe GRENIER <grenier@cgsecurity.org>

----------------------
Disk /dev/sdb - 8001 GB / 7452 GiB - CHS 972801 255 63
Partition Start End Size in sectors
D Linux filesys. data 2048 4982399 4980352 [1.42.6-24922]
D Linux Raid 2048 4982527 4980480 [md0]
D Linux Swap 4982528 9176687 4194160
D Linux Raid 4982528 9176831 4194304 [md1]



Structure: Ok. Use Up/Down Arrow keys to select partition
Use Left / Right Arrow keys to CHANGE partition characteristics:
P = Primary D= Deleted
Keys A: add partition, L load backup, T: change type,
Enter: to continue
md 0.90.0 B.Endian Raid 1: devices 0(8,17) 1(8,1) 2(8,33) 3(8,49) 4(8,68) 5(8,113) 6(8,97)* 7(8,81), 2550 MB / 2431 MiB
------------------------

What confuses me is that you are having two volumes in RAID 1 mode. Have you set them up this way?
To me it looks like a disk array (JBOD) represented by /dev/sdb with two RAID 1-arrays on top. Maybe I am wrong.
I have difficulties in imaging what happens if you put a RAID 1 software raid over a JBOD configuration of 8 x 1 TB.

I would suggest getting 8TB of free storage or less depending on how much you were using. You would then copy the content to that disk and you would verify afterwards that the rescued files can be read.

If everything is OK with the duplicated files, you could set the two partitions (volumes) to Primary and write that new partition table.
If you however repair the partition table and use your "Synology" in read/write mode you are playing roulette.

Another secure way of operating is to run ddrescue as described in the manual and cloning your /dev/sdb either to a file or a disk.

That would provide you with a duplicate of the current unrepaired disk environment.

[ipanini]

Thanks for your reply! Much appreciated!

1. I may have overlooked the option to "Notify me when a reply is posted", hence it took time before I came back to check for an answer. Sorry for that. Will a notification be sent by email?
2. In the mean time I have fixed the rescue PC to run from an SSD instead of booting off of a usb-stick, so booting at least is a lot faster.
3. I'm running ubuntu 20.04.1, and it seems TestDisk 7.1 is the latest available there in the repo. But I have downloaded 7.2 directly and am running it from a terminal.

Your answers / remarks:
- The 8 bay Synology has no temperature issues. In fact I run several Synologies, a two bay, 5 bay and 8 bay model and neither has ever had temperature issues. I guess it's not too hot there
- Not 8 disks with 1 TB each, but 100% 8 x 8TB disks
- running it is indeed a very poor expression, my point is that I do not have any experience with TestDisk, hence the term "ran" it. In fact it was "Analysing".

4. Attachments do seem to work now, maybe it had to do with me being too new a forum subscriber?
5.

What confuses me is that you are having two volumes in RAID 1 mode. Have you set them up this way?
To me it looks like a disk array (JBOD) represented by /dev/sdb with two RAID 1-arrays on top. Maybe I am wrong.
I have difficulties in imaging what happens if you put a RAID 1 software raid over a JBOD configuration of 8 x 1 TB.

=> To be very honest: I'm not 100% sure either. I found a *.dss file from another synology, so that is not going to help us here, but I was wondering a) if it contains diskstation storage manager information, and b) if it is "always" present somewhere, the idea being that I could then "retrieve" or copy it from the disks and consult the exact configuration.
I do seem to remember that I maybe had only a couple (two) drives at hand when I configured the Diskstation and added 6 others later, but I am not sure about that.

6. Your suggestion:

Code: Select all

I would suggest getting 8TB of free storage or less depending on how much you were using. You would then copy the content to that disk and you would verify afterwards that the rescued files can be read.

At the moment I'm freeing up a 8TB disk and wil create a copy of the content.
But...
As the Synology consists of 8 x 8 TB disks, and as the mishap caused **ALL** 8 x 8TB drives to be "unreadable" I would very much like to proceed in trying to "rescue" or "heal" the damaged disk.
TestDisk 7.2 reports an "Invalid RAID superblock" (see screenshot)

What I'm wondering:
- At the moment I'm freeing up an other 8TB drive that I will also connect (I hope this will work because I'm already at 9 drives so that will become 10 drives..) to the ubuntu machine
- I can then create a copy / clone that will contain a backup of the files of the damaged disk
So far so good.
- but as far as I understand, the clone or copy will not have the essential RAID information, but only the files
- So then, how will I be able to reconstruct the two volumes of my synology?

Also:
- As far as I can reconstruct my Synology had two volumes, one consisting of 2x 8TB drives, the other consisting of 6x 8TB drives.
- Since only a single drive has been damaged, it should be either volume one (2 x 8TB) or volume 2 (6 x 8TB) that is unreadable
- If this reasoning is correct, I should be able to assemble either volume one or volume two..
But how to find out the missing pieces of the puzzle?

Thanks a lot for any help or pointer!
If any more screenshots are needed, I will be glad to provide them.

Screenshot from 2024-05-25 13-34-28.png (130.36 KiB) Viewed 12936 times

Screenshot from 2024-05-25 16-44-52_invalid RAID superblock.png (68.9 KiB) Viewed 12936 times

[ipanini]

Ok, here is a follow up:
- I have freed up a 8TB disk to accomodate the files of the damaged 8TB disk /dev/sdc
- Connected the extra 8TB drive and mounted it
- Let TestDisk 7.2 create an image of /dev/sdc (This has taken approx 2 days)

To be clear: because my Synology is using btrfs, I have installed BTRFS tools on ubuntu, and formatted the 8TB rescue drive as a single BTRFS partition.
So I now have a 7.9 TB image.dd file on this BTRFS formatted drive.

Now I'd like to check my files, so I expected to be able to mount the image.dd file and then be able to read them, but this does not seem to work.

Here's what I get:

Code: Select all

erwin@erwin-HP-EliteDesk-800-G1-TWR:~$ sudo mount -o loop,ro /home/erwin/BU/image.dd /home/erwin/files
[sudo] password for erwin: 
NTFS signature is missing.
Failed to mount '/dev/loop7': Invalid argument
The device '/dev/loop7' doesn't seem to have a valid NTFS.
Maybe the wrong device is used? Or the whole disk instead of a
partition (e.g. /dev/sda, not /dev/sda1)? Or the other way around?
erwin@erwin-HP-EliteDesk-800-G1-TWR:~$

Thanks for helping out!

[recuperation]

Sorry, I can't help you anymore.

You do not know your previous configuration. If I understand your description correctly your machine is using some software that is emulating one or two virtual disks or volumes. Maybe your machine is using BTRFS for this particular task as BTRFS is able to span structures over a couple of disks.
Therefore you cannot just pick on of your 8 disks, pretending that is broken and running TestDisk against that particular disk.

Those disks must be mapped to some virtual device(s) but the device list in your screenshot does not show that device(s)

As far as I know TestDisk is not able to copy disks, only partitions but you pretend having copied a disk using TestDisk and now you seem to treat that partition like a disk that has a partition table.

Where is that line coming from?

> md 0.90.0 B.Endian Raid 1: devices 0(8,17) 1(8,1) 2(8,33) 3(8,49) 4(8,68) 5(8,113) 6(8,97)* 7(8,81), 2550 MB / 2431 MiB

It does not look plausible that you have some RAID 1 with a size of 2,55 TB when you are operating eight disks with a size of eight TB each...

[ipanini]

Thanks for sticking with me!

Code: Select all

> md 0.90.0 B.Endian Raid 1: devices 0(8,17) 1(8,1) 2(8,33) 3(8,49) 4(8,68) 5(8,113) 6(8,97)* 7(8,81), 2550 MB / 2431 MiB

This information comes from /dev/sdb

As far as I know, the Synology creates a couple of default partitions on each ((SHR) RAID) being (if not mistaken):
/dev/sdb1 linux-raid 2.37 GB
/dev/sdb2 linux-raid 2.00 GB (=> I think this is swap partition)
then always some unallocated part of approx 127.12 MiB
/dev/sdb3 linux-raid 7.27 TiB

The previously described partition structure is the same on all 8 x 8TB drives. (As visible in previous screenshots)
----------------------------

Question:
Is there some command line flag to have TestDisk write the log file to a separate log?
Is it allowed to run TestDisk simultaneously in multiple terminal windows?

If this is possible, I will:
- remove the "repair" disk that is foreign to the 8 diskset
- run TestDisk on all 8 disks (if this is allowed and possible)
- have the possibility to have a look at e.g log /dev/sdb log /dev/sdc log/dev/sde etc
I will also then be able to post and share this information

I had also a second thought:
- Since obviously TestDisk can read and copy the files on the damaged disk
- I could re-create a drive with the identical disk / partition layout as the other 7 disks
- then copy the file contents to the correct partition using TestDisk
The only question that would then remain: Is how to set the "label" of the new drive correctly (This label (shown in GParted) seems to always be DS2:3 where I know that DS2 is the hostname of my Synology. I do not know where the #3 comes from..

Again, thanks a lot for sticking with me!

[recuperation]

Question:
Is there some command line flag to have TestDisk write the log file to a separate log?

As far as I know you can only enforce the creation of a log file but not its location.

Is it allowed to run TestDisk simultaneously in multiple terminal windows?

That is not a supported way of application. I would not run TestDisk twice simultaneously against the same disk target but this is not what you are probably trying to do. If you are selecting the logfile creation the different versions will probably mess up your logfile.

If this is possible, I will:
- remove the "repair" disk that is foreign to the 8 diskset
- run TestDisk on all 8 disks (if this is allowed and possible)
- have the possibility to have a look at e.g log /dev/sdb log /dev/sdc log/dev/sde etc
I will also then be able to post and share this information

You can post your log file, that is certainly interesting for me, but I guess I can't help you anyway. I only have experiences with simple configurations. The use of BTRFS probably spanning multiple disk is not simple anymore.

I had also a second thought:
- Since obviously TestDisk can read and copy the files on the damaged disk

As I said, you cannot treat your array as individual entities anymore when your are using volume management features that combine individual disks.

- I could re-create a drive with the identical disk / partition layout as the other 7 disks

- then copy the file contents to the correct partition using TestDisk

See remark above!

The only question that would then remain: Is how to set the "label" of the new drive correctly (This label (shown in GParted) seems to always be DS2:3 where I know that DS2 is the hostname of my Synology. I do not know where the #3 comes from..

Sorry, I have no idea.