cgsecurity.org

Posted: **18 Feb 2021, 21:10**

I inadvertently overwrote the partition table on an Ubuntu system using LVM residing on, I believe a HW RAID1.
The system is/was running Ubuntu 18.04 LTS Server.

I was trying to add an external 10TB USB disk for local backups but I wrote a GPT partition table to the system disk instead of the USB disk.

I don't think I have done anything else that has modified the disks since updating the partition table.
The original machine is powered off right now.

I have used dd to make take an image copy of the disk and am trying out testdisk on that image file.

Below is the log file from an initial inspection of it.

Questions:
1. Is there a way to just extract the files from the LVM partition?
2. If I run testdisk from a liveDVD and have it update the GPT table would that make my original system bootable again?
3. Can I update the GPT table on the image file. Save it and then on a spare machine write that updated image to a raw disk to
make it bootable? If so, what tool to use for that?

Thanks for any help.

------------------------------------------------------

Using locale 'en_US.UTF-8'.

Thu Feb 18 13:54:55 2021
Command line: TestDisk /log /logname /home/john/testdiskLog.log /diske/backup_image_sda.img

TestDisk 7.0, Data Recovery Utility, April 2015
Christophe GRENIER <grenier@cgsecurity.org>
http://www.cgsecurity.org
OS: Linux, kernel 4.15.0-135-generic (#139-Ubuntu SMP Mon Jan 18 17:38:24 UTC 2021) x86_64
Compiler: GCC 7.2
ext2fs lib: 1.44.1, ntfs lib: libntfs-3g, reiserfs lib: none, ewf lib: none, curses lib: ncurses 6.0
User is not root!
Hard disk list
Disk /diske/backup_image_sda.img - 999 GB / 931 GiB - CHS 121535 255 63 (RO), sector size=512

Partition table type (auto): Intel
Media is opened in read-only.
Geometry from i386 MBR: head=255 sector=63
Disk /diske/backup_image_sda.img - 999 GB / 931 GiB (RO)
Partition table type: Intel

Analyse Disk /diske/backup_image_sda.img - 999 GB / 931 GiB - CHS 121535 255 63 (RO)
Geometry from i386 MBR: head=255 sector=63
BAD_RS LBA=1 0
Current partition structure:
1 P EFI GPT 0 0 2 121534 76 14 1952448511

Bad relative sector.
No partition is bootable

search_part()
Disk /diske/backup_image_sda.img - 999 GB / 931 GiB - CHS 121535 255 63 (RO)
Linux 6 127 58 25 126 37 305152
ext3 blocksize=1024 Sparse_SB, 156 MB / 149 MiB
Linux LVM 25 127 38 121533 254 63 1952034047
LVM2, 999 GB / 930 GiB

interface_write()
1 * Linux 6 127 58 25 126 37 305152
2 P Linux LVM 25 127 38 121533 254 63 1952034047
simulate write!
No extended partition

TestDisk exited normally.

Posted: **18 Feb 2021, 22:51**

jdehart1 wrote: 18 Feb 2021, 21:10 Questions:
1. Is there a way to just extract the files from the LVM partition?
2. If I run testdisk from a liveDVD and have it update the GPT table would that make my original system bootable again?
3. Can I update the GPT table on the image file. Save it and then on a spare machine write that updated image to a raw disk to
make it bootable? If so, what tool to use for that?

I am lacking linux experience and linux system recovery experience - so take every hint with a grain of salt.

First the report states that did not run as root. Running the report as root might give different results.

Answers:
1. You will have to try that out. Maybe it works if it's just about mirroring (Raid1). If LVM is not running in case of Raid5-array p.e. Testdisk would need to interpret LVM data to manage the member drives in able to pull out data. I doubt that.

2. The always repeating question of success guarantee cannot be answered because you can't rely on witness reports like yours. Furthermore, boot processes now can vary a lot with so many new boot techniques in place. I can't and don't want to support boot drive repairs - will typically answer recovery questions.

3. You can try that. dd or better ddrescue would be used to restore your image to a different drive. You will just have to reverse the backup process you triggered to generate your image file.

Posted: **18 Feb 2021, 23:12**

recuperation wrote: 18 Feb 2021, 22:51
jdehart1 wrote: 18 Feb 2021, 21:10 Questions:
1. Is there a way to just extract the files from the LVM partition?
2. If I run testdisk from a liveDVD and have it update the GPT table would that make my original system bootable again?
3. Can I update the GPT table on the image file. Save it and then on a spare machine write that updated image to a raw disk to
make it bootable? If so, what tool to use for that?
I am lacking linux experience and linux system recovery experience - so take every hint with a grain of salt.

First the report states that did not run as root. Running the report as root might give different results.

Answers:
1. You will have to try that out. Maybe it works if it's just about mirroring (Raid1). If LVM is not running in case of Raid5-array p.e. Testdisk would need to interpret LVM data to manage the member drives in able to pull out data. I doubt that.

2. The always repeating question of success guarantee cannot be answered because you can't rely on witness reports like yours. Furthermore, boot processes now can vary a lot with so many new boot techniques in place. I can't and don't want to support boot drive repairs - will typically answer recovery questions.

3. You can try that. dd or better ddrescue would be used to restore your image to a different drive. You will just have to reverse the backup process you triggered to generate your image file.

Thanks for the quick reply.
I will try #3 first before I move to #2.

Posted: **19 Feb 2021, 16:06**

I have recovered my files! Whew.

I was actually able to perform #1 from above, just extract the files from the LVM partition in the disk image.
Below I will give the detailed commands used.

To recap why I am here, I had overwritten my partition table with a blank GPT table.
The disk in question had an LVM partition that contained three filesystems of interest: /, /export and /users.

Step 0: DON'T PANIC. When I realized what I had done, I stopped immediately and made a 'dd' image copy of the disk.
Step 1: Continue to not Panic. In searching the internet, I learned about 'testdisk'.
Step 2: I built a spare machine with 4 1.2 TB disks. One was the system disk and 3 were spares for experimenting
on my 1TB disk image. I also attached a 10TB USB drive. I copied the disk image to the 3 spare drives and the 10TB drive.
Lots of backup copies just in case.
Step 3: Used 'testdisk' to find the partitions, write a proper partition table and write it back out to my disk image file.
I'll see if I can find where I did that in my log file, but here is a recreation of the steps.

Code: Select all

            > testdisk /diskc/backup_image_sda.img
            - Select disk image [Proceed]
            - Select Intel partition type [Enter]
            - [Analyse]
            - testdisk reports EFI GPT partition table, Bad relative sector, No partition is bootable.
            - [Quick Search]
            - testdisk finds two partitions: A bootable Linux partition and a Primary Linux LVM partition.
            - [Enter]
            - [Write], Y, [OK]
            - [Quit]
            - [Quit]
            > mv /diskc/backup_image_sda.img  /diskc/update_partition_table.img

Step 4: Learn about LVM and mounting an LVM partition from a disk image.
Step 5: Find the starting point (in bytes) and size (in bytes) of the LVM partition:

Code: Select all

john@forest2:~$ fdisk -l /diskc/update_partition_table.img
Disk /diskc/update_partition_table.img: 931 GiB, 999653638144 bytes, 1952448512 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device                             Boot  Start        End    Sectors   Size Id Type
/diskc/update_partition_table.img1 *    104448     409599     305152   149M 83 Linux
/diskc/update_partition_table.img2      409663 1952443709 1952034047 930.8G 8e Linux LVM
john@forest2:~$ sudo fdisk -l /diskc/update_partition_table.img
Disk /diskc/update_partition_table.img: 931 GiB, 999653638144 bytes, 1952448512 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device                             Boot  Start        End    Sectors   Size Id Type
/diskc/update_partition_table.img1 *    104448     409599     305152   149M 83 Linux
/diskc/update_partition_table.img2      409663 1952443709 1952034047 930.8G 8e Linux LVM
john@forest2:~$ bc
bc 1.07.1
Copyright 1991-1994, 1997, 1998, 2000, 2004, 2006, 2008, 2012-2017 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.
409663 * 512
209747456

1952034047 * 512
999441432064
quit

Step 6: Use those values with losetup to point a loop device at my LVM partition.

Code: Select all

john@forest2:~$ sudo losetup -v -f -o 209747456 --sizelimit 999441432064  /diskc/update_partition_table.img
john@forest2:~$ sudo losetup -a
/dev/loop1: [2049]:66981149 (/var/lib/snapd/snaps/gnome-3-34-1804_66.snap)
/dev/loop8: [2049]:66981153 (/var/lib/snapd/snaps/gnome-logs_103.snap)
/dev/loop6: [2049]:66981146 (/var/lib/snapd/snaps/core18_1988.snap)
/dev/loop4: [2049]:66981144 (/var/lib/snapd/snaps/gnome-3-26-1604_100.snap)
/dev/loop2: [2049]:66981154 (/var/lib/snapd/snaps/gnome-characters_570.snap)
/dev/loop0: [2049]:66981150 (/var/lib/snapd/snaps/gnome-calculator_826.snap)
/dev/loop9: [2081]:12 (/diskc/update_partition_table.img), offset 209747456, sizelimit 999441432064
/dev/loop7: [2049]:66981148 (/var/lib/snapd/snaps/gtk-common-themes_1514.snap)
/dev/loop5: [2049]:66981158 (/var/lib/snapd/snaps/gnome-system-monitor_157.snap)
/dev/loop3: [2049]:66981130 (/var/lib/snapd/snaps/core_10823.snap)
john@forest2:~$ sudo blkid /dev/loop9
/dev/loop9: UUID="q4m1Qd-GoEj-aewp-f8OP-rt1m-Hk9D-iufZpm" TYPE="LVM2_member"

Step 7: check the status of things with some 'blkid', 'pvs', and 'lsblk'

Code: Select all

john@forest2:~$ sudo blkid /dev/loop9
/dev/loop9: UUID="q4m1Qd-GoEj-aewp-f8OP-rt1m-Hk9D-iufZpm" TYPE="LVM2_member"
john@forest2:~$ sudo pvs
  WARNING: Not using lvmetad because duplicate PVs were found.
  WARNING: Use multipath or vgimportclone to resolve duplicate PVs?
  WARNING: After duplicates are resolved, run "pvscan --cache" to enable lvmetad.
  PV         VG         Fmt  Attr PSize   PFree
  /dev/loop9 vg_onlsrv3 lvm2 a--  930.80g    0
john@forest2:~$ sudo lsblk
NAME                   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0                    7:0    0   2.5M  1 loop /snap/gnome-calculator/826
loop1                    7:1    0   219M  1 loop /snap/gnome-3-34-1804/66
loop2                    7:2    0   276K  1 loop /snap/gnome-characters/570
loop3                    7:3    0  98.4M  1 loop /snap/core/10823
loop4                    7:4    0 140.7M  1 loop /snap/gnome-3-26-1604/100
loop5                    7:5    0   2.2M  1 loop /snap/gnome-system-monitor/157
loop6                    7:6    0  55.5M  1 loop /snap/core18/1988
loop7                    7:7    0  64.8M  1 loop /snap/gtk-common-themes/1514
loop8                    7:8    0   548K  1 loop /snap/gnome-logs/103
loop9                    7:9    0 930.8G  0 loop
├─vg_onlsrv3-lv_root   253:0    0   9.8G  0 lvm
├─vg_onlsrv3-lv_swap   253:1    0     8G  0 lvm
├─vg_onlsrv3-lv_export 253:2    0  48.8G  0 lvm
└─vg_onlsrv3-lv_users  253:3    0 864.2G  0 lvm
sda                      8:0    0   1.1T  0 disk
├─sda1                   8:1    0     1T  0 part /
├─sda2                   8:2    0     1K  0 part
└─sda5                   8:5    0    64G  0 part [SWAP]
sdb                      8:16   0   1.1T  0 disk
└─sdb1                   8:17   0   1.1T  0 part /diskb
sdc                      8:32   0   1.1T  0 disk
└─sdc1                   8:33   0   1.1T  0 part /diskc
sdd                      8:48   0   1.1T  0 disk
└─sdd1                   8:49   0   1.1T  0 part /diskd
sde                      8:64   0   9.1T  0 disk /diske
sr0                     11:0    1  1024M  0 rom

Step 8: Activate the volume group

Code: Select all

john@forest2:~$ sudo vgchange -a y vg_onlsrv3
  4 logical volume(s) in volume group "vg_onlsrv3" now active
john@forest2:~$ sudo dmsetup ls
vg_onlsrv3-lv_swap	(253:1)
vg_onlsrv3-lv_root	(253:0)
vg_onlsrv3-lv_export	(253:2)
vg_onlsrv3-lv_users	(253:3)

Step 9: Check the device manager view of volume group

Code: Select all

john@forest2:~$ sudo dmsetup info /dev/dm-0
Name:              vg_onlsrv3-lv_root
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        0
Event number:      0
Major, minor:      253, 0
Number of targets: 1
UUID: LVM-jS2YjC52D6fX4hsE4Ds6Ap6Tj7ypQ1dcNnQ75Dn3U7VVXE6cqUoqhV12pBwcNREs

john@forest2:~$ sudo dmsetup info /dev/dm-1
Name:              vg_onlsrv3-lv_swap
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        0
Event number:      0
Major, minor:      253, 1
Number of targets: 1
UUID: LVM-jS2YjC52D6fX4hsE4Ds6Ap6Tj7ypQ1dcWgs4itJkTM00GK1DSeQxFJ2zOj3Cpffx

john@forest2:~$ sudo dmsetup info /dev/dm-2
Name:              vg_onlsrv3-lv_export
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        0
Event number:      0
Major, minor:      253, 2
Number of targets: 1
UUID: LVM-jS2YjC52D6fX4hsE4Ds6Ap6Tj7ypQ1dcBtwgUxCw79IagR0hhg3jjr5exNSV8KVb

john@forest2:~$ sudo dmsetup info /dev/dm-3
Name:              vg_onlsrv3-lv_users
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        0
Event number:      0
Major, minor:      253, 3
Number of targets: 1
UUID: LVM-jS2YjC52D6fX4hsE4Ds6Ap6Tj7ypQ1dcfqDEPGOwNBms3o8rCxG8yAH9RbUwQiXS

Step 10: Make mount points for the three I am interested in (don't need swap).

Code: Select all

john@forest2:~$ sudo mkdir /onlsrv_root
john@forest2:~$ sudo mkdir /onlsrv_export
john@forest2:~$ sudo mkdir /onlsrv_users

Step 11: Mount them

Code: Select all

john@forest2:~$ sudo mount /dev/dm-0 /onlsrv_root/
john@forest2:~$ sudo mount /dev/dm-2 /onlsrv_export
john@forest2:~$ sudo mount /dev/dm-3 /onlsrv_users

Step 12: Check that they look like I expected:

Code: Select all

john@forest2:~$ ls /onlsrv_root/
backups                                dev         initrd.img.old  mnt                      rec      srv                   users
bacula-console.conf.ucftmp-jPTW5m89ug  etc         lib             mnt2                     root     sys                   usr
bacula-fd.conf.ucftmp-nF0KvuW8vK       export      lib64           offload                  run      tftpboot              var
bin                                    home        lost+found      packages.expandrive.gpg  sbin     tftpboot.export_link  vmlinuz
boot                                   initrd.img  media           proc                     scratch  tmp                   vmlinuz.old
john@forest2:~$ ls /onlsrv_export/
backup      ixp_files  keeboot_files    kernel_sources  lost+found       onl    simpana   UBUNTU_18_04_disklesskernel_for_tftpboot
ExpanDrive  ixp_logs   keeboot_kernels  KVM_Images      old-onlsrv.bkup  patch  tftpboot
john@forest2:~$ ls -1 /onlsrv_users | head 10
head: cannot open '10' for reading: No such file or directory
john@forest2:~$ ls -1 /onlsrv_users | head -10
0xsiow
aadil
aagatstein
aarellano
aarthi
aarti
aayuan
aayush
abdul
abdullah
john@forest2:~$ du -sh /onlsrv_
onlsrv_export/ onlsrv_root/   onlsrv_users/
john@forest2:~$ sudo du -sh /onlsrv_*
44G	/onlsrv_export
8.3G	/onlsrv_root
680G	/onlsrv_users
john@forest2:~$

Step 13: Copy Everything to my 10TB USB disk.
Step 14: Breath a sigh of relief.

And in case you are interested in the tear down of the above, here are the commands I used for that.

Code: Select all

john@forest2:~$ sudo umount /onlsrv_export
john@forest2:~$ sudo umount /onlsrv_root
john@forest2:~$ sudo umount /onlsrv_users
john@forest2:~$ sudo vgchange -a n vg_onlsrv3
  0 logical volume(s) in volume group "vg_onlsrv3" now active
john@forest2:~$ sudo losetup -a
/dev/loop1: [2049]:66981149 (/var/lib/snapd/snaps/gnome-3-34-1804_66.snap)
/dev/loop8: [2049]:66981153 (/var/lib/snapd/snaps/gnome-logs_103.snap)
/dev/loop6: [2049]:66981146 (/var/lib/snapd/snaps/core18_1988.snap)
/dev/loop4: [2049]:66981144 (/var/lib/snapd/snaps/gnome-3-26-1604_100.snap)
/dev/loop2: [2049]:66981154 (/var/lib/snapd/snaps/gnome-characters_570.snap)
/dev/loop0: [2049]:66981150 (/var/lib/snapd/snaps/gnome-calculator_826.snap)
/dev/loop9: [2081]:12 (/diskc/update_partition_table.img), offset 209747456, sizelimit 999441432064
/dev/loop7: [2049]:66981148 (/var/lib/snapd/snaps/gtk-common-themes_1514.snap)
/dev/loop5: [2049]:66981158 (/var/lib/snapd/snaps/gnome-system-monitor_157.snap)
/dev/loop3: [2049]:66981130 (/var/lib/snapd/snaps/core_10823.snap)
john@forest2:~$ sudo losetup -d /dev/loop9
john@forest2:~$

cgsecurity.org

Wiped out my Partition table

Wiped out my Partition table

Re: Wiped out my Partition table

Re: Wiped out my Partition table

Re: Wiped out my Partition table