CentOS 5/6 Major Failure - Data Lost

Using TestDisk to repair the filesystem
Forum rules
When asking for technical support:
- Search for posts on the same topic before posting a new question.
- Give clear, specific information in the title of your post.
- Include as many details as you can, MOST POSTS WILL GET ONLY ONE OR TWO ANSWERS.
- Post a follow up with a "Thank you" or "This worked!"
- When you learn something, use that knowledge to HELP ANOTHER USER LATER.
Before posting, please read https://www.cgsecurity.org/testdisk.pdf
Locked
Message
Author
szymonsg
Posts: 1
Joined: 11 Jan 2016, 07:13

CentOS 5/6 Major Failure - Data Lost

#1 Post by szymonsg »

Hi,

I have a question regarding a certain very major issue that occurred for myself twice.
Basically we were running 2 x CentOS (CentOS 5 and CentOS 6) samba file servers.
One of them was physical and one virtualised.

Physical Box: 2 x RAID 5 - LVM on top of those
Virtualised: 2 x Virtual Disks - LVM built on top

When the issue occurred both servers basically run out of space due to user activity
and all the processes hang. Due to panic IT guy who manages it instead of investigating
rebooted the box straight away. Once the box came up - it went straight away to GRUB.
Not detecting any file system at all. We were not able to get it up therefore we mounted
the disks on live image.

It appeared that half of the root file system was wiped out as well as all user data.
Unrecoverable or partially recoverable using 3rd party software.

The interesting thing is that it happened for both of the server - setup was basically
the same, however as mentioned above one was virtualised and one physical.

Facts:
1. The hardware disks are intact, there wasn't any physical failure.
2. Disk run out of space and server was rebooted. After reboot data was lost.
3. Only partial system data was inside when we mounted it on live image.

One of the boxes:
Linux <hostname> 2.6.18-406.el5 #1 SMP Tue Jun 2 17:25:57 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux

Does anyone have an idea on what could have happened? We are kind of stuck and unable to recover
even 50% of mission critical data. Any help would be appreciated.

EDIT:
[root@AP-MEL-CWENTOS6 ~]# lvscan
ACTIVE '/dev/vg_melst/lv_root' [50.00 GiB] inherit
ACTIVE '/dev/vg_melst/lv_home' [2.53 TiB] inherit
ACTIVE '/dev/vg_melst/lv_swap' [3.88 GiB] inherit
ACTIVE '/dev/vg_apmelcwentos6/lv_root' [35.63 GiB] inherit
ACTIVE '/dev/vg_apmelcwentos6/lv_swap' [3.88 GiB] inherit
wait no
[root@AP-MEL-CWENTOS6 ~]# pvscan
PV /dev/sdb2 VG vg_melst lvm2 [2.00 TiB / 0 free]
PV /dev/sdc VG vg_melst lvm2 [600.00 GiB / 5.00 GiB free]
PV /dev/sda2 VG vg_apmelcwentos6 lvm2 [39.51 GiB / 0 free]
Total: 3 [2.62 TiB] / in use: 3 [2.62 TiB] / in no VG: 0 [0 ]
pvscan
and
[root@AP-MEL-CWENTOS6 ~]# fdisk -l

Disk /dev/sdb: 2418.9 GB, 2418925580288 bytes
255 heads, 63 sectors/track, 294084 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0009a6ee

Device Boot Start End Blocks Id System
/dev/sdb1 * 1 64 512000 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sdb2 64 267350 2146969600 8e Linux LVM

Disk /dev/sdc: 644.2 GB, 644245094400 bytes
255 heads, 63 sectors/track, 78325 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/sda: 42.9 GB, 42949672960 bytes
255 heads, 63 sectors/track, 5221 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000cfe34

Device Boot Start End Blocks Id System
/dev/sda1 * 1 64 512000 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2 64 5222 41430016 8e Linux LVM

Disk /dev/mapper/vg_apmelcwentos6-lv_root: 38.3 GB, 38260441088 bytes
255 heads, 63 sectors/track, 4651 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/vg_apmelcwentos6-lv_swap: 4160 MB, 4160749568 bytes
255 heads, 63 sectors/track, 505 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/vg_melst-lv_root: 53.7 GB, 53687091200 bytes
255 heads, 63 sectors/track, 6527 cylinders

We can get some files back with PhotoRec, however 99% of them e.g. PSDs are corrupted etc.
"unexptected end of file"...

Anyone has idea on what might have happened?

Best Regards,
SzymonSG

Locked