Page 1 of 1

Dramatic failure on a RAID 5 Array

Posted: 26 Jul 2016, 12:24
by radix
Hello everybody.
Few days ago one of the servers I am administering (IBM x3650 with ORACLE LINUX 7 O.S.) experienced a serious failure.
Apparently, the problem was with the RAID controller. The system got frozen during the night, exactly when it was running the daily backup task.
Therefore, the backup was not completed and a full day of work for 60 internal users could be lost.
The message displayed during the boot was:

IBM serveRAID 8k-l at PCI Bus:v04, Dev:v00, Func: 00
Fatal Error: Controller Kernel PANIC !!
Unknown firmware error
Error Code: C7

Hence, our first diagnosys was that the controller had gone.
We replaced the controller but the message was the same, hence the problem could be with the mainboard.
We replaced the server but the message was again the same. Hence, the problem was with one of the HDDs.
Removed the first disk of the array (RAID 5 with 6 HDDs) the error did not appear any more and the boot went on.
Unfortunately, the server started in Emergency mode because of an error on /dev/sda at a certain block #.

Now I am running TestDisk to try to recover -as a miminum requirement- the ORACLE database files that are on the server and that contain all the work history of 2 companies.
Of course I have the backups, but they do not cover the last day of work (including the creation of something like 3K end-user invoices) and the Oracle Transaction Log files are on the same Array (therefore, unavailable).

This is the very first time I am using TestDisk to recover data from an ORACLE LINUX 7 server.
Any advice about TestDisk usage best practices will be extremely welcome....

One more info: when starting the emergency mode, the O.S. warns that "No Linux Partitions are found on /dev/sda".

Thanks a lot in advance for your kind help.

Re: Dramatic failure on a RAID 5 Array

Posted: 27 Jul 2016, 20:00
by cgrenier
If the partition is on an ext2/3/4 filesystem, try TestDisk, select /dev/sda, Advanced, select the partition, List and copy the file.
Otherwise, it may be possible to mount the partition in read-only (something like mount -o ro /dev/sdaX /mnt/XXX).