LVM2 on raid6 partition recovery Topic is solved

How to use TestDisk to recover lost partition
Forum rules
When asking for technical support:
- Search for posts on the same topic before posting a new question.
- Give clear, specific information in the title of your post.
- Include as many details as you can, MOST POSTS WILL GET ONLY ONE OR TWO ANSWERS.
- Post a follow up with a "Thank you" or "This worked!"
- When you learn something, use that knowledge to HELP ANOTHER USER LATER.
Before posting, please read https://www.cgsecurity.org/testdisk.pdf
Message
Author
MRovis
Posts: 18
Joined: 06 Mar 2012, 19:00
Contact:

Re: LVM2 on raid6 partition recovery

#11 Post by MRovis »

I hope it wouldn't be a problem if I, without cross-posting, but rather, drawing attention the my thread at this forum, tried to get help from my Gentoo mates on the:
https://forums.gentoo.org/
There's often been people who helpe me get me out in similar messes there.
And it's a huge community, pretty huge. I hope, to the extent of my linux raid issue, which seems obvious to me that my partition recovery boils down to, I hope that it is also advantageous for this forum.
Because I am stalled at this time.
Did a lot, but this hurdle stalls me.

User avatar
cgrenier
Site Admin
Posts: 5432
Joined: 18 Feb 2012, 15:08
Location: Le Perreux Sur Marne, France
Contact:

Re: LVM2 on raid6 partition recovery

#12 Post by cgrenier »

What gives

Code: Select all

mdadm --run /dev/md127
?

MRovis
Posts: 18
Joined: 06 Mar 2012, 19:00
Contact:

Re: LVM2 on raid6 partition recovery

#13 Post by MRovis »

Not yet gone anywhere.
Found and reading this:
http://www.issociate.de/board/post/5088 ... ssues.html
It's "all spares" to be found there, and it's the author of mdadm's wrote there.
Be back.

MRovis
Posts: 18
Joined: 06 Mar 2012, 19:00
Contact:

Re: LVM2 on raid6 partition recovery

#14 Post by MRovis »

Official archive with Neil Brown's suggestion in similar situation is this.
http://www.spinics.net/lists/raid/msg34840.html
And yes, I'll look deeper yet, and then ask what I will be intent on issueing
at the command line on the linux-raid mailing list, to which I just subscribed:
http://vger.kernel.org/vger-lists.html#linux-raid
I'd better spend a few more hours, but get my 532 GiB data partition back!
Even though I could restore each of the 5 drives in the state back when I dumped
their images, that is some two or more days' worth of work, plus redoing some of the
story that can be seen above, so this deep thinking is faster appoach, right? ;)

MRovis
Posts: 18
Joined: 06 Mar 2012, 19:00
Contact:

Re: LVM2 on raid6 partition recovery

#15 Post by MRovis »

I didn't wait for so long...
But I will inform with a link to this thread other users at the linux-raid mailing list that I mention in previous post, because I believe it can be of use to other mdadm users.
Here's how the story is hopefully nearing a fine ending.
In case there is no power outage (I owned a UPS, but it's... dead
now after all the years of use...), I might have recovered my data.
No, I'm not touching anything in my at8-g200 box, where the
resync is going on of /dev/md127.
And it will take long to hopefully accomplish.
Have a look how it went.

Code: Select all

at8-g200@sysresccd % mdadm -Q  /dev/sda6  
/dev/sda6: is not an md array
/dev/sda6: device 2 in 5 device undetected raid6 /dev/.tmp.md127.  Use mdadm --examine for more detail.
at8-g200@sysresccd %
OK. So --examine (-E) we will.
Here it is for all the members of this data raid6,
/dev/sda6 through /dev/sdf6, or /dev/sd[abcdf].
I don't understand why all but /dev/sda6 show at least one "faulty removed" member.
I can be back to read the explanation if anyone can tell us some time in the future.
Since I was given a lot by people like CG Security's author, through my Linux years,
when I can, I also explain things for the next Linux users.
For that same reason, while my data raid6 partition is resyncing (I really expect it to allow
the LVM2 to recognize it and to get my data back just fine), I'll conclude this narrative,
because I believe it might be useful to someone else in the future, just like Chad Walker's plea to which Neil Brown replied (I gave link further above) and he got his data back was useful to me.

Code: Select all

at8-g200@sysresccd % mdadm -E /dev/sda6
/dev/sda6:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 03a9e99f:6af97c47:82dd0df0:537ac377
  Creation Time : Fri Dec  3 17:28:25 2010
     Raid Level : raid6
  Used Dev Size : 186257408 (177.63 GiB 190.73 GB)
     Array Size : 558772224 (532.89 GiB 572.18 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 127

    Update Time : Thu Mar  1 10:58:46 2012
          State : active
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 958ba4bc - correct
         Events : 36125

         Layout : left-symmetric
     Chunk Size : 128K

      Number   Major   Minor   RaidDevice State
this     2       8        6        2      active sync   /dev/sda6

   0     0       8       86        0      active sync   /dev/sdf6
   1     1       8       54        1      active sync   /dev/sdd6
   2     2       8        6        2      active sync   /dev/sda6
   3     3       8       22        3      active sync   /dev/sdb6
   4     4       8       38        4      active sync   /dev/sdc6
at8-g200@sysresccd % mdadm -E /dev/sdb6
/dev/sdb6:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 03a9e99f:6af97c47:82dd0df0:537ac377
  Creation Time : Fri Dec  3 17:28:25 2010
     Raid Level : raid6
  Used Dev Size : 186257408 (177.63 GiB 190.73 GB)
     Array Size : 558772224 (532.89 GiB 572.18 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 127

    Update Time : Thu Mar  1 11:11:03 2012
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 3
  Spare Devices : 0
       Checksum : 958c3502 - correct
         Events : 36129

         Layout : left-symmetric
     Chunk Size : 128K

      Number   Major   Minor   RaidDevice State
this     3       8       22        3      active sync   /dev/sdb6

   0     0       8       86        0      active sync   /dev/sdf6
   1     1       0        0        1      faulty removed
   2     2       0        0        2      faulty removed
   3     3       8       22        3      active sync   /dev/sdb6
   4     4       0        0        4      faulty removed
at8-g200@sysresccd % mdadm -E /dev/sdc6
/dev/sdc6:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 03a9e99f:6af97c47:82dd0df0:537ac377
  Creation Time : Fri Dec  3 17:28:25 2010
     Raid Level : raid6
  Used Dev Size : 186257408 (177.63 GiB 190.73 GB)
     Array Size : 558772224 (532.89 GiB 572.18 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 127

    Update Time : Thu Mar  1 11:04:19 2012
          State : active
 Active Devices : 4
Working Devices : 4
 Failed Devices : 1
  Spare Devices : 0
       Checksum : 958ba63b - correct
         Events : 36126

         Layout : left-symmetric
     Chunk Size : 128K

      Number   Major   Minor   RaidDevice State
this     4       8       38        4      active sync   /dev/sdc6

   0     0       8       86        0      active sync   /dev/sdf6
   1     1       8       54        1      active sync   /dev/sdd6
   2     2       0        0        2      faulty removed
   3     3       8       22        3      active sync   /dev/sdb6
   4     4       8       38        4      active sync   /dev/sdc6
at8-g200@sysresccd % mdadm -E /dev/sdd6
/dev/sdd6:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 03a9e99f:6af97c47:82dd0df0:537ac377
  Creation Time : Fri Dec  3 17:28:25 2010
     Raid Level : raid6
  Used Dev Size : 186257408 (177.63 GiB 190.73 GB)
     Array Size : 558772224 (532.89 GiB 572.18 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 127

    Update Time : Thu Mar  1 11:04:19 2012
          State : active
 Active Devices : 4
Working Devices : 4
 Failed Devices : 1
  Spare Devices : 0
       Checksum : 958ba645 - correct
         Events : 36126

         Layout : left-symmetric
     Chunk Size : 128K

      Number   Major   Minor   RaidDevice State
this     1       8       54        1      active sync   /dev/sdd6

   0     0       8       86        0      active sync   /dev/sdf6
   1     1       8       54        1      active sync   /dev/sdd6
   2     2       0        0        2      faulty removed
   3     3       8       22        3      active sync   /dev/sdb6
   4     4       8       38        4      active sync   /dev/sdc6
at8-g200@sysresccd % mdadm -E /dev/sdf6
/dev/sdf6:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 03a9e99f:6af97c47:82dd0df0:537ac377
  Creation Time : Fri Dec  3 17:28:25 2010
     Raid Level : raid6
  Used Dev Size : 186257408 (177.63 GiB 190.73 GB)
     Array Size : 558772224 (532.89 GiB 572.18 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 127

    Update Time : Thu Mar  1 11:11:03 2012
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 3
  Spare Devices : 0
       Checksum : 958c353c - correct
         Events : 36129

         Layout : left-symmetric
     Chunk Size : 128K

      Number   Major   Minor   RaidDevice State
this     0       8       86        0      active sync   /dev/sdf6

   0     0       8       86        0      active sync   /dev/sdf6
   1     1       0        0        1      faulty removed
   2     2       0        0        2      faulty removed
   3     3       8       22        3      active sync   /dev/sdb6
   4     4       0        0        4      faulty removed
at8-g200@sysresccd %
There we need to start from /dev/md127 stopped, as it is below (and not showing
in /dev/mdstat), /dev/md127.

Code: Select all

at8-g200@sysresccd % mdadm -E /dev/md127
mdadm: cannot open /dev/md127: No such file or directory

at8-g200@sysresccd % cat /proc/mdstat   
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md126 : active raid5 sda5[0] sdf5[4] sdd5[3] sdc5[2] sdb5[1]
      9638400 blocks level 5, 128k chunk, algorithm 2 [5/5] [UUUUU]
      
md125 : active raid6 sdb2[3] sdf2[0] sdd2[1] sdc2[4] sda2[2]
      19277568 blocks level 6, 128k chunk, algorithm 2 [5/5] [UUUUU]
      
unused devices: <none>
at8-g200@sysresccd %

Code: Select all

at8-g200@sysresccd % mdadm -A /dev/md127
mdadm: /dev/md127 not identified in config file.
at8-g200@sysresccd %
Sure enough. It's not in /proc/mdstat.

Code: Select all

at8-g200@sysresccd % mdadm -A /dev/md127 /dev/sd[abcdf]6
mdadm: /dev/md127 assembled from 2 drives - not enough to start the array.
at8-g200@sysresccd %
That was not very explanatory. So "-v" is added from now on.

Code: Select all

at8-g200@sysresccd % mdadm -v -A /dev/md127 /dev/sd[abcdf]6
mdadm: looking for devices for /dev/md127
mdadm: cannot open device /dev/sda6: Device or resource busy
mdadm: /dev/sda6 has no superblock - assembly aborted
at8-g200@sysresccd %
Yeah. Forgot to stop it. Now stopping it first.

Code: Select all

at8-g200@sysresccd % mdadm -S /dev/md127                 
mdadm: stopped /dev/md127
at8-g200@sysresccd %
Here we'll see how the friend mdadm is more talkative.
Attempting to assemle all the members first.

Code: Select all

at8-g200@sysresccd % mdadm -v -A /dev/md127 /dev/sd[abcdf]6
mdadm: looking for devices for /dev/md127
mdadm: /dev/sda6 is identified as a member of /dev/md127, slot 2.
mdadm: /dev/sdb6 is identified as a member of /dev/md127, slot 3.
mdadm: /dev/sdc6 is identified as a member of /dev/md127, slot 4.
mdadm: /dev/sdd6 is identified as a member of /dev/md127, slot 1.
mdadm: /dev/sdf6 is identified as a member of /dev/md127, slot 0.
mdadm: added /dev/sdd6 to /dev/md127 as 1
mdadm: added /dev/sda6 to /dev/md127 as 2
mdadm: added /dev/sdb6 to /dev/md127 as 3
mdadm: added /dev/sdc6 to /dev/md127 as 4
mdadm: added /dev/sdf6 to /dev/md127 as 0
mdadm: /dev/md127 assembled from 2 drives - not enough to start the array.
at8-g200@sysresccd %
How this comes about, and why, I have no idea.

Code: Select all

at8-g200@sysresccd % mdadm -S /dev/md127                   
mdadm: stopped /dev/md127
at8-g200@sysresccd %
Doubling the verbosity below doesn't seem to say any more. No it doesn't.

Code: Select all

at8-g200@sysresccd % mdadm -v -v -A /dev/md127 /dev/sd[abcdf]6
mdadm: looking for devices for /dev/md127
mdadm: /dev/sda6 is identified as a member of /dev/md127, slot 2.
mdadm: /dev/sdb6 is identified as a member of /dev/md127, slot 3.
mdadm: /dev/sdc6 is identified as a member of /dev/md127, slot 4.
mdadm: /dev/sdd6 is identified as a member of /dev/md127, slot 1.
mdadm: /dev/sdf6 is identified as a member of /dev/md127, slot 0.
mdadm: added /dev/sdd6 to /dev/md127 as 1
mdadm: added /dev/sda6 to /dev/md127 as 2
mdadm: added /dev/sdb6 to /dev/md127 as 3
mdadm: added /dev/sdc6 to /dev/md127 as 4
mdadm: added /dev/sdf6 to /dev/md127 as 0
mdadm: /dev/md127 assembled from 2 drives - not enough to start the array.
at8-g200@sysresccd % mdadm -Ss /dev/md127                     
mdadm: stopped /dev/md127
at8-g200@sysresccd %
Let's try different combinations.
Pls. note that there is no rationale behind my different attempts.
Only mere guessing.
I think I have to leave the bash outputs below just as they were.
I like to be positive in my analyses, and not leave things for guessing.

Code: Select all

at8-g200@sysresccd % mdadm -v -v -A /dev/md127 /dev/sd[bcdf]6 
mdadm: looking for devices for /dev/md127
mdadm: /dev/sdb6 is identified as a member of /dev/md127, slot 3.
mdadm: /dev/sdc6 is identified as a member of /dev/md127, slot 4.
mdadm: /dev/sdd6 is identified as a member of /dev/md127, slot 1.
mdadm: /dev/sdf6 is identified as a member of /dev/md127, slot 0.
mdadm: added /dev/sdd6 to /dev/md127 as 1
mdadm: no uptodate device for slot 2 of /dev/md127
mdadm: added /dev/sdb6 to /dev/md127 as 3
mdadm: added /dev/sdc6 to /dev/md127 as 4
mdadm: added /dev/sdf6 to /dev/md127 as 0
mdadm: /dev/md127 assembled from 2 drives - not enough to start the array.
at8-g200@sysresccd % mdadm -Ss /dev/md127                    
mdadm: stopped /dev/md127
at8-g200@sysresccd %

Code: Select all

at8-g200@sysresccd % mdadm -v -v -A /dev/md127 /dev/sd[acdf]6
mdadm: looking for devices for /dev/md127
mdadm: /dev/sda6 is identified as a member of /dev/md127, slot 2.
mdadm: /dev/sdc6 is identified as a member of /dev/md127, slot 4.
mdadm: /dev/sdd6 is identified as a member of /dev/md127, slot 1.
mdadm: /dev/sdf6 is identified as a member of /dev/md127, slot 0.
mdadm: added /dev/sdd6 to /dev/md127 as 1
mdadm: added /dev/sda6 to /dev/md127 as 2
mdadm: no uptodate device for slot 3 of /dev/md127
mdadm: added /dev/sdc6 to /dev/md127 as 4
mdadm: added /dev/sdf6 to /dev/md127 as 0
mdadm: /dev/md127 assembled from 1 drive - not enough to start the array.
at8-g200@sysresccd % mdadm -Ss /dev/md127                    
mdadm: stopped /dev/md127
at8-g200@sysresccd %

Code: Select all

at8-g200@sysresccd % mdadm -v -v -A /dev/md127 /dev/sd[abdf]6 
mdadm: looking for devices for /dev/md127
mdadm: /dev/sda6 is identified as a member of /dev/md127, slot 2.
mdadm: /dev/sdb6 is identified as a member of /dev/md127, slot 3.
mdadm: /dev/sdd6 is identified as a member of /dev/md127, slot 1.
mdadm: /dev/sdf6 is identified as a member of /dev/md127, slot 0.
mdadm: added /dev/sdd6 to /dev/md127 as 1
mdadm: added /dev/sda6 to /dev/md127 as 2
mdadm: added /dev/sdb6 to /dev/md127 as 3
mdadm: no uptodate device for slot 4 of /dev/md127
mdadm: added /dev/sdf6 to /dev/md127 as 0
mdadm: /dev/md127 assembled from 2 drives - not enough to start the array.
at8-g200@sysresccd % mdadm -Ss /dev/md127                    
mdadm: stopped /dev/md127
at8-g200@sysresccd %

Code: Select all

at8-g200@sysresccd % mdadm -v -v -A /dev/md127 /dev/sd[abcf]6
mdadm: looking for devices for /dev/md127
mdadm: /dev/sda6 is identified as a member of /dev/md127, slot 2.
mdadm: /dev/sdb6 is identified as a member of /dev/md127, slot 3.
mdadm: /dev/sdc6 is identified as a member of /dev/md127, slot 4.
mdadm: /dev/sdf6 is identified as a member of /dev/md127, slot 0.
mdadm: no uptodate device for slot 1 of /dev/md127
mdadm: added /dev/sda6 to /dev/md127 as 2
mdadm: added /dev/sdb6 to /dev/md127 as 3
mdadm: added /dev/sdc6 to /dev/md127 as 4
mdadm: added /dev/sdf6 to /dev/md127 as 0
mdadm: /dev/md127 assembled from 2 drives - not enough to start the array.
at8-g200@sysresccd % mdadm -Ss /dev/md127                    
mdadm: stopped /dev/md127
at8-g200@sysresccd %

Code: Select all

at8-g200@sysresccd % mdadm -v -v -A /dev/md127 /dev/sd[abcd]6
mdadm: looking for devices for /dev/md127
mdadm: /dev/sda6 is identified as a member of /dev/md127, slot 2.
mdadm: /dev/sdb6 is identified as a member of /dev/md127, slot 3.
mdadm: /dev/sdc6 is identified as a member of /dev/md127, slot 4.
mdadm: /dev/sdd6 is identified as a member of /dev/md127, slot 1.
mdadm: no uptodate device for slot 0 of /dev/md127
mdadm: added /dev/sdd6 to /dev/md127 as 1
mdadm: added /dev/sda6 to /dev/md127 as 2
mdadm: added /dev/sdc6 to /dev/md127 as 4
mdadm: added /dev/sdb6 to /dev/md127 as 3
mdadm: /dev/md127 assembled from 1 drive - not enough to start the array.
at8-g200@sysresccd % mdadm -Ss /dev/md127                    
mdadm: stopped /dev/md127
at8-g200@sysresccd % mdadm -v -v -A /dev/md127 /dev/sd[bcf]6
mdadm: looking for devices for /dev/md127
mdadm: /dev/sdb6 is identified as a member of /dev/md127, slot 3.
mdadm: /dev/sdc6 is identified as a member of /dev/md127, slot 4.
mdadm: /dev/sdf6 is identified as a member of /dev/md127, slot 0.
mdadm: no uptodate device for slot 1 of /dev/md127
mdadm: no uptodate device for slot 2 of /dev/md127
mdadm: added /dev/sdb6 to /dev/md127 as 3
mdadm: added /dev/sdc6 to /dev/md127 as 4
mdadm: added /dev/sdf6 to /dev/md127 as 0
mdadm: /dev/md127 assembled from 2 drives - not enough to start the array.
at8-g200@sysresccd % mdadm -v -v -A /dev/md127 /dev/sd[acf]6 
mdadm: looking for devices for /dev/md127
mdadm: cannot open device /dev/sdc6: Device or resource busy
mdadm: /dev/sdc6 has no superblock - assembly aborted
at8-g200@sysresccd % mdadm -Ss /dev/md127                   
mdadm: stopped /dev/md127
at8-g200@sysresccd % mdadm -v -v -A /dev/md127 /dev/sd[acf]6
mdadm: looking for devices for /dev/md127
mdadm: /dev/sda6 is identified as a member of /dev/md127, slot 2.
mdadm: /dev/sdc6 is identified as a member of /dev/md127, slot 4.
mdadm: /dev/sdf6 is identified as a member of /dev/md127, slot 0.
mdadm: no uptodate device for slot 1 of /dev/md127
mdadm: added /dev/sda6 to /dev/md127 as 2
mdadm: no uptodate device for slot 3 of /dev/md127
mdadm: added /dev/sdc6 to /dev/md127 as 4
mdadm: added /dev/sdf6 to /dev/md127 as 0
mdadm: /dev/md127 assembled from 1 drive - not enough to start the array.
at8-g200@sysresccd % mdadm -Ss /dev/md127                   
mdadm: stopped /dev/md127
at8-g200@sysresccd % mdadm -v -v -A /dev/md127 /dev/sd[abf]6
mdadm: looking for devices for /dev/md127
mdadm: /dev/sda6 is identified as a member of /dev/md127, slot 2.
mdadm: /dev/sdb6 is identified as a member of /dev/md127, slot 3.
mdadm: /dev/sdf6 is identified as a member of /dev/md127, slot 0.
mdadm: no uptodate device for slot 1 of /dev/md127
mdadm: added /dev/sda6 to /dev/md127 as 2
mdadm: added /dev/sdb6 to /dev/md127 as 3
mdadm: no uptodate device for slot 4 of /dev/md127
mdadm: added /dev/sdf6 to /dev/md127 as 0
mdadm: /dev/md127 assembled from 2 drives - not enough to start the array.
at8-g200@sysresccd % mdadm -Ss /dev/md127                   
mdadm: stopped /dev/md127
at8-g200@sysresccd %

Code: Select all

at8-g200@sysresccd % mdadm -v -v -A /dev/md127 /dev/sd[adf]6
mdadm: looking for devices for /dev/md127
mdadm: /dev/sda6 is identified as a member of /dev/md127, slot 2.
mdadm: /dev/sdd6 is identified as a member of /dev/md127, slot 1.
mdadm: /dev/sdf6 is identified as a member of /dev/md127, slot 0.
mdadm: added /dev/sdd6 to /dev/md127 as 1
mdadm: added /dev/sda6 to /dev/md127 as 2
mdadm: no uptodate device for slot 3 of /dev/md127
mdadm: no uptodate device for slot 4 of /dev/md127
mdadm: added /dev/sdf6 to /dev/md127 as 0
mdadm: /dev/md127 assembled from 1 drive - not enough to start the array.
at8-g200@sysresccd % mdadm -Ss /dev/md127                   
mdadm: stopped /dev/md127
at8-g200@sysresccd %

Code: Select all

at8-g200@sysresccd % mdadm -v -v -A /dev/md127 /dev/sd[abc]6
mdadm: looking for devices for /dev/md127
mdadm: /dev/sda6 is identified as a member of /dev/md127, slot 2.
mdadm: /dev/sdb6 is identified as a member of /dev/md127, slot 3.
mdadm: /dev/sdc6 is identified as a member of /dev/md127, slot 4.
mdadm: no uptodate device for slot 0 of /dev/md127
mdadm: no uptodate device for slot 1 of /dev/md127
mdadm: added /dev/sda6 to /dev/md127 as 2
mdadm: added /dev/sdc6 to /dev/md127 as 4
mdadm: added /dev/sdb6 to /dev/md127 as 3
mdadm: /dev/md127 assembled from 1 drive - not enough to start the array.
at8-g200@sysresccd % mdadm -Ss /dev/md127                   
mdadm: stopped /dev/md127
at8-g200@sysresccd %

Code: Select all

at8-g200@sysresccd % mdadm -v -v -A /dev/md127 /dev/sd[abd]6
mdadm: looking for devices for /dev/md127
mdadm: /dev/sda6 is identified as a member of /dev/md127, slot 2.
mdadm: /dev/sdb6 is identified as a member of /dev/md127, slot 3.
mdadm: /dev/sdd6 is identified as a member of /dev/md127, slot 1.
mdadm: no uptodate device for slot 0 of /dev/md127
mdadm: added /dev/sdd6 to /dev/md127 as 1
mdadm: added /dev/sda6 to /dev/md127 as 2
mdadm: no uptodate device for slot 4 of /dev/md127
mdadm: added /dev/sdb6 to /dev/md127 as 3
mdadm: /dev/md127 assembled from 1 drive - not enough to start the array.
at8-g200@sysresccd % mdadm -Ss /dev/md127                   
mdadm: stopped /dev/md127
at8-g200@sysresccd %
And here comes the stroke of luck.
Of all the combinations above, and I don't know if there are other possible
combinations, won't spend any more time to investigate... Hey this has already
ruined my schedule as is!
This following combination is the only one that looks capable to be resuscitated.

Code: Select all

at8-g200@sysresccd % mdadm -v -v -A /dev/md127 /dev/sd[acd]6
mdadm: looking for devices for /dev/md127
mdadm: /dev/sda6 is identified as a member of /dev/md127, slot 2.
mdadm: /dev/sdc6 is identified as a member of /dev/md127, slot 4.
mdadm: /dev/sdd6 is identified as a member of /dev/md127, slot 1.
mdadm: device 2 in /dev/md127 has wrong state in superblock, but /dev/sda6 seems ok
mdadm: no uptodate device for slot 0 of /dev/md127
mdadm: added /dev/sda6 to /dev/md127 as 2
mdadm: no uptodate device for slot 3 of /dev/md127
mdadm: added /dev/sdc6 to /dev/md127 as 4
mdadm: added /dev/sdd6 to /dev/md127 as 1
mdadm: /dev/md127 assembled from 3 drives - not enough to start the array while not clean - consider --force.
at8-g200@sysresccd %
That's the Neil Brown's suggestion. Only enshrined in the code, not live as for Chad Walker!

Code: Select all

at8-g200@sysresccd % mdadm -v -v -A /dev/md127 /dev/sd[acd]6 --forceidx
mdadm: unrecognized option '--forceidx'
Usage: mdadm --help
  for help
at8-g200@sysresccd %
That's me being a little confused by the events. Of course, it's just "force", not "forceidx".

Code: Select all

at8-g200@sysresccd % mdadm -v -v -A /dev/md127 /dev/sd[acd]6 --force   
mdadm: looking for devices for /dev/md127
mdadm: cannot open device /dev/sda6: Device or resource busy
mdadm: /dev/sda6 has no superblock - assembly aborted
at8-g200@sysresccd %
Of course, Miro the Confused forgot to stop the array.

Code: Select all

at8-g200@sysresccd % mdadm -Ss /dev/md127                         
mdadm: stopped /dev/md127
at8-g200@sysresccd %
Here we go. You ain't getting away from me just yet. Be good, dear friend mdadm!

Code: Select all

at8-g200@sysresccd % mdadm -v -v -A /dev/md127 /dev/sd[acd]6 --force
mdadm: looking for devices for /dev/md127
mdadm: /dev/sda6 is identified as a member of /dev/md127, slot 2.
mdadm: /dev/sdc6 is identified as a member of /dev/md127, slot 4.
mdadm: /dev/sdd6 is identified as a member of /dev/md127, slot 1.
mdadm: clearing FAULTY flag for device 0 in /dev/md127 for /dev/sda6
mdadm: no uptodate device for slot 0 of /dev/md127
mdadm: added /dev/sda6 to /dev/md127 as 2
mdadm: no uptodate device for slot 3 of /dev/md127
mdadm: added /dev/sdc6 to /dev/md127 as 4
mdadm: added /dev/sdd6 to /dev/md127 as 1
mdadm: /dev/md127 has been started with 3 drives (out of 5).
at8-g200@sysresccd %
Right!

Code: Select all

at8-g200@sysresccd % mdadm -D /dev/md127                        
/dev/md127:
        Version : 0.90
  Creation Time : Fri Dec  3 17:28:25 2010
     Raid Level : raid6
     Array Size : 558772224 (532.89 GiB 572.18 GB)
  Used Dev Size : 186257408 (177.63 GiB 190.73 GB)
   Raid Devices : 5
  Total Devices : 3
Preferred Minor : 127
    Persistence : Superblock is persistent

    Update Time : Thu Mar  1 11:04:19 2012
          State : clean, degraded
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 128K

           UUID : 03a9e99f:6af97c47:82dd0df0:537ac377
         Events : 0.36126

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8       54        1      active sync   /dev/sdd6
       2       8        6        2      active sync   /dev/sda6
       3       0        0        3      removed
       4       8       38        4      active sync   /dev/sdc6
at8-g200@sysresccd %
Sure enough, now just add those two remaining members.

Code: Select all

at8-g200@sysresccd % mdadm  /dev/md127 -a /dev/sdb6
mdadm: re-added /dev/sdb6
at8-g200@sysresccd % mdadm  /dev/md127 -a /dev/sdf6
mdadm: added /dev/sdf6
at8-g200@sysresccd %
Voilà! The raid6 important data partition is alive.

Code: Select all

at8-g200@sysresccd % mdadm -D /dev/md127           
/dev/md127:
        Version : 0.90
  Creation Time : Fri Dec  3 17:28:25 2010
     Raid Level : raid6
     Array Size : 558772224 (532.89 GiB 572.18 GB)
  Used Dev Size : 186257408 (177.63 GiB 190.73 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 127
    Persistence : Superblock is persistent

    Update Time : Thu Mar  8 09:31:59 2012
          State : clean, degraded, recovering
 Active Devices : 3
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 2

         Layout : left-symmetric
     Chunk Size : 128K

 Rebuild Status : 0% complete

           UUID : 03a9e99f:6af97c47:82dd0df0:537ac377
         Events : 0.36131

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8       54        1      active sync   /dev/sdd6
       2       8        6        2      active sync   /dev/sda6
       6       8       22        3      spare rebuilding   /dev/sdb6
       4       8       38        4      active sync   /dev/sdc6

       5       8       86        -      spare   /dev/sdf6
at8-g200@sysresccd %

Code: Select all

root@sysresccd /root % mdadm -D /dev/md127 
/dev/md127:
        Version : 0.90
  Creation Time : Fri Dec  3 17:28:25 2010
     Raid Level : raid6
     Array Size : 558772224 (532.89 GiB 572.18 GB)
  Used Dev Size : 186257408 (177.63 GiB 190.73 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 127
    Persistence : Superblock is persistent

    Update Time : Thu Mar  8 10:10:44 2012
          State : clean, degraded, recovering
 Active Devices : 3
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 2

         Layout : left-symmetric
     Chunk Size : 128K

 Rebuild Status : 54% complete

           UUID : 03a9e99f:6af97c47:82dd0df0:537ac377
         Events : 0.36149

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8       54        1      active sync   /dev/sdd6
       2       8        6        2      active sync   /dev/sda6
       6       8       22        3      spare rebuilding   /dev/sdb6
       4       8       38        4      active sync   /dev/sdc6

       5       8       86        -      spare   /dev/sdf6
root@sysresccd /root %
In the half hour or more, since I began this current post, that's where we're at.
I'll be back just to tell you if LVM2 found its volume. Because the filesystem is on the LVM2 on top of /dev/md127.

MRovis
Posts: 18
Joined: 06 Mar 2012, 19:00
Contact:

Re: LVM2 on raid6 partition recovery

#16 Post by MRovis »

Well, now that there is one redundant drive synced, we can talk.

Code: Select all

at8-g200@sysresccd %  mdadm -D /dev/md127      
/dev/md127:
        Version : 0.90
  Creation Time : Fri Dec  3 17:28:25 2010
     Raid Level : raid6
     Array Size : 558772224 (532.89 GiB 572.18 GB)
  Used Dev Size : 186257408 (177.63 GiB 190.73 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 127
    Persistence : Superblock is persistent

    Update Time : Thu Mar  8 10:54:10 2012
          State : clean, degraded, recovering
 Active Devices : 4
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 128K

 Rebuild Status : 0% complete

           UUID : 03a9e99f:6af97c47:82dd0df0:537ac377
         Events : 0.36313

    Number   Major   Minor   RaidDevice State
       5       8       86        0      spare rebuilding   /dev/sdf6
       1       8       54        1      active sync   /dev/sdd6
       2       8        6        2      active sync   /dev/sda6
       3       8       22        3      active sync   /dev/sdb6
       4       8       38        4      active sync   /dev/sdc6
at8-g200@sysresccd %
As expected, the volume group...

Code: Select all

at8-g200@sysresccd % vgscan               
  Reading all physical volumes.  This may take a while...
  Found volume group "vg" using metadata type lvm2
at8-g200@sysresccd %
...and the logical volume..

Code: Select all

at8-g200@sysresccd % vgchange -ay              
  1 logical volume(s) in volume group "vg" now active
at8-g200@sysresccd %
...are there, and..

Code: Select all

at8-g200@sysresccd % mkdir /mnt/Cmn
at8-g200@sysresccd %
...will be...

Code: Select all

at8-g200@sysresccd % mount /dev/vg/Cmn /mnt/Cmn 
at8-g200@sysresccd %
mounted.

Code: Select all

at8-g200@sysresccd % ls -l  /mnt/Cmn           
total 2099252
drwxr-xr-x  3 root root        4096 Jan  9  2011 Audacity
drwxr-xr-x 10 1000  1000       4096 Feb 23 05:47 dLo
drwxr-xr-x  3 root root        4096 Jun 28  2011 DVD-CD.img
drwxr-xr-x  2 1000 users       4096 Feb 18 17:49 dwhelper
drwxr-xr-x  2 1000  1000       4096 Feb 27 04:58 Kaff
drwxr-xr-x  2 root root        4096 Dec 27  2010 lost+found
drwxr-xr-x  3 1000  1000       4096 Jan 26 12:27 m
-rwxr-xr-x  1 root root        1130 Nov 15 23:35 mdadm2old_at8-g250-c.sh
drwxr-xr-x 39 1000  1000      12288 Feb 28 14:50 MyVideos
-rw-r--r--  1 root root  2147483648 Dec  8 10:12 swapfile
drwxr-xr-x  4 1000  1000       4096 Dec 13 22:16 ThBird
at8-g200@sysresccd %
Data there! Recovered!

Code: Select all

at8-g200@sysresccd % df -h  /mnt/Cmn
Filesystem          Size  Used Avail Use% Mounted on
/dev/mapper/vg-Cmn  524G  426G   72G  86% /mnt/Cmn
at8-g200@sysresccd %
That would've been a lot to lose, right?
And what to think about people who lose their livelihood for political imprisonment?
What for the Syrians persecuted and killed in the first place, by their own rulers,
supported by world still somewhat sickly Russia, for the remnants of the monstrous communist
way of thinking and doing (so is my country Croatia, plagued by bolsheviks turned capitaists... on power)...
That is even worse, to lose lives, your own or your dearests', but also to lose livelihood is bad enough.
I can only pray for the Syrians, and for the political prisoners and politically persecuted (of which there are in my country of the former and of the latter --some are friends of mine--, and for that matter I might become one, but also in the Western countries, only it is better camouflaged than on the East).
And thinking of what God gives us, I am grateful, for my part.
God bless everyone who will be reading this!
Miroslav Rovis
Croatia
miro.rovis@gmail.com
m.rovis@inet.hr
01 660 2633
091 266 0202
http://www.CroatiaFidelis.hr (no English at this time)
http://www.exDeo.com (some English there)
http://groups.google.com/group/croatian-news/ (very little English there)
http://www.youtube.com/user/miroR2 (videos, so...)
What will come of this one:
http://www.youtube.com/user/prosvjednikkrcmarek (not mine, but a friend's profile, where I put lots of stuff when Google banned me to post videos longer than 15 minutes)
Here some English stuff too:
http://vimeo.com/user9621785

MRovis
Posts: 18
Joined: 06 Mar 2012, 19:00
Contact:

Re: LVM2 on raid6 partition recovery

#17 Post by MRovis »

cgrenier wrote:What gives

Code: Select all

mdadm --run /dev/md127
?
I am really sorry. I only right now, at this time in Central Europe:

Code: Select all

miro@at8-g250-c /  $ date
Thu Mar  8 12:26:15 CET 2012
miro@at8-g250-c / $ 
noticed your post!
Pls., I would've replied if I saw it earlier!
Thank you for the insight testdik gave me, although the route to recovery proves rightly different to take in my particular case.
I'm also afraid it is irrelevant to try your suggestion as it is all over.
And thanks for the nice Forum!

MRovis
Posts: 18
Joined: 06 Mar 2012, 19:00
Contact:

Re: LVM2 on raid6 partition recovery

#18 Post by MRovis »

Here, I point linux-raid users:
http://marc.info/?l=linux-raid&m=133121914429398&w=2
to this saga here that I wrote recovering my data on this forum.
Because, I believe there are questions as to how to prevent a loss like this one that nearly hit me hard!
I am dead tired.
So tired there is "a" missing for "Croatia" at end of that mail to linux-raid mailing list.

Locked