2

I have a raid5 array on 4 disks, sda1, sdb1, sdd1 and sde1. sdd1 was removed but is a healthy drive. While attempting to re-add it, I accidentally zeroed out the superblock for sdb1.

sdb1's data is consistent with sda1 and sde1, just missing it's superblock.

Is it possible to recover the superblock on that device and reassemble the array (degraded) without losing any data?

Below is the output of 'mdadm -E /dev/sd{a,d,e}1'.

Am I reading the output correctly in that sde1 is Device 0, sdd1 is Device 2 and sda1 is Device 3? Then that would mean that sdb1 was Device 1. Could I recreate the array with the following command without data loss:

$ sudo mdadm --create /dev/md0 --assume-clean --level=5 --raid-devices=4 /dev/sde1 /dev/sdb1 missing /dev/sda1

/dev/sda1: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : 81a36846:cf4f0489:219e2546:b1f5b90e Name : cowbell:0 Creation Time : Sun Sep 25 20:24:46 2011 Raid Level : raid5 Raid Devices : 4 Avail Dev Size : 3907025920 (1863.02 GiB 2000.40 GB) Array Size : 5860538880 (5589.05 GiB 6001.19 GB) Data Offset : 1024 sectors Super Offset : 8 sectors State : clean Device UUID : a0abbc72:f8bb1e2b:e8f81f5c:ed62a979 Internal Bitmap : 8 sectors from superblock Update Time : Tue Jul 23 18:05:09 2013 Checksum : 9f46a56 - correct Events : 717820 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 3 Array State : AA.A ('A' == active, '.' == missing) /dev/sdd1: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : 81a36846:cf4f0489:219e2546:b1f5b90e Name : cowbell:0 Creation Time : Sun Sep 25 20:24:46 2011 Raid Level : raid5 Raid Devices : 4 Avail Dev Size : 3907027053 (1863.02 GiB 2000.40 GB) Array Size : 5860538880 (5589.05 GiB 6001.19 GB) Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB) Data Offset : 2048 sectors Super Offset : 8 sectors State : active Device UUID : 79f5632d:46d5c083:a1c0130c:b83b0654 Internal Bitmap : 8 sectors from superblock Update Time : Tue Jul 23 17:42:02 2013 Checksum : a78d6f5b - correct Events : 717392 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 2 Array State : AAAA ('A' == active, '.' == missing) /dev/sde1: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : 81a36846:cf4f0489:219e2546:b1f5b90e Name : cowbell:0 Creation Time : Sun Sep 25 20:24:46 2011 Raid Level : raid5 Raid Devices : 4 Avail Dev Size : 3907027053 (1863.02 GiB 2000.40 GB) Array Size : 5860538880 (5589.05 GiB 6001.19 GB) Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB) Data Offset : 2048 sectors Super Offset : 8 sectors State : clean Device UUID : 6abd9160:c8ae382c:e6c73d01:37ac057b Internal Bitmap : 8 sectors from superblock Update Time : Tue Jul 23 18:05:09 2013 Checksum : 9f5d8fa6 - correct Events : 717820 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 0 Array State : AA.A ('A' == active, '.' == missing)
Improve this question
2
  • Before you do anything, have you made a backup? Commented Jul 24, 2013 at 17:36
  • I have a backup of the important things, yes. There are a bunch of things I'd rather not lose, but wouldn't be the end of the world if I did. Commented Jul 24, 2013 at 18:23

2 Answers 2

Reset to default
2

In addition to backups, you might also want to consider just making full dd copies of the disks before attempting any recovery.

That being said, looks like you're on the right track for recovery without data loss. You're interpreting the device numbers correctly. That command looks like what you need.

See my answer here - these things are surprisingly resilient against data loss, and the destruction of the superblock is not damaging to the ability to rebuild the array in the same geometry (see test 4 in that answer).

Improve this answer
4
  • Ok, so I executed the command above and the array started, but when I tried to mount it, I got the "wrong fs type, bad option, bad superblock on /dev/md0...". I created the array according to my output above, so I'm not sure what I did wrong. I don't recall changing any of the defaults like chunk size or anything, so what could be going wrong? Commented Jul 24, 2013 at 21:44
  • @JustinMiller Yeah, the left-symmetric layout and 512k chunk sizes are defaults, so that should match your initial build. Maybe take a look at some of the forensics work that Anton did in this answer, and compare what you're seeing on your system? Commented Jul 24, 2013 at 22:25
  • WOOOOOO! I got it mounted and all my data is there! Turns out the above command was fine, the superblocks just had the wrong data offsets. Thankfully I had the 'mdadm -E' from all but one of the drives and fortunately it only took 2 tries before I got the right one for sdb1 (it was same as one of the other drives added around the same time). I will write songs about you and Anton!!! Seriously though, the info that you both posted should, hands down, be on a blog about advanced RAID recovery. Invaluable. Commented Jul 25, 2013 at 12:44
  • @JustinMiller Awesome, great to hear it! Commented Jul 25, 2013 at 15:58
0

I would try to verify that the underlying physical disks are synced first before I continue re-creating MD device with assumtion it is clean. You can do that with 

mdadm -E /dev/sd[abc]1 | grep Event Events : 0.53120 Events : 0.53108 Events : 0.53120

The output shows that sda and sdc are synced while sdc is late.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.