2

I have a RAID 5 array built from partitions on 5 drives - /dev/sda1, /dev/sdb1, /dev/sdc1, /dev/sdd1, and /dev/sde1. When I assemble the array manually, it works fine.

Unfortunately, there seem to also be superblocks on /dev/sdb and /dev/sdc (probably something I messed up early on), and when the array assembles automatically at startup, it builds with /dev/sdb instead of /dev/sdb1 (the same with /dev/sdc), and fails to mount (with an invalid fs error). 

$ sudo mdadm --assemble --scan --force (presumably what happens on startup) mdadm: WARNING /dev/sdb1 and /dev/sdb appear to have very similar superblocks. If they are really different, please --zero the superblock on one If they are the same or overlap, please remove one from the DEVICE list in mdadm.conf. $ sudo mdadm --detail /dev/md7 /dev/md7: Version : 00.90 Creation Time : Wed Apr 6 18:17:07 2011 Raid Level : raid5 Array Size : 7814047744 (7452.06 GiB 8001.58 GB) Used Dev Size : 1953511936 (1863.01 GiB 2000.40 GB) Raid Devices : 5 Total Devices : 5 Preferred Minor : 7 Persistence : Superblock is persistent Update Time : Sat Apr 30 13:55:50 2011 State : clean Active Devices : 5 Working Devices : 5 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 64K UUID : 224009a1:e4173bf3:2490f10a:1455ce9c (local to host ravenscar) Events : 0.2 Number Major Minor RaidDevice State 0 8 1 0 active sync /dev/sda1 1 8 16 1 active sync /dev/sdb 2 8 32 2 active sync /dev/sdc 3 8 48 3 active sync /dev/sdd1 4 8 65 4 active sync /dev/sde1 $ sudo mdadm --examine /dev/sdb /dev/sdb: Magic : a92b4efc Version : 00.90.00 UUID : 224009a1:e4173bf3:2490f10a:1455ce9c (local to host ravenscar) Creation Time : Wed Apr 6 18:17:07 2011 Raid Level : raid5 Used Dev Size : 1953511936 (1863.01 GiB 2000.40 GB) Array Size : 7814047744 (7452.06 GiB 8001.58 GB) Raid Devices : 5 Total Devices : 5 Preferred Minor : 7 Update Time : Wed Apr 27 21:15:03 2011 State : clean Active Devices : 5 Working Devices : 5 Failed Devices : 0 Spare Devices : 0 Checksum : f83035d3 - correct Events : 2 Layout : left-symmetric Chunk Size : 64K Number Major Minor RaidDevice State this 1 8 17 1 active sync /dev/sdb1 0 0 8 1 0 active sync /dev/sda1 1 1 8 17 1 active sync /dev/sdb 2 2 8 33 2 active sync /dev/sdc 3 3 8 49 3 active sync /dev/sdd1 4 4 8 65 4 active sync /dev/sde1

Then I do:

$ sudo mdadm --stop /dev/md7 $ sudo mdadm --assemble /dev/md7 /dev/sd[abcde]1

And then it works.

My question: Is it safe to use the --zero-superblock option to remove those erroneous superblocks? what other measures should I take? Did I make the right choice building the array with partitions instead of drives (/dev/sda1 instead of /dev/sda, and so on).

Improve this question
1
  • What does /mdadm --examine /dev/sdb and /mdadm --examine /dev/sdb1 show? Commented Apr 30, 2011 at 17:36

4 Answers 4

Reset to default
2

Do not zero the super block. Edit as mdadm suggested /etc/mdadm.conf or /etc/mdadm/mdadm.conf and change DEVICE .... to:

DEVICE /dev/sda1 DEVICE /dev/sdb1 DEVICE /dev/sdc1 DEVICE /dev/sdd1 DEVICE /dev/sde1
Improve this answer
1

What seems to happen is MD sees the same superblock from two devices (sdb1 is a subset of sdb). Killing the superblock would kill the real superblock so don't do that.

MD v0.90 superblocks are stored at the end of the device on a 64k-aligned block. In your case sdb1 is the only partition and the superblock can be found at the end of both sdb and sdb1. Somehow (this appears to be an an old issue, see link at the end) MD got confused and started to think you have two copies of the superblock on two separate devices.

The fix is easy if you have some extra space on that partition - you can simply wipe the superbblock then repartition the disk ending the partition 64k before the end of the disk; MD won't see a suberblock on sdb and assemble the array properly using sdb1. If you cannot take out 64k from sdb, you'll have to rebuild the array. If this isn't your boot disk, you can also use a newer superblock version. v.1.1 and v.1.2 both store the superblock somewhere on the beginning of the array, which will solve this specific problem.

I've just experienced the exact same problem on a raid1. Using "fdisk -u" (use sectors as units) the partition was ending on sector 143374743. I subtracted 128 from this number (128 512-byte sectors = 64k) and used the resulting number as the end of the partition. I could then verify that MD wouldn't see the superblock on sda with:

# mdadm --misc -E /dev/sda

It seems the recent behavioral change of using sectors and dropping DOS compatibility in partitioning tools led to having the last partition always end at the very end of the disk (rather than on a cylinder boundary). Some other parameters may come into play as - for instance - I have a running system which does show the duplicate superblocks when disks are queried directly, and still mounts its array properly.

For more info on MD superblock formats, see:

https://raid.wiki.kernel.org/index.php/RAID_superblock_formats#The_version-0.90_Superblock_Format

The Debian FAQ (point 11.) mention this issue:

http://git.debian.org/?p=pkg-mdadm/mdadm.git;a=blob_plain;f=debian/FAQ;hb=HEAD

Improve this answer
0

If you don't have any data on the array I'd just wipe it and start over. If you do have data on it I'd back it up before playing around w/ the superblocks.

Personally, I'd get the data off of it and dd a few sectors of /dev/zero over the beginning of each disk, then repartition and rebuild the array. I'd feel more comfortable knowing that everything was setup soundly to begin with.

As an aside: That's a pretty big array to be doing RAID-5 on, to my mind. Keep an eye on the rebuild time. In the event of a single disk failure you'll have catastrophic data loss if you have another error while reading during rebuild. You may want to consider RAID-6.

Improve this answer
3
  • wiping isn't really an option, unfortunately. And yeah, I've heard that about raid 5. we should be fine though, right? Commented Apr 30, 2011 at 18:29
  • Your circumstance scares me. It may work fine for you, but I'd be wary. Starting a new 7TB volume off with a checkered past sounds like a formula for a bad future, even if it seems like an innocuous problem. re: a 7TB RAID-5 array - You'll be fine if you end up being fine. If the "luck" of an unrecoverable read error happens during a rebuild, someday in the future, you won't be fine. It's like winning the lottery, except in reverse. Using RAID-5 means you're buying more tickets in the "lose all my data in a rebuild lottery" than with RAID-6 or RAID-10. Commented Apr 30, 2011 at 18:34
  • The superblock can be at the beginning or at the end of the partition, depending on the mdadm version. Commented Jun 7, 2011 at 3:46
0

Unluckily I deleted the RAID-Superblock when reading following massage: 

mdadm: WARNING /dev/sdb1 and /dev/sdb appear to have very similar superblocks. If they are really different, please --zero the superblock on one If they are the same or overlap, please remove one from the DEVICE list in mdadm.conf.

Afterwards my RAID1 couldn't start any more.

Luckily, creating new RAID1 with identical parameters as original done, did not delete old one: http://lists.debian.org/debian-user-french/2006/03/msg00607.html via http://kevin.deldycke.com/2007/03/how-to-recover-a-raid-array-after-having-zero-ized-superblocks/

As partly thumb/misleading/unhelpful above mdadm message is, as smart is mdadm with recreation.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.