[ssc@ripley-nas[user@host ~]$ dmesg | egrep '^da.*?: .*?MB ' da0: 10240MB (20971520 512 byte sectors: 255H 63S/T 1305C) da1: 476940MB (976773168 512 byte sectors: 255H 63S/T 60801C) da2: 476940MB (976773168 512 byte sectors: 255H 63S/T 60801C) da3: 476940MB (976773168 512 byte sectors: 255H 63S/T 60801C) da4: 476940MB (976773168 512 byte sectors: 255H 63S/T 60801C) da5: 476938MB (976771055 512 byte sectors: 255H 63S/T 60801C) <-- da6: 476938MB (976771055 512 byte sectors: 255H 63S/T 60801C) <-- da7: 476938MB (976771055 512 byte sectors: 255H 63S/T 60801C) <-- da8: 476940MB (976773168 512 byte sectors: 255H 63S/T 60801C) da9: 152627MB (312581808 512 byte sectors: 255H 63S/T 19457C) da10: 152627MB (312581808 512 byte sectors: 255H 63S/T 19457C) da11: 152627MB (312581808 512 byte sectors: 255H 63S/T 19457C) da12: 152627MB (312581808 512 byte sectors: 255H 63S/T 19457C)
20111213-1930+1100 Update 02:
This did take a while indeed. I've spent months with several open computer cases on my desk with various amounts of harddrive stacks hanging out and also slept a few nights with earplugs, because I could not shut down the machine before going to bed as it was running some lengthy critical operation. However, I prevailed at last! :-) I've also learned a lot in the process and I would like to share that knowledge here for anyone in a similar situation.
This article is already much longer than anyone with a ZFS file server out of action has the time to read, so I will go into details here and create a conclusion section with the essential findings further below.
I dug deep in the obsolete hardware box to assemble enough storage space to move the stuff off the single 500GB drives to which the defective drives were mirrored. I also had to rip out a few hard drives out of their USB cases, so I could connect them over SATA directly. There was some more, unrelated issues involved and some of the old drives started to fail when I put them back into action requiring a zpool replace, but I'll skip on that.
Tip: At some stage, there was a total of about 30 hard drives involved in this. With that much hardware, it is an enormous help to have them stacked properly; cables coming loose or hard drive falling off your desk surely won't help in the process and might cause further damage to your data integrity.
I spent a couple of minutes creating some make-shift cardboard hard drive fixtures which really helped to keep things sorted:
Ironically, when I connected the old drives the first time, I realized there's an old zpool on there I must have created for testing with an older version of some, but not all of the personal data that's gone missing, so while the data loss was somewhat reduced, this meant additional shifting back and forth of files.
Finally, I mirrored the problematic drives to backup drives, used those for the zpool and left the original ones disconnected. The backup drives have a newer firmware, at least SeaTools does not report any required firmware updates. I did the mirroring with a simple dd from one device to the other, e.g.
sudo dd if=/dev/sda of=/dev/sde
I believe ZFS does notice the hardware change (by some hard drive UUID or whatever), but doesn't seem to care.
The zpool however was still in the same state, insufficient replicas / corrupted data.
As mentioned in the HPA Wikipedia article mentioned earlier, the presence of a host protected area is reported when Linux boots and can be investigated using hdparm. As far as I know, there is no hdparm tool available on FreeBSD, but by this time, I anyway had FreeBSD 8.2 and Debian 6.0 installed as dual-boot system, so I booted into Linux:
user@host:~$ for i in {a..l}; do sudo hdparm -N /dev/sd$i; done ... /dev/sdd: max sectors = 976773168/976773168, HPA is disabled /dev/sde: max sectors = 976771055/976773168, HPA is enabled /dev/sdf: max sectors = 976771055/976773168, HPA is enabled /dev/sdg: max sectors = 976771055/976773168, HPA is enabled /dev/sdh: max sectors = 976773168/976773168, HPA is disabled ...
So the problem obviously was that the new motherboard created a HPA of a couple of megabytes at the end of the drive which 'hid' the upper two ZFS labels, i.e. prevented ZFS from seeing them.
Dabbling with the HPA seems a dangerous business. From the hdparm man page, parameter -N:
Get/set max visible number of sectors, also known as the Host Protected Area setting. ... To change the current max (VERY DANGEROUS, DATA LOSS IS EXTREMELY LIKELY), a new value should be provided (in base10) immediately following the -N option. This value is specified as a count of sectors, rather than the "max sector address" of the drive. Drives have the concept of a temporary (volatile) setting which is lost on the next hardware reset, as well as a more permanent (non-volatile) value which survives resets and power cycles. By default, -N affects only the temporary (volatile) setting. To change the permanent (non-volatile) value, prepend a leading p character immediately before the first digit of the value. Drives are supposed to allow only a single permanent change per session. A hardware reset (or power cycle) is required before another permanent -N operation can succeed. ...
In my case, the HPA is removed like this:
user@host:~$ sudo hdparm -Np976773168 /dev/sde /dev/sde: setting max visible sectors to 976773168 (permanent) max sectors = 976773168/976773168, HPA is disabled
and in the same way for the other drives with an HPA. If you get the wrong drive or something about the size parameter you specify is not plausible, hdparm is smart enough to figure:
user@host:~$ sudo hdparm -Np976773168 /dev/sdx /dev/sdx: setting max visible sectors to 976773168 (permanent) Use of -Nnnnnn is VERY DANGEROUS. You have requested reducing the apparent size of the drive. This is a BAD idea, and can easily destroy all of the drive's contents. Please supply the --yes-i-know-what-i-am-doing flag if you really want this. Program aborted.
After that, I restarted the FreeBSD 7.2 virtual machine on which the zpool had been originally created and zpool status reported a working pool again. YAY! :-)
I exported the pool on the virtual system and re-imported it on the host FreeBSD 8.2 system.
Some more major hardware upgrades, another motherboard swap, a ZFS pool update to ZFS 4 / 15, a thorough scrubbing and now my zpool consists of 8x1TB plus 8x500GB raidz2 parts:
[user@host ~]$ sudo zpool status pool: zpool state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM zpool ONLINE 0 0 0 raidz2 ONLINE 0 0 0 ad0 ONLINE 0 0 0 ad1 ONLINE 0 0 0 ad2 ONLINE 0 0 0 ad3 ONLINE 0 0 0 ad8 ONLINE 0 0 0 ad10 ONLINE 0 0 0 ad14 ONLINE 0 0 0 ad16 ONLINE 0 0 0 raidz2 ONLINE 0 0 0 da0 ONLINE 0 0 0 da1 ONLINE 0 0 0 da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 da4 ONLINE 0 0 0 da5 ONLINE 0 0 0 da6 ONLINE 0 0 0 da7 ONLINE 0 0 0 errors: No known data errors [user@host ~]$ df -h Filesystem Size Used Avail Capacity Mounted on /dev/label/root 29G 13G 14G 49% / devfs 1.0K 1.0K 0B 100% /dev zpool 8.0T 3.6T 4.5T 44% /mnt/zpool
Conclusion:
Question: Why did the zpool fail ?
Answer: A new motherboard created a host protected area (HPA), a small section usually located at the end of the harddrive. ZFS maintains 4 labels with partition meta information and the HPA prevented ZFS from seeing the upper two.
Solution: use hdparm under Linux to remove the HPA.
The problem did not only occur with the new motherboard, I had a similar issue when connecting the drives to an SAS controller card. The solution is the same, though.
As a last word, it seems to me ZFS pools are very, very hard to kill. The guys from Sun from who created that system have all the reason the call it the last word in filesystems. Respect!
