(LVM) Cannot restore Volume Group with 1 PVs marked as missing. Couldn't find device with uuid, but the pv exists and blkid finds the same uuid

Question

This problem is on a kvm/qemu hypervisor with CentOS 7, where we host multiple virtual machines.

The Disk /dev/sda( 1918.2 GB) is a raid6 LSI MEGARAID.

[root@mia3 ~]# fdisk -l Disk /dev/sda: 1918.2 GB, 1918239768576 bytes, 3746562048 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0x51549cd4 Device Boot Start End Blocks Id System /dev/sda1 * 2048 1026047 512000 83 Linux /dev/sda2 1026048 135243775 67108864 82 Linux swap / Solaris /dev/sda3 135243776 449816575 157286400 83 Linux /dev/sda4 449816576 3746562047 1648372736 8e Linux LVM 

This is our disk configuration: sda1 - /boot, sda2 - swap, sda3 - / (root), sda4 - PV for our LVM with thin provisioning for our VMs.

After a power failure, our data on the root partition got corrupted. We had to reinstall the OS in order to bring up the server. We didn't touch the /dev/sda4 partition. Before that, we mounted the root partition in rescue mode and made a backup for the /etc/lvm/ folder.

After reinstalling Centos we tried to recover LVM using vgcfgrestore, but we got the error:

[root@mia3 ~]# vgcfgrestore -f /etc/lvm/backup/vg_kvm vg_kvm -v No devices found. /dev/sda4: Checksum error at offset 808960 Couldn't read volume group metadata from /dev/sda4. Metadata location on /dev/sda4 at 808960 has invalid summary for VG. Failed to read metadata summary from /dev/sda4 Failed to scan VG from /dev/sda4 Couldn't find device with uuid HNB35U-ilI4-WWAK-1Q4D-Netj-t68p-Po4vCT. Consider using option --force to restore Volume Group vg_kvm with thin volumes. Restore failed. Scanning all devices to initialize lvmetad. /dev/sda4: Checksum error at offset 808960 Couldn't read volume group metadata from /dev/sda4. Metadata location on /dev/sda4 at 808960 has invalid summary for VG. Failed to read metadata summary from /dev/sda4 Failed to scan VG from /dev/sda4 [root@mia3 ~]# 

So we tried to readd the PV and ovewrite the metadata, but without success:

 [root@mia3 ~]# pvcreate -f /dev/sda4 --restorefile /root/lvm/backup/vg_kvm --uuid HNB35U-ilI4-WWAK-1Q4D-Netj-t68p-Po4vCT -vv devices/global_filter not found in config: defaulting to global_filter = [ "a|.*/|" ] global/lvmetad_update_wait_time not found in config: defaulting to 10 devices/filter not found in config: defaulting to filter = [ "a|.*/|" ] devices/cache not found in config: defaulting to /etc/lvm/cache/.cache metadata/record_lvs_history not found in config: defaulting to 0 File-based locking selected. Couldn't find device with uuid HNB35U-ilI4-WWAK-1Q4D-Netj-t68p-Po4vCT. Stack vg_kvm/lvpool:0[0] on LV vg_kvm/lvpool_tdata:0. Adding vg_kvm/lvpool:0 as an user of vg_kvm/lvpool_tdata. Adding vg_kvm/lvpool:0 as an user of vg_kvm/lvpool_tmeta. Adding vg_kvm/vsv11999-dm9oryxunpw8po9p-1xwrkrotul6i5k3g:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12001-dmzdbquhkhrydlnf-kqjcbeyvz1c5xwun:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12034-dnadddzaxsxoq2rx-iwxjz6wr2vmqhsj3:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12036-dbgt9zilw2tg9sqd-c0xa9h1uxigjuqg9:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12037-dctrjirji3agbl9l-dtzyf8jrb8lip0m7:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12038-dm5mulwohkvz5ara-kccgnknpjdi11tdy:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12039-d1lnb9qcyjaqmodk-mnl22loaflo24kpm:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12040-daejvderic4uvccj-wjnydccgzohyaekq:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12046-dswief4uk4gthxa1-ie5s7up6vbf37p68:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12056-du7hvvsp3lw7jgja-uap8l1jpe29llrpu:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12080-dwydbngcjptnnryn-hzz0jimwfl6yau2x:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12094-d8cmko7aio2f1fyo-warkvewdzfeedme5:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12097-dllp7oepp4homdj9-jm6n2itmvy29k9xf:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12098-dcj6mnnidej8c80u-r3bkk3pfu8g9sf3o:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12103-dwtyfq1quemcij56-4rwivmmtzxwk1ydb:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12106-dngrxbyoapwac5sm-bstrvsbxzrztefob:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12107-dq3h0wgnzkasoles-hv12r8qmyobojubp:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12109-ddq6cmtow9qwuaka-v9a1omyljy0xujk4:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12110-dvaaktx2bng99yl9-f5xh28mlg7u3mo5r:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12111-db0wlufykbcbb2mp-dxhgxxmivthvqwwp:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12112-dyfnblgyajfpidqs-ny0xpnxhzfzwc1db:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12116-dowxexkxa0ygsgki-e4dih6xcnybtdgx2:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12117-dwd0weu8ig1dvukx-fly3n8dyseklhntv:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12118-dz9hbvnbtsihufm5-paqhexr1tuld8bnd:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12119-dof5n90vez6smo1j-h1xmccyzl7qotery:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12120-d9q0y85z4shpw64o-6zx5tmxhajpcxtkv:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12121-dac4vuig8akyryd0-0e4rf2snm04r5mdy:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12122-d99yza6ssbhxxkru-16ptyp8btexq6wkb:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12124-d4ud5tu2ueajr0vc-r9pv77cjusnjue21:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12126-dqjmtfzubhqvheij-blecac71kpwsmy62:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12127-d5sgcmdodjaqtvp9-oqbroobow5tzn0va:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12130-dtf7rf4suf8bov9t-8lb0agstvcpann2e:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12131-dwkveordvertdam3-3g4qwtf5ydhy0hug:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12134-dgpxbewt3xlkxlqb-j5ad04pt2fndqqmw:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12135-d55krjg3dotss3e4-dwbbjmo0shwwwzv2:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12138-dyz9yl6swxbaxfrj-1pgri05suzqyagqq:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12140-dykgcwqaauznoxrm-4ibft4womwuvpwkx:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12142-dwtlkzoke0a05u1o-rqxjogzuvjvfttxs:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12144-dn3gyu3uroppwgzg-wraejm2ynwnmtmd4:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12162-dalw90et3l9vbath-eo5womktdromb5sn:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12211-dyyizuuag0mz4eok-je7du6slil1jlfzp:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12329-d5xmlj2iqsuekgkm-cg6qplkwk48yiyeo:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12333-drqvvb7fktr4n52b-estbavvooqdljpvo:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12334-dbdf0lvefbgpqbxa-bmpnzainjdfvuavv:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12339-d3yfeg1upsfoxbd3-nbxbccudjdtcgztt:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12354-d1ctzl88dwv9znri-pzfp6weunhpdhvgn:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12361-dbug4gr7r8mbsnva-yamgras92xuqin66:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12088-dqphwqqeuwxtmm7y-4rurc6jm0lvcj57q:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12089-dbhimrcqvjkaxvxc-3vpljlh2xkojhb9e:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12090-dndlzsmpj8qgm2gb-wm4sg3d6n2cq3tpm:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12091-dyyxcz5cf4w4cu6z-gja2kph6egpjvyzh:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12092-dnllln6x7uud75ss-um9zelymhc04ohd6:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12442-dyf2cgwdgaluli0s-cahwc74lrstt61gl:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12456-dmiuo7guhru4snsf-yt55qun0wx5mtfkg:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12087-dxbtb8w1g3tx5akk-bdqcsnlathemkrig:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12234-dq82wyoapvxvh7zz-maaekbfd7h9ixks8:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12588-das5kos8qt1ziktl-aky0pxwg8ecv9b48:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12771-djaikmniei1vxia5-jhqasdwc763s3pmb:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12784-d3k6aplejbvjrlcw-uwo81r0wkmeiptso:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12139-dpnhdmg2odfabvfn-zqgiw42wfsgbbkrc:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12137-dxtiqptvdwcrrcwa-22fkxorf6hl44evf:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12866-d37htmu48aopiafv-sli9m0lt3dz0yeyl:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12096-dz9pfno05wqq8vbg-sq8i5na5z8lrnm6n:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12363-dtlh9w3tza99quoj-dlicfndz6e3pbfu0:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12890-dmaaydjldw3c13jk-iskdygh5pczveo3r:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12320-dqmecms9nep6muns-xmmkziafa8mz1hta:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12936-dhdfa36ovcdzfafp-ultnlmom4scugsw1:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12964-dctbp9xgo2gl8w6i-kbw9cinhoieea8av:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12984-dwqjvkqc3aznhb4o-criyexpxvdndoscq:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12143-d3s1toagczaiq4ma-oupwnk3eg27r41ze:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13019-d5yrsi5a58bdzalc-i6mbum8zthk2arhs:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13022-djszrasae5w7xegh-ehqsutbhrwd7qgsi:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13027-dq6pg99jd2fphljg-u15x8sonbwukaree:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12041-dlsmtmfa6fp4wcuc-bfhftkdhwfqa7sqb:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13039-djtjimojqi3iaufp-csbinqtsvebzzmqx:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13061-duocptl9qdtvwubp-feryqeejht4j24ql:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13064-dexwkomdh62by80s-aymbuetu6p0btn4d:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13076-drd4nrhy4u7fq1tn-f1kt8ubk2jch4ol3:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13082-ds9gjikqyg9nftua-rrhk4dlrr8cp3la7:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12759-du4yducr4ghi8eun-bzivqxabwaltaddm:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12123-dzb9rktypp0jqinx-adtlzed02wpccxoa:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13118-d0lj9gpvqfmox63i-9jstjcjkmemszrla:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13121-du0r39n6wvnrclox-bviz76ig9vvexota:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13133-d2fk6lg6bsppbkcs-hwk5bmsvqxa0y0bs:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13204-dl6hzsmswetitsb4-bais6lpjbbhfkexn:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13230-dtiyrmijg8mde5xy-vobukov1qxg8bx5j:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13254-d3sdhrrgcvbwrsyp-yxaozxmt3clqzxxc:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13271-drutw4ihq774xhs6-ecpwfmzcrqy0vvsm:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13266-dmuj5cfeu2hrgfxw-2tpflycnwoedyidb:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13380-d99ciziibe7rt3es-7fkh56myy79oushw:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13487-d30yxx4wgfxcb9te-chc2bxojs7tvcwz3:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13502-dc2oivdodzo94jj3-1qmpba9ti4zzbfeg:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13277-deudjco25w9cd3rv-re1ha5qkozses9i3:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12208-dfqhoqmtuwzktwqm-n9s5daovhtnyvyeh:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13481-dthcwgpolombhfed-dkmesvsvmt1vem0d:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13585-ddvwhm2vcqpq66uy-1brs98bctgr6bcq9:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13586-dadacaexxn8nkuy3-fyx8zp5mursbqhns:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13587-dc0fmskwdbkmz7mg-ra53wgfksiv7tsqk:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13569-dk6v0pyvz8eoohl5-qyndiwn8zriir75s:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13603-d66mz2amabmd63fj-rkcqmtvfnlfmywou:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13613-dykuo32mnkgf9nce-hhkhn118dhsedoc8:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13623-dksoizdtoglihaax-23o0zltutliyoo2q:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13629-dxbpqqernfvufxrz-iv6jujdxm78hxjbn:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13610-drrehnh36oipacdf-frnumdhlt823mntg:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13654-dmlnhhqilztxff5a-4gawkysgtkze4st7:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13669-dk3ukgidvj0l4x4x-9d3vfao5qejzecdn:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13672-d1r1608gu96b0ujo-nn7ejmxkbfce40m4:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13689-dhpzh8w7jlwi0zza-aqizb6tbeuk9k1ca:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12477-dejmu0gihtllnj97-gaxwgicndwp25unx:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13710-dv2bktawped5fz44-0bkdtrkp8tk2vynu:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12125-d5d9pebts1rue7jt-vwnkapewjkfszked:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13718-dmimkn9h1ax3t4am-8tdnpznya2reek8g:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13545-dzaq2rjqfiyf5azl-kw35xomndhltjnrs:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13723-daleipfjwu7zm7aj-s8jo6yhxwz4efl2y:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13726-dpdqvmnzkvxkiz4y-cfudhed6m12hnlmj:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12136-d0qvvamyx5qq1g6o-awtlh6pbotvlacyc:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13258-du2kuwgkwhypcfzz-i6h00rd0exzw4b69:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13755-dqglblzy3akyohph-wtev5tlrs5lkkejd:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13758-dhfrybljxildcky1-ktotkou3mpvxfjnq:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13759-dxqbq7phpjgoa1ed-dwtly1dkuorvlyli:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13762-di54bo3us4p8gefa-34s2uhcye1rbdbvw:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13767-dwtsyqxhqbp8fa9w-us8xvwm9q6nqkmzx:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13768-dadnlkrau2cwhlbd-cj8liz4hfolpmjfa:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13769-dt4nwlcmpfqyenob-jvpfd5jut8kvcm8u:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13781-dvckr0uqfwrsvmc6-zklf1olhlhzzfppp:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12133-dp55dttaolpbd6n8-rlhoezgituqkyrov:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12042-dnncyzlyhfxyaxkt-metx0ei0zlz3d2sn:0 as an user of vg_kvm/lvpool. metadata/pvmetadataignore not found in config: defaulting to 0 metadata/pvmetadatasize not found in config: defaulting to 255 metadata/pvmetadatacopies not found in config: defaulting to 1 report/output_format not found in config: defaulting to basic log/report_command_log not found in config: defaulting to 0 Locking /run/lock/lvm/P_orphans WB /dev/sda4: size is 3296745472 sectors /dev/sda4: using cached size 3296745472 sectors /dev/sda: size is 3746562048 sectors /dev/sda1: size is 1024000 sectors /dev/sda1: using cached size 1024000 sectors /dev/sda2: size is 134217728 sectors /dev/sda2: using cached size 134217728 sectors /dev/sda3: size is 314572800 sectors /dev/sda3: using cached size 314572800 sectors /dev/sda4: using cached size 3296745472 sectors /dev/sda4: using cached size 3296745472 sectors Locking #orphans_lvm1 already done Reading VG #orphans_lvm1 Locking #orphans_pool already done Reading VG #orphans_pool Locking #orphans_lvm2 already done Reading VG #orphans_lvm2 Processing device /dev/sda1. Processing device /dev/sda2. Processing device /dev/sda3. Processing device /dev/sda4. /dev/sda4: lvm2 label detected at sector 1 /dev/sda4: Checksum error at offset 808960 Couldn't read volume group metadata from /dev/sda4. Metadata location on /dev/sda4 at 808960 has invalid summary for VG. Failed to read metadata summary from /dev/sda4 Failed to scan VG from /dev/sda4 Unlocking /run/lock/lvm/P_orphans [root@mia3 ~]# 

Note the error:

 Couldn't find device with uuid HNB35U-ilI4-WWAK-1Q4D-Netj-t68p-Po4vCT. 

The PV (/dev/sda4) has the same UUID:

[root@mia3 ~]# blkid /dev/sda1: LABEL="boot" UUID="087755bd-9434-4597-953b-c056fd9cb285" TYPE="xfs" /dev/sda2: LABEL="swap" UUID="c7fd2030-0441-4a4e-8462-0aeffc272bfa" TYPE="swap" /dev/sda3: LABEL="root" UUID="3ee19cf6-87b5-40b5-a7dd-13d507efa734" TYPE="xfs" /dev/sda4: UUID="HNB35U-ilI4-WWAK-1Q4D-Netj-t68p-Po4vCT" TYPE="LVM2_member" [root@mia3 ~]# 

Here is a small part of the content for /etc/lvm/backup/vg_kvm:

vg_kvm { id = "i3usZz-vdt5-9yaU-Kbcl-XDWJ-OFr1-UhZRBa" seqno = 1349 format = "lvm2" # informational status = ["RESIZEABLE", "READ", "WRITE"] flags = [] extent_size = 8192 # 4 Megabytes max_lv = 0 max_pv = 0 metadata_copies = 0 physical_volumes { pv0 { id = "HNB35U-ilI4-WWAK-1Q4D-Netj-t68p-Po4vCT" device = "/dev/sda4" # Hint only status = ["ALLOCATABLE"] flags = [] dev_size = 3296745472 # 1.53517 Terabytes pe_start = 2048 pe_count = 402434 # 1.53516 Terabytes } } logical_volumes { lvpool { id = "DHqs7X-95DA-ADNm-WTd9-cXDf-HR5a-MoHLRx" status = ["READ", "WRITE", "VISIBLE"] flags = [] creation_time = 1575901618 # 2019-12-09 16:26:58 +0200 creation_host = "mia3" segment_count = 1 segment1 { start_extent = 0 extent_count = 401081 # 1.53 Terabytes type = "thin-pool" metadata = "lvpool_tmeta" pool = "lvpool_tdata" transaction_id = 667 chunk_size = 2048 # 1024 Kilobytes discards = "passdown" zero_new_blocks = 1 } } vsv11999-dm9oryxunpw8po9p-1xwrkrotul6i5k3g { id = "vBco77-gPno-S0cu-QZdx-9Nnk-twXR-cBBAzW" status = ["READ", "WRITE", "VISIBLE"] flags = [] creation_time = 1575902920 # 2019-12-09 16:48:40 +0200 creation_host = "mia3" segment_count = 1 segment1 { start_extent = 0 extent_count = 6400 # 25 Gigabytes type = "thin" thin_pool = "lvpool" transaction_id = 0 device_id = 1 } } vsv12001-dmzdbquhkhrydlnf-kqjcbeyvz1c5xwun { id = "xqDzS2-iHXO-2zEi-LRHO-0FDR-hvcM-oRdFeD" status = ["READ", "WRITE", "VISIBLE"] flags = [] creation_time = 1575904408 # 2019-12-09 17:13:28 +0200 creation_host = "mia3" segment_count = 1 segment1 { start_extent = 0 extent_count = 3840 # 15 Gigabytes type = "thin" thin_pool = "lvpool" transaction_id = 1 device_id = 2 } } vsv12034-dnadddzaxsxoq2rx-iwxjz6wr2vmqhsj3 { id = "VIDuPN-SN4Z-V1qm-gzI9-sSnV-WGcp-NBvgUa" status = ["READ", "WRITE", "VISIBLE"] flags = [] creation_time = 1575986509 # 2019-12-10 16:01:49 +0200 creation_host = "mia3" segment_count = 1 segment1 { start_extent = 0 extent_count = 3840 # 15 Gigabytes type = "thin" thin_pool = "lvpool" transaction_id = 6 device_id = 5 } } vsv12036-dbgt9zilw2tg9sqd-c0xa9h1uxigjuqg9 { id = "XnnwPm-e3jE-VU9W-7QIz-3Sz5-HDSM-sEe7NY" status = ["READ", "WRITE", "VISIBLE"] flags = [] creation_time = 1575988785 # 2019-12-10 16:39:45 +0200 creation_host = "mia3" segment_count = 1 segment1 { start_extent = 0 extent_count = 3840 # 15 Gigabytes type = "thin" thin_pool = "lvpool" transaction_id = 8 device_id = 7 } } vsv12037-dctrjirji3agbl9l-dtzyf8jrb8lip0m7 { id = "ysHM90-Vmof-Mrwj-Nb1s-HPER-g3v5-hdutBz" status = ["READ", "WRITE", "VISIBLE"] flags = [] creation_time = 1575989058 # 2019-12-10 16:44:18 +0200 creation_host = "mia3" segment_count = 1 segment1 { start_extent = 0 extent_count = 3840 # 15 Gigabytes type = "thin" thin_pool = "lvpool" transaction_id = 9 device_id = 8 } } 

All the data (root partitions for our VMs) are still there, we have checked with disk recovery tools like testdisk, but we can't recover the lvm. What are we doing wrong? Is there a possibility of recovering?

#

UPDATE

Using Centos6 CD and rescue mode I have managed to restore the PV and VG. However, I can't reactivate the VG and LV pool, we're getting the error:

 Executing: /usr/sbin/thin_check -q --clear-needs-check-flag /dev/mapper/vg_kvm-lvpool_tmeta /usr/sbin/thin_check failed: 1 Check of pool vg_kvm/lvpool failed (status:1). Manual repair required! 

Answer 1

It appears Red Hat 7 and 8 have similar issues when trying to use vgcfgrestore: The pvcreate command fails with "Metadata location has invalid summary for VG" error . CentOS shares a base with Red Hat, so...

One answer there was "We have a solution for this problem upstream and in RHEL8, Unfortunately, it is too invasive to consider back porting to RHEL7 at this stage." Another person in that bug said they fixed it by booting Red Hat 6, and then vgcfgrestore worked.

Perhaps you can boot a new sysrescue CD from a USB stick and do it?

And as I said in a comment, either backup the whole volume with dd, or at least do something like dd if=/dev/sda4 of=/somewere/backup-first100MB.img bs=1M count=100 so that you have the header info.

Answer 2

It worked with Centos6 rescue. After restoring, I booted into normal centos 7 and the vg was there. However, now I can't activate the lv (thin pool):

[root@mia3 ~]# lvchange -ay /dev/vg_kvm/lvpool Check of pool vg_kvm/lvpool failed (status:1). Manual repair required! [root@mia3 ~]# 

Here is the command again with -vvv

 [root@mia3 ~]# lvchange -ay /dev/vg_kvm/lvpool -vv devices/global_filter not found in config: defaulting to global_filter = [ "a|.*/|" ] global/lvmetad_update_wait_time not found in config: defaulting to 10 Setting devices/filter to filter = [ "a|/dev/sda.*|", "a|/dev/disk/by-id/dm-uuid-.*mpath-.*|", "r|.*|" ] devices/cache not found in config: defaulting to /etc/lvm/cache/.cache metadata/record_lvs_history not found in config: defaulting to 0 File-based locking selected. report/output_format not found in config: defaulting to basic log/report_command_log not found in config: defaulting to 0 Obtaining the complete list of VGs before processing their LVs Processing VG vg_kvm i3usZz-vdt5-9yaU-Kbcl-XDWJ-OFr1-UhZRBa Locking /run/lock/lvm/V_vg_kvm RB Reading VG vg_kvm i3usZz-vdt5-9yaU-Kbcl-XDWJ-OFr1-UhZRBa Stack vg_kvm/lvpool:0[0] on LV vg_kvm/lvpool_tdata:0. Adding vg_kvm/lvpool:0 as an user of vg_kvm/lvpool_tdata. Adding vg_kvm/lvpool:0 as an user of vg_kvm/lvpool_tmeta. Adding vg_kvm/vsv11999-dm9oryxunpw8po9p-1xwrkrotul6i5k3g:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12001-dmzdbquhkhrydlnf-kqjcbeyvz1c5xwun:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12034-dnadddzaxsxoq2rx-iwxjz6wr2vmqhsj3:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12036-dbgt9zilw2tg9sqd-c0xa9h1uxigjuqg9:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12037-dctrjirji3agbl9l-dtzyf8jrb8lip0m7:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12038-dm5mulwohkvz5ara-kccgnknpjdi11tdy:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12039-d1lnb9qcyjaqmodk-mnl22loaflo24kpm:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12040-daejvderic4uvccj-wjnydccgzohyaekq:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12046-dswief4uk4gthxa1-ie5s7up6vbf37p68:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12056-du7hvvsp3lw7jgja-uap8l1jpe29llrpu:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12080-dwydbngcjptnnryn-hzz0jimwfl6yau2x:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12094-d8cmko7aio2f1fyo-warkvewdzfeedme5:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12097-dllp7oepp4homdj9-jm6n2itmvy29k9xf:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12098-dcj6mnnidej8c80u-r3bkk3pfu8g9sf3o:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12103-dwtyfq1quemcij56-4rwivmmtzxwk1ydb:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12106-dngrxbyoapwac5sm-bstrvsbxzrztefob:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12107-dq3h0wgnzkasoles-hv12r8qmyobojubp:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12109-ddq6cmtow9qwuaka-v9a1omyljy0xujk4:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12110-dvaaktx2bng99yl9-f5xh28mlg7u3mo5r:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12111-db0wlufykbcbb2mp-dxhgxxmivthvqwwp:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12112-dyfnblgyajfpidqs-ny0xpnxhzfzwc1db:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12116-dowxexkxa0ygsgki-e4dih6xcnybtdgx2:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12117-dwd0weu8ig1dvukx-fly3n8dyseklhntv:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12118-dz9hbvnbtsihufm5-paqhexr1tuld8bnd:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12119-dof5n90vez6smo1j-h1xmccyzl7qotery:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12120-d9q0y85z4shpw64o-6zx5tmxhajpcxtkv:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12121-dac4vuig8akyryd0-0e4rf2snm04r5mdy:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12122-d99yza6ssbhxxkru-16ptyp8btexq6wkb:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12124-d4ud5tu2ueajr0vc-r9pv77cjusnjue21:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12126-dqjmtfzubhqvheij-blecac71kpwsmy62:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12127-d5sgcmdodjaqtvp9-oqbroobow5tzn0va:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12130-dtf7rf4suf8bov9t-8lb0agstvcpann2e:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12131-dwkveordvertdam3-3g4qwtf5ydhy0hug:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12134-dgpxbewt3xlkxlqb-j5ad04pt2fndqqmw:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12135-d55krjg3dotss3e4-dwbbjmo0shwwwzv2:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12138-dyz9yl6swxbaxfrj-1pgri05suzqyagqq:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12140-dykgcwqaauznoxrm-4ibft4womwuvpwkx:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12142-dwtlkzoke0a05u1o-rqxjogzuvjvfttxs:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12144-dn3gyu3uroppwgzg-wraejm2ynwnmtmd4:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12162-dalw90et3l9vbath-eo5womktdromb5sn:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12211-dyyizuuag0mz4eok-je7du6slil1jlfzp:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12329-d5xmlj2iqsuekgkm-cg6qplkwk48yiyeo:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12333-drqvvb7fktr4n52b-estbavvooqdljpvo:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12334-dbdf0lvefbgpqbxa-bmpnzainjdfvuavv:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12339-d3yfeg1upsfoxbd3-nbxbccudjdtcgztt:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12354-d1ctzl88dwv9znri-pzfp6weunhpdhvgn:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12361-dbug4gr7r8mbsnva-yamgras92xuqin66:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12088-dqphwqqeuwxtmm7y-4rurc6jm0lvcj57q:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12089-dbhimrcqvjkaxvxc-3vpljlh2xkojhb9e:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12090-dndlzsmpj8qgm2gb-wm4sg3d6n2cq3tpm:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12091-dyyxcz5cf4w4cu6z-gja2kph6egpjvyzh:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12092-dnllln6x7uud75ss-um9zelymhc04ohd6:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12442-dyf2cgwdgaluli0s-cahwc74lrstt61gl:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12456-dmiuo7guhru4snsf-yt55qun0wx5mtfkg:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12087-dxbtb8w1g3tx5akk-bdqcsnlathemkrig:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12234-dq82wyoapvxvh7zz-maaekbfd7h9ixks8:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12588-das5kos8qt1ziktl-aky0pxwg8ecv9b48:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12771-djaikmniei1vxia5-jhqasdwc763s3pmb:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12784-d3k6aplejbvjrlcw-uwo81r0wkmeiptso:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12139-dpnhdmg2odfabvfn-zqgiw42wfsgbbkrc:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12137-dxtiqptvdwcrrcwa-22fkxorf6hl44evf:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12866-d37htmu48aopiafv-sli9m0lt3dz0yeyl:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12096-dz9pfno05wqq8vbg-sq8i5na5z8lrnm6n:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12363-dtlh9w3tza99quoj-dlicfndz6e3pbfu0:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12890-dmaaydjldw3c13jk-iskdygh5pczveo3r:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12320-dqmecms9nep6muns-xmmkziafa8mz1hta:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12936-dhdfa36ovcdzfafp-ultnlmom4scugsw1:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12964-dctbp9xgo2gl8w6i-kbw9cinhoieea8av:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12984-dwqjvkqc3aznhb4o-criyexpxvdndoscq:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12143-d3s1toagczaiq4ma-oupwnk3eg27r41ze:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13019-d5yrsi5a58bdzalc-i6mbum8zthk2arhs:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13022-djszrasae5w7xegh-ehqsutbhrwd7qgsi:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13027-dq6pg99jd2fphljg-u15x8sonbwukaree:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12041-dlsmtmfa6fp4wcuc-bfhftkdhwfqa7sqb:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13039-djtjimojqi3iaufp-csbinqtsvebzzmqx:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13061-duocptl9qdtvwubp-feryqeejht4j24ql:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13064-dexwkomdh62by80s-aymbuetu6p0btn4d:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13076-drd4nrhy4u7fq1tn-f1kt8ubk2jch4ol3:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13082-ds9gjikqyg9nftua-rrhk4dlrr8cp3la7:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12759-du4yducr4ghi8eun-bzivqxabwaltaddm:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12123-dzb9rktypp0jqinx-adtlzed02wpccxoa:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13118-d0lj9gpvqfmox63i-9jstjcjkmemszrla:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13121-du0r39n6wvnrclox-bviz76ig9vvexota:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13133-d2fk6lg6bsppbkcs-hwk5bmsvqxa0y0bs:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13204-dl6hzsmswetitsb4-bais6lpjbbhfkexn:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13230-dtiyrmijg8mde5xy-vobukov1qxg8bx5j:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13254-d3sdhrrgcvbwrsyp-yxaozxmt3clqzxxc:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13271-drutw4ihq774xhs6-ecpwfmzcrqy0vvsm:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13266-dmuj5cfeu2hrgfxw-2tpflycnwoedyidb:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13380-d99ciziibe7rt3es-7fkh56myy79oushw:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13487-d30yxx4wgfxcb9te-chc2bxojs7tvcwz3:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13502-dc2oivdodzo94jj3-1qmpba9ti4zzbfeg:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13277-deudjco25w9cd3rv-re1ha5qkozses9i3:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12208-dfqhoqmtuwzktwqm-n9s5daovhtnyvyeh:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13481-dthcwgpolombhfed-dkmesvsvmt1vem0d:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13585-ddvwhm2vcqpq66uy-1brs98bctgr6bcq9:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13586-dadacaexxn8nkuy3-fyx8zp5mursbqhns:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13587-dc0fmskwdbkmz7mg-ra53wgfksiv7tsqk:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13569-dk6v0pyvz8eoohl5-qyndiwn8zriir75s:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13603-d66mz2amabmd63fj-rkcqmtvfnlfmywou:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13613-dykuo32mnkgf9nce-hhkhn118dhsedoc8:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13623-dksoizdtoglihaax-23o0zltutliyoo2q:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13629-dxbpqqernfvufxrz-iv6jujdxm78hxjbn:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13610-drrehnh36oipacdf-frnumdhlt823mntg:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13654-dmlnhhqilztxff5a-4gawkysgtkze4st7:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13669-dk3ukgidvj0l4x4x-9d3vfao5qejzecdn:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13672-d1r1608gu96b0ujo-nn7ejmxkbfce40m4:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13689-dhpzh8w7jlwi0zza-aqizb6tbeuk9k1ca:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12477-dejmu0gihtllnj97-gaxwgicndwp25unx:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13710-dv2bktawped5fz44-0bkdtrkp8tk2vynu:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12125-d5d9pebts1rue7jt-vwnkapewjkfszked:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13718-dmimkn9h1ax3t4am-8tdnpznya2reek8g:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13545-dzaq2rjqfiyf5azl-kw35xomndhltjnrs:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13723-daleipfjwu7zm7aj-s8jo6yhxwz4efl2y:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13726-dpdqvmnzkvxkiz4y-cfudhed6m12hnlmj:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12136-d0qvvamyx5qq1g6o-awtlh6pbotvlacyc:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13258-du2kuwgkwhypcfzz-i6h00rd0exzw4b69:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13755-dqglblzy3akyohph-wtev5tlrs5lkkejd:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13758-dhfrybljxildcky1-ktotkou3mpvxfjnq:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13759-dxqbq7phpjgoa1ed-dwtly1dkuorvlyli:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13762-di54bo3us4p8gefa-34s2uhcye1rbdbvw:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13767-dwtsyqxhqbp8fa9w-us8xvwm9q6nqkmzx:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13768-dadnlkrau2cwhlbd-cj8liz4hfolpmjfa:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13769-dt4nwlcmpfqyenob-jvpfd5jut8kvcm8u:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv13781-dvckr0uqfwrsvmc6-zklf1olhlhzzfppp:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12133-dp55dttaolpbd6n8-rlhoezgituqkyrov:0 as an user of vg_kvm/lvpool. Adding vg_kvm/vsv12042-dnncyzlyhfxyaxkt-metx0ei0zlz3d2sn:0 as an user of vg_kvm/lvpool. /dev/sda4: size is 3296745472 sectors Adding vg_kvm/lvpool to the list of LVs to be processed. Processing LV lvpool in VG vg_kvm. Activating logical volume vg_kvm/lvpool exclusively. Locking /run/lock/lvm/A_vg_kvm WB vg_kvm/lvpool is not active vg_kvm/vsv11999-dm9oryxunpw8po9p-1xwrkrotul6i5k3g is not active vg_kvm/vsv12001-dmzdbquhkhrydlnf-kqjcbeyvz1c5xwun is not active vg_kvm/vsv12034-dnadddzaxsxoq2rx-iwxjz6wr2vmqhsj3 is not active vg_kvm/vsv12036-dbgt9zilw2tg9sqd-c0xa9h1uxigjuqg9 is not active vg_kvm/vsv12037-dctrjirji3agbl9l-dtzyf8jrb8lip0m7 is not active vg_kvm/vsv12038-dm5mulwohkvz5ara-kccgnknpjdi11tdy is not active vg_kvm/vsv12039-d1lnb9qcyjaqmodk-mnl22loaflo24kpm is not active vg_kvm/vsv12040-daejvderic4uvccj-wjnydccgzohyaekq is not active vg_kvm/vsv12046-dswief4uk4gthxa1-ie5s7up6vbf37p68 is not active vg_kvm/vsv12056-du7hvvsp3lw7jgja-uap8l1jpe29llrpu is not active vg_kvm/vsv12080-dwydbngcjptnnryn-hzz0jimwfl6yau2x is not active vg_kvm/vsv12094-d8cmko7aio2f1fyo-warkvewdzfeedme5 is not active vg_kvm/vsv12097-dllp7oepp4homdj9-jm6n2itmvy29k9xf is not active vg_kvm/vsv12098-dcj6mnnidej8c80u-r3bkk3pfu8g9sf3o is not active vg_kvm/vsv12103-dwtyfq1quemcij56-4rwivmmtzxwk1ydb is not active vg_kvm/vsv12106-dngrxbyoapwac5sm-bstrvsbxzrztefob is not active vg_kvm/vsv12107-dq3h0wgnzkasoles-hv12r8qmyobojubp is not active vg_kvm/vsv12109-ddq6cmtow9qwuaka-v9a1omyljy0xujk4 is not active vg_kvm/vsv12110-dvaaktx2bng99yl9-f5xh28mlg7u3mo5r is not active vg_kvm/vsv12111-db0wlufykbcbb2mp-dxhgxxmivthvqwwp is not active vg_kvm/vsv12112-dyfnblgyajfpidqs-ny0xpnxhzfzwc1db is not active vg_kvm/vsv12116-dowxexkxa0ygsgki-e4dih6xcnybtdgx2 is not active vg_kvm/vsv12117-dwd0weu8ig1dvukx-fly3n8dyseklhntv is not active vg_kvm/vsv12118-dz9hbvnbtsihufm5-paqhexr1tuld8bnd is not active vg_kvm/vsv12119-dof5n90vez6smo1j-h1xmccyzl7qotery is not active vg_kvm/vsv12120-d9q0y85z4shpw64o-6zx5tmxhajpcxtkv is not active vg_kvm/vsv12121-dac4vuig8akyryd0-0e4rf2snm04r5mdy is not active vg_kvm/vsv12122-d99yza6ssbhxxkru-16ptyp8btexq6wkb is not active vg_kvm/vsv12124-d4ud5tu2ueajr0vc-r9pv77cjusnjue21 is not active vg_kvm/vsv12126-dqjmtfzubhqvheij-blecac71kpwsmy62 is not active vg_kvm/vsv12127-d5sgcmdodjaqtvp9-oqbroobow5tzn0va is not active vg_kvm/vsv12130-dtf7rf4suf8bov9t-8lb0agstvcpann2e is not active vg_kvm/vsv12131-dwkveordvertdam3-3g4qwtf5ydhy0hug is not active vg_kvm/vsv12134-dgpxbewt3xlkxlqb-j5ad04pt2fndqqmw is not active vg_kvm/vsv12135-d55krjg3dotss3e4-dwbbjmo0shwwwzv2 is not active vg_kvm/vsv12138-dyz9yl6swxbaxfrj-1pgri05suzqyagqq is not active vg_kvm/vsv12140-dykgcwqaauznoxrm-4ibft4womwuvpwkx is not active vg_kvm/vsv12142-dwtlkzoke0a05u1o-rqxjogzuvjvfttxs is not active vg_kvm/vsv12144-dn3gyu3uroppwgzg-wraejm2ynwnmtmd4 is not active vg_kvm/vsv12162-dalw90et3l9vbath-eo5womktdromb5sn is not active vg_kvm/vsv12211-dyyizuuag0mz4eok-je7du6slil1jlfzp is not active vg_kvm/vsv12329-d5xmlj2iqsuekgkm-cg6qplkwk48yiyeo is not active vg_kvm/vsv12333-drqvvb7fktr4n52b-estbavvooqdljpvo is not active vg_kvm/vsv12334-dbdf0lvefbgpqbxa-bmpnzainjdfvuavv is not active vg_kvm/vsv12339-d3yfeg1upsfoxbd3-nbxbccudjdtcgztt is not active vg_kvm/vsv12354-d1ctzl88dwv9znri-pzfp6weunhpdhvgn is not active vg_kvm/vsv12361-dbug4gr7r8mbsnva-yamgras92xuqin66 is not active vg_kvm/vsv12088-dqphwqqeuwxtmm7y-4rurc6jm0lvcj57q is not active vg_kvm/vsv12089-dbhimrcqvjkaxvxc-3vpljlh2xkojhb9e is not active vg_kvm/vsv12090-dndlzsmpj8qgm2gb-wm4sg3d6n2cq3tpm is not active vg_kvm/vsv12091-dyyxcz5cf4w4cu6z-gja2kph6egpjvyzh is not active vg_kvm/vsv12092-dnllln6x7uud75ss-um9zelymhc04ohd6 is not active vg_kvm/vsv12442-dyf2cgwdgaluli0s-cahwc74lrstt61gl is not active vg_kvm/vsv12456-dmiuo7guhru4snsf-yt55qun0wx5mtfkg is not active vg_kvm/vsv12087-dxbtb8w1g3tx5akk-bdqcsnlathemkrig is not active vg_kvm/vsv12234-dq82wyoapvxvh7zz-maaekbfd7h9ixks8 is not active vg_kvm/vsv12588-das5kos8qt1ziktl-aky0pxwg8ecv9b48 is not active vg_kvm/vsv12771-djaikmniei1vxia5-jhqasdwc763s3pmb is not active vg_kvm/vsv12784-d3k6aplejbvjrlcw-uwo81r0wkmeiptso is not active vg_kvm/vsv12139-dpnhdmg2odfabvfn-zqgiw42wfsgbbkrc is not active vg_kvm/vsv12137-dxtiqptvdwcrrcwa-22fkxorf6hl44evf is not active vg_kvm/vsv12866-d37htmu48aopiafv-sli9m0lt3dz0yeyl is not active vg_kvm/vsv12096-dz9pfno05wqq8vbg-sq8i5na5z8lrnm6n is not active vg_kvm/vsv12363-dtlh9w3tza99quoj-dlicfndz6e3pbfu0 is not active vg_kvm/vsv12890-dmaaydjldw3c13jk-iskdygh5pczveo3r is not active vg_kvm/vsv12320-dqmecms9nep6muns-xmmkziafa8mz1hta is not active vg_kvm/vsv12936-dhdfa36ovcdzfafp-ultnlmom4scugsw1 is not active vg_kvm/vsv12964-dctbp9xgo2gl8w6i-kbw9cinhoieea8av is not active vg_kvm/vsv12984-dwqjvkqc3aznhb4o-criyexpxvdndoscq is not active vg_kvm/vsv12143-d3s1toagczaiq4ma-oupwnk3eg27r41ze is not active vg_kvm/vsv13019-d5yrsi5a58bdzalc-i6mbum8zthk2arhs is not active vg_kvm/vsv13022-djszrasae5w7xegh-ehqsutbhrwd7qgsi is not active vg_kvm/vsv13027-dq6pg99jd2fphljg-u15x8sonbwukaree is not active vg_kvm/vsv12041-dlsmtmfa6fp4wcuc-bfhftkdhwfqa7sqb is not active vg_kvm/vsv13039-djtjimojqi3iaufp-csbinqtsvebzzmqx is not active vg_kvm/vsv13061-duocptl9qdtvwubp-feryqeejht4j24ql is not active vg_kvm/vsv13064-dexwkomdh62by80s-aymbuetu6p0btn4d is not active vg_kvm/vsv13076-drd4nrhy4u7fq1tn-f1kt8ubk2jch4ol3 is not active vg_kvm/vsv13082-ds9gjikqyg9nftua-rrhk4dlrr8cp3la7 is not active vg_kvm/vsv12759-du4yducr4ghi8eun-bzivqxabwaltaddm is not active vg_kvm/vsv12123-dzb9rktypp0jqinx-adtlzed02wpccxoa is not active vg_kvm/vsv13118-d0lj9gpvqfmox63i-9jstjcjkmemszrla is not active vg_kvm/vsv13121-du0r39n6wvnrclox-bviz76ig9vvexota is not active vg_kvm/vsv13133-d2fk6lg6bsppbkcs-hwk5bmsvqxa0y0bs is not active vg_kvm/vsv13204-dl6hzsmswetitsb4-bais6lpjbbhfkexn is not active vg_kvm/vsv13230-dtiyrmijg8mde5xy-vobukov1qxg8bx5j is not active vg_kvm/vsv13254-d3sdhrrgcvbwrsyp-yxaozxmt3clqzxxc is not active vg_kvm/vsv13271-drutw4ihq774xhs6-ecpwfmzcrqy0vvsm is not active vg_kvm/vsv13266-dmuj5cfeu2hrgfxw-2tpflycnwoedyidb is not active vg_kvm/vsv13380-d99ciziibe7rt3es-7fkh56myy79oushw is not active vg_kvm/vsv13487-d30yxx4wgfxcb9te-chc2bxojs7tvcwz3 is not active vg_kvm/vsv13502-dc2oivdodzo94jj3-1qmpba9ti4zzbfeg is not active vg_kvm/vsv13277-deudjco25w9cd3rv-re1ha5qkozses9i3 is not active vg_kvm/vsv12208-dfqhoqmtuwzktwqm-n9s5daovhtnyvyeh is not active vg_kvm/vsv13481-dthcwgpolombhfed-dkmesvsvmt1vem0d is not active vg_kvm/vsv13585-ddvwhm2vcqpq66uy-1brs98bctgr6bcq9 is not active vg_kvm/vsv13586-dadacaexxn8nkuy3-fyx8zp5mursbqhns is not active vg_kvm/vsv13587-dc0fmskwdbkmz7mg-ra53wgfksiv7tsqk is not active vg_kvm/vsv13569-dk6v0pyvz8eoohl5-qyndiwn8zriir75s is not active vg_kvm/vsv13603-d66mz2amabmd63fj-rkcqmtvfnlfmywou is not active vg_kvm/vsv13613-dykuo32mnkgf9nce-hhkhn118dhsedoc8 is not active vg_kvm/vsv13623-dksoizdtoglihaax-23o0zltutliyoo2q is not active vg_kvm/vsv13629-dxbpqqernfvufxrz-iv6jujdxm78hxjbn is not active vg_kvm/vsv13610-drrehnh36oipacdf-frnumdhlt823mntg is not active vg_kvm/vsv13654-dmlnhhqilztxff5a-4gawkysgtkze4st7 is not active vg_kvm/vsv13669-dk3ukgidvj0l4x4x-9d3vfao5qejzecdn is not active vg_kvm/vsv13672-d1r1608gu96b0ujo-nn7ejmxkbfce40m4 is not active vg_kvm/vsv13689-dhpzh8w7jlwi0zza-aqizb6tbeuk9k1ca is not active vg_kvm/vsv12477-dejmu0gihtllnj97-gaxwgicndwp25unx is not active vg_kvm/vsv13710-dv2bktawped5fz44-0bkdtrkp8tk2vynu is not active vg_kvm/vsv12125-d5d9pebts1rue7jt-vwnkapewjkfszked is not active vg_kvm/vsv13718-dmimkn9h1ax3t4am-8tdnpznya2reek8g is not active vg_kvm/vsv13545-dzaq2rjqfiyf5azl-kw35xomndhltjnrs is not active vg_kvm/vsv13723-daleipfjwu7zm7aj-s8jo6yhxwz4efl2y is not active vg_kvm/vsv13726-dpdqvmnzkvxkiz4y-cfudhed6m12hnlmj is not active vg_kvm/vsv12136-d0qvvamyx5qq1g6o-awtlh6pbotvlacyc is not active vg_kvm/vsv13258-du2kuwgkwhypcfzz-i6h00rd0exzw4b69 is not active vg_kvm/vsv13755-dqglblzy3akyohph-wtev5tlrs5lkkejd is not active vg_kvm/vsv13758-dhfrybljxildcky1-ktotkou3mpvxfjnq is not active vg_kvm/vsv13759-dxqbq7phpjgoa1ed-dwtly1dkuorvlyli is not active vg_kvm/vsv13762-di54bo3us4p8gefa-34s2uhcye1rbdbvw is not active vg_kvm/vsv13767-dwtsyqxhqbp8fa9w-us8xvwm9q6nqkmzx is not active vg_kvm/vsv13768-dadnlkrau2cwhlbd-cj8liz4hfolpmjfa is not active vg_kvm/vsv13769-dt4nwlcmpfqyenob-jvpfd5jut8kvcm8u is not active vg_kvm/vsv13781-dvckr0uqfwrsvmc6-zklf1olhlhzzfppp is not active vg_kvm/vsv12133-dp55dttaolpbd6n8-rlhoezgituqkyrov is not active vg_kvm/vsv12042-dnncyzlyhfxyaxkt-metx0ei0zlz3d2sn is not active vg_kvm/lvpool is not active vg_kvm/lvpool_tmeta is not active vg_kvm/lvpool_tdata is not active Locking LV i3usZzvdt59yaUKbclXDWJOFr1UhZRBaDHqs7X95DAADNmWTd9cXDfHR5aMoHLRx (EX) activation/volume_list configuration setting not defined: Checking only host tags for vg_kvm/lvpool. /dev/sda4: read_ahead is 256 sectors Getting driver version Getting target version for thin-pool Found thin-pool target v1.21.0. Getting target version for thin-pool Found thin-pool target v1.21.0. Getting target version for linear Found linear target v1.3.0. Getting target version for striped Found striped target v1.6.0. Creating vg_kvm-lvpool_tmeta Loading table for vg_kvm-lvpool_tmeta (253:0). Resuming vg_kvm-lvpool_tmeta (253:0). Creating vg_kvm-lvpool_tdata Loading table for vg_kvm-lvpool_tdata (253:1). Resuming vg_kvm-lvpool_tdata (253:1). global/thin_check_executable not found in config: defaulting to /usr/sbin/thin_check global/thin_check_options not found in config: defaulting to thin_check_options = [ "-q", "--clear-needs-check-flag" ] Executing: /usr/sbin/thin_check -q --clear-needs-check-flag /dev/mapper/vg_kvm-lvpool_tmeta /usr/sbin/thin_check failed: 1 Check of pool vg_kvm/lvpool failed (status:1). Manual repair required! Removing vg_kvm-lvpool_tdata (253:1) Removing vg_kvm-lvpool_tmeta (253:0) Unlocking /run/lock/lvm/A_vg_kvm Unlocking /run/lock/lvm/V_vg_kvm 

Answer 3

You can wipe the first 2k of the PV with the error before restoring your VG meta backup. The default metadatasize of LVM is 1MB, so this shouldn't reach your data. The dd will invalidate the header so LVM will think its an empty disk, then you can restore the PV and VG without it complaining about the state of the existing (invalid) PV/VG meta:

~]# dd if=/dev/zero bs=1k count=2 of=/dev/sdX ~]# sync # force to disk, LVM might use directio ~]# pvcreate -ff --uuid 06kvOm-xN2Y-iX1y-Tt69-LfFh-ltkG-TtRsgy --restorefile data /dev/bcache0 Physical volume "/dev/sdX" successfully created. ~]# vgcfgrestore --force data Restored volume group data 

It worked for me, be careful with the dd command!