1

I have a dedicated Centos 6.9 linux server with a company that crashed due to hardware failure and has been restored with the old HDD's on a usb caddy. I am trying to move lvm partitions over to the new internal hard disks. I have created a new volume group and same size logical volumes. The partitions are currently being used by kvm virtual machines. I have taken a snaphot of the lv. Then I am using something like the following to copy the partitions.

dd if=/dev/OldVolGroup/lv__snap of=/dev/NewVolGroup/lv bs=1M

This seems to work well with the following speed:

8192+0 records in 8192+0 records out 8589934592 bytes (8.6 GB) copied, 264.61 s, 32.5 MB/s

However, the ram and then swap slowly starts to get swallowed up. The box has 24GB of RAM and around 4GB spare when I start. It has a 2GB lv for swap.

I was able to turn some kvm machines off to free more memory, around 7-8GB and it just managed to finish the 8GB transfer without using all resources.

Can anyone tell me why it uses up so much memory and then swap? Is this normal?

Improve this question
3
  • try to use dd with direct IO "man dd | less +/direct" Commented Nov 29, 2017 at 11:42
  • thanks @c4f4t0r that's interesting. I have researched and still can't tell whether I need to actually set a flag on the hard disk not to cache or just an option on dd? Do you know? Have you used this for large disks? Commented Nov 30, 2017 at 14:00
  • when testing with dd you can oflag and iflag to control the os caching and simulate direct IO Commented Nov 30, 2017 at 15:06

1 Answer 1

Reset to default
0

It's because dd reads into cache before writing. If you have top running you'll see that it's cache that is being used up before memory. All you can really do is try a larger block size. It's not a good thing to be doing on a running server but obviously it's what you're working with.

Improve this answer
2
  • Thanks for the information. I have added an extra swap lv partition (20G). Is this a cache buffer? So do you think it will have a cap or level out? I'm wondering how someone has gone about transferring their 1.5T partition or similar? Commented Nov 29, 2017 at 12:48
  • I think it will still use free memory first then swap, as that's what kernel memory management does. Remember, dd is a system level tool that is pretty much as old as POSIX itself and 1TB filesystems just didn't exist when it was written. There are a few possible alternatives such as ddrescue and pv which may be friendlier. Also the comment from @c4f4t0r might be useful. Commented Nov 29, 2017 at 13:03

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.