DEV Community

Cover image for rsync - 10 examples in 11 days (Day 03)
m4r4v
m4r4v

Posted on • Edited on

rsync - 10 examples in 11 days (Day 03)

#linux #rsync #bash #tutorial

Day 03

10 examples in 11 days

Exclude and Include

Now it's time to explain the exclude and include option. This options are great if you want to get control of what needs to be sync.

With the --exclude option, you can exclude a file, a directory and even a list of files/directories listed inside a file. Let's check out the the rsync syntax with the --exclude option.

I will create a file called exclude-me.txt inside ~/Users folder

iamgroot@laptop:~$ touch ~/Users/exclude-me.txt && echo "Hello World" >> ~/Users/exclude-me.txt
Enter fullscreen mode Exit fullscreen mode

Now Let's check the size and files inside ~/Users and ~/Sync/Users folders

iamgroot@laptop:~$ stat -c "%s %n" ~/Users/* && stat -c "%s %n" ~/Sync/Users/* 12 /home/iamgroot/Users/exclude-me.txt 151 /home/iamgroot/Users/user01.json 57 /home/iamgroot/Users/user02.json 69 /home/iamgroot/Users/user03.json 43 /home/iamgroot/Users/user.config 151 /home/iamgroot/Sync/Users/user01.json 57 /home/iamgroot/Sync/Users/user02.json 69 /home/iamgroot/Sync/Users/user03.json 88 /home/iamgroot/Sync/Users/user.config 4096 /home/iamgroot/Sync/Users/Users
Enter fullscreen mode Exit fullscreen mode

To exclude a file, we need to wrap the file name in quotes after the --exclude option. This way, rsync leaves the file in source and doesn't include it inside destination.

iamgroot@laptop:~$ rsync -vhr --exclude 'exclude-me.txt' ~/Users ~/Sync/ sending incremental file list sent 146 bytes received 17 bytes 326.00 bytes/sec total size is 320 speedup is 1.96
Enter fullscreen mode Exit fullscreen mode

Let's check if the file exclude-me.txt was copied from source to destination

iamgroot@laptop:~$ stat -c "%s %n" ~/Users/* && stat -c "%s %n" ~/Sync/Users/* 12 /home/iamgroot/Users/exclude-me.txt 151 /home/iamgroot/Users/user01.json 57 /home/iamgroot/Users/user02.json 69 /home/iamgroot/Users/user03.json 43 /home/iamgroot/Users/user.config 151 /home/iamgroot/Sync/Users/user01.json 57 /home/iamgroot/Sync/Users/user02.json 69 /home/iamgroot/Sync/Users/user03.json 88 /home/iamgroot/Sync/Users/user.config 4096 /home/iamgroot/Sync/Users/Users
Enter fullscreen mode Exit fullscreen mode

It wasn't included. Like I said, this option comes very handy. It also works with directories. Let's create a folder and include multiple files in it

iamgroot@laptop:~$ mkdir ~/Users/exclude-dir && touch ~/Users/exclude-dir/e{1..5}.txt && ls -a ~/Users/exclude-dir . .. e1.txt e2.txt e3.txt e4.txt e5.txt
Enter fullscreen mode Exit fullscreen mode

Now we have a folder called exclude-dir with 5 files calles e1.txt, e2.txt, etc

Let's sync them excluding this folder

iamgroot@laptop:~$ rsync -vhr --exclude 'exclude-dir' ~/Users ~/Sync sending incremental file list Users/exclude-me.txt Users/user.config Users/user01.json Users/user02.json Users/user03.json sent 731 bytes received 112 bytes 1.69K bytes/sec total size is 332 speedup is 0.39 jorge@code-lab:~$ stat -c "%s %n" ~/Users/* && stat -c "%s %n" ~/Sync/Users/* 4096 /home/jorge/Users/exclude-dir 12 /home/jorge/Users/exclude-me.txt 151 /home/jorge/Users/user01.json 57 /home/jorge/Users/user02.json 69 /home/jorge/Users/user03.json 43 /home/jorge/Users/user.config 12 /home/jorge/Sync/Users/exclude-me.txt 151 /home/jorge/Sync/Users/user01.json 57 /home/jorge/Sync/Users/user02.json 69 /home/jorge/Sync/Users/user03.json 43 /home/jorge/Sync/Users/user.config
Enter fullscreen mode Exit fullscreen mode

We excluded the exclude-dir folder but it copied the exclude-me.txt file that we previously did not wanted to include at destination. How can we tell rsync to exclude this file as well?, easy, just tell rsync you want to eclude it as well

I will remove the file from destination and repeat the rsync command and check what was sync into destination

iamgroot@laptop:~$ rm ~/Sync/Users/exclude.me-txt iamgroot@laptop:~$ rsync -vhr --exclude 'exclude-dir' --exclude 'exclude-me.txt' ~/Users ~/Sync sending incremental file list sent 150 bytes received 17 bytes 334.00 bytes/sec total size is 320 speedup is 1.92 iamgroot@laptop:~$ stat -c "%n" ~/Sync/Users/* /home/iamgroot/Sync/Users/user01.json /home/iamgroot/Sync/Users/user02.json /home/iamgroot/Sync/Users/user03.json /home/iamgroot/Sync/Users/user.config
Enter fullscreen mode Exit fullscreen mode

Great, we achieve what we wanted.

If you really think about it, using mutiple --exclude options is not very clean and specially if you are working with multiple files/folders. There is a simple way to use --exclude in the same rsync line and its using curly braces

iamgroot@laptop:~$ rsync -vhr --exclude={'exclude-dir','exclude-me.txt'} ~/Users ~/Sync sending incremental file list sent 150 bytes received 17 bytes 334.00 bytes/sec total size is 320 speedup is 1.92
Enter fullscreen mode Exit fullscreen mode

Note: You have to watch out and not to include whitespaces between the files/folders isnide the curly braces, if you include a whitespace it will sync all files/folders

GOOD rsync -vhr --exclude={'exclude-dir','exclude-me.txt'} ~/Users ~/Sync

NOT GOOD rsync -vhr --exclude={'exclude-dir', 'exclude-me.txt'} ~/Users ~/Sync

Again, if we have multiple, by multiple a mean hundreds or even more, using this technique is not so handy.

The good news is that rsync have a way to do this easily hehe (well done rsync!!!).

Today, we are very used to use git and just like git has .gitignore with rsync is pretty much the same thing. We can create a file and list all the files we want to exclude. To achieve this, we have the --exclude-from option.

I will create a file called rsync-ignore.txt and inside of it I will list all files that we want to exclude, notice that I will include the rsync-ignore.txt file as well

iamgroot@laptop:~$ touch ~/Users/rsync-ignore.txt && echo -e "exclude-dir\nexclude-me.txt\nrsync-ignore.txt" >> ~/Users/rsync-ignore.txt && cat ~/Users/rsync-ignore.txt exclude-dir exclude-me.txt rsync-ignore.txt
Enter fullscreen mode Exit fullscreen mode 
iamgroot@laptop:~$ rsync -vha --exclude-from='/home/iamgroot/Users/rsync-ignore.txt' ~/Users ~/Sync sending incremental file list Users/ sent 165 bytes received 20 bytes 370.00 bytes/sec total size is 320 speedup is 1.73
Enter fullscreen mode Exit fullscreen mode

notice I included the full path of the --exclude-from, using (~/) will not work

iamgroot@laptop:~$ ls -h ~/Sync/Users user01.json user02.json user03.json user.config
Enter fullscreen mode Exit fullscreen mode

As you can see, everything worked just as expected. Now I will point out some other options to be used with the --exclude option.

What about the Include option?

The include option behaves pretty much the same as --exclude, is just the opposite. It's useful when you mix them along the same line. Let's see an example

iamgroot@laptop:~$ rsync -avhm --include '*.json' --exclude '*.txt' ~/Users ~/Sync building file list ... done Users/ Users/user.config Users/user01.json Users/user02.json Users/user03.json sent 695 bytes received 95 bytes 1.58K bytes/sec total size is 320 speedup is 0.41
Enter fullscreen mode Exit fullscreen mode

What this did was:

  • included all .json files
  • excluded all the .txt files
  • using the -m option it didn't copy the exclude-dir folder since it was empty because inside there were only .txt files and got empty

Below are some more options to be used.

Exclude Files using the wildcard (*)

This option let's you specify a pattern you want to exclude. To achieve this, you need to use the wilcard (*), it can also be used to exclude specific filetypes (it's all about how you use the wildcard. Let's see an example:

iamgroot@laptop:~$ rsync -avh --exclude '*.json' ~/Users ~/Sync
Enter fullscreen mode Exit fullscreen mode

The above example will exclude all .json files. The * can be used at the beggining, at the end, in the middle, etc. i.e. rsync -avh --exclude '*ignore*' ~/Users ~/Sync will ignore the *rsync-ignore.txt* file

Exclude Files based in File Size

This option let's you specify between the maximum or minimum size of the files you want to exclude.

iamgroot@laptop:~$ rsync -avh --min-size=1K --max-size=50K ~/Users ~/Sync
Enter fullscreen mode Exit fullscreen mode

The above example will exclude any file bigger than 1K and smaller than 50K

Here is what the man page says about it

--max-size=SIZE

This tells rsync to avoid transferring any file that is larger than the specified SIZE. The SIZE value can be suffixed with a string to indicate a size multiplier, and may be a fractional value
(e.g. "--max-size=1.5m").

This option is a transfer rule, not an exclude, so it doesn’t affect the data that goes into the file-lists, and thus it doesn’t affect deletions. It just limits the files that the receiver requests to be transferred.

The suffixes are as follows: "K" (or "KiB") is a kibibyte (1024), "M" (or "MiB") is a mebibyte (1024*1024), and "G" (or "GiB") is a gibibyte (1024*1024*1024). If you want the multiplier to be 1000 instead of 1024, use "KB", "MB", or "GB". (Note: lower-case is also accepted for all values.) Finally, if the suffix ends in either "+1" or "-1", the value will be offset by one byte in the indicated direction.

Examples: --max-size=1.5mb-1 is 1499999 bytes, and --max-size=2g+1 is 2147483649 bytes.

Note that rsync versions prior to 3.1.0 did not allow --max-size=0.

--min-size=SIZE

This tells rsync to avoid transferring any file that is smaller than the specified SIZE, which can help in not transferring small, junk files. See the --max-size option for a description of SIZE and other information.

Note that rsync versions prior to 3.1.0 did not allow --min-size=0.

Ok, that'll be for today's example, thanks for reading!!!

Follow, ❤ or 🦄

Top comments (0)