rsync notes

What...

rsync is possibly one of the most useful tools to the Unix administrator. If you aren't using it, learn it. If you are, you can always probably use it better.

The rsync man page. Required, if not reading, at least reference. MOST of what you need is there.

This page is my usage notes, things I learned the hard way, and will probably forget if I don't put them some place where I can find them.


--filter Include/Exclude statements

This looks so easy. And for very simple cases, it can be. But for more complex cases, it gets really weird really fast. Personally, I like to put the include/exclude statements into a separate file, then use a:
    --filter="merge file.txt"
rsync option to use them. This can simplify your command line to something like this:
    rsync $OPTS --filter="merge file.txt" $SOURCE $DEST
But here's an example...

I have a system that has ten copies of the same kind of data. One directory holds the production data, the other nine are development copies -- here, the data isn't interesting, it's the programs and scripts to process the data that are important.

Here's an rsync filter file that will grab just individual directories from certain subdirectories:

+ /
- /mnt
- /proc
- /sql_extract
- /tmp
- /dev
- /LDU1
- /opt/ATMDIALOG_BACKUP
- /lpp
- /usr/java*/
- /usr/lpp/
+ /APP
+ /APP/APP551
+ /APP/APP623
+ /APP/APP623/BATCH/
+ /APP/APP623/SPECFILES/
+ /APP/APP623/DATAFILES/
+ /APP/APP623/HELPFILES/
+ /APP/APP623/REPORTSPECS/
- /APP/APP623/*
+ /APP/APP650
+ /APP/APP650/BATCH/
+ /APP/APP650/SPECFILES/
+ /APP/APP650/DATAFILES/
+ /APP/APP650/HELPFILES/
+ /APP/APP650/REPORTSPECS/
- /APP/APP650/*
+ /APP/APP777
+ /APP/APP777/BATCH/
+ /APP/APP777/SPECFILES/
+ /APP/APP777/DATAFILES/
+ /APP/APP777/HELPFILES/
+ /APP/APP777/REPORTSPECS/
- /APP/APP777/*
+ /APP/APP789
+ /APP/APP789/BATCH/
+ /APP/APP789/SPECFILES/
+ /APP/APP789/DATAFILES/
+ /APP/APP789/HELPFILES/
+ /APP/APP789/REPORTSPECS/
- /APP/APP789/*
+ /APP/APP810
+ /APP/APP810/BATCH/
+ /APP/APP810/SPECFILES/
+ /APP/APP810/DATAFILES/
+ /APP/APP810/HELPFILES/
+ /APP/APP810/REPORTSPECS/
- /APP/APP810/*
+ /APP/APP952
+ /APP/APP952/BATCH/
+ /APP/APP952/SPECFILES/
+ /APP/APP952/DATAFILES/
+ /APP/APP952/HELPFILES/
+ /APP/APP952/REPORTSPECS/
- /APP/APP952/*
+ /APP/APP953
+ /APP/APP953/BATCH/
+ /APP/APP953/SPECFILES/
+ /APP/APP953/DATAFILES/
+ /APP/APP953/HELPFILES/
+ /APP/APP953/REPORTSPECS/
- /APP/APP953/*
- /APP/*
Here's the magic -- Everything UP TO the desired directory must first be "+" marked. So, if you want /APP/APP952/BATCH but not the rest of /APP/APP952/, the answer is you want to have:
+ /
+ /APP
+ /APP/APP952
+ /APP/APP952/BATCH
- /APP/APP952/*
- /APP/*
to make it work. This will NOT work:
- /			# BROKE!
+ /APP/APP952/BATCH	# BROKE!
as you are excluding everything...and it will never look in /APP/APP952 to find the BATCH directory.

Hopefully, this saves some hair pulling.


rsync delta-transfer may not be your friend

rsync's big claim to fame has alwasys been its ability to find what needs to be sent and copy over just the parts of files that changed, they call it their "Delta-xfer algorithm".

Newsflash: this is something you very possibly want to disable.

In real life, we've seen systems spend far more time trying to figure out how little to transfer than simply transfering entire files if a change has been made. In fact, we saw HUGE problems with this at an employer of mine -- sometimes the transfers would go fast, other times slow. But overall, things were ALWAYS faster if we used the -W option.

--whole-file, -W         copy files whole (w/o delta-xfer algorithm)
I'd suggest benchmarking it before making a decision, but I have yet to find something that it made better. Most likely, it was a bigger win back in the dial-up and slow Internet days, but when moving big files with lots of little changes over fat (but high-latency) pipes, I've never seen things go slower when I add a -W to the rsync command line.
 

Holland Consulting home page
Contact Holland Consulting


since November 10, 2017

(C)opyright  2017, 2022 Nick Holland, Holland Consulting

Published: 11/10/2017
Revised: 8/9/2022