The rsync man page. Required, if not reading, at least reference. MOST of what you need is there.
This page is my usage notes, things I learned the hard way, and will probably forget if I don't put them some place where I can find them.
--filter="merge file.txt"rsync option to use them. This can simplify your command line to something like this:
rsync $OPTS --filter="merge file.txt" $SOURCE $DESTBut here's an example...
I have a system that has ten copies of the same kind of data. One directory holds the production data, the other nine are development copies -- here, the data isn't interesting, it's the programs and scripts to process the data that are important.
Here's an rsync filter file that will grab just individual directories from certain subdirectories:
Here's the magic -- Everything UP TO the desired directory must first be "+" marked. So, if you want /APP/APP952/BATCH but not the rest of /APP/APP952/, the answer is you want to have:+ / - /mnt - /proc - /sql_extract - /tmp - /dev - /LDU1 - /opt/ATMDIALOG_BACKUP - /lpp - /usr/java*/ - /usr/lpp/ + /APP + /APP/APP551 + /APP/APP623 + /APP/APP623/BATCH/ + /APP/APP623/SPECFILES/ + /APP/APP623/DATAFILES/ + /APP/APP623/HELPFILES/ + /APP/APP623/REPORTSPECS/ - /APP/APP623/* + /APP/APP650 + /APP/APP650/BATCH/ + /APP/APP650/SPECFILES/ + /APP/APP650/DATAFILES/ + /APP/APP650/HELPFILES/ + /APP/APP650/REPORTSPECS/ - /APP/APP650/* + /APP/APP777 + /APP/APP777/BATCH/ + /APP/APP777/SPECFILES/ + /APP/APP777/DATAFILES/ + /APP/APP777/HELPFILES/ + /APP/APP777/REPORTSPECS/ - /APP/APP777/* + /APP/APP789 + /APP/APP789/BATCH/ + /APP/APP789/SPECFILES/ + /APP/APP789/DATAFILES/ + /APP/APP789/HELPFILES/ + /APP/APP789/REPORTSPECS/ - /APP/APP789/* + /APP/APP810 + /APP/APP810/BATCH/ + /APP/APP810/SPECFILES/ + /APP/APP810/DATAFILES/ + /APP/APP810/HELPFILES/ + /APP/APP810/REPORTSPECS/ - /APP/APP810/* + /APP/APP952 + /APP/APP952/BATCH/ + /APP/APP952/SPECFILES/ + /APP/APP952/DATAFILES/ + /APP/APP952/HELPFILES/ + /APP/APP952/REPORTSPECS/ - /APP/APP952/* + /APP/APP953 + /APP/APP953/BATCH/ + /APP/APP953/SPECFILES/ + /APP/APP953/DATAFILES/ + /APP/APP953/HELPFILES/ + /APP/APP953/REPORTSPECS/ - /APP/APP953/* - /APP/*
to make it work. This will NOT work:+ / + /APP + /APP/APP952 + /APP/APP952/BATCH - /APP/APP952/* - /APP/*
as you are excluding everything...and it will never look in /APP/APP952 to find the BATCH directory.- / # BROKE! + /APP/APP952/BATCH # BROKE!
Hopefully, this saves some hair pulling.
Newsflash: this is something you very possibly want to disable.
In real life, we've seen systems spend far more time trying to figure out how little to transfer than simply transfering entire files if a change has been made. In fact, we saw HUGE problems with this at an employer of mine -- sometimes the transfers would go fast, other times slow. But overall, things were ALWAYS faster if we used the -W option.
I'd suggest benchmarking it before making a decision, but I have yet to find something that it made better. Most likely, it was a bigger win back in the dial-up and slow Internet days, but when moving big files with lots of little changes over fat (but high-latency) pipes, I've never seen things go slower when I add a -W to the rsync command line.--whole-file, -W copy files whole (w/o delta-xfer algorithm)
Holland Consulting home
Contact Holland Consulting
since November 10, 2017
(C)opyright 2017, 2022 Nick Holland, Holland Consulting