Skip to content

rsync

Great way to sync one location to another, local or remote. Note that this does not mean full synchronization, two commands with reversed source and destinations are required to accomplish that.

Syntax Examples

Giving additional ssh options

rsync -e 'ssh -o ConnectTimeout=10 -o PasswordAuthentication=no' -Rai /home target:/

Exclude Filters

Exclude filters are kinda weird.

  • They're case sensitive and there's no way to be case insensitive.
  • They are relative to the root of the source URI. EG, rsync --exclude="Desktop/" ~/ remotehost:~/

Here is an example of what to use in --exclude-from=file.txt

**Cache
**Caches
**cache
**caches
**/.dropbox
**Previews.lrdata
**/Library/Application\ Support/Google/Chrome

Include Filters

Include filters should be specified before exclude filters if you have very broad exclude filters. For instance, to only transfer mp4 files:

rsync -ai --include='*.mp4' --exclude="*" remote-server:/media/movies/ ~/Movies/

If you exclude first, nothing will be transferred.

Long and Partial Transfers

If you're doing transfers which you'd like to monitor and risk being cut off, use this syntax:

rsync -e ssh -az --partial --progress ./foo remotehost:~/bar/

This will resume broken file transfers where they were left off, and give you completion statistics with transfer rate, percent complete and estimated time left.

rsync can be used to create a hard linked local copy of a whole tree. This is useful if you don't have GNU cp where the same could be done with simply cp -lrp. On OS X with homebrew, GNU cp can be installed via brew install coreutils and accessed via gcp. See also ls -la /usr/local/opt/coreutils/bin/.

Slashes are really really important here; this won't work if you get them wrong. Absolute paths must be given, thus ${PWD} and ${HOME} vs ~

rsync -aP --link-dest="${PWD}/src" ./src/ dst #recursively hard link ./src to dst

For example:

rsync -aivv --link-dest="${HOME}/Dropbox" ${HOME}/Dropbox/some_dir ${HOME}/temp/

This will create the directory ${HOME}/temp/some_dir and hard link all the files from the source into the destination. It should only take a few seconds. Lines with 'hf' indicate a hard linked file. Lines with 'cd' indicate 'created directory'.

rsync can copy not only data, but also filesystem attributes, and if these differ between the link-dest and the src, a hard link may not be created but instead a copy of the file from the local filesystem is made and correct metadata is applied from the source.

Backup to remote host with timestamp

The following example copies files from /local/host/src into /remote/host/path-2 but hard links those files against the data in /remote/host/path-1 on the receiving side if any files are identical. This avoids transferring data, is an efficient use of disk space for files that will be archived (IE: not changed in-place), and allows deletion of older copies of the data while keeping newer copies.

rsync -aP --link-dest="/remote/host/path-1" "/local/host/src/" "$REMOTE_HOST":/remote/host/path-2/

Or for a daily datestamped backup using GNU date (this example will not work with BSD date like macOS has):

rsync -aivv --link-dest="/archive/path/$(date -d "-1 day" "+%F")/" /src/data/ "${REMOTE_HOST}:/archive/path/$(date "+%F")/"

Move files to another server in small batches

This is useful if you want to gradually clear up disk space rather than waiting until the end of a transfer of a large number of files to clear up disk space in one large operation.

while date ;
files=$(find /srv/backups/scribe/./ -type f -mtime +400 | head -n 500) ;
echo md5 of files ${#files} is $(echo ${files} | md5sum) ;
[ ! -z "${files}" ] ; do
  sudo rsync --bwlimit 20000 -RaPi --remove-source-files ${files} root@10.2.17.7:/srv/backups/scribe-sea/ ; echo sleeping ;
  sleep 10 ;
done ;

This relies on gnu date, so use gdate if used on OS X.

rsync -aPiv \
  --remove-source-files \
  --bwlimit 20000 \
  --exclude="**$(date -d "1 month ago" "+%Y-%m")**" \
  --exclude="**$(date "+%Y-%m")**" \
  --no-links \
  /srv/backups/scribe/* \
  root@10.2.17.7:/srv/backups/scribe-sea/

Reduce time precision during comparison

This is useful for rsyncing to FAT filesystems where time precision is 2 seconds.

rsync --modify-window=1 # allow 1 second of difference in timestamps

Connect as a normal user and escalate using sudo

Many times you have to copy files that your remote user does not have access to without sudo. You can perform this hoop-jump with ease using the following syntax:

rsync --rsync-path='sudo rsync' $REMOTE_HOSTNAME:/etc/kubernetes/admin.conf "$HOME/.kube/config"

Use rsync with find

You can use rsync --files-from=- to read files from stdin. You have to make sure the path used in find matches the source path for rsync.

cd /tank/photos/ && find . -type f -print0 | rsync -ia0 --files-from=- . remote-host:/dest/

rsync to fat32

fat32 has 4GiB - 1 byte (0xFFFFFFFF, or 4294967295 bytes) file size limit. It also has some other shortcomings that I have not worked through yet, so this example is far from perfect.

rsync --max-size=4GiB-1

See Also