I’m using rsync to sync files from my computer to a FAT formatted SD card. Using the --update flag to rsync makes it “skip files that are newer on the receiver”. It seems that it should work as follows: After the first sync, any subsequent syncs will only transfer those files which changed in the meantime. However, I noticed that transfer times are usually longer than expected, which led me to think that things are not working as they should.
As --update relies only on modification date, obviously something is wrong with it. After ruling out several other possibilities, I’ve guessed that the modification date of some files get mangled. A quick check about FAT revealed that FAT can only save modification times in 2 second granularity. I’ve also used rsync’s --archive flag, which among other things attempts to preserve modifications times of the files it copies. But what about FAT’s lower granularity for modification time? That was apparently the culprit. Some files when copied got a modification time which was up to 1 second before the original modification time! Hence, every time I’ve synced, from rsync’s perspective, the target was older than the source, and hence needs to be overwritten.
This can be demonstrated by the following session:
$ dd if=/dev/urandom of=test-even bs=1M count=100 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 6.20716 s, 16.9 MB/s $ cp test-even test-odd $ touch -t 201701180000.00 test-even $ touch -t 201701180000.01 test-odd $ stat -c %Y test-* 1484690400 1484690401
We’ve created two files test-even and test-odd with random content. We’ve set the modification time of test-even to be on an even second, and for test-odd to be on an odd second (one second apart). Simply syncing the files to a FAT partitioned device works as expected:
$ rsync -av --update test-{even,odd} /media/guyru/4E09-6457
sending incremental file list
test-even
test-odd
sent 209,766,550 bytes received 54 bytes 7,627,876.51 bytes/sec
total size is 209,715,200 speedup is 1.00
Resyncing immediately also works:
$ rsync -avv --update test-{even,odd} /media/guyru/4E09-6457
sending incremental file list
delta-transmission disabled for local transfer or --whole-file
test-even is uptodate
test-odd is uptodate
total: matches=0 hash_hits=0 false_alarms=0 data=0
sent 78 bytes received 136 bytes 428.00 bytes/sec
total size is 209,715,200 speedup is 979,977.57
Both files are reported as being up-to-date. But what happens if we unmount and remount the device and then sync?
$ umount /media/guyru/4E09-6457
$ udisksctl mount --block-device /dev/disk/by-uuid/4E09-6457
Mounted /dev/sdd1 at /media/guyru/4E09-6457.
$ time rsync -avv --update test-{even,odd} /media/guyru/4E09-6457
sending incremental file list
delta-transmission disabled for local transfer or --whole-file
test-even is uptodate
test-odd
total: matches=0 hash_hits=0 false_alarms=0 data=104857600
sent 104,883,314 bytes received 131 bytes 7,769,144.07 bytes/sec
total size is 209,715,200 speedup is 2.00
This time test-odd was reported as needing to be updated. So what changed? My guess is that while the device was mounted, modifcation dates were cached using normal granularity. Once I’ve remounted the file-system, the cache was purged, the the low-granularity modification date was used. That meant test-odd now had a modificaiton time which was one second before the original.
So how can it be avoided? Probably the trivial solution is to tell rsync not to copy over the modification date. This means that the copied files would probably anyway get a new modification time which is more than 1 second later than the original, hence even when rounding the modification time as FAT does it will still work. But a much better solution is built right into rsync: --modify-window
When comparing two timestamps, rsync treats the timestamps as
being equal if they differ by no more than the modify-window
value. This is normally 0 (for an exact match), but you may
find it useful to set this to a larger value in some situations.
In particular, when transferring to or from an MS Windows FAT
filesystem (which represents times with a 2-second resolution),
–modify-window=1 is useful (allowing times to differ by up to 1
second).
By using --modify-window=1 we can overcome the problem, as the files will be deemed as modified at the same time. Indeed it solves the problem:
$ rsync -avv --update --modify-window=1 test-{even,odd} /media/guyru/4E09-6457
sending incremental file list
delta-transmission disabled for local transfer or --whole-file
test-even is uptodate
test-odd is uptodate
total: matches=0 hash_hits=0 false_alarms=0 data=0
sent 82 bytes received 140 bytes 444.00 bytes/sec
total size is 209,715,200 speedup is 944,663.06