[SOLVED] I found a minor flaw in my Rsync Back-Up strategy

[Dennola4]

Hi guys,

back on Sept 11, of 2010, several awesome Salix devs helped me to revise a back-up strategy using rsync which has been working ever since. The link to that extremely tedious thread is here: http://www.salixos.org/forum/viewtopic.php?f=16&t=1257

Using the solution from the above thread, I created for myself an instructional document, which I follow point-for-point when it is time to do a back-up. The document reads:

Code: Select all

----------------------------------------------------------------------------------
1) plug the destination drive in and MOUNT all partitions that appear on desktop
   (should be two).
----------------------------------------------------------------------------------
2) identify which is the ROOT partition and which is the HOME partition. One will
   be /media/disk and one will be /media/disk-1.
----------------------------------------------------------------------------------
3) Become root@darkstar with su -
----------------------------------------------------------------------------------
4) If ROOT partition is /media/disk-1, then run:
   #rsync -avx / /media/disk-1

   If ROOT partition is /media/disk, then run:
   #rsync -avx / /media/disk
----------------------------------------------------------------------------------
5) If HOME partition is /media/disk-1, then run:
   #rsync -avx --exclude=.gvfs --exclude=.gnupg /home/dennis/ /media/disk-1/dennis

   If HOME partition is /media/disk, then run:
   #rsync -avx --exclude=.gvfs --exclude=.gnupg /home/dennis/ /media/disk/dennis
----------------------------------------------------------------------------------

This solution has saved my butt many times, by giving me an updated, bootable backup HDD (sdb) identical to my existing HDD (sda). Now a few times over the past few years I have mistyped the rsync command and corrupted the back-up. In these instances I use the dd command to copy sda to sdb, as the HDD's are identical pieces of hardware. So really, I have a pretty foolproof back-up system.

And now the point of all of this: The other day, after a few months worth of rsync back-ups, I decided to show a friend of mine how Meld Difference Viewer works. I plugged in sdb (the back-up) via enclosure, then compared each partition as corresponds to those on sda. They were not identical. What I found was that while any newly added or moved file on sda is getting copied to the corresponding location on sdb, the old file is not being removed. So, for instance, if I were to move /home/dennis/foo to /home/dennis/bar on sda, then run my rsync commands above, the result is that I now have both /home/dennis/foo AND /home/dennis/bar on sdb, where I should just have /home/dennis/bar.

This is problematic for obvious reasons. Please, how do I fix it?

Dennis in New Orleans

[Shador]

--del an alias for --delete-during
--delete delete extraneous files from dest dirs
--delete-before receiver deletes before transfer (default)
--delete-during receiver deletes during xfer, not before
--delete-delay find deletions during, delete after
--delete-after receiver deletes after transfer, not before
--delete-excluded also delete excluded files from dest dirs

[Dennola4]

Hello again Shador,

I have obviously read the man pages for rsync, but it's a lot of involved reading and I just wasn't sure that -- delete was *precisely* the option I needed. Past experience has taught me to ask an authority when in doubt rather than "just trying stuff". Anyway, per you suggestion I tested it and yes, it's just what I needed.

Thanks as always, you're a great help.


Dennis in New Orleans

[Shador]

No problem, you're welcome.
Yes, it's quite long, but fortunately there are also many examples in there. I recommend that if you're unsure whether an option works you just setup some test directories and run the rsync command on those to test whether it does what you want. I'd recommend that for a backup strategy anyway.

[thenktor]

I recommend that if you're unsure whether an option works you just setup some test directories and run the rsync command on those to test whether it does what you want.

Especially because rsync is a powerful tool and it can be hard to find the right options.

[Dennola4]

Actually just prior to your post I had already done what you suggested....

....I created ~/test/foo and /etc/test/foo on sda, ran rsync, then checked that they were copied to the corresponding directories on sdb. I then moved the test files on sda respectively to ~/test2/foo and /usr/test/foo, ran rsync with the new --delete command, and yes....the first two were deleted and the latter two were added.

So yes, I concur that testing is the only way to know for sure. I guess my reluctance to experiment was the possibility of late night blunder in which I somehow reversed the source and destination, copying sdb to sda, sort of like an if/of reversal when using the dd command.

But on that note, thanks again for the help. It's a cool thing to be able to use an OS designed and maintained by people that care about what they are doing and have great attitudes in general. Fingers crossed the Salix team stays pure and doesn't succumb to the Dark Side.

[mimosa]

I actually found the "extremely tedious" thread you link to at the top of this one made for very useful reading. I recently bought a large external disk, and mean to get to grips with backup. I'll certainly be looking back over that thread when I get round to it.

The business with the trailing slash is indeed confusing, and the issue occurs with cp and mv too. As I remember, those two commands treat the issue differently. I get it wrong all the time.

[Shador]

The business with the trailing slash is indeed confusing, and the issue occurs with cp and mv too. As I remember, those two commands treat the issue differently. I get it wrong all the time.

No, actually not. The trailing slash is almost completely meaningless with cp and mv. If the destination of cp or mv does not contain a trailing slash (e.g. cp a b) the behaviour is to copy to the directory b if such a directory does exist, i.e. a duplicate named b/a of the file a is created. Otherwise if no directory b exists a duplicate of a named b is created in the same directory.
Now adding a trailing slash doesn't change much (e.g. cp a dir/). The only difference is now that dir must be an already existent directory. This means if no directory exists with the name dir/ the copy does fail. Otherwise it behaves exactly as "cp a dir" when there is a directory dir. I.e. appending a slash does only make sure you're really copying to a directory and fails otherwise.
For mv the behaviour is exactly the same it just doesn't keep the source file. The behaviours is also exactly the same with a trailing slash on the source argument. This is because generally on Unix systems a directory can be referred by either name or name/, but a file just by name. Therefor consequently name means work on file or directory and name/ means just work on directory.

With rsync the trailing slash has a different and what I would call real meaning. The first big difference is here that we're talking with rsync about a trailing slash with only the source (not destination) argument. A trailing slash with the destination is actually completely meaningless with rsync as it just operates on directories. Now the slash with rsync distinguishes between copying the source directory + its content (rsync dir ...) vs. just the content (rsync dir/). That's happening also we're actually referring the exactly same filesystem object (i.e. the directory named dir). But that behavior is relatively easy to memorize as well. With rsync dir we're sort of ignoring the directory nature of dir and therefor copy it as well. With the dir/ we actually refer to dir being a container of other files as a directory therefor we're only copying the content.

[Dennola4]

Glad to hear that the thread is useful.

Over the years my system has become very personalized. There are launchers pointing to scripts that do all sorts of things, permissions changed all over the place, system files edited, daemons running, and so on so that my computer will do what I need it to do. I now take this set-up for granted, but if I were ever to corrupt my system and lose all of the work I put into this customization it would take a lot of time and effort to get it back to the way I have it now.

So I wanted my backup solution to be able to not only save my personal data; I wanted the entire system to be copied identically to another location so that in the event of a meltdown I could just pop in the backup and boot up as though it never happened. Incidentally, this has happened at least four times over the past couple years. I fry my system, pull out my backup, pop it in, then boot up and immediately back it up onto the corrupted HDD.

What I did was purchase a HDD identical in architecture to the one in my laptop, wiped it clean with dban, then used the dd command to copy the entire contents of my HDD block for block to the new usb-connected mass storage device. This created a bootable clone of my existing HDD. Once a month or so I connect the backup HDD via usb (the cord costs $5-$10 US), run the rsync command, and any changes made to the source are quickly made to the destination. Back up time: 5 minutes, maybe 10 if I have downloaded a lot of big files. I use an external storage device for archiving all of my excess big files (pictures, music, video, etc) to keep my working HDD lean. It's only 80 gigs and it's still only half full.

I'm sure there are actual IT people and such out there with more elegant solutions.... cron jobs backing up to ftp sites and the like, but for my simple needs and with my limited knowledge setting this was quite an accomplishment.... and it has been working fine ever since.

[mimosa]

@ Shador

I was thinking of directories not files, but you are still right. In order to get the behaviour produced in rsync by a trailing slash, for both mv and cp, you have to do

Code: Select all

cp -r foo/* bar

and rm behaves in the same way. However mv is the odd one out in that it doesn't need the -r switch.