Hi,
Les Mikesell wrote on 2014-06-26 10:47:19 -0500 [Re: [BackupPC-users]
Uneffective rsync transfers?]:
> On Thu, Jun 26, 2014 at 7:40 AM, Franti??ek Starý <frantisek AT stary DOT
> net> wrote:
> >
> > I've started experimenting with rsync xfer method and I've found that the
> > rsync transfers always the changes against the last backup.
>
> You don't need to experiment to find that - see $Conf{IncrLevels} in
> the documentation.
and it's not "the last backup", it's "the reference backup". Of course, if
you're only doing full backups, that's the same thing.
> > Second full backup (when 0 files changed):
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > full backup started for directory /testdir/ (baseline backup #0)
> > [...]
> > Got fatal error during xfer (No files dumped for share /testdir/)
> > Backup aborted (No files dumped for share /testdir/)
> > Not saving this as a partial backup since it has fewer files than the prior
> > one (got 0 and 0 files versus 0)
> >
> > Shouldn't be this considered as "non error" status? It means no files were
> > changed.
>
> That's optional, See $Conf{BackupZeroFilesIsFatal}
No, I believe the objection is correct. BackupZeroFilesIsFatal is for a share
with 0 files, not 0 *changed* files. If there's something there, but it
hasn't changed since the last backup, that is not an error. Apparently, rsync
(or rather the rsync XferMethod) is complaining about not having any files in
the backup - possibly due to failing hard linking.
> > Second full backup (when 1 file added):
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > [...]
> > Done: 1 files, 1048576 bytes.
> >
> > It seems to be okay. Only the new file was transferred. But when I look in
> > the web gui at this full backup, I see only this file. The other files
> > (testfile1,2,3) are not here. Shouldn't be the full backup really full
> > (merged somehow with the previous full backup)?
>
> Full
rsync
> backups should link unchanged files from the previous full into
> the current tree.
Other XferMethods will "coincidentally" create a link to the same pool file
because the content matches (hard link count permitting). They are, however,
unaware that the file is unchanged. Only rsync creates "same" log entries -
other XferMethods only have "pool". You should check your XferLogs for this.
It would seem they might not contain "same" entries, although they really
should.
> Is your xfer log full of 'can't link' errors?
Since the quote was from the xfer log, apparently it isn't. And we're talking
about a different piece of code than usual - it would seem to be the
XferMethod that is failing to create the hard links, not BackupPC_link (though
it's hard to imagine *how* it would fail in such a way).
> > Fourth full backup (when 0 files changed after the previous backup):
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > [...]
> > And again. Only the files that are not present in the previous backup are
> > present.
> >
> > Is this a normal behavior?
>
> No, you have some problem with the filesystem. Does it support hard links?
In particular, does Perl link() report success although the file system is not
creating hard links?
Honestly, if there were a bug in BackupPC that would re-transfer unchanged
data every second backup, someone would have noticed ;-). There are people
here doing large backups over slow links who depend on the bandwidth savings
from rsync transfer.
The analysis that hard links may be failing fits what you are seeing very
well, though from the code I can't see how BackupPC
a) would not notice,
b) still complain about a zero file backup.
Linking would need to *appear* to be working for BackupPC to even start.
Of course, it's hard to analyse the code when you don't know which code to
look at. Which version of BackupPC are you using on which distribution (or
rather: where did you get it from)? What type of file system is your pool on,
and how do you access it?
A different possibility would be that the code for building a view of the last
backup is not working correctly for you, though I can't even begin to imagine
why it wouldn't.
> > Is there a way to configure backuppc to transfer
> > only changed files and merge them with the previous backup? I want to backup
> > 200GB of email files.
>
> Yes, that is the way it is supposed to work. If the files are mbox
> format (many messages in one file) it can be rather slow to merge the
> changes but it does work. Maildir format works more efficiently.
The important thing to note here is that with large mbox files, you'll have an
individual copy after every change and not much savings from BackupPC pooling.
Maildir on the other hand fits in with BackupPC pooling perfectly.
Regards,
Holger
------------------------------------------------------------------------------
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
|
