Re: [Bacula-users] File jobs purged when they should not

Please respond to the bottom of the message, or inline.  This makes it easier 
to follow the discussion.

On Nov 20, 2012, at 2:06 PM, Felip Moll wrote:

> 2012/11/20 Melissari, Ryan Joseph <rmelissari AT tamu DOT edu>
> I didn't see this in your configuration you sent to the list, but I saw you 
> mention a question about multiple pools.  I also saw you have retention 
> settings defined in both client and pool resources.  Volume retention is the 
> only one out of the three that is a pool attribute. Job and client retention 
> should be placed in the client resource.  Since job and client retention are 
> client resources, when a client resource is called (assuming autoprune is 
> enabled) it will go and prune any job on all of its pools.
> 
> You can define these values in the pool resource if you want, but just 
> remember that they apply to the client and not the pool itself.  The only 
> safe way I know of to backup clients to multiple pools with separate 
> retention times are to setup multiple client resources for each client.
> 
> 
> ----- Original Message -----
> > From: "Felip Moll" <lipixx AT gmail DOT com>
> > To: "lst hoe02" <lst_hoe02 AT kwsoft DOT de>
> > Cc: bacula-users AT lists.sourceforge DOT net
> > Sent: Tuesday, November 20, 2012 10:54:11 AM
> > Subject: Re: [Bacula-users] File jobs purged when they should not
> >
> >
> > This is correct. I took care of the priority for purging.
> >
> > By the way, as you noticed, I must be able to restore individually
> > the files of the backup done at 01-nov 22.00h, from 01 nov to 01
> > dec.
> >
> > On 06 november, the files of the backup were purged.. so NO 1 month
> > restore.. and was a full backup.
> >
> >
> >
> >
> > 2012/11/20 < lst_hoe02 AT kwsoft DOT de >
> >
> >
> >
> > Zitat von Felip Moll < lipixx AT gmail DOT com >:
> >
> >
> > > Hi people,
> > >
> > > Recently I found a problem when trying to recover my files from a
> > > Job. The
> > > Files of a Job are purged when they shouldn't be.
> > >
> >
> >
> > > #######################
> > > ####################### RELEVANT CONFIGURATIONS
> > > #######################
> > >
> > > Job {
> > > Name = "Backup Dades : Dominator 6m"
> > > Client = dominator-fd
> > > JobDefs = "JobTemplate"
> > > Fileset = "home"
> > > Schedule = "CicleMensual-6mesos-22h"
> > > Storage = Autochanger
> > > Pool = Cintes-Data-6mesos
> > > }
> > >
> > > Schedule {
> > > Name = "CicleMensual-6mesos-22h"
> > > Run = Full january 1 at 22:00
> > > Run = Full july 1 at 22:00
> > > Run = Incremental feb-jun 1 at 22:00
> > > Run = Incremental aug-dec 1 at 22:00
> > > }
> > >
> > > Client {
> > > Name = dominator-fd
> > > Address = dominator.domain.com
> > > FDPort = 9102
> > > Catalog = MyCatalog
> > > Password = "****" # password for FileDaemon
> > > File Retention = 10 years # 10 years
> > > Job Retention = 10 years # six months
> > > AutoPrune = yes # Prune expired Jobs/Files
> > > }
> > >
> > >
> > >
> > > Pool {
> > > Name = Cintes-Data-6mesos
> > > Pool Type = Backup
> > > Recycle = yes
> > > AutoPrune = yes
> > > File Retention = 1 month
> > > Job Retention = 6 months
> > > Volume Retention = 12 months
> > > }
> > >
> > > #######################
> > > #######################
> > > #######################
> > >
> > > #######################
> > > ####################### What happens when I try to restore
> > > #######################
> > > Enter JobId(s), comma separated, to restore: 7444
> > > You have selected the following JobId: 7444
> > >
> > > Building directory tree for JobId(s) 7444 ...
> > >
> > > For one or more of the JobIds selected, no files were found,
> > > so file selection is not possible.
> > > Most likely your retention policy pruned the files.
> > >
> > > Do you want to restore all the files? (yes|no): no
> > >
> >
> > You should be aware of the "priority" for purging. From my knowledge
> > the pool purging has priority against the setting for the client.
> > Furthermore the purge is inclusive as purging jobs do purge all
> > related file records and so on. For your example you have a "File
> > Retention = 1 month" and a "Job Retention = 6 months" which means
> > after 1 month you can only restore the full job without file
> > selection
> > (that's what bacula suggested).

> So, my problem is that the retention period that I defined in the pools are 
> applied to the client regardless of the pool where the backup job is done?

No.

First, if you know this, sorry, but:

Retention refers to the Catalog, Not to backups.  Keep that in mind when 
reading the following paragraphs.

File, Job, and Volume retention each refer to meta data, not backups.  These 
values refer to how long database records in the Catalog should be retained.

Here's what I wrote about this a while back 
(http://dan.langille.org/2010/02/08/bacula-retention-periods/):

"The result of this means you can’t restore a given file after the File 
retention passed. If the Job retention has not passed, you can still restore 
the whole job."

Similarly, after Job retention has passed, you cannot restore an entire job.

In both cases, the backup persists.  You just cannot easily access using the 
Catalog.  You'd have to use tools such as bls or bextract.

This situation highlights both the strength and the weakness of the catalog.  
Consider how much work is is doing for you during a restore.

Also note, that the lesser of all the retention periods applies when it comes 
to recycling.

>>From 
>>http://www.bacula.org/5.2.x-manuals/en/main/main/Configuring_Director.html#SECTION0022130000000000000000
>> :

Under 'The Client Resource', File Retention = time-period-specification

"File records may actually be retained for a shorter period than you specify on 
this directive if you specify either a shorter Job Retention or a shorter 
Volume Retention period. The shortest retention period of the three takes 
precedence."

Please read through all that again, and make sure.  I hope I cannot be 
misunderstood.  :)

> So, probably, the pool defined for a 7 days retention is pruning all files 
> from the backups defined to run in a different pool, eg. 4 weeks, 1 month, 5 
> years pools... "messing up" these jobs.

Yes.  It sounds like Bacula is doing as instructed.  Or more precisely, as 
permitted by the specified Retention periods.

> I have some clients with TiB of data, and I would like to have a list of 
> files to recover in a 7 days period, and every 7 days prune oldest jobs and 
> files. I want also that this clients backup their data to the pool of 1 month 
> and 5y, but not saving files... so from what you said, i must define multiple 
> clients for every "real client".

Multiple clients are not required.

I was unable to follow your requirements.  Perhaps if you express them in terms 
of:

* I want to be able to restore an individual file that was backed up X days ago.
* I want to keep all full backups for X day.

Keep in mind: database records are cheap.  A few GB of disk space is a cheap 
price to pay to be able to restore any file backed up in the past 3 years.

> Is it correct?

Now that we know the problem, I think we can move on to defining your retention 
periods based on the sample statements I suggested above.

:)


-- 
Dan Langille - http://langille.org


------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users