Front page

Does obnam check the physical existence of chunks on backup?

c12cfa16730b49c5b3cc11ead075a0e1
SNAPLINE CHICAGO WALLET

From: Dennis Jacobfeuerborn <dennisml@conversis.de>
Date: Wed, 24 Feb 2016 22:49:37 +0100

   Hi,
   since I'm still struggling with the dreaded "R43272X: Repository doesn't
   contain chunk" error I finally formed a hypothesis why this error is so
   persistent.
   Is it possible that during a backup when obnam checks if a chunk already
   exists it only checks the index and not the physical presence of the
   chunk on disk?
   If that were the case then a missing chunk would not only mean that the
   first generation referencing that chunk would miss data but all future
   generations would miss that data as well unless the file on the client
   changes.
   
   Regards,
     Dennis
   
   _______________________________________________
   obnam-support mailing list
   obnam-support@obnam.org
   http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-support-obnam.org
From: Dennis Jacobfeuerborn <dennisml@conversis.de>
Date: Wed, 24 Feb 2016 22:49:37 +0100

   Hi,
   since I'm still struggling with the dreaded "R43272X: Repository doesn't
   contain chunk" error I finally formed a hypothesis why this error is so
   persistent.
   Is it possible that during a backup when obnam checks if a chunk already
   exists it only checks the index and not the physical presence of the
   chunk on disk?
   If that were the case then a missing chunk would not only mean that the
   first generation referencing that chunk would miss data but all future
   generations would miss that data as well unless the file on the client
   changes.
   
   Regards,
     Dennis
   
   _______________________________________________
   obnam-support mailing list
   obnam-support@obnam.org
   http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-support-obnam.org
From: Lars Wirzenius <liw@liw.fi>
Date: Thu, 25 Feb 2016 08:09:38 +0200

   On Wed, Feb 24, 2016 at 10:49:37PM +0100, Dennis Jacobfeuerborn wrote:
   > Is it possible that during a backup when obnam checks if a chunk already
   > exists it only checks the index and not the physical presence of the
   > chunk on disk?
   > If that were the case then a missing chunk would not only mean that the
   > first generation referencing that chunk would miss data but all future
   > generations would miss that data as well unless the file on the client
   > changes.
   
   That is exactly what happens, yes.
   
   Obnam could check, during backup, that the actual chunk exists, but
   that would add at least one extra round trip to the repository, and
   that'd make backups over high-latency links much slower.
   
   BTW, I would love it if someone could change obnam fsck to remove
   references to missing chunks.
From: Dennis Jacobfeuerborn <dennisml@conversis.de>
Date: Fri, 26 Feb 2016 04:54:26 +0100

   On 25.02.2016 07:09, Lars Wirzenius wrote:
   > On Wed, Feb 24, 2016 at 10:49:37PM +0100, Dennis Jacobfeuerborn wrote:
   >> Is it possible that during a backup when obnam checks if a chunk already
   >> exists it only checks the index and not the physical presence of the
   >> chunk on disk?
   >> If that were the case then a missing chunk would not only mean that the
   >> first generation referencing that chunk would miss data but all future
   >> generations would miss that data as well unless the file on the client
   >> changes.
   > 
   > That is exactly what happens, yes.
   > 
   > Obnam could check, during backup, that the actual chunk exists, but
   > that would add at least one extra round trip to the repository, and
   > that'd make backups over high-latency links much slower.
   
   In my case I always use pull backups and the repository is local so the
   performance impact should be much smaller. The worst case scenario in
   this case is lots of small files since this would still mean a lot of
   additional IOPS for the device. In the regular case the number of
   look-ups on disk shouldn't slow things down to much though I think.
   
   I'm thinking about implementing an option that allows the user to chose
   to either simply check for the existence of the chunk on diks or also
   verify that the checksum of the chunk on disk actually is correct which
   would be the slowest but also the most secure option.
   
   > BTW, I would love it if someone could change obnam fsck to remove
   > references to missing chunks.
   
   I'm going to look into that over the weekend. In the repo.has_chunk()
   function there is a test _is_in_tree_chunk_id(chunk_id) and I'm not sure
   I understand what it means for a chunk to be "in tree". When I run an
   fsck operation this function seems to always evaluate to false for all
   chunks. In what case would this function actually return true?
   
   Regards,
     Dennis
   
   
   _______________________________________________
   obnam-support mailing list
   obnam-support@obnam.org
   http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-support-obnam.org
From: Roland Mas <lolando@debian.org>
Date: Sat, 27 Feb 2016 00:07:20 +0100

   Dennis Jacobfeuerborn, 2016-02-26 04:54:26 +0100 :
   
   [...]
   
   > I'm thinking about implementing an option that allows the user to chose
   > to either simply check for the existence of the chunk on diks or also
   > verify that the checksum of the chunk on disk actually is correct which
   > would be the slowest but also the most secure option.
   
   When BackupPC backups using the rsync protocol, there's a tweak that
   allows not to redownload the whole file if the remotely computed
   checksum matches the local one.  It comes with a setting that specifies
   a probability of a recheck.  Maybe that would be better than a simple
   boolean setting?
   
   Roland.
From: Dennis Jacobfeuerborn <dennisml@conversis.de>
Date: Sat, 27 Feb 2016 05:00:29 +0100

   On 25.02.2016 07:09, Lars Wirzenius wrote:
   > On Wed, Feb 24, 2016 at 10:49:37PM +0100, Dennis Jacobfeuerborn wrote:
   >> Is it possible that during a backup when obnam checks if a chunk already
   >> exists it only checks the index and not the physical presence of the
   >> chunk on disk?
   >> If that were the case then a missing chunk would not only mean that the
   >> first generation referencing that chunk would miss data but all future
   >> generations would miss that data as well unless the file on the client
   >> changes.
   > 
   > That is exactly what happens, yes.
   > 
   > Obnam could check, during backup, that the actual chunk exists, but
   > that would add at least one extra round trip to the repository, and
   > that'd make backups over high-latency links much slower.
   > 
   > BTW, I would love it if someone could change obnam fsck to remove
   > references to missing chunks.
   
   So I've looked into this an while removing the missing chunk from the
   index is easy enough it doesn't seem to make much of a difference.
   Since on the next backup the metadata of the file still matches the it
   isn't even considered to be backed up so all future backups will still
   have that file with a missing chunk.
   
   There are two strategies the I see here right now:
   
   1) Don't just remove the chunk from the index but also find all the
   references to that chunk in all generations and clients and remove the
   files from these generations. The problem is that this is potentially
   time consuming and this would delete more data than necessary after all
   being able to restore 99% of the file might still be better than not
   being able to restore the file at all.
   
   2) The other way would be to create a field in the metadata
   "force_backup" that defaults to 0 but can be set to 1 by fsck. If the
   backup process tries to compare the metadata it will not match and the
   file will be backed up again.
   
   I haven't looked too deeply yet into how the chunks, chunk id's,
   checksums, etc. reference each other but potentially it could be
   possible to heal previous generations by re-creating the chunk with the
   same checksum (provided the content of the file hasn't changed) by
   performing the chunk checking i mentioned in my previous mail but only
   for the force_backup=1 case and regenerating the chunk that is missing.
   
   Regards,
     Dennis
   
   
   
   _______________________________________________
   obnam-support mailing list
   obnam-support@obnam.org
   http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-support-obnam.org
From: Lars Wirzenius <liw@liw.fi>
Date: Sat, 2 Apr 2016 18:56:46 +0300

   On Fri, Feb 26, 2016 at 04:54:26AM +0100, Dennis Jacobfeuerborn wrote:
   > I'm going to look into that over the weekend. In the repo.has_chunk()
   > function there is a test _is_in_tree_chunk_id(chunk_id) and I'm not sure
   > I understand what it means for a chunk to be "in tree". When I run an
   > fsck operation this function seems to always evaluate to false for all
   > chunks. In what case would this function actually return true?
   
   The in-tree chunks were a failed attempt at optimising Obnam by having
   it store small chunks directly in the B-trees. It turned out to not
   work well.
   
   You can ignore that now, I think. Obnam hasn't been creating in-tree
   chunks for many, many years.
From: Lars Wirzenius <liw@liw.fi>
Date: Sat, 2 Apr 2016 19:03:25 +0300

   On Sat, Feb 27, 2016 at 05:00:29AM +0100, Dennis Jacobfeuerborn wrote:
   > On 25.02.2016 07:09, Lars Wirzenius wrote:
   > > On Wed, Feb 24, 2016 at 10:49:37PM +0100, Dennis Jacobfeuerborn wrote:
   > >> Is it possible that during a backup when obnam checks if a chunk already
   > >> exists it only checks the index and not the physical presence of the
   > >> chunk on disk?
   > >> If that were the case then a missing chunk would not only mean that the
   > >> first generation referencing that chunk would miss data but all future
   > >> generations would miss that data as well unless the file on the client
   > >> changes.
   > > 
   > > That is exactly what happens, yes.
   > > 
   > > Obnam could check, during backup, that the actual chunk exists, but
   > > that would add at least one extra round trip to the repository, and
   > > that'd make backups over high-latency links much slower.
   > > 
   > > BTW, I would love it if someone could change obnam fsck to remove
   > > references to missing chunks.
   > 
   
   > So I've looked into this an while removing the missing chunk from the
   > index is easy enough it doesn't seem to make much of a difference.
   > Since on the next backup the metadata of the file still matches the it
   > isn't even considered to be backed up so all future backups will still
   > have that file with a missing chunk.
   
   This is a good point.
   
   > There are two strategies the I see here right now:
   
   The first strategy (remove all files in all generations that refer to
   the missing chunk) is clearly bad. The second strategy as such is not
   going to work, as it would change the on-disk repository format, and
   instantly make older versions of Obnam incompatible with them, and
   require everyone to upgrade their Obnam and possibly re-backup
   everything.
   
   However, a variant on that which should work would be to change a
   suitable metadata field to an "impossible" value. For example, set
   file size to 0 and the list of chunks to empty. Live data either has a
   zero size as well (so it's correct to not re-backup) or it's not an
   empty file, resulting in a backup.
   
   Would that work for you?
   
   > I haven't looked too deeply yet into how the chunks, chunk id's,
   > checksums, etc. reference each other but potentially it could be
   > possible to heal previous generations by re-creating the chunk with the
   > same checksum (provided the content of the file hasn't changed) by
   > performing the chunk checking i mentioned in my previous mail but only
   > for the force_backup=1 case and regenerating the chunk that is missing.
   
   Re-creating a chunk by finding the same data again would be lovely,
   yes. However, it's going to be a bit of work, I suspect.