Front page

Node cannot be found | Couldn't create lock

a75546e18f2b44d395315ca7c2c10bb9
REPAY EQUIPMENT CUBIC

From: "r.c.r" <rcr@gmx.ch>
Date: Fri, 6 Jan 2017 01:00:08 +0100

   Hi there!
   
   Recently I have started to use obnam to do my backups.
   I am doing the backups from 3 clients to one repository. 2 repos 
   have roughly 1 TB of data, the third one just over 100 MB, but most 
   of the data is twice, so the final backup directory is not much over 
   1 TB.
   I backed up the clients one by one manually first, and then 
   continued manual backups for a few days before setting up 
   cron/anacron routines. Few days ago, the backups started to fail.
   
   First, there seemed to be a problem with the locking/unlocking of 
   the clients:
   2017-01-02 09:22:37 INFO Unlocking client [client1]
   2017-01-02 09:22:45 INFO Locking client [client1]
   2017-01-02 09:22:45 INFO Locking client [client2]
   2017-01-02 09:22:45 INFO Locking client [client*]
   2017-01-02 09:23:45 INFO Unlocking client [client1]
   2017-01-02 09:23:45 INFO Unlocking client [client2]
   2017-01-02 09:23:45 INFO Successfully unlocked
   2017-01-02 09:23:45 CRITICAL R681AEX: Couldn't create lock 
   4417895082264505752/lock: timeout
   
   Initially, I could fix that with force-lock.
   However, now it fails due to a missing chunk, when I do a backup of 
   any of the clients:
   
   on client 3:
   obnam backup --config /home/xxxx/.config/obnam/data.conf
   00h05m42s 6478 files 17.10 GiB scanned: making checkpoint: adding 
   chunks to shared B-treesERROR: Node 0x2fa8b cannot be found in the 
   node store chunksums: 2: No such file: None
   
   on client 1
   obnam backup --config /home/rolf/.config/obnam/obnam-data.conf
   00h02m27s 73462 files 114.95 GiB scanned: committing changes to 
   repository: adding chunks to shared B-treesERROR: Node 0x2fa8f 
   cannot be found in the node store chunksums: 2: No such file: None
   obnam force-lock --config /home/rolf/.config/obnam/obnam-data.conf
   obnam backup --config /home/rolf/.config/obnam/obnam-data.conf
   00h02m31s 73450 files 114.95 GiB scanned: committing changes to 
   repository: adding chunks to shared B-treesERROR: Node 0x2fa80 
   cannot be found in the node store chunksums: 2: No such file: None
   
   Running obnam fsck doesn't fix the problem. When running obnam 
   backup again, the problem is still there:
   
   2017-01-06 00:51:10 INFO Attempting to unlock client because of error
   2017-01-06 00:51:10 INFO Unlocking client [client1]
   2017-01-06 00:51:10 INFO Attempting to unlock shared trees because 
   of error
   2017-01-06 00:51:10 INFO Successfully unlocked
   2017-01-06 00:51:10 CRITICAL Node 0x2fa80 cannot be found in the 
   node store chunksums: 2: No such file: None
   Traceback (most recent call last):
      File "/usr/lib/python2.7/dist-packages/obnamlib/app.py", line 
   207, in process_args
        cliapp.Application.process_args(self, args)
      File "/usr/lib/python2.7/dist-packages/cliapp/app.py", line 570, 
   in process_args
        method(args[1:])
      File 
   "/usr/lib/python2.7/dist-packages/obnamlib/plugins/backup_plugin.py", 
   line 169, in backup
        self.finish_generation()
      File 
   "/usr/lib/python2.7/dist-packages/obnamlib/plugins/backup_plugin.py", 
   line 251, in finish_generation
        self.add_chunks_to_shared()
      File 
   "/usr/lib/python2.7/dist-packages/obnamlib/plugins/backup_plugin.py", 
   line 344, in add_chunks_to_shared
        self.repo.put_chunk_into_indexes(chunkid, token, self.client_name)
      File 
   "/usr/lib/python2.7/dist-packages/obnamlib/fmt_6/repo_fmt_6.py", 
   line 785, in put_chunk_into_indexes
        self._chunksums.add(token, chunk_id, client_id)
      File 
   "/usr/lib/python2.7/dist-packages/obnamlib/fmt_6/checksumtree.py", 
   line 57, in add
        self.tree.insert(key, '')
      File "/usr/lib/python2.7/dist-packages/larch/tree.py", line 281, 
   in insert
        kids = self._insert_into_index(self.root, key, value)
      File "/usr/lib/python2.7/dist-packages/larch/tree.py", line 317, 
   in _insert_into_index
        child = self._get_node(new_index[child_key])
      File "/usr/lib/python2.7/dist-packages/larch/tree.py", line 111, 
   in _get_node
        return self.node_store.get_node(node_id)
      File "/usr/lib/python2.7/dist-packages/larch/nodestore_disk.py", 
   line 240, in get_node
        raise larch.NodeMissing(self.dirname, node_id, error=e)
   NodeMissing: Node 0x2fa80 cannot be found in the node store 
   chunksums: 2: No such file: None
   
   I suspect, there must be some corruption in the repository that 
   cause some files or informations go missing.
   
   Can you advise me how to fix/repair and prevent this issue?
   Thanks!
   
   _______________________________________________
   obnam-support mailing list
   obnam-support@obnam.org
   http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-support-obnam.org
From: Lars Wirzenius <liw@liw.fi>
Date: Fri, 6 Jan 2017 10:33:08 +0200

   On Fri, Jan 06, 2017 at 01:00:08AM +0100, r.c.r wrote:
   > Can you advise me how to fix/repair and prevent this issue?
   
   What version of Obnam are you using? Are you using local disk or an
   sftp server for the repository? What version of the paramiko Python
   library are you using? What sftp server are you using?
From: "r.c.r" <rcr@gmx.ch>
Date: Mon, 9 Jan 2017 20:43:37 +0100

   Dear Lars
   Sorry about dropping the support-list...
   
   I briefly clarify my set-up:
   On 2 clients I use(d) debian jessie with obnam from backports (v. 1.17).
   The 3rd client runs linuxmint 17.2 with obnam 1.6.
   The backup goes to a simple server on the local network. The server 
   is based on a raspberry 3, running headless rasbian (jessie). It 
   also runs obnam v. 1.17 (though I guess this should be irrelevant 
   for the issue, as I only acces the repository from the clients).
   The clients 1 & 2 run Python 2.7.9, so does the raspbian server. 
   Client 3 runs python 2.7.6
   
   Follwing your advice, I have attempted to update the three clients, 
   using the latest deb package from code.liw.fi.
   The 2 clients running debian jessie I succeeded in installing the 
   latest version.
   The client running linuxmint failed to install the package, because 
   python-tracing is pre . I tried tried to install all the other obnam 
   deb packages back to 1.17, but no luck.
   
   After upgrading the 2 clients to obnam 1.21, I run obnam backup 
   again, but the original error persisted.
   I also tried obnam fsck and obnam fsck-fix, but both commands 
   ultimately failed due to node not found. However, obnam fsck only 
   failed from one client only after almost 15 hours of checking.
   
   Do you have to use the same version of obnam for each client if you 
   share a repository?
   
   
   
   
   
   On 06/01/17 09:33, Lars Wirzenius wrote:
   > On Fri, Jan 06, 2017 at 01:00:08AM +0100, r.c.r wrote:
   >> Can you advise me how to fix/repair and prevent this issue?
   > What version of Obnam are you using? Are you using local disk or an
   > sftp server for the repository? What version of the paramiko Python
   > library are you using? What sftp server are you using?
   >
   >
   
   
   _______________________________________________
   obnam-support mailing list
   obnam-support@obnam.org
   http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-support-obnam.org
From: Lars Wirzenius <liw@liw.fi>
Date: Sat, 21 Jan 2017 16:52:25 +0200

   I'm afraid my suspicion is that you ran into a bug in Obnam prior to
   1.6.1 that results in nodes going missing from the backup repository.
   This FAQ entry will hopefully be of use, though admittedly not of much
   consolation:
   
   http://obnam.org/faq/missing-node/
   
   Sorry about the inconvenience.
   
   On Mon, Jan 09, 2017 at 08:43:37PM +0100, r.c.r wrote:
   > Dear Lars
   > Sorry about dropping the support-list...
   > 
   > I briefly clarify my set-up:
   > On 2 clients I use(d) debian jessie with obnam from backports (v. 1.17).
   > The 3rd client runs linuxmint 17.2 with obnam 1.6.
   > The backup goes to a simple server on the local network. The server is based
   > on a raspberry 3, running headless rasbian (jessie). It also runs obnam v.
   > 1.17 (though I guess this should be irrelevant for the issue, as I only
   > acces the repository from the clients).
   > The clients 1 & 2 run Python 2.7.9, so does the raspbian server. Client 3
   > runs python 2.7.6
   > 
   > Follwing your advice, I have attempted to update the three clients, using
   > the latest deb package from code.liw.fi.
   > The 2 clients running debian jessie I succeeded in installing the latest
   > version.
   > The client running linuxmint failed to install the package, because
   > python-tracing is pre . I tried tried to install all the other obnam deb
   > packages back to 1.17, but no luck.
   > 
   > After upgrading the 2 clients to obnam 1.21, I run obnam backup again, but
   > the original error persisted.
   > I also tried obnam fsck and obnam fsck-fix, but both commands ultimately
   > failed due to node not found. However, obnam fsck only failed from one
   > client only after almost 15 hours of checking.
   > 
   > Do you have to use the same version of obnam for each client if you share a
   > repository?
   > 
   > 
   > 
   > 
   > 
   > On 06/01/17 09:33, Lars Wirzenius wrote:
   > > On Fri, Jan 06, 2017 at 01:00:08AM +0100, r.c.r wrote:
   > > > Can you advise me how to fix/repair and prevent this issue?
   > > What version of Obnam are you using? Are you using local disk or an
   > > sftp server for the repository? What version of the paramiko Python
   > > library are you using? What sftp server are you using?
   > > 
   > > 
   > 
   > 
   > _______________________________________________
   > obnam-support mailing list
   > obnam-support@obnam.org
   > http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-support-obnam.org
   >
From: R C Rupp <rcr@gmx.ch>
Date: Thu, 2 Feb 2017 10:00:09 +0100

   Ok, I can live with that.
   
   I cannot update the obnam package on the linuxmint machine at all, as 
   the required dependencies of python are outdated. A reason more to 
   update the the whole machine... :)
   
   I run the backup for now on separate repositories for each client, so 
   the backup with obnam 1.6 will not break all the backups it it fails 
   again...
   
   Thanks for all the help!
   
   
   
   On 21/01/17 15:52, Lars Wirzenius wrote:
   > I'm afraid my suspicion is that you ran into a bug in Obnam prior to
   > 1.6.1 that results in nodes going missing from the backup repository.
   > This FAQ entry will hopefully be of use, though admittedly not of much
   > consolation:
   >
   > http://obnam.org/faq/missing-node/
   >
   > Sorry about the inconvenience.
   >
   > On Mon, Jan 09, 2017 at 08:43:37PM +0100, r.c.r wrote:
   >> Dear Lars
   >> Sorry about dropping the support-list...
   >>
   >> I briefly clarify my set-up:
   >> On 2 clients I use(d) debian jessie with obnam from backports (v. 1.17).
   >> The 3rd client runs linuxmint 17.2 with obnam 1.6.
   >> The backup goes to a simple server on the local network. The server is based
   >> on a raspberry 3, running headless rasbian (jessie). It also runs obnam v.
   >> 1.17 (though I guess this should be irrelevant for the issue, as I only
   >> acces the repository from the clients).
   >> The clients 1 & 2 run Python 2.7.9, so does the raspbian server. Client 3
   >> runs python 2.7.6
   >>
   >> Follwing your advice, I have attempted to update the three clients, using
   >> the latest deb package from code.liw.fi.
   >> The 2 clients running debian jessie I succeeded in installing the latest
   >> version.
   >> The client running linuxmint failed to install the package, because
   >> python-tracing is pre . I tried tried to install all the other obnam deb
   >> packages back to 1.17, but no luck.
   >>
   >> After upgrading the 2 clients to obnam 1.21, I run obnam backup again, but
   >> the original error persisted.
   >> I also tried obnam fsck and obnam fsck-fix, but both commands ultimately
   >> failed due to node not found. However, obnam fsck only failed from one
   >> client only after almost 15 hours of checking.
   >>
   >> Do you have to use the same version of obnam for each client if you share a
   >> repository?
   >>
   >>
   >>
   >>
   >>
   >> On 06/01/17 09:33, Lars Wirzenius wrote:
   >>> On Fri, Jan 06, 2017 at 01:00:08AM +0100, r.c.r wrote:
   >>>> Can you advise me how to fix/repair and prevent this issue?
   >>> What version of Obnam are you using? Are you using local disk or an
   >>> sftp server for the repository? What version of the paramiko Python
   >>> library are you using? What sftp server are you using?
   >>>
   >>>
   >>
   >> _______________________________________________
   >> obnam-support mailing list
   >> obnam-support@obnam.org
   >> http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-support-obnam.org
   >>
   
   
   _______________________________________________
   obnam-support mailing list
   obnam-support@obnam.org
   http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-support-obnam.org