Front page

Kdirstat merged to Qdirstat

74ddc3e0bee44cc49b27368cda74de23
INDOORS TAMBOURINE SNOWCAP

From: SanskritFritz <sanskritfritz@gmail.com>
Date: Thu, 26 Jan 2017 22:42:44 +0100

   Hi Ian.
   I'm writing this to you because the command kdirstat captured my
   imagination. However when I was about to try this, I learned that
   Kdirstat is deprecated, but instead the same author wrote Qdirstat. I
   installed it, but sadly this new program cannot interpret Obnam's
   kdirstat file. Could you please look into this? I find the idea of
   being able to inspect all files with Qdirstat amazing.
   Thanks
   SanskritFritz
   
   _______________________________________________
   obnam-dev mailing list
   obnam-dev@obnam.org
   http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org
From: Ian Campbell <ijc@hellion.org.uk>
Date: Fri, 27 Jan 2017 12:06:27 +0000

   On Thu, 2017-01-26 at 22:42 +0100, SanskritFritz wrote:
   > Hi Ian.
   > I'm writing this to you because the command kdirstat captured my
   > imagination. However when I was about to try this, I learned that
   > Kdirstat is deprecated, but instead the same author wrote Qdirstat. I
   > installed it, but sadly this new program cannot interpret Obnam's
   > kdirstat file. Could you please look into this? I find the idea of
   > being able to inspect all files with Qdirstat amazing.
   > Thanks
   > 
   
   I'm afraid I'm still using kdirstat (actually k4dirstat) and hadn't
   heard of qdirstat until today. I've no time to work on this in the
   short term, so please feel free to take it on.
   
   Ian.
   
   _______________________________________________
   obnam-dev mailing list
   obnam-dev@obnam.org
   http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org
From: SanskritFritz <sanskritfritz@gmail.com>
Date: Fri, 27 Jan 2017 15:00:49 +0100

   On Fri, Jan 27, 2017 at 1:06 PM, Ian Campbell <ijc@hellion.org.uk> wrote:
   > I'm afraid I'm still using kdirstat (actually k4dirstat) and hadn't
   > heard of qdirstat until today. I've no time to work on this in the
   > short term, so please feel free to take it on.
   
   I see thanks. Fair enough, I'll see what I can do.
   
   _______________________________________________
   obnam-dev mailing list
   obnam-dev@obnam.org
   http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org
From: SanskritFritz <sanskritfritz@gmail.com>
Date: Fri, 27 Jan 2017 20:58:50 +0100

   On Fri, Jan 27, 2017 at 3:00 PM, SanskritFritz <sanskritfritz@gmail.com> wrote:
   > On Fri, Jan 27, 2017 at 1:06 PM, Ian Campbell <ijc@hellion.org.uk> wrote:
   >> I'm afraid I'm still using kdirstat (actually k4dirstat) and hadn't
   >> heard of qdirstat until today. I've no time to work on this in the
   >> short term, so please feel free to take it on.
   >
   > I see thanks. Fair enough, I'll see what I can do.
   
   Ok, I have to write again because I suspect the problem I found is
   also relevant with k4dirstat and others. I found that qdirstat is able
   to read the output provided by obnam if the backup root is not root
   (/). My backup's root directory was exactly that "/". Qdirstat logfile
   revealed that the parent directory was missing, namely "/". When I
   tried another repository where the root was /Common/Pictures, I was
   able to present the results in qdirstat. So, maybe this problem is
   present in k4dirstat too. The solution is simple: obnam kdirstat
   should write the root directory into the first line like this:
   
   # Generated by obnam Generation 14997 (2016-11-14 23:15:04 +0100 -
   2016-11-14 23:21:33 +0100)
   
   # Do not edit!
   #
   # Type  path            size    mtime           <optional fields>
   
   D /    4K    0x5852ecf9
   L    /bin    7    0x560c3562
   ...
   
   After editing the obnam generated file by inserting the root dir,
   qdirstat happily presented my backup.
   
   Can you test this with k4dirstat?
   I hope to fix this some day and present a patch too, but my python-fu
   is quite weak yet.
   
   _______________________________________________
   obnam-dev mailing list
   obnam-dev@obnam.org
   http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org
From: SanskritFritz <sanskritfritz@gmail.com>
Date: Sat, 28 Jan 2017 01:10:09 +0100

   On Fri, Jan 27, 2017 at 8:58 PM, SanskritFritz <sanskritfritz@gmail.com> wrote:
   > I hope to fix this some day and present a patch too, but my python-fu
   > is quite weak yet.
   
   OK, I think I found the cause and hereby present patch to you. I have
   tested this with qdirstat and the result is good.
   
   diff --git a/obnamlib/plugins/show_plugin.py b/obnamlib/plugins/show_plugin.py
   index 995a958c..79ccde10 100644
   --- a/obnamlib/plugins/show_plugin.py
   +++ b/obnamlib/plugins/show_plugin.py
   @@ -314,9 +314,6 @@ class ShowPlugin(obnamlib.ObnamPlugin):
            enc_filename = enc_filename.replace(" ", "%20")
            enc_filename = enc_filename.replace("\t", "%09")
   
   -        if filename == "/":
   -            return
   -
            self.app.output.write(
                "%s%s\t%d\t%#x\n" %
                (mode_str, enc_filename, size, mtime_sec))
   
   _______________________________________________
   obnam-dev mailing list
   obnam-dev@obnam.org
   http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org
From: Ian Campbell <ijc@hellion.org.uk>
Date: Sun, 29 Jan 2017 10:00:52 +0000

   On Sat, 2017-01-28 at 01:10 +0100, SanskritFritz wrote:
   > On Fri, Jan 27, 2017 at 8:58 PM, SanskritFritz <sanskritfritz@gmail.c
   > om> wrote:
   > > I hope to fix this some day and present a patch too, but my python-
   > fu
   > > is quite weak yet.
   > 
   > OK, I think I found the cause and hereby present patch to you. I have
   > tested this with qdirstat and the result is good.
   > 
   > diff --git a/obnamlib/plugins/show_plugin.py
   > b/obnamlib/plugins/show_plugin.py
   > index 995a958c..79ccde10 100644
   > --- a/obnamlib/plugins/show_plugin.py
   > +++ b/obnamlib/plugins/show_plugin.py
   > @@ -314,9 +314,6 @@ class ShowPlugin(obnamlib.ObnamPlugin):
   >          enc_filename = enc_filename.replace(" ", "%20")
   >          enc_filename = enc_filename.replace("\t", "%09")
   > 
   > -        if filename == "/":
   > -            return
   > -
   >          self.app.output.write(
   >              "%s%s\t%d\t%#x\n" %
   >              (mode_str, enc_filename, size, mtime_sec))
   
   
   I monkey patched this into my local install (i.e. I directly edited
   /usr/lib/python2.7/dist-packages/obnamlib/plugins/show_plugin.py, which
    is very naughty of me) and for a non-/ root backup the diff in the
   kdirstat is:
   
   $ diff -u before after 
   --- before	2017-01-29 09:46:23.546378982 +0000
   +++ after	2017-01-29 09:45:44.378570390 +0000
   @@ -5,6 +5,7 @@
    #
    # Type  path            size    mtime           <optional fields>
    
   +D /	4096	0x587c8af7
    D /local	4096	0x50559ae2
    D /local/scratch	4096	0x50559afe
    D /local/scratch/ijc	4096	0x50559b18
   
   Unfortunately loading the "after" version into k4dirstat produces a UI
   which shows only / itself and none of its children :-( Loading the
   "before" version works. (this backup contains nothing of interest, so
   I've attach both files)
   
   I don't have any / rooted backups to try with unfortunately.
   
   Perhaps what is needed instead of the `file == "/"` check is a check
   something like "is the filename under the backup root prefix" (i.e.
   `not filename.startswith(backuproot)`, but I don't know where to get
   the backuproot from).
   
   I think the issues likely correspond to this bit of
   https://github.com/shundhammer/qdirstat/blob/master/doc/cache-file-format.txt
   
       Path or Name
       ------------
               
       Either an absolute path (starting with "/") or only a base name
       relative to the last preceding full path in the file.
   
   and confusion as to what it means to be the basename of "/". Obnam
   always outputs absolute paths though so should be ok. Maybe this is
   something where shundhammer could advise? Given that obnam follows the
   file format spec (I believe) perhaps it is even a *dirstat bug.
   
   Ian.
From: Ian Campbell <ijc@hellion.org.uk>
Date: Sun, 29 Jan 2017 18:19:45 +0000

   On Sun, 2017-01-29 at 18:52 +0100, SanskritFritz wrote:
   > I really recommend uding qdirstat after this :D
   
   Yes, I would switch right away if only it were packaged in Debian (/me
   goes hunting for ITP/RFP bugs...).
   
   Given this I think your patch is indeed the best approach. I'd
   recommend reposting as a standalone thread with a subject "[PATCH]
   Concise explanation of the change" (so Lars sees it) and a commit
   message explaining the circumstances in the body before the patch.
   
   Ian.
   
   _______________________________________________
   obnam-dev mailing list
   obnam-dev@obnam.org
   http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org
From: SanskritFritz <sanskritfritz@gmail.com>
Date: Sun, 29 Jan 2017 18:52:12 +0100

   On Sun, Jan 29, 2017 at 11:00 AM, Ian Campbell <ijc@hellion.org.uk> wrote:
   > Unfortunately loading the "after" version into k4dirstat produces a UI
   > which shows only / itself and none of its children :-( Loading the
   > "before" version works. (this backup contains nothing of interest, so
   > I've attach both files)
   
   Qdirstat performs better, it shows everything as intended, the
   "before" file starts with /local whereas the "after" file starts with
   /, followed by /local as subdir.
   
   > I don't have any / rooted backups to try with unfortunately.
   
   obnam backup --root=/ --repository=obnamroot --exclude='^/[^/]*/.*$'
   
   > Perhaps what is needed instead of the `file == "/"` check is a check
   > something like "is the filename under the backup root prefix" (i.e.
   > `not filename.startswith(backuproot)`, but I don't know where to get
   > the backuproot from).
   >
   > I think the issues likely correspond to this bit of
   > https://github.com/shundhammer/qdirstat/blob/master/doc/cache-file-format.txt
   >
   >     Path or Name
   >     ------------
   >
   >     Either an absolute path (starting with "/") or only a base name
   >     relative to the last preceding full path in the file.
   >
   > and confusion as to what it means to be the basename of "/". Obnam
   > always outputs absolute paths though so should be ok. Maybe this is
   > something where shundhammer could advise? Given that obnam follows the
   > file format spec (I believe) perhaps it is even a *dirstat bug.
   
   That could very well be the case. However I think listing the root
   directory from a non-root backup is exactly what obnam should do
   anyway. Even "obnam ls" does this, it lists "/" as the first directory
   backed up:
   
   root@HomeC /Backup# obnam backup --repository="/Backup/obnam_Pictures"
   --exclude-from="/Backup/obnam_Pictures_exclude"
   --root="/Common/Pictures"
   root@HomeC /Backup# obnam ls --repository=/Backup/obnam_Pictures
   Generation 4711 (2016-11-20 23:42:31 +0100 - 2016-11-20 23:43:10 +0100)
   drwxr-xr-x    21 root     root           4096 2016-11-16 18:37:52 /
   drwxr-xr-x    11 root     root           4096 2015-09-11 15:19:36 /Common
   drwxrwxr-x     9 frank    users          4096 2016-11-19 22:09:00
   /Common/Pictures
   ...
   
   So, I think the `file == "/"` check actually circumvents the k4dirstat
   bug you described. I really recommend uding qdirstat after this :D
   
   _______________________________________________
   obnam-dev mailing list
   obnam-dev@obnam.org
   http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org