Front page

"exclude-if-present" option

2173d4e7d8a4497da071ca9d46be611b
BLACKJACK HURRICANE STEAMSHIP

From: Michele Alessandrini <m.alessandrini@univpm.it>
Date: Thu, 29 Jun 2017 12:08:29 +0200

   Hi, I just implemented the "exclude-if-present" feature we were 
   discussing yesterday. Is the patch format correct?
   I'd be happy if you can review it and consider if it's worth. I tested 
   it and it seems to work correctly.
   Actually it's not been very difficult, because I copied the 
   "exclude-caches" plugin (very similar) and adapted for the new case.
   
   The "./check" suite mostly passes (I had to fight a little with pep8), 
   but it gives some errors at the end, I suspect they're not related to 
   code, I report the output at the end. Maybe I have not all the needed 
   programs installed? I followed the README, and by the way I use debian 
   testing so I conveniently found all your packages right in aptitude.
   
   About unit tests, I found there is no similar tests for exclusion cases, 
   so it would be difficult for me to write one from scratch, and I added 
   it to "without-tests" file together with the other exclude plugins. I 
   hope this is not a big problem.
   
   Now about the documentation: I modified the english markdown manual, 
   obviously I cannot modify german and french. By the way I'm Italian, it 
   would be nice if I could contribute a translation, but I'm not sure I'd 
   have the needed time, and secondly I think technical things are better 
   in original english. But let me know what you think.
   
   A note: the German PDF fails to build on my computer.
   
   A doubt: where is the man page? Because it needs to be updated too, but 
   I can't find its sources.
   
   A suggestion: adding "*.pyc" to .gitignore, otherwise the git status is 
   a mess with all the compiled python files.
   
   Bye and thanks, hope to hear from you
   Michele (male, if you are wondering)
   
   -------------------------------------------------------------------------------------------
   Failed scenarios:
        - backup non-basic filesystem objects
        - backup notices when extended attribute value changes
        - Browsing backups with FUSE plugin
        - use repository format 6
        - use repository format 6 with in-tree data
        - create and do a check of a kdirstat cache of interesting objects
   ERROR: Test suite FAILED in 6 scenarios
   ERROR: Command '['yarn', '-s', 'yarns/obnam.sh', '--env', 
   'REPOSITORY_FORMAT=6', '--require-assumptions', 
   'yarns/0010-introduction.yarn', 'yarns/0020-test-environment.yarn', 
   'yarns/0030-basics.yarn', 'yarns/0040-generations.yarn', 
   'yarns/0050-multiple-clients.yarn', 'yarns/0060-encryption.yarn', 
   'yarns/0070-compression.yarn', 'yarns/0080-verify.yarn', 
   'yarns/0090-lock-handling.yarn', 'yarns/0100-fuse.yarn', 
   'yarns/0110-administration.yarn', 'yarns/0120-corrupt-repo.yarn', 
   'yarns/0200-repo-formats.yarn', 'yarns/0300-kdirstat-integration.yarn', 
   'yarns/9000-implements.yarn']' returned non-zero exit status 1
   
    
    
    
    
   La seguente informativa e' inserita in automatico dal sistema al fine esclusivo della realizzazione dei fini istituzionali dell'ente. 
    
    
   INVESTI NELLA RICERCA 
    
   il 5 per mille all'Universita' Politecnica delle Marche e' un investimento per i giovani, per il loro futuro - C.F. 00382520427 
    
   http://www.univpm.it/5_per_mille
From: Jan Niggemann <jn@hz6.de>
Date: Thu, 29 Jun 2017 22:59:41 +0200

   Hi Michele,
   
   please post the error message you get when trying to build the German manual.
   
   Jan
   
   
   Am 29. Juni 2017 12:08:29 MESZ schrieb Michele Alessandrini <m.alessandrini@univpm.it>:
   >Hi, I just implemented the "exclude-if-present" feature we were 
   >discussing yesterday. Is the patch format correct?
   >I'd be happy if you can review it and consider if it's worth. I tested 
   >it and it seems to work correctly.
   >Actually it's not been very difficult, because I copied the 
   >"exclude-caches" plugin (very similar) and adapted for the new case.
   >
   >The "./check" suite mostly passes (I had to fight a little with pep8), 
   >but it gives some errors at the end, I suspect they're not related to 
   >code, I report the output at the end. Maybe I have not all the needed 
   >programs installed? I followed the README, and by the way I use debian 
   >testing so I conveniently found all your packages right in aptitude.
   >
   >About unit tests, I found there is no similar tests for exclusion
   >cases, 
   >so it would be difficult for me to write one from scratch, and I added 
   >it to "without-tests" file together with the other exclude plugins. I 
   >hope this is not a big problem.
   >
   >Now about the documentation: I modified the english markdown manual, 
   >obviously I cannot modify german and french. By the way I'm Italian, it
   >
   >would be nice if I could contribute a translation, but I'm not sure I'd
   >
   >have the needed time, and secondly I think technical things are better 
   >in original english. But let me know what you think.
   >
   >A note: the German PDF fails to build on my computer.
   >
   >A doubt: where is the man page? Because it needs to be updated too, but
   >
   >I can't find its sources.
   >
   >A suggestion: adding "*.pyc" to .gitignore, otherwise the git status is
   >
   >a mess with all the compiled python files.
   >
   >Bye and thanks, hope to hear from you
   >Michele (male, if you are wondering)
   >
   >-------------------------------------------------------------------------------------------
   >Failed scenarios:
   >     - backup non-basic filesystem objects
   >     - backup notices when extended attribute value changes
   >     - Browsing backups with FUSE plugin
   >     - use repository format 6
   >     - use repository format 6 with in-tree data
   >     - create and do a check of a kdirstat cache of interesting objects
   >ERROR: Test suite FAILED in 6 scenarios
   >ERROR: Command '['yarn', '-s', 'yarns/obnam.sh', '--env', 
   >'REPOSITORY_FORMAT=6', '--require-assumptions', 
   >'yarns/0010-introduction.yarn', 'yarns/0020-test-environment.yarn', 
   >'yarns/0030-basics.yarn', 'yarns/0040-generations.yarn', 
   >'yarns/0050-multiple-clients.yarn', 'yarns/0060-encryption.yarn', 
   >'yarns/0070-compression.yarn', 'yarns/0080-verify.yarn', 
   >'yarns/0090-lock-handling.yarn', 'yarns/0100-fuse.yarn', 
   >'yarns/0110-administration.yarn', 'yarns/0120-corrupt-repo.yarn', 
   >'yarns/0200-repo-formats.yarn', 'yarns/0300-kdirstat-integration.yarn',
   >
   >'yarns/9000-implements.yarn']' returned non-zero exit status 1
   >
   > 
   > 
   > 
   > 
   >La seguente informativa e' inserita in automatico dal sistema al fine
   >esclusivo della realizzazione dei fini istituzionali dell'ente. 
   > 
   > 
   >INVESTI NELLA RICERCA 
   > 
   >il 5 per mille all'Universita' Politecnica delle Marche e' un
   >investimento per i giovani, per il loro futuro - C.F. 00382520427 
   > 
   >http://www.univpm.it/5_per_mille
From: Michele Alessandrini <m.alessandrini@univpm.it>
Date: Fri, 30 Jun 2017 09:13:03 +0200

   Il 29/06/2017 22:59, Jan Niggemann ha scritto:
   > Hi Michele,
   >
   > please post the error message you get when trying to build the German
   > manual.
   >
   
   It's a latex error about babel:
   
   ./format-de-pdf obnam-manual.de.pdf \
   	"Backups mit Obnam" "Lars Wirzenius (liw@liw.fi)" "Version 
   obnam-1.22-2-g50fbcd08" "Übersetzung: Jan Niggemann (jn@hz6.de)" \
   	de/010-einleitung.mdwn de/015-uebersicht.mdwn de/020-konzepte.mdwn 
   de/040-installation.mdwn de/060-sichern.mdwn 
   de/070-wiederherstellen.mdwn de/080-loeschen.mdwn 
   de/090-ueberpruefen.mdwn de/100-mehrere-clients.mdwn 
   de/110-verschluesselung.mdwn de/120-verschiedenes.mdwn 
   de/130-fallbeispiele.mdwn de/140-fehlersuche.mdwn 
   de/150-konfiguration.mdwn de/155-repo.mdwn de/160-tuning.mdwn 
   de/700-mitarbeit.mdwn de/800-fehler.mdwn de/801-fehler.mdwn 
   de/900-siehe-auch.mdwn de/980-juristenzeug.mdwn de/990-klappentext.mdwn
   ! Package babel Error: You haven't specified a language option.
   
   See the babel package documentation for explanation.
   Type  H <return>  for immediate help.
     ...
   
   l.343 ...ry to proceed from here, type x to quit.}
   
   pandoc: Error producing PDF
   
   
    
    
    
    
   La seguente informativa e' inserita in automatico dal sistema al fine esclusivo della realizzazione dei fini istituzionali dell'ente. 
    
    
   INVESTI NELLA RICERCA 
    
   il 5 per mille all'Universita' Politecnica delle Marche e' un investimento per i giovani, per il loro futuro - C.F. 00382520427 
    
   http://www.univpm.it/5_per_mille 
   
   _______________________________________________
   obnam-dev mailing list
   obnam-dev@obnam.org
   http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org
From: Jan Niggemann <jn@hz6.de>
Date: Fri, 30 Jun 2017 18:28:59 +0200

   Zitat von Michele Alessandrini <m.alessandrini@univpm.it>:
   > ! Package babel Error: You haven't specified a language option.
   What kind of system are you building on?
   German language has funny diacritics like ä, ö or ü, are  
   language-specific packages like texlive-lang-german or similar  
   installed?
   
   -- jan
From: Lars Wirzenius <liw@liw.fi>
Date: Mon, 3 Jul 2017 08:33:10 +0300

   I've not had time to try this yet, but from reading the patch it looks
   good. A few comments below.
   
   On Thu, Jun 29, 2017 at 12:08:29PM +0200, Michele Alessandrini wrote:
   > The "./check" suite mostly passes (I had to fight a little with pep8), but
   > it gives some errors at the end, I suspect they're not related to code, I
   > report the output at the end. Maybe I have not all the needed programs
   > installed? I followed the README, and by the way I use debian testing so I
   > conveniently found all your packages right in aptitude.
   
   I'll have a look and see if I can figure out what's wrong. The README
   may be out of date, but debian/control should be up to date.
   
   > About unit tests, I found there is no similar tests for exclusion cases, so
   > it would be difficult for me to write one from scratch, and I added it to
   > "without-tests" file together with the other exclude plugins. I hope this is
   > not a big problem.
   
   Yeah, that's OK. In principle it'd be good to add a yarn test for
   integration testing, so that there is _some_ assurance the new feature
   work. Adapting the "exclude cache directories" section in
   yarns/0030-basic.yarn should be fairly easy.
   
   (Yarn in my tool for integration testing. It's in the cmdtest package.)
   
   > Now about the documentation: I modified the english markdown manual,
   > obviously I cannot modify german and french. By the way I'm Italian, it
   > would be nice if I could contribute a translation, but I'm not sure I'd have
   > the needed time, and secondly I think technical things are better in
   > original english. But let me know what you think.
   
   I'd love for the Obnam manual to be translated to Italian, and other
   languages. While I prefer English myself, a lot of people don't know
   much or any English.
   
   > A note: the German PDF fails to build on my computer.
   
   I believe Jan has responsded to this. I tend to install the
   texlive-full package, since life is too short to find the minimal
   required set of packages from the TeX universe.
   
   > A doubt: where is the man page? Because it needs to be updated too, but I
   > can't find its sources.
   
   See obnam.1.in for the English version. It gets filled in with
   information about options automatically during the build process,
   using the help text for the options.
   
   > A suggestion: adding "*.pyc" to .gitignore, otherwise the git status is a
   > mess with all the compiled python files.
   
   I prefer to add *.pyc to my global git ignore file, instead of adding
   it to .gitignore in every Python project separately.
   
   > +    def config_loaded(self):
   > +        if len(self.app.settings['exclude-if-present']) > 0:
   
   This would feel more Pythonic to me like this:
   
       if self.app.settings['exclude-if-present']:
   
   Not a big deal.
From: Michele Alessandrini <m.alessandrini@univpm.it>
Date: Mon, 3 Jul 2017 10:05:33 +0200

   Il 30/06/2017 18:28, Jan Niggemann ha scritto:
   >
   > Zitat von Michele Alessandrini <m.alessandrini@univpm.it>:
   >> ! Package babel Error: You haven't specified a language option.
   > What kind of system are you building on?
   > German language has funny diacritics like ä, ö or ü, are
   > language-specific packages like texlive-lang-german or similar installed?
   >
   > -- jan
   >
   
   Hi, after trying installing all of the latex packages with no success, I 
   found the working solution is setting "lang=de" instead of "lang=german" 
   in the format-de-pdf script.
   
   
    
    
    
    
   La seguente informativa e' inserita in automatico dal sistema al fine esclusivo della realizzazione dei fini istituzionali dell'ente. 
    
    
   INVESTI NELLA RICERCA 
    
   il 5 per mille all'Universita' Politecnica delle Marche e' un investimento per i giovani, per il loro futuro - C.F. 00382520427 
    
   http://www.univpm.it/5_per_mille 
   
   _______________________________________________
   obnam-dev mailing list
   obnam-dev@obnam.org
   http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org
From: Michele Alessandrini <m.alessandrini@univpm.it>
Date: Mon, 3 Jul 2017 10:47:20 +0200

   >
   > Yeah, that's OK. In principle it'd be good to add a yarn test for
   > integration testing, so that there is _some_ assurance the new feature
   > work. Adapting the "exclude cache directories" section in
   > yarns/0030-basic.yarn should be fairly easy.
   >
   > (Yarn in my tool for integration testing. It's in the cmdtest package.)
   >
   
   Ok. Now only 2 yarns fail, and they're about "repository format 6", 
   maybe it's something experimental. I'll add my yarn test as you say.
   
   
   > I'd love for the Obnam manual to be translated to Italian, and other
   > languages. While I prefer English myself, a lot of people don't know
   > much or any English.
   >
   
   No problem, I'd be happy to do it, I'll plan to do it soon.
   
   
   >
   >> +    def config_loaded(self):
   >> +        if len(self.app.settings['exclude-if-present']) > 0:
   >
   > This would feel more Pythonic to me like this:
   >
   >     if self.app.settings['exclude-if-present']:
   >
   
   Very well, I changed it and improved the description for the man page 
   (set a more readable "metavar"). I'll send a revised patch together with 
   the yarn integration.
   
   Thanks
   Michele
   
    
    
    
    
   La seguente informativa e' inserita in automatico dal sistema al fine esclusivo della realizzazione dei fini istituzionali dell'ente. 
    
    
   INVESTI NELLA RICERCA 
    
   il 5 per mille all'Universita' Politecnica delle Marche e' un investimento per i giovani, per il loro futuro - C.F. 00382520427 
    
   http://www.univpm.it/5_per_mille 
   
   _______________________________________________
   obnam-dev mailing list
   obnam-dev@obnam.org
   http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org
From: Michele Alessandrini <m.alessandrini@univpm.it>
Date: Mon, 3 Jul 2017 15:42:26 +0200

   Hi, I made some changes to the first patch:
   - modified python code as suggested by Lars
   - improved option description string for man page
   - added a test in the yarns
   
   I managed to add a test among the other scenarios by using only the 
   existing "IMPLEMENTS" recipe. I'm confident it passes (it was not 
   passing in the first tries for syntax mistakes). I confirm that I have 
   an error for 2 scenarios only, I report the output here if you are 
   interested.
   
   Bye and thanks, let me know!
   Michele
   
   
   ----------------------------------------------------------
   
   ERROR: In scenario "use repository format 6"
   step "THEN restored data in R1 matches T/manifest-1" failed,
   with exit code 1:
   Standard output from shell command:
        --- 
   /tmp/tmpWC8Ic9/use_repository_format_6/datadir/original-manifest 
   2017-07-03 15:28:21.712932607 +0200
        +++ 
   /tmp/tmpWC8Ic9/use_repository_format_6/datadir/restored-manifest 
   2017-07-03 15:28:21.708932607 +0200
        @@ -1,7 +1,7 @@
         Name: .
         Mtime: 2013-12-28 21:11:07 +0000
         Mode: 40775
        -Nlink: 4
        +Nlink: 1
   
         Name: fifo
         Mtime: 2013-12-28 21:11:07 +0000
        @@ -111,10 +111,10 @@
         Name: dir-empty
         Mtime: 2013-12-28 21:11:07 +0000
         Mode: 40775
        -Nlink: 2
        +Nlink: 1
   
         Name: symlink-target-dir
         Mtime: 2013-12-28 21:11:07 +0000
         Mode: 40775
        -Nlink: 2
        +Nlink: 1
   
   
   Standard error from shell command:
        + GNUPGHOME=/tmp/tmpWC8Ic9/use_repository_format_6/datadir/dot-gnupg
        + mkdir -p /tmp/tmpWC8Ic9/use_repository_format_6/datadir/dot-gnupg
        + rsync -a --delete /home/michele/devel/obnam/test-gpghome/. 
   /tmp/tmpWC8Ic9/use_repository_format_6/datadir/dot-gnupg/.
        + cd /tmp/tmpWC8Ic9/use_repository_format_6/datadir/R1
        + true
        + ls
        + wc -l
        + cd tmp
        + true
        + ls
        + wc -l
        + cd tmpbr5YS5
        + true
        + + wcls
         -l
        + cd data
        + true
        + ls
        + wc -l
        + break
        + summain -r --exclude=Ino --exclude=Dev --exclude=Uid 
   --exclude=Username --exclude=Gid --exclude=Group --checksum=SHA1 .
        + normalise_manifest_times
        + sed /^Mtime:/s/\.[0-9]* / /
        + normalise_manifest_times 
   /tmp/tmpWC8Ic9/use_repository_format_6/datadir/T/manifest-1
        + sed /^Mtime:/s/\.[0-9]* / / 
   /tmp/tmpWC8Ic9/use_repository_format_6/datadir/T/manifest-1
        + diff -u 
   /tmp/tmpWC8Ic9/use_repository_format_6/datadir/original-manifest 
   /tmp/tmpWC8Ic9/use_repository_format_6/datadir/restored-manifest
   
   ERROR: In scenario "use repository format 6 with in-tree data"
   step "THEN restored data in R1 matches T/manifest-1" failed,
   with exit code 1:
   Standard output from shell command:
        --- 
   /tmp/tmpYuAW3Y/use_repository_format_6_with_in-tree_data/datadir/original-manifest 
   2017-07-03 15:28:23.648932644 +0200
        +++ 
   /tmp/tmpYuAW3Y/use_repository_format_6_with_in-tree_data/datadir/restored-manifest 
   2017-07-03 15:28:23.644932644 +0200
        @@ -1,7 +1,7 @@
         Name: .
         Mtime: 2014-01-07 18:36:19 +0000
         Mode: 40775
        -Nlink: 4
        +Nlink: 1
   
         Name: fifo
         Mtime: 2014-01-07 18:36:19 +0000
        @@ -111,10 +111,10 @@
         Name: dir-empty
         Mtime: 2014-01-07 18:36:19 +0000
         Mode: 40775
        -Nlink: 2
        +Nlink: 1
   
         Name: symlink-target-dir
         Mtime: 2014-01-07 18:36:19 +0000
         Mode: 40775
        -Nlink: 2
        +Nlink: 1
   
   
   Standard error from shell command:
        + 
   GNUPGHOME=/tmp/tmpYuAW3Y/use_repository_format_6_with_in-tree_data/datadir/dot-gnupg
        + mkdir -p 
   /tmp/tmpYuAW3Y/use_repository_format_6_with_in-tree_data/datadir/dot-gnupg
        + rsync -a --delete /home/michele/devel/obnam/test-gpghome/. 
   /tmp/tmpYuAW3Y/use_repository_format_6_with_in-tree_data/datadir/dot-gnupg/.
        + cd 
   /tmp/tmpYuAW3Y/use_repository_format_6_with_in-tree_data/datadir/R1
        + true
        + + wc -l
        ls
        + cd tmp
        + true
        + + wc -l
        ls
        + cd tmpvR_VsS
        + true
        + ls+
        wc -l
        + cd data
        + true
        + + wc -l
        ls
        + break
        + summain -r --exclude=Ino --exclude=Dev+  --exclude=Uid 
   --exclude=Username --exclude=Gid --exclude=Group --checksum=SHA1 .
        normalise_manifest_times
        + sed /^Mtime:/s/\.[0-9]* / /
        + normalise_manifest_times 
   /tmp/tmpYuAW3Y/use_repository_format_6_with_in-tree_data/datadir/T/manifest-1
        + sed /^Mtime:/s/\.[0-9]* / / 
   /tmp/tmpYuAW3Y/use_repository_format_6_with_in-tree_data/datadir/T/manifest-1
        + diff -u 
   /tmp/tmpYuAW3Y/use_repository_format_6_with_in-tree_data/datadir/original-manifest 
   /tmp/tmpYuAW3Y/use_repository_format_6_with_in-tree_data/datadir/restored-manifest
   
   Failed scenarios:
        - use repository format 6
        - use repository format 6 with in-tree data
   ERROR: Test suite FAILED in 2 scenarios
   ERROR: Command '['yarn', '-s', 'yarns/obnam.sh', '--env', 
   'REPOSITORY_FORMAT=6', 'yarns/0010-introduction.yarn', 
   'yarns/0020-test-environment.yarn', 'yarns/0040-generations.yarn', 
   'yarns/0050-multiple-clients.yarn', 'yarns/0060-encryption.yarn', 
   'yarns/0070-compression.yarn', 'yarns/0080-verify.yarn', 
   'yarns/0090-lock-handling.yarn', 'yarns/0100-fuse.yarn', 
   'yarns/0110-administration.yarn', 'yarns/0120-corrupt-repo.yarn', 
   'yarns/0200-repo-formats.yarn', 'yarns/0300-kdirstat-integration.yarn', 
   'yarns/9000-implements.yarn', 'yarns/0030-basics.yarn']' returned 
   non-zero exit status 1
   
   
    
    
    
    
   La seguente informativa e' inserita in automatico dal sistema al fine esclusivo della realizzazione dei fini istituzionali dell'ente. 
    
    
   INVESTI NELLA RICERCA 
    
   il 5 per mille all'Universita' Politecnica delle Marche e' un investimento per i giovani, per il loro futuro - C.F. 00382520427 
    
   http://www.univpm.it/5_per_mille