[darcs-devel] [issue1678] performance regression in check/whatsnew/record (2.3.x)

[Kevin Quick]

Tried to update the bug with numbers, but got this:

MOD_PYTHON ERROR

ProcessId:      3595
Interpreter:    'bugs.darcs.net'

ServerName:     'bugs.darcs.net'
DocumentRoot:   '/var/lib/roundup/trackers/darcs'

URI:            '/issue1678'
Location:       None
Directory:      '/var/lib/roundup/trackers/'
Filename:       '/var/lib/roundup/trackers/darcs/dummy.py'
PathInfo:       '/issue1678'

Phase:          'PythonHandler'
Handler:        'roundup.cgi.apache'

Traceback (most recent call last):

  File "/usr/lib/python2.5/site-packages/mod_python/importer.py", line 1537, in HandlerDispatch
    default=default_handler, arg=req, silent=hlist.silent)

  File "/usr/lib/python2.5/site-packages/mod_python/importer.py", line 1229, in _process_target
    result = _execute_target(config, req, object, arg)

  File "/usr/lib/python2.5/site-packages/mod_python/importer.py", line 1128, in _execute_target
    result = object(arg)

  File "/usr/lib/python2.5/site-packages/roundup/cgi/apache.py", line 95, in handler
    _tracker = roundup.instance.open(_home, not _debug)

  File "/usr/lib/python2.5/site-packages/roundup/instance.py", line 257, in open

  File "/usr/lib/python2.5/site-packages/roundup/instance.py", line 54, in __init__

  File "/usr/lib/python2.5/site-packages/roundup/instance.py", line 79, in get_backend_name

IOError: [Errno 24] Too many open files: '/var/lib/roundup/trackers/darcs/db/backend_name'


Sending them via this email instead.  Here are the numbers for both
the unrepaired and the repaired repo.  Note that only the 2.2.0
numbers are valid for the unfixed repo are valid: 2.3.1 and the
current darcs both abort with the zip file error.


=== dtest ===

                || darcs2.2.0        | darcs2.3.1        | darcs.net       
================++===================+===================+=================
     get (full) ||      30.6s 154.0M |       3.9s 159.0M |     29.4s 159.0M
get (lazy, x10) ||     107.2s   7.0M |       3.1s   9.0M |    118.2s   8.0M
       pull 100 ||      16.4s  23.0M |       8.8s  29.0M |         -      -
       annotate ||          -      - |          -      - |         -      -
         wh x50 ||     290.4s   0.0M |       0.3s   0.0M |     38.4s   0.0M
     wh mod x50 ||     290.5s  10.0M |       8.4s   0.0M |     38.4s   0.0M
      wh -l x20 ||     368.2s  83.0M |       5.1s  74.0M |     16.9s   0.0M
          check ||      27.4s 274.0M |       5.0s 275.0M |         -      -
         repair ||      27.2s 272.0M |       4.8s 275.0M |     77.4s 392.0M
      pull 1000 ||      49.1s  85.0M |       0.4s  84.0M |     39.5s  84.0M

=== drepair ===

                || darcs2.2.0        | darcs2.3.1        | darcs.net       
================++===================+===================+=================
     get (full) ||      29.6s 153.0M |      36.3s 160.0M |     32.8s 159.0M
get (lazy, x10) ||     114.4s   5.0M |     114.3s   8.0M |    118.7s   7.0M
       pull 100 ||      18.3s  28.0M |      18.0s  25.0M |      7.2s  29.0M
       annotate ||          -      - |          -      - |         -      -
         wh x50 ||     317.5s   0.0M |       7.8s   0.0M |      9.8s   0.0M
     wh mod x50 ||     316.7s  11.0M |       8.0s   6.0M |     10.1s   4.0M
      wh -l x20 ||     412.3s 145.0M |     403.0s 128.0M |     12.3s   5.0M
          check ||      27.6s 224.0M |      34.7s 275.0M |     76.7s 410.0M
         repair ||      27.5s 275.0M |      34.1s 274.0M |     78.0s 392.0M
      pull 1000 ||      48.7s  84.0M |      49.3s  86.0M |     39.6s  84.0M


          Type: darcs
        Format: hashed
      Pristine: HashedPristine
   Num Patches: 2913
 Max dir depth: 5
         Files: 4842


-KQ


On Sun, Jan 17, 2010 at 04:46:12PM +0000, Petr Ročkai wrote:
> 
> Petr Ročkai <me at mornfall.net> added the comment:
> 
> > === darcs ===
> >
> >                 ||  darcs-2.2.1        | darcs-2.3.1       |  darcs
> > ================++=====================+===================+===============
> >           check ||        11.7s 183.0M |       11.0s 55.0M |  42.4s 258.0M
> >          repair ||        11.8s 183.0M |       11.1s 55.0M |  44.2s 272.0M
> >
> >
> > === ghc-hashed ===
> >
> >                 ||  darcs-2.2.1         | darcs-2.3.1        |   darcs
> > ================++======================+====================+===============
> >           check ||       172.3s 1210.0M |      160.5s 146.0M |       -      -
> >          repair ||       173.0s 1287.0M |      127.7s 146.0M | 1190.6s 355.0M
> >
> > For details, see
> >  - http://lists.osuosl.org/pipermail/darcs-users/2010-January/022832.html
> >  - http://lists.osuosl.org/pipermail/darcs-users/2010-January/022833.html
> I have fixed the worst part of this problem in hashed-storage. Currently
> only available in its darcs version.
> 
> On my machine: darcs-2.3 = 1:53, darcs-HEAD = 2:49 (on ghc-hashed).
> 
> That's still a substantial difference, but much less than 2:06 against
> almost 20 minutes. I don't think I will have the time to bring this
> further down in time for darcs 2.4, so we'll have to accept this as a
> price for the refactoring. (The primary cause for this was my crusade to
> remove SlurpDirectory which is not yet complete, but it is reasonably
> close to completion, so maybe in 2.5 we can get rid of that...)
> 
> Yours,
>    Petr.
> 
> __________________________________
> Darcs bug tracker <bugs at darcs.net>
> <http://bugs.darcs.net/issue1678>
> __________________________________