[darcs-users] Memory usage of Record

[Jason Dagit]

On Mon, Aug 31, 2009 at 9:23 AM, Reinier Lamers<tux_rocker at reinier.de> wrote:
> Hi all,
>
> On Sunday 30 August 2009 05:14:26 Jason Dagit wrote:
>> Hello,
>>
>> Recently I noticed that if you take a large code base and try to
>> import it into darcs that the performance is pretty dismal.  I was
>> thinking this was once optimized and perhaps it required a special
>> case like --all or something.
>
> That optimization was removed because with it you can record patches that are
> too big for the rest of the darcs machinery.

Okay.  I forgot to mention it, but I did find one bug in roundup about
that before I posted here.
>> By removing just the bangs, the profiler says that darcs is now doing
>> work 40% of the time.  That's a 10% increase.
>
> Did the maximum memory use drop or rise because of this modification?

It seemed to remain about the same total usage, I think.  It certainly
wasn't a huge difference.  On the other hand, given how much memory it
was using I may not have noticed a change of, say, 10%.

> So this was a version of darcs that would record addfile patches without the
> corresponding hunk?

Exactly.

> Would it be possible to incrementally write the pending patch, writing hunks
> that the user has agreed to to disk immediately? Or would that require a major
> overhaul of the whole path selection mechanism?

In this case, the pending patch is already written.  What is taking
all the memory is more like a tentative patch.  In memory, darcs is
constructing the addfiles + hunks that you want to view/record.  I was
imagining storing the user selections in some temp file.  Mainly
because I'm not sure what existing file would be an appropriate one to
piggy back on.

>> Does anyone else have suggestions?  Does anyone know if Petr's work
>> will make a difference here?  I heard rumors of a darcs-hs that uses
>> hashed-storage more aggressively.  But, I doubt it impacts the reading
>> of pending?
>
> AFAIK Petr has a zoo of repositories that perform poorly. They may be useful
> for benchmarking.

Thanks for the reminder.  I would like to focus on this performance
case for now though.  Once we efficiently handle pending I will move
on to other things.

Thanks,
Jason