[darcs-users] darcs patch: switch Darcs.Patch.FileName to be ByteString.Char8 int...

Jason Dagit dagit at codersbase.com
Thu Oct 1 06:15:21 UTC 2009


Thanks Simon and Duncan.

I'm still trying to wrap my head around this so I have some questions below.

On Fri, Sep 25, 2009 at 6:58 AM, Simon Marlow <marlowsd at gmail.com> wrote:

> On 25/09/2009 14:25, Jason Dagit wrote:
>
>>
>>
>> On Fri, Sep 25, 2009 at 4:34 AM, Simon Marlow <marlowsd at gmail.com
>> <mailto:marlowsd at gmail.com>> wrote:
>>
>>    On 24/09/2009 04:57, Jason Dagit wrote:
>>
>>        I ran this more with normal builds and just the GC statistics.
>>          Here is
>>        what we see:
>>        unmodified darcs:
>>        469 MB total memory in use (3 MB lost due to fragmentation)
>>        Total time   26.51s  ( 29.76s elapsed)
>>        Productivity  85.6% of total user, 76.2% of total elapsed
>>
>>        469 MB total memory in use (3 MB lost due to fragmentation)
>>        Total time   26.59s  ( 32.47s elapsed)
>>        Productivity  85.6% of total user, 70.1% of total elapsed
>>
>>        469 MB total memory in use (3 MB lost due to fragmentation)
>>        Total time   26.54s  ( 29.95s elapsed)
>>        Productivity  85.5% of total user, 75.8% of total elapsed
>>
>>        With my patch applied:
>>        554 MB total memory in use (4 MB lost due to fragmentation)
>>        Total time   23.30s  ( 31.56s elapsed)
>>        Productivity  83.1% of total user, 61.3% of total elapsed
>>
>>        554 MB total memory in use (4 MB lost due to fragmentation)
>>        Total time   22.85s  ( 26.33s elapsed)
>>        Productivity  82.9% of total user, 71.9% of total elapsed
>>
>>        554 MB total memory in use (4 MB lost due to fragmentation)
>>        Total time   22.88s  ( 26.38s elapsed)
>>        Productivity  82.8% of total user, 71.8% of total elapsed
>>
>>        Now that the profiler is disabled the productivity with my
>>        changes is
>>        less, the run-time is maybe improved by 2-3 seconds, and the memory
>>        usage has increased by almost 100 megs.  I can only assume that the
>>        profiling is interfering with my results a fair bit.
>>
>>
>>    The profile graphs in your previous message showed a residency of
>>    around ~30M before your patch, and ~6M after your patch.  Which
>>    seems like a worthwhile saving.  I presume those graphs were from a
>>    different test case?  If not, then something very strange is going
>>    on.  If they are from a different test case, then do the numbers
>>    stand up when using the non-profiled darcs?
>>
>>
>> Thanks for trying to help me understand the profiling.
>>
>> My test case has not changed at all.  The only thing I've been changing
>> is the darcs source code.
>>
>> Do you have suggestions on determining which strange thing may be
>> happening?
>>
>
> Well, if the heap profile is showing a ~30M residency and yet the runtime
> is using 500MB, there's about 400MB unaccounted for.
>
> Ah - so one problem is that the contents of ByteStrings are not accounted
> for in the heap profile, for unpleasant technical reasons to do with the way
> they're stored.  Unfortunately it's not easy to fix, because the GC doesn't
> retain information about which ByteStrings are live, only that the 4KB block
> in which they live is still alive.
>

Does this also mean that the space consumed by ByteStrings is not considered
for the purposes of memory pressure?  I've always noticed that darcs tends
to allocate a lot of virtual memory, and now this discussion makes me wonder
if that's partially because the GC isn't collecting ByteStrings as eagerly
as it does other garbage?  If this is the case, is there a way to specify
the memory pressure of a value?  Basically, I'm saying, "Can I give hints to
the GC about the relative need to collect something once it becomes
garbage?"

Or perhaps I'm completely misunderstanding the problem here.

I've been looking at taking some dynamic library injection code that Ian
wrote for linux and porting it to OS X so that I can track how much memory
we allocate via mmap (which I believe should catch everything that is
unaccounted for in this case).

Is there a way to convince the RTS to collect more eagerly or run any
finalizers (or whatever it would take) to get any ByteStrings cleaned up
more eagerly?  This might make for a nice experiment.


>
> The underlying problem in your example may be fragmentation, due to the way
> that ByteStrings are pinned and hence hold on to the whole 4KB block in
> which they were allocated until they die.  Duncan has been thinking about
> how to improve the situation, but I'm not sure of the current status -
> Duncan?
>

What is it about fragmentation that is an issue?  We are definitely using a
lot of bytestrings and a lot of mmap memory.  If you told me that the GC
simply isn't tracking this memory in the heap profile then it makes a lot of
sense what I'm seeing.  If you're suggesting something else is going on in
addition to the deficient tracking then I need to spend some time
understanding the problem better.

Thanks,
Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osuosl.org/pipermail/darcs-users/attachments/20090930/ba302d08/attachment-0001.htm>


More information about the darcs-users mailing list