[darcs-users] Current Pull Attempting to Overallocate

David Roundy droundy at jdj5.mit.edu
Wed Jan 28 17:13:23 UTC 2004


On Wed, Jan 28, 2004 at 08:09:26AM -0800, Paul Snively wrote:
> *** malloc: vm_allocate(size=524288000) failed (error code=3)
> *** malloc[563]: error: Can't allocate region
> Ugh on length 4
...
> Oddly, darcs still seems to work OK, so far, even with the allocation 
> failures. But they make me nervous. And 524,288,000 is an insane amount 
> of even virtual space to try to allocate in one go, IMHO.
> 
> Thoughts?

This is a very ugly trick, which I'll probably have to give up on.  The
problem it tried to fix was that the ghc garbage collector never decreases
the size of its heap, which means that if darcs ever grows its ghc heap
beyond the amount of RAM available, it will be forever swapping until it
exits, even if the amount of memory actually used goes back down.

About half of the memory darcs uses isn't allocated by ghc itself, but
instead is malloced.  If I can keep this memory out of the ghc heap, then
when memory usage drops back down, it'll can drop back to half of its peak
value, which is VERY nice if its peak value happened to be about twice the
physical memory available.

The trick I use (which is in fpstring.c for the curious) is to allocate a
very large chunk of memory, then allocate a second large chunk of memory,
realloc the second large chunk to its desired size, and finally deallocate
the first large chunk.  The net result is that the allocated memory is far
from the ghc heap... or at least that seems to be the net result, and it
works well under linux.  I get the same errors you do on MacOS X, which is
presumably because MacOS is less optimistic than linux is with respect to
out of memory conditions.  Linux will just let the allocation succeed and
hope the memory isn't actually used, which works out fine in this case,
since darcs doesn't use the memory.

If anyone has any suggestions as to how one might be able to (nicely?)
control the location of malloced memory, that would be great... In the
meantime, I guess I'll have to go back to using a simple malloc.  :(
-- 
David Roundy
http://civet.berkeley.edu/droundy/




More information about the darcs-users mailing list