[darcs-devel] [issue471] Too much memory used!
Abram Hindle
bugs at darcs.net
Thu Jun 14 10:16:28 PDT 2007
New submission from Abram Hindle <abez at abez.ca>:
Repository: http://churchturing.org/w/test.case/
Problem: It uses too much memory to record changes and to push changes.
In a similar repo I run out of memory pushing changes around.
realloc: resource exhausted (out of memory)
I think it hits near a 2GB limit and then dies. Regardless it shouldn't
use that memory for such a minor change.
The change is to a 70kloc file with additions every 20 lines.
My machines:
linux 2.4.25 i686 , Glasgow Haskell Compiler, Version 6.4, for
Haskell 98, compiled by GHC version 5.04.3
linux 2.6.15-27-686, Glasgow Haskell Compiler, Version 6.4.1, for
Haskell 98, compiled by GHC version 6.4.1
This is with darcs 1.0.9 stable, compiled from the repo.
Thanks,
abram
p.s. attached is the chat log and a
--- CHATLOGS! ----
21:54 < abez> Does anyone have problems with running out of memory on a
push or a pull?
21:55 < kc5tja> That depends. I've had darcs attempt to solve the
halting problem while trying to merge several parallel branches of code
once, but it never ran out of memory.
21:55 < abez> I had a 70kloc file which recieved changes every 20 lines
for one patch
21:55 < abez> It must've hit the 2gb limit
21:56 < abez> realloc: resource exhausted (out of memory)
21:56 < kc5tja> What version of darcs?
21:56 < abez> 1.9
21:56 * kc5tja attempts to replicate.
21:56 < abez> 1.0.9
21:57 -!- MarcWebe1 [n=marc at 217.224.137.40] has joined #darcs
21:59 < kc5tja> OK, commiting the source file now. It's definitely
taking some time to commit the initial import.
21:59 < kc5tja> Just finished.
21:59 < kc5tja> Now to make a script that alters one character randomly
every 20 lines.
22:05 < abez> I got it up to 1200mb
22:06 < abez> (VIRT and 700 RES)
22:06 -!- Jerri [n=gerhard at dslb-084-056-036-081.pools.arcor-ip.net] has
joined #darcs
22:12 < kc5tja> So, let me see if my math is right.
22:12 < kc5tja> 70,000 LOC, one change every 20 lines, that's about 3500
changes total for the file.
22:12 < kc5tja> According to my diff, I have about 5000 changes total
(not sure why, but either way, I've confirmed it's one change per line,
for more lines than my estimate)
22:12 < kc5tja> Here goes nothing....
22:15 < kc5tja> M ./70kloc.src -3498 +3462
22:15 < kc5tja> That's what darcs w -l reports, so I'm close. diff was
giving me some extra lines in its output.
22:15 < kc5tja> Took 2 minutes to generate the w -l report.
22:15 < kc5tja> now for the commit
22:18 -!- clanehin [n=clanehin at cpe-069-134-236-134.nc.res.rr.com] has
quit [Remote closed the connection]
22:18 -!- MarcWeber [n=marc at pD9E09F07.dip.t-dialin.net] has quit [Read
error: 110 (Connection timed out)]
22:20 -!- Jerri_ [n=gerhard at dslb-084-056-053-202.pools.arcor-ip.net] has
quit [Read error: 110 (Connection timed out)]
22:20 < abez> kc5tja: I just tried to commit a change to a test repo and
I accidently modified the test file. Instead of selecting d to record
the current selections, I pressed s
22:20 < abez> kc5tja: now I'm waiting for linux to tell darcs it is out
of memory
22:20 < abez> 1219m V with 685m RES
22:20 < kc5tja> Ditto, my darcs record is still in progress.
22:21 < kc5tja> However, it's at 78MB total, and has been stable.
22:22 < abez> I had the same behaviour on a 2.4 kernel and on an ubuntu
Dapper system
22:22 < abez> 13810 abez 19 0 1240m 649m 61m R 3.6 85.8
1:50.81 darcs
22:22 < kc5tja> What kernel version? I don't even know what I'm at. ;D
22:23 < kc5tja> Ooh, did you compile darcs yourself? If so, what
version of GHC did you use?
22:23 < kc5tja> 2.6.17-11 here (GoboLinux)
22:23 < kc5tja> GHC 6.6 (but no patch level indicated.)
22:28 < abez> kc5tja: linux 2.4.25 i686 , Glasgow Haskell Compiler,
Version 6.4, for Haskell 98, compiled by GHC version 5.04.3
22:28 < abez> kc5tja: 2.6.15-27-686, Glasgow Haskell Compiler, Version
6.4.1, for Haskell 98, compiled by GHC version 6.4.1
22:28 < abez> and this happened with 1.0.3 and 1.0.7
22:30 < kc5tja> Yeah, the fact that I have a later version of GHC might
have an impact (perhaps better garbage collection).
22:30 < kc5tja> It finished!
22:30 < kc5tja> darcs record -a -m "BUG: BOMBS AWAY" 334.39s user
89.29s system 90% cpu 7:46.38 total
22:32 < abez> kc5tja: http://churchturing.org/w/test.case/ here's my
test case
22:32 < lambdabot> Title: Index of /w/test.case
22:33 < kc5tja> Mine isn't much different from yours. Written in Python
though.
22:34 < kc5tja> kc5tja at aldeberan ~/tmp/bug]cat 70kloc.py
22:34 < kc5tja> #!/usr/bin/python
22:34 < kc5tja> l = "akhdflakdflakhdf KLSHFDKLSHFDKL KLjhdflaskjhdflksJ
fLSD FSKLJHFDSKLjhfd skljhfsd fKJH"
22:34 < kc5tja> open("70kloc.src", "w").write( "\n".join([l for x in
xrange(70000)]))
22:34 < kc5tja> That generates the file.
22:35 < kc5tja> http://pastebin.ca/565651
22:35 < kc5tja> That is the modifier script.
-------------------
Top Memory Log:
13512 abez 17 0 48256 47m 3064 R 92.0 6.2 0:08.67 darcs
13512 abez 16 0 50764 49m 3404 R 35.2 6.5 0:13.75 darcs
13512 abez 15 0 81300 79m 3404 R 99.3 10.5 0:19.74 darcs
13512 abez 19 0 179m 179m 3404 R 99.0 23.7 0:22.72 darcs
13512 abez 20 0 311m 311m 3404 R 96.0 41.2 0:25.61 darcs
13512 abez 17 0 461m 461m 3404 R 98.7 61.0 0:28.58 darcs
13512 abez 16 0 643m 626m 3524 D 36.1 82.8 0:32.53 darcs
13599 abez 18 0 255m 255m 3368 R 99.0 33.7 0:23.73 darcs
13599 abez 16 0 407m 407m 3368 R 99.0 53.8 0:26.71 darcs
13599 abez 16 0 558m 558m 3368 R 97.7 73.8 0:29.65 darcs
13599 abez 19 0 645m 645m 3368 R 62.8 85.3 0:31.54 darcs
13599 abez 16 0 751m 697m 3792 D 28.0 92.2 0:34.03 darcs
push
13649 abez 20 0 84568 82m 3396 R 99.0 10.9 0:08.27 darcs
13649 abez 19 0 208m 208m 3396 R 97.3 27.6 0:11.20 darcs
13649 abez 17 0 365m 365m 3396 R 99.0 48.3 0:14.18 darcs
13649 abez 16 0 520m 520m 3396 R 99.3 68.7 0:17.17 darcs
13649 abez 17 0 703m 702m 3288 S 35.5 92.8 0:21.14 darcs
13649 abez 20 0 703m 684m 3796 R 58.8 90.5 0:22.91 darcs
13649 abez 16 0 721m 699m 4420 D 85.1 92.4 0:28.45 darcs
13649 abez 17 0 740m 684m 4496 R 32.2 90.4 0:29.42 darcs
13649 abez 16 0 928m 688m 4752 R 22.6 91.0 0:35.47 darcs
13649 abez 16 0 928m 688m 4752 R 22.6 91.0 0:35.47 darcs
13649 abez 18 0 993m 663m 4752 D 13.3 87.7 0:37.11 darcs
13649 abez 14 0 1119m 697m 4752 D 37.5 92.1 0:40.19 darcs
13649 abez 14 0 1119m 697m 4752 D 37.5 92.1 0:40.19 darcs
13649 abez 14 0 1158m 704m 4752 R 33.9 93.1 0:41.21 darcs
13649 abez 17 0 1201m 703m 4752 R 36.7 93.0 0:42.38 darcs
13649 abez 15 0 1245m 704m 4752 R 41.2 93.1 0:43.62 darcs
13649 abez 14 0 899m 339m 4752 R 27.6 44.8 0:44.45 darcs
13649 abez 13 0 644m 339m 4752 R 0.7 44.8 0:44.48 darcs
... then I got bored and later ran out of memory :( ...
----------
messages: 1717
nosy: EricKow, abez, beschmi, droundy, tommy
status: unread
title: Too much memory used!
____________________________________
Darcs issue tracker <bugs at darcs.net>
<http://bugs.darcs.net/issue471>
____________________________________
More information about the darcs-devel
mailing list