[darcs-users] increasingly slow record times

[Nathan Gray]

I have been experimenting with putting images (20000 of them) into a
darcs repository.  This is mainly to help me keep track of when
individual images have been updated, and to ease deployment.

I was amazed to see how rapidly the speed of 'darcs record'
diminished.

I wrote a quick test example to show the rate of records.  I ran the
test with 3000 patches, but the rate continues to decrease.  To record
all 20000 of my images took about 24 hours.

  mkdir image_test
  cd image_test
  darcs init
  for number in `seq 1 3000`
  do
    echo "generating image $number"
    convert -background white -fill black -size 150x100 -gravity center label:$number $number.jpg
    echo `date +"%s.%N"` >> create.log
  done
  for number in `seq 1 3000`
  do
    darcs add $number.jpg
    darcs record --all --author image_test -m "initial import of image $number" $number.jpg
    echo `date +"%s.%N"` >> record.log
  done
  cat create.log | perl -e 'my $a; while (my $b = <STDIN>) { 
    chomp($b); 
    print ++$c . "\t" . (60 / ($b - $a)) . "\n" if $a; $a = $b 
  }' > create_rate.txt
  echo 'set term png; set output "create_rate.png"; plot "create_rate.txt"' | gnuplot -
  cat record.log | perl -e 'my $a; while (my $b = <STDIN>) { 
    chomp($b); 
    print ++$c . "\t" . (60 / ($b - $a)) . "\n" if $a; $a = $b 
  }' > record_rate.txt
  echo 'set term png; set output "record_rate.png"; plot "record_rate.txt"' | gnuplot -
  display record_rate.png create_rate.png
  cd ..

It appears that darcs repositories with non-binary files do the same
thing, which may mean that the slow-down has to do with the number of
files, rather than the number of commits.

mkdir text_test; cd text_test; darcs init; for number in `seq 1 3000`; do echo "generating file $number"; echo "this is file $number" > $number.txt; echo `date +"%s.%N"` >> create.log; done; for number in `seq 1 3000`; do darcs add $number.txt; darcs record --all --author text_test -m "initial import of file $number" $number.txt; echo `date +"%s.%N"` >> record.log; done; cat record.log | perl -e 'my $a; while (my $b = <STDIN>) { chomp($b); print ++$c . "\t" . (60 / ($b - $a)) .  "\n" if $a; $a = $b }' > record_rate.txt; echo 'set term png; set output "record_rate.png"; plot "record_rate.txt"' | gnuplot -; display record_rate.png; cd ..

My attempts at changing a single file 3000 times results in a repo
with less than 100 patches, because most of the time the change is not
detected.  I am not sure how to overcome this problem and get useful
benchmarking.

mkdir text_update_test; cd text_update_test; darcs init; touch 1.txt; darcs add 1.txt; for number in `seq 1 3000`; do echo "this is change $number" > 1.txt; cat 1.txt; darcs record --all --author text_update_test -m "updated to number $number" 1.txt; echo `date +"%s.%N"` >> record.log; done; cat record.log | perl -e 'my $a; while (my $b = <STDIN>) { chomp($b); print ++$c . "\t" . (60 / ($b - $a)) . "\n" if $a; $a = $b }' > record_rate.txt; echo 'set term png; set output "record_rate.png"; plot "record_rate.txt"' | gnuplot -; display record_rate.png; cd ..

I am attaching the rate graphs for all three tests.

-kolibrie

-------------- next part --------------
A non-text attachment was scrubbed...
Name: record_rate_image.png
Type: image/png
Size: 4783 bytes
Desc: not available
Url : http://lists.osuosl.org/pipermail/darcs-users/attachments/20071005/39d829d9/attachment.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: record_rate_text.png
Type: image/png
Size: 4628 bytes
Desc: not available
Url : http://lists.osuosl.org/pipermail/darcs-users/attachments/20071005/39d829d9/attachment-0001.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: record_rate_update_text.png
Type: image/png
Size: 7966 bytes
Desc: not available
Url : http://lists.osuosl.org/pipermail/darcs-users/attachments/20071005/39d829d9/attachment-0002.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://lists.osuosl.org/pipermail/darcs-users/attachments/20071005/39d829d9/attachment.pgp