[darcs-devel] Use System.Directory.copyFile for file copying

[Jason Dagit]

Fantastic analysis.  Exactly what I was looking for.  I'm now convinced :)

The icing on the cake would be to check this on various platforms but
I'm convinced enough already.

Thanks!
Jason

On 7/30/07, Kevin Quick <quick at sparq.org> wrote:
> On Sat, 28 Jul 2007 17:23:13 -0700, "Jason Dagit"
> <dagit at codersbase.com> wrote:
> > I think it would be nice to see a benchmark between the old and new
> > where there are thousands of tiny little files.  Is there a noticeable
> > performance difference?  The reason for lots of little files is
> > because of the permission copying.  I don't know that it would affect
> > the performance, but if the permission changes require a lot of
> > function calls then it could have a big impact.
> >
> > Jason
>
> Attached is copytest.hs.  It is given three directory names and
> copies from the first to the second using System.Directory.copyFile,
> then copies from the first to the third using readfilePS >>=
> writeFilePS.  It then reports the elapsed time for each test.
>
> To build (assuming it is located in the top-level of the darcs source
> tree):
>
> $ ghc --make -isrc copytest src/fpstring.c -lz
>
> I used two input sets.  The first directory set had 998 small files (50
> bytes to 500 bytes) in the source directory.  The second set had 4 big
> files:
>
> $ ls -lh copytestdir/big1
> total 83M
> -rw-rw-r-- 1 kquick kquick 16M Jul 30 22:15 bigfile1
> -rw-rw-r-- 1 kquick kquick 14M Jul 30 22:15 bigfile2
> -rw-rw-r-- 1 kquick kquick 30M Jul 30 22:15 bigfile3
> -rw-rw-r-- 1 kquick kquick 25M Jul 30 22:16 bigfile4
>
> Test runs:
>
> $ for X in 1 2 3 4 5 6 ; do rm copytestdir/small{2,3}/*; ./copytest
> copytestdir/small?; done
> Copy of 998 files:
>    via System.Directory.copyFile: 0.160895s
>    via readFilePS >= writeFilePS: 0.153626s
> Copy of 998 files:
>    via System.Directory.copyFile: 0.161436s
>    via readFilePS >= writeFilePS: 0.153718s
> Copy of 998 files:
>    via System.Directory.copyFile: 0.163191s
>    via readFilePS >= writeFilePS: 0.155526s
> Copy of 998 files:
>    via System.Directory.copyFile: 0.16112s
>    via readFilePS >= writeFilePS: 0.156255s
> Copy of 998 files:
>    via System.Directory.copyFile: 0.162132s
>    via readFilePS >= writeFilePS: 0.157913s
> Copy of 998 files:
>    via System.Directory.copyFile: 0.163213s
>    via readFilePS >= writeFilePS: 0.157451s
> $
>
>
> $ for X in 1 2 3 4 5 6 ; do rm copytestdir/big{2,3}/*; ./copytest
> copytestdir/ big?; done
> Copy of 4 files:
>    via System.Directory.copyFile: 10.418745s
>    via readFilePS >= writeFilePS: 11.420843s
> Copy of 4 files:
>    via System.Directory.copyFile: 8.318079s
>    via readFilePS >= writeFilePS: 16.533595s
> Copy of 4 files:
>    via System.Directory.copyFile: 8.384256s
>    via readFilePS >= writeFilePS: 11.35574s
> Copy of 4 files:
>    via System.Directory.copyFile: 7.752898s
>    via readFilePS >= writeFilePS: 14.43615s
> Copy of 4 files:
>    via System.Directory.copyFile: 8.029765s
>    via readFilePS >= writeFilePS: 14.187116s
> Copy of 4 files:
>    via System.Directory.copyFile: 7.85273s
>    via readFilePS >= writeFilePS: 12.406907s
>
>
> >From the above, I conclude that System.Directory.copyFile is better at
> actually copying the file data, and that the overhead of copying
> permissions (visible from copying small files) is quite small (about 8
> us/file).
>
>
> --
> --
> Kevin Quick
> quick at org after sparq
>
> _______________________________________________
> darcs-devel mailing list
> darcs-devel at darcs.net
> http://lists.osuosl.org/mailman/listinfo/darcs-devel
>
>
>