[darcs-users] breaking hashed files into multiple directories

[zooko]

On Sep 1, 2008, at 16:23 PM, David Roundy wrote:

> I suppose this would be a good entry-level project for someone, doing
> the standard create 256 or so subdirectories based on the first couple
> (or more?) hex characters in the hash, and then distribute the files
> into those directories.

If somebody out there wants a trickier and more innovative project  
that is less likely to succeed, you could see if a compressing,  
archiving tool would serve the same purpose as a filesystem directory  
(or nested set of directories) while also offering space and maybe  
even time advantages.

The idea is, make a single compressed archive file which contains a  
bunch of "files" inside of it, where each file is (just like in the  
current hashed format), a patch whose filename is a hash and whose  
contents is the patch.

If the compressing archive tool that you use can efficient extract  
individual files from the archive, then this might be efficient  
enough, or even more efficient, in speed.

I'm personally interested in FreeArc because it has among the best  
compression ratio and speed among all such tools, because it is open  
source, and because it has a Haskell interface:

http://freearc.org

However, I don't know if it offers this functionality (extraction of  
individual files from a compressed archive) at all, and if it does I  
don't know how efficient it would be.

I do know the FreeARC doesn't compile "out of the box" on either of  
my two main development platforms: 64-bit Ubuntu or 64-bit Mac OS X,  
so there may be some porting work to be done even to start this job,  
and certainly there would be a lot of porting and testing and smooth  
upgrade issues to work out before such a thing could be shipped in  
darcs itself.

Regards,

Zooko
---
http://allmydata.org -- Tahoe, the Least-Authority Filesystem
http://allmydata.com -- back up all your files for $5/month