[darcs-devel] Filesystem in DB, but data in filesystem

[Salvatore Insalaco]

2007/7/29, Eric Y. Kow <eric.kow at gmail.com>:
> Now, if we do something like this, do we still need the filesystem in a
> proper DB or would an ad-hoc DB (aka text file) do the job just as well?

I was thinking about that too. In this case we can avoid the use of a
relational db, and use native Haskell data structures backed by a
serialization strategy; I was thinking about Data.Binary for
efficiency reason, but if we want to continue a long-standing
tradition of readable text files in Darcs, we can do it.

> I guess... that would at least 'fix' my scare scenario, people using
> Unison to synchronise files (because I know some non-expert darcs users
> who do make heavy use of it).  You rm a file, record and unison.  Either
> Unison or the user gets confused and puts the file back into pristine.
> Pristine is hereby corrupted.  But this kind of thing would be handled
> correctly, right?

Sure. The new file will just be ignored. We could provide in darcs
optimize a tool to scan the filesystem for files not in file index,
and delete them.

>
> > 5) Filenames are made up, so no find / sed issues.
>
> It reduces the risk anyway; some stupid sed tricks might not involve
> looking at filenames

Well, the really stupid sed tricks could mangle a single-file storage
too (and in worse ways...).

>
> Are there any kinds of issues with the table getting out of synch with
> the data, that is any fancy new kinds of corruption that can sneak in
> as a result of this scheme?

We have two scenarios of out-of-syncness:
1) We have a file in the file index that is no more on the filesystem.
As you said before we can complain, but forcing a transactional
behaviour, writing the file index only after flushing the file
insertion in the filesystem, we can mostly avoid it.
2) We have a file on the filesystem that is no more in the file index.
This is a no-problem situation, as we just don't see the file.

So we can prevent lot of problems just by writing the file index to
disk after additions and before deletions.

Salvatore