[darcs-users] Re: meta robots nofollow in darcs.cgi?

[Mark Stosberg]

David Roundy wrote:
 > Hello users of darcs.cgi (and Will!),
 >
 > I just noticed that google was browsing the darcs repository history
 > chewing up cpu on the darcs.net server.  I fixed it (or hope it will
 > be fixed when google notices) by adding a robots.txt file, but this
 > made me wonder whether we should add a <META NAME="ROBOTS"
 > CONTENT="NOFOLLOW"> tag to the output of darcs.cgi?
 >
 > It seems to me that *almost* never would you like a robot to be
 > indexing the contents of your repository, since the required calls
 > to annotate are slow, and there are vast numbers of links in any
 > reasonably-sized repository.  But it may be that some users would
 > prefer to have their repository histories indexed.
 >
 > Any thoughts?

I think this should be a user option, perhaps with the default being
no indexing for safety.

I have gotten useful answers from Google because it was able to search
in published source code, so I think it should be an option. Although
if 'annotate' is the only command used to display the source...and
it's always slow, maybe there is no good way currently to have the
source searchable with decent performance.

     Mark