[darcs-users] meta robots nofollow in darcs.cgi?

David Roundy droundy at abridgegame.org
Sun Feb 20 14:10:17 UTC 2005


Hello users of darcs.cgi (and Will!),

I just noticed that google was browsing the darcs repository history
chewing up cpu on the darcs.net server.  I fixed it (or hope it will be
fixed when google notices) by adding a robots.txt file, but this made me
wonder whether we should add a <META NAME="ROBOTS" CONTENT="NOFOLLOW"> tag
to the output of darcs.cgi?

It seems to me that *almost* never would you like a robot to be indexing
the contents of your repository, since the required calls to annotate
are slow, and there are vast numbers of links in any reasonably-sized
repository.  But it may be that some users would prefer to have their
repository histories indexed.

Any thoughts?

I lean towards adding the meta tag, since that way users are less likely to
get bitten by robots accessing computationally expensive cgi pages.  I'd go
with "NOFOLLOW", but not "NOINDEX", so users could still find your
repository if it's being linked to, but robots won't end up trying to index
every change to every file and directory.
-- 
David Roundy
http://www.darcs.net




More information about the darcs-users mailing list