[bouncer-dev] Progress, roadmap
emsearcy at osuosl.org
Thu Jun 12 21:54:10 UTC 2008
Been a while since this list has seen any activity ... so here's a
long email to make up for it.
I wanted to post some thoughts on Bouncer direction, and some needs we
have at the OSL that we're going to start working on. Most of this
I've already discussed with Mike Morgan (morgamic).
First off is the idea (actually morgamic's) of bouncer as virtual
filesystem. Bouncer can easily get a list files available. The
Mozilla fork needs a local copy of the mirror in order to do hash
checks (since they are partial file, it cannot be stored in the DB
like v2 does). Even if a local copy is unavailable (perhaps a certain
install may only want per-file touch checks), the file list could
still be built via a one-argument rsync call to the master mirror.
So, how would this change operation? Rather than adding new product/
version/os/lang tuples, tied to a mirror location that would then be
checked, the new files can be detected automatically and added.
Sentry could then automatically start polling remote mirrors to see
which files match.
For simple mirrors, like Mozilla, where the structure is well
standardized, this would be all they would need. The change would
just be made in the original download chooser on the site to point to
a file path on the bouncer server instead of passing args:
http://download.server/path/to/file (redirects to) http://some.mirror/base/path/to/file
The idea of having a p/v/os/l tuple that ties to a filename could
still be implemented, and would be in a sense an extra redirect on the
bouncer side, /query?args -> /path/to/file. The advantage with moving
the template layer `higher' would be that: a) clients like Mozilla
that don't need it don't have to use it b) the templates don't have to
be used in sentry runs, just for download bounces---sentry just needs
to do filesystem-to-filesystem comparisons (think simple).
I've already started sketching out a DB design for this, with an added
feature (transparent if you don't want it) for multiple projects
within a single bouncer install. This basically means that for
different subdirectories in your filesystem, you can have different
mirrorsets and p/v/os/l links. Think using bouncer on a general-
purpose mirror that hosts lots of projects. If your bandwidth started
getting taxed, you could pass off requests to other mirrors.
That `once its taxed, start passing off' idea requires one last
feature I plan on putting in: tiers (also transparent if you don't use
them). The tier system has two purposes: failover to non-primary
mirrors (like other people's mirrors?) only if you run out of
bandwidth, and a failover for the GeoIP setup so that at some capacity
you fail over into a different zone. So, if S. America only has two
mirrors, it will start passing requests to N.America only once the
`capacity' is used up.
In addition to dropping most of my former sysadmin duties to work on
this, I'll be hopefully passing some tasks off to a volunteer student
who wanted to help with some OSL development project.
I'll hopefully start using the Trac site more too: posting roadmaps,
creating docs in the wiki.
This will likely happen as a rewrite of the v1 sentry. The codebase
is smaller and more in the spirit of how this version would work than
the v2 sentry.py application is. The admin utility can be rewritten
from the ground up to conform to the new DB structure with little
effort using CakePHP. Also, I haven't put much thought into how the
repo should be restructured to account for the `yet-another-fork'
nature of bouncer.
OSU Open Source Lab
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 186 bytes
Desc: This is a digitally signed message part
Url : http://lists.osuosl.org/pipermail/bouncer-dev/attachments/20080612/36585526/attachment.pgp
More information about the bouncer-dev