[bouncer-dev] Progress, roadmap

Eric Searcy emsearcy at osuosl.org
Thu Jun 12 21:54:10 UTC 2008

Been a while since this list has seen any activity ... so here's a  
long email to make up for it.

I wanted to post some thoughts on Bouncer direction, and some needs we  
have at the OSL that we're going to start working on.  Most of this  
I've already discussed with Mike Morgan (morgamic).

First off is the idea (actually morgamic's) of bouncer as virtual  
filesystem.  Bouncer can easily get a list files available.  The  
Mozilla fork needs a local copy of the mirror in order to do hash  
checks (since they are partial file, it cannot be stored in the DB  
like v2 does).  Even if a local copy is unavailable (perhaps a certain  
install may only want per-file touch checks), the file list could  
still be built via a one-argument rsync call to the master mirror.   
So, how would this change operation?  Rather than adding new product/ 
version/os/lang tuples, tied to a mirror location that would then be  
checked, the new files can be detected automatically and added.   
Sentry could then automatically start polling remote mirrors to see  
which files match.

For simple mirrors, like Mozilla, where the structure is well  
standardized, this would be all they would need.  The change would  
just be made in the original download chooser on the site to point to  
a file path on the bouncer server instead of passing args:

http://download.server/path/to/file (redirects to) http://some.mirror/base/path/to/file

The idea of having a p/v/os/l tuple that ties to a filename could  
still be implemented, and would be in a sense an extra redirect on the  
bouncer side, /query?args -> /path/to/file.  The advantage with moving  
the template layer `higher' would be that: a) clients like Mozilla  
that don't need it don't have to use it b) the templates don't have to  
be used in sentry runs, just for download bounces---sentry just needs  
to do filesystem-to-filesystem comparisons (think simple).

I've already started sketching out a DB design for this, with an added  
feature (transparent if you don't want it) for multiple projects  
within a single bouncer install.  This basically means that for  
different subdirectories in your filesystem, you can have different  
mirrorsets and p/v/os/l links.  Think using bouncer on a general- 
purpose mirror that hosts lots of projects.  If your bandwidth started  
getting taxed, you could pass off requests to other mirrors.

That `once its taxed, start passing off' idea requires one last  
feature I plan on putting in: tiers (also transparent if you don't use  
them).  The tier system has two purposes: failover to non-primary  
mirrors (like other people's mirrors?) only if you run out of  
bandwidth, and a failover for the GeoIP setup so that at some capacity  
you fail over into a different zone.  So, if S. America only has two  
mirrors, it will start passing requests to N.America only once the  
`capacity' is used up.

In addition to dropping most of my former sysadmin duties to work on  
this, I'll be hopefully passing some tasks off to a volunteer student  
who wanted to help with some OSL development project.

I'll hopefully start using the Trac site more too: posting roadmaps,  
creating docs in the wiki.

This will likely happen as a rewrite of the v1 sentry.  The codebase  
is smaller and more in the spirit of how this version would work than  
the v2 sentry.py application is.  The admin utility can be rewritten  
from the ground up to conform to the new DB structure with little  
effort using CakePHP.  Also, I haven't put much thought into how the  
repo should be restructured to account for the `yet-another-fork'  
nature of bouncer.

Comments, concerns?

Eric Searcy
OSU Open Source Lab

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
Url : http://lists.osuosl.org/pipermail/bouncer-dev/attachments/20080612/36585526/attachment.pgp 

More information about the bouncer-dev mailing list