From emsearcy at osuosl.org Thu Jun 12 21:54:10 2008 From: emsearcy at osuosl.org (Eric Searcy) Date: Thu, 12 Jun 2008 14:54:10 -0700 Subject: [bouncer-dev] Progress, roadmap Message-ID: Been a while since this list has seen any activity ... so here's a long email to make up for it. I wanted to post some thoughts on Bouncer direction, and some needs we have at the OSL that we're going to start working on. Most of this I've already discussed with Mike Morgan (morgamic). First off is the idea (actually morgamic's) of bouncer as virtual filesystem. Bouncer can easily get a list files available. The Mozilla fork needs a local copy of the mirror in order to do hash checks (since they are partial file, it cannot be stored in the DB like v2 does). Even if a local copy is unavailable (perhaps a certain install may only want per-file touch checks), the file list could still be built via a one-argument rsync call to the master mirror. So, how would this change operation? Rather than adding new product/ version/os/lang tuples, tied to a mirror location that would then be checked, the new files can be detected automatically and added. Sentry could then automatically start polling remote mirrors to see which files match. For simple mirrors, like Mozilla, where the structure is well standardized, this would be all they would need. The change would just be made in the original download chooser on the site to point to a file path on the bouncer server instead of passing args: http://download.server/path/to/file (redirects to) http://some.mirror/base/path/to/file The idea of having a p/v/os/l tuple that ties to a filename could still be implemented, and would be in a sense an extra redirect on the bouncer side, /query?args -> /path/to/file. The advantage with moving the template layer `higher' would be that: a) clients like Mozilla that don't need it don't have to use it b) the templates don't have to be used in sentry runs, just for download bounces---sentry just needs to do filesystem-to-filesystem comparisons (think simple). I've already started sketching out a DB design for this, with an added feature (transparent if you don't want it) for multiple projects within a single bouncer install. This basically means that for different subdirectories in your filesystem, you can have different mirrorsets and p/v/os/l links. Think using bouncer on a general- purpose mirror that hosts lots of projects. If your bandwidth started getting taxed, you could pass off requests to other mirrors. That `once its taxed, start passing off' idea requires one last feature I plan on putting in: tiers (also transparent if you don't use them). The tier system has two purposes: failover to non-primary mirrors (like other people's mirrors?) only if you run out of bandwidth, and a failover for the GeoIP setup so that at some capacity you fail over into a different zone. So, if S. America only has two mirrors, it will start passing requests to N.America only once the `capacity' is used up. In addition to dropping most of my former sysadmin duties to work on this, I'll be hopefully passing some tasks off to a volunteer student who wanted to help with some OSL development project. I'll hopefully start using the Trac site more too: posting roadmaps, creating docs in the wiki. This will likely happen as a rewrite of the v1 sentry. The codebase is smaller and more in the spirit of how this version would work than the v2 sentry.py application is. The admin utility can be rewritten from the ground up to conform to the new DB structure with little effort using CakePHP. Also, I haven't put much thought into how the repo should be restructured to account for the `yet-another-fork' nature of bouncer. Comments, concerns? -- Eric Searcy OSU Open Source Lab -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 186 bytes Desc: This is a digitally signed message part Url : http://lists.osuosl.org/pipermail/bouncer-dev/attachments/20080612/36585526/attachment.pgp