[gsoc-dev] FTP Mirror syncing architecture: Questions and suggestions

[Lance Albertson]

On Sun, Mar 16, 2014 at 4:28 PM, Pranjal Mittal <mittal.pranjal at gmail.com>wrote:

> *Have some questions (Q)  & suggestions (S)*
>
> (Q) For every project that we add to the master we are creating a new unix
> user for that project same as the project name? How is this helpful to us?
>

No, this only really happens if we're acting as a master for said
project. Normally a regular mirrored project does not require a user.
Sometimes this isn't the case however, but generally that's the rule of
thumb.

However, that being said, having a user isn't necessary if the app is
written with a good API. We only created users so that we gave them more
control on syncing in an easy way since we didn't have an API. Perhaps a
better focused feature for this might be better. Basically we'd like to
give people who we're acting as a master for more control on how they sync.
They mainly need to be able to do the following:

   1. ability to manually trigger a sync
   2. set where the sync comes from (rsync configuration)
   3. optionally do other setups to prep their repo (this rarely is a
   requirement, would need to look for an example of this)


> (Q) What does the *trigger-$project script *do? I remember that we use
> trigger-set to set a trigger on master so that slaves can notice and
> initiate a pull-sync, but what does trigger-$project do? Is it used to
> manually tell slaves to pull from master?
>

Its just a wrapper script that does "trigger-set $project". In the case
of being a master mirror, after they do a manual sync, it tells the slaves
to initiate a sync of the master.


> (S) I think we might want to avoid doing so many steps [2] to add a
> project on the master. These steps could be wrapped into a single command
> through a bash script.
>
> It would be convenient to have to specify a project_name, source_location
> / path, authorized_ssh keys, etc just once while adding a project.
>

I agree. The way I envision it is that we have a backend that has an
exposed API. Then a CLI/Web tool can talk to the API to add/remove/modify
projects. Making it as simple as possible is ideal/


> (S) An interactive web form could be another convenient way to add
> projects?
>

Yes, but I'd like to have the initial focus of this project be on the
backend and CLI tools. If written correctly a web gui should be easy, but
that's sort of a project in of its own. Even a simple ssh trigger would be
neat to implement for something really simple.


> (Q) What are the awstats [1] about and how are they being collected? Most
> of the numbers I see are fixed at 0. Are we collecting these stats yet?
>
> [1] https://awstats.osuosl.org/list/2007
> [2] http://pastebin.osuosl.org/6186/
>

We only run awstats on select projects because of the sheer volume of
traffic these hosts do. Our awstats server would constantly have problems
processing all the logs. Ideally we should have a more scalable solution
to show stats in some form.


> Looking foward to answers/comments!
>

Hopefully that helps. I'm going to try and update the docs some more to
clear up a few things. Thanks for the questions!

-- 
Lance Albertson
Director
Oregon State University | Open Source Lab
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osuosl.org/pipermail/gsoc-dev/attachments/20140317/e8b74833/attachment-0001.html>