[gsoc-dev] [Mirror Syncing] Storing parameters when adding new projects; Jobstores & Config files

[Pranjal Mittal]

Hello Lance,

Its been exciting to brainstorm on Mirror Syncing architecture over the
last week. My exams are over now which means that I can spend more time on
the project. I have written a blog post [1] which briefly gives an account
of my work and understanding so far. (Work in progress: updated recently to
add more details)

Some time back we had a discussion on whether to use configuration files or
databases for storing project information. *Scheduling a project* for
syncing periodically, obviously calls for storing the project parameters
like source location, destination, scheduled sync times. Fortunately,
APScheduler already "ties up" with a job store internally to make things
easy for us. Our objective is to specify at the beginning (before starting
the scheduler) a suitable Job Store that will be used among the available
options [2]. This "tie-up" between "apscheduler" and "jobstores" is helpful
for us as we do  not have to worry about installing a db, defining a
database schema, and writing to it as it is handled by the scheduler
itself. (which is a amazing feature in my opinion).

   - I need your suggestions on which Job Store could be ideal for our
   purpose? (Options in [2])
   - Should we be also adding the parameters to config files too (apart
   from jobstores)? Not sure how that will help us?

Till now I have done experiments to add a project. (which just "writes" to
jobstores initially). Need to figure out how to modify and delete as well.
I am also trying to write a fast wrapper over rsync using subprocess [3]
and interface the rsync code with the add_project code. (+ test it live)

 I think we could initially use subprocess wrappers over rsync and later on
 to make things faster, write CFFI wrappers over librsync. (I'll keep that
lower on priority at the moment).

Looking forward to your feedback!


[1]
http://www.pranjalmittal.in/2014/05/google-summer-of-code-2014-mirror.html
[2] http://pythonhosted.org/APScheduler/#job-stores
[3] https://github.com/pramttl/msync-api-poc/blob/master/sync_utilities.py
[4] https://github.com/pramttl/msync-api-poc/


Thanks,
Kind Regards,
- Pranjal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osuosl.org/pipermail/gsoc-dev/attachments/20140511/108f99fa/attachment.html>