[gsoc-dev] Re-implementation of mirror-sync Api to provide Asynchronous capabilities

[Pranjal Mittal]

Hi Gorantla,

Thanks for introducing!

Lance and Ken would be able to tell you more about what is required for us
in the mirror syncing project but I would say performance is definitely
important. Klein looks just like Flask for twisted so I am curious about
it. What are the key differences (both pros and cons)? I will have a look
in greater detail and get back to you. Since Master node should be a
multi-core system, ability to handle rsyncs from various upstream sources
asynchronously (in parallel) should increase performance for sure. Some
rsync(s) can take more time depending on the bandwidth available between
upstream sources and master node so it does make sense to have some sort of
timesharing between rsyncs / have each rsync wrapper call [1] run in an
asynchronous manner to increase performance.

I would suggest to discuss or ask questions on IRC (freenode) at
*#osuosl-gsoc* channel for quick help.


P.S:
I assumed you have an understanding of Mirror Syncing terminology, if not
just have a look at how mirroring works for various projects like Fedora,
Debian, etc and also read [2] which is little outdated but will help in
understanding some important terminology and how mirror syncing currently
works at OSL. (We are planning to gradually migrate to [3] after the
project becomes stable and is tested)

[1]
https://github.com/pramttl/mirror-sync-api/blob/develop/utils/syncing.py#L7
[2] http://mirror-sync.osuosl.org/en/latest/
[3] https://github.com/pramttl/mirror-sync-api



On Thu, Nov 13, 2014 at 12:30 AM, Gorantla sai <gorantla.sai5 at gmail.com>
wrote:

> Hello ,
>
>     I'm a Cs Undergraduate student at IIT BHU. I'm interested in
> networking , web scraping and web applications and i use python for most of
> my work . I know Python , JavaScript , C++ , Bash and I'm familiar with
> Unix systems .I like the  Open-source philosophy .  And i would like to
> contribute to OSUOSL .
>
>              I have seen your  *mirror-sync project
> <https://github.com/pramttl/mirror-sync-api>*  which uses Rsync and Flask
> . While rsync have  async I/O capabilities,  Flask  is based on WSGI
> <http://legacy.python.org/dev/peps/pep-0333/)> which is a fundamentally
> synchronous and blocking API . So I would like to *re-implement* the
> mirror-sync project using Klein <https://github.com/twisted/klein>  which
> is based on twisted which gives it *Async* capabilities .
>
>          Any  questions and suggestions are welcome.
>
> Thanks ,
> Gorantla Sai .
>
> _______________________________________________
> gsoc-dev mailing list
> gsoc-dev at lists.osuosl.org
> http://lists.osuosl.org/mailman/listinfo/gsoc-dev
>
>


hth,
Pranjal Mittal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osuosl.org/pipermail/gsoc-dev/attachments/20141113/481f2cb4/attachment.html>