[Pydra] Return values of ParallelTask, etc.

Yin QIU allenchue at gmail.com
Fri May 29 18:23:01 UTC 2009


On Sat, May 30, 2009 at 2:02 AM, Peter Krenesky <peter at osuosl.org> wrote:
>
> When a task finishes here is the flow of calls:
>
> 1)  This is how the master is notified that the worker finished the task.
>
>    Task -> Worker.work_complete(...) - >Master.send_results(...)
>
>
> 2) At this point the master has the results.  It will handle anything
> internally such as scheduling.  If it was a subtask then the Main-Worker
> is notified
>
>    Master -> Worker.receive_results(..) ->
> ParallelTask._work_unit_complete(...)
>
>
> When the workunit is assigned it can't be a blocking call.  It returns
> right away, with at most a confirmation that the workunit was assigned.
> The workunit may take a long time to finish, so it handles
> notification.  There is fault tolerance in both the Master and Worker to
> ensure that if either goes away the workunit is never dropped.
>

Oops! I should have identified the function call flow myself. Anyway
thanks for your fast response.

>
> Theres a few issues here that i know need to be worked on, but I've been
> putting off for other things because they are mostly coding style changes.
>
>    1) The naming convention - its not very clear which functions call what.
>    2) files need to be organized and modularized so they are easier to
> deal with.  Master.py has grown very large and functions should be
> grouped into smaller components.

Actually I am trying to separate the scheduling logic to another file
named sched.py. Hope this helps modularization.

>    2) There are additional functions that deal with sending Failures
> and Returning work from disconnected Workers.  The results passing calls
> need to be unified, and reworked to expect a list of results.  This is a
> requirement for improved slicing which will result in lists of workunits.
>
> -Peter
> _______________________________________________
> Pydra mailing list
> Pydra at osuosl.org
> http://lists.osuosl.org/mailman/listinfo/pydra
>



-- 
Yin Qiu
Nanjing University, China


More information about the Pydra mailing list