[Pydra] Return values of ParallelTask, etc.
allenchue at gmail.com
Fri May 29 18:23:01 UTC 2009
On Sat, May 30, 2009 at 2:02 AM, Peter Krenesky <peter at osuosl.org> wrote:
> When a task finishes here is the flow of calls:
> 1) This is how the master is notified that the worker finished the task.
> Task -> Worker.work_complete(...) - >Master.send_results(...)
> 2) At this point the master has the results. It will handle anything
> internally such as scheduling. If it was a subtask then the Main-Worker
> is notified
> Master -> Worker.receive_results(..) ->
> When the workunit is assigned it can't be a blocking call. It returns
> right away, with at most a confirmation that the workunit was assigned.
> The workunit may take a long time to finish, so it handles
> notification. There is fault tolerance in both the Master and Worker to
> ensure that if either goes away the workunit is never dropped.
Oops! I should have identified the function call flow myself. Anyway
thanks for your fast response.
> Theres a few issues here that i know need to be worked on, but I've been
> putting off for other things because they are mostly coding style changes.
> 1) The naming convention - its not very clear which functions call what.
> 2) files need to be organized and modularized so they are easier to
> deal with. Master.py has grown very large and functions should be
> grouped into smaller components.
Actually I am trying to separate the scheduling logic to another file
named sched.py. Hope this helps modularization.
> 2) There are additional functions that deal with sending Failures
> and Returning work from disconnected Workers. The results passing calls
> need to be unified, and reworked to expect a list of results. This is a
> requirement for improved slicing which will result in lists of workunits.
> Pydra mailing list
> Pydra at osuosl.org
Nanjing University, China
More information about the Pydra