[Pydra] Return values of ParallelTask, etc.

Peter Krenesky peter at osuosl.org
Fri May 29 18:25:41 UTC 2009


Yin QIU wrote:
> On Sat, May 30, 2009 at 2:02 AM, Peter Krenesky <peter at osuosl.org> wrote:
>   
>> When a task finishes here is the flow of calls:
>>
>> 1)  This is how the master is notified that the worker finished the task.
>>
>>    Task -> Worker.work_complete(...) - >Master.send_results(...)
>>
>>
>> 2) At this point the master has the results.  It will handle anything
>> internally such as scheduling.  If it was a subtask then the Main-Worker
>> is notified
>>
>>    Master -> Worker.receive_results(..) ->
>> ParallelTask._work_unit_complete(...)
>>
>>
>> When the workunit is assigned it can't be a blocking call.  It returns
>> right away, with at most a confirmation that the workunit was assigned.
>> The workunit may take a long time to finish, so it handles
>> notification.  There is fault tolerance in both the Master and Worker to
>> ensure that if either goes away the workunit is never dropped.
>>
>>     
>
> Oops! I should have identified the function call flow myself. Anyway
> thanks for your fast response.
>
>   
>> Theres a few issues here that i know need to be worked on, but I've been
>> putting off for other things because they are mostly coding style changes.
>>
>>    1) The naming convention - its not very clear which functions call what.
>>    2) files need to be organized and modularized so they are easier to
>> deal with.  Master.py has grown very large and functions should be
>> grouped into smaller components.
>>     
>
> Actually I am trying to separate the scheduling logic to another file
> named sched.py. Hope this helps modularization.
>   
Yes, thats good.

>>    2) There are additional functions that deal with sending Failures
>> and Returning work from disconnected Workers.  The results passing calls
>> need to be unified, and reworked to expect a list of results.  This is a
>> requirement for improved slicing which will result in lists of workunits.
>>
>> -Peter
>> _______________________________________________
>> Pydra mailing list
>> Pydra at osuosl.org
>> http://lists.osuosl.org/mailman/listinfo/pydra
>>
>>     
>
>
>
>   



More information about the Pydra mailing list