[Pydra] map-reduce - parallel reduce stage (and more)

Peter Krenesky peter at osuosl.org
Wed Jul 1 15:50:20 UTC 2009


This email got lost somehow meant it to go out a few days ago....

It looks good.  A few things...

== self.available_workers ==
The concept of pre-assigned workers will go away with the new
scheduler.  The scheduler will either queue requests or pull requests
from the tasks.  This doesn't need to be updated until the scheduler is
finished.


== Task as Mapper or Reducer
My objections to not allowing Task to be used as a Mapper or Reducer
still stand.  It's not going to hold
up merging in the code though because changing this doesn't alter how
the rest of the code works.


== intermediate file ==
Are you planning on replacing this with the input/output helper
interface, once it is built?



More information about the Pydra mailing list