[Pydra] Updates on task packaging and task sync
peter at osuosl.org
Sun Aug 23 15:14:04 UTC 2009
I'm excited to get this merged in. I hope to get enough of the
remaining node refactor code finished tonight that i can merge it back
into master. I'll be merging before all the fault tolerance is added
because I want to consolidate all the branches (sync, node-refactor, dist).
Yin QIU wrote:
> I just pushed some changes to my public repo. I managed to add
> preliminary support for keeping multiple versions of a task package.
> There are now two folders holding the task code, namely tasks_cache
> and tasks_cache_internal. The former is publicly known and is for
> deployment usage; the latter is used by TaskManager internally and is
> thus hidden from the outside world.
> tasks_cache always contains the latest code. We can either drop files
> to this folder or put contents into it with certain API (not available
> yet). TaskManager keeps monitoring tasks_cache, and if it notices
> updates, copies the latest task code into tasks_cache_internal, where
> it places the code in a subdirectory with the SHA1 hash of the code as
> the directory's name.
> I've performed a simple test against this new feature. I put a
> modified task package while running an older version of the package.
> This resulted in two different task packages in tasks_cache_internal.
> There is currently no cleanup mechanism yet. That is, once a task
> package is created in tasks_cache_internal, there is no automatic way
> to remove it after it expires. This issue will be resolved after we
> let the scheduler emit TASK_STARTED and TASK_STOPPED signals and
> handle these signals in TaskManager.
More information about the Pydra