[Pydra] Updates on task packaging and task sync

Yin QIU qiuyin at gmail.com
Sun Aug 23 04:15:01 UTC 2009


I just pushed some changes to my public repo. I managed to add
preliminary support for keeping multiple versions of a task package.

There are now two folders holding the task code, namely tasks_cache
and tasks_cache_internal. The former is publicly known and is for
deployment usage; the latter is used by TaskManager internally and is
thus hidden from the outside world.

tasks_cache always contains the latest code. We can either drop files
to this folder or put contents into it with certain API (not available
yet). TaskManager keeps monitoring tasks_cache, and if it notices
updates, copies the latest task code into tasks_cache_internal, where
it places the code in a subdirectory with the SHA1 hash of the code as
the directory's name.

I've performed a simple test against this new feature. I put a
modified task package while running an older version of the package.
This resulted in two different task packages in tasks_cache_internal.

There is currently no cleanup mechanism yet. That is, once a task
package is created in tasks_cache_internal, there is no automatic way
to remove it after it expires. This issue will be resolved after we
let the scheduler emit TASK_STARTED and TASK_STOPPED signals and
handle these signals in TaskManager.

Nanjing University, China

More information about the Pydra mailing list