Adding new choice to --on-error - #1974

Open

AlexTate wants to merge 30 commits into

common-workflow-language:mainfrom

AlexTate:on-error-abort

Commits on Nov 13, 2024

Adding inputs for new "kill" mode of the on-error parameter

AlexTate
authored and
mr-c
committed
572e865
Adding kill switch trigger. If a job's processStatus != "success" and runtimeContext.on_error = "kill", then the switch is activated. WorkflowKillSwitch is raised so it can be handled at the workfl…

AlexTate
authored and
mr-c
committed
d7b7712
Actively running jobs respond to the kill switch by checking the switch's status in the monitor function. The monitor function, up to this point, has been for gathering memory usage statistics via …

AlexTate
authored and
mr-c
committed
788a89d
When the WorkflowKillSwitch exception reaches the TaskQueue, try to prevent pending tasks from starting by simply draining the queue. This is a very loose policy, but since kill switch response is …

AlexTate
authored and
mr-c
committed
de73938
Propagation of the WorkflowKillSwitch exception stops once it reaches an executor. The workflow_eval_lock release had to be moved to the finally block in MultithreadedJobExecutor.run_jobs(). Otherw…

AlexTate
authored and
mr-c
committed
a9a2ab6
Don't create runtime_context.kill_switch by default
mr-c
committed
0a00582
This commit brings kill switch initialization and monitoring to the TaskQueue. This helps to better synchronize the kill switch event and avoid adding/executing tasks after the switch has been set.

AlexTate
authored and
mr-c
committed
beaa3f4
JobBase._execute() previously skipped some important post-job actions when exiting due to kill switch. Those actions have been placed under a `finally` block so that they are executed by both the "…

AlexTate
authored and
mr-c
committed
b52333d
Adding a unit test. Two outcomes are measured:

AlexTate
authored and
mr-c
committed
a1ab910
Formatting corrections provided by `make cleanup` which are relevant to this issue. Other changes were offered by the tool, but they are outside the scope of this issue.

AlexTate
authored and
mr-c
committed
5924cfe
cleanups
mr-c
committed
1bee55f
DO NOT MERGE, let all tests run
mr-c
committed
7eb2c0d
Removing irrelevant debug line from test body

AlexTate
authored and
mr-c
committed
03cde29
Removing CPU resource constraints for the test. My intention is to have MultithreadedJobExecutor ignore allocated resources when deciding whether to run the next parallel job. The steps in this wor…

AlexTate
authored and
mr-c
committed
2b9d76a
Using a different approach to remove the "one job per core" resource constraint. The current ResourceRequirement implementation doesn't allow {coresMin: 0}. However, this can still be achieved with…

AlexTate
authored and
mr-c
committed
36f12cf
Adding suggested changes to the --on-error help string per @mr-c. Formatting compliance updates in test_parallel.py

AlexTate
authored and
mr-c
committed
791fe7b
Missed running `make mypy` before pushing last commit. Opting to disable type checking on selectResources() because it's just an implementation detail for the test

AlexTate
authored and
mr-c
committed
93948fb
don't cancel jobs; typecheck the test just in case
mr-c
committed
d938bf8
test_on_error_kill: allow for some longer runtime due to busy systems
mr-c
committed
5d92d2d
Fixing ReceiveScatterOutput to allow outputs to be collected from successful steps when 1) these steps are upstream from a scattered subworkflow, 2) the workflow kill switch is activated by one of …

AlexTate
authored and
mr-c
committed
fe53fb8
Simplifying value checks for RuntimeContext.on_error in parallel_steps() and WorkflowJob.job().

AlexTate
authored and
mr-c
committed
b639dfa
Adding __repr__ to WorkflowStep, WorkflowJob, and WorkflowJobStep classes. This can be VERY helpful while debugging, particularly when unraveling callback chains.

AlexTate
authored and
mr-c
committed
8e0b6f2
It is critical for WorkflowJob.processStatus to update in WorkflowJob.receive_output(). Otherwise, MultithreadedJobExecutor.run_jobs() will likely stop iterating over the topmost WorkflowJob.job() …

AlexTate
authored and
mr-c
committed
32b22d8
Simplifying the approach in on-error_kill.cwl. It now activates the kill switch via a ToolTimeLimit requirement. It also uses a much longer timeout which will hopefully be sufficient for the CI ser…

AlexTate
authored and
mr-c
committed
6485cff
Reformatting and type hint updates

AlexTate
authored and
mr-c
committed
beb6af7
Formatting mistake fix

AlexTate
authored and
mr-c
committed
50f2793
Adding instance ID to repr strings, improving readability of WorkflowStep's repr string, and adding docstrings.

AlexTate
authored and
mr-c
committed
20fd703
Changing TaskQueue constructor to accept the kill_switch threading.Event as an argument rather than an entire RuntimeContext, per @mr-c

AlexTate
authored and
mr-c
committed
e8f9284
codecov: don't fail CI if there is an error
mr-c
committed
f17f678

Commits on Nov 20, 2024

Merge branch 'main' into on-error-abort
mr-c
authored
e139fae

Navigation Menu

Uh oh!

Uh oh!

Adding new choice to --on-error - #1974

Adding new choice to --on-error#1974
AlexTate wants to merge 30 commits into
common-workflow-language:mainfrom
AlexTate:on-error-abort

Commits on Nov 13, 2024

Adding inputs for new "kill" mode of the on-error parameter

Adding kill switch trigger. If a job's processStatus != "success" and runtimeContext.on_error = "kill", then the switch is activated. WorkflowKillSwitch is raised so it can be handled at the workfl…

Actively running jobs respond to the kill switch by checking the switch's status in the monitor function. The monitor function, up to this point, has been for gathering memory usage statistics via …

When the WorkflowKillSwitch exception reaches the TaskQueue, try to prevent pending tasks from starting by simply draining the queue. This is a very loose policy, but since kill switch response is …

Propagation of the WorkflowKillSwitch exception stops once it reaches an executor. The workflow_eval_lock release had to be moved to the finally block in MultithreadedJobExecutor.run_jobs(). Otherw…

Don't create runtime_context.kill_switch by default

This commit brings kill switch initialization and monitoring to the TaskQueue. This helps to better synchronize the kill switch event and avoid adding/executing tasks after the switch has been set.

JobBase._execute() previously skipped some important post-job actions when exiting due to kill switch. Those actions have been placed under a `finally` block so that they are executed by both the "…

Adding a unit test. Two outcomes are measured:

Formatting corrections provided by `make cleanup` which are relevant to this issue. Other changes were offered by the tool, but they are outside the scope of this issue.

cleanups

DO NOT MERGE, let all tests run

Removing irrelevant debug line from test body

Removing CPU resource constraints for the test. My intention is to have MultithreadedJobExecutor ignore allocated resources when deciding whether to run the next parallel job. The steps in this wor…

Using a different approach to remove the "one job per core" resource constraint. The current ResourceRequirement implementation doesn't allow {coresMin: 0}. However, this can still be achieved with…

Adding suggested changes to the --on-error help string per @mr-c. Formatting compliance updates in test_parallel.py

Missed running `make mypy` before pushing last commit. Opting to disable type checking on selectResources() because it's just an implementation detail for the test

don't cancel jobs; typecheck the test just in case

test_on_error_kill: allow for some longer runtime due to busy systems

Fixing ReceiveScatterOutput to allow outputs to be collected from successful steps when 1) these steps are upstream from a scattered subworkflow, 2) the workflow kill switch is activated by one of …

Simplifying value checks for RuntimeContext.on_error in parallel_steps() and WorkflowJob.job().

Adding repr to WorkflowStep, WorkflowJob, and WorkflowJobStep classes. This can be VERY helpful while debugging, particularly when unraveling callback chains.

It is critical for WorkflowJob.processStatus to update in WorkflowJob.receive_output(). Otherwise, MultithreadedJobExecutor.run_jobs() will likely stop iterating over the topmost WorkflowJob.job() …

Simplifying the approach in on-error_kill.cwl. It now activates the kill switch via a ToolTimeLimit requirement. It also uses a much longer timeout which will hopefully be sufficient for the CI ser…

Reformatting and type hint updates

Formatting mistake fix

Adding instance ID to repr strings, improving readability of WorkflowStep's repr string, and adding docstrings.

Changing TaskQueue constructor to accept the kill_switch threading.Event as an argument rather than an entire RuntimeContext, per @mr-c

codecov: don't fail CI if there is an error

Commits on Nov 20, 2024

Merge branch 'main' into on-error-abort