Description
When an afterLease hook fails with on_failure: endLease or encounters an unexpected error, the exporter gets permanently stuck in AfterLeaseHookFailed status and cannot accept new leases.
Reproduction
- Configure an exporter with an afterLease hook that fails with
on_failure: endLease:
hooks:
afterLease:
script: "exit 1"
timeout: 10
onFailure: endLease
- Create a lease and wait for it to complete
- The afterLease hook fails (expected)
- Check exporter status:
kubectl get exporters.jumpstarter.dev -n <namespace>
Expected: Exporter transitions AfterLeaseHookFailed → Available
Actual: Exporter stays in AfterLeaseHookFailed permanently. Subsequent lease attempts show "none are ready (still cleaning up previous lease)"
Root Cause
The run_after_lease_hook() finally block calls request_lease_release(), which sends AVAILABLE status to the controller via gRPC but doesn't update the exporter's internal _exporter_status field. The serve() loop clears _lease_context correctly, but the stale internal status prevents new lease assignments.
The happy path and on_failure: warn paths work correctly because they call report_status(AVAILABLE, ...) in the try block. The on_failure: endLease and unexpected error paths only report AFTER_LEASE_HOOK_FAILED with no subsequent transition.
Impact
- Exporters become unusable after first hook failure with
endLease
- Requires exporter restart to recover
on_failure: exit is not affected (correctly shuts down)
on_failure: warn is not affected (reports AVAILABLE in try block)
Description
When an afterLease hook fails with
on_failure: endLeaseor encounters an unexpected error, the exporter gets permanently stuck inAfterLeaseHookFailedstatus and cannot accept new leases.Reproduction
on_failure: endLease:Expected: Exporter transitions
AfterLeaseHookFailed→AvailableActual: Exporter stays in
AfterLeaseHookFailedpermanently. Subsequent lease attempts show "none are ready (still cleaning up previous lease)"Root Cause
The
run_after_lease_hook()finally block callsrequest_lease_release(), which sendsAVAILABLEstatus to the controller via gRPC but doesn't update the exporter's internal_exporter_statusfield. The serve() loop clears_lease_contextcorrectly, but the stale internal status prevents new lease assignments.The happy path and
on_failure: warnpaths work correctly because they callreport_status(AVAILABLE, ...)in the try block. Theon_failure: endLeaseand unexpected error paths only reportAFTER_LEASE_HOOK_FAILEDwith no subsequent transition.Impact
endLeaseon_failure: exitis not affected (correctly shuts down)on_failure: warnis not affected (reports AVAILABLE in try block)