Skip to content

Exporter stuck in AfterLeaseHookFailed status after hook failure with endLease #870

Description

@evakhoni

Description

When an afterLease hook fails with on_failure: endLease or encounters an unexpected error, the exporter gets permanently stuck in AfterLeaseHookFailed status and cannot accept new leases.

Reproduction

  1. Configure an exporter with an afterLease hook that fails with on_failure: endLease:
hooks:
  afterLease:
    script: "exit 1"
    timeout: 10
    onFailure: endLease
  1. Create a lease and wait for it to complete
  2. The afterLease hook fails (expected)
  3. Check exporter status:
kubectl get exporters.jumpstarter.dev -n <namespace>

Expected: Exporter transitions AfterLeaseHookFailedAvailable

Actual: Exporter stays in AfterLeaseHookFailed permanently. Subsequent lease attempts show "none are ready (still cleaning up previous lease)"

Root Cause

The run_after_lease_hook() finally block calls request_lease_release(), which sends AVAILABLE status to the controller via gRPC but doesn't update the exporter's internal _exporter_status field. The serve() loop clears _lease_context correctly, but the stale internal status prevents new lease assignments.

The happy path and on_failure: warn paths work correctly because they call report_status(AVAILABLE, ...) in the try block. The on_failure: endLease and unexpected error paths only report AFTER_LEASE_HOOK_FAILED with no subsequent transition.

Impact

  • Exporters become unusable after first hook failure with endLease
  • Requires exporter restart to recover
  • on_failure: exit is not affected (correctly shuts down)
  • on_failure: warn is not affected (reports AVAILABLE in try block)

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions