Expunge and Decommission disks in planner (#7286)

This PR partially fixes #6999,  and fixes #7098 and #7502.

A few significant changes are made in this PR:

 1.  Where possible execution steps no longer error out and cause
the executor itself to early return without running all steps. Instead a
`StepWarning` is returned for each step that contains errors and the
executor
runs every step.
2. Some fallible steps have become decoupled in that they do not rely on
each other and can safely run independently. This was mostly true,
except for
the bulk of the work in this PR which ensures that when a a disk is
expunged, it
is not immediately decommissioned unless the sled is also expunged. The
planner
waits to see that the expungement has been observed by a sled-agent on
an active
sled with an expunged disk before it marks it decommissioned.
3. The executor only decommissions physical disks if their disposition
is expunged and they are `ready_for_cleanup` in the blueprint. This
completes the feedback loop
between planning, inventory collection, and execution. Previously disk
decommissioning was done for *all* expunged disks. This was racy in
that a separate nexus may have expunged a disk that was not yet ready
to be decommissioned, and it would be picked up in the decommissioning
step of the first nexus's executor. There is more information in the
following [commit
comments](3ee1546#commitcomment-149432816).
4. A Disposition field have been added to the table output for disks.

Steps 1 and 2 above partially resolve #6999 and resolve #7502. While I
was modifying the datastore
to implement step 3 I also updated the `modified_time` of the physical
disks
table for necessary transactions. This fixes #7502.

This PR also fixes a planner bug where we only check the planning input
for
decommissioned disks even if there was a change in the parent blueprint,
but no
corresponding change to the planning input.

Author: andrewjstone

dev-tools/omdb/tests/successes.out

@@ -1342,19 +1342,19 @@ parent:    <none>
   sled: ..........<REDACTED_UUID>........... (active)
 
     physical disks at generation 2:
-    ---------------------------------------------------
-    vendor        model              serial            
-    ---------------------------------------------------
-    nexus-tests   nexus-test-model   nexus-test-disk-10
-    nexus-tests   nexus-test-model   nexus-test-disk-11
-    nexus-tests   nexus-test-model   nexus-test-disk-12
-    nexus-tests   nexus-test-model   nexus-test-disk-13
-    nexus-tests   nexus-test-model   nexus-test-disk-14
-    nexus-tests   nexus-test-model   nexus-test-disk-15
-    nexus-tests   nexus-test-model   nexus-test-disk-16
-    nexus-tests   nexus-test-model   nexus-test-disk-17
-    nexus-tests   nexus-test-model   nexus-test-disk-18
-    nexus-tests   nexus-test-model   nexus-test-disk-19
+    -----------------------------------------------------------------
+    vendor        model              serial               disposition
+    -----------------------------------------------------------------
+    nexus-tests   nexus-test-model   nexus-test-disk-10   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-11   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-12   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-13   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-14   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-15   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-16   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-17   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-18   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-19   in service 
 
 
     datasets at generation 2:
@@ -1373,19 +1373,19 @@ parent:    <none>
   sled: ..........<REDACTED_UUID>........... (active)
 
     physical disks at generation 2:
-    --------------------------------------------------
-    vendor        model              serial           
-    --------------------------------------------------
-    nexus-tests   nexus-test-model   nexus-test-disk-0
-    nexus-tests   nexus-test-model   nexus-test-disk-1
-    nexus-tests   nexus-test-model   nexus-test-disk-2
-    nexus-tests   nexus-test-model   nexus-test-disk-3
-    nexus-tests   nexus-test-model   nexus-test-disk-4
-    nexus-tests   nexus-test-model   nexus-test-disk-5
-    nexus-tests   nexus-test-model   nexus-test-disk-6
-    nexus-tests   nexus-test-model   nexus-test-disk-7
-    nexus-tests   nexus-test-model   nexus-test-disk-8
-    nexus-tests   nexus-test-model   nexus-test-disk-9
+    ----------------------------------------------------------------
+    vendor        model              serial              disposition
+    ----------------------------------------------------------------
+    nexus-tests   nexus-test-model   nexus-test-disk-0   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-1   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-2   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-3   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-4   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-5   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-6   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-7   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-8   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-9   in service 
 
 
     datasets at generation 2:
@@ -1438,19 +1438,19 @@ parent:    <none>
   sled: ..........<REDACTED_UUID>........... (active)
 
     physical disks at generation 2:
-    ---------------------------------------------------
-    vendor        model              serial            
-    ---------------------------------------------------
-    nexus-tests   nexus-test-model   nexus-test-disk-10
-    nexus-tests   nexus-test-model   nexus-test-disk-11
-    nexus-tests   nexus-test-model   nexus-test-disk-12
-    nexus-tests   nexus-test-model   nexus-test-disk-13
-    nexus-tests   nexus-test-model   nexus-test-disk-14
-    nexus-tests   nexus-test-model   nexus-test-disk-15
-    nexus-tests   nexus-test-model   nexus-test-disk-16
-    nexus-tests   nexus-test-model   nexus-test-disk-17
-    nexus-tests   nexus-test-model   nexus-test-disk-18
-    nexus-tests   nexus-test-model   nexus-test-disk-19
+    -----------------------------------------------------------------
+    vendor        model              serial               disposition
+    -----------------------------------------------------------------
+    nexus-tests   nexus-test-model   nexus-test-disk-10   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-11   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-12   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-13   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-14   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-15   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-16   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-17   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-18   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-19   in service 
 
 
     datasets at generation 2:
@@ -1469,19 +1469,19 @@ parent:    <none>
   sled: ..........<REDACTED_UUID>........... (active)
 
     physical disks at generation 2:
-    --------------------------------------------------
-    vendor        model              serial           
-    --------------------------------------------------
-    nexus-tests   nexus-test-model   nexus-test-disk-0
-    nexus-tests   nexus-test-model   nexus-test-disk-1
-    nexus-tests   nexus-test-model   nexus-test-disk-2
-    nexus-tests   nexus-test-model   nexus-test-disk-3
-    nexus-tests   nexus-test-model   nexus-test-disk-4
-    nexus-tests   nexus-test-model   nexus-test-disk-5
-    nexus-tests   nexus-test-model   nexus-test-disk-6
-    nexus-tests   nexus-test-model   nexus-test-disk-7
-    nexus-tests   nexus-test-model   nexus-test-disk-8
-    nexus-tests   nexus-test-model   nexus-test-disk-9
+    ----------------------------------------------------------------
+    vendor        model              serial              disposition
+    ----------------------------------------------------------------
+    nexus-tests   nexus-test-model   nexus-test-disk-0   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-1   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-2   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-3   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-4   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-5   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-6   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-7   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-8   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-9   in service 
 
 
     datasets at generation 2:
@@ -1536,37 +1536,37 @@ to:   blueprint ......<REDACTED_BLUEPRINT_ID>.......
   sled ..........<REDACTED_UUID>........... (active):
 
     physical disks at generation 2:
-    ---------------------------------------------------
-    vendor        model              serial            
-    ---------------------------------------------------
-    nexus-tests   nexus-test-model   nexus-test-disk-10
-    nexus-tests   nexus-test-model   nexus-test-disk-11
-    nexus-tests   nexus-test-model   nexus-test-disk-12
-    nexus-tests   nexus-test-model   nexus-test-disk-13
-    nexus-tests   nexus-test-model   nexus-test-disk-14
-    nexus-tests   nexus-test-model   nexus-test-disk-15
-    nexus-tests   nexus-test-model   nexus-test-disk-16
-    nexus-tests   nexus-test-model   nexus-test-disk-17
-    nexus-tests   nexus-test-model   nexus-test-disk-18
-    nexus-tests   nexus-test-model   nexus-test-disk-19
+    -----------------------------------------------------------------
+    vendor        model              serial               disposition
+    -----------------------------------------------------------------
+    nexus-tests   nexus-test-model   nexus-test-disk-10   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-11   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-12   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-13   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-14   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-15   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-16   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-17   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-18   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-19   in service 
 
 
   sled ..........<REDACTED_UUID>........... (active):
 
     physical disks at generation 2:
-    --------------------------------------------------
-    vendor        model              serial           
-    --------------------------------------------------
-    nexus-tests   nexus-test-model   nexus-test-disk-0
-    nexus-tests   nexus-test-model   nexus-test-disk-1
-    nexus-tests   nexus-test-model   nexus-test-disk-2
-    nexus-tests   nexus-test-model   nexus-test-disk-3
-    nexus-tests   nexus-test-model   nexus-test-disk-4
-    nexus-tests   nexus-test-model   nexus-test-disk-5
-    nexus-tests   nexus-test-model   nexus-test-disk-6
-    nexus-tests   nexus-test-model   nexus-test-disk-7
-    nexus-tests   nexus-test-model   nexus-test-disk-8
-    nexus-tests   nexus-test-model   nexus-test-disk-9
+    ----------------------------------------------------------------
+    vendor        model              serial              disposition
+    ----------------------------------------------------------------
+    nexus-tests   nexus-test-model   nexus-test-disk-0   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-1   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-2   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-3   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-4   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-5   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-6   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-7   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-8   in service 
+    nexus-tests   nexus-test-model   nexus-test-disk-9   in service 
 
 
     datasets at generation 2:

dev-tools/reconfigurator-cli/tests/output/cmd-example-stdout

@@ -69,19 +69,19 @@ parent:    02697f74-b14a-4418-90f0-c28b2a3a6aa9
   sled: 2eb69596-f081-4e2d-9425-9994926e0832 (active)
 
     physical disks at generation 2:
-    ----------------------------------------------------------------------
-    vendor        model        serial                                     
-    ----------------------------------------------------------------------
-    fake-vendor   fake-model   serial-088ed702-551e-453b-80d7-57700372a844
-    fake-vendor   fake-model   serial-09e51697-abad-47c0-a193-eaf74bc5d3cd
-    fake-vendor   fake-model   serial-3a512d49-edbe-47f3-8d0b-6051bfdc4044
-    fake-vendor   fake-model   serial-40517680-aa77-413c-bcf4-b9041dcf6612
-    fake-vendor   fake-model   serial-78d3cb96-9295-4644-bf78-2e32191c71f9
-    fake-vendor   fake-model   serial-853595e7-77da-404e-bc35-aba77478d55c
-    fake-vendor   fake-model   serial-8926e0e7-65d9-4e2e-ac6d-f1298af81ef1
-    fake-vendor   fake-model   serial-9c0b9151-17f3-4857-94cc-b5bfcd402326
-    fake-vendor   fake-model   serial-d61354fa-48d2-47c6-90bf-546e3ed1708b
-    fake-vendor   fake-model   serial-d792c8cb-7490-40cb-bb1c-d4917242edf4
+    ------------------------------------------------------------------------------------
+    vendor        model        serial                                        disposition
+    ------------------------------------------------------------------------------------
+    fake-vendor   fake-model   serial-088ed702-551e-453b-80d7-57700372a844   in service 
+    fake-vendor   fake-model   serial-09e51697-abad-47c0-a193-eaf74bc5d3cd   in service 
+    fake-vendor   fake-model   serial-3a512d49-edbe-47f3-8d0b-6051bfdc4044   in service 
+    fake-vendor   fake-model   serial-40517680-aa77-413c-bcf4-b9041dcf6612   in service 
+    fake-vendor   fake-model   serial-78d3cb96-9295-4644-bf78-2e32191c71f9   in service 
+    fake-vendor   fake-model   serial-853595e7-77da-404e-bc35-aba77478d55c   in service 
+    fake-vendor   fake-model   serial-8926e0e7-65d9-4e2e-ac6d-f1298af81ef1   in service 
+    fake-vendor   fake-model   serial-9c0b9151-17f3-4857-94cc-b5bfcd402326   in service 
+    fake-vendor   fake-model   serial-d61354fa-48d2-47c6-90bf-546e3ed1708b   in service 
+    fake-vendor   fake-model   serial-d792c8cb-7490-40cb-bb1c-d4917242edf4   in service 
 
 
     datasets at generation 2:
@@ -165,19 +165,19 @@ parent:    02697f74-b14a-4418-90f0-c28b2a3a6aa9
   sled: 32d8d836-4d8a-4e54-8fa9-f31d79c42646 (active)
 
     physical disks at generation 2:
-    ----------------------------------------------------------------------
-    vendor        model        serial                                     
-    ----------------------------------------------------------------------
-    fake-vendor   fake-model   serial-128b0f04-229b-48dc-9c5c-555cb5723ed8
-    fake-vendor   fake-model   serial-43ae0f4e-b0cf-4d74-8636-df0567ba01e6
-    fake-vendor   fake-model   serial-4e9806d0-41cd-48c2-86ef-7f815c3ce3b1
-    fake-vendor   fake-model   serial-70bb6d98-111f-4015-9d97-9ef1b2d6dcac
-    fake-vendor   fake-model   serial-7ce5029f-703c-4c08-8164-9af9cf1acf23
-    fake-vendor   fake-model   serial-b113c11f-44e6-4fb4-a56e-1d91bd652faf
-    fake-vendor   fake-model   serial-bf149c80-2498-481c-9989-6344da914081
-    fake-vendor   fake-model   serial-c69b6237-09f9-45aa-962c-5dbdd1d894be
-    fake-vendor   fake-model   serial-ccd5a87b-00ae-42ad-85da-b37d70436cb1
-    fake-vendor   fake-model   serial-d7410a1c-e01d-49a4-be9c-f861f086760a
+    ------------------------------------------------------------------------------------
+    vendor        model        serial                                        disposition
+    ------------------------------------------------------------------------------------
+    fake-vendor   fake-model   serial-128b0f04-229b-48dc-9c5c-555cb5723ed8   in service 
+    fake-vendor   fake-model   serial-43ae0f4e-b0cf-4d74-8636-df0567ba01e6   in service 
+    fake-vendor   fake-model   serial-4e9806d0-41cd-48c2-86ef-7f815c3ce3b1   in service 
+    fake-vendor   fake-model   serial-70bb6d98-111f-4015-9d97-9ef1b2d6dcac   in service 
+    fake-vendor   fake-model   serial-7ce5029f-703c-4c08-8164-9af9cf1acf23   in service 
+    fake-vendor   fake-model   serial-b113c11f-44e6-4fb4-a56e-1d91bd652faf   in service 
+    fake-vendor   fake-model   serial-bf149c80-2498-481c-9989-6344da914081   in service 
+    fake-vendor   fake-model   serial-c69b6237-09f9-45aa-962c-5dbdd1d894be   in service 
+    fake-vendor   fake-model   serial-ccd5a87b-00ae-42ad-85da-b37d70436cb1   in service 
+    fake-vendor   fake-model   serial-d7410a1c-e01d-49a4-be9c-f861f086760a   in service 
 
 
     datasets at generation 2:
@@ -258,19 +258,19 @@ parent:    02697f74-b14a-4418-90f0-c28b2a3a6aa9
   sled: 89d02b1b-478c-401a-8e28-7a26f74fa41b (active)
 
     physical disks at generation 2:
-    ----------------------------------------------------------------------
-    vendor        model        serial                                     
-    ----------------------------------------------------------------------
-    fake-vendor   fake-model   serial-44fa7024-c2bc-4d2c-b478-c4997e4aece8
-    fake-vendor   fake-model   serial-5265edc6-debf-4687-a758-a9746893ebd3
-    fake-vendor   fake-model   serial-532fbd69-b472-4445-86af-4c4c85afb313
-    fake-vendor   fake-model   serial-54fd6fa6-ce3c-4abe-8c9d-7e107e159e84
-    fake-vendor   fake-model   serial-8562317c-4736-4cfc-9292-7dcab96a6fee
-    fake-vendor   fake-model   serial-9a1327e4-d11b-4d98-8454-8c41862e9832
-    fake-vendor   fake-model   serial-bf9d6692-64bc-459a-87dd-e7a83080a210
-    fake-vendor   fake-model   serial-ce1c13f3-bef2-4306-b0f2-4e39bd4a18b6
-    fake-vendor   fake-model   serial-f931ec80-a3e3-4adb-a8ba-fa5adbd2294c
-    fake-vendor   fake-model   serial-fe1d5b9f-8db7-4e2d-bf17-c4b80e1f897c
+    ------------------------------------------------------------------------------------
+    vendor        model        serial                                        disposition
+    ------------------------------------------------------------------------------------
+    fake-vendor   fake-model   serial-44fa7024-c2bc-4d2c-b478-c4997e4aece8   in service 
+    fake-vendor   fake-model   serial-5265edc6-debf-4687-a758-a9746893ebd3   in service 
+    fake-vendor   fake-model   serial-532fbd69-b472-4445-86af-4c4c85afb313   in service 
+    fake-vendor   fake-model   serial-54fd6fa6-ce3c-4abe-8c9d-7e107e159e84   in service 
+    fake-vendor   fake-model   serial-8562317c-4736-4cfc-9292-7dcab96a6fee   in service 
+    fake-vendor   fake-model   serial-9a1327e4-d11b-4d98-8454-8c41862e9832   in service 
+    fake-vendor   fake-model   serial-bf9d6692-64bc-459a-87dd-e7a83080a210   in service 
+    fake-vendor   fake-model   serial-ce1c13f3-bef2-4306-b0f2-4e39bd4a18b6   in service 
+    fake-vendor   fake-model   serial-f931ec80-a3e3-4adb-a8ba-fa5adbd2294c   in service 
+    fake-vendor   fake-model   serial-fe1d5b9f-8db7-4e2d-bf17-c4b80e1f897c   in service 
 
 
     datasets at generation 2:
@@ -413,13 +413,13 @@ parent:    02697f74-b14a-4418-90f0-c28b2a3a6aa9
   sled: 89d02b1b-478c-401a-8e28-7a26f74fa41b (active)
 
     physical disks at generation 2:
-    ----------------------------------------------------------------------
-    vendor        model        serial                                     
-    ----------------------------------------------------------------------
-    fake-vendor   fake-model   serial-44fa7024-c2bc-4d2c-b478-c4997e4aece8
-    fake-vendor   fake-model   serial-8562317c-4736-4cfc-9292-7dcab96a6fee
-    fake-vendor   fake-model   serial-ce1c13f3-bef2-4306-b0f2-4e39bd4a18b6
-    fake-vendor   fake-model   serial-f931ec80-a3e3-4adb-a8ba-fa5adbd2294c
+    ------------------------------------------------------------------------------------
+    vendor        model        serial                                        disposition
+    ------------------------------------------------------------------------------------
+    fake-vendor   fake-model   serial-44fa7024-c2bc-4d2c-b478-c4997e4aece8   in service 
+    fake-vendor   fake-model   serial-8562317c-4736-4cfc-9292-7dcab96a6fee   in service 
+    fake-vendor   fake-model   serial-ce1c13f3-bef2-4306-b0f2-4e39bd4a18b6   in service 
+    fake-vendor   fake-model   serial-f931ec80-a3e3-4adb-a8ba-fa5adbd2294c   in service 
 
 
     datasets at generation 2: