openstack/ironic
Code Issues Proposed changes

Add hold steps

Browse Source 
* Updates API version to 1.85 to permit an ``unhold`` verb
* Adds the ``deploy hold`` and ``clean hold`` provision states
  to the internal state machine.
* Adds on documentation on steps to help provide greater clarity
  to Ironic's users on how to utilize steps. It should be noted
  this documentation also includes the power state reserved step
  names from the DPU functionality patch.
* Fixes the state machine diagram. Changes type to PNG as SVG
  rendering is broken due to python libraries utilized for SVG
  generation which do not work on more recent Python versions.

Change-Id: I34f58f4e77e7757b89247fd64f5fcde26f679453
This commit is contained in:
Julia Kreger 2023-03-30 09:07:40 -07:00
parent f69e9da1d0
commit c4e3100d5c
24 changed files with 540 additions and 535 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
doc/source/admin/index.rst
View File

@ -58,6 +58,7 @@ Advanced Topics
Tuning Ironic <tuning> Tuning Ironic <tuning>
Role Based Access Control <secure-rbac> Role Based Access Control <secure-rbac>
Deploying with Anaconda <anaconda-deploy-interface> Deploying with Anaconda <anaconda-deploy-interface>
Steps <steps>
.. toctree:: .. toctree::
:hidden: :hidden:

94
doc/source/admin/steps.rst Normal file
View File

@ -0,0 +1,94 @@
=====
Steps
=====
What are steps?
===============
Steps are exactly that, steps to achieve a goal, and in most cases, they
are what an operator requested.
However, originally they were the internal list of actions to achieve to
perform *automated cleaning*. The conductor would determine a list of
steps or actions to take by generating a list of steps from data the
conductor via drivers, the ``ironic-python-agent``, and any loaded
hardware managers determined to be needed.
As time passed and Ironic's capabilities were extended, this was extended
to *manual cleaning*, and later into *deploy steps*, and *deploy templates*
allowing an operator to request for firmware to be updated by a driver, or
RAID to be configured by the agent prior to the machine being released
to the end user for use.
Reserved Functional Steps
=========================
In the execution of the cleaning, and deployment steps frameworks, some step
names are reserved for specific functions which can be invoked by a user to
perform specific actions.
+-----------+----------------------------------------------------------+
| Step Name | Description |
+-----------+----------------------------------------------------------+
| hold | Pauses the execution of the steps by moving the node |
| | from the current ``deploy wait`` or ``clean wait`` state |
| | to the appropriate "hold" state, such as ``deploy hold`` |
| | or ``clean hold``. The process can be resumed by sending |
| | a ``unhold`` verb to the provision state API endpoint |
| | which will result in the process resuming upon the next |
| | heartbeat operation. During this time, heartbeat |
| | operations will continue be recorded by Ironic, but will |
| | not be acted upon, preventing the node from timing out. |
| | |
| | This step cannot be used against a child node in the |
| | context of being requested when executing against a |
| | parent node. |
| | |
| | The use case for this verb is if you have external |
| | automation or processes which need to be executed in the |
| | entire process to achieve the overall goal. |
+-----------+----------------------------------------------------------+
| power_on | Powers on the node, which may be useful if a node's |
| | power must be toggled multiple times to enable |
| | embedded behavior such as to boot from network. |
| | This step can be executed against child nodes. |
+-----------+----------------------------------------------------------+
| power_off | Turn the node power off via the conductor. |
| | This step can be used against child nodes. When used |
| | outside of the context of a child node, any agent token |
| | metadata is also removed as so the machine can reboot |
| | back to the agent, if applicable. |
+-----------+----------------------------------------------------------+
| reboot | Reboot the node utilizing the conductor. This generally |
| | signals for power to be turned off and back on, however |
| | driver specific code may request an CPU interrupt based |
| | reset. This step can be executed on child nodes. |
+-----------+----------------------------------------------------------+
In the these cases, the interface upon which the method is expected is
ignored, and the step is acted upon based upon just the step's name.
Example
-------
In this example, we utilize the cleaning step ``erase_devices`` and then
trigger hold of the node. In this specific case the node will enter
a ``clean hold`` state.
.. code-block:: json
{
"target":"clean",
"clean_steps": [{
"interface": "deploy",
"step": "erase_devices"
},
{
"interface": "deploy",
"step": "hold"
}]
}
Once you have completed whatever action which needed to be performed while
the node was in a held state, you will need to issue an unhold provision
state command, via the API or command line to inform the node to proceed.

9
doc/source/contributor/webapi-version-history.rst
View File

@ -2,6 +2,15 @@
REST API Version History REST API Version History
======================== ========================
1.85 (Bobcat)
-------------
This version adds a new provision state change verb called ``unhold``
to be utilized with the new ``provision_state`` values ``clean hold``
and ``deploy hold``. The verb instructs Ironic to remove the node
from it's present hold and to resume it's prior cleaning or
deployment process.
1.84 (Bobcat) 1.84 (Bobcat)
------------- -------------

BIN
doc/source/images/states.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 531 KiB

511
doc/source/images/states.svg
View File

@ -1,511 +0,0 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by graphviz version 2.38.0 (20140413.2041)
-->
<!-- Title: Ironic states Pages: 1 -->
<svg width="2609pt" height="855pt"
viewBox="0.00 0.00 2609.29 855.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 851)">
<title>Ironic states</title>
<polygon fill="white" stroke="none" points="-4,4 -4,-851 2605.29,-851 2605.29,4 -4,4"/>
<!-- enroll -->
<g id="node1" class="node"><title>enroll</title>
<ellipse fill="none" stroke="black" stroke-width="1.7" cx="27" cy="-237" rx="27" ry="18"/>
<text text-anchor="middle" x="27" y="-234.2" font-family="Times,serif" font-size="11.00">enroll</text>
</g>
<!-- verifying -->
<g id="node2" class="node"><title>verifying</title>
<ellipse fill="none" stroke="black" cx="208.675" cy="-237" rx="33.8507" ry="18"/>
<text text-anchor="middle" x="208.675" y="-234.2" font-family="Times,serif" font-size="11.00" fill="gray">verifying</text>
</g>
<!-- enroll&#45;&gt;verifying -->
<g id="edge1" class="edge"><title>enroll&#45;&gt;verifying</title>
<path fill="none" stroke="black" d="M54.1263,-237C83.0573,-237 130.15,-237 164.565,-237"/>
<polygon fill="black" stroke="black" points="164.805,-240.5 174.805,-237 164.805,-233.5 164.805,-240.5"/>
<text text-anchor="middle" x="114.5" y="-240.4" font-family="Times,serif" font-size="12.00">manage (via API)</text>
</g>
<!-- verifying&#45;&gt;enroll -->
<g id="edge17" class="edge"><title>verifying&#45;&gt;enroll</title>
<path fill="none" stroke="black" d="M181.26,-226.345C173.566,-223.736 165.059,-221.316 157,-220 119.716,-213.913 109.138,-213.075 72,-220 67.7433,-220.794 63.368,-221.985 59.1046,-223.375"/>
<polygon fill="black" stroke="black" points="57.6647,-220.175 49.4683,-226.888 60.0622,-226.751 57.6647,-220.175"/>
<text text-anchor="middle" x="114.5" y="-223.4" font-family="Times,serif" font-size="12.00" fill="gray">fail</text>
</g>
<!-- manageable -->
<g id="node3" class="node"><title>manageable</title>
<ellipse fill="none" stroke="black" stroke-width="1.7" cx="344.443" cy="-237" rx="42.1875" ry="18"/>
<text text-anchor="middle" x="344.443" y="-234.2" font-family="Times,serif" font-size="11.00">manageable</text>
</g>
<!-- verifying&#45;&gt;manageable -->
<g id="edge16" class="edge"><title>verifying&#45;&gt;manageable</title>
<path fill="none" stroke="black" d="M242.398,-237C257.259,-237 275.195,-237 291.805,-237"/>
<polygon fill="black" stroke="black" points="291.937,-240.5 301.937,-237 291.937,-233.5 291.937,-240.5"/>
<text text-anchor="middle" x="272.35" y="-240.4" font-family="Times,serif" font-size="12.00" fill="gray">done</text>
</g>
<!-- cleaning -->
<g id="node4" class="node"><title>cleaning</title>
<ellipse fill="none" stroke="black" cx="549.037" cy="-290" rx="32.4445" ry="18"/>
<text text-anchor="middle" x="549.037" y="-287.2" font-family="Times,serif" font-size="11.00" fill="gray">cleaning</text>
</g>
<!-- manageable&#45;&gt;cleaning -->
<g id="edge2" class="edge"><title>manageable&#45;&gt;cleaning</title>
<path fill="none" stroke="black" d="M359.543,-253.899C370.458,-265.604 386.688,-280.249 404.537,-287 437.687,-299.537 478.297,-299.069 507.845,-296.178"/>
<polygon fill="black" stroke="black" points="508.252,-299.654 517.805,-295.071 507.479,-292.697 508.252,-299.654"/>
<text text-anchor="middle" x="446.537" y="-300.4" font-family="Times,serif" font-size="12.00">provide (via API)</text>
</g>
<!-- manageable&#45;&gt;cleaning -->
<g id="edge3" class="edge"><title>manageable&#45;&gt;cleaning</title>
<path fill="none" stroke="black" d="M381.682,-245.647C389.232,-247.431 397.142,-249.288 404.537,-251 441.827,-259.634 451.682,-259.667 488.537,-270 496.052,-272.107 504.019,-274.6 511.597,-277.099"/>
<polygon fill="black" stroke="black" points="510.67,-280.48 521.264,-280.355 512.904,-273.846 510.67,-280.48"/>
<text text-anchor="middle" x="446.537" y="-273.4" font-family="Times,serif" font-size="12.00">clean (via API)</text>
</g>
<!-- inspecting -->
<g id="node5" class="node"><title>inspecting</title>
<ellipse fill="none" stroke="black" cx="549.037" cy="-102" rx="37.0671" ry="18"/>
<text text-anchor="middle" x="549.037" y="-99.2" font-family="Times,serif" font-size="11.00" fill="gray">inspecting</text>
</g>
<!-- manageable&#45;&gt;inspecting -->
<g id="edge4" class="edge"><title>manageable&#45;&gt;inspecting</title>
<path fill="none" stroke="black" d="M351.504,-218.864C359.877,-196.636 377.133,-159.415 404.537,-139 408.707,-135.894 464.841,-121.955 505.643,-112.104"/>
<polygon fill="black" stroke="black" points="506.482,-115.502 515.385,-109.759 504.844,-108.696 506.482,-115.502"/>
<text text-anchor="middle" x="446.537" y="-142.4" font-family="Times,serif" font-size="12.00">inspect (via API)</text>
</g>
<!-- adopting -->
<g id="node6" class="node"><title>adopting</title>
<ellipse fill="none" stroke="black" cx="549.037" cy="-438" rx="32.4445" ry="18"/>
<text text-anchor="middle" x="549.037" y="-435.2" font-family="Times,serif" font-size="11.00" fill="gray">adopting</text>
</g>
<!-- manageable&#45;&gt;adopting -->
<g id="edge5" class="edge"><title>manageable&#45;&gt;adopting</title>
<path fill="none" stroke="black" d="M346.454,-255.026C349.374,-289.184 360.7,-363.451 404.537,-403 432.164,-427.924 474.573,-435.839 506.132,-438.018"/>
<polygon fill="black" stroke="black" points="506.213,-441.527 516.382,-438.554 506.578,-434.537 506.213,-441.527"/>
<text text-anchor="middle" x="446.537" y="-439.4" font-family="Times,serif" font-size="12.00">adopt (via API)</text>
</g>
<!-- cleaning&#45;&gt;manageable -->
<g id="edge30" class="edge"><title>cleaning&#45;&gt;manageable</title>
<path fill="none" stroke="black" d="M536.055,-273.182C525.345,-259.722 508.317,-241.844 488.537,-234 457.988,-221.885 420.811,-223.089 392.009,-227.083"/>
<polygon fill="black" stroke="black" points="391.152,-223.674 381.804,-228.661 392.222,-230.592 391.152,-223.674"/>
<text text-anchor="middle" x="446.537" y="-237.4" font-family="Times,serif" font-size="12.00" fill="gray">manage</text>
</g>
<!-- available -->
<g id="node7" class="node"><title>available</title>
<ellipse fill="none" stroke="black" stroke-width="1.7" cx="755.232" cy="-461" rx="34.054" ry="18"/>
<text text-anchor="middle" x="755.232" y="-458.2" font-family="Times,serif" font-size="11.00">available</text>
</g>
<!-- cleaning&#45;&gt;available -->
<g id="edge27" class="edge"><title>cleaning&#45;&gt;available</title>
<path fill="none" stroke="black" d="M562.458,-306.567C573.694,-320.818 591.239,-341.335 609.537,-356 643.392,-383.132 659.691,-378.154 694.537,-404 708.003,-413.988 721.629,-426.752 732.456,-437.669"/>
<polygon fill="black" stroke="black" points="729.986,-440.149 739.468,-444.875 735.003,-435.267 729.986,-440.149"/>
<text text-anchor="middle" x="652.037" y="-407.4" font-family="Times,serif" font-size="12.00" fill="gray">done</text>
</g>
<!-- clean failed -->
<g id="node17" class="node"><title>clean failed</title>
<ellipse fill="none" stroke="black" cx="956.23" cy="-257" rx="41.4846" ry="18"/>
<text text-anchor="middle" x="956.23" y="-254.2" font-family="Times,serif" font-size="11.00" fill="red">clean failed</text>
</g>
<!-- cleaning&#45;&gt;clean failed -->
<g id="edge28" class="edge"><title>cleaning&#45;&gt;clean failed</title>
<path fill="none" stroke="black" d="M581.627,-290.825C610.914,-291.395 655.693,-291.767 694.537,-290 782.539,-285.996 804.985,-286.191 891.927,-272 897.854,-271.033 904.05,-269.82 910.147,-268.506"/>
<polygon fill="black" stroke="black" points="911.268,-271.841 920.251,-266.223 909.725,-265.013 911.268,-271.841"/>
<text text-anchor="middle" x="755.232" y="-292.4" font-family="Times,serif" font-size="12.00" fill="gray">fail</text>
</g>
<!-- clean wait -->
<g id="node18" class="node"><title>clean wait</title>
<ellipse fill="none" stroke="black" cx="755.232" cy="-250" rx="37.7689" ry="18"/>
<text text-anchor="middle" x="755.232" y="-247.2" font-family="Times,serif" font-size="11.00" fill="gray">clean wait</text>
</g>
<!-- cleaning&#45;&gt;clean wait -->
<g id="edge29" class="edge"><title>cleaning&#45;&gt;clean wait</title>
<path fill="none" stroke="black" d="M579.355,-283.408C588.96,-281.298 599.692,-278.994 609.537,-277 642.885,-270.247 680.686,-263.215 709.287,-258.021"/>
<polygon fill="black" stroke="black" points="710.165,-261.419 719.382,-256.194 708.919,-254.53 710.165,-261.419"/>
<text text-anchor="middle" x="652.037" y="-280.4" font-family="Times,serif" font-size="12.00" fill="gray">wait</text>
</g>
<!-- inspecting&#45;&gt;manageable -->
<g id="edge37" class="edge"><title>inspecting&#45;&gt;manageable</title>
<path fill="none" stroke="black" d="M517.555,-92.3444C486.468,-84.4713 438.156,-77.8882 404.537,-100 367.983,-124.042 353.94,-175.998 348.616,-208.567"/>
<polygon fill="black" stroke="black" points="345.103,-208.401 347.125,-218.801 352.03,-209.41 345.103,-208.401"/>
<text text-anchor="middle" x="446.537" y="-103.4" font-family="Times,serif" font-size="12.00" fill="gray">done</text>
</g>
<!-- inspect failed -->
<g id="node19" class="node"><title>inspect failed</title>
<ellipse fill="none" stroke="black" cx="956.23" cy="-76" rx="46.1069" ry="18"/>
<text text-anchor="middle" x="956.23" y="-73.2" font-family="Times,serif" font-size="11.00" fill="red">inspect failed</text>
</g>
<!-- inspecting&#45;&gt;inspect failed -->
<g id="edge38" class="edge"><title>inspecting&#45;&gt;inspect failed</title>
<path fill="none" stroke="black" d="M586.394,-102.893C649.051,-103.948 781.307,-104.054 891.927,-90 896.767,-89.3851 901.782,-88.5771 906.775,-87.659"/>
<polygon fill="black" stroke="black" points="907.801,-91.024 916.929,-85.6465 906.44,-84.1576 907.801,-91.024"/>
<text text-anchor="middle" x="755.232" y="-106.4" font-family="Times,serif" font-size="12.00" fill="gray">fail</text>
</g>
<!-- inspect wait -->
<g id="node20" class="node"><title>inspect wait</title>
<ellipse fill="none" stroke="black" cx="755.232" cy="-18" rx="42.8909" ry="18"/>
<text text-anchor="middle" x="755.232" y="-15.2" font-family="Times,serif" font-size="11.00" fill="gray">inspect wait</text>
</g>
<!-- inspecting&#45;&gt;inspect wait -->
<g id="edge39" class="edge"><title>inspecting&#45;&gt;inspect wait</title>
<path fill="none" stroke="black" d="M564.269,-85.3671C575.396,-73.4368 591.894,-57.948 609.537,-49 626.012,-40.6446 669.636,-31.8943 704.709,-25.8292"/>
<polygon fill="black" stroke="black" points="705.538,-29.2387 714.811,-24.1146 704.367,-22.3374 705.538,-29.2387"/>
<text text-anchor="middle" x="652.037" y="-52.4" font-family="Times,serif" font-size="12.00" fill="gray">wait</text>
</g>
<!-- active -->
<g id="node9" class="node"><title>active</title>
<ellipse fill="none" stroke="black" stroke-width="1.7" cx="1167.64" cy="-559" rx="27" ry="18"/>
<text text-anchor="middle" x="1167.64" y="-556.2" font-family="Times,serif" font-size="11.00">active</text>
</g>
<!-- adopting&#45;&gt;active -->
<g id="edge45" class="edge"><title>adopting&#45;&gt;active</title>
<path fill="none" stroke="black" d="M581.489,-436.996C614.22,-436.006 666.965,-434.561 712.537,-434 750.485,-433.533 760.482,-427.825 797.927,-434 927.7,-455.399 1073.22,-516.489 1135.57,-544.566"/>
<polygon fill="black" stroke="black" points="1134.3,-547.831 1144.85,-548.778 1137.19,-541.457 1134.3,-547.831"/>
<text text-anchor="middle" x="853.927" y="-457.4" font-family="Times,serif" font-size="12.00" fill="gray">done</text>
</g>
<!-- adopt failed -->
<g id="node21" class="node"><title>adopt failed</title>
<ellipse fill="none" stroke="black" cx="755.232" cy="-515" rx="41.4846" ry="18"/>
<text text-anchor="middle" x="755.232" y="-512.2" font-family="Times,serif" font-size="11.00" fill="red">adopt failed</text>
</g>
<!-- adopting&#45;&gt;adopt failed -->
<g id="edge46" class="edge"><title>adopting&#45;&gt;adopt failed</title>
<path fill="none" stroke="black" d="M568.577,-452.632C582.662,-463.509 601.016,-477.109 609.537,-481 639.747,-494.793 676.151,-503.31 704.886,-508.371"/>
<polygon fill="black" stroke="black" points="704.659,-511.882 715.1,-510.082 705.816,-504.978 704.659,-511.882"/>
<text text-anchor="middle" x="652.037" y="-509.4" font-family="Times,serif" font-size="12.00" fill="gray">fail</text>
</g>
<!-- available&#45;&gt;manageable -->
<g id="edge7" class="edge"><title>available&#45;&gt;manageable</title>
<path fill="none" stroke="black" d="M726.023,-470.944C660.612,-492.018 497.13,-532.998 404.537,-453 376.12,-428.448 356.939,-317.941 349.238,-265.193"/>
<polygon fill="black" stroke="black" points="352.684,-264.568 347.81,-255.161 345.754,-265.554 352.684,-264.568"/>
<text text-anchor="middle" x="549.037" y="-503.4" font-family="Times,serif" font-size="12.00">manage (via API)</text>
</g>
<!-- deploying -->
<g id="node8" class="node"><title>deploying</title>
<ellipse fill="none" stroke="black" cx="956.23" cy="-678" rx="35.4579" ry="18"/>
<text text-anchor="middle" x="956.23" y="-675.2" font-family="Times,serif" font-size="11.00" fill="gray">deploying</text>
</g>
<!-- available&#45;&gt;deploying -->
<g id="edge6" class="edge"><title>available&#45;&gt;deploying</title>
<path fill="none" stroke="black" d="M779.208,-474.106C785.558,-478.19 792.257,-482.956 797.927,-488 856.136,-539.775 911.05,-613.91 937.958,-652.482"/>
<polygon fill="black" stroke="black" points="935.311,-654.808 943.877,-661.043 941.069,-650.827 935.311,-654.808"/>
<text text-anchor="middle" x="853.927" y="-589.4" font-family="Times,serif" font-size="12.00">active (via API)</text>
</g>
<!-- deploying&#45;&gt;active -->
<g id="edge20" class="edge"><title>deploying&#45;&gt;active</title>
<path fill="none" stroke="black" d="M960.849,-660.046C967.407,-632.827 984.101,-581.969 1020.53,-560 1054.39,-539.582 1101.19,-543.658 1132.61,-549.97"/>
<polygon fill="black" stroke="black" points="1132.12,-553.445 1142.64,-552.164 1133.62,-546.607 1132.12,-553.445"/>
<text text-anchor="middle" x="1061.03" y="-563.4" font-family="Times,serif" font-size="12.00" fill="gray">done</text>
</g>
<!-- deploy failed -->
<g id="node15" class="node"><title>deploy failed</title>
<ellipse fill="none" stroke="black" cx="1374.25" cy="-727" rx="44.498" ry="18"/>
<text text-anchor="middle" x="1374.25" y="-724.2" font-family="Times,serif" font-size="11.00" fill="red">deploy failed</text>
</g>
<!-- deploying&#45;&gt;deploy failed -->
<g id="edge18" class="edge"><title>deploying&#45;&gt;deploy failed</title>
<path fill="none" stroke="black" d="M966.444,-695.258C976.924,-712.883 995.833,-739.27 1020.53,-751 1123.58,-799.935 1262.91,-765.165 1331.53,-742.438"/>
<polygon fill="black" stroke="black" points="1332.78,-745.71 1341.13,-739.184 1330.53,-739.08 1332.78,-745.71"/>
<text text-anchor="middle" x="1167.64" y="-778.4" font-family="Times,serif" font-size="12.00" fill="gray">fail</text>
</g>
<!-- wait call&#45;back -->
<g id="node16" class="node"><title>wait call&#45;back</title>
<ellipse fill="none" stroke="black" cx="1167.64" cy="-727" rx="48.2143" ry="18"/>
<text text-anchor="middle" x="1167.64" y="-724.2" font-family="Times,serif" font-size="11.00" fill="gray">wait call&#45;back</text>
</g>
<!-- deploying&#45;&gt;wait call&#45;back -->
<g id="edge19" class="edge"><title>deploying&#45;&gt;wait call&#45;back</title>
<path fill="none" stroke="black" d="M972.756,-693.955C984.658,-704.992 1002.12,-718.772 1020.53,-725 1049.75,-734.882 1084.16,-735.829 1112.25,-734.099"/>
<polygon fill="black" stroke="black" points="1112.58,-737.585 1122.3,-733.352 1112.06,-730.604 1112.58,-737.585"/>
<text text-anchor="middle" x="1061.03" y="-737.4" font-family="Times,serif" font-size="12.00" fill="gray">wait</text>
</g>
<!-- active&#45;&gt;deploying -->
<g id="edge8" class="edge"><title>active&#45;&gt;deploying</title>
<path fill="none" stroke="black" d="M1142.29,-565.768C1102.09,-576.997 1025.73,-598.588 1020.53,-602 1001.04,-614.808 984.407,-635.423 973.091,-651.954"/>
<polygon fill="black" stroke="black" points="969.946,-650.365 967.367,-660.641 975.791,-654.217 969.946,-650.365"/>
<text text-anchor="middle" x="1061.03" y="-605.4" font-family="Times,serif" font-size="12.00">rebuild (via API)</text>
</g>
<!-- deleting -->
<g id="node10" class="node"><title>deleting</title>
<ellipse fill="none" stroke="black" cx="2398.02" cy="-690" rx="31.0408" ry="18"/>
<text text-anchor="middle" x="2398.02" y="-687.2" font-family="Times,serif" font-size="11.00" fill="gray">deleting</text>
</g>
<!-- active&#45;&gt;deleting -->
<g id="edge9" class="edge"><title>active&#45;&gt;deleting</title>
<path fill="none" stroke="black" d="M1188.2,-571.071C1231.19,-596.846 1338,-656.725 1436.75,-679 1526.73,-699.299 2174.51,-696.944 2266.75,-698 2303.2,-698.417 2312.43,-700.99 2348.75,-698 2351.83,-697.747 2355,-697.404 2358.19,-697.003"/>
<polygon fill="black" stroke="black" points="2358.88,-700.441 2368.28,-695.559 2357.88,-693.511 2358.88,-700.441"/>
<text text-anchor="middle" x="1767.53" y="-698.4" font-family="Times,serif" font-size="12.00">deleted (via API)</text>
</g>
<!-- rescuing -->
<g id="node11" class="node"><title>rescuing</title>
<ellipse fill="none" stroke="black" cx="1374.25" cy="-556" rx="32.4445" ry="18"/>
<text text-anchor="middle" x="1374.25" y="-553.2" font-family="Times,serif" font-size="11.00" fill="gray">rescuing</text>
</g>
<!-- active&#45;&gt;rescuing -->
<g id="edge10" class="edge"><title>active&#45;&gt;rescuing</title>
<path fill="none" stroke="black" d="M1194.97,-558.614C1229.52,-558.107 1290.56,-557.212 1331.46,-556.613"/>
<polygon fill="black" stroke="black" points="1331.7,-560.11 1341.65,-556.463 1331.6,-553.11 1331.7,-560.11"/>
<text text-anchor="middle" x="1272.75" y="-561.4" font-family="Times,serif" font-size="12.00">rescue (via API)</text>
</g>
<!-- deleting&#45;&gt;cleaning -->
<g id="edge36" class="edge"><title>deleting&#45;&gt;cleaning</title>
<path fill="none" stroke="black" d="M2381.96,-674.272C2354.68,-644.879 2297.11,-578.548 2266.75,-512 2251.81,-479.236 2275.59,-457.018 2248.75,-433 1991.45,-202.67 1822.08,-345 1476.75,-345 754.232,-345 754.232,-345 754.232,-345 692.502,-345 623.776,-321.275 583.872,-304.989"/>
<polygon fill="black" stroke="black" points="585.186,-301.745 574.609,-301.13 582.494,-308.207 585.186,-301.745"/>
<text text-anchor="middle" x="1475.75" y="-348.4" font-family="Times,serif" font-size="12.00" fill="gray">clean</text>
</g>
<!-- error -->
<g id="node12" class="node"><title>error</title>
<ellipse fill="none" stroke="black" stroke-width="1.7" cx="2574.29" cy="-726" rx="27" ry="18"/>
<text text-anchor="middle" x="2574.29" y="-723.2" font-family="Times,serif" font-size="11.00" fill="red">error</text>
</g>
<!-- deleting&#45;&gt;error -->
<g id="edge35" class="edge"><title>deleting&#45;&gt;error</title>
<path fill="none" stroke="black" d="M2427.24,-683.396C2454.35,-678.41 2495.98,-674.361 2529.29,-687 2538.21,-690.383 2546.48,-696.492 2553.39,-702.887"/>
<polygon fill="black" stroke="black" points="2551.3,-705.748 2560.83,-710.378 2556.26,-700.816 2551.3,-705.748"/>
<text text-anchor="middle" x="2488.29" y="-690.4" font-family="Times,serif" font-size="12.00" fill="gray">error</text>
</g>
<!-- rescue -->
<g id="node13" class="node"><title>rescue</title>
<ellipse fill="none" stroke="black" stroke-width="1.7" cx="1668.53" cy="-642" rx="27.824" ry="18"/>
<text text-anchor="middle" x="1668.53" y="-639.2" font-family="Times,serif" font-size="11.00">rescue</text>
</g>
<!-- rescuing&#45;&gt;rescue -->
<g id="edge49" class="edge"><title>rescuing&#45;&gt;rescue</title>
<path fill="none" stroke="black" d="M1393.93,-570.387C1405.64,-578.743 1421.38,-588.81 1436.75,-595 1502.19,-621.37 1584.26,-633.545 1630.81,-638.68"/>
<polygon fill="black" stroke="black" points="1630.71,-642.188 1641.02,-639.755 1631.44,-635.227 1630.71,-642.188"/>
<text text-anchor="middle" x="1475.75" y="-621.4" font-family="Times,serif" font-size="12.00" fill="gray">done</text>
</g>
<!-- rescue wait -->
<g id="node22" class="node"><title>rescue wait</title>
<ellipse fill="none" stroke="black" cx="1573.64" cy="-527" rx="40.7822" ry="18"/>
<text text-anchor="middle" x="1573.64" y="-524.2" font-family="Times,serif" font-size="11.00" fill="gray">rescue wait</text>
</g>
<!-- rescuing&#45;&gt;rescue wait -->
<g id="edge50" class="edge"><title>rescuing&#45;&gt;rescue wait</title>
<path fill="none" stroke="black" d="M1406.09,-551.469C1437.8,-546.809 1487.64,-539.488 1524.54,-534.066"/>
<polygon fill="black" stroke="black" points="1525.19,-537.508 1534.58,-532.591 1524.17,-530.582 1525.19,-537.508"/>
<text text-anchor="middle" x="1475.75" y="-549.4" font-family="Times,serif" font-size="12.00" fill="gray">wait</text>
</g>
<!-- rescue failed -->
<g id="node23" class="node"><title>rescue failed</title>
<ellipse fill="none" stroke="black" cx="1767.53" cy="-515" rx="44.498" ry="18"/>
<text text-anchor="middle" x="1767.53" y="-512.2" font-family="Times,serif" font-size="11.00" fill="red">rescue failed</text>
</g>
<!-- rescuing&#45;&gt;rescue failed -->
<g id="edge51" class="edge"><title>rescuing&#45;&gt;rescue failed</title>
<path fill="none" stroke="black" d="M1405.84,-560.174C1415.69,-561.332 1426.66,-562.428 1436.75,-563 1555.87,-569.758 1589.67,-576.294 1704.53,-544 1713.6,-541.449 1723,-537.695 1731.66,-533.729"/>
<polygon fill="black" stroke="black" points="1733.21,-536.868 1740.71,-529.388 1730.18,-530.557 1733.21,-536.868"/>
<text text-anchor="middle" x="1573.64" y="-571.4" font-family="Times,serif" font-size="12.00" fill="gray">fail</text>
</g>
<!-- error&#45;&gt;deploying -->
<g id="edge11" class="edge"><title>error&#45;&gt;deploying</title>
<path fill="none" stroke="black" d="M2560.15,-741.474C2533.22,-771.337 2468.43,-834 2399.02,-834 1166.64,-834 1166.64,-834 1166.64,-834 1096.77,-834 1074.52,-820.342 1020.53,-776 997.081,-756.739 979.397,-726.455 968.762,-704.649"/>
<polygon fill="black" stroke="black" points="971.926,-703.152 964.508,-695.588 965.59,-706.127 971.926,-703.152"/>
<text text-anchor="middle" x="1767.53" y="-837.4" font-family="Times,serif" font-size="12.00">rebuild (via API)</text>
</g>
<!-- error&#45;&gt;deleting -->
<g id="edge12" class="edge"><title>error&#45;&gt;deleting</title>
<path fill="none" stroke="black" d="M2547.55,-722.349C2521.97,-718.523 2481.69,-711.966 2447.29,-704 2443.48,-703.117 2439.52,-702.114 2435.59,-701.06"/>
<polygon fill="black" stroke="black" points="2436.52,-697.686 2425.95,-698.374 2434.64,-704.43 2436.52,-697.686"/>
<text text-anchor="middle" x="2488.29" y="-722.4" font-family="Times,serif" font-size="12.00">deleted (via API)</text>
</g>
<!-- rescue&#45;&gt;deleting -->
<g id="edge14" class="edge"><title>rescue&#45;&gt;deleting</title>
<path fill="none" stroke="black" d="M1696.28,-643.765C1806.35,-651.028 2224.63,-678.625 2356.76,-687.344"/>
<polygon fill="black" stroke="black" points="2356.65,-690.844 2366.86,-688.01 2357.11,-683.859 2356.65,-690.844"/>
<text text-anchor="middle" x="1979.53" y="-668.4" font-family="Times,serif" font-size="12.00">deleted (via API)</text>
</g>
<!-- rescue&#45;&gt;rescuing -->
<g id="edge13" class="edge"><title>rescue&#45;&gt;rescuing</title>
<path fill="none" stroke="black" d="M1643.2,-649.349C1598.28,-661.361 1501.02,-679.917 1436.75,-638 1420.75,-627.572 1429.71,-614.628 1418.75,-599 1413.57,-591.627 1407.05,-584.42 1400.65,-578.078"/>
<polygon fill="black" stroke="black" points="1402.93,-575.418 1393.27,-571.072 1398.11,-580.495 1402.93,-575.418"/>
<text text-anchor="middle" x="1475.75" y="-665.4" font-family="Times,serif" font-size="12.00">rescue (via API)</text>
</g>
<!-- unrescuing -->
<g id="node14" class="node"><title>unrescuing</title>
<ellipse fill="none" stroke="black" cx="1979.53" cy="-476" rx="39.1741" ry="18"/>
<text text-anchor="middle" x="1979.53" y="-473.2" font-family="Times,serif" font-size="11.00" fill="gray">unrescuing</text>
</g>
<!-- rescue&#45;&gt;unrescuing -->
<g id="edge15" class="edge"><title>rescue&#45;&gt;unrescuing</title>
<path fill="none" stroke="black" d="M1690.86,-631.201C1734.38,-609.079 1836.39,-556.752 1920.53,-510 1928.92,-505.337 1937.91,-500.161 1946.24,-495.282"/>
<polygon fill="black" stroke="black" points="1948.12,-498.234 1954.96,-490.144 1944.57,-492.202 1948.12,-498.234"/>
<text text-anchor="middle" x="1767.53" y="-618.4" font-family="Times,serif" font-size="12.00">unrescue (via API)</text>
</g>
<!-- unrescuing&#45;&gt;active -->
<g id="edge59" class="edge"><title>unrescuing&#45;&gt;active</title>
<path fill="none" stroke="black" d="M1952.29,-462.999C1920.22,-447.83 1863.87,-423.491 1812.53,-413 1648.88,-379.562 1595.84,-367.145 1436.75,-418 1342.62,-448.085 1242.24,-509.962 1195.49,-540.758"/>
<polygon fill="black" stroke="black" points="1193.53,-537.859 1187.13,-546.308 1197.4,-543.69 1193.53,-537.859"/>
<text text-anchor="middle" x="1573.64" y="-395.4" font-family="Times,serif" font-size="12.00" fill="gray">done</text>
</g>
<!-- unrescue failed -->
<g id="node24" class="node"><title>unrescue failed</title>
<ellipse fill="none" stroke="black" cx="2197.64" cy="-460" rx="51.2277" ry="18"/>
<text text-anchor="middle" x="2197.64" y="-457.2" font-family="Times,serif" font-size="11.00" fill="red">unrescue failed</text>
</g>
<!-- unrescuing&#45;&gt;unrescue failed -->
<g id="edge60" class="edge"><title>unrescuing&#45;&gt;unrescue failed</title>
<path fill="none" stroke="black" d="M2018.17,-473.213C2050.89,-470.791 2098.87,-467.239 2136.84,-464.427"/>
<polygon fill="black" stroke="black" points="2137.55,-467.885 2147.26,-463.656 2137.03,-460.904 2137.55,-467.885"/>
<text text-anchor="middle" x="2083.53" y="-474.4" font-family="Times,serif" font-size="12.00" fill="gray">fail</text>
</g>
<!-- deploy failed&#45;&gt;deploying -->
<g id="edge24" class="edge"><title>deploy failed&#45;&gt;deploying</title>
<path fill="none" stroke="black" d="M1338.95,-716.026C1307.33,-706.452 1258.9,-693.158 1215.75,-687 1141.3,-676.375 1053.63,-675.799 1001.89,-676.672"/>
<polygon fill="black" stroke="black" points="1001.66,-673.176 991.726,-676.87 1001.79,-680.175 1001.66,-673.176"/>
<text text-anchor="middle" x="1167.64" y="-690.4" font-family="Times,serif" font-size="12.00">rebuild (via API)</text>
</g>
<!-- deploy failed&#45;&gt;deploying -->
<g id="edge25" class="edge"><title>deploy failed&#45;&gt;deploying</title>
<path fill="none" stroke="black" d="M1353.19,-711.136C1341.63,-702.649 1326.5,-692.613 1311.75,-686 1271.51,-667.966 1259.48,-666.601 1215.75,-661 1140.2,-651.325 1051.49,-661.887 1000.08,-670.147"/>
<polygon fill="black" stroke="black" points="999.297,-666.729 990.002,-671.815 1000.44,-673.635 999.297,-666.729"/>
<text text-anchor="middle" x="1167.64" y="-664.4" font-family="Times,serif" font-size="12.00">active (via API)</text>
</g>
<!-- deploy failed&#45;&gt;deleting -->
<g id="edge26" class="edge"><title>deploy failed&#45;&gt;deleting</title>
<path fill="none" stroke="black" d="M1418.69,-725.813C1497.28,-723.664 1668.29,-718.983 1812.53,-715 1860.53,-713.675 1872.54,-713.774 1920.53,-712 2082.06,-706.028 2274.29,-696.373 2356.49,-692.123"/>
<polygon fill="black" stroke="black" points="2356.84,-695.609 2366.65,-691.596 2356.48,-688.619 2356.84,-695.609"/>
<text text-anchor="middle" x="1875.53" y="-717.4" font-family="Times,serif" font-size="12.00">deleted (via API)</text>
</g>
<!-- wait call&#45;back&#45;&gt;deploying -->
<g id="edge21" class="edge"><title>wait call&#45;back&#45;&gt;deploying</title>
<path fill="none" stroke="black" d="M1129.94,-715.739C1120.7,-713.058 1110.79,-710.308 1101.53,-708 1067.59,-699.533 1028.75,-691.554 999.918,-685.956"/>
<polygon fill="black" stroke="black" points="1000.26,-682.457 989.777,-684.005 998.936,-689.331 1000.26,-682.457"/>
<text text-anchor="middle" x="1061.03" y="-711.4" font-family="Times,serif" font-size="12.00" fill="gray">resume</text>
</g>
<!-- wait call&#45;back&#45;&gt;deleting -->
<g id="edge23" class="edge"><title>wait call&#45;back&#45;&gt;deleting</title>
<path fill="none" stroke="black" d="M1198.4,-741.04C1237.57,-758.205 1309,-785 1373.25,-785 1373.25,-785 1373.25,-785 1669.53,-785 1973.71,-785 2049.15,-753.526 2348.75,-701 2352.16,-700.403 2355.69,-699.722 2359.21,-698.999"/>
<polygon fill="black" stroke="black" points="2360.01,-702.408 2369.04,-696.88 2358.53,-695.565 2360.01,-702.408"/>
<text text-anchor="middle" x="1767.53" y="-787.4" font-family="Times,serif" font-size="12.00">deleted (via API)</text>
</g>
<!-- wait call&#45;back&#45;&gt;deploy failed -->
<g id="edge22" class="edge"><title>wait call&#45;back&#45;&gt;deploy failed</title>
<path fill="none" stroke="black" d="M1215.84,-727C1246.72,-727 1287.15,-727 1319.31,-727"/>
<polygon fill="black" stroke="black" points="1319.43,-730.5 1329.43,-727 1319.43,-723.5 1319.43,-730.5"/>
<text text-anchor="middle" x="1272.75" y="-730.4" font-family="Times,serif" font-size="12.00" fill="gray">fail</text>
</g>
<!-- clean failed&#45;&gt;manageable -->
<g id="edge34" class="edge"><title>clean failed&#45;&gt;manageable</title>
<path fill="none" stroke="black" d="M929.475,-243.134C918.26,-237.755 904.767,-232.143 891.927,-229 851.265,-219.045 839.745,-224.941 797.927,-223 623.273,-214.894 577.797,-197.529 404.537,-221 399.338,-221.704 393.947,-222.732 388.632,-223.932"/>
<polygon fill="black" stroke="black" points="387.444,-220.62 378.573,-226.412 389.12,-227.416 387.444,-220.62"/>
<text text-anchor="middle" x="652.037" y="-220.4" font-family="Times,serif" font-size="12.00">manage (via API)</text>
</g>
<!-- clean wait&#45;&gt;cleaning -->
<g id="edge33" class="edge"><title>clean wait&#45;&gt;cleaning</title>
<path fill="none" stroke="black" d="M721.076,-241.942C691.098,-236.2 646.179,-231.488 609.537,-244 595.752,-248.707 582.682,-258.048 572.322,-267.089"/>
<polygon fill="black" stroke="black" points="569.949,-264.517 564.96,-273.864 574.689,-269.668 569.949,-264.517"/>
<text text-anchor="middle" x="652.037" y="-247.4" font-family="Times,serif" font-size="12.00" fill="gray">resume</text>
</g>
<!-- clean wait&#45;&gt;clean failed -->
<g id="edge31" class="edge"><title>clean wait&#45;&gt;clean failed</title>
<path fill="none" stroke="black" d="M793.165,-251.98C800.695,-252.349 808.565,-252.709 815.927,-253 845.273,-254.159 878.109,-255.118 904.528,-255.806"/>
<polygon fill="black" stroke="black" points="904.6,-259.309 914.686,-256.066 904.779,-252.311 904.6,-259.309"/>
<text text-anchor="middle" x="853.927" y="-258.4" font-family="Times,serif" font-size="12.00" fill="gray">fail</text>
</g>
<!-- clean wait&#45;&gt;clean failed -->
<g id="edge32" class="edge"><title>clean wait&#45;&gt;clean failed</title>
<path fill="none" stroke="black" d="M788.212,-241.07C797.112,-238.977 806.833,-237.052 815.927,-236 849.481,-232.118 858.582,-230.613 891.927,-236 899.723,-237.259 907.859,-239.338 915.599,-241.711"/>
<polygon fill="black" stroke="black" points="914.544,-245.048 925.137,-244.833 916.722,-238.395 914.544,-245.048"/>
<text text-anchor="middle" x="853.927" y="-239.4" font-family="Times,serif" font-size="12.00">abort (via API)</text>
</g>
<!-- inspect failed&#45;&gt;manageable -->
<g id="edge40" class="edge"><title>inspect failed&#45;&gt;manageable</title>
<path fill="none" stroke="black" d="M920.607,-87.7177C911.35,-90.6363 901.309,-93.6109 891.927,-96 813.207,-116.046 791.832,-113.365 712.537,-131 574.8,-161.632 537.055,-159.537 404.537,-208 396.437,-210.962 387.977,-214.698 380.074,-218.494"/>
<polygon fill="black" stroke="black" points="378.514,-215.361 371.106,-222.936 381.62,-221.634 378.514,-215.361"/>
<text text-anchor="middle" x="652.037" y="-155.4" font-family="Times,serif" font-size="12.00">manage (via API)</text>
</g>
<!-- inspect failed&#45;&gt;inspecting -->
<g id="edge41" class="edge"><title>inspect failed&#45;&gt;inspecting</title>
<path fill="none" stroke="black" d="M910.04,-78.389C861.291,-81.0172 781.392,-85.472 712.537,-90 673.312,-92.5795 628.79,-95.8944 596.28,-98.3863"/>
<polygon fill="black" stroke="black" points="595.87,-94.9074 586.169,-99.1645 596.408,-101.887 595.87,-94.9074"/>
<text text-anchor="middle" x="755.232" y="-93.4" font-family="Times,serif" font-size="12.00">inspect (via API)</text>
</g>
<!-- inspect wait&#45;&gt;manageable -->
<g id="edge42" class="edge"><title>inspect wait&#45;&gt;manageable</title>
<path fill="none" stroke="black" d="M714.208,-12.8694C642.652,-6.06784 492.105,-2.86119 404.537,-81 367.231,-114.289 353.367,-173.626 348.293,-208.739"/>
<polygon fill="black" stroke="black" points="344.804,-208.435 346.985,-218.803 351.745,-209.337 344.804,-208.435"/>
<text text-anchor="middle" x="549.037" y="-30.4" font-family="Times,serif" font-size="12.00" fill="gray">done</text>
</g>
<!-- inspect wait&#45;&gt;inspect failed -->
<g id="edge43" class="edge"><title>inspect wait&#45;&gt;inspect failed</title>
<path fill="none" stroke="black" d="M794.746,-24.8927C822.183,-30.2627 859.749,-38.5546 891.927,-49 900.095,-51.6514 908.667,-54.9424 916.771,-58.3047"/>
<polygon fill="black" stroke="black" points="915.447,-61.545 926.019,-62.2505 918.195,-55.1066 915.447,-61.545"/>
<text text-anchor="middle" x="853.927" y="-52.4" font-family="Times,serif" font-size="12.00" fill="gray">fail</text>
</g>
<!-- inspect wait&#45;&gt;inspect failed -->
<g id="edge44" class="edge"><title>inspect wait&#45;&gt;inspect failed</title>
<path fill="none" stroke="black" d="M791.951,-8.80514C820.179,-3.36397 860.004,0.178118 891.927,-13 910.324,-20.5943 926.187,-36.4115 937.432,-50.2964"/>
<polygon fill="black" stroke="black" points="934.948,-52.8076 943.818,-58.6012 940.497,-48.5405 934.948,-52.8076"/>
<text text-anchor="middle" x="853.927" y="-16.4" font-family="Times,serif" font-size="12.00">abort (via API)</text>
</g>
<!-- adopt failed&#45;&gt;manageable -->
<g id="edge48" class="edge"><title>adopt failed&#45;&gt;manageable</title>
<path fill="none" stroke="black" d="M727.614,-501.184C722.123,-497.451 716.741,-493.029 712.537,-488 699.249,-472.105 709.749,-459.066 694.537,-445 665.334,-417.996 643.124,-438.307 609.537,-417 599.715,-410.769 601.023,-404.733 591.537,-398 517.241,-345.261 475.997,-370.522 404.537,-314 386.806,-299.976 371.43,-279.504 360.811,-263.194"/>
<polygon fill="black" stroke="black" points="363.706,-261.224 355.415,-254.628 357.783,-264.955 363.706,-261.224"/>
<text text-anchor="middle" x="549.037" y="-401.4" font-family="Times,serif" font-size="12.00">manage (via API)</text>
</g>
<!-- adopt failed&#45;&gt;adopting -->
<g id="edge47" class="edge"><title>adopt failed&#45;&gt;adopting</title>
<path fill="none" stroke="black" d="M728.923,-500.649C723.246,-496.889 717.454,-492.59 712.537,-488 702.79,-478.902 705.886,-470.999 694.537,-464 677.654,-453.587 627.891,-446.226 591.467,-442.054"/>
<polygon fill="black" stroke="black" points="591.438,-438.53 581.115,-440.912 590.67,-445.487 591.438,-438.53"/>
<text text-anchor="middle" x="652.037" y="-467.4" font-family="Times,serif" font-size="12.00">adopt (via API)</text>
</g>
<!-- rescue wait&#45;&gt;deleting -->
<g id="edge55" class="edge"><title>rescue wait&#45;&gt;deleting</title>
<path fill="none" stroke="black" d="M1607.54,-537.303C1615.67,-539.669 1624.38,-542.057 1632.53,-544 1711.24,-562.758 2270.01,-657.373 2348.75,-676 2352.56,-676.901 2356.52,-677.915 2360.45,-678.977"/>
<polygon fill="black" stroke="black" points="2359.52,-682.349 2370.09,-681.672 2361.4,-675.608 2359.52,-682.349"/>
<text text-anchor="middle" x="1979.53" y="-618.4" font-family="Times,serif" font-size="12.00">deleted (via API)</text>
</g>
<!-- rescue wait&#45;&gt;rescuing -->
<g id="edge52" class="edge"><title>rescue wait&#45;&gt;rescuing</title>
<path fill="none" stroke="black" d="M1537.61,-518.274C1509.75,-512.851 1470.07,-508.536 1436.75,-518 1424.2,-521.563 1411.71,-528.494 1401.29,-535.472"/>
<polygon fill="black" stroke="black" points="1399.23,-532.645 1393.07,-541.271 1403.26,-538.364 1399.23,-532.645"/>
<text text-anchor="middle" x="1475.75" y="-521.4" font-family="Times,serif" font-size="12.00" fill="gray">resume</text>
</g>
<!-- rescue wait&#45;&gt;rescue failed -->
<g id="edge53" class="edge"><title>rescue wait&#45;&gt;rescue failed</title>
<path fill="none" stroke="black" d="M1614.51,-527.873C1640.24,-528.07 1674.42,-527.646 1704.53,-525 1708.45,-524.655 1712.49,-524.209 1716.56,-523.697"/>
<polygon fill="black" stroke="black" points="1717.18,-527.143 1726.61,-522.308 1716.23,-520.209 1717.18,-527.143"/>
<text text-anchor="middle" x="1668.53" y="-530.4" font-family="Times,serif" font-size="12.00" fill="gray">fail</text>
</g>
<!-- rescue wait&#45;&gt;rescue failed -->
<g id="edge54" class="edge"><title>rescue wait&#45;&gt;rescue failed</title>
<path fill="none" stroke="black" d="M1604.53,-515.19C1613.4,-512.236 1623.23,-509.487 1632.53,-508 1659.82,-503.636 1690.6,-504.821 1715.77,-507.348"/>
<polygon fill="black" stroke="black" points="1715.51,-510.841 1725.83,-508.459 1716.28,-503.883 1715.51,-510.841"/>
<text text-anchor="middle" x="1668.53" y="-511.4" font-family="Times,serif" font-size="12.00">abort (via API)</text>
</g>
<!-- rescue failed&#45;&gt;deleting -->
<g id="edge58" class="edge"><title>rescue failed&#45;&gt;deleting</title>
<path fill="none" stroke="black" d="M1811.51,-512.199C1931.12,-505.29 2264.6,-492.32 2348.75,-556 2381.56,-580.826 2392.11,-629.985 2395.48,-661.36"/>
<polygon fill="black" stroke="black" points="2392.02,-661.997 2396.4,-671.648 2399,-661.376 2392.02,-661.997"/>
<text text-anchor="middle" x="2083.53" y="-511.4" font-family="Times,serif" font-size="12.00">deleted (via API)</text>
</g>
<!-- rescue failed&#45;&gt;rescuing -->
<g id="edge56" class="edge"><title>rescue failed&#45;&gt;rescuing</title>
<path fill="none" stroke="black" d="M1729.71,-505.399C1683.68,-494.593 1602.44,-479.466 1532.75,-487 1489.32,-491.694 1476.28,-489.434 1436.75,-508 1423.07,-514.423 1409.63,-524.364 1398.82,-533.536"/>
<polygon fill="black" stroke="black" points="1396.3,-531.094 1391.11,-540.33 1400.93,-536.346 1396.3,-531.094"/>
<text text-anchor="middle" x="1573.64" y="-490.4" font-family="Times,serif" font-size="12.00">rescue (via API)</text>
</g>
<!-- rescue failed&#45;&gt;unrescuing -->
<g id="edge57" class="edge"><title>rescue failed&#45;&gt;unrescuing</title>
<path fill="none" stroke="black" d="M1799.66,-502.402C1809.4,-498.88 1820.29,-495.372 1830.53,-493 1863.29,-485.409 1900.95,-481.144 1930.09,-478.784"/>
<polygon fill="black" stroke="black" points="1930.7,-482.247 1940.41,-477.998 1930.17,-475.267 1930.7,-482.247"/>
<text text-anchor="middle" x="1875.53" y="-496.4" font-family="Times,serif" font-size="12.00">unrescue (via API)</text>
</g>
<!-- unrescue failed&#45;&gt;deleting -->
<g id="edge63" class="edge"><title>unrescue failed&#45;&gt;deleting</title>
<path fill="none" stroke="black" d="M2248.88,-458.967C2281.5,-461.04 2322.66,-469.253 2348.75,-495 2372.24,-518.169 2387.38,-613.595 2393.68,-661.824"/>
<polygon fill="black" stroke="black" points="2390.22,-662.424 2394.95,-671.905 2397.17,-661.544 2390.22,-662.424"/>
<text text-anchor="middle" x="2307.75" y="-498.4" font-family="Times,serif" font-size="12.00">deleted (via API)</text>
</g>
<!-- unrescue failed&#45;&gt;rescuing -->
<g id="edge61" class="edge"><title>unrescue failed&#45;&gt;rescuing</title>
<path fill="none" stroke="black" d="M2147.71,-456.123C2112.39,-453.524 2063.55,-450.342 2020.53,-449 1760.73,-440.894 1668.86,-367.033 1436.75,-484 1416.68,-494.112 1400.51,-513.952 1389.79,-530.249"/>
<polygon fill="black" stroke="black" points="1386.75,-528.519 1384.41,-538.854 1392.68,-532.231 1386.75,-528.519"/>
<text text-anchor="middle" x="1767.53" y="-432.4" font-family="Times,serif" font-size="12.00">rescue (via API)</text>
</g>
<!-- unrescue failed&#45;&gt;unrescuing -->
<g id="edge62" class="edge"><title>unrescue failed&#45;&gt;unrescuing</title>
<path fill="none" stroke="black" d="M2154.85,-450.106C2122.68,-444.096 2077.31,-439.13 2038.53,-448 2029.91,-449.971 2021.08,-453.396 2013,-457.187"/>
<polygon fill="black" stroke="black" points="2011.28,-454.134 2003.9,-461.743 2014.41,-460.393 2011.28,-454.134"/>
<text text-anchor="middle" x="2083.53" y="-451.4" font-family="Times,serif" font-size="12.00">unrescue (via API)</text>
</g>
</g>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 39 KiB

2
doc/source/user/states.rst
View File

@ -17,7 +17,7 @@ API-initiated-transitions that are possible from non-stable states.
The events for these API-initiated transitions are indicated with '(via API)'. The events for these API-initiated transitions are indicated with '(via API)'.
Internally, the conductor initiates the other transitions (depicted in gray). Internally, the conductor initiates the other transitions (depicted in gray).
.. figure:: ../images/states.svg .. figure:: ../images/states.png
:width: 660px :width: 660px
:align: left :align: left
:alt: Ironic state transitions :alt: Ironic state transitions

10
ironic/api/controllers/v1/node.py
View File

@ -122,7 +122,8 @@ _DEFAULT_RETURN_FIELDS = ['instance_uuid', 'maintenance', 'power_state',
PROVISION_ACTION_STATES = (ir_states.VERBS['manage'], PROVISION_ACTION_STATES = (ir_states.VERBS['manage'],
ir_states.VERBS['provide'], ir_states.VERBS['provide'],
ir_states.VERBS['abort'], ir_states.VERBS['abort'],
ir_states.VERBS['adopt']) ir_states.VERBS['adopt'],
ir_states.VERBS['unhold'])
_NODES_CONTROLLER_RESERVED_WORDS = None _NODES_CONTROLLER_RESERVED_WORDS = None
@ -1129,6 +1130,13 @@ class NodeStatesController(rest.RestController):
if not api_utils.allow_inspect_abort(): if not api_utils.allow_inspect_abort():
raise exception.NotAcceptable() raise exception.NotAcceptable()
if target == ir_states.VERBS['unhold']:
# NOTE(TheJulia): There is no solid reason to do state checks
# as well, other than add additional complexity as multiple
# states are involved.
if not api_utils.allow_unhold_verb():
raise exception.NotAcceptable()
self._do_provision_action(rpc_node, target, configdrive, clean_steps, self._do_provision_action(rpc_node, target, configdrive, clean_steps,
deploy_steps, rescue_password, deploy_steps, rescue_password,
disable_ramdisk) disable_ramdisk)

5
ironic/api/controllers/v1/utils.py
View File

@ -1951,6 +1951,11 @@ def allow_status_in_heartbeat():
return api.request.version.minor >= versions.MINOR_72_HEARTBEAT_STATUS return api.request.version.minor >= versions.MINOR_72_HEARTBEAT_STATUS
def allow_unhold_verb():
"""Check if the unhold verb may be passed to the API"""
return api.request.version.minor >= versions.MINOR_85_UNHOLD_VERB
def check_allow_deploy_steps(target, deploy_steps): def check_allow_deploy_steps(target, deploy_steps):
"""Check if deploy steps are allowed""" """Check if deploy steps are allowed"""

4
ironic/api/controllers/v1/versions.py
View File

@ -122,6 +122,7 @@ BASE_VERSION = 1
# v1.82: Add node sharding capability # v1.82: Add node sharding capability
# v1.83: Add child node modeling # v1.83: Add child node modeling
# v1.84: Add ramdisk callback to continue inspection. # v1.84: Add ramdisk callback to continue inspection.
# v1.85: Add unhold verb
MINOR_0_JUNO = 0 MINOR_0_JUNO = 0
MINOR_1_INITIAL_VERSION = 1 MINOR_1_INITIAL_VERSION = 1
MINOR_2_AVAILABLE_STATE = 2 MINOR_2_AVAILABLE_STATE = 2
@ -207,6 +208,7 @@ MINOR_81_NODE_INVENTORY = 81
MINOR_82_NODE_SHARD = 82 MINOR_82_NODE_SHARD = 82
MINOR_83_PARENT_CHILD_NODES = 83 MINOR_83_PARENT_CHILD_NODES = 83
MINOR_84_CONTINUE_INSPECTION = 84 MINOR_84_CONTINUE_INSPECTION = 84
MINOR_85_UNHOLD_VERB = 85
# When adding another version, update: # When adding another version, update:
# - MINOR_MAX_VERSION # - MINOR_MAX_VERSION
@ -214,7 +216,7 @@ MINOR_84_CONTINUE_INSPECTION = 84
# explanation of what changed in the new version # explanation of what changed in the new version
# - common/release_mappings.py, RELEASE_MAPPING['master']['api'] # - common/release_mappings.py, RELEASE_MAPPING['master']['api']
MINOR_MAX_VERSION = MINOR_84_CONTINUE_INSPECTION MINOR_MAX_VERSION = MINOR_85_UNHOLD_VERB
# String representations of the minor and maximum versions # String representations of the minor and maximum versions
_MIN_VERSION_STRING = '{}.{}'.format(BASE_VERSION, MINOR_1_INITIAL_VERSION) _MIN_VERSION_STRING = '{}.{}'.format(BASE_VERSION, MINOR_1_INITIAL_VERSION)

2
ironic/common/release_mappings.py
View File

@ -574,7 +574,7 @@ RELEASE_MAPPING = {
} }
}, },
'master': { 'master': {
'api': '1.84', 'api': '1.85',
'rpc': '1.56', 'rpc': '1.56',
'objects': { 'objects': {
'Allocation': ['1.1'], 'Allocation': ['1.1'],

25
ironic/common/states.py
View File

@ -52,6 +52,7 @@ VERBS = {
'adopt': 'adopt', 'adopt': 'adopt',
'rescue': 'rescue', 'rescue': 'rescue',
'unrescue': 'unrescue', 'unrescue': 'unrescue',
'unhold': 'unhold',
} }
""" Mapping of state-changing events that are PUT to the REST API """ Mapping of state-changing events that are PUT to the REST API
@ -129,6 +130,9 @@ This is mainly a target provision state used during deployment. A successfully
deployed node should go to ACTIVE status. deployed node should go to ACTIVE status.
""" """
DEPLOYHOLD = 'deploy hold'
""" Node is being held by a deploy step. """
DELETING = 'deleting' DELETING = 'deleting'
""" Node is actively being torn down. """ """ Node is actively being torn down. """
@ -159,6 +163,9 @@ the driver to finish a cleaning step.
CLEANFAIL = 'clean failed' CLEANFAIL = 'clean failed'
""" Node failed cleaning. This requires operator intervention to resolve. """ """ Node failed cleaning. This requires operator intervention to resolve. """
CLEANHOLD = 'clean hold'
""" Node is a holding state due to a clean step. """
ERROR = 'error' ERROR = 'error'
""" An error occurred during node processing. """ An error occurred during node processing.
@ -354,11 +361,13 @@ machine.add_state(VERIFYING, target=MANAGEABLE, **watchers)
machine.add_state(DEPLOYING, target=ACTIVE, **watchers) machine.add_state(DEPLOYING, target=ACTIVE, **watchers)
machine.add_state(DEPLOYWAIT, target=ACTIVE, **watchers) machine.add_state(DEPLOYWAIT, target=ACTIVE, **watchers)
machine.add_state(DEPLOYFAIL, target=ACTIVE, **watchers) machine.add_state(DEPLOYFAIL, target=ACTIVE, **watchers)
machine.add_state(DEPLOYHOLD, target=ACTIVE, **watchers)
# Add clean* states # Add clean* states
machine.add_state(CLEANING, target=AVAILABLE, **watchers) machine.add_state(CLEANING, target=AVAILABLE, **watchers)
machine.add_state(CLEANWAIT, target=AVAILABLE, **watchers) machine.add_state(CLEANWAIT, target=AVAILABLE, **watchers)
machine.add_state(CLEANFAIL, target=AVAILABLE, **watchers) machine.add_state(CLEANFAIL, target=AVAILABLE, **watchers)
machine.add_state(CLEANHOLD, target=AVAILABLE, **watchers)
# Add delete* states # Add delete* states
machine.add_state(DELETING, target=AVAILABLE, **watchers) machine.add_state(DELETING, target=AVAILABLE, **watchers)
@ -393,11 +402,19 @@ machine.add_transition(DEPLOYFAIL, DEPLOYING, 'deploy')
# A deployment may also wait on external callbacks # A deployment may also wait on external callbacks
machine.add_transition(DEPLOYING, DEPLOYWAIT, 'wait') machine.add_transition(DEPLOYING, DEPLOYWAIT, 'wait')
machine.add_transition(DEPLOYING, DEPLOYHOLD, 'hold')
machine.add_transition(DEPLOYWAIT, DEPLOYHOLD, 'hold')
machine.add_transition(DEPLOYWAIT, DEPLOYING, 'resume') machine.add_transition(DEPLOYWAIT, DEPLOYING, 'resume')
# A deployment waiting on callback may time out # A deployment waiting on callback may time out
machine.add_transition(DEPLOYWAIT, DEPLOYFAIL, 'fail') machine.add_transition(DEPLOYWAIT, DEPLOYFAIL, 'fail')
# Return the node into a deploying state from holding
machine.add_transition(DEPLOYHOLD, DEPLOYWAIT, 'unhold')
# A node in deploy hold may also be aborted
machine.add_transition(DEPLOYHOLD, DEPLOYFAIL, 'abort')
# A deployment may complete # A deployment may complete
machine.add_transition(DEPLOYING, ACTIVE, 'done') machine.add_transition(DEPLOYING, ACTIVE, 'done')
@ -435,8 +452,16 @@ machine.add_transition(CLEANWAIT, CLEANFAIL, 'abort')
# Cleaning may also wait on external callbacks # Cleaning may also wait on external callbacks
machine.add_transition(CLEANING, CLEANWAIT, 'wait') machine.add_transition(CLEANING, CLEANWAIT, 'wait')
machine.add_transition(CLEANING, CLEANHOLD, 'hold')
machine.add_transition(CLEANWAIT, CLEANHOLD, 'hold')
machine.add_transition(CLEANWAIT, CLEANING, 'resume') machine.add_transition(CLEANWAIT, CLEANING, 'resume')
# A node in a clean hold step may also be aborted
machine.add_transition(CLEANHOLD, CLEANFAIL, 'abort')
# Return the node back to cleaning
machine.add_transition(CLEANHOLD, CLEANWAIT, 'unhold')
# An operator may want to move a CLEANFAIL node to MANAGEABLE, to perform # An operator may want to move a CLEANFAIL node to MANAGEABLE, to perform
# other actions like cleaning # other actions like cleaning
machine.add_transition(CLEANFAIL, MANAGEABLE, 'manage') machine.add_transition(CLEANFAIL, MANAGEABLE, 'manage')

16
ironic/conductor/cleaning.py
View File

@ -159,7 +159,6 @@ def do_next_clean_step(task, step_index, disable_ramdisk=None):
LOG.info('Executing %(kind)s cleaning on node %(node)s, remaining steps: ' LOG.info('Executing %(kind)s cleaning on node %(node)s, remaining steps: '
'%(steps)s', {'node': node.uuid, 'steps': steps, '%(steps)s', {'node': node.uuid, 'steps': steps,
'kind': 'manual' if manual_clean else 'automated'}) 'kind': 'manual' if manual_clean else 'automated'})
# Execute each step until we hit an async step or run out of steps # Execute each step until we hit an async step or run out of steps
for ind, step in enumerate(steps): for ind, step in enumerate(steps):
# Save which step we're about to start so we can restart # Save which step we're about to start so we can restart
@ -171,10 +170,20 @@ def do_next_clean_step(task, step_index, disable_ramdisk=None):
result = None result = None
try: try:
if not eocn: if not eocn:
interface = getattr(task.driver, step.get('interface'))
LOG.info('Executing %(step)s on node %(node)s', LOG.info('Executing %(step)s on node %(node)s',
{'step': step, 'node': node.uuid}) {'step': step, 'node': node.uuid})
if not conductor_steps.use_reserved_step_handler(task, step): use_step_handler = conductor_steps.use_reserved_step_handler(
task, step)
if use_step_handler:
if use_step_handler == conductor_steps.EXIT_STEPS:
# Exit the step, i.e. hold step
return
# if use_step_handler == conductor_steps.USED_HANDLER
# Then we have completed the needful in the handler,
# but since there is no other value to check now,
# we know we just need to skip execute_deploy_step
else:
interface = getattr(task.driver, step.get('interface'))
result = interface.execute_clean_step(task, step) result = interface.execute_clean_step(task, step)
else: else:
LOG.info('Executing %(step)s on child nodes for node ' LOG.info('Executing %(step)s on child nodes for node '
@ -234,7 +243,6 @@ def do_next_clean_step(task, step_index, disable_ramdisk=None):
return utils.cleaning_error_handler(task, msg) return utils.cleaning_error_handler(task, msg)
LOG.info('Node %(node)s finished clean step %(step)s', LOG.info('Node %(node)s finished clean step %(step)s',
{'node': node.uuid, 'step': step}) {'node': node.uuid, 'step': step})
if CONF.agent.deploy_logs_collect == 'always' and not disable_ramdisk: if CONF.agent.deploy_logs_collect == 'always' and not disable_ramdisk:
driver_utils.collect_ramdisk_logs(task.node, label='cleaning') driver_utils.collect_ramdisk_logs(task.node, label='cleaning')

13
ironic/conductor/deployments.py
View File

@ -278,7 +278,18 @@ def do_next_deploy_step(task, step_index):
interface = getattr(task.driver, step.get('interface')) interface = getattr(task.driver, step.get('interface'))
LOG.info('Executing %(step)s on node %(node)s', LOG.info('Executing %(step)s on node %(node)s',
{'step': step, 'node': node.uuid}) {'step': step, 'node': node.uuid})
if not conductor_steps.use_reserved_step_handler(task, step): use_step_handler = conductor_steps.use_reserved_step_handler(
task, step)
if use_step_handler:
if use_step_handler == conductor_steps.EXIT_STEPS:
# Exit the step, i.e. hold step
return
# if use_step_handler == conductor_steps.USED_HANDLER
# Then we have completed the needful in the handler,
# but since there is no other value to check now,
# we know we just need to skip execute_deploy_step
else:
interface = getattr(task.driver, step.get('interface'))
result = interface.execute_deploy_step(task, step) result = interface.execute_deploy_step(task, step)
else: else:
LOG.info('Executing %(step)s on child nodes for node ' LOG.info('Executing %(step)s on child nodes for node '

27
ironic/conductor/manager.py
View File

@ -1304,10 +1304,21 @@ class ConductorManager(base_manager.BaseConductorManager):
if (action == states.VERBS['abort'] if (action == states.VERBS['abort']
and node.provision_state in (states.CLEANWAIT, and node.provision_state in (states.CLEANWAIT,
states.RESCUEWAIT, states.RESCUEWAIT,
states.INSPECTWAIT)): states.INSPECTWAIT,
states.CLEANHOLD,
states.DEPLOYHOLD)):
self._do_abort(task) self._do_abort(task)
return return
if (action == states.VERBS['unhold']
and node.provision_state in (states.CLEANHOLD,
states.DEPLOYHOLD)):
# NOTE(TheJulia): Release the node from the hold, which
# allows the next heartbeat action to pick the node back
# up and it continue it's operation.
task.process_event('unhold')
return
try: try:
task.process_event(action) task.process_event(action)
except exception.InvalidState: except exception.InvalidState:
@ -1319,7 +1330,7 @@ class ConductorManager(base_manager.BaseConductorManager):
"""Handle node abort for certain states.""" """Handle node abort for certain states."""
node = task.node node = task.node
if node.provision_state == states.CLEANWAIT: if node.provision_state in (states.CLEANWAIT, states.CLEANHOLD):
# Check if the clean step is abortable; if so abort it. # Check if the clean step is abortable; if so abort it.
# Otherwise, indicate in that clean step, that cleaning # Otherwise, indicate in that clean step, that cleaning
# should be aborted after that step is done. # should be aborted after that step is done.
@ -1368,6 +1379,18 @@ class ConductorManager(base_manager.BaseConductorManager):
if node.provision_state == states.INSPECTWAIT: if node.provision_state == states.INSPECTWAIT:
return inspection.abort_inspection(task) return inspection.abort_inspection(task)
if node.provision_state == states.DEPLOYHOLD:
# Immediately break agent API interaction
# and align with do_node_tear_down
utils.remove_agent_url(task.node)
# And then call the teardown
task.process_event(
'abort',
callback=self._spawn_worker,
call_args=(self._do_node_tear_down, task,
task.node.provision_state),
err_handler=utils.provisioning_error_handler)
@METRICS.timer('ConductorManager._sync_power_states') @METRICS.timer('ConductorManager._sync_power_states')
@periodics.periodic(spacing=CONF.conductor.sync_power_state_interval, @periodics.periodic(spacing=CONF.conductor.sync_power_state_interval,
enabled=CONF.conductor.sync_power_state_interval > 0) enabled=CONF.conductor.sync_power_state_interval > 0)

19
ironic/conductor/steps.py
View File

@ -80,6 +80,10 @@ RESERVED_STEP_HANDLER_MAPPING = {
'reboot': [utils.node_power_action, states.REBOOT], 'reboot': [utils.node_power_action, states.REBOOT],
} }
# values to enable declariation of how to handle reserved step names
USED_HANDLER = 'used_handler'
EXIT_STEPS = 'exit_steps'
def _clean_step_key(step): def _clean_step_key(step):
"""Sort by priority, then interface priority in event of tie. """Sort by priority, then interface priority in event of tie.
@ -811,7 +815,11 @@ def validate_user_deploy_steps_and_templates(task, deploy_steps=None,
def use_reserved_step_handler(task, step): def use_reserved_step_handler(task, step):
"""Returns True if reserved step execution is used, otherwise False. """Returns guidance for reserved step execution or process is used.
This method is utilized to handle some specific cases with the execution
of steps. For example, reserved step names, or reserved names which
have specific meaning in the state machine.
:param task: a TaskManager object. :param task: a TaskManager object.
:param step: The requested step. :param step: The requested step.
@ -822,5 +830,10 @@ def use_reserved_step_handler(task, step):
method = call_to_use[0] method = call_to_use[0]
parameter = call_to_use[1] parameter = call_to_use[1]
method(task, parameter) method(task, parameter)
return True return USED_HANDLER
return False if step_name == 'hold':
task.process_event('hold')
return EXIT_STEPS
# If we've reached this point, we're going to return None as
# there is no work for us to do. This allows the caller to
# take its normal path.

9
ironic/drivers/modules/agent_base.py
View File

@ -84,19 +84,22 @@ VENDOR_PROPERTIES = {
} }
__HEARTBEAT_RECORD_ONLY = (states.ENROLL, states.MANAGEABLE, states.AVAILABLE, __HEARTBEAT_RECORD_ONLY = (states.ENROLL, states.MANAGEABLE, states.AVAILABLE,
states.CLEANING, states.DEPLOYING, states.RESCUING) states.CLEANING, states.DEPLOYING, states.RESCUING,
states.DEPLOYHOLD, states.CLEANHOLD)
_HEARTBEAT_RECORD_ONLY = frozenset(__HEARTBEAT_RECORD_ONLY) _HEARTBEAT_RECORD_ONLY = frozenset(__HEARTBEAT_RECORD_ONLY)
_HEARTBEAT_ALLOWED = (states.DEPLOYWAIT, states.CLEANWAIT, states.RESCUEWAIT, _HEARTBEAT_ALLOWED = (states.DEPLOYWAIT, states.CLEANWAIT, states.RESCUEWAIT,
# These are allowed but don't cause any actions since # These are allowed but don't cause any actions since
# they're also in HEARTBEAT_RECORD_ONLY. # they're also in HEARTBEAT_RECORD_ONLY.
states.DEPLOYING, states.CLEANING, states.RESCUING) states.DEPLOYING, states.CLEANING, states.RESCUING,
states.DEPLOYHOLD, states.CLEANHOLD)
HEARTBEAT_ALLOWED = frozenset(_HEARTBEAT_ALLOWED) HEARTBEAT_ALLOWED = frozenset(_HEARTBEAT_ALLOWED)
_FASTTRACK_HEARTBEAT_ALLOWED = (states.DEPLOYWAIT, states.CLEANWAIT, _FASTTRACK_HEARTBEAT_ALLOWED = (states.DEPLOYWAIT, states.CLEANWAIT,
states.RESCUEWAIT, states.ENROLL, states.RESCUEWAIT, states.ENROLL,
states.MANAGEABLE, states.AVAILABLE, states.MANAGEABLE, states.AVAILABLE,
states.DEPLOYING) states.DEPLOYING, states.CLEANHOLD,
states.DEPLOYHOLD)
FASTTRACK_HEARTBEAT_ALLOWED = frozenset(_FASTTRACK_HEARTBEAT_ALLOWED) FASTTRACK_HEARTBEAT_ALLOWED = frozenset(_FASTTRACK_HEARTBEAT_ALLOWED)

86
ironic/tests/unit/api/controllers/v1/test_node.py
View File

@ -6403,6 +6403,92 @@ ORHMKeXMO8fcK0By7CiMKwHSXCoEQgfQhWwpMdSsO8LgHCjh87DQc= """
self.assertEqual(http_client.BAD_REQUEST, ret.status_code) self.assertEqual(http_client.BAD_REQUEST, ret.status_code)
self.assertEqual(0, mock_dpa.call_count) self.assertEqual(0, mock_dpa.call_count)
@mock.patch.object(rpcapi.ConductorAPI, 'do_provisioning_action',
autospec=True)
def test_unhold_cleanhold(self, mock_dpa):
self.node.provision_state = states.CLEANHOLD
self.node.save()
ret = self.put_json('/nodes/%s/states/provision' % self.node.uuid,
{'target': states.VERBS['unhold']},
headers={api_base.Version.string: "1.85"})
self.assertEqual(http_client.ACCEPTED, ret.status_code)
self.assertEqual(b'', ret.body)
mock_dpa.assert_called_once_with(mock.ANY, mock.ANY, self.node.uuid,
states.VERBS['unhold'],
'test-topic')
@mock.patch.object(rpcapi.ConductorAPI, 'do_provisioning_action',
autospec=True)
def test_abort_cleanhold(self, mock_dpa):
self.node.provision_state = states.CLEANHOLD
self.node.save()
ret = self.put_json('/nodes/%s/states/provision' % self.node.uuid,
{'target': states.VERBS['abort']},
headers={api_base.Version.string: "1.85"})
self.assertEqual(http_client.ACCEPTED, ret.status_code)
self.assertEqual(b'', ret.body)
mock_dpa.assert_called_once_with(mock.ANY, mock.ANY, self.node.uuid,
states.VERBS['abort'],
'test-topic')
@mock.patch.object(rpcapi.ConductorAPI, 'do_provisioning_action',
autospec=True)
def test_unhold_deployhold(self, mock_dpa):
self.node.provision_state = states.DEPLOYHOLD
self.node.save()
ret = self.put_json('/nodes/%s/states/provision' % self.node.uuid,
{'target': states.VERBS['unhold']},
headers={api_base.Version.string: "1.85"})
self.assertEqual(http_client.ACCEPTED, ret.status_code)
self.assertEqual(b'', ret.body)
mock_dpa.assert_called_once_with(mock.ANY, mock.ANY, self.node.uuid,
states.VERBS['unhold'],
'test-topic')
@mock.patch.object(rpcapi.ConductorAPI, 'do_provisioning_action',
autospec=True)
def test_abort_deployhold(self, mock_dpa):
self.node.provision_state = states.DEPLOYHOLD
self.node.save()
ret = self.put_json('/nodes/%s/states/provision' % self.node.uuid,
{'target': states.VERBS['abort']},
headers={api_base.Version.string: "1.85"})
self.assertEqual(http_client.ACCEPTED, ret.status_code)
self.assertEqual(b'', ret.body)
mock_dpa.assert_called_once_with(mock.ANY, mock.ANY, self.node.uuid,
states.VERBS['abort'],
'test-topic')
@mock.patch.object(rpcapi.ConductorAPI, 'do_provisioning_action',
autospec=True)
def test_unhold_cleanhold_not_allowed(self, mock_dpa):
self.node.provision_state = states.CLEANHOLD
self.node.save()
ret = self.put_json('/nodes/%s/states/provision' % self.node.uuid,
{'target': states.VERBS['unhold']},
headers={api_base.Version.string: "1.84"},
expect_errors=True)
self.assertEqual(http_client.NOT_ACCEPTABLE, ret.status_code)
mock_dpa.assert_not_called()
@mock.patch.object(rpcapi.ConductorAPI, 'do_provisioning_action',
autospec=True)
def test_unhold_deployhold_not_allowed(self, mock_dpa):
self.node.provision_state = states.DEPLOYHOLD
self.node.save()
ret = self.put_json('/nodes/%s/states/provision' % self.node.uuid,
{'target': states.VERBS['unhold']},
headers={api_base.Version.string: "1.84"},
expect_errors=True)
self.assertEqual(http_client.NOT_ACCEPTABLE, ret.status_code)
mock_dpa.assert_not_called()
def test_set_console_mode_enabled(self): def test_set_console_mode_enabled(self):
with mock.patch.object(rpcapi.ConductorAPI, with mock.patch.object(rpcapi.ConductorAPI,
'set_console_mode', 'set_console_mode',

51
ironic/tests/unit/conductor/test_cleaning.py
View File

@ -1136,6 +1136,36 @@ class DoNodeCleanTestCase(db_base.DbTestCase):
def test__do_next_clean_step_manual_bad_step_return_value(self): def test__do_next_clean_step_manual_bad_step_return_value(self):
self._do_next_clean_step_bad_step_return_value(manual=True) self._do_next_clean_step_bad_step_return_value(manual=True)
def _test_do_next_clean_step_handles_hold(self, start_state):
node = obj_utils.create_test_node(
self.context, driver='fake-hardware',
provision_state=start_state,
driver_internal_info={
'clean_steps': [
{
'step': 'hold',
'priority': 10,
'interface': 'power'
}
],
'clean_step_index': None},
clean_step=None)
with task_manager.acquire(
self.context, node.uuid, shared=False) as task:
cleaning.do_next_clean_step(task, 0)
node.refresh()
self.assertEqual(states.CLEANHOLD, node.provision_state)
def test_do_next_clean_step_handles_hold_from_active(self):
# Start is from the conductor
self._test_do_next_clean_step_handles_hold(states.CLEANING)
def test_do_next_clean_step_handles_hold_from_wait(self):
# Start is the continuation from a heartbeat.
self._test_do_next_clean_step_handles_hold(states.CLEANWAIT)
@mock.patch.object(cleaning, 'do_next_clean_step', autospec=True) @mock.patch.object(cleaning, 'do_next_clean_step', autospec=True)
def _continue_node_clean(self, mock_next_step, skip=True): def _continue_node_clean(self, mock_next_step, skip=True):
# test that skipping current step mechanism works # test that skipping current step mechanism works
@ -1168,7 +1198,8 @@ class DoNodeCleanTestCase(db_base.DbTestCase):
class DoNodeCleanAbortTestCase(db_base.DbTestCase): class DoNodeCleanAbortTestCase(db_base.DbTestCase):
@mock.patch.object(fake.FakeDeploy, 'tear_down_cleaning', autospec=True) @mock.patch.object(fake.FakeDeploy, 'tear_down_cleaning', autospec=True)
def _test_do_node_clean_abort(self, clean_step, tear_mock): def _test_do_node_clean_abort(self, clean_step,
tear_mock=None):
node = obj_utils.create_test_node( node = obj_utils.create_test_node(
self.context, driver='fake-hardware', self.context, driver='fake-hardware',
provision_state=states.CLEANWAIT, provision_state=states.CLEANWAIT,
@ -1228,6 +1259,24 @@ class DoNodeCleanAbortTestCase(db_base.DbTestCase):
self.assertTrue(task.node.maintenance) self.assertTrue(task.node.maintenance)
self.assertEqual('clean failure', task.node.fault) self.assertEqual('clean failure', task.node.fault)
@mock.patch.object(fake.FakeDeploy, 'tear_down_cleaning', autospec=True)
def test__do_node_cleanhold_abort_tear_down_fail(self, tear_mock):
tear_mock.side_effect = Exception('Surprise')
node = obj_utils.create_test_node(
self.context, driver='fake-hardware',
provision_state=states.CLEANHOLD,
target_provision_state=states.MANAGEABLE,
clean_step={'step': 'hold', 'abortable': True})
with task_manager.acquire(self.context, node.uuid) as task:
cleaning.do_node_clean_abort(task)
tear_mock.assert_called_once_with(task.driver.deploy, task)
self.assertIsNotNone(task.node.last_error)
self.assertIsNotNone(task.node.maintenance_reason)
self.assertTrue(task.node.maintenance)
self.assertEqual('clean failure', task.node.fault)
class DoNodeCleanTestChildNodes(db_base.DbTestCase): class DoNodeCleanTestChildNodes(db_base.DbTestCase):
def setUp(self): def setUp(self):

28
ironic/tests/unit/conductor/test_deployments.py
View File

@ -966,6 +966,34 @@ class DoNextDeployStepTestCase(mgr_utils.ServiceSetUpMixin,
mock_execute.assert_called_once_with( mock_execute.assert_called_once_with(
mock.ANY, mock.ANY, self.deploy_steps[0]) mock.ANY, mock.ANY, self.deploy_steps[0])
def _test_do_next_deploy_step_handles(self, start_state):
node = obj_utils.create_test_node(
self.context, driver='fake-hardware',
provision_state=start_state,
driver_internal_info={
'deploy_steps': [
{
'step': 'hold',
'priority': 10,
'interface': 'deploy'
}
],
'deploy_step_index': None},
clean_step=None)
with task_manager.acquire(
self.context, node.uuid, shared=False) as task:
deployments.do_next_deploy_step(task, 0)
node.refresh()
self.assertEqual(states.DEPLOYHOLD, node.provision_state)
def test_do_next_deploy_step_handles_hold_from_active(self):
# Prior step/action was out of the conductor
self._test_do_next_deploy_step_handles(states.DEPLOYING)
def test_do_next_deploy_step_handles_hold_from_wait(self):
# Prior step was async in a wait state
self._test_do_next_deploy_step_handles(states.DEPLOYWAIT)
@mock.patch.object(deployments, 'do_next_deploy_step', autospec=True) @mock.patch.object(deployments, 'do_next_deploy_step', autospec=True)
@mock.patch.object(conductor_steps, '_get_steps', autospec=True) @mock.patch.object(conductor_steps, '_get_steps', autospec=True)
def _continue_node_deploy(self, mock__get_steps, mock_next_step, def _continue_node_deploy(self, mock__get_steps, mock_next_step,

132
ironic/tests/unit/conductor/test_manager.py
View File

@ -2764,6 +2764,53 @@ class DoProvisioningActionTestCase(mgr_utils.ServiceSetUpMixin,
self.assertEqual(states.CLEANWAIT, node.provision_state) self.assertEqual(states.CLEANWAIT, node.provision_state)
self.assertEqual(states.AVAILABLE, node.target_provision_state) self.assertEqual(states.AVAILABLE, node.target_provision_state)
@mock.patch('ironic.conductor.manager.ConductorManager._spawn_worker',
autospec=True)
def _do_provision_action_abort_from_cleanhold(self, mock_spawn,
manual=False):
tgt_prov_state = states.MANAGEABLE if manual else states.AVAILABLE
node = obj_utils.create_test_node(
self.context, driver='fake-hardware',
provision_state=states.CLEANHOLD,
target_provision_state=tgt_prov_state)
self._start_service()
self.service.do_provisioning_action(self.context, node.uuid, 'abort')
node.refresh()
# Node will be moved to tgt_prov_state after cleaning, not tested here
self.assertEqual(states.CLEANFAIL, node.provision_state)
self.assertEqual(tgt_prov_state, node.target_provision_state)
self.assertEqual('By request, the clean operation was aborted',
node.last_error)
mock_spawn.assert_called_with(
self.service, cleaning.do_node_clean_abort, mock.ANY)
def test_do_provision_action_abort_cleanhold_automated_clean(self):
self._do_provision_action_abort_from_cleanhold()
def test_do_provision_action_abort_cleanhold_manual_clean(self):
self._do_provision_action_abort_from_cleanhold(manual=True)
@mock.patch('ironic.conductor.manager.ConductorManager._spawn_worker',
autospec=True)
def test_do_provision_action_abort_from_deployhold(self, mock_spawn):
node = obj_utils.create_test_node(
self.context, driver='fake-hardware',
provision_state=states.DEPLOYHOLD,
driver_internal_info={
'agent_url': 'https://foo.bar/'
})
self._start_service()
self.service.do_provisioning_action(self.context, node.uuid, 'abort')
node.refresh()
mock_spawn.assert_called_with(
self.service,
self.service._do_node_tear_down,
mock.ANY,
states.DEPLOYHOLD)
self.assertNotIn('agent_url', node.driver_internal_info)
@mgr_utils.mock_record_keepalive @mgr_utils.mock_record_keepalive
class DoNodeCleanTestCase(mgr_utils.ServiceSetUpMixin, db_base.DbTestCase): class DoNodeCleanTestCase(mgr_utils.ServiceSetUpMixin, db_base.DbTestCase):
@ -3059,6 +3106,90 @@ class DoNodeCleanTestCase(mgr_utils.ServiceSetUpMixin, db_base.DbTestCase):
self.service.continue_node_clean(self.context, node.uuid) self.service.continue_node_clean(self.context, node.uuid)
self._stop_service() self._stop_service()
@mock.patch('ironic.conductor.manager.ConductorManager._spawn_worker',
autospec=True)
def test_do_provision_action_unlocks_cleaning_manual(self, mock_spawn):
node = obj_utils.create_test_node(
self.context, driver='fake-hardware',
provision_state=states.CLEANHOLD,
target_provision_state=states.MANAGEABLE,
driver_internal_info={
'clean_steps': [
{'step': 'hold', 'priority': 9, 'interface': 'power'},
{'step': 'update_firmware', 'priority': 10,
'interface': 'power'}
],
'clean_step_index': 1
}
)
self._start_service()
self.service.do_provisioning_action(self.context, node.uuid, 'unhold')
node.refresh()
self.assertIsNone(node.last_error)
self.assertEqual(states.CLEANWAIT, node.provision_state)
self.service.continue_node_clean(self.context, node.uuid)
node.refresh()
self.assertIsNone(node.last_error)
self.assertEqual(states.CLEANING, node.provision_state)
self._stop_service()
@mock.patch('ironic.conductor.manager.ConductorManager._spawn_worker',
autospec=True)
def test_do_provision_action_unlocks_cleaning_automated(self, mock_spawn):
node = obj_utils.create_test_node(
self.context, driver='fake-hardware',
provision_state=states.CLEANHOLD,
# NOTE(TheJulia): This actually tests should this occur with
# automated cleaning. While not an explicit feature, we don't
# want things to go sideways in this case.
target_provision_state=states.AVAILABLE,
driver_internal_info={
'clean_steps': [
{'step': 'hold', 'priority': 9, 'interface': 'power'},
{'step': 'update_firmware', 'priority': 10,
'interface': 'power'}
],
'clean_step_index': 1
}
)
self._start_service()
self.service.do_provisioning_action(self.context, node.uuid, 'unhold')
node.refresh()
self.assertIsNone(node.last_error)
self.assertEqual(states.CLEANWAIT, node.provision_state)
self.service.continue_node_clean(self.context, node.uuid)
node.refresh()
self.assertIsNone(node.last_error)
self.assertEqual(states.CLEANING, node.provision_state)
self._stop_service()
@mock.patch('ironic.conductor.manager.ConductorManager._spawn_worker',
autospec=True)
def test_do_provision_action_unlocks_deploying(self, mock_spawn):
node = obj_utils.create_test_node(
self.context, driver='fake-hardware',
provision_state=states.DEPLOYHOLD,
target_provision_state=states.ACTIVE,
driver_internal_info={
'clean_steps': [
{'step': 'hold', 'priority': 9, 'interface': 'power'},
{'step': 'update_firmware', 'priority': 10,
'interface': 'power'}
],
'deploy_step_index': 1
}
)
self._start_service()
self.service.do_provisioning_action(self.context, node.uuid, 'unhold')
node.refresh()
self.assertIsNone(node.last_error)
self.assertEqual(states.DEPLOYWAIT, node.provision_state)
self.service.continue_node_deploy(self.context, node.uuid)
node.refresh()
self.assertIsNone(node.last_error)
self.assertEqual(states.DEPLOYING, node.provision_state)
self._stop_service()
class DoNodeRescueTestCase(mgr_utils.CommonMixIn, mgr_utils.ServiceSetUpMixin, class DoNodeRescueTestCase(mgr_utils.CommonMixIn, mgr_utils.ServiceSetUpMixin,
db_base.DbTestCase): db_base.DbTestCase):
@ -8275,7 +8406,6 @@ class NodeHistoryRecordCleanupTestCase(mgr_utils.ServiceSetUpMixin,
self.assertEqual(8, len(events)) self.assertEqual(8, len(events))
self.service._manage_node_history(self.context) self.service._manage_node_history(self.context)
events = objects.NodeHistory.list(self.context) events = objects.NodeHistory.list(self.context)
print(events)
self.assertEqual(6, len(events)) self.assertEqual(6, len(events))
events = objects.NodeHistory.list_by_node_id(self.context, 10) events = objects.NodeHistory.list_by_node_id(self.context, 10)
self.assertEqual(2, len(events)) self.assertEqual(2, len(events))

8
ironic/tests/unit/drivers/modules/test_agent_base.py
View File

@ -231,7 +231,8 @@ class HeartbeatMixinTest(AgentDeployMixinBaseTest):
autospec=True) autospec=True)
def test_heartbeat_noops_in_wrong_state(self, next_step_mock, log_mock): def test_heartbeat_noops_in_wrong_state(self, next_step_mock, log_mock):
allowed = {states.DEPLOYWAIT, states.CLEANWAIT, states.RESCUEWAIT, allowed = {states.DEPLOYWAIT, states.CLEANWAIT, states.RESCUEWAIT,
states.DEPLOYING, states.CLEANING, states.RESCUING} states.DEPLOYING, states.CLEANING, states.RESCUING,
states.DEPLOYHOLD, states.CLEANHOLD}
for state in set(states.machine.states) - allowed: for state in set(states.machine.states) - allowed:
for m in (next_step_mock, log_mock): for m in (next_step_mock, log_mock):
m.reset_mock() m.reset_mock()
@ -463,8 +464,9 @@ class HeartbeatMixinTest(AgentDeployMixinBaseTest):
task, 'Node failed to perform rescue operation: some failure') task, 'Node failed to perform rescue operation: some failure')
@mock.patch.object(agent_base.LOG, 'error', autospec=True) @mock.patch.object(agent_base.LOG, 'error', autospec=True)
def test_heartbeat_records_cleaning_deploying(self, log_mock): def test_heartbeat_records_when_appropriate(self, log_mock):
for provision_state in (states.CLEANING, states.DEPLOYING): for provision_state in (states.CLEANING, states.DEPLOYING,
states.CLEANHOLD, states.DEPLOYHOLD):
self.node.driver_internal_info = {} self.node.driver_internal_info = {}
self.node.provision_state = provision_state self.node.provision_state = provision_state
self.node.save() self.node.save()

18
releasenotes/notes/add-hold-states-7be5804d6f3a119a.yaml Normal file
View File

@ -0,0 +1,18 @@
---
features:
- |
Adds a ``clean hold`` and a ``deploy hold`` provision state in which
baremetal nodes can be put in utilizing specialed ``hold`` cleaning
and deployment steps. Allowing for patterns and processes where
Ironic's work is intentionally paused to allow for any external or
operator processes to take place. In these new states, a ``unhold``
provision state verb can be used to inform Ironic to proceed.
The ``abort`` verb is also a possible option should operators wish
to start over.
- |
Adds the ability to send an ``unhold`` provision state verb utilizing
API version *1.84*.
other:
- |
Fixes the generated state machine diagram and updates it to match the
current state of the code.

2
tools/states_to_dot.py
View File

@ -1,4 +1,4 @@
#!/usr/bin/env python #!/usr/bin/env python3
# Copyright (C) 2014 Yahoo! Inc. All Rights Reserved. # Copyright (C) 2014 Yahoo! Inc. All Rights Reserved.
# #

3
tox.ini
View File

@ -35,6 +35,7 @@ deps = {[testenv]deps}
-r{toxinidir}/driver-requirements.txt -r{toxinidir}/driver-requirements.txt
[testenv:genstates] [testenv:genstates]
allowlist_externals = {toxinidir}/tools/states_to_dot.py
deps = {[testenv]deps} deps = {[testenv]deps}
pydot2 pydot2
commands = {toxinidir}/tools/states_to_dot.py -f {toxinidir}/doc/source/images/states.svg --format svg commands = {toxinidir}/tools/states_to_dot.py -f {toxinidir}/doc/source/images/states.svg --format svg
@ -92,7 +93,7 @@ deps =
-c{env:TOX_CONSTRAINTS_FILE:https://releases.openstack.org/constraints/upper/master} -c{env:TOX_CONSTRAINTS_FILE:https://releases.openstack.org/constraints/upper/master}
-r{toxinidir}/requirements.txt -r{toxinidir}/requirements.txt
-r{toxinidir}/doc/requirements.txt -r{toxinidir}/doc/requirements.txt
commands = sphinx-build -b html -W doc/source doc/build/html commands = sphinx-build -b html doc/source doc/build/html
[testenv:pdf-docs] [testenv:pdf-docs]
allowlist_externals = make allowlist_externals = make