airship
/
specs
Code Issues Proposed changes

Move specs: airship-in-a-bottle to airship-specs

Browse Source 
Moves the two blueprint/spec documents that existed in
airship-in-a-bottle to the airship-specs. The implemented spec was not
reformatted to the spec template. The other spec (in approved folder)
was minimally updated to the spec template.

Change-Id: I7468579e2fa3077ee1144e5294eba97d8e4ced05
changes/45/587945/1
Bryan Strassner 5 years ago
parent 6e0a18e7fa
commit bfbfd56c81
2 changed files with 1189 additions and 0 deletions
Show Stats Download Patch File Download Diff File

620
specs/approved/workflow_node-teardown.rst
Unescape View File

@ -0,0 +1,620 @@
..
Copyright 2018 AT&T Intellectual Property.
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
.. index::
single: Teardown node
single: workflow;redeploy_server
single: Drydock
single: Promenade
single: Shipyard
.. _node-teardown:
=====================
Airship Node Teardown
=====================
Shipyard is the entrypoint for Airship actions, including the need to redeploy a
server. The first part of redeploying a server is the graceful teardown of the
software running on the server; specifically Kubernetes and etcd are of
critical concern. It is the duty of Shipyard to orchestrate the teardown of the
server, followed by steps to deploy the desired new configuration. This design
covers only the first portion - node teardown
Links
=====
None
Problem description
===================
When redeploying a physical host (server) using the Airship Platform,
it is necessary to trigger a sequence of steps to prevent undesired behaviors
when the server is redeployed. This blueprint intends to document the
interaction that must occur between Airship components to teardown a server.
Impacted components
===================
Drydock
Promenade
Shipyard
Proposed change
===============
Shipyard node teardown Process
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#. (Existing) Shipyard receives request to redeploy_server, specifying a target
server.
#. (Existing) Shipyard performs preflight, design reference lookup, and
validation steps.
#. (New) Shipyard invokes Promenade to decommission a node.
#. (New) Shipyard invokes Drydock to destroy the node - setting a node
filter to restrict to a single server.
#. (New) Shipyard invokes Promenade to remove the node from the Kubernetes
cluster.
Assumption:
node_id is the hostname of the server, and is also the identifier that both
Drydock and Promenade use to identify the appropriate parts - hosts and k8s
nodes. This convention is set by the join script produced by promenade.
Drydock Destroy Node
--------------------
The API/interface for destroy node already exists. The implementation within
Drydock needs to be developed. This interface will need to accept both the
specified node_id and the design_id to retrieve from Deckhand.
Using the provided node_id (hardware node), and the design_id, Drydock will
reset the hardware to a re-provisionable state.
By default, all local storage should be wiped (per datacenter policy for
wiping before re-use).
An option to allow for only the OS disk to be wiped should be supported, such
that other local storage is left intact, and could be remounted without data
loss. e.g.: --preserve-local-storage
The target node should be shut down.
The target node should be removed from the provisioner (e.g. MaaS)
Responses
~~~~~~~~~
The responses from this functionality should follow the pattern set by prepare
nodes, and other Drydock functionality. The Drydock status responses used for
all async invocations will be utilized for this functionality.
Promenade Decommission Node
---------------------------
Performs steps that will result in the specified node being cleanly
disassociated from Kubernetes, and ready for the server to be destroyed.
Users of the decommission node API should be aware of the long timeout values
that may occur while awaiting promenade to complete the appropriate steps.
At this time, Promenade is a stateless service and doesn't use any database
storage. As such, requests to Promenade are synchronous.
.. code:: json
POST /nodes/{node_id}/decommission
{
rel : "design",
href: "deckhand+https://{{deckhand_url}}/revisions/{{revision_id}}/rendered-documents",
type: "application/x-yaml"
}
Such that the design reference body is the design indicated when the
redeploy_server action is invoked through Shipyard.
Query Parameters:
- drain-node-timeout: A whole number timeout in seconds to be used for the
drain node step (default: none). In the case of no value being provided,
the drain node step will use its default.
- drain-node-grace-period: A whole number in seconds indicating the
grace-period that will be provided to the drain node step. (default: none).
If no value is specified, the drain node step will use its default.
- clear-labels-timeout: A whole number timeout in seconds to be used for the
clear labels step. (default: none). If no value is specified, clear labels
will use its own default.
- remove-etcd-timeout: A whole number timeout in seconds to be used for the
remove etcd from nodes step. (default: none). If no value is specified,
remove-etcd will use its own default.
- etcd-ready-timeout: A whole number in seconds indicating how long the
decommission node request should allow for etcd clusters to become stable
(default: 600).
Process
~~~~~~~
Acting upon the node specified by the invocation and the design reference
details:
#. Drain the Kubernetes node.
#. Clear the Kubernetes labels on the node.
#. Remove etcd nodes from their clusters (if impacted).
- if the node being decommissioned contains etcd nodes, Promenade will
attempt to gracefully have those nodes leave the etcd cluster.
#. Ensure that etcd cluster(s) are in a stable state.
- Polls for status every 30 seconds up to the etcd-ready-timeout, or the
cluster meets the defined minimum functionality for the site.
- A new document: promenade/EtcdClusters/v1 that will specify details about
the etcd clusters deployed in the site, including: identifiers,
credentials, and thresholds for minimum functionality.
- This process should ignore the node being torn down from any calculation
of health
#. Shutdown the kubelet.
- If this is not possible because the node is in a state of disarray such
that it cannot schedule the daemonset to run, this step may fail, but
should not hold up the process, as the Drydock dismantling of the node
will shut the kubelet down.
Responses
~~~~~~~~~
All responses will be form of the Airship Status response.
- Success: Code: 200, reason: Success
Indicates that all steps are successful.
- Failure: Code: 404, reason: NotFound
Indicates that the target node is not discoverable by Promenade.
- Failure: Code: 500, reason: DisassociateStepFailure
The details section should detail the successes and failures further. Any
4xx series errors from the individual steps would manifest as a 500 here.
Promenade Drain Node
--------------------
Drain the Kubernetes node for the target node. This will ensure that this node
is no longer the target of any pod scheduling, and evicts or deletes the
running pods. In the case of notes running DaemonSet manged pods, or pods
that would prevent a drain from occurring, Promenade may be required to provide
the `ignore-daemonsets` option or `force` option to attempt to drain the node
as fully as possible.
By default, the drain node will utilize a grace period for pods of 1800
seconds and a total timeout of 3600 seconds (1 hour). Clients of this
functionality should be prepared for a long timeout.
.. code:: json
POST /nodes/{node_id}/drain
Query Paramters:
- timeout: a whole number in seconds (default = 3600). This value is the total
timeout for the kubectl drain command.
- grace-period: a whole number in seconds (default = 1800). This value is the
grace period used by kubectl drain. Grace period must be less than timeout.
.. note::
This POST has no message body
Example command being used for drain (reference only)
`kubectl drain --force --timeout 3600s --grace-period 1800 --ignore-daemonsets --delete-local-data n1`
https://git.openstack.org/cgit/openstack/airship-promenade/tree/promenade/templates/roles/common/usr/local/bin/promenade-teardown
Responses
~~~~~~~~~
All responses will be form of the Airship Status response.
- Success: Code: 200, reason: Success
Indicates that the drain node has successfully concluded, and that no pods
are currently running
- Failure: Status response, code: 400, reason: BadRequest
A request was made with parameters that cannot work - e.g. grace-period is
set to a value larger than the timeout value.
- Failure: Status response, code: 404, reason: NotFound
The specified node is not discoverable by Promenade
- Failure: Status response, code: 500, reason: DrainNodeError
There was a processing exception raised while trying to drain a node. The
details section should indicate the underlying cause if it can be
determined.
Promenade Clear Labels
----------------------
Removes the labels that have been added to the target kubernetes node.
.. code:: json
POST /nodes/{node_id}/clear-labels
Query Parameters:
- timeout: A whole number in seconds allowed for the pods to settle/move
following removal of labels. (Default = 1800)
.. note::
This POST has no message body
Responses
~~~~~~~~~
All responses will be form of the UCP Status response.
- Success: Code: 200, reason: Success
All labels have been removed from the specified Kubernetes node.
- Failure: Code: 404, reason: NotFound
The specified node is not discoverable by Promenade
- Failure: Code: 500, reason: ClearLabelsError
There was a failure to clear labels that prevented completion. The details
section should provide more information about the cause of this failure.
Promenade Remove etcd Node
~~~~~~~~~~~~~~~~~~~~~~~~~~
Checks if the node specified contains any etcd nodes. If so, this API will
trigger that etcd node to leave the associated etcd cluster::
POST /nodes/{node_id}/remove-etcd
{
rel : "design",
href: "deckhand+https://{{deckhand_url}}/revisions/{{revision_id}}/rendered-documents",
type: "application/x-yaml"
}
Query Parameters:
- timeout: A whole number in seconds allowed for the removal of etcd nodes
from the targe node. (Default = 1800)
Responses
~~~~~~~~~
All responses will be form of the UCP Status response.
- Success: Code: 200, reason: Success
All etcd nodes have been removed from the specified node.
- Failure: Code: 404, reason: NotFound
The specified node is not discoverable by Promenade
- Failure: Code: 500, reason: RemoveEtcdError
There was a failure to remove etcd from the target node that prevented
completion within the specified timeout, or that etcd prevented removal of
the node because it would result in the cluster being broken. The details
section should provide more information about the cause of this failure.
Promenade Check etcd
~~~~~~~~~~~~~~~~~~~~
Retrieves the current interpreted state of etcd.
GET /etcd-cluster-health-statuses?design_ref={the design ref}
Where the design_ref parameter is required for appropriate operation, and is in
the same format as used for the join-scripts API.
Query Parameters:
- design_ref: (Required) the design reference to be used to discover etcd
instances.
Responses
~~~~~~~~~
All responses will be form of the UCP Status response.
- Success: Code: 200, reason: Success
The status of each etcd in the site will be returned in the details section.
Valid values for status are: Healthy, Unhealthy
https://github.com/att-comdev/ucp-integration/blob/master/docs/source/api-conventions.rst#status-responses
.. code:: json
{ "...": "... standard status response ...",
"details": {
"errorCount": {{n}},
"messageList": [
{ "message": "Healthy",
"error": false,
"kind": "HealthMessage",
"name": "{{the name of the etcd service}}"
},
{ "message": "Unhealthy"
"error": false,
"kind": "HealthMessage",
"name": "{{the name of the etcd service}}"
},
{ "message": "Unable to access Etcd"
"error": true,
"kind": "HealthMessage",
"name": "{{the name of the etcd service}}"
}
]
}
...
}
- Failure: Code: 400, reason: MissingDesignRef
Returned if the design_ref parameter is not specified
- Failure: Code: 404, reason: NotFound
Returned if the specified etcd could not be located
- Failure: Code: 500, reason: EtcdNotAccessible
Returned if the specified etcd responded with an invalid health response
(Not just simply unhealthy - that's a 200).
Promenade Shutdown Kubelet
--------------------------
Shuts down the kubelet on the specified node. This is accomplished by Promenade
setting the label `promenade-decomission: enabled` on the node, which will
trigger a newly-developed daemonset to run something like:
`systemctl disable kubelet && systemctl stop kubelet`.
This daemonset will effectively sit dormant until nodes have the appropriate
label added, and then perform the kubelet teardown.
.. code:: json
POST /nodes/{node_id}/shutdown-kubelet
.. note::
This POST has no message body
Responses
~~~~~~~~~
All responses will be form of the UCP Status response.
- Success: Code: 200, reason: Success
The kubelet has been successfully shutdown
- Failure: Code: 404, reason: NotFound
The specified node is not discoverable by Promenade
- Failure: Code: 500, reason: ShutdownKubeletError
The specified node's kubelet fails to shutdown. The details section of the
status response should contain reasonable information about the source of
this failure
Promenade Delete Node from Cluster
----------------------------------
Updates the Kubernetes cluster, removing the specified node. Promenade should
check that the node is drained/cordoned and has no labels other than
`promenade-decomission: enabled`. In either of these cases, the API should
respond with a 409 Conflict response.
.. code:: json
POST /nodes/{node_id}/remove-from-cluster
.. note::
This POST has no message body
Responses
~~~~~~~~~
All responses will be form of the UCP Status response.
- Success: Code: 200, reason: Success
The specified node has been removed from the Kubernetes cluster.
- Failure: Code: 404, reason: NotFound
The specified node is not discoverable by Promenade
- Failure: Code: 409, reason: Conflict
The specified node cannot be deleted due to checks that the node is
drained/cordoned and has no labels (other than possibly
`promenade-decomission: enabled`).
- Failure: Code: 500, reason: DeleteNodeError
The specified node cannot be removed from the cluster due to an error from
Kubernetes. The details section of the status response should contain more
information about the failure.
Shipyard Tag Releases
---------------------
Shipyard will need to mark Deckhand revisions with tags when there are
successful deploy_site or update_site actions to be able to determine the last
known good design. This is related to issue 16 for Shipyard, which utilizes the
same need.
.. note::
Repeated from https://github.com/att-comdev/shipyard/issues/16
When multiple configdocs commits have been done since the last deployment,
there is no ready means to determine what's being done to the site. Shipyard
should reject deploy site or update site requests that have had multiple
commits since the last site true-up action. An option to override this guard
should be allowed for the actions in the form of a parameter to the action.
The configdocs API should provide a way to see what's been changed since the
last site true-up, not just the last commit of configdocs. This might be
accommodated by new deckhand tags like the 'commit' tag, but for
'site true-up' or similar applied by the deploy and update site commands.
The design for issue 16 includes the bare-minimum marking of Deckhand
revisions. This design is as follows:
Scenario
~~~~~~~~
Multiple commits occur between site actions (deploy_site, update_site) - those
actions that attempt to bring a site into compliance with a site design.
When this occurs, the current system of being able to only see what has changed
between committed and the the buffer versions (configdocs diff) is insufficient
to be able to investigate what has changed since the last successful (or
unsuccessful) site action.
To accommodate this, Shipyard needs several enhancements.
Enhancements
~~~~~~~~~~~~
#. Deckhand revision tags for site actions
Using the tagging facility provided by Deckhand, Shipyard will tag the end
of site actions.
Upon completing a site action successfully tag the revision being used with
the tag site-action-success, and a body of dag_id:<dag_id>
Upon completion of a site action unsuccessfully, tag the revision being used
with the tag site-action-failure, and a body of dag_id:<dag_id>
The completion tags should only be applied upon failure if the site action
gets past document validation successfully (i.e. gets to the point where it
can start making changes via the other UCP components)
This could result in a single revision having both site-action-success and
site-action-failure if a later re-invocation of a site action is successful.
#. Check for intermediate committed revisions
Upon running a site action, before tagging the revision with the site action
tag(s), the dag needs to check to see if there are committed revisions that
do not have an associated site-action tag. If there are any committed
revisions since the last site action other than the current revision being
used (between them), then the action should not be allowed to proceed (stop
before triggering validations). For the calculation of intermediate
committed revisions, assume revision 0 if there are no revisions with a
site-action tag (null case)
If the action is invoked with a parameter of
allow-intermediate-commits=true, then this check should log that the
intermediate committed revisions check is being skipped and not take any
other action.
#. Support action parameter of allow-intermediate-commits=true|false
In the CLI for create action, the --param option supports adding parameters
to actions. The parameters passed should be relayed by the CLI to the API
and ultimately to the invocation of the DAG. The DAG as noted above will
check for the presense of allow-intermediate-commits=true. This needs to be
tested to work.
#. Shipyard needs to support retrieving configdocs and rendered documents for
the last successful site action, and last site action (successful or not
successful)
--successful-site-action
--last-site-action
These options would be mutually exclusive of --buffer or --committed
#. Shipyard diff (shipyard get configdocs)
Needs to support an option to do the diff of the buffer vs. the last
successful site action and the last site action (succesful or not
successful).
Currently there are no options to select which versions to diff (always
buffer vs. committed)
support:
--base-version=committed | successful-site-action | last-site-action (Default = committed)
--diff-version=buffer | committed | successful-site-action | last-site-action (Default = buffer)
Equivalent query parameters need to be implemented in the API.
Because the implementation of this design will result in the tagging of
successful site-actions, Shipyard will be able to determine the correct
revision to use while attempting to teardown a node.
If the request to teardown a node indicates a revision that doesn't exist, the
command to do so (e.g. redeploy_server) should not continue, but rather fail
due to a missing precondition.
The invocation of the Promenade and Drydock steps in this design will utilize
the appropriate tag based on the request (default is successful-site-action) to
determine the revision of the Deckhand documents used as the design-ref.
Shipyard redeploy_server Action
-------------------------------
The redeploy_server action currently accepts a target node. Additional
supported parameters are needed:
#. preserve-local-storage=true which will instruct Drydock to only wipe the
OS drive, and any other local storage will not be wiped. This would allow
for the drives to be remounted to the server upon re-provisioning. The
default behavior is that local storage is not preserved.
#. target-revision=committed | successful-site-action | last-site-action
This will indicate which revision of the design will be used as the
reference for what should be re-provisioned after the teardown.
The default is successful-site-action, which is the closest representation
to the last-known-good state.
These should be accepted as parameters to the action API/CLI and modify the
behavior of the redeploy_server DAG.
Security impact
---------------
None. This change introduces no new security concerns outside of established
patterns for RBAC controls around API endpoints.
Performance impact
------------------
As this is an on-demand action, there is no expected performance impact to
existing processes, although tearing down a host may result in temporary
degraded service capacity in the case of needing to move workloads to different
hosts, or a more simple case of reduced capacity.
Alternatives
------------
N/A
Implementation
==============
None at this time
Dependencies
============
None.
References
==========
None

569
specs/implemented/deployment-grouping-baremetal.rst
Unescape View File

@ -0,0 +1,569 @@
..
Copyright 2018 AT&T Intellectual Property.
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
.. index::
single: Deployment grouping
single: workflow
single: Shipyard
single: Drydock
.. _deployment-grouping-baremetal:
=======================================
Deployment Grouping for Baremetal Nodes
=======================================
One of the primary functionalities of the Undercloud Platform is the deployment
of baremetal nodes as part of site deployment and upgrade. This blueprint aims
to define how deployment strategies can be applied to the workflow during these
actions.
.. note::
This document has been moved from the airship-in-a-bottle project, and is
previously implemented. The format of this document diverges from the
standard template for airship-specs.
Overview
--------
When Shipyard is invoked for a deploy_site or update_site action, there are
three primary stages:
1. Preparation and Validation
2. Baremetal and Network Deployment
3. Software Deployment
During the Baremetal and Network Deployment stage, the deploy_site or
update_site workflow (and perhaps other workflows in the future) invokes
Drydock to verify the site, prepare the site, prepare the nodes, and deploy the
nodes. Each of these steps is described in the `Drydock Orchestrator Readme`_
.. _Drydock Orchestrator Readme: https://git.openstack.org/cgit/openstack/airship-drydock/plain/drydock_provisioner/orchestrator/readme.md
The prepare nodes and deploy nodes steps each involve intensive and potentially
time consuming operations on the target nodes, orchestrated by Drydock and
MAAS. These steps need to be approached and managed such that grouping,
ordering, and criticality of success of nodes can be managed in support of
fault tolerant site deployments and updates.
For the purposes of this document `phase of deployment` refer to the prepare
nodes and deploy nodes steps of the Baremetal and Network deployment.
Some factors that advise this solution:
1. Limits to the amount of parallelization that can occur due to a centralized
MAAS system.
2. Faults in the hardware, preventing operational nodes.
3. Miswiring or configuration of network hardware.
4. Incorrect site design causing a mismatch against the hardware.
5. Criticality of particular nodes to the realization of the site design.
6. Desired configurability within the framework of the UCP declarative site
design.
7. Improved visibility into the current state of node deployment.
8. A desire to begin the deployment of nodes before the finish of the
preparation of nodes -- i.e. start deploying nodes as soon as they are ready
to be deployed. Note: This design will not achieve new forms of
task parallelization within Drydock; this is recognized as a desired
functionality.
Solution
--------
Updates supporting this solution will require changes to Shipyard for changed
workflows and Drydock for the desired node targeting, and for retrieval of
diagnostic and result information.
.. index::
single: Shipyard Documents; DeploymentStrategy
Deployment Strategy Document (Shipyard)
---------------------------------------
To accommodate the needed changes, this design introduces a new
DeploymentStrategy document into the site design to be read and utilized
by the workflows for update_site and deploy_site.
Groups
~~~~~~
Groups are named sets of nodes that will be deployed together. The fields of a
group are:
name
Required. The identifying name of the group.
critical
Required. Indicates if this group is required to continue to additional
phases of deployment.
depends_on
Required, may be empty list. Group names that must be successful before this
group can be processed.
selectors
Required, may be empty list. A list of identifying information to indicate
the nodes that are members of this group.
success_criteria
Optional. Criteria that must evaluate to be true before a group is considered
successfully complete with a phase of deployment.
Criticality
'''''''''''
- Field: critical
- Valid values: true | false
Each group is required to indicate true or false for the `critical` field.
This drives the behavior after the deployment of baremetal nodes. If any
groups that are marked as `critical: true` fail to meet that group's success
criteria, the workflow should halt after the deployment of baremetal nodes. A
group that cannot be processed due to a parent dependency failing will be
considered failed, regardless of the success criteria.
Dependencies
''''''''''''
- Field: depends_on
- Valid values: [] or a list of group names
Each group specifies a list of depends_on groups, or an empty list. All
identified groups must complete successfully for the phase of deployment before
the current group is allowed to be processed by the current phase.
- A failure (based on success criteria) of a group prevents any groups
dependent upon the failed group from being attempted.
- Circular dependencies will be rejected as invalid during document validation.
- There is no guarantee of ordering among groups that have their dependencies
met. Any group that is ready for deployment based on declared dependencies
will execute. Execution of groups is serialized - two groups will not deploy
at the same time.
Selectors
'''''''''
- Field: selectors
- Valid values: [] or a list of selectors
The list of selectors indicate the nodes that will be included in a group.
Each selector has four available filtering values: node_names, node_tags,
node_labels, and rack_names. Each selector is an intersection of this
critera, while the list of selectors is a union of the individual selectors.
- Omitting a criterion from a selector, or using empty list means that criterion
is ignored.
- Having a completely empty list of selectors, or a selector that has no
criteria specified indicates ALL nodes.
- A collection of selectors that results in no nodes being identified will be
processed as if 100% of nodes successfully deployed (avoiding division by
zero), but would fail the minimum or maximum nodes criteria (still counts as
0 nodes)
- There is no validation against the same node being in multiple groups,
however the workflow will not resubmit nodes that have already completed or
failed in this deployment to Drydock twice, since it keeps track of each node
uniquely. The success or failure of those nodes excluded from submission to
Drydock will still be used for the success criteria calculation.
E.g.::
selectors:
- node_names:
- node01
- node02
rack_names:
- rack01
node_tags:
- control
- node_names:
- node04
node_labels:
- ucp_control_plane: enabled
Will indicate (not really SQL, just for illustration)::
SELECT nodes
WHERE node_name in ('node01', 'node02')
AND rack_name in ('rack01')
AND node_tags in ('control')
UNION
SELECT nodes
WHERE node_name in ('node04')
AND node_label in ('ucp_control_plane: enabled')
Success Criteria
''''''''''''''''
- Field: success_criteria
- Valid values: for possible values, see below
Each group optionally contains success criteria which is used to indicate if
the deployment of that group is successful. The values that may be specified:
percent_successful_nodes
The calculated success rate of nodes completing the deployment phase.
E.g.: 75 would mean that 3 of 4 nodes must complete the phase successfully.
This is useful for groups that have larger numbers of nodes, and do not
have critical minimums or are not sensitive to an arbitrary number of nodes
not working.
minimum_successful_nodes
An integer indicating how many nodes must complete the phase to be considered
successful.
maximum_failed_nodes
An integer indicating a number of nodes that are allowed to have failed the
deployment phase and still consider that group successful.
When no criteria are specified, it means that no checks are done - processing
continues as if nothing is wrong.
When more than one criterion is specified, each is evaluated separately - if
any fail, the group is considered failed.
Example Deployment Strategy Document
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This example shows a deployment strategy with 5 groups: control-nodes,
compute-nodes-1, compute-nodes-2, monitoring-nodes, and ntp-node.
::
---
schema: shipyard/DeploymentStrategy/v1
metadata:
schema: metadata/Document/v1
name: deployment-strategy
layeringDefinition:
abstract: false
layer: global
storagePolicy: cleartext
data:
groups:
- name: control-nodes
critical: true
depends_on:
- ntp-node
selectors:
- node_names: []
node_labels: []
node_tags:
- control
rack_names:
- rack03
success_criteria:
percent_successful_nodes: 90
minimum_successful_nodes: 3
maximum_failed_nodes: 1
- name: compute-nodes-1
critical: false
depends_on:
- control-nodes
selectors:
- node_names: []
node_labels: []
rack_names:
- rack01
node_tags:
- compute
success_criteria:
percent_successful_nodes: 50
- name: compute-nodes-2
critical: false
depends_on:
- control-nodes
selectors:
- node_names: []
node_labels: []
rack_names:
- rack02
node_tags:
- compute
success_criteria:
percent_successful_nodes: 50
- name: monitoring-nodes
critical: false
depends_on: []
selectors:
- node_names: []
node_labels: []
node_tags:
- monitoring
rack_names:
- rack03
- rack02
- rack01
- name: ntp-node
critical: true
depends_on: []
selectors:
- node_names:
- ntp01
node_labels: []
node_tags: []
rack_names: []
success_criteria:
minimum_successful_nodes: 1
The ordering of groups, as defined by the dependencies (``depends-on``
fields)::
__________ __________________
| ntp-node | | monitoring-nodes |
---------- ------------------
|
____V__________
| control-nodes |
---------------
|_________________________
| |
______V__________ ______V__________
| compute-nodes-1 | | compute-nodes-2 |
----------------- -----------------
Given this, the order of execution could be:
- ntp-node > monitoring-nodes > control-nodes > compute-nodes-1 > compute-nodes-2
- ntp-node > control-nodes > compute-nodes-2 > compute-nodes-1 > monitoring-nodes
- monitoring-nodes > ntp-node > control-nodes > compute-nodes-1 > compute-nodes-2
- and many more ... the only guarantee is that ntp-node will run some time
before control-nodes, which will run sometime before both of the
compute-nodes. Monitoring-nodes can run at any time.
Also of note are the various combinations of selectors and the varied use of
success criteria.
Deployment Configuration Document (Shipyard)
--------------------------------------------
The existing deployment-configuration document that is used by the workflows
will also be modified to use the existing deployment_strategy field to provide
the name of the deployment-straegy document that will be used.
The default value for the name of the DeploymentStrategy document will be
``deployment-strategy``.
Drydock Changes
---------------
API and CLI
~~~~~~~~~~~
- A new API needs to be provided that accepts a node filter (i.e. selector,
above) and returns a list of node names that result from analysis of the
design. Input to this API will also need to include a design reference.
- Drydock needs to provide a "tree" output of tasks rooted at the requested
parent task. This will provide the needed success/failure status for nodes
that have been prepared/deployed.
Documentation
~~~~~~~~~~~~~
Drydock documentation will be updated to match the introduction of new APIs
Shipyard Changes
----------------
API and CLI
~~~~~~~~~~~
- The commit configdocs api will need to be enhanced to look up the
DeploymentStrategy by using the DeploymentConfiguration.
- The DeploymentStrategy document will need to be validated to ensure there are
no circular dependencies in the groups' declared dependencies (perhaps
NetworkX_).
- A new API endpoint (and matching CLI) is desired to retrieve the status of
nodes as known to Drydock/MAAS and their MAAS status. The existing node list
API in Drydock provides a JSON output that can be utilized for this purpose.
Workflow
~~~~~~~~
The deploy_site and update_site workflows will be modified to utilize the
DeploymentStrategy.
- The deployment configuration step will be enhanced to also read the
deployment strategy and pass the information on a new xcom for use by the
baremetal nodes step (see below)
- The prepare nodes and deploy nodes steps will be combined to perform both as
part of the resolution of an overall ``baremetal nodes`` step.
The baremetal nodes step will introduce functionality that reads in the
deployment strategy (from the prior xcom), and can orchestrate the calls to
Drydock to enact the grouping, ordering and and success evaluation.
Note that Drydock will serialize tasks; there is no parallelization of
prepare/deploy at this time.
Needed Functionality
''''''''''''''''''''
- function to formulate the ordered groups based on dependencies (perhaps
NetworkX_)
- function to evaluate success/failure against the success criteria for a group
based on the result list of succeeded or failed nodes.
- function to mark groups as success or failure (including failed due to
dependency failure), as well as keep track of the (if any) successful and
failed nodes.
- function to get a group that is ready to execute, or 'Done' when all groups
are either complete or failed.
- function to formulate the node filter for Drydock based on a group's
selectors
- function to orchestrate processing groups, moving to the next group (or being
done) when a prior group completes or fails.
- function to summarize the success/failed nodes for a group (primarily for
reporting to the logs at this time).
Process
'''''''
The baremetal nodes step (preparation and deployment of nodes) will proceed as
follows:
1. Each group's selector will be sent to Drydock to determine the list of
nodes that are a part of that group.
- An overall status will be kept for each unique node (not started |
prepared | success | failure).
- When sending a task to Drydock for processing, the nodes associated with
that group will be sent as a simple `node_name` node filter. This will
allow for this list to exclude nodes that have a status that is not
congruent for the task being performed.
- prepare nodes valid status: not started
- deploy nodes valid status: prepared
2. In a processing loop, groups that are ready to be processed based on their
dependencies (and the success criteria of groups they are dependent upon)
will be selected for processing until there are no more groups that can be
processed. The processing will consist of preparing and then deploying the
group.
- The selected group will be prepared and then deployed before selecting
another group for processing.
- Any nodes that failed as part of that group will be excluded from
subsequent deployment or preparation of that node for this deployment.
- Excluding nodes that are already processed addresses groups that have
overlapping lists of nodes due to the group's selectors, and prevents
sending them to Drydock for re-processing.
- Evaluation of the success criteria will use the full set of nodes
identified by the selector. This means that if a node was previously
successfully deployed, that same node will count as "successful" when
evaluating the success criteria.
- The success criteria will be evaluated after the group's prepare step and
the deploy step. A failure to meet the success criteria in a prepare step
will cause the deploy step for that group to be skipped (and marked as
failed).
- Any nodes that fail during the prepare step, will not be used in the
corresponding deploy step.
- Upon completion (success, partial success, or failure) of a prepare step,
the nodes that were sent for preparation will be marked in the unique list
of nodes (above) with their appropriate status: prepared or failure
- Upon completion of a group's deployment step, the nodes status will be
updated to their current status: success or failure.
4. Before the end of the baremetal nodes step, following all eligible group
processing, a report will be logged to indicate the success/failure of
groups and the status of the individual nodes. Note that it is possible for
individual nodes to be left in `not started` state if they were only part of
groups that were never allowed to process due to dependencies and success
criteria.
5. At the end of the baremetal nodes step, if any nodes that have failed
due to timeout, dependency failure, or success criteria failure and are
marked as critical will trigger an Airflow Exception, resulting in a failed
deployment.
Notes:
- The timeout values specified for the prepare nodes and deploy nodes steps
will be used to put bounds on the individual calls to Drydock. A failure
based on these values will be treated as a failure for the group; we need to
be vigilant on if this will lead to indeterminate states for nodes that mess
with further processing. (e.g. Timed out, but the requested work still
continued to completion)
Example Processing
''''''''''''''''''
Using the defined deployment strategy in the above example, the following is
an example of how it may process::
Start
|
| prepare ntp-node <SUCCESS>
| deploy ntp-node <SUCCESS>
V
| prepare control-nodes <SUCCESS>
| deploy control-nodes <SUCCESS>
V
| prepare monitoring-nodes <SUCCESS>
| deploy monitoring-nodes <SUCCESS>
V
| prepare compute-nodes-2 <SUCCESS>
| deploy compute-nodes-2 <SUCCESS>
V
| prepare compute-nodes-1 <SUCCESS>
| deploy compute-nodes-1 <SUCCESS>
|
Finish (success)
If there were a failure in preparing the ntp-node, the following would be the
result::
Start
|
| prepare ntp-node <FAILED>
| deploy ntp-node <FAILED, due to prepare failure>
V
| prepare control-nodes <FAILED, due to dependency>
| deploy control-nodes <FAILED, due to dependency>
V
| prepare monitoring-nodes <SUCCESS>
| deploy monitoring-nodes <SUCCESS>
V
| prepare compute-nodes-2 <FAILED, due to dependency>
| deploy compute-nodes-2 <FAILED, due to dependency>
V
| prepare compute-nodes-1 <FAILED, due to dependency>
| deploy compute-nodes-1 <FAILED, due to dependency>
|
Finish (failed due to critical group failed)
If a failure occurred during the deploy of compute-nodes-2, the following would
result::
Start
|
| prepare ntp-node <SUCCESS>
| deploy ntp-node <SUCCESS>
V
| prepare control-nodes <SUCCESS>
| deploy control-nodes <SUCCESS>
V
| prepare monitoring-nodes <SUCCESS>
| deploy monitoring-nodes <SUCCESS>
V
| prepare compute-nodes-2 <SUCCESS>
| deploy compute-nodes-2 <FAILED>
V
| prepare compute-nodes-1 <SUCCESS>
| deploy compute-nodes-1 <SUCCESS>
|
Finish (success with some nodes/groups failed)
Schemas
~~~~~~~
A new schema will need to be provided by Shipyard to validate the
DeploymentStrategy document.
Documentation
~~~~~~~~~~~~~
The Shipyard action documentation will need to include details defining the
DeploymentStrategy document (mostly as defined here), as well as the update to
the DeploymentConfiguration document to contain the name of the
DeploymentStrategy document.
.. _NetworkX: https://networkx.github.io/documentation/networkx-1.9/reference/generated/networkx.algorithms.dag.topological_sort.html
Write Preview
Loading…
Cancel
Save