From c5064ef2eb47a4872a2b46380ca1640be9670b04 Mon Sep 17 00:00:00 2001 From: Roman Gorshunov Date: Wed, 13 Mar 2019 16:12:45 +0100 Subject: [PATCH] Fix docs renderring, enforce instructions and template This patch applies various documentation renderring fixes, and enforces application of the instructions and the template for the file names. In addition to that it adds a requirement to add patches related to the spec under specified Gerrit topics. Change-Id: I36199cf78c30f2ee75c2d716b8919ceae2ab7c42 --- .gitignore | 1 + ...os.rst => airship_multi_linux_distros.rst} | 20 +- specs/approved/data_config_generator.rst | 186 ++++++++++-------- .../approved/divingbell_ansible_framework.rst | 2 + .../drydock_support_bios_configuration.rst | 8 +- specs/approved/k8s_external_facing_api.rst | 2 +- ...{pegleg-secrets.rst => pegleg_secrets.rst} | 4 +- specs/approved/workflow_node-teardown.rst | 89 ++++----- specs/instructions.rst | 58 +++--- specs/template.rst | 8 +- 10 files changed, 189 insertions(+), 189 deletions(-) rename specs/approved/{multi-linux-distros.rst => airship_multi_linux_distros.rst} (96%) rename specs/approved/{pegleg-secrets.rst => pegleg_secrets.rst} (99%) diff --git a/.gitignore b/.gitignore index dbd8953..dd10b51 100644 --- a/.gitignore +++ b/.gitignore @@ -5,3 +5,4 @@ /AUTHORS /ChangeLog .tox +.vscode/ diff --git a/specs/approved/multi-linux-distros.rst b/specs/approved/airship_multi_linux_distros.rst similarity index 96% rename from specs/approved/multi-linux-distros.rst rename to specs/approved/airship_multi_linux_distros.rst index 520a1bc..3a6aae5 100644 --- a/specs/approved/multi-linux-distros.rst +++ b/specs/approved/airship_multi_linux_distros.rst @@ -5,18 +5,9 @@ http://creativecommons.org/licenses/by/3.0/legalcode .. index:: - single: template - single: creating specs - -.. note:: - - Blueprints are written using ReSTructured text. - -Add index directives to help others find your spec. E.g.:: - - .. index:: - single: template - single: creating specs + single: Airship + single: multi-linux-distros + single: containers =========================================== Airship Multiple Linux Distribution Support @@ -30,8 +21,9 @@ and other Linux distro's as new plugins. Links ===== -The work to author and implement this spec is tracked in Storyboard: -https://storyboard.openstack.org/#!/story/2003699 +The work to author and implement this spec is tracked in Storyboard +`2003699 `_ and uses Gerrit +topics ``airship_suse``, ``airship_rhel`` and similar. Problem description =================== diff --git a/specs/approved/data_config_generator.rst b/specs/approved/data_config_generator.rst index 9d98f6e..e88f8d6 100644 --- a/specs/approved/data_config_generator.rst +++ b/specs/approved/data_config_generator.rst @@ -150,34 +150,40 @@ Overall Architecture - Raw rack information from plugin: + :: + vlan_network_data: oam: subnet: 12.0.0.64/26 vlan: '1321' + - Rules to define gateway, ip ranges from subnet: + :: + rule_ip_alloc_offset: name: ip_alloc_offset ip_alloc_offset: default: 10 gateway: 1 - The above rule specify the ip offset to considered to define ip address for gateway, reserved - and static ip ranges from the subnet pool. - So ip range for 12.0.0.64/26 is : 12.0.0.65 ~ 12.0.0.126 - The rule "ip_alloc_offset" now helps to define additional information as follows: - - gateway: 12.0.0.65 (the first offset as defined by the field 'gateway') - - reserved ip ranges: 12.0.0.65 ~ 12.0.0.76 (the range is defined by adding - "default" to start ip range) - - static ip ranges: 12.0.0.77 ~ 12.0.0.126 (it follows the rule that we need - to skip first 10 ip addresses as defined by "default") + The above rule specify the ip offset to considered to define ip address for gateway, reserved + and static ip ranges from the subnet pool. + So ip range for 12.0.0.64/26 is : 12.0.0.65 ~ 12.0.0.126 + The rule "ip_alloc_offset" now helps to define additional information as follows: + + - gateway: 12.0.0.65 (the first offset as defined by the field 'gateway') + - reserved ip ranges: 12.0.0.65 ~ 12.0.0.76 (the range is defined by adding + "default" to start ip range) + - static ip ranges: 12.0.0.77 ~ 12.0.0.126 (it follows the rule that we need + to skip first 10 ip addresses as defined by "default") - Intermediary YAML file information generated after applying the above rules to the raw rack information: -:: + :: network: vlan_network_data: @@ -192,13 +198,13 @@ Overall Architecture static_end: 12.0.0.126 ----+ vlan: '1321' --- + -- - - J2 templates for specifying oam network data: It represents the format in - which the site manifests will be generated with values obtained from - Intermediary YAML + - J2 templates for specifying oam network data: It represents the format in + which the site manifests will be generated with values obtained from + Intermediary YAML -:: + :: --- schema: 'drydock/Network/v1' @@ -230,12 +236,12 @@ Overall Architecture end: {{ data['network']['vlan_network_data']['oam']['static_end'] }} ... --- + -- - - OAM Network information in site manifests after applying intermediary YAML to J2 - templates.: + - OAM Network information in site manifests after applying intermediary YAML to J2 + templates.: -:: + :: --- schema: 'drydock/Network/v1' @@ -267,7 +273,7 @@ Overall Architecture end: 12.0.0.126 ... --- + -- Security impact --------------- @@ -304,106 +310,114 @@ plugins. A. Excel Based Data Source. - - Gather the following input files: + - Gather the following input files: - 1) Excel based site Engineering package. This file contains detail specification - covering IPMI, Public IPs, Private IPs, VLAN, Site Details, etc. + 1) Excel based site Engineering package. This file contains detail specification + covering IPMI, Public IPs, Private IPs, VLAN, Site Details, etc. + 2) Excel Specification to aid parsing of the above Excel file. It contains + details about specific rows and columns in various sheet which contain the + necessary information to build site manifests. + 3) Site specific configuration file containing additional configuration like + proxy, bgp information, interface names, etc. + 4) Intermediary YAML file. In this cases Site Engineering Package and Excel + specification are not required. - 2) Excel Specification to aid parsing of the above Excel file. It contains - details about specific rows and columns in various sheet which contain the - necessary information to build site manifests. + B. Remote Data Source - 3) Site specific configuration file containing additional configuration like - proxy, bgp information, interface names, etc. + - Gather the following input information: - 4) Intermediary YAML file. In this cases Site Engineering Package and Excel - specification are not required. - - B. Remote Data Source - - - Gather the following input information: - - 1) End point configuration file containing credentials to enable its access. - Each end-point type shall have their access governed by their respective plugins - and associated configuration file. - - 2) Site specific configuration file containing additional configuration like - proxy, bgp information, interface names, etc. These will be used if information - extracted from remote site is insufficient. + 1) End point configuration file containing credentials to enable its access. + Each end-point type shall have their access governed by their respective plugins + and associated configuration file. + 2) Site specific configuration file containing additional configuration like + proxy, bgp information, interface names, etc. These will be used if information + extracted from remote site is insufficient. * Program execution - 1) CLI Options: - -g, --generate_intermediary Dump intermediary file from passed Excel and - Excel spec. - -m, --generate_manifests Generate manifests from the generated - intermediary file. - -x, --excel PATH Path to engineering Excel file, to be passed - with generate_intermediary. The -s option is - mandatory with this option. Multiple engineering - files can be used. For example: -x file1.xls -x file2.xls - -s, --exel_spec PATH Path to Excel spec, to be passed with - generate_intermediary. The -x option is - mandatory along with this option. - -i, --intermediary PATH Path to intermediary file,to be passed - with generate_manifests. The -g and -x options - are not required with this option. - -d, --site_config PATH Path to the site specific YAML file [required] - -l, --loglevel INTEGER Loglevel NOTSET:0 ,DEBUG:10, INFO:20, - WARNING:30, ERROR:40, CRITICAL:50 [default:20] - -e, --end_point_config File containing end-point configurations like user-name - password, certificates, URL, etc. - --help Show this message and exit. + 1. CLI Options: - 2) Example: + +-----------------------------+-----------------------------------------------------------+ + | -g, --generate_intermediary | Dump intermediary file from passed Excel and | + | | Excel spec. | + +-----------------------------+-----------------------------------------------------------+ + | -m, --generate_manifests | Generate manifests from the generated | + | | intermediary file. | + +-----------------------------+-----------------------------------------------------------+ + | -x, --excel PATH | Path to engineering Excel file, to be passed | + | | with generate_intermediary. The -s option is | + | | mandatory with this option. Multiple engineering | + | | files can be used. For example: -x file1.xls -x file2.xls | + +-----------------------------+-----------------------------------------------------------+ + | -s, --exel_spec PATH | Path to Excel spec, to be passed with | + | | generate_intermediary. The -x option is | + | | mandatory along with this option. | + +-----------------------------+-----------------------------------------------------------+ + | -i, --intermediary PATH | Path to intermediary file,to be passed | + | | with generate_manifests. The -g and -x options | + | | are not required with this option. | + +-----------------------------+-----------------------------------------------------------+ + | -d, --site_config PATH | Path to the site specific YAML file [required] | + +-----------------------------+-----------------------------------------------------------+ + | -l, --loglevel INTEGER | Loglevel NOTSET:0 ,DEBUG:10, INFO:20, | + | | WARNING:30, ERROR:40, CRITICAL:50 [default:20] | + +-----------------------------+-----------------------------------------------------------+ + | -e, --end_point_config | File containing end-point configurations like user-name | + | | password, certificates, URL, etc. | + +-----------------------------+-----------------------------------------------------------+ + | --help | Show this message and exit. | + +-----------------------------+-----------------------------------------------------------+ - 2-1) Using Excel spec as input data source: + 2. Example: - Generate Intermediary: spyglass -g -x -s -d + 1) Using Excel spec as input data source: - Generate Manifest & Intermediary: spyglass -mg -x -s -d + Generate Intermediary: ``spyglass -g -x -s -d `` - Generate Manifest with Intermediary: spyglass -m -i + Generate Manifest & Intermediary: ``spyglass -mg -x -s -d `` + Generate Manifest with Intermediary: ``spyglass -m -i `` - 2-1) Using external data source as input: + 2) Using external data source as input: - Generate Manifest and Intermediary : spyglass -m -g -e -d - Generate Manifest : spyglass -m -e -d + Generate Manifest and Intermediary: ``spyglass -m -g -e -d `` - Note: The end_point_config shall include attributes of the external data source that are - necessary for its access. Each external data source type shall have its own plugin to configure - its corresponding credentials. + Generate Manifest: ``spyglass -m -e -d `` + + .. note:: + + The end_point_config shall include attributes of the external data source that are + necessary for its access. Each external data source type shall have its own plugin to configure + its corresponding credentials. * Program output: + a) Site Manifests: As an initial release, the program shall output manifest files for "airship-seaworthy" site. For example: baremetal, deployment, networks, pki, etc. - Reference:https://github.com/openstack/airship-treasuremap/tree/master/site/airship-seaworthy + Reference: https://github.com/openstack/airship-treasuremap/tree/master/site/airship-seaworthy b) Intermediary YAML: Containing aggregated site information generated from data sources that is used to generate the above site manifests. Future Work ============ -1) Schema based manifest generation instead of Jinja2 templates. It shall -be possible to cleanly transition to this schema based generation keeping a unique -mapping between schema and generated manifests. Currently this is managed by -considering a mapping of j2 templates with schemas and site type. - -2) UI editor for intermediary YAML +1. Schema based manifest generation instead of Jinja2 templates. It shall + be possible to cleanly transition to this schema based generation keeping a unique + mapping between schema and generated manifests. Currently this is managed by + considering a mapping of j2 templates with schemas and site type. +2. UI editor for intermediary YAML Alternatives ============ -1) Schema based manifest generation instead of Jinja2 templates. -2) Develop the data source plugins as an extension to Pegleg. +1. Schema based manifest generation instead of Jinja2 templates. +2. Develop the data source plugins as an extension to Pegleg. Dependencies ============ -1) Availability of a repository to store Jinja2 templates. -2) Availability of a repository to store generated manifests. +1. Availability of a repository to store Jinja2 templates. +2. Availability of a repository to store generated manifests. References ========== None - diff --git a/specs/approved/divingbell_ansible_framework.rst b/specs/approved/divingbell_ansible_framework.rst index 6658382..6acc3c8 100644 --- a/specs/approved/divingbell_ansible_framework.rst +++ b/specs/approved/divingbell_ansible_framework.rst @@ -60,6 +60,7 @@ A separate directory structure needs to be created for adding the playbooks. Each Divingbell config can be a separate role within the playbook structure. :: + - playbooks/ - roles/ - systcl/ @@ -83,6 +84,7 @@ With Divingbell DaemonSet running on each host mounted at ``hostPath``, ``hosts`` should be defined as given below within the ``master.yml``. :: + hosts: all connection: chroot diff --git a/specs/approved/drydock_support_bios_configuration.rst b/specs/approved/drydock_support_bios_configuration.rst index 25fbf67..81c1f96 100644 --- a/specs/approved/drydock_support_bios_configuration.rst +++ b/specs/approved/drydock_support_bios_configuration.rst @@ -193,14 +193,10 @@ Work Items ---------- - Update Hardware profile schema to support new attribute bios_setting - - Update Hardware profile objects - - Update Orchestrator action PrepareNodes to call OOB driver for BIOS configuration - - Update Redfish OOB driver to support new action ConfigBIOS - - Add unit test cases Assignee(s): @@ -215,8 +211,8 @@ Other contributors: Dependencies ============ -This spec depends on ``Introduce Redfish based OOB Driver for Drydock`` -https://storyboard.openstack.org/#!/story/2003007 +This spec depends on `Introduce Redfish based OOB Driver for Drydock `_ +story. References ========== diff --git a/specs/approved/k8s_external_facing_api.rst b/specs/approved/k8s_external_facing_api.rst index f41ba9c..64d8c1c 100644 --- a/specs/approved/k8s_external_facing_api.rst +++ b/specs/approved/k8s_external_facing_api.rst @@ -45,7 +45,7 @@ Impacted components The following Airship components would be impacted by this solution: #. Promenade - Maintenance of the chart for external facing Kubernetes API -servers + servers Proposed change =============== diff --git a/specs/approved/pegleg-secrets.rst b/specs/approved/pegleg_secrets.rst similarity index 99% rename from specs/approved/pegleg-secrets.rst rename to specs/approved/pegleg_secrets.rst index 9fbb3cc..6277461 100644 --- a/specs/approved/pegleg-secrets.rst +++ b/specs/approved/pegleg_secrets.rst @@ -5,8 +5,8 @@ http://creativecommons.org/licenses/by/3.0/legalcode .. index:: - single: template - single: creating specs + single: Pegleg + single: Security ======================================= Pegleg Secret Generation and Encryption diff --git a/specs/approved/workflow_node-teardown.rst b/specs/approved/workflow_node-teardown.rst index db9d587..95ffbff 100644 --- a/specs/approved/workflow_node-teardown.rst +++ b/specs/approved/workflow_node-teardown.rst @@ -150,21 +150,26 @@ details: #. Drain the Kubernetes node. #. Clear the Kubernetes labels on the node. #. Remove etcd nodes from their clusters (if impacted). + - if the node being decommissioned contains etcd nodes, Promenade will - attempt to gracefully have those nodes leave the etcd cluster. + attempt to gracefully have those nodes leave the etcd cluster. + #. Ensure that etcd cluster(s) are in a stable state. + - Polls for status every 30 seconds up to the etcd-ready-timeout, or the - cluster meets the defined minimum functionality for the site. + cluster meets the defined minimum functionality for the site. - A new document: promenade/EtcdClusters/v1 that will specify details about - the etcd clusters deployed in the site, including: identifiers, - credentials, and thresholds for minimum functionality. + the etcd clusters deployed in the site, including: identifiers, + credentials, and thresholds for minimum functionality. - This process should ignore the node being torn down from any calculation - of health + of health + #. Shutdown the kubelet. + - If this is not possible because the node is in a state of disarray such - that it cannot schedule the daemonset to run, this step may fail, but - should not hold up the process, as the Drydock dismantling of the node - will shut the kubelet down. + that it cannot schedule the daemonset to run, this step may fail, but + should not hold up the process, as the Drydock dismantling of the node + will shut the kubelet down. Responses ~~~~~~~~~ @@ -173,11 +178,9 @@ All responses will be form of the Airship Status response. - Success: Code: 200, reason: Success Indicates that all steps are successful. - - Failure: Code: 404, reason: NotFound Indicates that the target node is not discoverable by Promenade. - - Failure: Code: 500, reason: DisassociateStepFailure The details section should detail the successes and failures further. Any @@ -223,16 +226,13 @@ All responses will be form of the Airship Status response. Indicates that the drain node has successfully concluded, and that no pods are currently running - - Failure: Status response, code: 400, reason: BadRequest A request was made with parameters that cannot work - e.g. grace-period is set to a value larger than the timeout value. - - Failure: Status response, code: 404, reason: NotFound The specified node is not discoverable by Promenade - - Failure: Status response, code: 500, reason: DrainNodeError There was a processing exception raised while trying to drain a node. The @@ -263,11 +263,9 @@ All responses will be form of the Airship Status response. - Success: Code: 200, reason: Success All labels have been removed from the specified Kubernetes node. - - Failure: Code: 404, reason: NotFound The specified node is not discoverable by Promenade - - Failure: Code: 500, reason: ClearLabelsError There was a failure to clear labels that prevented completion. The details @@ -298,11 +296,9 @@ All responses will be form of the Airship Status response. - Success: Code: 200, reason: Success All etcd nodes have been removed from the specified node. - - Failure: Code: 404, reason: NotFound The specified node is not discoverable by Promenade - - Failure: Code: 500, reason: RemoveEtcdError There was a failure to remove etcd from the target node that prevented @@ -315,7 +311,7 @@ Promenade Check etcd ~~~~~~~~~~~~~~~~~~~~ Retrieves the current interpreted state of etcd. -GET /etcd-cluster-health-statuses?design_ref={the design ref} + GET /etcd-cluster-health-statuses?design_ref={the design ref} Where the design_ref parameter is required for appropriate operation, and is in the same format as used for the join-scripts API. @@ -334,42 +330,40 @@ All responses will be form of the Airship Status response. The status of each etcd in the site will be returned in the details section. Valid values for status are: Healthy, Unhealthy -https://github.com/openstack/airship-in-a-bottle/blob/master/doc/source/api-conventions.rst#status-responses + https://github.com/openstack/airship-in-a-bottle/blob/master/doc/source/api-conventions.rst#status-responses -.. code:: json + .. code:: json - { "...": "... standard status response ...", - "details": { - "errorCount": {{n}}, - "messageList": [ - { "message": "Healthy", - "error": false, - "kind": "HealthMessage", - "name": "{{the name of the etcd service}}" - }, - { "message": "Unhealthy" - "error": false, - "kind": "HealthMessage", - "name": "{{the name of the etcd service}}" - }, - { "message": "Unable to access Etcd" - "error": true, - "kind": "HealthMessage", - "name": "{{the name of the etcd service}}" - } - ] - } - ... - } + { "...": "... standard status response ...", + "details": { + "errorCount": {{n}}, + "messageList": [ + { "message": "Healthy", + "error": false, + "kind": "HealthMessage", + "name": "{{the name of the etcd service}}" + }, + { "message": "Unhealthy" + "error": false, + "kind": "HealthMessage", + "name": "{{the name of the etcd service}}" + }, + { "message": "Unable to access Etcd" + "error": true, + "kind": "HealthMessage", + "name": "{{the name of the etcd service}}" + } + ] + } + ... + } - Failure: Code: 400, reason: MissingDesignRef Returned if the design_ref parameter is not specified - - Failure: Code: 404, reason: NotFound Returned if the specified etcd could not be located - - Failure: Code: 500, reason: EtcdNotAccessible Returned if the specified etcd responded with an invalid health response @@ -400,11 +394,9 @@ All responses will be form of the Airship Status response. - Success: Code: 200, reason: Success The kubelet has been successfully shutdown - - Failure: Code: 404, reason: NotFound The specified node is not discoverable by Promenade - - Failure: Code: 500, reason: ShutdownKubeletError The specified node's kubelet fails to shutdown. The details section of the @@ -433,17 +425,14 @@ All responses will be form of the Airship Status response. - Success: Code: 200, reason: Success The specified node has been removed from the Kubernetes cluster. - - Failure: Code: 404, reason: NotFound The specified node is not discoverable by Promenade - - Failure: Code: 409, reason: Conflict The specified node cannot be deleted due to checks that the node is drained/cordoned and has no labels (other than possibly `promenade-decomission: enabled`). - - Failure: Code: 500, reason: DeleteNodeError The specified node cannot be removed from the cluster due to an error from diff --git a/specs/instructions.rst b/specs/instructions.rst index 5fa31a8..a76051b 100644 --- a/specs/instructions.rst +++ b/specs/instructions.rst @@ -20,6 +20,12 @@ Instructions a short explanation. - New specs for review should be placed in the ``approved`` subfolder, where they will undergo review and approval in Gerrit_. +- Test if the spec file renders correctly in a web-browser by running + ``make docs`` command and opening ``doc/build/html/index.html`` in a + web-browser. Ubuntu needs the following packages to be installed:: + + apt-get install -y make tox gcc python3-dev + - Specs that have finished implementation should be moved to the ``implemented`` subfolder. @@ -50,38 +56,38 @@ Use the following guidelines to determine the category to use for a document: 1) For new functionality and features, the best choice for a category is to match a functional duty of Airship. -site-definition - Parts of the platform that support the definition of a site, including - management of the yaml definitions, document authoring and translation, and - the collation of source documents. + site-definition + Parts of the platform that support the definition of a site, including + management of the yaml definitions, document authoring and translation, and + the collation of source documents. -genesis - Used for the steps related to preparation and deployment of the genesis node - of an Airship deployment. + genesis + Used for the steps related to preparation and deployment of the genesis node + of an Airship deployment. -baremetal - Those changes to Airflow that provide for the lifecycle of bare metal - components of the system - provisioning, maintenance, and teardown. This - includes booting, hardware and network configuration, operating system, and - other host-level management + baremetal + Those changes to Airflow that provide for the lifecycle of bare metal + components of the system - provisioning, maintenance, and teardown. This + includes booting, hardware and network configuration, operating system, and + other host-level management -k8s - For functionality that is about interfacing with Kubernetes directly, other - than the initial setup that is done during genesis. + k8s + For functionality that is about interfacing with Kubernetes directly, other + than the initial setup that is done during genesis. -software - Functionality that is related to the deployment or redeployment of workload - onto the Kubernetes cluster. + software + Functionality that is related to the deployment or redeployment of workload + onto the Kubernetes cluster. -workflow - Changes to existing workflows to provide new functionality and creation of - new workflows that span multiple other areas (e.g. baremetal, k8s, software), - or those changes that are new arrangements of existing functionality in one - or more of those other areas. + workflow + Changes to existing workflows to provide new functionality and creation of + new workflows that span multiple other areas (e.g. baremetal, k8s, software), + or those changes that are new arrangements of existing functionality in one + or more of those other areas. -administration - Security, logging, auditing, monitoring, and those things related to site - administrative functions of the Airship platform. + administration + Security, logging, auditing, monitoring, and those things related to site + administrative functions of the Airship platform. 2) For specs that are not feature focused, the component of the system may be the best choice for a category, e.g. ``shipyard``, ``armada`` etc... diff --git a/specs/template.rst b/specs/template.rst index 01577ee..6309548 100644 --- a/specs/template.rst +++ b/specs/template.rst @@ -12,7 +12,7 @@ Blueprints are written using ReSTructured text. -Add index directives to help others find your spec. E.g.:: +Add *index* directives to help others find your spec by keywords. E.g.:: .. index:: single: template @@ -27,9 +27,9 @@ Introduction paragraph -- What is this blueprint about? Links ===== -Include pertinent links to where the work is being tracked (e.g. Storyboard), -as well as any other foundational information that may lend clarity to this -blueprint +Include pertinent links to where the work is being tracked (e.g. Storyboard ID +and Gerrit topics), as well as any other foundational information that may lend +clarity to this blueprint Problem description ===================