Added the feature in airflow to verify that MaaS list
of BM hosts is not empty for shipyard update_site action.
If the MaaS Machine list is empty, and the
continue-on-fail parameter is not set to true (the default
value is false), it will fail the shipyard steps to
parepare and re-deploy the missing nodes in MaaS through
DD.
Caveat: this US did not have the requirement to compare
the list of nodes in MaaS with the expected site Design.
It simply checks for empty node list, and decide based
on that.
Change-Id: I5ba4a107fe2ae43728e5941570b6c88a436d7b12
Converts the input to the method that deploys nodes to a list, to
protect against a set being used as input, which leads to a problem when
serializing the data in node filter passed to the Drydock client.
Change-Id: I6a5a0ec0ea9ed09b1911c462fec9dc3793dd1c26
This PS updates the version of helm used to 2.10, the current version
targeted by armada and OSH:
* https://review.openstack.org/#/c/597296
Change-Id: Ib40c02f5e2c05cd55af5d702e732c7937b2ee922
Signed-off-by: Pete Birley <pete@port.direct>
Retries of a step may execute on different workers.
Each worker only exposes the logs that were generated by that worker.
When assembling the URL for the step, we need to ensure
that we include the retry number as part of the key lookup for a step,
so that it can use the correct worker reference.
Updated ActionsHelper.get_step to use try_number.
The test is included.
Change-Id: I2f8cf9ed70ce344f53ecdcc4edbec924ba6a00c9
Includes guiding information to direct users of redeploy_server
through preparatory steps to minimize undesired impacts.
Change-Id: Idae8b0beab667af05b34acb2bf1be590627d911a
Before this change check_intermidiate_commit return False if revision
has site-action-failure tag only.
Co-Authored-by: Bryan Strassner <bryan.strassner@gmail.com>
Co-Authored-by: Serge Kovaleff <sk607s@att.com>
Change-Id: I8524f599741dab743df9e1a2638b25e04c87da7c
When processing a deployment group, the the deployment of nodes was
using the same input and a success against the success_criteria
evaluated after preparing nodes. This lead to situations where nodes
failed to prepare, but were assumed (and thusly failed) for deployment.
This was especially problematic when a timeout was triggered by Shipyard
before Drydock had finished preparing.
This change will only attempt to deploy nodes that were positively
identified as prepared by Drydock. When the timeout scenario is reached,
since there will have been no positive confirmation of successful nodes,
the deployment of nodes will not be attempted. This will also prevent
attempting to deploy nodes that have expicitly failed to prepare.
Additionally, added some TODOs around the concept of cancelling tasks in
Drydock when Shipyard stops due to a timeout, however, this kind of
functionality does not yet exist, so the TODOs serve as a placeholder.
Change-Id: I582abcec62407dc2903d8a4477ea891a9397f1fb
Invoking make charts initializes the clean recipe, resulting in the
deletion of the temp directory, $(BUILD_DIR), which is created when
make is invoked. Since this is the location the helm client is
installed by tools/helm_install.sh, the script fails. This commit
removes the statement that deletes the temp directory created when make
is invoked.
Change-Id: I87d41df979a5a4daeb539bcc1330f126a5a407c7
This commit updates Shipyard's default RBAC policy to include two
additional roles:
- admin_ucp
- admin_ucp_viewer
The default policy is implemented with this in mind:
- The 'admin' and 'admin_ucp' roles have access to all of Shipyard's
APIs.
- The 'admin_ucp_viewer' role only has access to Shipyard's GET,
LIST, and AUDIT APIs
Automated Shipyard RBAC tests are found here [0].
[0] https://github.com/att-comdev/airship-tempest-plugin/tree/master/airship_tempest_plugin/tests/api/shipyard/rbac
Change-Id: I5cf8910441c7a80829dd00320d817416ca22ff98
In the corner case that the tmp directory would not be created, the old
form could attempt to delete much more than intended. This changes to
more simply fail in the case of a missing tmp directory
Change-Id: I3a4cd800e329cd477904d00f6dcb77bbbc2ff90b
Currently, the shipyard.sh script does not set the no_proxy
environment variable for the Shipyard container. Invoking the CLI
to perform actions results in a 404 error unless the variable is set.
This change adds localhost and .svc.cluster.local to the no_proxy
environment variable.
Change-Id: Ie0ae52d4ecb51eafe983a7a452b61a01c2c9426e
In the case of a temporary failure to lookup nodes, this introduces a
nominal number of retries to possibly bridge the failure. By default 2
reties, spaced 30 seconds apart will be attempted. Some exceptions may
not benefit from retries; this change doesn't account for the myriad
situations thay may benefit from short circuiting.
Change-Id: Icbedf5c1d8ca485c36ef6ba31e8c0201f0f28b6d
Sets a default of 10M for body size so that ingress will allow more than
1M before issuing a 413 response.
Change-Id: I832c41685135b556e9f6c81d6be04f7497328b3b
This change modifies the internal Keystone API port in the Shipyard
chart from 80 to 5000 and removes the default admin port to match
the Keystone chart provided by OpenStack-Helm.
Change-Id: I8e3b4846ddee9995182eade73146418f0cb70895
The Shipyard API docs and shipyard_api_client incorrectly reference
the wrong API endpoints for some APIs. This commit updates the API
docs to reference the correct API endpoints for:
- getting action validation details
- triggering a control action against an activity
The following is updated in the shipyard_api_client.py:
- API endpoint for getting action validation details
Change-Id: I04b770acfd64f331efce6f83f51cb41d0818a6b2
This PS updates the make scripts for charts to use the current version
of helm targetted by ariship.
Change-Id: Iaf49b28516e2b5b5d1fe063b54171a321bd1d64f
Signed-off-by: Pete Birley <pete@port.direct>
Shipyard was passing a complete design reference dictionary to the
Drydock client, as it would for calling the Drydock API directly. This
causes an error to be reported (and subsequently ignored) during
validation of the DeploymentStrategy document. This change corrects the
behavior by passing only the href to the Drydock Client from the
deployment strategy validator
Change-Id: Idca0a69ec4ea11bf2bc4b520eb1512c0bdcd481b
This change adds the global zuul pep8 tox job, which runs both
bandit and pep8 using tox. This also removes the two other airship
specific lint-pep8 and bandit zuul jobs since they are both covered
by the default openstack global one.
Change-Id: Iebea5b872f78762d6f401b574d53965b2e1c090b
Adds the functionality to redeploy a server in an unguarded fashion,
meaning that the server will not be pre-validated to be in a state that
workloads have been removed.
This is the first targeted action for Shipyard, so a refactoring of the
validators to support more flexibility has been done.
Also adds RBAC controls for specific actions being created, rather than
a binary create action privilege.
Change-Id: I39e3dab6595c5187affb9f2b6edadd0f5f925b0c
Adds a #nosec exclusion to a known Bandit error that has been evaluated
as low severity, and included comments about how the severity is further
reduced.
Adds a target Python version for Bandit job so that it does not fail to
scan several more files that are not working with a Python 2 run of
Bandit.
Change-Id: I251abd092b3049a663b8758bbec0926f4b4836f7
Sets the run_id for a DAG invoked in Airflow to the same ULID assigned
to it in Shipyard. While this was already happening as a parameter to
the DAG being invoked, by making it the run_id, further correlation is
possible, at a level that both Shipyard and the Airflow framework are
aware.
As part of making this change, fragility was uncovered in the
rest_api_plugin that expedited the need to switch to the built-in, but
experimental airflow API to trigger a dag (one of two API endpoints
provided - this is important later in this story). In any case, the 3rd
party rest_api_plugin was removed.
As a result of the rest_api_plugin being removed:
1) the simpleton helm test to check the api of airflow was also removed
(it used the version endpoint of this plugin). As the built-in api
provides no version endpoint or similarly accessible-without-being-stateful
endpoint, the helm test had no new place to look for something to call.
2) Some clean up of exclusions and documentation was possible - test
coverage, security exclusions, left over documentation remnants
Change-Id: I0b68496a8500408b776b4acc12888aa017c4c7d2
Regenerates the policy file to update it to include the new sample
policies that can be manipulated
Change-Id: I37f3319d2323d6bf24183b41f54f2e471d07cecd
Adds options to the configuration of Shipyard to direct oslo_policy to
the location of the /etc/shipyard/policy.yaml file (default location)
allowing for override of default policies via chart or chart override.
Change-Id: I5cf68994c40aa835a631f5b6f67363a2b8a8af0a
Changes repeated use of strings to a list of constant values for the
policies used to validate access to the APIs of Shipyard.
Change-Id: Ie1cac7b0587ddcf907e81ffee14fa43042b812b5
Because the type of the query was generated by sqlalchemy.sql.text(),
the TextClause object did not have a split() method, and crashed out
trying to reformat into one line. Using the str() wrapper provides a
string that can be properly split and joined.
Change-Id: I1ed9e39d7ebf3904d3d233330ee57082ad02c5f3
Cleans up Shipyard logs a bit, where queries were nice and easy to read,
but multi-line and needlessly on a separate line from the logging
headers.
Change-Id: I2fdff634dad097ef30207edae5205cb6c7226602
Updates the rest api plugin used from [0] to remove background
processing logic that triggers an observed security vulnerability due to
use of os.system(). The background process support was used only for log
retrieval, which Shipyard does in a different way (not this plugin).
[0] https://github.com/teamclairvoyant/airflow-rest-api-plugin
Change-Id: I6967938c1f29678137ea27d01b4a639bc3acc6d5
During the publish of an image post-merge, an error was being rasied due
to a conflict between the ansible-supplied docker-py and the pip
installed docker. This removes the pip installed docker to allow
publishing to proceed.
Change-Id: I6a1ff54ed2d2af85cbe44ed50b1c637dee9adf54
A new Shipyard site statuses API and CLI supporting nodes
provisoning status and node power state. This API
can be further developed to support new status
requirements by expanding the filters option.
Change-Id: I620aefd82d4a17b616f3f253265605e519506257
