Updates the rest api plugin used from [0] to remove background
processing logic that triggers an observed security vulnerability due to
use of os.system(). The background process support was used only for log
retrieval, which Shipyard does in a different way (not this plugin).
[0] https://github.com/teamclairvoyant/airflow-rest-api-plugin
Change-Id: I6967938c1f29678137ea27d01b4a639bc3acc6d5
During the publish of an image post-merge, an error was being rasied due
to a conflict between the ansible-supplied docker-py and the pip
installed docker. This removes the pip installed docker to allow
publishing to proceed.
Change-Id: I6a1ff54ed2d2af85cbe44ed50b1c637dee9adf54
A new Shipyard site statuses API and CLI supporting nodes
provisoning status and node power state. This API
can be further developed to support new status
requirements by expanding the filters option.
Change-Id: I620aefd82d4a17b616f3f253265605e519506257
Use of memcached, which is enabled by default in the Shipyard Helm
chart, requires the cryptography package. Word is that cryptography
had a compatibility issue with python3 in the past, but v2.3 seems
to be working fine and getting memcached integration to work.
Change-Id: I6b5c9983f41aec874e45fd81bd8a30faa88fc316
We want to default to running all tox environments under python 3, so
set the basepython value in each environment.
We do not want to specify a minor version number, because we do not
want to have to update the file every time we upgrade python.
We do not want to set the override once in testenv, because that
breaks the more specific versions used in default environments like
py35 and py36.
Change-Id: If0f86ecb4c18216daf158101dbd3663bda06956f
Updates the imports for the dags and operators to support both "as
deployed" and "as tested" package configurations. This allows for a
simple test to be added that at least imorts and checks the dags to
ensure they contain steps.
A future refactor may eliminate the need for some/much of this by moving the
operators away from the plugin appraoch such that they can be statically
built into the airflow pod and used like a third party library instead
of being appended to the airflow plugins. For now though, this maintains
the status quo for the way these are used in a deployed way.
Change-Id: I437ff9c583358188e27de0e2f6987c38ca85ab2f
l is to let user customize the base image of the component
by passing FROM=myimage during the build process. This would let any
project leveraging Airship ensure that the base image is matching the
security requirements for that project and still use the same Dockerfile.
This will also ease the control of the /etc/apt/source.list
and thereby the result of apt-get update/upgrade procedure.
2. The above goal is achievable by using docker-ce feature such as:
ARG FROM="defaultbaseimage:xx"
FROM ${FROM}
For this reason, the installation of docker.io in the Zuul gating is beeing
replaced by docker-ce.
3. Third Goal is to bring consistency with the other compoenents leveraging
Helm such as the openstack-helm and potentially use bindep the same way
the LOCI images are to ensure
4. The new syntax in the Dockerfile is still commented out until the associated
image builder have been updated to use docker-ce as they have been for the LOCI
images.
Change-Id: Ife7e1be53c7c139bdc42dee42f0798e83f4fd271
Adds better info to the error returne when retreiving the validations
from another Airship component. Adds tests to cover the success and
failure flows of this same logic
Change-Id: Id7fb389a3905f3e0659d4a7eec0e0658e00f3f28
Provides different fields in the values.yaml to use for the node
selector keys/values used in deployments vs jobs.
Change-Id: I12d7c6257aea0ac00cd77cd3f6331a2b7380b589
Updates the workflows to have considerably more parallelization of the
early steps, and unwraps some of the operators from their sub-dags, in
an attempt to simplify the representation of the workflow and
potentially increase performance.
Change-Id: I6ce987b32399e261a2383233bd192b0e49514791
This patchset updates osh-infra-deploy-docker.yaml playbook
to align with role rename change in osh-infra causing pre-run
to fail: https://review.openstack.org/#/c/578703/6
Change-Id: Ie119f953e77fc748cc784b8cd9f509892e55927d
When checking for deployed nodes, the kubernetes join check was
only performing a negative check - and would wait for up to the timeout
even in the event that nodes that were not part of the current
processing before proceeding. This had the drawback of being overall
likely to add wait time in any complex deployment scenario, as well as
(and more importantly) miss the case where a node never started to try
to join, and assume that was a success.
This patchset flips the logic to positively look for an expected set of
nodes instead, and will not wait upon nodes that are not currently being
checked. The end result should remedy both of the drawbacks listed
above.
Change-Id: Ib07e4e2677ec4f773d695d57893fdfa5e4b7ff76
A KeyError was not being caught by the ValueError exception handler when
removing a node from a list of successes. This provides a safer
exception handler.
Change-Id: I3f7b5146009f4f05ee893919a73e41e182dea9f9
Adds the secret to support TLS for the Shipyard API
Change-Id: I34d753bc0c65b00df54aeb32ff66eef5bf2c4c6e
Co-Authored-By: Pete Birley <pete@port.direct>
Signed-off-by: Pete Birley <pete@port.direct>
In the case of being able to gather informational task info, if there is
any exception, report the exception and move on, rather than failing
hard.
Change-Id: I677f9c375549f0ff421aa322c561a8bc7cb848d0
Updated configurations to point to openstack-helm-infra
for reference to helm-toolkit as helm-toolkit has been
removed from the openstack-helm repo [0]
Also aligned with changes to the keystone user set up in
OSH using Helm ToolKit so as to get pass Helm Lint.
Updated Makefile targets to install helm dynamically
[0] https://review.openstack.org/#/c/558065/
Change-Id: I0a0813516f9ad176ff005b4693e6b933013a99fd
Updates the Shipyard/Airflow workflow for deploy_site and
update_site to use the deployment group/deployment strategy
information from the design.
This allows for baremetal nodes to be deployed in a design-
specified order, with criticality and success criteria driving
the success and failure of deployment.
Includes refactoring of service endpoints to reduce the need
for so much data passing.
Change-Id: Ib5e9fca535ca74d1819fe46959695acfed5b65c2
- Don't change directory so that the volume mounted in the
container at /home/shipayard/host is predictable
- Update image default to point at airshipit repo
Change-Id: If8b0988925921d7b54ec74ac73e12289d89d392e
