Adding DISTRO parameter for makefile to invoke distribution specific
Dockerfile and build image accordingly.
Also qualifying image job name to distinguish ubuntu vs opensuse
check jobs and passing distro specific variable.
Similar logic for pushing distro specific images.
Added new document section for multiple distro support.
Change-Id: I215f8a107cab8770181c0472e0cc29053b2b978e
A recent Shipyard change [0] introduced user-context tracing support in
Shipyard, which requires the passing of an additional parameter to the
Deckhand and Drydock API clients. The Deckhand [1] and Drydock [2]
clients were updated to handle the additional parameter, but the
versions of those components used by Shipyard was not updated. This
change uplifts the versions of Deckhand and Drydock in requirements.txt
for the Airflow component of Shipyard to a version that supports the
aforementioned, additional, user-context parameter.
[0] https://review.openstack.org/633873
[1] https://review.openstack.org/634068
[2] https://review.openstack.org/638115
Change-Id: I096191f436c5123b6c263fe92c0d69b9e4085c66
The current script is fragile to the introduction of a DEBUG logging
level. This change addresses the specifics of logging records being
returned when invoking the airflow command enough to make the dag_status
result end up being the last result as the script desires, so the script
is returned to working order, even if the logging level is set to DEBUG.
This change goes a little further to the point of removing all records
that indicate a logging level of DEBUG|INFO|WARN|ERROR.
Change-Id: Ic72358097e3a476e20fa3713008d0d0bcd35463e
The airflow image has a requirements file that is separate from the one
located in src/bin/shipyard_airflow, which was missed in a previous
uplift patch [0]. This change updates all versions in the image
requirements file to match the source code requirements.
[0] https://review.openstack.org/652736
Change-Id: I238bba4038e5cd8364711c2c5d1f78f51fc76948
Updates the dependencies on Armada, Drydock, and Deckhand to current
values. The primary reason for this is to take advantage of a new set of
Armada requirements that allows for a reduced image size (no longer
includes grpc tools)
Change-Id: I8f208ab91664479a7af96c543a4e72d8f6d154ef
Change to use ubuntu base image instead of python
Refactor Shipyard Dockerfile to reduce image size significantly
BREAKING CHANGE: The `make images` PYTHON_BASE_IMAGE arg is now renamed to BASE_IMAGE.
Change-Id: I3338dfbbb91b5514fa4fd205bdfc4136d0abc2e5
Changes to make the docker image build to include the workflows from
Shipyard, rather than adding them to the container during Helm install
of Shipyard. This also removes the "prod" switch, as it is now always
built the same way, with the workflows in place.
Change-Id: I4acd6195cbec32193e15621e75ccaeb9879455f5
Updates to Airflow 1.10.1; See (1), (2) for some notes
Related, and additionally: configures Airflow to restore logging of
workflow steps to a console/sdtout logger, supporting the desired
ability to attach logging and monitoring to standard container
mechanisms. This does not change the behavior of also logging to the
airflow-arranged log files for steps and DAG runs.
A side effect of updating to 1.10.1 includes a major decrease in
resource usage by the Airflow scheudler process (reducing from ~ 1 core
fully consumed to less than 5% of a core consumed YMMV, but significant)
Additional adjustment downward of resources allocated, threads produced,
and frequency of polling leads to an overall significant reduction in
resource usage.
Airship note:
Because Airflow 1.10.0 and 1.10.1 use compatible versions of celery and
dag_run information, updating from 1.10.0 - 1.10.1 in place is possible
if airflow-worker pods are allowed to continue to run.
(1) https://github.com/apache/incubator-airflow/blob/master/UPDATING.md
(2) https://github.com/apache/incubator-airflow/releases/tag/1.10.1
Change-Id: I9b024e3996c528c7b74e2888191d48c7a45a1f04
This PS enables shipyard to use PBR as rest
of the other OpenStack project.
Currently if shipyard is used as a requirement in another
project, the dependencies for shipyard doesn't get installed.
This PS fixes this bug.
Change-Id: I34d452649a886af9d865d4ad51eeab70399fe395
Now that Deckhand supports rolling back to revision 0 with
https://review.openstack.org/#/c/592725/ it is preferrable to
exercise this functionality instead of deleting all revisions
in Deckhand's database, so that previous revision history
isn't lost.
Deckhand is fast-forwarded to
0b5aa2e98a1ab5ab8a58c9dec3c1f88ef00d17a9 which contains
the commit required for SY to be able to make use of this
new change in Deckhand's client, as well as to align with
Promenade's Deckhand fast-forward of Deckhand in
https://review.openstack.org/#/c/604459/
Depends-On: https://review.openstack.org/#/c/592725/
Change-Id: Iebbfeb5e6c74b3e2f09b2137ec14482cc8fa4f44
Updates the image building to Airflow 1.10, including necessary
configuration changes and a general update of dependencies.
Airflow 1.10 includes many enhancements and bugfixes since 1.9 [0]
This change introduces many "unused" configuration parameters to satsify
Airflow's expectations[1]. An ugly, but likely harmless change to the
log output with interleaved newline characters from Airflow steps[2].
Changes to the chart and other dependendencies have also been introduced
to match this update.
[0] https://github.com/apache/incubator-airflow/blob/master/CHANGELOG.txt
[1] https://issues.apache.org/jira/browse/AIRFLOW-3099
[2] https://issues.apache.org/jira/browse/AIRFLOW-1917
Change-Id: I179dcf1f0369650b8c4519f704abb7fb495f4248
1) UCP -> Airship
2) readthedocs.org -> readthedocs.io (there is redirect)
3) http -> https
4) attcomdev -> airshipit (repo on quay.io)
5) att-comdev -> openstack/airship-* (repo on github/openstack git)
6) many URLs have been verified and adjusted to be current
7) no need for 'en/latest/' path in URL of the RTD
8) added more info to some setup.cfg and setup.py files
9) ucp-integration docs are now in airship-in-a-bottle
10) various other minor fixes
Change-Id: I4b8cc6ddf491e35d600a83f5f82d7717108e31dd
1) Use OCI Image Specs for labels instead of custom 'commit-id=xxxxx'
or legacy "Label Schema"
2) Fix missing git commit id labels on images (.revision)
3) Add human-readable title (.title) of the image, URL (.url), and
a few other properties (annotations) according to the latest Specs
Change-Id: Iedd64fa70016880d5df241d27d8dad101e10130b
l is to let user customize the base image of the component
by passing FROM=myimage during the build process. This would let any
project leveraging Airship ensure that the base image is matching the
security requirements for that project and still use the same Dockerfile.
This will also ease the control of the /etc/apt/source.list
and thereby the result of apt-get update/upgrade procedure.
2. The above goal is achievable by using docker-ce feature such as:
ARG FROM="defaultbaseimage:xx"
FROM ${FROM}
For this reason, the installation of docker.io in the Zuul gating is beeing
replaced by docker-ce.
3. Third Goal is to bring consistency with the other compoenents leveraging
Helm such as the openstack-helm and potentially use bindep the same way
the LOCI images are to ensure
4. The new syntax in the Dockerfile is still commented out until the associated
image builder have been updated to use docker-ce as they have been for the LOCI
images.
Change-Id: Ife7e1be53c7c139bdc42dee42f0798e83f4fd271
Updates the Shipyard/Airflow workflow for deploy_site and
update_site to use the deployment group/deployment strategy
information from the design.
This allows for baremetal nodes to be deployed in a design-
specified order, with criticality and success criteria driving
the success and failure of deployment.
Includes refactoring of service endpoints to reduce the need
for so much data passing.
Change-Id: Ib5e9fca535ca74d1819fe46959695acfed5b65c2
This patchset uplifts Deckhand commit to include fix
for document replacement.
This uplift is the same one used in Promenade [0].
The Shipyard README.md was renamed to README.rst for
2 reasons:
* To fix an issue with [1] breaking.
* To update content for OpenStack migration.
[0] I32d22110749b334d1fbf19f910e41ab0b7ff3a16
[1] http://logs.openstack.org/96/571296/1/check/airship-shipyard-pep8/76f6cb1/job-output.txt.gz#_2018-05-30_20_19_47_616403
Change-Id: Ie15f4234d504223e961c525339c87fca3a883ffc
Adds the validation of the existence of a deployment
configuration and a deployment strategy document to
the checks before submitting a site action for
processing.
Change-Id: I61bf67759bd919dcc31208370cb1be5a777baf54
Refactor Shipyard to be better able to leverage common
packages and conform with the target UCP standard layout.
This change supports the same tox entrypoints at
the root level, but the preferred approach is to use make
targets defined in the Makefile such as 'make tests' and
'make lint'
The previous tox.ini has moved and been
tailored to the specifics of each subproject at
src/bin/*/tox.ini
Autotmatic generation of the policy and configuration
files has been removed from the sphinx build for now
but these files will be automatically generated locally
into the docs source by using a 'make docs' command.
This may need to be revisited later to re-enable the
automatic generation of these files such that readthedocs
would still support the project layout.
Change-Id: Ifdc1cd4cf35fb3c5923414c677b781a60a9bae42
This reduces the number of layers built by the Dockerfile and puts
more of hte dependency management into a common place.
Change-Id: I416ac11749f3fe8803919994e807f46d64493b5a
As mentioned in [0], the best practice is NOT to use ENV for
DEBIAN_FRONTEND as it will persists in the final image. It
appears that the common consensus is to use ARG instead so
that noninteractive DEBIAN_FRONTEND is set only during build.
[0] https://github.com/moby/moby/issues/4032
Change-Id: I6980e140ee5a51596a2f9ba4e1471e2c98599846
This patch set is meant to create a workflow that will allow us
to upgrade the airflow worker without causing disruption to the
current running workflow.
Note that we will set the update strategy for airflow worker
to 'OnDelete'. The 'OnDelete' update strategy implements the legacy
(1.6 and prior) behavior. When we select this update strategy, the
statefulSet controller will not automatically update Pods when a
modification is made to the StatefulSet’s '.spec.template field'.
This strategy can be selected by setting the '.spec.template.updateStrategy.type'
to 'OnDelete'. Refer to [0] for more information.
[0] https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#creating-a-statefulset
Change-Id: I1f6c3564b7fba6abe422b86e36818eb2cd3454ea
The one that we are currently point to is missing critical
fixes [0]. Hence we will need to pin to the current latest
deckhand merged commit which contains the fix for the deckhand
client.
[0] https://review.gerrithub.io/#/c/402182/
Change-Id: Ic523f675e527dc1b129131b75e0c0716d086f40e
We need a side car container to perform log rotation
on the log files. Logs shall be retained for 30 days.
This is the default setting and can be changed by updating
values.yaml
Also cleaned up README.md
Change-Id: I39a7797e96abd349160d753f8917f7f78f7d8797
Updates dockerfile to use python 3.5 base image and
reorders to take advantage of image layering caching better
Removes some outdated and unused files
adds .editorconfig file
adds .vscode to gitignore
Change-Id: I08571b7176f0cb9fab42f072326c6c14d86d518a
The story requires Shipyard to get the rendered document from
Deckhand before proceeding with the rest of the workflow.
This is to ensure that the schema validation at Deckhand is
good and to fail early if need be.
This p.s. attempts to make use of the deckhand client to get
the list of rendered documents. Note that we will need to install
the DeckHand client libraries in the Airflow image in order for
this to happen.
Change-Id: I22f4165be101edfa4cef9bead6dd3d3765146c6e
Airflow version 1.9.0 was released a few weeks ago
This Patch Set will bump the version from 1.8.2 to 1.9.0
Change-Id: Ief9e33f64d0bacea4628dedc1a94342f60cfecea
This P.S. will allow user to execute Shipyard CLI using a
Shipyard docker container.
The updates made to the Dockerfile and entrypoint.sh are
aligned with those in Armada and Drydock.
Change-Id: I59ccd2d41fab19781f51334cb1a69b69a48d3c49
Set up and do a rudimentary test of the images created
by the shipyard project to see if the images are okish.
Change-Id: I1f1c8fbcfeeafff66764ae99a176ff6a6766edce
This PS migrates the Shipyard and Airflow Dockerfiles
into this repo and adds a Makefile
We will run the following command from the root directory
to build the airflow and shipyard images
$ sudo make build_airflow
$ sudo make build_shipyard
Change-Id: I9a9fb761ce193b1c5b9c5d9589982366eb73e396
