649 Commits

Author SHA1 Message Date
Carter, Matthew (mc981n)
222b074cb1 Be configuration driven when referencing document names/schemas
Currently, any document name or schema referenced in the Shipyard
code base is a hard-coded string. Often times, these strings are
repeated throughout the code. This patch set adds a new configuration
section to shipyard.conf to define document names and schemas so they
can then be referenced in the Shipyard code via the oslo
configuration object. This functionality will be important for
upcoming Shipyard features which will call for more documents to be
validated as well as some new Shipyard-created docs.

Change-Id: I34ae8cd578bab730d004c3d176e3817b5a45c89e
2019-05-02 10:47:38 -05:00
Zuul
1d519520c8 Merge "Don't log OS_PASSWORD in auth.log" 2019-04-29 18:44:37 +00:00
Zuul
902ee1b7af Merge "Update ha_celery policy pattern" 2019-04-29 18:14:19 +00:00
Crank, Daniel (dc6350)
0ab1bf552d Don't log OS_PASSWORD in auth.log
By default, all sudo commands are logged to auth.log with their
full command line. Previously, Shipyard scripts called 'sudo docker'
with -e OS_PASSWORD=foo, resulting in the password value appearing
in auth.log in plaintext. This change adds -E to the sudo command
to pass the user's environment through, and removes the value from
-e OS_PASSWORD which tells Docker to use the environment value
directly. This prevents the password value from being logged.

Change-Id: Ifcf7f6525876144a609ff42be42da57a3f7f6f60
2019-04-26 17:17:06 -05:00
Zuul
8d0b3f671f Merge "CI: Add Airskiff check" 2019-04-26 15:03:50 +00:00
Zuul
f395a2efb9 Merge "Return error if execution of the step fails" 2019-04-24 17:48:36 +00:00
anthony.bellino
d294a1939a Update ha_celery policy pattern
Updating pattern to include the default queue and all celery matching queues.

Change-Id: Id85951e9d84c2e0d43a06b3509ff364f71110092
2019-04-22 18:11:47 +00:00
Drew Walters
5556227634 CI: Add Airskiff check
This commit adds a non-voting Zuul job that deploys Memcached using the
Airship-Treasuremap airskiff site and submitted Shipyard changes.

[0] https://airship-treasuremap.readthedocs.io/en/latest/airskiff.html

Change-Id: Ibe9c6c94406bcdac4c39493b694535baf8ad2719
2019-04-22 16:39:58 +00:00
OpenDev Sysadmins
46875d8ac4 OpenDev Migration Patch
This commit was bulk generated and pushed by the OpenDev sysadmins
as a part of the Git hosting and code review systems migration
detailed in these mailing list posts:

http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html
http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html

Attempts have been made to correct repository namespaces and
hostnames based on simple pattern matching, but it's possible some
were updated incorrectly or missed entirely. Please reach out to us
via the contact information listed at https://opendev.org/ with any
questions you may have.
2019-04-19 19:52:20 +00:00
Crank, Daniel (dc6350)
70328e267b Airflow: Update Deckhand / Drydock version
A recent Shipyard change [0] introduced user-context tracing support in
Shipyard, which requires the passing of an additional parameter to the
Deckhand and Drydock API clients. The Deckhand [1] and Drydock [2]
clients were updated to handle the additional parameter, but the
versions of those components used by Shipyard was not updated. This
change uplifts the versions of Deckhand and Drydock in requirements.txt
for the Airflow component of Shipyard to a version that supports the
aforementioned, additional, user-context parameter.

[0] https://review.openstack.org/633873
[1] https://review.openstack.org/634068
[2] https://review.openstack.org/638115

Change-Id: I096191f436c5123b6c263fe92c0d69b9e4085c66
2019-04-18 11:50:41 -05:00
Bryan Strassner
b575df40e6 Update mq queue mirror directive to match vhost
Updates the definition for queue mirroring to use the same vhost as the
queue itself is defined to be.

Change-Id: Ibb631d7adb237fa17c5e853a9e9e35297a525782
2019-04-17 16:30:14 -05:00
Zuul
aa909ed2eb Merge "CI: Add chart build job for latest Helm toolkit" 2019-04-17 16:28:31 +00:00
Bryan Strassner
6f995d1fb2 Update upgrade_airflow_worker.sh script
The current script is fragile to the introduction of a DEBUG logging
level. This change addresses the specifics of logging records being
returned when invoking the airflow command enough to make the dag_status
result end up being the last result as the script desires, so the script
is returned to working order, even if the logging level is set to DEBUG.

This change goes a little further to the point of removing all records
that indicate a logging level of DEBUG|INFO|WARN|ERROR.

Change-Id: Ic72358097e3a476e20fa3713008d0d0bcd35463e
2019-04-17 08:36:35 -05:00
Evgeny L
4a065c3270 Return error if execution of the step fails
When Shipyard fails to execute some of the steps, the status
of the action is always "Complete", determine the success
of the execution based not only on action status, but
also on the status of every step within the action.

Change-Id: If7f71b55c2aed0322edef9c811ed1906400b0913
2019-04-16 21:48:07 +00:00
Zuul
b36bf8abb7 Merge "Shipyard: Add pod/container security context - deployment-shipyard" 2019-04-16 15:49:57 +00:00
Drew Walters
962126ed4c airflow: Update image reqs to match src reqs
The airflow image has a requirements file that is separate from the one
located in src/bin/shipyard_airflow, which was missed in a previous
uplift patch [0]. This change updates all versions in the image
requirements file to match the source code requirements.

[0] https://review.openstack.org/652736

Change-Id: I238bba4038e5cd8364711c2c5d1f78f51fc76948
2019-04-15 19:25:28 +00:00
Zuul
ad93fc1b02 Merge "images: Update default image tag to latest" 2019-04-15 19:01:49 +00:00
Drew Walters
cdad9e7cbc airflow: Update Armada version
A recent Shipyard change [0] introduced user-context tracing support in
Shipyard, which requires the passing of an additional parameter to the
Armada API client. The Armada client was updated to handle the
additional parameter [1]; however, the Shipyard version Armada uses was
not updated. This change uplifts the version of Armada in
requirements.txt for the Airflow component of Shipyard to a version that
supports the aforementioned, additional, user-context parameter.

[0] https://review.openstack.org/633873
[1] https://review.openstack.org/638124

Change-Id: I7dfbda842eb136858fbb6d891b96d057a977c913
2019-04-15 12:14:45 -05:00
Smruti Soumitra Khuntia
9c5270b616 User context tracing through logging
This PS adds entry in log for user id and passes on the context
maker to other Airship components from Shipyard during API call.

This will ensure easy tracing of user and context through log
tracing.

Change-Id: Ib9bfa8f20b641f8bb6c2dca967d9388e30d5735c
2019-04-04 13:19:02 +00:00
Scott Hussey
7ff21610a5 (zuul) Fix image publish job
- Fix issue in post pipeline image publish job introduced
  by Ansible update

Change-Id: Ia97f2927980f5ee5c2d5adf23a5a016b8d3b1c9f
2019-04-03 15:23:27 -05:00
Drew Walters
95179f53ff CI: Add chart build job for latest Helm toolkit
This commit introduces a non-voting job to lint Helm charts against the
latest version of Helm toolkit from OpenStack-Helm Infra. This job
should serve as an indicator of when it's safe to advance the version of
Helm toolkit used by Airship.

Additionally, this commit modifies all Helm chart lint jobs to run on
each commit, regardless of the files modified by a change. This should
not introduce a noticeable difference in CI runtime, as these jobs
execute quicker than the tox jobs.

Change-Id: Ife3b285a2f9a160df9724e9dcb230243d06be3ab
2019-04-03 14:26:37 +00:00
Zuul
279ca0f700 Merge "Adds HTTP Security Headers" 2019-04-02 18:33:24 +00:00
Rahul Khiyani
25defd8ca7 Shipyard: Add pod/container security context
- deployment-shipyard

This updates the shipyard chart to include the pod
security context on the pod template.

This also adds the container security context to set
allowPrivilegeEscalation to false and readOnlyRootFilesystem to true

Change-Id: Idb1b848847eaec2b6e24389c063b7ece2973c4dc
2019-04-02 15:37:59 +00:00
Zuul
8cfc2b228d Merge "Remove Shipyard queries for Tiller information" 2019-04-01 18:17:05 +00:00
Nishant kumar
afd2788729 Remove Shipyard queries for Tiller information
Currently Shipyard is attempting to retrieve Tiller information to pass
to Armada. Once the dependent change is merged, Tiller will be moved
into the same pod as Armada, and this information will be retrieved
from the Armada configuration file. Therefore, Shipyard should no
longer be retrieving this information for Armada.

Co-Authored-By: Michael Beaver <michaelbeaver64@gmail.com>
Depends-On: https://review.openstack.org/#/c/632788/
Change-Id: I689e396ef9f184525f09f819f13c1490344481f2
2019-03-28 16:35:29 -05:00
Drew Walters
fc16d9c2e3 tools: Update Helm to v2.13.1
Helm v2.13.1 has been released [0]. This change uplifts Helm to v2.13.1
to match other Airship projects and pins the Shipyard chart linting gate
to the latest version of Helm toolkit.

[0] https://github.com/helm/helm/releases/tag/v2.13.1

Depends-On: https://review.openstack.org/647812
Change-Id: Id9b15ad5f3253332ef0f8129b62a7b7510ab475c
2019-03-28 15:25:17 +00:00
Drew Walters
3af62864d6 images: Update default image tag to latest
This commit changes the default image tag for Shipyard and Airflow
images from `untagged` to `latest` to match other Airship projects.

Change-Id: Id1f9d56784e51f0631280188ba1b898fde8f56ea
2019-03-22 18:42:48 +00:00
Aaron Sheffield
12de088b9d Updating Docker Gate use of zuul.newrev
- Zuul updated ansible to 2.7, no longer allows missing variables.
- Using default value when it isn't available.

Based on Aaron Sheffield's PS for Pegleg: https://review.openstack.org/#/c/645631/

Change-Id: I02495bc793021b429e0be62ecac0ed45e930484c
2019-03-22 12:53:55 -05:00
Zuul
041cbcbafe Merge "tools: generalize execute_shipyard_action" 2019-03-12 19:59:35 +00:00
Zuul
38956c7b27 Merge "[Database] Shipyard DB changes" 2019-03-07 21:21:39 +00:00
Nishant Kumar
d9f145e2a6 [Database] Shipyard DB changes
- Use helm-toolkit for DB initialization [0]
- Create DB auxiliary Job for shipyard specific
  additonal DB operations
- Refactor Job dependencies

[0] https://review.openstack.org/#/c/635348/

Depends-On: https://review.openstack.org/#/c/635348/

Change-Id: I093671f9bce747b491f22dd8f38f597bd9dae9af
2019-03-07 15:16:39 +00:00
Dustin Specker
c234b72abd tools: generalize execute_shipyard_action
This enables passing any combination of actions and parameters to this
script.  This also makes it easier to add scripts for additional actions
by not having to modify execute_shipyard_action.

Updated redeploy_server to pass in servers as a param.

Change-Id: I590183c28e8c66997ab85470902dbe9576c5c5f6
2019-02-28 15:16:06 -06:00
Dustin Specker
f4f57a1bbf tools: fix execute_shipyard_action if condition
Before if condition was looking for non-existence of ${server}. `server`
is never defined, so this condition is always true.

Now if condition is looking for existence of ${servers}, which is
sometimes defined.

Change-Id: I31cd4a2765705176a9d5239a01b04ff341b36540
2019-02-28 00:11:32 +00:00
Dustin Specker
2d10945dd8 tools: fix execute typo
Change-Id: I41e45b236472d34eba2c434755373f7a6b706100
2019-02-25 19:16:21 +00:00
Bryan Strassner
62cada246d Update dependencies on Armada, Drydock, Deckhand
Updates the dependencies on Armada, Drydock, and Deckhand to current
values. The primary reason for this is to take advantage of a new set of
Armada requirements that allows for a reduced image size (no longer
includes grpc tools)

Change-Id: I8f208ab91664479a7af96c543a4e72d8f6d154ef
2019-02-22 22:36:21 +00:00
Bryan Strassner
067db5f2a0 [FIX] Fix document build
Adds the SLUGIFY_USES_TEXT_UNIDECODE=yes option to the appropriate tox
jobs to allow document builds/regeneration of directories occur in
alignment with license requirements

Change-Id: Ie0d25183e6759d1ae48aa45c504479e1d0592d52
2019-02-22 15:43:08 -06:00
Bryan Strassner
919868e16c Use UBUNTU_BASE_IMAGE instead of BASE_IMAGE
Offering better alignment with some downstream build jobs.

Change-Id: I90332926b4e5dd04b00b6008d06d36624086e313
2019-02-06 16:55:00 -06:00
Zuul
c7472d7f60 Merge "Switch to ubuntu base image" 2019-02-04 17:23:22 +00:00
Bryan Strassner
8550346b78 Switch to ubuntu base image
Change to use ubuntu base image instead of python
Refactor Shipyard Dockerfile to reduce image size significantly

BREAKING CHANGE: The `make images` PYTHON_BASE_IMAGE arg is now renamed to BASE_IMAGE.

Change-Id: I3338dfbbb91b5514fa4fd205bdfc4136d0abc2e5
2019-02-02 08:21:55 -06:00
Samantha Blanco
1f76ba1f1c Adds HTTP Security Headers
Adds HTTP security headers to Shipyard API handling

Change-Id: Ia68d2364d40c0a1e528f0f27d1ce999981a0b5bf
2019-01-29 15:41:27 -05:00
Bryan Strassner
a11e962eef Move Airflow web container into Shipyard pod
Moves the airflow web server container from its own pod into the
Shipyard pod. This removes exposed network surface area from the
Shipyard suite of software. Shipyard, after this change accesses the
Airflow API using localhost in the same k8s pod.

Change-Id: Ied4bd415a8d78c393b7256ead27a6a2176f4a2d6
2019-01-29 09:41:16 -06:00
Bryan Strassner
9725b0f337 Build workflows into Airflow image
Changes to make the docker image build to include the workflows from
Shipyard, rather than adding them to the container during Helm install
of Shipyard. This also removes the "prod" switch, as it is now always
built the same way, with the workflows in place.

Change-Id: I4acd6195cbec32193e15621e75ccaeb9879455f5
2019-01-29 09:41:16 -06:00
Bryan Strassner
6b75c7119a Move airflow scheduler to worker statefulset
Moves the airflow sceduler to a container in the airflow-worker
statefulset so that its version lifecycle matches that of the worker.
Leaves the stand-alone scheduler in place to support upgradability from
prior installations that included a standalone scheduler. New
installations are advised to turn off the scheduler template from
rendering using the values.yaml flag.

This is an attempt to make disruptive upgrades to airflow less impactful
to a "update_site" action from Shipyard.

Additionally this removes the template for airflow-flower, which is not in use.

Change-Id: I0608793ee6aba1eb3ce0f5e9567655287014a0ca
2019-01-29 09:41:16 -06:00
Zuul
6bd02eea84 Merge "CI: Add chart build gate" 2019-01-29 15:36:03 +00:00
Zuul
377378febd Merge "Adding filename to logging message format for troubleshooting purpose" 2019-01-28 20:24:15 +00:00
Drew Walters
5a32c2d1fd CI: Add chart build gate
This commit introduces a chart build gate that triggers when changes are
made to charts in the charts/ directory.

Change-Id: Iec1c6da785d26e2e7273b37fd1028f3bf47ec199
2019-01-28 11:27:26 -06:00
Drew Walters
ebf4ee4785 CI: Remove OpenStack-Helm-Infra CI job
Currently, the airship-shipyard-ubuntu job deploys K8s using outdated
playbooks. The job fails due to a change [0] that introduced an AppArmor
requirement to Calico. This change removes the failing job, as its
success does not depend on changes in Shipyard.

In the future, jobs requiring K8s should be deployed using the new
Minikube gate scripts from the OpenStack-Helm infra repository. They
will be added in future changes once they are required. Additionally, a
future change will introduce a Helm linting gate for charts hosted in
the Shipyard repository.

[0] https://review.openstack.org/614805
[1] https://git.openstack.org/cgit/openstack/openstack-helm-infra/tree/tools/deployment/common/005-deploy-k8s.sh

Change-Id: I72aaff9050eab844ef0856794b059de14e2a4a28
2019-01-28 09:20:17 -06:00
pd2839
ce564e7eca Adding filename to logging message format for troubleshooting purpose
This change is realted to Airflow

Change-Id: If8fd5983fd589ddf07b960d38e0081cfb0237003
2019-01-27 22:12:56 -06:00
Zuul
1474d7856f Merge "Raise specific errors during create configdocs" 2019-01-22 20:04:51 +00:00
Zuul
ce8e8264a3 Merge "fix typos in API.rst & output.py" 2019-01-22 20:01:11 +00:00