Commit Graph

113 Commits (c6e50da9e520ac132ba4a6a3286b461fb6cede51)

Author SHA1 Message Date
Zuul c6e50da9e5 Merge "Switch jobs to use fedora-34 nodes" 2021-07-28 06:03:53 +00:00
James E. Blair a32c31bc3c Add docs
Change-Id: I2975607a51860df88863454785ae5d189f6f2831
2021-07-23 10:27:04 -07:00
James E. Blair 245f6d4713 Support fingergw
Change-Id: I5bb8eca947e387f3236c9fb8e0ca6a91772dc338
2021-07-23 10:27:02 -07:00
James E. Blair 5358c6dc0e Support imagePrefix and versions
Allow the user to specify an image prefix and a version tag.

This updates the functional test to account for a recent change
in Zuul which returns incomplete builds on the API endpoint.

Change-Id: I49e82c790a518c1c9f5a463250b5b785eacde55a
2021-07-23 09:47:08 -07:00
James E. Blair ce1f8b338c Add keystore password support
Zuul now requires a keystore password set in zuul.conf.

Change-Id: I2269b2dce156265a084b2a68687a233131933b00
2021-07-20 13:20:45 -07:00
James E. Blair a0b182199e Add merger support
Also, update web and executors to use the replica count from the
spec.

Change-Id: Ifb9ce9d82a18e5b4dd2e30dd7968c2aa32cbeb4f
2021-07-20 13:20:40 -07:00
James E. Blair e503cef9d5 Pass through extra scheduler config options
This lets users specify params like "relative_priority" without
the operator needing to know about them.

Change-Id: I8a82725e51e8c7172063e0fe944a9ee9f13bada1
2021-07-20 13:20:37 -07:00
James E. Blair 088e663782 Support externally managed Zookeeper and DB
The external zookeeper config was mostly a stub and most of the system
actually assumed internal management.  Flesh out the support for external
management.  Do the same for the database.

This also passes through any additional keys in the zookeeper and database
sections of the spec.

Change-Id: I2014616e6366f8d93283e0a6ca38b288aa16a558
2021-07-20 13:20:33 -07:00
James E. Blair b6d8f1eb53 Move ingress to functional test
The operator currently creates an ingress object, but this was really
just to aid testing.  The operator probably shouldn't really create one
since doing so may be site-specific.

Having the ingress greatly aids local and functional testing though, so
move it to the functional test playbook.

Change-Id: Ic6296120f8c107ecc29556fdab4d6cddb5216a50
2021-07-20 13:20:07 -07:00
James E. Blair 1b699279a4 Run a git server in k8s for functional tests
To make the functional tests easier to run locally, run a git server
in k8s instead of on the local machine.

Also, update the secret creation to work with kind as well as minikube.

Change-Id: I4c00129c7105405b0f6256e769486e79c0b173fe
2021-07-20 13:19:59 -07:00
James E. Blair e5879666bd Bump API version to v1alpha2
To accomodate the CR syntax changes.

Change-Id: Ie646a9c58655b0987a8fb0d4c914cd8b93190b1b
2021-07-20 13:16:58 -07:00
James E. Blair eff9f360f7 Use kopf operator framework
This switches from the ansible/dhall operator framework to kopf,
an operator framework written in pure Python.  This allows us to:

* Build the operator application as a Python app.
* Build the operator image using the opendev python builder images.
* Run the operator as a Python CLI program "zuul-operator".
* Write procedural Python code to handle operator tasks (such as
  creating new nodepool launchers when providers are added).
* Use Jinja for templating config files and k8s resource files
  (direct pythonic manipulation of resources is an option too).

The new CR nearly matches the existing one, with some minor differences.

Some missing features and documentation are added in the commits
immediately following; they should be reviewed and merged as a unit.

Also, fx waiting for scheduler to settle in functional test since
we changed this log line in Zuul.

Change-Id: Ib37b67e3444b7cd44692d48eee77775ee9049e9f

Change-Id: I70ec31ecd8fe264118215944022b2e7b513dced9
2021-07-20 13:16:07 -07:00
Mohammed Naser 2b7d8cf7d1 Switch jobs to use fedora-34 nodes
Fedora 32 is now EOL, we should test against the newly released
version of Fedora which is 34.

Depends-On: https://review.opendev.org/c/openstack/project-config/+/795604
Change-Id: I40a9d7377a4bf5c7ef88109f9a12d35641820ca7
2021-06-09 14:16:36 -04:00
James E. Blair 0366b867bf Upload release images on tag
This uploads release images to dockerhub when we tag the repo.

Change-Id: Ic44a9d29a1e888d8d7874205b0025f286912b5e8
2021-04-10 18:37:28 -07:00
Tristan Cacqueray ee8a7b2cfb Remove command args override and update operator-framework to v1.4.2
This change fixes an issue where zuul services can't create
a lockfile.

Another issue happening with older operator-framework and newer
kubernetes version is being fixed by updating the operator-framework
following this migration documentation:
  https://sdk.operatorframework.io/docs/building-operators/ansible/migration/

Change-Id: I6adfb907184112f0b7debb050975f76cd1dd4b01
2021-04-06 17:23:56 +00:00
Daniel Pawlik 2c367363e8 Replace Fedora 31 with Fedora 32 in zuul-operator-functional
Fedora 31 is EOL and it should be replaced to new version.

Change-Id: Ib51035850528666ece39a1c3f689d73bce29b802
2020-12-17 09:37:52 +01:00
Tristan Cacqueray 3454835c63 Add zuul-operator-upload-image job
It seems like the zuul-operator-promote-image job does not
upload the image. This change adds a new gate job to upload
the image and fix the promote job.

Depends-On: https://review.opendev.org/727868
Change-Id: Ic3b41e3b29182135f71dc1adf8c6ebd9c26d798d
2020-12-04 16:38:32 +00:00
Tristan Cacqueray 76619d839a Add reformat changes to the blame ignore list
This change prevents git blame from refering the previous
reformatting changes.

Change-Id: I7bcd18e23f0f74b0a09192ee70587703d1fe7eb8
2020-07-08 13:20:57 -04:00
Tristan Cacqueray 06b4062e4b Update to dhall lang v17
This change updates the dhall code to the latest version.

* Support for record pun where `{ x = x }` can now be written as `{ x }`
* Indentation and function definitions are more compact

This change has been generated after installing dhall v1.33 using this
command: find . -name "*.dhall" -exec dhall --ascii format --inplace {} \;

Change-Id: I3b6560f26e28622aa51150dc8083d127d89a8a7b
2020-07-08 16:26:42 +00:00
Tristan Cacqueray 0fbffe66f2 Use ensure-pip role to unblock the CI
This change ensures the pip tool is installed as it is no longer
provided by the OpenDev test instance.

Change-Id: Ie9ebf946fbf847f5574c3e140f43bdb1c9949137
2020-07-07 15:00:36 +00:00
Zuul ae56155d26 Merge "Refactor ZooKeeper service configuration" 2020-05-05 17:00:56 +00:00
Zuul ce4251d9da Merge "Add support for global image prefix" 2020-05-05 17:00:55 +00:00
Zuul 0b1d0c1445 Merge "Add support for custom container image name" 2020-05-05 17:00:55 +00:00
Zuul f9c29acef3 Merge "test: refactor run tasks to include file" 2020-05-05 16:51:47 +00:00
Zuul d1213e97ff Merge "Add zuul-preview service" 2020-05-05 16:41:51 +00:00
Zuul e0d6b03dad Merge "Add registry tls secret provided by cert-manager" 2020-05-05 16:40:58 +00:00
Zuul 0a426628ae Merge "Add gearman tls secret provided by cert-manager" 2020-05-05 16:38:01 +00:00
Zuul 82a277abc3 Merge "Add initial withCertManager input toggle" 2020-05-05 16:35:20 +00:00
Zuul 7f4ceb8ee1 Merge "Refactor Zuul services" 2020-05-05 14:23:38 +00:00
Zuul c173eaa7bf Merge "Refactor backend services" 2020-05-05 14:23:37 +00:00
Zuul ffa539721c Merge "Refactor deployment and statefulset functions" 2020-05-05 14:23:37 +00:00
Zuul 26024c7ba1 Merge "Refactor kubernetes resources constructor to the function file" 2020-05-05 14:17:59 +00:00
Zuul 6cf9fb67d3 Merge "Refactor file creation to function files" 2020-05-05 14:17:42 +00:00
vass 4f182cf2b6 Use ensure-* roles
The install-* roles in zuul-jobs have been renamed to ensure-*, follow
the rename.

Change-Id: I1089398b91b9f26aeb876584256e0309aa36fb25
2020-04-27 21:53:47 +02:00
Tristan Cacqueray 65f206af17 Refactor ZooKeeper service configuration
This change simplifies ZooKeeper service configuration to
better describe the situation where the user provides
a zookeeper service.

Change-Id: Ic872b0e4a07acc2dedd79723e6189af2374da58a
2020-04-17 19:33:55 +00:00
Tristan Cacqueray ea6652e4e6 Add support for global image prefix
This change adds an imagePrefix attribute to enable using
custom images for all the service at once using a single attribute
which default to "docker.io/zuul".

Change-Id: Ia1a5dd4aa9de410937f59a769019dcdb34167944
2020-04-17 16:50:44 +00:00
Tristan Cacqueray ba2691fb47 Add support for custom container image name
This change threads the components input to their resources
function so that the container can be configured with the
optional custom image name when set.

Change-Id: I496cc0237f378057b8125f673a760dcda80ecd99
2020-04-17 15:55:41 +00:00
Tristan Cacqueray 344a0b763b test: refactor run tasks to include file
This change move tasks to dedicated files to
improve the readability of the run playbooks.

Change-Id: Iaf4829daf5f89f22b200d089ef8fd4eeee3e91bf
2020-04-15 18:05:22 +00:00
Tristan Cacqueray 9c53503051 Add zuul-preview service
This change adds an optional preview configuration to the spec:

  preview:
    image: docker.io/zuul/zuul-preview:latest
    count: 0

Change-Id: Id9d902b3b9f25b1bf3fa459634396d7520681417
2020-04-15 15:15:34 +00:00
Tristan Cacqueray 62b5ca9ad8 Add registry tls secret provided by cert-manager
This change adds a Cerficiate resource to manage
the registry tls secret with the cert-manager service.

This change also splits the registry rw user to a
dedicated secret to enable separate creation of the passwords.

Change-Id: I673ea8db31fd2926c82a4288fd9362f225794da8
2020-04-15 00:15:09 +00:00
Tristan Cacqueray 5196ced54b Add gearman tls secret provided by cert-manager
This change adds a Certificate resource to manage
the gearman tls secret with the cert-manager service.

To keep things simple, this change also merges the client
and server certificates into one secret.

Change-Id: I26e1075ccc5d9ff18bd5d2c68ffdf97244f3f87c
2020-04-15 00:07:56 +00:00
Tristan Cacqueray 20f634230d Add initial withCertManager input toggle
This change adds a new input toggle to enable using a cert-manager service.
The operator currently only setup a selfSigned CA.

Change-Id: Ifc63768a87f9508c66e4414d5286bae2969985e7
2020-04-15 00:04:21 +00:00
Tristan Cacqueray f75f52fa8d Refactor Zuul services
This change refactors the Zuul component to dedicated file
for easier maintainance.

Change-Id: I3815abb698b2e42f24289436a718d95cec7be607
2020-04-14 23:58:05 +00:00
Tristan Cacqueray a4efa4baf1 Refactor backend services
This change moves the optional Backend service to dedicated
components file for easier maintainance.

Change-Id: If3f1ce2f473ae916a64ba3a88fd72f2508288f1a
2020-04-14 23:57:59 +00:00
Tristan Cacqueray 4b68c92a0e Refactor deployment and statefulset functions
This change refactors the mkDeployment and mkStatefulset function to
enable spliting the component to their own file. The main difference
is that the method needs an extra input for the application name.

Change-Id: I1e7ac6d91f92c0e8db99ec42c76d0a41ede87d84
2020-04-14 23:55:12 +00:00
Tristan Cacqueray 66094e6c78 Refactor kubernetes resources constructor to the function file
This change refactors the mkService function for easier maintainance
and usage outside of the resource function.

Change-Id: I33ee419a66a928e953ad2ceea286301082f457aa
2020-04-14 23:53:31 +00:00
Tristan Cacqueray 3bb0f0d7d0 Refactor file creation to function files
This change splits the mkConf into dedicated functions file for
easier maintainance.

Change-Id: I7cbb3432b9ed88af8294d4bb2af7853d623cccf1
2020-04-14 17:11:28 +00:00
Tristan Cacqueray 5b6ee968b9 Add TLS configuration to ZooKeeper service
This change enables ZooKeeper TLS configuration when no
zookeeper external connection is provided.

Depends-On: https://review.opendev.org/712816
Depends-On: https://review.opendev.org/712531
Depends-On: https://review.opendev.org/712733
Change-Id: I2bef362bf7fba16fcd99229417c3e30494b2988d
2020-04-14 11:37:42 +00:00
Tristan Cacqueray c6d35be4d6 Add zuul-registry deployment
This change adds an optional registry configuration to the spec:

  registry:
    image: docker.io/zuul/zuul-registry:latest
    count: 0
    storage-size: 20
    public-url: https://registry:9000

The operator expect a {{ cr_name }}-registry-tls secret to be provided
for tls and user configuration. If the secret is missing, the operator
creates self signed certificates and generates the user password.

Depends-On: https://review.opendev.org/710644
Change-Id: I0c054485b0ad01d53ddcff93f7bcbf34d1810325
2020-04-14 11:22:20 +00:00
Tristan Cacqueray 4a12041754 Add a zuul-ensure-database-passwords role
This change replaces the hardcoded internal database password with a
secret environment key provided by the ansible role.

This change also adds the missing DB and ZK environment key required
by the web service.

Change-Id: I4f04732491c627e16988e81bc8ba9cccc78b2da1
2020-04-14 11:14:35 +00:00