Fedora Atomic 27 has end of life for a while, it's time to replace it
with Fedora Atomic 29 now.

Task: 36356
Story: 2006441

Change-Id: Iab131745854b0b908be17bd17c7510cd54dde1f5
(cherry picked from commit 703de97cd469f8d077c247e55034f723343c50d0)

[stable branch policy requiring a backport rather than an 'in branch'
revert means that the following revert SHA refers to the parent patch,
not to this branch]

This reverts commit e8d0ee1b14.

This commit is reverted for two reasons:

* It is undesirable that the end user can inject proxy config into
  the magnum-conductor service via the cluster template.

* The proxy settings for the magnum-conductor service may not be
  the same as those which are required in the cluster template for
  the end user VM.

Systemd, docker and podman all include native mechanisms for setting
environment variables for proecesses, and this should be used by the
cloud operator / deployment tooling to configure the required proxy
settings for the magnum-conductor service.

In particular this patch makes it impossible for the cloud operator
to specify their own http_proxy via the environment, the user supplied
cluster template setting will always be used.

conflicts: magnum/drivers/heat/template_def.py

Change-Id: I33da19ad6764bedcf15f2a08381063e2471f8991
(cherry picked from commit 79699f23cd)
(cherry picked from commit 1c4f8127f1)

This fixes an issue with --registry-enabled in k8s_fedora_atomic where
the registry container fails to start in the minion due to two missing
heat parameters: TRUSTEE_USERNAME and TRUSTEE_DOMAIN_ID.

Change-Id: Ib93a7c0f761d047da3408703a5cf4208821acb33
Task: 23067
Story: 2003033
(cherry picked from commit 00522c5ba20bdd8939e3ff1d8daca48c2507dde7)
(cherry picked from commit 9b53aac3774420122fb85ad0977224caf8182c07)

There's a regression[0] in bandit 1.6.0 which causes bandit to stop
respecting excluded directories, and our tests throw a bunch of
violations. Blacklist this version, but allow newer versions as there is
already a pull request[1] to fix it, and I expect it will be included in
the next release.

Also fix the requirements job which was broken by
https://review.opendev.org/657890 adding a cap on Sphinx on Python 2.

[0] https://github.com/PyCQA/bandit/issues/488
[1] https://github.com/PyCQA/bandit/pull/489

Co-Authored-By: Jake Yip <jake.yip@unimelb.edu.au>

Task: 33401
Story: 2005740

Change-Id: I34dc36c5236debc42424073af2c2d2104e18179a
(cherry picked from commit 913636b6b1fdcdbcfed951ad1ca68a5f84f7b8e0)
(cherry picked from commit eec7184fbc2de00d30752f16160c9553bdd2df7d)

These periodic jobs are failing, let's disable the periodic jobs.
They have been broken since ages without fixing.
There's no record of a successful run under Zuul v3.
Last images at http://tarballs.openstack.org/magnum/images/
are from 2017.

This only backports the disabling of the jobs.

Change-Id: I01122fa029b4124d912e80ea43bca07b8f2ebe5c
(cherry picked from commit 42c75c4d3d)

stable/rocky branch should use rocky-stable container - protect against
changes from stein-dev.

Change-Id: I08d6606a5c4138866b1c0250b5ea2bc539b06e26

This commit was bulk generated and pushed by the OpenDev sysadmins
as a part of the Git hosting and code review systems migration
detailed in these mailing list posts:

http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html
http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html

Attempts have been made to correct repository namespaces and
hostnames based on simple pattern matching, but it's possible some
were updated incorrectly or missed entirely. Please reach out to us
via the contact information listed at https://opendev.org/ with any
questions you may have.

This is a mechanically generated change to replace openstack.org
git:// URLs with https:// equivalents.

This is in aid of a planned future move of the git hosting
infrastructure to a self-hosted instance of gitea (https://gitea.io),
which does not support the git wire protocol at this stage.

This update should result in no functional change.

For more information see the thread at

 http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003825.html

Change-Id: I93d7ecdaefefd289782d175b99c6c14940d8efd3

The script [1] that writes the ca.key depends in the apiserver to be
running and the script to start the apiserver [0] needs the ca.key to
exist.

Write the ca_key before all other scripts that depend on the apiserver.

story: 2005254
task: 30051

[0]
https://github.com/openstack/magnum/blob/master/magnum/drivers/common/templates/kubernetes/fragments/enable-services-master.sh
[1]
https://github.com/openstack/magnum/blob/master/magnum/drivers/k8s_fedora_atomic_v1/templates/kubecluster.yaml#L843

Change-Id: If532ccc4673225eb1b7e7cab77a30950ee5ee695
Signed-off-by: Spyros Trigazis <spyridon.trigazis@cern.ch>
(cherry picked from commit 13e8c11f78)

The scripts run by cloud-init for the master and minion nodes currently
write proxy environment variables into /bin/bashrc when they are defined.

These variables will only be introduced into the running environment
when a new bash shell is started. The /bin/sh used by the fragment
scripts will ignore /etc/bashrc, so the new shells invoked per fragment
will not have the http proxy variables present. This means that the
master/minion node deployment fails when behind an http proxy.

This patch adds explicit exports for HTTP_PROXY and HTTPS_PROXY when those
variables are bith defined, and not empty.

Change-Id: Id05c90d5bf99d720ae6002b38d3291e364e1e0c4
(cherry picked from commit d6706c0c00)

Fixes the problem with Mesos cluster creation where the
nodes_affinity_policy was not properly conveyed as it is required
in order to create the corresponding server group in Nova.

Change-Id: Ie8d73247ba95f20e24d6cae27963d18b35f8715a
story: 2005116
(cherry picked from commit a47f5a39944920f694114c0e1cb964bbf60c93ba)

On node reboot, kubelet and kube-proxy set
iptables -P FORWARD DROP which doesn't work with
flannel in the way we use it.
Add a systemd unit to set the rule to ACCEPT after
flannel,docker,kubelet,kube-proxy.

Squashed in this patch, is the release notes patch [0]

[0] I07771f2c4711b0b86a53610517abdc3dad270574 which is
(cherry picked from commit e6b3325120)

Change-Id: I7f6200a4966fda1cc701749bf1f37ddc492390c5
Co-Authored-By: Spyros Trigazis <spyridon.trigazis@cern.ch>
(cherry picked from commit cf5f78e5be)

This patch makes the regex more broad to cover both services and ingresses that
create load balancers in octavia. In cloud-provider-openstack repo, the load
balancer description is defined in:

- f056677572/pkg/cloudprovider/providers/openstack/openstack_loadbalancer.go (L459)
- f056677572/pkg/ingress/controller/openstack/octavia.go (L316)

The floating ip description is defined in:

- f056677572/pkg/cloudprovider/providers/openstack/openstack_loadbalancer.go (L1063)
- f056677572/pkg/ingress/controller/openstack/neutron.go (L72)

Change-Id: I7ce38e38de0d66fac340e287ce520de4c4bf679d
Story: 2004839
(cherry picked from commit d2f36a2ed171f6be2c57d98a038edf1e97cc88a5)

To get a better cluster template versioning and relieve the pain
of maintaining public cluster template, the patch is proposing
that the name of cluster template can be changed.

A folllowing patch/spec will be proposed to add a new field
'deprecated' to allow ops to hide old/deprecated templates.

Task: 26889
Story: 2003960

Change-Id: Id1db81d35bc3dccff0fac481be7801de200d52de

- Add "octavia" as one of the "ingress_controller" options.
- Add label "octavia_ingress_controller_tag".
- Use external network ID in the heat templates.

Story: 2004838
Change-Id: I7d889a054cd5feb2eeef523b20607a6c7630d777
(cherry picked from commit a941822c8ecffa50f3b16ff137eba3f7c9897ca5)

When user creates LoadBalancer type service in k8s cluster, a floating
ip may be created and associated with the load balancer VIP. Magnum
now could delete the load balancers automatically in the cluster
pre-delete method, should also remove the floating ip as needed.

This patch depends on the github PR for cloud-provider-openstack:
https://github.com/kubernetes/cloud-provider-openstack/pull/433

Story: 2004836
Change-Id: Ia553aff4e66033346c6bfe120a72992bec79e136
(cherry picked from commit f63761a804b240dd0e33832e1e57d9cdb3873277)

Now cloud-provider-openstack of Kubernetes has a webhook to support
Keystone authorization and authentication. With this feature, user
can use a new label 'keystone-auth-enabled' to enable the keystone
authN and authZ.

DocImpact
Task: 21637
Story: 1755770

Change-Id: I3d21ad8f55c0d7308a302f62db9e9af147a604f8
(cherry picked from commit 59da4e25a6a31e296f8ad734395a791015769424)

Add heat_container_agent_tag label to allow users select the
heat-agent tag. Stein default: stein-dev

story: 2003992
task: 26936

Change-Id: I6a8d8dbb2ec7bd4b7d01fa7cd790a8966ea88f73
Signed-off-by: Spyros Trigazis <spyridon.trigazis@cern.ch>
(cherry picked from commit c98e9525c7db34734afb29d1b9fb409a08d16ef7)

There are 2 changes included in this patch:

1. Using cluster ip instead of fixed ip for grafana service to
make sure the address is reachable.

2. Move node exporter to prometheus-monitoring namespace and
make it as a DaemonSet to collect metrics from master node.

Task: 28468
Story: 2004590

Change-Id: I9090c6dc4b38e1a1466c4c3a6a827d95c089fb41
(cherry picked from commit b6936894c427af2a7e8aa0e5fba79327bfb1469e)

HTTP(S) proxy can be specified when creating the template.

https://docs.openstack.org/magnum/latest/admin/magnum-proxy.html

However, it is not being utilized when talking to a public etcd discovery
service, which result in failed cluster creation. We need to be able to
use HTTP(S) proxy when services are running behind a firewall.

Change-Id: I13d86b0dc7c232a51149107f0412219388d8c2cd
(cherry picked from commit ffc61816c8)

Change-Id: I77f46ac03cfd3f1e86db6e3327b37c27d8f0edab
(cherry picked from commit 6390e0dbd3)

* Use the external cloud-provider [0]
* Label master nodes
* Make the script the deploys the cloud-provider and clusterroles
  for the apiserver a SoftwareDeployment
* Rename kube_openstack_config to cloud-config,
  for cinder to workm the kubelet expects the cloud config name only
  like this. Keep a copy of kube_openstack_config for backwards
  compatibility.

Change-Id: Ife5558f1db4e581b64cc4a8ffead151f7b405702
Task: 22361
Story: 2002652
Co-Authored-By: Spyros Trigazis <spyridon.trigazis@cern.ch>
(cherry picked from commit 6c61a1a949615f6dc1df36f3098cd97466ac7238)

- Start workers as soon as the master VM is created, rather than
  waiting all the services ready.
- Move all the SoftwareDeployment outside of kubemaster stack.
- Tweak the scripts in SoftwareDeployment so that they can be combined
  into a single script.

Story: 2004573
Task: 28347
Change-Id: Ie48861253615c8f60b34a2c1e9ad6b91d3ae685e
Co-Authored-By: Lingxian Kong <anlin.kong@gmail.com>
(cherry picked from commit cae7fa21b63d471bb5bbc878fee68cace7a7d4a6)

For k8s cluster, the loadbalancers created for LoadBalancer type
services should be deleted before the cluster deletion.

Change-Id: I75f44187b7be7d0ffb6a8f195f755de4b1564335
Closes-Bug: #1712062
(cherry picked from commit e18ced4d5c)
Depends-On:  https://review.openstack.org/#/c/625766/

Adding the client enables the manipulation of Octavia
resources with Magnum such as during cluster deletion,
being able to clean up non-heat created resouces.

Change-Id: I976ab136e24b98d447d61028ce07d0f5dd9d255a
story: 2004259
task: 27795
(cherry picked from commit 9a6698fb4535e408b6c4a522088197af0ab4aa4d)

The functional jobs are currently deploying with an empty list of
admission control list which means that the service token controller
does not go up resulting in the cluster failing to go up.

This patch drops that so it uses the default values of the admission
controllers in order to get the cluster to go up cleanly.

Change-Id: I0fdd65a9859f34e202016c37620f553623ef8a3e
(cherry picked from commit 3646a59eac)

The API for delete_namespaced_service requires a body which when
missing will fail the functional tests.

This patch addresses that issue by adding an empty body in order
for the delete to work properly.

Change-Id: I3b4a4bb08f60d9d18368dd3faa84ab1348acb543
Story: #2002589
Task: #28341
(cherry picked from commit e0fd3f9ef1)

This patch changes the intergration jobs to use nodes which are
at VEXXHOST that come with supported nested virtualization in
order to allow for the functional jobs to finish on-time.

Change-Id: Ie275caac1a40fb3f10a0653b66611d2ba9d1c470
Story: #2002589
Task: #28341
(cherry picked from commit 2e2ebaf679)

At the moment, if a cluster fails to be created, we fall back to
getting the node information from Heat directly.  However, this
behaviour doesn't work at the moment because `self.cluster` is
a copy of the API record on-create which does not have the stack
ID yet.

This patch makes an extra HTTP request to get the `stack_id` in
order to get the server IPs and be able to pull down the correct
information.

Story: #2002589
Task: #28341
Change-Id: I3fb3542f8ab63f38a23094d579d3df1fcb99a497
(cherry picked from commit d6cc77b16f)

* Added support for www_authenticate_uri in ContextHook.
* Made code path consistent with keystone.py impl.

Story: 2004271
Task: 28073

Change-Id: I7e3f23964a55be3255e87a4c4af7bae0a1415676
(cherry picked from commit f27bde71719905e6f274a1a57799595780bc50c2)

We do currently not support www_authentication_uri at all, which
is the new standard, as auth_uri has long been deprecated.

* Make sure we support both auth_uri and www_authenticate_uri.
* Switched to www_authenticate_uri for devstack.
* Fixed a bug where a bad exception would be thrown if auth_uri
  was not set.
  
Story: 2004271
Task: 27819

Change-Id: Ibc932d35f3d6ba2ac7ffb6193aa37bd4a3d4422e
(cherry picked from commit 718cb9c9b475a705783c0cd07a0c02b9be33f0c6)

Switch to systemd logging to take advantage of some of the newer
logging features.

Story: 2004272
Task: 27820

Change-Id: I475bf26e24b3a725f68c7da355807374bf1e189b
(cherry picked from commit daa7d0495119f02abfe53142ca237a4084db5297)

The cluster name is useful to identify resources created in different
k8s clusters, especially in the cloud environment, the cluster name is
always injected into the name of the cloud resources(e.g. the load
balancer, volume, etc.), which is helpful for the cluster resource
clean up.

The magnum cluster UUID is used as the value of '--cluster-name' option.

Story: 2004242
Task: 27766
Change-Id: I245a8869948a0b8bfa8d5cc32e7fb9277477026a
(cherry picked from commit 5d1eab9d9f896f6adf5a31a17c43995377a93f78)