4637 Commits

Author SHA1 Message Date
Theodoros Tsioutsias
18c77a288d ng-2: Adapt existing cluster APIs and conductor
This changes the existing cluster APIs and the cluster conductor to
take into consideration nodegroups:

* create: now creates the default nodegroups for the cluster
* update: updates the default nodegroups of the cluster
* delete: deletes also the nodegroups that belong to the cluster
* cluster_resize: takes into account the nodegroup provided by the API

story: 2005266

Change-Id: I5478c83ca316f8f09625607d5ae9d9f3c02eb65a
2019-03-28 10:31:01 +00:00
Feilong Wang
70f1dbd9c7 Kubernetes images release
Releases:

v1.11.9
v1.12.7
v1.13.5
v1.14.0

Change-Id: I07539727744ad4dc7b6f665ef14cdbe1b6c00611
2019-03-28 10:03:02 +13:00
Feilong Wang
1f5dc1aa91 [fedora-atomic-k8s] Allow all traffic from master to worker nodes
In Rocky release, the k8s workers security group was wide opened but
in Stein release it is more restrictive which prevent the access of
Kubnertes dashboard(and other serivces) via the command:

  $ kubectl proxy

This patch can fix it by allowing traffic from master security group
to workers security group.

Co-Authored: Feilong Wang<flwang@catalyst.net.nz>

Task: 30171
Story: 2005294

Change-Id: I546cd7324b87b267e945477c78539ea80534538f
2019-03-26 10:30:22 -04:00
Zuul
a6c8c399e9 Merge "Add API ref for <ClusterID>/actions/resize" 2019-03-25 10:09:58 +00:00
Zuul
8a87054725 Merge "Replace openstack.org git:// URLs with https://" 2019-03-25 10:09:56 +00:00
Feilong Wang
66c6666a83 Add API ref for <ClusterID>/actions/resize
Task: 29737
Story: 2005054

Change-Id: I5511303c8c08a330bdfc0104290c6f8f1831e4b2
2019-03-24 21:11:50 +00:00
Zuul
714ee99756 Merge "Update master for stable/stein" 2019-03-24 20:48:51 +00:00
Zuul
e63611ad1a Merge "ng-1: Add nodegroup representation" 2019-03-24 20:48:47 +00:00
Ian Wienand
ae6933ffce Replace openstack.org git:// URLs with https://
This is a mechanically generated change to replace openstack.org
git:// URLs with https:// equivalents.

This is in aid of a planned future move of the git hosting
infrastructure to a self-hosted instance of gitea (https://gitea.io),
which does not support the git wire protocol at this stage.

This update should result in no functional change.

For more information see the thread at

 http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003825.html

Change-Id: Ie288c147a3cbdd19abd257bf14972c316db6d67c
2019-03-24 20:34:05 +00:00
Zuul
f1f96e5835 Merge "add python 3.7 unit test job" 2019-03-22 17:24:41 +00:00
5e0672a477 Update master for stable/stein
Add file to the reno documentation build to show release notes for
stable/stein.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/stein.

Change-Id: Ib327c9320ec306098769040df8188e8968913ef4
Sem-Ver: feature
2019-03-21 21:38:41 +00:00
Zuul
5c586aed3c Merge "Fix openstack-cloud-controller-manager restarts" 2019-03-21 21:18:34 +00:00
Theodoros Tsioutsias
0607c7a9d6 ng-1: Add nodegroup representation
This adds the object and db schema changes needed for supporting
nodegroups.

story: 2005266

Change-Id: Ibf10277a52aa94c4b217cf3b364844b04baab1e0
2019-03-21 16:19:56 +00:00
Diogo Guerra
a46d2ffc91 [k8s] Install prometheus monitoring with helm
The Kubernetes Helm repository includes in its stable distribution
a prometheus-operator Chart.
This stable/prometheus-operator chart can be used to install all the
dependencies and some default configurations to use prometheus.
The installed extra charts are:
  * stable/prometheus-node-exporter (data scraping)
  * stable/prometheus (prometheus and alertmanager server)
  * stable/grafana (visualization dashboard)
  * stable/prometheus-operator (supervision and simple configuration)

The prometheus-operator is installed by using the label
monitoring_enabled=True. Also, the label grafana_admin_passwd can be
used to set the admin password for access to the grafana dashboard

This patch allows for transferral of prometheus monitoring maintenance
work to be done by the kubernetes/helm team.

Task: 28544
Story: 2004623
depends_on: I99d3a78085ba10030200f12bbfe58a72964e2326
Change-Id: I80d590785bf30f9d634debeaf51c0d4cce0aeb93
Signed-off-by: Diogo Guerra <dy090.guerra@gmail.com>
8.0.0.0rc1
2019-03-21 13:25:04 +01:00
Zuul
d1957c71dc Merge "Improve floating IP allocation" 2019-03-20 18:12:43 +00:00
Diogo Guerra
21acb8dc9a Fix openstack-cloud-controller-manager restarts
Openstack-cloud-controller-manager restarts several times during
cluster creation.

This happens because cloud-controller-manager starts running before
needed secrets exist in kubernetes. Cloud-controller-manager lists secrets 
and if the secrets exists it uses it and moves on, but if the secret 
doesn't exist it starts a watch until it does. As this is not allowed the
pod fails.

This is triggered by Issue 
https://github.com/kubernetes/cloud-provider-openstack/issues/545

Story: 2005270

Change-Id: If8f34dc45b3b8a76e3d561ed41b4d0a783ceecb5
Signed-off-by: Diogo Guerra <dy090.guerra@gmail.com>
2019-03-20 14:55:23 +01:00
Zuul
342023e870 Merge "Migrate legacy jobs to Ubuntu Bionic" 2019-03-20 08:15:57 +00:00
Lingxian Kong
c47fde0cbe Improve floating IP allocation
- Never allocate floating IP for etcd service.
- Introduce a new label `master_lb_floating_ip_enabled` which controls
  if Magnum allocates floating IP for the master load balancer. This
  label only takes effect when the `master_lb_enabled` is set. The
  default value is the same with `floating_ip_enabled`.
- The `floating_ip_enabled` property now only controls if Magnum
  should allocate the floating IPs for the master and worker nodes.

Change-Id: I0a232406deaf112b0cb9e445735d7b49206c676d
Story: #2005153
Task: #29868
2019-03-20 18:44:45 +13:00
Zuul
0cd35dbcca Merge "Support <ClusterID>/actions/resize API" 2019-03-19 22:16:15 +00:00
Feilong Wang
15ecdb8033 Support <ClusterID>/actions/resize API
Now an OpenStack driver for Kubernetes Cluster Autoscaler is being
proposed to support autoscaling when running k8s cluster on top of
OpenStack. However, currently there is no way in Magnum to let
the external consumer to control which node will be removed. The
alternative option is calling Heat API directly but obviously it
is not the best solution and it's confusing k8s community. So with
this patch, we're going to add a new API:

POST <ClusterID>/actions/resize

And the post body will be:

{
    "node_count": 3,
    "nodes_to_remove": ["dd9cc5ed-3a2b-11e9-9233-fa163e46bcc2"],
    "nodegroup": "production_group"
}

The API will be working in a declarative way. For example, there
are 3 nodes in the cluser now, user can propose an API request
like above. Magnum will call Heat to remove the node
dd9cc5ed-3a2b-11e9-9233-fa163e46bcc2 firstly, then bring the node
count back to 3 again.

Task: 29563
Story: 2005052

Change-Id: I7e36ce82c3f442976cc498153950b19c56a1759f
2019-03-19 20:13:17 +00:00
Spyros Trigazis
13e8c11f78 k8s_fedora: Add ca_key before all deployments
The script [1] that writes the ca.key depends in the apiserver to be
running and the script to start the apiserver [0] needs the ca.key to
exist.

Write the ca_key before all other scripts that depend on the apiserver.

story: 2005254
task: 30051

[0]
https://github.com/openstack/magnum/blob/master/magnum/drivers/common/templates/kubernetes/fragments/enable-services-master.sh
[1]
https://github.com/openstack/magnum/blob/master/magnum/drivers/k8s_fedora_atomic_v1/templates/kubecluster.yaml#L843

Change-Id: If532ccc4673225eb1b7e7cab77a30950ee5ee695
Signed-off-by: Spyros Trigazis <spyridon.trigazis@cern.ch>
2019-03-18 10:48:06 +01:00
Zuul
0da8288ada Merge "ci: Disable functional tests" 2019-03-13 11:26:21 +00:00
ghanshyam
b5a6ee1dc1 Migrate legacy jobs to Ubuntu Bionic
We have migrated the zuulv3 job to Bionic during Dec/Jan month.
 - http://lists.openstack.org/pipermail/openstack-discuss/2018-December/000837.html
 - https://etherpad.openstack.org/p/devstack-bionic
But that effort does not move all gate job to Bionic as there are
large amount of jobs are still legacy jobs. All the legacy jobs still
use Xenial as nodeset.

As per the decided runtime for Stein, we need to test everything on openstack
CI/CD on Bionic - https://governance.openstack.org/tc/reference/runtimes/stein.html

Below patch move the legacy base jobs to bionic which will move the derived jobs
automatically to bionic. These jobs are modified with branch variant so that they will use
Bionic node from stein onwards and xenial for all other stable branches
until stable/rocky.
- https://review.openstack.org/#/c/639096

This commit remove the overridden nodeset from magnum legacy jobs
so that it will start using the nodeset defined in parent job.

More Details: 
- https://etherpad.openstack.org/p/legacy-job-bionic
- http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003614.html

Depends-On: https://review.openstack.org/#/c/641886/
Change-Id: Ia5f037432f4c5925f916e19cbe8a3253869674d9
2019-03-13 01:24:50 +00:00
Zuul
e6f4969539 Merge "[fedora-atomic-k8s] Adding Node Problem Detector" 2019-03-12 22:05:22 +00:00
Feilong Wang
c39f1150e5 [fedora-atomic-k8s] Adding Node Problem Detector
Deploying Node Problem Detector to all nodes to detect problems which
can be leverage by auto healing. This is the first step of enabling
the auto healing feature.

Task: 29886
Story: 2004782

Change-Id: I1b6075025c5f369821b4136783e68b16535dc6ef
2019-03-11 22:39:50 +00:00
Zuul
988cbb8b49 Merge "Add missing ws separator between words" 2019-03-11 22:17:41 +00:00
Spyros Trigazis
16c2a4cfe3 ci: Disable functional tests
We currently run only vexxhost with nested
virtualization. Due to a kernel change all
functional jobs are failing.

Change-Id: I9ab45da36dbc5618587b4795658b4f4bb264f2c8
Signed-off-by: Spyros Trigazis <spyridon.trigazis@cern.ch>
2019-03-11 20:20:22 +01:00
Jonathan Rosser
2595fda3e3 Ensure http proxy environment is available during 'atomic install' for k8s
The scripts run by cloud-init for the master and minion nodes currently
write proxy environment variables into /bin/bashrc when they are defined.

These variables will only be introduced into the running environment
when a new bash shell is started. The /bin/sh used by the fragment
scripts will ignore /etc/bashrc, so the new shells invoked per fragment
will not have the http proxy variables present. This means that the
master/minion node deployment fails when behind an http proxy.

This patch adds explicit exports for HTTP_PROXY and HTTPS_PROXY when those
variables are defined, and not empty.

Task: 29863
Change-Id: Id05c90d5bf99d720ae6002b38d3291e364e1e0c4
2019-03-07 22:16:38 +00:00
Zuul
90dfeaa491 Merge "Fix swarm functional job" 2019-03-07 21:37:46 +00:00
Zuul
24775e0eb3 Merge "Update min tox version to 2.0" 2019-03-07 21:37:45 +00:00
Zuul
f0175f6aac Merge "[k8s] Make flannel self-hosted" 2019-03-07 21:37:40 +00:00
Zuul
722fc56eb3 Merge "Return health_status for cluster listing" 2019-03-07 11:05:58 +00:00
Zuul
373286368d Merge "make sure to set node_affinity_policy for Mesos template definition" 2019-03-06 21:10:57 +00:00
Zuul
c11c40a04d Merge "Fix prometheus installation script" 2019-03-06 15:44:39 +00:00
Zuul
6505aa360d Merge "Do not exit in the enable-helm-tiller script" 2019-03-06 09:46:49 +00:00
Spyros Trigazis
2ab874a5be [k8s] Make flannel self-hosted
Similar to calico, deploy flannel as a DS.
Flannel can use the kubernetes API to store
data, so it doesn't need to contact the etcd
server directly anymore.

This patch drops to relatively large files for
flannel's config, flannel-config-service.sh and
write-flannel-config.sh. All required config is
in the manifests.

Additional options to the controller manager:
--allocate-node-cidrs=true and --cluster-cidr.

Change-Id: I4f1129e155e2602299394b5866165260f4ea0df8
story: 2002751
task: 24870
2019-03-05 18:33:45 +01:00
Nguyen Hai Truong
18fc68dd26 Update min tox version to 2.0
The commands used by constraints need at least tox 2.0.
Update to reflect reality, which should help with local running of
constraints targets.

Change-Id: Iece749b90ec90bec1f5324bc351878e6252720ed
2019-03-05 11:56:54 +11:00
Feilong Wang
83c8b13bf0 Release k8s v1.11.8, v1.12.6 and v1.13.4
Release new k8s version because of CVE-2019-1002100[1]

[1] https://discuss.kubernetes.io/t/kubernetes-security-announcement-v1-11-8-1-12-6-1-13-4-released-to-address-medium-severity-cve-2019-1002100/5147

Task: 29789
Story: 2005124

Change-Id: I6435a10b05932ea71e825e944d53859eba374e91
2019-03-03 20:55:47 +00:00
Guang Yee
a47f5a3994 make sure to set node_affinity_policy for Mesos template definition
Fixes the problem with Mesos cluster creation where the
nodes_affinity_policy was not properly conveyed as it is required
in order to create the corresponding server group in Nova.

Change-Id: Ie8d73247ba95f20e24d6cae27963d18b35f8715a
story: 2005116
2019-03-01 15:49:06 -08:00
Zuul
e256f87d1a Merge "[k8s-fedora-atomic] Use ClusterIP for prometheus service" 2019-03-01 02:36:49 +00:00
Feilong Wang
e4b05bbd1a Fix swarm functional job
Now swarm functional job failed due to a a regression issue caused by
If11ba863a2aa538efe1e3e850084bdd33afd27d2 This patch fixes.

Task: 29766
Story: 2004195

Change-Id: I830ab66775e0dd57766cdab25d06500d85651dc1
2019-03-01 14:36:33 +13:00
Lingxian Kong
2cf4df0850 Fix prometheus installation script
- Fix the indent in the file.
- Use 'kubectl apply' instead of 'kubectl create' for more robust
  service restart.
- Do not retry infinitely when Prometheus datasource already injected
  into Grafana

Story: #2005117
Task: #29765

Change-Id: I5857fe62f922d27860946fd318296950834a8797
2019-03-01 14:16:36 +13:00
Feilong Wang
8c8cd7d199 Return health_status for cluster listing
Task: 29761
Story: 2002742

Change-Id: If702584fabe1402257b45db281561a5f5b83b972
2019-03-01 12:08:01 +13:00
Lingxian Kong
3695536085 Do not exit in the enable-helm-tiller script
The scripts included in the Heat kube_cluster_config resource should not exit
if the particular step is skipped.

Change-Id: I2d4cf54631c8ed3a9eb30b3e6c8e1af0007e23d5
Story: #2005109
Task: #29743
2019-03-01 12:03:52 +13:00
Zuul
57a3b73fa0 Merge "Fix async reserved word in python3.7" 2019-02-28 17:03:18 +00:00
Zuul
c181fce90d Merge "FakeLoopingCall raises IOError" 2019-02-28 17:03:13 +00:00
Zuul
6d85d7be56 Merge "python3 fix: decode binary cert data if encountered" 2019-02-28 11:28:40 +00:00
Theodoros Tsioutsias
14b46ea22b FakeLoopingCall raises IOError
All unittests using FakeLoopingCall raise an IOError if an initial
delay is not specified, because the default initial_dealy is -1.
Changing the default initial delay to 0.

story: 2005112
task: 29748
Change-Id: I6cbae0996c2347e25d8be617e4b3fd93f4d9cc95
2019-02-28 10:01:17 +00:00
Zuul
d76ab4da80 Merge "[k8s-fedora-atomic] Security group definition for worker nodes" 2019-02-27 23:59:12 +00:00
Lingxian Kong
31c82625d6 [k8s-fedora-atomic] Security group definition for worker nodes
Defines more strict security group rules for kubernetes worker nodes. The
ports that are open by default: default port range(30000-32767) for
external service ports; kubelet healthcheck port; Calico BGP network ports;
flannel overlay network ports. The cluster admin should manually config the
security group on the nodes where Traefik is allowed.

Story: #2005082
Task: #29661
Change-Id: Idbc67cb95133d3a4029105e6d4dc92519c816288
2019-02-27 22:15:46 +00:00