Commit Graph

44 Commits (c98e9525c7db34734afb29d1b9fb409a08d16ef7)

Author SHA1 Message Date
Zuul 3fcf6439ec Merge "[k8s] Add kubelet to the master nodes" 5 years ago
Spyros Trigazis 6390e0dbd3 [k8s] Add kubelet to the master nodes
Add kubelet on the master nodes. This work was
done already for calico, this patch applies the
same config when calico is used as well.

story: 2003521
task: 24797

Change-Id: Id33fb59ef23da740712d9a9b7ec4205bd6579b35
5 years ago
Sergey Filatov 85981d893d Remove deprecated `tls-ca-file` option from kube-apiserver
tls-ca-file flag is unused and was removed from kube-apiserver
in kubernetes 1.11. This means that any cluster with this
option specified will fail on apiserver start
Pull request on flag removal:

Task: 24858
Story: 2003566

Change-Id: I9c192b94056629a949ee92d867e8cda5c4ff6810
5 years ago
Zuul 3a50a242d3 Merge "[k8s] Add proxy to master and set cluster-cidr" 5 years ago
Spyros Trigazis 4f121e50c5 [k8s] Add proxy to master and set cluster-cidr
1. pods with host network can not reach coredns or any svc or resolve
their own hostname
2. If webhooks are deployed in the cluster, the apiserver needs to
contact them, which means kube-proxy is required in the master node with
the cluster-cidr set.

Change-Id: Icb8e7c3b8c75a3ab087c818c8580c0c8a9111d30
story: 2003460
task: 24719
5 years ago
Spyros Trigazis 77a220671f Fix enable_cloud_provider check
The statement in configure-kubernetes-master and minion
that is checking weather to enable the cloud provider needs
to be split into two and use one '='.

Change-Id: I64b2d5be10058b2d03c406519b3d80e212844d15
story: 1775358
5 years ago
Zuul 62029fa562 Merge "Change Kubelet flexvolume directory" 5 years ago
Feilong Wang eb930a3f1a Create /etc/kubernetes/manifests on k8s master
When using calico network driver for k8s, kubelet will be
enabled/installed on master node. So we need to make sure
the /etc/kubernetes/manifests directory is accessible. Same
thing has been done for minion node.

Task: 23211
Story: 2003103

Change-Id: I33ed0ccc224179f1f8fb7968e340cbbb9805cafc
5 years ago
Kien Nguyen 1b0fbc2074 Change Kubelet flexvolume directory
In these environments, the Kubelet needs to be told to use
a different flexvolume plugin directory that is accessible
and writeable (rw). By default, it's /usr/libexec/kubernetes/\
kubelet-plugins/volume/exec/. It raised read-only directory error
when creating.

The patch simply change flexvolume dir to accessible and
writeable one.

Change-Id: Iaa470890547a2ccf734e37498e0c5286e815ff97
Task: 22565
Story: 2002723
5 years ago
Spyros Trigazis 974399a912 k8s_fedora: Add cloud_provider_enabled label
Add 'cloud_provider_enabled' label for the k8s_fedora_atomic
driver. Defaults to true. For specific kubernetes versions if
'cinder' is selected as a 'volume_driver', it is implied that
the cloud provider will be enabled since they are combined.

The motivation for this change is that in environments with
high load to the OpenStack APIs, users might want to disable
the cloud provider.

story: 1775358
task: 1775358

Change-Id: I2920f699654af1f4ba45644ab60a04a3f70918fe
5 years ago
Zuul efe1fabd37 Merge "Sync service account keys for multi masters" 5 years ago
Feilong Wang 043c57da74 Sync service account keys for multi masters
Multi master deployments for k8s driver use different service account
keys for each api/controller manager server which leads to 401 errors
for service accounts. This patch will create a signed cert and private
key for k8s service account keys explicitly, dedicatedly for the k8s
cluster to avoid the inconsistent keys issue.

Task:  21653
Story: 1766546

Change-Id: I61547405f866d3c5a84da63de66724b55af1066a
5 years ago
Bharat Kunwar ec58c23361 Add option to specify Cgroup driver for Kubelet
This patch allows specification of Cgroup driver for Kubelet service.
The necessity of this patch was realised after upgrading Docker to the
new community edition (17.3+) which defaults to  `cgroupfs` Cgroup
driver but on the other hand, Fedora Atomic (version 27) comes with
1.13. Cgroup drivers for Docker need to be identical for the two
services, Docker and Kubelet, need to be able to work together.

Story: 2002533
Task: 22079
Change-Id: Ia4b38a63ede59e18c8edb01e93acbb66f1e0b0e4
5 years ago
Zuul 1fc9d6c252 Merge "k8s_fedora: Add flannel to master nodes" 5 years ago
Feilong Wang 79c002ce7a Add calico-node on k8s master node
By current design, pods under kube-system will run on minion nodes. And
given now we're not running kubelet on master node, so calico-node is
not running on k8s master node. As a result, kubectl proxy is not
working to access dashboard. And it's confirmed with calico team that
the calico-node container must be running on master node if user want
to use kubectl proxy, see [1]. So, the solution is enabling kubelet
on master but disallow the other pods scheduled on master with

Besides, this patch includes another fix about running calico on
Fedora Atomic. Because Fedora Atomic is using NetworkManager, it
manipulates the routing table for interfaces in the default network
namespace where Calico veth pairs are anchored for connections to
containers. This can interfere with the Calico agent’s ability to
route correctly. Please see more information about this at [2].


Closes-Bug: #1751978

Change-Id: Iacd964806a28b3ca6ba3e037c60060f0957d44aa
5 years ago
Spyros Trigazis 405b0c2028 k8s_fedora: Add flannel to master nodes
To allow ther api server access pods, we need
flannel to be running on the master node.
* Run flannel on the master node in a system

Change-Id: Ic0996ba36e335e970f3d2255840b24a8b4f738b8
Closes-Bug: #1757936
5 years ago
Spyros Trigazis 205e8adafa k8s_fedora: Add kubelet authentication/authorization
* disable kubelet anonymous-auth
* enable kubelet webhook-(token) authorization
* disable kubelet cadvisor and read-only ports
* listen kubelet only on internal ipv4 ip
* update kubelet certs
* Update heapster RBAC to access kubelets
* update api config to access kubelet over https

Closes-Bug: #1758672
Change-Id: I2c6046ce5921a63a2d56f51435433497b1ff30ba
5 years ago
Ricardo Rocha 4efb58b28d k8s: allow passing extra options to kube daemons
Define a set of new labels to pass additional options to the kubernetes
daemons - kubelet_options, kubeapi_options, kubescheduler_options,
kubecontroller_options, kubeproxy_options.

In all cases the default value is "", meaning no extra options are
passed to the daemons.

Change-Id: Idabe33b1365c7530edc53d1a81dee3c857a4ea47
Closes-Bug: #1701223
5 years ago
Costin Gamenț 5a34d7d830 Check CERT_MANAGER_API if True or False
Follow-up on "Change 529818" to check variable value "True" or "False".

Change-Id: Id01ff344320983653672c9f8df12ae4038953352
Related-bug: 1734318
5 years ago
Ricardo Rocha faa9e90402 [k8s] allow enabling kubernetes cert manager api
Add a new label 'cert_manager_api' to kubernetes clusters controlling the
enable/disable of the kubernetes certificate manager api.

The same cluster cert/key pair is used by this api. The heat agent is used
to install the key in the master node(s), as this is required for kubernetes
to later sign new certificate requests.

The master template init order is changed so the heat agent is launched
previous to enabling the services - the controller manager requires the CA key
to be locally available before being launched.

Change-Id: Ibf85147316e3a194d8a3f92cbb4ae9ce8e16c98f
Partial-Bug: #1734318
5 years ago
Spyros Trigazis 2329cb7fb4 k8s: Fix kubelet, add RBAC and pass e2e tests
Due to a few several small connected patches for the
fedora atomic driver, this patch includes 4 smaller patches.

Patch 1:
k8s: Do not start kubelet and kube-proxy on master

Patch [1], misses the removal of kubelet and kube-proxy from and therefore they are started if they
exist in the image or the script will fail.
Closes-Bug: #1726482

Patch 2:
k8s: Set require-kubeconfig when needed

From kubernetes 1.8 [1] --require-kubeconfig is deprecated and
in kubernetes 1.9 it is removed.

Add --require-kubeconfig only for k8s <= 1.8.


Closes-Bug: #1718926

Patch 3:
k8s_fedora: Add RBAC configuration

* Make certificates and kubeconfigs compatible
  with NodeAuthorizer [1].
* Add CoreDNS roles and rolebindings.
* Create the system:kube-apiserver-to-kubelet ClusterRole.
* Bind the system:kube-apiserver-to-kubelet ClusterRole to
  the kubernetes user.
* remove creation of kube-system namespaces, it is created
  by default
* update client cert generation in the conductor with
  kubernetes' requirements
* Add --insecure-bind-address= to work on
  multi-master too. The controller manager on each
  node needs to contact the apiserver (on the same node)


Closes-Bug: #1742420
Depends-On: If43c3d0a0d83c42ff1fceffe4bcc333b31dbdaab

Patch 4:
k8s_fedora: Update coredns config to pass e2e

To pass the e2e conformance tests, coredns needs to
be configured with POD-MODE verified. Otherwise, pods
won't be resolvable [1].

Closes-Bug: #1738633

Change-Id: Ibd5245ca0f5a11e1d67a2514cebb2ffe8aa5e7de
5 years ago
Spyros Trigazis 2f69309eca k8s_atomic: Remove kubelet and kube-proxy from master
Currently we start kubelet as unscheduled. Before
containerizing kube. we needed to run kubelet before as
unscheduled to run the controller-manager, scheduler and
kube-proxy as static pods. There is no such need anymore.

Change-Id: I0e36606427530756d8084b643ba43880541bbe44
Partially-Implements: blueprint run-kube-as-container
Closes-Bug: #1726482
6 years ago
Spyros Trigazis 69bb03fcbe k8s_fedora: Add container_infra_prefix label
Add a label to prefix all container image use by magnum:
* kubernetes components
* coredns
* node-exporter
* kubernetes-dashboard

Using this label all containers will be pulled from the specified
registry and group in the registry.

* grafana
* prometheus

Closes-Bug: #1712810
Change-Id: Iefe02f5ebc97787ee80431e0f16f73ae8444bdc0
6 years ago
Jenkins 26a0e8a9b6 Merge "Fix usage of --kubelet-preferred-address arg for apiserver" 6 years ago
Mathieu Velten 46255dd4b1 Add a kube_tag label to control the k8s containers to pull
Separate the tag from which to pull from the kubernetes version.
With the current state the tag and the version happen to be the
the same. But, it is not decided yet in the fedoraproject how the
images are going to be tag. Finally, operators might want to try
their own container images with custom tags.

Depends-On: Icddb8ed1598f2ba1f782622f86fb6083953c3b3f
Implements: blueprint run-kube-as-container

Change-Id: I4c4bc055d7df5e65aede93464bff51e6d5971504
6 years ago
Mathieu Velten 005eeb575d Launch kube-proxy as a system container
Following up of

Depends-On: I9a7d00cddb456b885b6de28cfb3d33d2e16cc348
Implements: blueprint run-kube-as-container

Change-Id: Icddb8ed1598f2ba1f782622f86fb6083953c3b3f
6 years ago
Mathieu Velten d003e80a3a Launch k8s scheduler & controller-manager as system containers
Following up of

Depends-On: I22918c0b06ca34d96ee68ac43fabcd5c0b281950
Implements: blueprint run-kube-as-container

Change-Id: I9a7d00cddb456b885b6de28cfb3d33d2e16cc348
6 years ago
Mathieu Velten 024f2c0241 Use atomic containers for kubelet & apiserver
Use system containers based on fedora rawhide from
projectatomic [1]. Until the fedoraproject updated
the tags properly we mirror our containers in [2].
System containers are meant to be drop in replacements
of the fedora kubernetes binaries.

Update k8s to 1.7.4 to match the version in the containers.


Implements: blueprint run-kube-as-container

Change-Id: I22918c0b06ca34d96ee68ac43fabcd5c0b281950
6 years ago
yatin 8e36613965 Fix usage of --kubelet-preferred-address arg for apiserver fixed it only for
tls based cluster, we need kubectl exec/log to work with
tls-disabled clusters as well.

Change-Id: Iae2d4bc9af7fc55ab0ce2db97c6b7cf61479a2ff
Closes-Bug: #1668337
6 years ago
Mathieu Velten 4a39ad699b Move all kubernetes files in /etc/kubernetes
Kubernetes uses cetificates, kubeconfig and the kubernetes openstack
cloud provider configuration from /srv/kubernetes and /etc/sysconfig.

The upstream kubernetes system containers used with atomic hosts
mounts /etc/kubernetes, we can unify the location of all kubernetes
configuration and also be able to use the upstream containers

Implements: blueprint run-kube-as-container

Change-Id: I9b2da390745836d9a66b7c8fc995a35cb74993e9
6 years ago
Mathieu Velten 94ce3b00bd k8s: Fix apiserver configuration
kube-apiserver configuration was modified using sed and / as a
delimiter. However, the arguments were containing / so the sed command
was breaking.

Furthermore, the KUBE_API_ARGS was added without quotes which was
working when using the file in  a systemd unit, it does not work when
sourcing with . or source.

Implements: blueprint run-kube-as-container

Change-Id: I49714f51159ea1bf4e9918bbec82749443cc1751
6 years ago
PanFengyun 0ae88f6312 Specified cgroup driver
Closes-Bug: #1693525
Change-Id: Icdb4f807c7e2352c1b38e42a7186ea4009136df8
6 years ago
Ricardo Rocha 7c35c8fe40 Add CoreDNS deployment in kubernetes atomic
Enable internal cluster DNS by deploying CoreDNS in the kube-system
namespace. It covers dns queries for both the cluster and external,
acting as a proxy with a cache layer in front.

Version of CoreDNS hard-coded to 007, image taken from dockerhub.

Related-Bug: #1692449

Change-Id: I0a9703b531fe872416dcd79fa7d4d27c1ea61586
6 years ago
Jongsoo Yoon bb376f2a48 Add Command for using default docker log-driver
 Container Log file which is located in /var/log/containers cannot be
founded when k8s cluster is created based on Fedora or CentOS
It is because docker set log-driver "journald" as it's default

 Added Command into both configure-kubernetes-master and minion
It search string "--log-driver=journald" in /etc/sysconfig/docker
and then remove it.
After that, docker'll write logs into it's default.

Closes-Bug: #1690717

Change-Id: Ie8449c04c792e17e084187e5e1853c0f957717ce
6 years ago
Mathieu Velten c0787b4e94 Fix usage of the trustee user in K8S Cinder plugin
Closes-Bug: #1672667
Change-Id: I702818777ea4664ecd560c4b7a02431c86988e17
6 years ago
Vijendar Komalla 396439f703 Set k8s apiserver preferred address type arg
Currently not able to run kubectl exec/logs commands
with a k8s cluster created on devstack. This is due
to the fact that apiserver is not able to resolve
the worker node by hostname. This change fixes the
issue by passing --kubelet-preferred-address-types
argument to apiserver.

Change-Id: I9d328626723d11372a6d912fae4edd33b8f01277
Closes-Bug: #1668337
6 years ago
Mathieu Velten 42e36895ef Missing root-ca-file parameter for proper service account support
Change-Id: I8d581b1fbffdb4b8bc64457da6faae6d45dfc594
Closes-Bug: 1666599
6 years ago
Bertrand NOEL 1f3b0500b7 K8S: Allows to specify admission control plugins to enable
If nothing is specified a set of recommended default plugins is used,
which includes the ServiceAccount one.

Change-Id: I1383aae09ba68f8e83b07e3eaae40ab071f7be94
Closes-Bug: #1646489
6 years ago
Mathieu Velten ef84906b04 Use Kubernetes config to launch services pods
Mount certs, sysconfig and /srv/kubernetes in the pods in case an argument
references it (cinder support for example).

Partially-Implements: blueprint run-kube-as-container
Change-Id: If7b0f4ad956ed9492a3889b5eb9d287a235b747d
6 years ago
yatin e904a8af5c [k8s_fedora_atomic] Enable TLS in Etcd cluster
With this patch following are done:-
- Configure Etcd with TLS support

Configure Following to commuicate with TLS enabled Etcd:-
- Flannel

Etcd also listens at, so on master nodes
etcdctl and kube apiserver can communicate without using

if TLS_DISABLED="True" then TLS is not enabled for etcd.

Change-Id: I2147b67c4e346a4415e1f76c19ac68e94cb0a0fa
Partially-Implements: blueprint secure-etcd-cluster-coe
7 years ago
Dirk Mueller 80fc5a2d42 Add bashate checks to pep8 step
Similarly to pep8 checks, this allows enforcing a consistent
style of the shell scripts accross modfications. For now
only the indentation is enforced to reduce code churn.

Closes-Bug: 1648099
Change-Id: Ie66cbe1aea4bd01a8bba8833ef6cbd2cff6a7c6a
7 years ago
PanFengyun 1876211d8c Fix the config args of kubernetes service
Many parameters of magnum templates use '_' instead of '-', but
all the args of kubernetes service use '-'. So let's format the

Change-Id: I80b356f2b853008e18edce3123add89b8ef7ea26
7 years ago
Ton Ngo 324f4aca7d Fix K8s load balancer with LBaaS v1
Fix node name and auth_url

Update the url to Keystone v2 which has been changed.
The name of the node registered in the kube-apiserver
was also changed at some point to use the IP instead of the
Nova instance name as was done originally, and this
broke the Kubernetes plugin code.  Change the node name
back to the Nova instance name in the option
--hostname-override for kubelet.

Some update to the document.

With this patch, the load balancer works with Magnum Newton
and later, along with the image fedora-atomic-latest.
Important notes:
1. The current image has Kubernetes release 1.2 and this only
works with neutron LBaaS v1.  Support for LBaaS v2 requires
Kubernetes release 1.3 or later.  Magnum support for 1.3
is still in development.
2. LBaaS v1 has been removed in Newton and is only available
in Mitaka or by custom installation (likely requires some hacking).
This means to get the load balancer feature, you will want to
install Openstack Mitaka and Magnum Newton.

Change-Id: Ica9d92c8d7410bf30832005687ecce4a90ef6c58
Closes-Bug: #1524025
7 years ago
Spyros Trigazis 2c635692ae Split k8s atomic vm and ironic drivers
The 2 k8s atomic drivers we currently support are added to the
same driver. This breaks ironic support with the stevedore
work I'm currently doing.

With stevedore, we can choose only one driver based on the
server_type, os and coe. We won't be able to pick a driver and
then choose an implementation bases on server_type.

Partially-Implements: blueprint magnum-baremetal-full-support
Co-Authored-By: Spyros Trigazis <>
Change-Id: Ic1b8103551f48f85baa2ed9ff32d5b70b1fab84e
7 years ago