Commit Graph

512 Commits (c98e9525c7db34734afb29d1b9fb409a08d16ef7)

Author SHA1 Message Date
Kirsten G d9e590bdc6 Cache barbican certs for periodic tasks
Added configuration parameter, temp_cache_dir, to magnum.conf with
default value of "/var/lib/magnum/certificate-cache". This local
directory will hold cached cluster TLS credentials that are generated
during periodic tasks, to reduce load as the number of clusters
increases. If the temp_cache_dir does not exist, the certificates
will be created as tempfiles.

Closes-Bug: #1659545

Change-Id: I8808c4098a7c8d22dbfc841142c9f9c8b976dde1
5 years ago
Zuul 7aa0c0a285 Merge "federation api: api endpoints" 5 years ago
Clenimar Filemon ec950be894 federation api: api endpoints
this commit introduces a new '/federations'
endpoint to Magnum API, as well as its controllers,
entities and conductor handlers.

this corresponds to the first phase of the
federation-api spec. please refer to [1] for more


Change-Id: I662ac2d6ddec07b50712109541486fd26c5d21de
Partially-Implements: blueprint federation-api
5 years ago
Spyros Trigazis 2329cb7fb4 k8s: Fix kubelet, add RBAC and pass e2e tests
Due to a few several small connected patches for the
fedora atomic driver, this patch includes 4 smaller patches.

Patch 1:
k8s: Do not start kubelet and kube-proxy on master

Patch [1], misses the removal of kubelet and kube-proxy from and therefore they are started if they
exist in the image or the script will fail.
Closes-Bug: #1726482

Patch 2:
k8s: Set require-kubeconfig when needed

From kubernetes 1.8 [1] --require-kubeconfig is deprecated and
in kubernetes 1.9 it is removed.

Add --require-kubeconfig only for k8s <= 1.8.


Closes-Bug: #1718926

Patch 3:
k8s_fedora: Add RBAC configuration

* Make certificates and kubeconfigs compatible
  with NodeAuthorizer [1].
* Add CoreDNS roles and rolebindings.
* Create the system:kube-apiserver-to-kubelet ClusterRole.
* Bind the system:kube-apiserver-to-kubelet ClusterRole to
  the kubernetes user.
* remove creation of kube-system namespaces, it is created
  by default
* update client cert generation in the conductor with
  kubernetes' requirements
* Add --insecure-bind-address= to work on
  multi-master too. The controller manager on each
  node needs to contact the apiserver (on the same node)


Closes-Bug: #1742420
Depends-On: If43c3d0a0d83c42ff1fceffe4bcc333b31dbdaab

Patch 4:
k8s_fedora: Update coredns config to pass e2e

To pass the e2e conformance tests, coredns needs to
be configured with POD-MODE verified. Otherwise, pods
won't be resolvable [1].

Closes-Bug: #1738633

Change-Id: Ibd5245ca0f5a11e1d67a2514cebb2ffe8aa5e7de
5 years ago
yatin 553316e40f Fix: functional CI Jobs
After [1] jobs are return false(SUCCESS) status due
to wrong EXIT_CODE.
After [2] kubernetes client is updated to v4.0.0 and
no longer contains ConfiugrationObject so we need create
instance of Configuration class.

Also don't use local to create variable as local
 can only be used in a function.


Change-Id: Ida5aac40b234a358b2a13b2e51a41d0242031ebb
5 years ago
coldmoment ba8ad5e37f Add a hacking rule for string interpolation at logging
String interpolation should be delayed to be handled
by the logging code, rather than being done at the point
of the logging call.
See the oslo i18n guideline

Change-Id: I8a4f5f896865aebbff88ee894f0081e58cfce9ef
6 years ago
yuanpeng 71d25456d2 Remove log translations
Log messages are no longer being translated. This removes all use of
the _LE, _LI, and _LW translation markers to simplify logging and to
avoid confusion with new contributions.


Change-Id: If1f4bd2f6be967368f52fb367c5a428d3eb58a9d
6 years ago
Jenkins ed173776ca Merge "Fix CVE-2016-7404" 6 years ago
Davanum Srinivas e634b55637 Switch to kubernetes upstream python client
For a really long time, we generated and maintained our very own python
client generated from kubernetes swagger json files. Now in Kubernetes
Community there is a concerted effort to organize an official python
client (also generated from swagger) for everyone to use. So let us
please switch over from our python-k8sclient and use the community
driven python client. I have ported all of our end-to-end tests and got
them working in kubernetes client-python project upstream so we should
be protected from regressions.

Implements: blueprint replace-k8sclient-with-upstream-kubernetes-client

Depends-On: I72359f2b811392008eb5267812bf343797b1553a
Change-Id: Ib81a69cfdc25198e259e3b3d4081c92c01fd1bc5
6 years ago
Johannes Grassler e93d82e8b3 Fix CVE-2016-7404
This commit addresses multiple potential vulnerabilities in
Magnum. It makes the following changes:

* Permissions for /etc/sysconfig/heat-params inside Magnum
  created instances are tightened to 0600 (used to be 0755).
* Certificate retrieval is modified to work without the need
  for a Keystone trust.
* The cluster's Keystone trust id is only passed into
  instances for clusters where that is actually needed. This
  prevents the trustee user from consuming the trust in cases
  where it is not needed.
* The configuration setting trust/cluster_user_trust (False by
  default) is introduced. It needs to be explicitely enabled
  by the cloud operator to allow clusters that need the
  trust_id to be passed into instances to work. Without this
  setting, attempts to create such clusters will fail.

Please note, that none of these changes apply to existing
clusters. They will have to be deleted and rebuilt to benefit
from these changes.

Change-Id: I643d408cde0d6e30812cf6429fb7118184793400
6 years ago
Jenkins 4e1ada7914 Merge "Integrate OSProfiler in Magnum" 6 years ago
Jason Dunsmore a65ef7d3c3 Add an API to rotate a cluster CA certificate
This will give admins a way to revoke access to an existing cluster
once a user has been granted access.

Bumped the API microversion to 1.5 for the new endpoint.

Deprecated policy certificate:get in favor of certificate:get_ca for
clarity and consistency.

Depends-On: Ie960464e45445e195e75b91e8d65a4046eb21e93
Implements: blueprint revoke-cluster-cert
Change-Id: Ief28bef3a79f212acf4166e443a96e5419fbb757
6 years ago
Tovin Seven 32d088b2c1 Integrate OSProfiler in Magnum
* Add osprofiler wsgi middleware. This middleware is used for 2 things:
  1) It checks that person who wants to trace is trusted and knows
     secret HMAC key.
  2) It starts tracing in case of proper trace headers
     and adds first wsgi trace point, with info about HTTP request

* Add initialization of osprofiler at start of service
  Currently that includes oslo.messaging notifer instance creation
  to send Ceilometer backend notifications.

* Traces HTTP/RPC/DB API calls


Co-Authored-By: Hieu LE <>
Implements: blueprint osprofiler-support-in-magnum
Change-Id: I7d68995aab81d365433950aada078ef1fcd5469b
6 years ago
Spyros Trigazis 97d5e6ebe7 Move scale managers at driver level
Following up cluster drivers implementation, move the scale managers
at driver level. This change is needed to add the driver field

Change-Id: Ia854f2354c51b5fa47095bb4cb118416f3f01a33
Implements: blueprint bay-drivers
6 years ago
Spyros Trigazis f997a332da Move monitors at driver level
Following changes for cluster-drivers, move coe specific monitors
at driver level. This change is needed to add the driver field

Change-Id: Id4658b8f7400bf3c86c8ff81756fb33d1211a0b3
Implements: blueprint bay-drivers
6 years ago
Danil Golov 9cbb142c2e Add cluster record to db right after API request
This commit changes the incorrect behavior of cluster create workflow.
Now db record with status CREATE_IN_PROGRESS is created right after
related API request.

Change-Id: I11692c4126823d49672ba5172fa45774bf0ce544
Closes-bug: #1640729
7 years ago
Randall Burt 759c1b3b2b Move cluster status updates into driver
This is an alternative implementation to:

This version implements an earlier proposal from the
spec that adds a driver method for synchronizing
cluster state. This method is optional so that drivers
that do not wish to leverage the existing periodic
synchronization task can do so in whatever manner
they wish and Magnum will not force them to do anything

1. add an update_cluster_status method to the driver
2. implment update_cluster_status for Heat drivers
   using the existing tested logic
3. Remove cluster status updates from the cluster conductor
   in favor of the periodic sync_cluster_status task - this
   should avoid timeouts and race conditions possible in the
   previous implementation
4. Update the periodic sync_cluster_status method to use
   the driver to update cluster status rather than calling
   Heat directly

Change-Id: Iae0ec7af2542343cc51e85f0efd21086d693e540
Partial-Blueprint: bp-driver-consolodation
7 years ago
Randall Burt 7890725c52 Refactor driver interface (pt 1)
Refactor driver interface to encapsulate the orchestration
strategy. This first patch only refactors the main driver
operations. A follow-on will handle the state synchronization
and removing the poller from the conductor.

1. Make driver interface abstract
2. Move external cluster operations into driver interface
3. Make Heat-based driver abstract and update based on
   driver interface changes
4. Move Heat driver code into its own module
5. Update existing Heat drivers based on interface changes

Change-Id: Icfa72e27dc496862d950ac608885567c911f47f2
Partial-Blueprint: bp-driver-consolodation
7 years ago
Jenkins 7fc0935a3e Merge "Move cluster delete method to driver" 7 years ago
fengbeihong fc55b01678 Fix magnum cluster-update error
Closes-Bug: #1636050
Change-Id: I498b0ff477f451e48a95881b8b63f3840016ac3f
7 years ago
murali allada 0d6deae21b Move cluster delete method to driver
This patch moves the cluster deletion logic into driver to conform
with the driver spec.

Change-Id: I8a4f9a94b70feddae4a137942bbb98fa36388676
Partially-Implements: blueprint bay-drivers
7 years ago
Jenkins ba11900215 Merge "Cluster Drivers" 7 years ago
murali allada 104501cfe6 Cluster Drivers
- Dynamically load drivers using stevedore
- Changed the entry points to reference drivers instead of
  template definitions
- Implement Create and update driver operations

Change-Id: I5c3259404c796e1935c872cf3109ffecae3cee02
Partially-Implements: blueprint bay-drivers
7 years ago
Wenzhi Yu 4a7d265aeb Implement mesos cluster smart scale down
We currently allow Magnum to scale down mesos cluster by removing nodes
from the cluster's ResourceGroup by updating the heat stack that created
the cluster. The problem with this approach is that Heat decides which
nodes to delete, and all containers on that node will also be deleted.

The smart cluster scale down feature has been implemented for k8s bays(
for k8s cluster, we'll ask Heat to delete nodes that have NO CONTAINERS
on them).

This patch proposes a similar implementation for a mesos cluster.

Change-Id: I00cda7f35c9db978bdc604cf86603ef58e339256
Implements: blueprint mesos-smart-bay-scale-down
7 years ago
Hieu LE 6d6cfb87b6 Centralize config option: docker section
Centralize config option of docker section.
Replace oslo_conf cfg to magnum.conf.

Change-Id: Ic8b2b2f0f9bfc6b02d75f57fab00c548b9d8f482
Implements: blueprint centralize-config-magnum
7 years ago
Hieu LE 8f9eeb801a Centralize config option: cluster_heat section
Centralize config option of Cluster Heat section.
Replace oslo_conf cfg to magnum.conf.

Change-Id: I9118eeb17061a0aa26269ea9deaba28e79f28b76
Implements: blueprint centralize-config-magnum
7 years ago
Hieu LE 4f8b26f822 Centralize config option: conductor section
Centralize config option of Conductor section.
Replace oslo_conf cfg to magnum.conf.

Change-Id: I7deea1cba16495af2081bbb12444da7e62bdf235
Implements: blueprint centralize-config-magnum
7 years ago
Madhuri Kumari 9f7295475d Add exceptions to cluster db to show failures
After changing create to async operation, all the exceptions goes hidden
and no db entry is created to show the actual failure. One has to go through
the logs to find the actual error. This patch creates a db entry in cluster
table to show the actual error in column 'status_reason'.

Change-Id: Iad6e8bfce7326b34dea04914e4552f87d2796e86
Closes-bug: #1623387
7 years ago
Jaycen Grant 729c2d0ab4 Rename Bay DB, Object, and internal usage to Cluster
This is patch 3 of 3 to change the internal usage of the terms
Bay and BayModel.  This patch updates Bay to Cluster in DB and
Object as well as all the usages.  No functionality should be
changed by this patch, just naming and db updates.

Change-Id: Ife04b0f944ded03ca932d70e09e6766d09cf5d9f
Implements: blueprint rename-bay-to-cluster
7 years ago
Cao Xuan Hoang 33bd24252f Clean imports in code
This patch set modifies lines which are importing objects
instead of modules. As per openstack import guide lines, user should
import modules in a file not objects.

Closes-Bug: #1620161

Change-Id: I7ec9022a6b1cec36c678a2cec2a1856e70a51c68
7 years ago
Jaycen Grant 0b7c6401dd Rename BayModel DB, Object, and internal usage to ClusterTemplate
This patch is the first of 3 patches to change the internal
usage of the terms Bay and BayModel. This patch updates
BayModel to ClusterTemplate. No functionality should be
changed by this patch, just naming and db updates.

Change-Id: I0803e81be6482962be2878a8ea2c7480f89111ac
Implements: blueprint rename-bay-to-cluster
7 years ago
Jenkins 42abb07835 Merge "Rename bay to cluster in certificate object and references" 7 years ago
Jaycen Grant 8e0de76aff Rename bay to cluster in certificate object and references
This is patch #2 of 3 to rename the term bay to cluster within
the internal references and objects of magnum. This patch changes
all references to the certificate objects bay_uuid field to
cluster_uuid.  Certifcate does not have a db table so no db
changes were made. No functionality is changed by this patch,
just internal naming.

Change-Id: I68a3b87b75b49de43a7855355807b50a4ae695f3
Implements: blueprint rename-bay-to-cluster
7 years ago
Stephen Watson 6ead3e4780 Updates CONF usage from bay to cluster.
Renames bay-related CONF options to their respective cluster names.
Adds release notes for CONF changes.

Change-Id: I7bbe0927e54c1f40a47bfdea448a88b467fef106
Implements: blueprint rename-bay-to-cluster
7 years ago
Jenkins 5258fddf9d Merge "Include version info in bay/cluster show operation" 7 years ago
Jenkins 4650a4b5a1 Merge "Fix bay status: after bay-delete status is not DELETE_IN_PROGRESS" 7 years ago
Vijendar Komalla 50bc376c4d Include version info in bay/cluster show operation
Currently bay-show operation does not return bay/cluster
version information. This change contain changes to return
bay/cluster version and container version info.

Change-Id: Ie12b6583e6d85faa3607f87295c04d72698034a5
Closes-Bug: #1613413
7 years ago
ztetfger 23b2610973 Fix bay status: after bay-delete status is not DELETE_IN_PROGRESS
After bay-delete, the bay status will not change until the bay is
deleted. To change the bay status, should be added.

Change-Id: Ib16e9896cdf980c9f9c1f2fc581aa723dfc52ac0
Closes-Bug: #1615891
7 years ago
yatin 272903ff9e Revert "Update mesos slave to mesos agent"
This reverts commit 15162ce33a.

Mesos services were not working, so reverted the patch
which renamed slave to agent.

Change-Id: I94a9a36e649f48f8e31386226226261ef459ce25
Closes-Bug: #1617407
7 years ago
Jaycen Grant eaddb942fd Rename Bay to Cluster in api
This is the first of several patches to add new Cluster commands
that will replace the Bay terminalogy in Magnum. This patch adds
the new Cluster and ClusterTemplate commands in addition to the
Bay and Baymodel commands.  Additional patches will be created
for client, docs, and additional functional tests.

Change-Id: Ie686281a6f98a1a9931158d2a79eee6ac21ed9a1
Implements: blueprint rename-bay-to-cluster
7 years ago
Jenkins 5372cd92b4 Merge "Rollback bay on update failure" 7 years ago
Jenkins 9b45c1b828 Merge "Set bay status: DELETE_IN_PROGRESS before updated by poll" 7 years ago
Wenzhi Yu 63b5c21c8d Rollback bay on update failure
There is a rollback mechanism in heat after the stack
update failed. There should be a rollback mechanism in
magnum after bay update failed.

This patch add new microversion 1.3 to add rollback
support for Magnum bay, user can enable rollback on bay
update failure by specifying microversion 1.3 in header(
{'OpenStack-API-Version': 'container-infra 1.3'}) and
passing 'rollback=True'(http://XXX/v1/bays/XXX/?rollback=True)
when issuing bay update reqeust.

Change-Id: Idd02769f98078702404a11dc9f7a3339ce4e22eb
Partially-Implements: blueprint bay-rollback-on-update-failure
7 years ago
yatin karel 52f30927af Set bay status: DELETE_IN_PROGRESS before updated by poll
Set bay status to DELETE_IN_PROGRESS as soon as delete stack
request returns. Currently this status is updated by Heat

Change-Id: I44a3cea1a5d6e9735b0b74637d7c62db2e9cffa3
Closes-Bug: #1612408
7 years ago
yatin karel fb0057c105 Add i18n translation for Log messages
Added i18n translation to all log messages. Also added
hacking check for this to ensure in future all log
messages are translated.

Change-Id: I77fbb18ffe47b12232125cc1fa7aae97ea9b1214
Closes-Bug: #1603759
7 years ago
Vijendar Komalla bf30b9b4cb Support for async bay operations
Current implementation of magnum bay operations are synchronous
and as a result API requests are blocked until response from HEAT
service is received. With this change bay-create, bay-update and
bay-delete calls will be asynchronous.
Please note that with this change bay-create/bay-update api calls
will return bay uuid instead of bay object and also microversion
1.2 is added for new behavior.

Change-Id: I4ca1f9f386b6417726154c466e7a9104b6e6e5e1
Closes-Bug: #1588425
7 years ago
Spyros Trigazis e6a29fb252 Remove ReplicationController object
Following the removal of service [1], pod [2] and container [3], remove
COE specific object ReplicationController.

This change also removes k8s_conductor.

[1] I4f06bb779caa0ad369a2b96b4714e1bf2db8acc6
[2] I8c2499ccb97aae39d80868ce02fbef292d762c10
[3] I288fa7a9717519b1ae8195820975676d99b4d6d2

Change-Id: Ica100c8d2dfdd7dc709feb1f5cdc5a3f3d6c7318
Partially-Implements: blueprint delete-container-endpoint
Partially-Implements: blueprint bay-drivers
7 years ago
Tom Cammann 40aa6550f1 Remove container object
Following on from removing the k8s specific APIs in
I1f6f04a35dfbb39f217487fea104ded035b75569 the objects associated with
these APIs need removal.

Remove the container object, drop the db table and remove references to
the container object. The docker_conductor has also been removed as this
was used for managing containers using Magnum objects.

Change-Id: I288fa7a9717519b1ae8195820975676d99b4d6d2
Partially-Implements: blueprint delete-container-endpoint
Co-Authored-By: Spyros Trigazis <>
7 years ago
PanFengyun 828aff671d Fix some simple mistake
1.The configure is different to the output in install guide.
2.k8s_api has list_namespaced_pod() rather than list_namespaced_pods().
3.Before division, we should check whether the divisor is zero.

Change-Id: I9ca676eac4e3dcba4dcc5553bb5502896d73960e
7 years ago
Bin-Lu 4427374c06 Fix the permission of these files -rwxr-xr-x
As shown above,these files have executable attribute that need be removed.

Change-Id: I1955360afe17ec83c7b759d44a5bf60d9b055594
7 years ago