63 Commits

Author SHA1 Message Date
Zuul
251ed8f219 Merge "Change stx-ceph-manager base image" 2024-08-05 15:02:58 +00:00
Erickson Silva de Oliveira
f21f6ef838 Change stx-ceph-manager base image
The stx-ceph-manager image, which is used for rook-ceph,
was with ceph v14.2.2, which is the bare-metal version.
However, rook-ceph uses ceph v18.2.2.

Therefore, to maintain version compatibility, the base image
was updated to docker.io/openstackhelm/ceph-config-helper:
ubuntu_jammy_18.2.2-1-20240312 and all necessary adjustments
were made to the Dockerfile so that all STX packages could be
installed successfully.

Test Plan:
- PASS: Build stx-ceph-manager image
- PASS: Change the stx-ceph-manager deployment in
	the rook-ceph app to use this image
- PASS: Access the stx-ceph-manager pod and check
	the ceph version
- PASS: Check the pod logs to verify if there
	are any errors

Story: 2011066
Task: 50702

Depends-On: https://review.opendev.org/c/starlingx/utilities/+/924883
Depends-On: https://review.opendev.org/c/starlingx/app-rook-ceph/+/924884

Change-Id: I30371e7ed586f5b956668517bc1f9e5ed6b5cdce
Signed-off-by: Erickson Silva de Oliveira <Erickson.SilvadeOliveira@windriver.com>
2024-08-01 23:08:33 -03:00
Erickson Silva de Oliveira
8905b49eaf Get certificate with mgr name instead of hostname
In bare-metal ceph, mgr's were created with the hostname,
however with rook-ceph they are created with letters.
Therefore, it is not possible to get the certificate
from restful API module because the hostname is passed
instead of the mgr name.

To fix this, the "ceph mgr metadata" command has been
replaced with "ceph mgr dump", which returns the name
of the active mgr.

Test Plan:
- PASS: Build stx-ceph-manager image
- PASS: Change the stx-ceph-manager deployment in
	the rook-ceph app to use this image
- PASS: Build rook-ceph app
- PASS: Apply app on Standard (IPV6)
- PASS: Check the pod's logs to verify that
	the certificate was obtained
- PASS: Using bare-metal ceph, check the logs in
        /var/log/ceph-manager.log for any errors.

Story: 2011066
Task: 50656

Change-Id: I4f57e1d0cc45143c1ea29c826e45bae59e27efdc
Signed-off-by: Erickson Silva de Oliveira <Erickson.SilvadeOliveira@windriver.com>
2024-07-30 18:36:54 -03:00
Scott Little
afab49269d Remove CentOS/OpenSUSE build support
StarlingX stopped supporting CentOS builds in the after release 7.0.
This update will strip CentOS from our code base.  It will also remove
references to the failed OpenSUSE feature as well.

Story: 2011110
Task: 49964
Change-Id: I99d3b6d6de95434a7040f5a1967db9a04191a25f
Signed-off-by: Scott Little <scott.little@windriver.com>
2024-04-26 14:19:03 -04:00
Erickson Silva
1d6755c0eb Add missing dependencies in stx-ceph-manager debian image
stx-ceph-manager images are actually being generated for both
CentOS and Debian images. When comparing both images, we can
notice that Debian images are missing the following dependencies:
python-cephclient and ceph-manager. Therefore, rook-ceph cannot
be applied.

To resolve this, dependencies were added to the Dockerfile of
the stx-ceph-manager image.

Test Plan:
 PASS: Build stx-ceph-manager image
 PASS: Push image to local registry
 PASS: Apply rook-ceph-apps
 PASS: Check if ceph is healthy

Closes-Bug: 2039564

Change-Id: Ie03cbead00f26bf972c7e4796d9f8472b58b91c9
Signed-off-by: Erickson Silva <Erickson.SilvadeOliveira@windriver.com>
2023-10-17 11:43:50 +00:00
David Bastos
1f6d1236ce Update debian package versions to use git commits
The Debian packaging has been changed to reflect all the
git commits under the directory, and not just the commits
to the metadata folder.

Package builder concatenates the number of revisions added since
the one defined in the BASE_SRCREV to the name of the .deb file
generated. Thus, any new submission to the code in these directories
increments this revision counter in the filename.

Test Plan:
PASS: Created new code submissions to package python-cephclient
and verify that the version in the package name was incremented.
PASS: Created new code submissions to package ceph-manager and
verify that the version in the package name was incremented.

Story: 2010550
Task: 47510

Signed-by: David Bastos <david.barbosabastos@windriver.com>
Change-Id: I365f01789bb129f869ce52a3372bb99bbc5ab548
2023-03-09 13:44:09 +00:00
Zuul
509afb27d4 Merge "Ceph-manager: Change timeout usage for ceph commands" 2023-01-18 19:24:22 +00:00
Felipe Sanches Zanoni
21236a3d4b Ceph-manager: Change timeout usage for ceph commands
Ceph-manager is using cephclient to get information from ceph and
the command 'ceph restful list-keys --connect-timeout 15' is
timing out and crashing, generating core dumps.

Removed the '-connect-timeout' parameter and added 'timeout'
argument in check_output function.
This way no core dumps were seen.

Test Plan:
  PASS: Fresh install AIO-SX, stop ceph services, watch ceph-manager.log
        file looking for timed out commands and verify if there is any
        core dump from ceph.

Closes-bug: 1999985

Signed-off-by: Felipe Sanches Zanoni <Felipe.SanchesZanoni@windriver.com>
Change-Id: Ie119cacd2409de07ed700fd554df03d4187a252d
Signed-off-by: Erickson Silva de Oliveira <Erickson.SilvadeOliveira@windriver.com>
2023-01-18 18:50:45 +00:00
Al Bailey
93318327fc Update tox.ini to work with tox 4
This change will allow this repo to pass zuul now
that this has merged:
https://review.opendev.org/c/zuul/zuul-jobs/+/866943

Tox 4 deprecated whitelist_externals.
Replace whitelist_externals with allowlist_externals

Partial-Bug: #2000399

Signed-off-by: Al Bailey <al.bailey@windriver.com>
Change-Id: Ic19cac494dd1d1d256d8f6c2113f9ad6a36944cc
2022-12-27 01:31:42 +00:00
Zuul
47d8b341a8 Merge "Adding retry when getting ceph services in cephmgr" 2022-12-07 20:04:59 +00:00
Gustavo Ornaghi Antunes
ba8889bd79 Adding retry when getting ceph services in cephmgr
After bootstrap system, it takes a while to provide ceph
services available. This can cause problems when updating
via nfv for example.
To fix this, added some attempts to get the service URL.
Now, the system is checking and waiting for the ceph services url
to initialization continue.

Test Plan:
PASS: Check if the message "Service url successfully retrieved"
      is displayed after "Could not get service url" in the
      ceph-manager logs.

Closes-Bug: 1997758

Signed-off-by: Gustavo Ornaghi Antunes <gustavo.ornaghiantunes@windriver.com>
Change-Id: I7d47d9c88da1afd909eb65ce510c0a562e6912b6
2022-12-05 14:42:00 +00:00
Zuul
465ae995db Merge "Add stx-ceph-manager docker image" 2022-11-18 13:30:26 +00:00
Zuul
a65a04d19d Merge "Use hostname on restful service certificate" 2022-11-15 20:36:48 +00:00
Luiz Felipe Kina
681af65049 Add stx-ceph-manager docker image
This creates the debian docker image file of stx-ceph-manager.

Test Plan:
Debian:
PASS: build-pkgs, build-image, deploy. Unlocked, enabled, available
and platform-integ-apps applied. Inspected ceph-manager.log for
correct application

Story: 2009831
Task: 46734

Signed-off-by: Luiz Felipe Kina <LuizFelipe.EiskeKina@windriver.com>
Change-Id: I8179901ce2030236b0a4d5d207f9fdba01271be0
2022-11-07 16:13:22 -05:00
Daian Cardoso Sganderlla
2617f00165 Use hostname on restful service certificate
Force the use of hostname on getting the certificates
for restful service, instead of the IP address.
This is necessary due to the update of the Ceph version
in rook-ceph app. New Ceph version started to store the IP
number instead of the hostname in restful service infos.
We need a new function to return the hostname in order to
get the key and certificate files.

Story: 2009138
Task: 46735

Test Plan:
PASS: AIO-SX - created an image with this version of code
and use it to get service certificate

Signed-off-by: Daian Cardoso Sganderlla <Daian.CardosoSganderlla@windriver.com>
Change-Id: I06ac2b40662ef1df4b86d82d2cdf298d746d886b
2022-11-04 13:50:12 +00:00
Alyson Deives Pereira
190e824490 Remove RPC server from ceph-manager
These RPC methods were called by sysinv. However, due to changes in [1]
and [2],these rpc methods have no effect anymore:
- get_primary_tier_size: always returns 0.
- get_tiers_size: always returns {}.
- is_cluster_up: this rpc is not called by sysinv.

Moreover, as mentioned in [2], there is still parts of code in sysinv
that are marked with "TODO(CephPoolsDecouple)" which is not in use
anymore and can be removed, which include methods that perform calls
to ceph-manager rpc methods.

Therefore, sysinv does not need to call these rpc methods anymore and
the rpc server from ceph-manager can be removed.
A change proposed in [3] replaces these rpc calls with their respective
current return values.

Story: 2010087
Task: 46520

Test Plan:
PASS: AIO-SX: manually replaced these and [3] files into a Debian
installation and no crashes happened.
PASS: rebuild the whole system with this and [3] modifications with no
crashes.
PASS: List rabbitmq queues and confirm that sysinv.ceph-manager queues
are not listed
PASS: Configure a ceph storage backend with no errors
PASS: Check no error is found in ceph-manager.log and sysinv.log
PASS: Confirm no alarm is raised


[1] https://review.opendev.org/c/starlingx/utilities/+/820933
[2] https://review.opendev.org/c/starlingx/config/+/620448
[3] https://review.opendev.org/c/starlingx/config/+/860690

Signed-off-by: Alyson Deives Pereira <alyson.deivespereira@windriver.com>
Change-Id: I30bedf6c46519e0dab89cffed821063b0ff67bd6
2022-10-27 19:27:59 +00:00
Isac Souza
958237c229 Refactor ceph-manager code to use sysinv-api instead of RPC
The monitor module of ceph-manager has support for executing
certain actions during an upgrade. This code is intended
to support alarm filtering or other actions necessary to
migrate ceph-related code from one version to another.

The code currently uses an RPC call to conductor in order
to fetch upgrade information from sysinv. This patch changes
ceph-manager to use the REST API instead of an RCP. The main
reason for this migration is a planned change on sysinv to
use ZeroMQ for RPC, instead of RabbitMQ. This change on
sysinv would require changes on ceph-manager to use the new
protocol. Using the REST API has other advantages over RPC,
such as decoupling and resilience.

Story: 2010087
Task: 46225

Test Plan:
1. Unit tests for the new code
2. Installed the new code and verified the code works by
   forcing the upgrade check to run (with and without an
   active upgrade)
3. Fresh installation of ISO

Signed-off-by: Isac Souza <IsacSacchi.Souza@windriver.com>
Change-Id: Ifd763825cd184192f100a119db6c3aee92708706
2022-10-07 20:39:58 +00:00
Thiago Brito
c213a3832d Add flake8-import-order and use python3.9 on tox
Improving the code quality of utilities by adding the flake8 check on
zuul and adding the flake8-import-order plugin to standardize imports.
Also, defaults testenv to python3.9 configuration that should be used
for now on with the debian migration.

Story: 2010100
Task: 45671
Signed-off-by: Thiago Brito <thiago.brito@windriver.com>
Change-Id: I007302bdfb873aed4ebb7b9023db2461282b9fba
2022-09-13 21:49:41 +00:00
Dan Voiculeasa
c5ccb52f4e Update ceph-manager rabbit configuration
This work is part of Debian integration effort.

It is observed in the logs that ceph-manager cannot connect
to rabbitmq.
Oslo.messaging 5 is used on CentOS and 12 on Debian.
We see that the interface for reading configuration from files and
passing it to messaging was changed:
b0d3bfceb8

Ceph-manager uses the configuration from sysinv.conf. Ceph-manager
code is common between CentOS and Debian. Oslo.messaging module
doesn't provide a version variable, thus instead of spliting the
logic I used a common interface for configuring the rabbit driver.

Tests on AIO-SX.
CentOS:
PASS: build-pkgs, build-image, deploy
PASS: unlocked enabled available and platform-integ-apps applied
PASS: inspected ceph-manager.log for patched/non-patched, no difference
was observed.
Debian:
- live patch controller, ceph-manager.log no longer shows connection
refused errors
- logs look ok

Story: 2009101
Task: 44787
Signed-off-by: Dan Voiculeasa <dan.voiculeasa@windriver.com>
Change-Id: Ie2e7b6be4a54bca45f5ee8e9b952632310302485
2022-03-17 10:30:45 +02:00
Vinicius Lopes da Silva
dce0552541 Clean ceph quota polls
STX does not support managed pool quotas. This is legacy code left over
from the project seed code that was never properly cleaned up. Removing
as this pollutes the logs with false positive errors.

Test plan:
    PASS: Verify that no alarms are raised related to quotas.
    PASS: Confirm that no quota messages are present in the logs.

Partial-Bug: 1959894

Signed-off-by: Daian Cardoso Sganderlla <Daian.CardosoSganderlla@windriver.com>

Change-Id: Iff3b9df68766f2c05dd16d2b8e20962a10e0a7b0
2022-02-07 17:42:09 +00:00
Zuul
bb56982ee1 Merge "Add debian package for ceph/ceph-manager" 2022-01-25 17:05:37 +00:00
Ramon Gazoni Lacerda
52eeba8c32 Add debian package for ceph/ceph-manager
Add debian packaging infrastructure for ceph/ceph-manager to build
a debian package.

Test Plan:
PASS: Built package
PASS: Built iso
PASS: Installed controller

Story: 2009101
Task: 44189

Signed-off-by: Ramon Gazoni Lacerda <Ramon.GazoniLacerda@windriver.com>
Change-Id: Id4f9b05c8fefb6e7538fe4011c67d3cdab6af21b
2022-01-24 09:39:38 -03:00
Vinicius Lopes da Silva
41e65d98f1 Increase timeout value for ceph CLI
When a monitor election is in process and ceph CLI commands are issued,
they might hang until the end of the election occurs. Sometimes the
election is quick and sometimes it might take up until 11s.
Since current code awaits up until 5s, sometimes an error appears in
STX due to timeout.

The timeout value needs to increase, I believe 15s of timeout is enough
for most cases.

Testing performed:
2+2:
Set up two scripts making requests to ceph. One had a small timeout
and another one a big timeout.
Started both scripts.
Locked storage-0.
Script having small timeout showed error and the other one didn't.

Story: 2009074
Task: 44159

Signed-off-by: Vinicius Lopes da Silva <vinicius.lopesdasilva@windriver.com>
Change-Id: I618ed74c8f63ca77caa0893016f90b67fdfee7e0
2021-12-09 19:18:58 -03:00
Vinicius Lopes da Silva
be55832311 Fix code to support new json format when calling osd crush tree
When doing host-lock in storage nodes when ceph cluster is in warning
state, it verifies the state of the storage node. Doing this involves
calling 'pg dump_stuck' and 'osd crush tree' but their json format has
changed and current code doesn't support this format.

The fix was made to support the new format.

Testing performed:
STANDARD+STORAGE: Set ceph-cluster in warning state and successfully
locked storage-1. To put the cluster in warning state I created a pool
having size 3 and set nosizechange to true. This allows pgs_list to not
be empty and test the loop iteration (which calls 'osd crush tree').

Story: 2009074
Task: 44128

Signed-off-by: Vinicius Lopes da Silva <vinicius.lopesdasilva@windriver.com>
Change-Id: I1b0f76e89222e2de0f95858955ef07ba479a6822
2021-12-06 11:36:34 -03:00
Vinicius Lopes da Silva
65f223f0d4 Add body argument to enable_msgr2 call
The _request method is always deleting 'body' parameter from kwargs.
In case a given parameter do not have a body, it will raise an
exception.
The fix adds a 'body' parameter to enable_msgr2 so it can be deleted by
_request method.

This problem was found during upgrade of AIO-SX after method
enable_msgr2 was created. The test was made only for AIO-SX but this
solves the problem in all configs.

Testing performed:
 - Upgrade of AIO-SX

Story: 2009074
Task: 43989

Signed-off-by: Vinicius Lopes da Silva <vinicius.lopesdasilva@windriver.com>
Change-Id: I4d3949a29ad9800508a6833701eff927f8c1e76d
2021-11-16 14:15:27 -03:00
Vinicius Lopes da Silva
d58e4bc824 Method to enable msgr2
This method will enable msgr2 (v2 protocol) in ceph cluster.
This is required starting in Nautilus release.

Testing performed:
- Upgrade of 2+2+2
- Upgrade of AIO-DX
- Upgrade of AIO-SX
- Fresh install of AIO-SX
- Fresh install of AIO-DX
- Fresh install of 2+2

Story: 2009074
Task: 43943

Signed-off-by: Vinicius Lopes da Silva
<vinicius.lopesdasilva@windriver.com>
Change-Id: I9ff05f95410d4d03e3a200efc69685a9b3d419fb
2021-11-14 18:16:06 -03:00
Zuul
5900b5d71b Merge "Re-enable important py3k checks for utilities" 2021-11-09 15:28:32 +00:00
Bernardo Decco
5862f6a009 Re-enable important py3k checks for utilities
Re-enabling some of the disabled tox warnings present on
the pylint.rc file
Re-enabling:

W1619: old-division
W1636: map-builtin-not-iterating
W1638: range-builtin-not-iterating
W1645: exception-message-attribute

Note: W1645 was suppressed inline since the self refersto the class
attribute and not the Exception.message from BaseException PEP 352

Test Plan: Sanity test run on AIO-SX:

PASS: test_system_health_pre_session[pods]
PASS: test_system_health_pre_session[alarms]
PASS: test_system_health_pre_session[system_apps]
PASS: test_wr_analytics[deploy_and_remove]
PASS: test_horizon_host_inventory_display
PASS: test_lock_unlock_host[controller]
PASS: test_pod_to_pod_connection
PASS: test_pod_to_service_connection
PASS: test_host_to_service_connection

Story: 2006796
Task: 43376
Signed-off-by: Bernardo Decco <bernardo.deccodesiqueira@windriver.com>
Change-Id: I23ea193a3e794f28dcf0ea00f9a2aedb17283d1e
2021-10-21 12:34:24 +00:00
Charles Short
ddf814efcf Add debian packaging for python-cephclient
Add debian packaging infrastructure for python-cephclient.

Story: 2009101
Task: 43048

Signed-off-by: Charles Short <charles.short@windriver.com>
Change-Id: I164020eaae198da0ff96b6e9fb05845501869f4f
2021-09-28 09:45:24 -04:00
Daniel Pinto Barros
6bdea932bd Update ceph-manager to recover TLS certificate
Code changes in client.py to recreate the TLS CA certificate
temporary file if it had been unexpectedly removed or not found.
Without this temporary file, ceph-manager can not check health of
ceph by ceph-api and raise Ceph-Down alarm even if the cluster were
operational.

The exception thrown by the request lib is an IOError raised in the
cert_verify method. The _refresh_session is a method that set the
certification file if it has not been set before. But if the
certification file has been set and after it were removed, the method
do not try to reset it. To avoid duplicated code, a flag has been
added to this method to force the recover of the TLS certification.

Closes-Bug: #1940471

Signed-off-by: Daniel Pinto Barros <DanielPinto.Barros@windriver.com>
Change-Id: Icb15337e1e76ae92cd62631190a6739960f6245d
2021-08-19 13:52:55 -04:00
Chen, Haochuan Z
6829ac785f Enable bandit for utilities
Story: 2008929
Task: 42530

Change-Id: Id63488414ae16b384549fcf8cbd9805b5b1a81ad
Signed-off-by: Chen, Haochuan Z <haochuan.z.chen@intel.com>
2021-06-17 01:45:52 +00:00
Chen, Haochuan Z
88a678f142 Add packages to stx-ceph-manager image
This update installs ceph-mgr, ceph-mon, ceph-osd packages as part
of stx-ceph-manager image.

Partial-Bug: 1920882

Change-Id: I4afde8b1476e14453fac8561f1edde7360b8ee96
Signed-off-by: Chen, Haochuan Z <haochuan.z.chen@intel.com>
2021-04-02 15:36:01 +08:00
Zuul
39d8f05bf8 Merge "Build image for ceph-manager" 2021-01-18 15:23:02 +00:00
Don Penney
467851e8d5 Add auto-version for remaining stx/utilities packages
Update remaining StarlingX packages with hardcoded TIS_PATCH_VER to
use PKG_GITREVCOUNT where possible, with offsets as needed to ensure
the version is incremented above the hardcoded version.

Change-Id: Ia1ade77e12b948040f06806a5417d005488e3bb9
Story: 2008455
Task: 41460
Signed-off-by: Don Penney <don.penney@windriver.com>
2020-12-17 13:20:49 -05:00
Martin, Chen
b58d9d20d3 Build image for ceph-manager
For host-based ceph cluster, serivce manager launch service mgr-restful-plugin
which launch ceph-mgr and daemon ceph-manager. Ceph-manager daemon polls
ceph cluster status by ceph-mgr restful module and raise or clean alarm to
to fault manager.

For rook, build a image named stx-ceph-manager, use this image to make a
deployment, which will take the same task polling containerized ceph cluster
status and raise or clean alarm.

Story: 2005527
Task: 41338

Change-Id: Iaaedfc0c7198e102eb4b8c94ab759e9b209e6bfd
Signed-off-by: Martin, Chen <haochuan.z.chen@intel.com>
2020-12-01 14:15:04 +08:00
Dan Voiculeasa
f666e60281 Change ceph manager port
Free port 5001 to be used by keystone.

Story: 2007347
Task: 39393

Depends-On: Id789591bf22931494e970aaf3b12e9e5cbe223fa
Change-Id: I45ee810c9b4686d98c246c3a73f21f0de4ba76a1
Signed-off-by: Dan Voiculeasa <dan.voiculeasa@windriver.com>
2020-04-14 11:01:52 +03:00
Hang Li
c17b773754 Fix the error link for utilities
The link to the webpage in utilities are
error,so redirect them to the correct link.

Change-Id: I51e5c877eed2d12f81be29cf0bbff82d9f5be09b
Closes-Bug: #1852838
Signed-off-by: Hang Li <lihang0166@fiberhome.com>
2019-11-19 20:02:28 +08:00
Stefan Dinescu
d838e22ac6 Don't raise missing ceph peer alarm on simplex
A missing ceph peer alarm was raised on an AIO-SX node.
Since an AIO-SX node only has one node, such an alarm makes
no sense on this kind of setup and it should not be raised.

Note: also added some extra empty lines for improved code
      readability.

Change-Id: Ie6debccad4667e840859eba3d5e369e8932e782a
Closes-bug: 1846287
Signed-off-by: Stefan Dinescu <stefan.dinescu@windriver.com>
2019-10-09 11:11:30 +03:00
Daniel Badea
2e7d2550b9 python-cephclient: fix urlparse function call
urlparse import for python3 adds urlparse function to the
global namespace instead of a module name. Existing code
calling urlparse.urlparse() fails because of that.

Replace urlparse.urlparse() with urlparse().

Change-Id: I61c749b9beb8b5a09e82c8bc93087348b2c20d73
Closes-bug: 1828470
Signed-off-by: Daniel Badea <daniel.badea@windriver.com>
2019-09-17 11:10:22 +00:00
Daniel Badea
613276c944 python-cephclient: fix python 3 tox
Fix python 3 tox error caused urlparse import in commit
9f73cd4f9bf4a8f284642b7940ac41d22b637d3e "python-cephclient:
use configured restful api plugin"

Closes-bug: 1828470
Signed-off-by: Daniel Badea <daniel.badea@windriver.com>
Change-Id: I58c338cec72132aade7a4b4c1e2abeb887b27c86
2019-09-16 19:55:31 +00:00
Zuul
a77b9eec8f Merge "python-cephclient: use configured restful api plugin" 2019-09-16 16:36:41 +00:00
Dan Voiculeasa
e32b790684 ceph-manager: raise alarms when OSD is down even if health OK
If ceph status reports HEALTH_OK then OSD down alarm is not raised.
Same for OSD out.

Example scenario:
The disk might fail which means the osd will be in down state.
Ceph status shows a health warning, for example HEALTH_WARN.
An alarm is raised by ceph-manager.
After some time the disk will be marked out. Ceph health becomes
HEALTH_OK and alarms are cleared.
The user might never replace the disk thus the OSD state is still down
yet no alarm is raised.

Raise alarms even when status is HEALTH_OK to let the user know that
the OSDs are still down or out.

Closes-Bug: 1841903
Change-Id: I4380183ce0cd2e41fbf12d0f9f20a4328293882c
Signed-off-by: Dan Voiculeasa <dan.voiculeasa@windriver.com>
2019-09-10 08:54:46 -04:00
Daniel Badea
9f73cd4f9b python-cephclient: use configured restful api plugin
ceph-mgr restful plugin is using self-signed certificate when providing
HTTPS access to Ceph REST API.

Instead of retrieving and using this certificate python-cephclient is
currently a shortcut and disables verifying HTTPS requests for the
entire requests/urllib3 library. This was meant to be temporary shortcut
until proper handling of ceph-mgr restful plugin HTTPS certificates is
implemented.

This commit implements automatic python-cephclient restful plugin
certificate retrieval such that it is no longer necessary to disable
requests/urllib3 certificates verification.

Two options were available:

1. provide path to certificate file when creating an instance of
   CephClient() or CephWrapper() then use that value when creating
   a request session ('verify' attribute).

   This delegates the responsibility of providing a valid certificate to
   the caller/user of python-cephclient library. Because it implies an
   API update all StarlingX components using python-cephclient need to
   be updated.

   The certificate file itself is created when mgr-restful-plugin
   is started before ceph-mgr restful plugin service is configured
   to use it.

2. add support for retrieving the certificate by using 'ceph' commands
   similar to how user credentials and restful plugin endpoint are
   discovered.

   If there is an error in getting the certificate then the session
   certificate verification is temporarily disabled until the next
   request is made. This means that if the corresponding Ceph config-key
   'mgr/restful/{hostname}/crt' is removed then python-cephclient will
   incur the overhead of running 'ceph config-key get' before each
   request but this is an unlikely scenario in our case.

Option #2 was selected because it doesn't change existing API.

Change-Id: I68acb3e1d2fb8e2bb07c8d67e65b02d55a6716ca
Depends-on: I6e8ca93c7b51546d134a6eb221c282961ba50afa
Closes-bug: 1828470
Signed-off-by: Daniel Badea <daniel.badea@windriver.com>
2019-09-09 06:55:06 +00:00
Daniel Badea
4026b5a23f python-cephclient: populate items list for all nodes except osd
cephclient wrapper is converting a flat list of dictionaries returned
by Ceph Mimic's osd_crush_tree() to nested dictionaries (actual tree)
as expected sysinv. While doing this it looks at the "children"
attribute and if there's none then it skips populating current
node "items".

For storage nodes that don't have any attached OSDs the corresponding
tree entry will not have an "items" attribute. When sysinv tries to
get OSD's by storage it tries to access it and crashes.

Fix by creating empty "items" attribute unless node type is "osd".

Closes-Bug: 1834539
Change-Id: Icc5988407c9773d10d2cd1078e08ae213075f793
Signed-off-by: Daniel Badea <daniel.badea@windriver.com>
2019-07-24 16:41:10 +00:00
Daniel Badea
3f3e61f148 python-cephclient: delete finished requests
ceph-mgr REST API supports synchronous and asynchronous requests.
In asynchronous mode clients can run multiple requests in parallel
then poll to get status of finished requests.

ceph-mgr restful plugin keeps a list of requests that were initiated
by the client and forwarded towards ceph-mgr. It expects the client
to delete finished requests after retrieving current status.

python-cephclient is making synchronous requests (using POST to
"/request?wait=1") but the server is converting them asynchronus
then polls for status on its side. So after getting a response back
the client is still expected to DELETE "/request?id=..."

Currently it's not doing that and ceph-mgr restful plugin is
accumulating a list of all requests ever made by python-cephclient

Change-Id: If8d5c8b27135fde45116e05bb04b655d9574c5ca
Closes-Bug: 1828549
Signed-off-by: Daniel Badea <daniel.badea@windriver.com>
2019-05-10 12:18:47 +00:00
Daniel Badea
2e4832103c ceph-manager: fix tox issues
Fix pep8 issues and remove py27 section because there is no test
defined.

Depends-On: I7c6bff4d8986c1fd75c3c9d353557c5eafcdcde0
Change-Id: I7b534e31868e53ec479c2321d6883604c12aa6d3
Signed-off-by: Daniel Badea <daniel.badea@windriver.com>
2019-04-26 08:54:44 +00:00
Changcheng Liu
a1144eceed ceph-manager: update mgr-restful-plugin endpoint
ceph-rest-api was removed in Ceph Mimic. Update endpoint to remove
 "api/v0.1" path that's not needed by ceph-mgr restful plugin.

Story: 2003605
Task: 28860

Depends-On: I1952a12032bcd08d17786bd817d1e15ce36d5afd
Change-Id: I7c6bff4d8986c1fd75c3c9d353557c5eafcdcde0
Co-Authored-By: Daniel Badea <daniel.badea@windriver.com>
Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>
Signed-off-by: Daniel Badea <daniel.badea@windriver.com>
2019-04-26 08:51:44 +00:00
Daniel Badea
ea0a829ab2 python-cephclient: rewrite to use ceph-mgr restful plugin
Not using any of the existing python-cephclient package.
(it's a new package sharing the same name). Released
under Apache-2.0 license as part of StarlingX.

Additional commits:
- create session on first API request
- add missing osd_remove function
- add timeout when retrieving url and password
- sanitize osd ID input
- osd_crush_tree discards json output

Story: 2003605
Task: 28860

Depends-On: I31fb9aac89c44bbce24939197446caa987d395cb
Change-Id: I1952a12032bcd08d17786bd817d1e15ce36d5afd
Signed-off-by: Daniel Badea <daniel.badea@windriver.com>
2019-04-26 08:51:19 +00:00
Changcheng Liu
55111603e1 ceph: remove unused ceph jewel components
* python-cephclient: remove ceph jewel client
* ceph: remove unused ceph-rest-api service
    ceph-rest-api was removed in Ceph Mimic
* ceph: remove unused osd-wait-status script

Story: 2003605
Task: 28861

Depends-On: Ibfbecf0a8beb38009b9d7192ca9455a841402040
Change-Id: Ia79c2f03054588fe5057107d9b2856ee2e821881
Co-Authored-By: Daniel Badea <daniel.badea@windriver.com>
Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>
Signed-off-by: Daniel Badea <daniel.badea@windriver.com>
2019-04-24 15:20:19 +00:00
Eric Barrett
abd4587682 Enable Flake8 Docstring Errors
Flake8 currently ignores the following errors:
H401: docstring should not start with a space
H404: multi line docstring should start without a leading new line
H405: multi line docstring summary not separated with an empty line
Enable them for more consistent formatting of docstrings

Change-Id: I385e28e9c6eca3c02a3def51ff64b00b7a63a853
Story: 2004515
Task: 30076
Signed-off-by: Eric Barrett <eric.barrett@windriver.com>
2019-04-18 11:50:45 -04:00