2175 Commits

Author SHA1 Message Date
Mingyuan Qi
3997132003 Add test class for kube_app AppOperator
Add test class and setup method.
Mock method/operator will be added in individual commits which use
that method or operator.

Story: 2007082
Task: 37998

Change-Id: I60a4b235417f288adb4a13577b9779d79169a233
Signed-off-by: Mingyuan Qi <mingyuan.qi@intel.com>
2020-01-15 10:56:31 +08:00
Zuul
f768a3f15d Merge "Revert "configure kubernetes to use containerd as CRI"" 2020-01-14 21:27:25 +00:00
Don Penney
be3208886a Revert "configure kubernetes to use containerd as CRI"
This reverts commit c229d8ac7e893ecce406cb15c8d1e35988e101e2.

Reverting due to https://bugs.launchpad.net/starlingx/+bug/1859686

Change-Id: I6534fb83c202827aaf1365d92ea6acbf4e531d84
2020-01-14 20:37:52 +00:00
Zuul
d120dec156 Merge "Add sample unit test for helm data generation" 2020-01-14 20:08:56 +00:00
Zuul
c1a9836abb Merge "Adding pylint checking to the sysinv unit tests" 2020-01-14 18:51:33 +00:00
Zuul
f7b90b61ad Merge "Monitor the datanetwork for non-OpenStack work node" 2020-01-14 14:35:20 +00:00
Al Bailey
6258c07dde Adding pylint checking to the sysinv unit tests
pylint checks were excluding the unit tests folder.
It is now included.

The following pylint errors needed to be fixed in the unit tests
 - wildcard import (tests.db)
 - duplicate-key (fornodeid was listed twice in one test structure)
 - function-redefined (test_storage_tier.py)
 - dangerous-default-value (using a list as a default)
 - no-member (rename a class to end in Mixin to avoid these checks)

Change-Id: I1095d54542e1e7b3d07865e1e7b4cd46f14dc407
Story: 2007082
Task: 38081
Signed-off-by: Al Bailey <Al.Bailey@windriver.com>
2020-01-14 08:32:12 -06:00
Zuul
7867465026 Merge "stx-monitor app: enhanced missing elastic label error message" 2020-01-14 13:24:29 +00:00
Zuul
e207e7ac9f Merge "Additional condition is comparison when doing config check out of date" 2020-01-13 21:27:09 +00:00
Zuul
7e6f6b3318 Merge "configure kubernetes to use containerd as CRI" 2020-01-13 19:22:23 +00:00
abdelfattah saafan
745aec251e stx-monitor app: enhanced missing elastic label error message
When attempting to apply stx-monitor app without the required elastic
labels, user will get an error message detailing what is missing.
This label was enhanced to be more readable and provide instructions
on how to apply the missing labels.

Moved utility methods up to the super class LabelTestCase. Also
moved host definitions to each of the test classes since
LabelAssignTestCase require hosts to be locked and
StxAppLabelsTest require hosts to be unlocked.

Added the following enhanced logic:
1. All controller nodes are required to have all 4 elastic labels
2. At least one worker node should have elastic-master label.
3. Add label value validation and print error message for invalid
values.

Change-Id: I0b2bc7b87d2e8bf78bfbba455d499c52e1a72349
Closes-Bug: 1856763
Signed-off-by: abdelfattah saafan <abdelfattah.saafan@windriver.com>
2020-01-13 13:01:33 -06:00
Sun Austin
238b196d3a Additional condition is comparison when doing config check out of date
This will fix below scenario:
Controller-0 is installed and some configs were applied. Config
f5d83753-c485-4221-b4aa-e986f5421d6d is the target config and requests
boot.
After controller-0 is unlocked and reboot.another request with uuid
29c89b7e-51fe-40cb-8700-e22c83ed5fa9 is coming,then the config_target is
set to a9c89b7e-51fe-40cb-8700-e22c83ed5fa9;
Once 29c89b7e-51fe-40cb-8700-e22c83ed5fa9 was applied,the config should
be cleared, but because 29c89b7e-51fe-40cb-8700-e22c83ed5fa9 is not same
as a9c89b7e-51fe-40cb-8700-e22c83ed5fa9,the alarm can not be cleared.

In this fix, tracking list for reboot config and clear it once config is
applied. the out of date config will be clear if reboot config tracking
is empty and config_applied w/ reboot config flag is equal config_target

Closes-Bug: 1851874
Change-Id: Iabeab338bc3fb4615cefcff9e4ae9402e4216321
Signed-off-by: Sun Austin <austin.sun@intel.com>
2020-01-11 16:21:21 +08:00
Bart Wensley
af308b40d5 Add sample unit test for helm data generation
Adding unit test for nova_api_proxy helm data generation.

Change-Id: Icd62ad584e2f3bb0332455ab8458cd8fa585ae09
Story: 2007082
Task: 37994
Signed-off-by: Bart Wensley <barton.wensley@windriver.com>
2020-01-10 08:07:22 -06:00
Al Bailey
89e2975eb8 Add missing requirements for sysinv
The sysinv code has imports for python components but does
not have all of those specified as rpm dependencies.

Adding:
 python-jsonpatch (used by several sysinv/api/controllers/v1/*)
 python-keystoneauth1 (used by sysinv/common/fm)
 python-keystonemiddleware (used by sysinv/api/acl)
 python-oslo-serialization (used by sysinv/conductor/manager)
 python-oslo-service (used by sysinv/common/wsgi_service)
 python-paste (used by api config file)
 python-psutil (used by sysinv/api/controllers/v1/host)
 python-requests (used by sysinv/common/ceph)
 python-retrying (used by sysinv/agent/manager)
 python-stevedore (used by sysinv/puppet)
 python-webob (used by sysinv/api/hooks)

This is a protective commit, since this only works because other
packages are pulling in the dependencies, and those packages can be
updated or removed in the future, thus breaking sysinv.

Change-Id: I9457e3817ae8d219ef0c582b0b0177510a8ec4b0
Story: 2004515
Task: 37985
Signed-off-by: Al Bailey <Al.Bailey@windriver.com>
2020-01-09 12:33:29 -06:00
Zuul
9ca199621c Merge "Deprecate sysinv.openstack.common.lockutils file" 2020-01-08 22:31:59 +00:00
Kevin Smith
a6b821acad Elastic: lower elasticsearch curator limits.
Lower the elasticsearch curator limits to take into account the
elasticsearch cluster.routing.allocation.disk settings and increase
the cronjob frequency to run every half hour.

Change-Id: I7eee7d6ad8a1eb2210a4a9c10c8ff8782783c192
Closes-Bug: 1858436
Signed-off-by: Kevin Smith <kevin.smith@windriver.com>
2020-01-08 14:44:45 -05:00
Al Bailey
b1d0e2caf3 Deprecate sysinv.openstack.common.lockutils file
Removing sysinv.openstack.common.lockutils
The newer lockutils file is located in oslo_concurrency.
Only unit tests were using the older lockutils.

The following pylint error codes are no longer suppressed:
  E0611: no-name-in-module
  E1124: redundant-keyword-arg

Change-Id: I060523571e94857beb7a65cef72bbb367aa1bbc5
Story: 2006796
Task: 37962
Signed-off-by: Al Bailey <Al.Bailey@windriver.com>
2020-01-08 09:17:03 -06:00
Zuul
b52a3cac2e Merge "Prevent provisioning sriov without mgmt iface" 2020-01-07 17:03:04 +00:00
Zuul
9f84b37d62 Merge "Add kubernetes health checks" 2020-01-06 18:06:58 +00:00
Zuul
ee93284575 Merge "Skip generating hieradata for uninventoried host" 2020-01-06 15:46:22 +00:00
Zuul
2e2cf30e3e Merge "Remove sqlalchemy-migrate 0.7.3 patching" 2020-01-06 15:10:12 +00:00
Zuul
be1658ba5a Merge "Remove glance storage backend" 2020-01-06 14:52:07 +00:00
Al Bailey
bbbcb55d12 Remove sqlalchemy-migrate 0.7.3 patching
This is similar to upstream submissions in 2013 such as
https://review.opendev.org/#/c/56705/

We currently build, test and ship with a newer version
of sqlalchemy-migrate 0.11.0 so we do not need to handle
0.7.3 workarounds and patches.

This review also fixes: tests.db.sqlalchemy.test_migrations
which would fail when tested in isolation, due to its environment
not being initialized properly, but working if certain other tests
were run first.

Some skipped tests are now also enabled.

Change-Id: I6ce73491f7a2471f25c7691dab8b53dc354619a2
Story: 2004515
Task: 37954
Signed-off-by: Al Bailey <Al.Bailey@windriver.com>
2020-01-03 09:39:20 -06:00
Zuul
16912dc68d Merge "Removing mox from sysinv in StarlingX" 2020-01-02 21:34:28 +00:00
marvin
c337d55590 Monitor the datanetwork for non-OpenStack work node
Update the lmon to support datanetwork interface monitoring
and use collectd to control the alarm information. Now lmon
will obtain the list of interfaces from /etc/lmon/lmon.conf
which can be generated by puppet.

Change-Id: Id886c935d54158eca44d6685f1e282c20367b9a4
Story: #2002948
Task: #37326
Depends-on: https://review.opendev.org/#/c/694927
Signed-off-by: marvin <weifei.yu@intel.com>
2019-12-26 04:02:46 +00:00
Joseph Richard
063e398afe Skip generating hieradata for uninventoried host
When _config_apply_runtime_manifest is run with force=True, it will
generate hieradata for all hosts, even hosts that have yet to come
online and are still uninventoried.  Because the node is still
uninventoried, the hieradata will have an incomplete network config
with only the loopback interface configured.
When those hosts come online for the first time, they will mount
/opt/platform and check for their hieradata, and because it exists,
apply their puppet manifests with that hieradata, which will bring down
all non-loopback interfaces and make the node go offline and become
unreachable.
This commit adds a check to only generate hieradata for inventoried
hosts, even when force=True.

Closes-bug: 1856617
Change-Id: I37c611e9a1047d5651844c71351f40d939977b6f
Signed-off-by: Joseph Richard <joseph.richard@windriver.com>
2019-12-20 14:16:57 -05:00
Joseph Richard
a9686eccf9 Prevent provisioning sriov without mgmt iface
This commit adds a semantic check to prevent provisioning an sriov
interface without a mgmt interface configured.  This is necessary to
prevent an invalid network config being generated and applied, which
causes the node to lose connectivity over mgmt and become unreachable.

Closes-bug: 1853329
Change-Id: I783447a2214a1b0f4d698ac20037f8f1e8083958
Signed-off-by: Joseph Richard <joseph.richard@windriver.com>
2019-12-20 13:38:35 -05:00
Bart Wensley
b5c78afe28 Add kubernetes health checks
Adding kubernetes health checks when a kubernetes upgrade is
started to ensure the kubernetes cluster is healthy. Also
adding a new "system health-query-kube-upgrade" command to
do the health checks without starting a kubernetes upgrade.

Change-Id: Ib3a491e25a6cb523cfd9f08e74879e5c0a25d4f7
Story: 2006781
Task: 37583
Depends-On: https://review.opendev.org/#/c/700197/
Signed-off-by: Bart Wensley <barton.wensley@windriver.com>
2019-12-20 10:50:33 -06:00
Zuul
d09c66a421 Merge "Import subprocess from eventlet.green package" 2019-12-20 14:37:08 +00:00
Al Bailey
4501e6ee86 Removing mox from sysinv in StarlingX
Mox is only being used by unit tests.

There is an openstack governance directive to remove mox:
    https://storyboard.openstack.org/#!/story/2001546

Many unit tests were able to be updated based on
similar work in ironic. Ex:
    https://review.opendev.org/#/c/50729/

Removed two oslo incubator files under sysinv.openstack.common.fixture:
    mockpatch (unused)
    moxstubout

Change-Id: Ie90a9c9e5a3028575d016f6eccf0ad118495d482
Story: 2006796
Task: 37851
Signed-off-by: Al Bailey <Al.Bailey@windriver.com>
2019-12-20 08:04:22 -06:00
Zuul
5914adea65 Merge "Override subcloud dcdbsync admin endpoint for DC access" 2019-12-19 22:02:02 +00:00
Bart Wensley
24e3cbe321 Import subprocess from eventlet.green package
The python kubernetes client requires a newer version of eventlet.
However, newer versions of eventlet have an issue with the subprocess
module, which requires subprocess to be imported from eventlet.green
instead of being imported directly. See:
https://github.com/eventlet/eventlet/issues/413

To allow eventlet to be upversioned (from 0.18.4 to 0.24.1), the
subprocess imports are being changed for code that runs with
eventlet.

Change-Id: Idf221c5c1fa542fbe2d9231dff441ec9289a93f1
Story: 2006781
Task: 37583
Signed-off-by: Bart Wensley <barton.wensley@windriver.com>
2019-12-19 15:29:42 -06:00
Zuul
6b732d2197 Merge "Update and expose openstack keystone admin endpoint" 2019-12-19 21:24:54 +00:00
Zuul
21ce64d769 Merge "Move incorrectly located stand-alone scripts" 2019-12-19 18:57:40 +00:00
Zuul
1623310564 Merge "Remove eventlet_backdoor from sysinv" 2019-12-19 18:27:58 +00:00
Bart Wensley
5321e8f3a9 Move incorrectly located stand-alone scripts
Moving incorrectly located stand-alone scripts from the
sysinv/cmd directory to the scripts directory. These scripts
appeared to be using eventlet due to the __init__.py file in the
sysinv/cmd directory, but were not, as they are moved to the
/bin directory at installation time.

This is being done in preparation for upversioning eventlet.

Change-Id: I511d61e14b3eb1511cee03aa0e36ffd1277e79ab
Story: 2006781
Task: 37583
Signed-off-by: Bart Wensley <barton.wensley@windriver.com>
2019-12-19 12:01:50 -06:00
Al Bailey
3044cbcd73 Remove eventlet_backdoor from sysinv
The eventlet_backdoor was part of oslo incubator and
is now located in oslo_service.

There is no code in StarlingX that uses eventlet_backdoor.

Change-Id: I7ed15c7ad5261dac07c4c786ae14eddb147ff71d
Story: 2006796
Task: 37826
Signed-off-by: Al Bailey <Al.Bailey@windriver.com>
2019-12-19 09:51:41 -06:00
Bart Wensley
b14f933c75 Revert "Import subprocess from eventlet.green package"
This reverts commit 45d62bf02e309b7c5e3356ff39a28b749fbcc633.
This commit is causing issues with partition management due
to the manage-partitions command not using eventlet. It could
also be causing issues with the query_pci_id command for the
same reason.

Change-Id: I6088b14d0c860c44b363e2e78903744f88308c6b
2019-12-19 14:45:50 +00:00
Bart Wensley
45d62bf02e Import subprocess from eventlet.green package
The python kubernetes client requires a newer version of eventlet.
However, newer versions of eventlet have an issue with the subprocess
module, which requires subprocess to be imported from eventlet.green
instead of being imported directly. See:
https://github.com/eventlet/eventlet/issues/413

To allow eventlet to be upversioned (from 0.18.4 to 0.24.1), the
subprocess imports are being changed.

Change-Id: I884e8e539864f26b3593e9c15a29bbd099c94453
Story: 2006781
Task: 37583
Signed-off-by: Bart Wensley <barton.wensley@windriver.com>
2019-12-18 18:55:56 +00:00
Saul Wold
335b755b39 Zuul: remove stx-build-rpms template
Since OBS is not working correctly for CentOS and we are stopping openSUSE
for now, no point it having this template run those OBS jobs.

Change-Id: I75310fbeff26bd4d6f022750e3921da3ec5c82cf
Signed-off-by: Saul Wold <sgw@linux.intel.com>
2019-12-18 06:52:31 -08:00
Zuul
f33fbf1c82 Merge "Increase size of service-parameter field value" 2019-12-17 20:07:17 +00:00
Zuul
7be53a6490 Merge "Add k8s cert rotation script to sysinv pkg" 2019-12-17 18:47:26 +00:00
Zuul
cc87b1638a Merge "Use oslo utils version of uuidutils." 2019-12-17 18:24:36 +00:00
Zuul
555be95dbd Merge "Hugepage percent allocation." 2019-12-17 17:56:39 +00:00
Kristine Bujold
1a220c2e7d Increase size of service-parameter field value
This submit increases the size of the “value” field for the
service-parameter table from 255 to 4096 characters. This is
to accommodate registries that can be saved as a list in
this field.

This change was tested in lab with a size of 4096 and 4097
for no_proxy's value.

Closes-Bug: 1856236
Change-Id: I5067f7ee1e5f8b532045d0736402080bad39be72
Signed-off-by: Kristine Bujold <kristine.bujold@windriver.com>
2019-12-17 12:53:21 -05:00
Al Bailey
591de6e402 Use oslo utils version of uuidutils.
The old oslo incubator uuidutils is being removed and the
version in oslo_utils is being used instead.

Change-Id: Ie57c55974ec34e04d4604f84e3e882cc000bb92b
Story: 2006796
Task: 37425
Signed-off-by: Al Bailey <Al.Bailey@windriver.com>
2019-12-17 08:30:37 -06:00
Andrew Troake
766de0c034 Hugepage percent allocation.
Users have the option to allocate hugepage memory by # of
pages or percent of memory available.
Percent allocation is allocated on boot.

Story: 2006755
Task: 37245

CLI:
Allocating by percentage:
system host-memory-modify -2M 50% controller-0 0
Allocating by # of pages
system host-memory-modify -2M 50 controller-0 0

Testcases:
Allocating hugepage memory by percentage
Allocating hugepage memory by number of pages
Correct percentage is allocated if available memory
    changes
100% of hugepage memory allocates correctly
Check error is raised if user attempts to allocate
    percentage not in range of 0%-100%
Confirm that Horizon GUI can set as both percent and
    integer
Confirm that hugepage changes are made before unlock
    and after unlock

Change-Id: I6f1165c22e409e261c0b065f7d6cf5407647b56e
Signed-off-by: Andrew Troake <andrew.troake@windriver.com>
2019-12-16 15:16:56 -05:00
Zuul
fbfb03fbf9 Merge "Deprecate the sysinv.openstack.common utils files" 2019-12-16 19:53:19 +00:00
Zuul
5a86cb5b1f Merge "Update list of skipped bandit plugins." 2019-12-16 19:51:20 +00:00
Zuul
84f1aa0655 Merge "Convert from testrepository to stestr" 2019-12-16 19:38:27 +00:00