Browse Source

Use bundletester for amulet test execution

Switch to using bundletester for execution of functional tests,
leveraging tox to build out test virtualenvs.

Rename amulet tests inline with gate-*, dev-* and dfs-*
naming standards.

Update README to refer to functional testing section of the charm
guide.

Also remove brittle auth conf checks which are failing at
cinder-ceph master. Even without those explicit checks, if auth
fails, functional tests will still fail and catch issues in gate.

Change-Id: I7b0009fa0e29dcaf7d6e5607af7b397e437aac79
changes/04/344004/5
Ryan Beisner 2 years ago
parent
commit
e460db6f64

+ 1
- 1
Makefile View File

@@ -10,7 +10,7 @@ test:
10 10
 
11 11
 functional_test:
12 12
 	@echo Starting amulet deployment tests...
13
-	@juju test -v -p AMULET_HTTP_PROXY,AMULET_OS_VIP --timeout 2700
13
+	@tox -e func27
14 14
 
15 15
 bin/charm_helpers_sync.py:
16 16
 	@mkdir -p bin

+ 16
- 0
test-requirements.txt View File

@@ -7,3 +7,19 @@ flake8>=2.2.4,<=2.4.1
7 7
 os-testr>=0.4.1
8 8
 charm-tools>=2.0.0
9 9
 requests==2.6.0
10
+# BEGIN: Amulet OpenStack Charm Helper Requirements
11
+# Liberty client lower constraints
12
+amulet>=1.14.3,<2.0
13
+bundletester>=0.6.1,<1.0
14
+python-ceilometerclient>=1.5.0,<2.0
15
+python-cinderclient>=1.4.0,<2.0
16
+python-glanceclient>=1.1.0,<2.0
17
+python-heatclient>=0.8.0,<1.0
18
+python-keystoneclient>=1.7.1,<2.0
19
+python-neutronclient>=3.1.0,<4.0
20
+python-novaclient>=2.30.1,<3.0
21
+python-openstackclient>=1.7.0,<2.0
22
+python-swiftclient>=2.6.0,<3.0
23
+pika>=0.10.0,<1.0
24
+distro-info
25
+# END: Amulet OpenStack Charm Helper Requirements

+ 0
- 31
tests/00-setup View File

@@ -1,31 +0,0 @@
1
-#!/bin/bash
2
-#
3
-# Copyright 2016 Canonical Ltd
4
-#
5
-# Licensed under the Apache License, Version 2.0 (the "License");
6
-# you may not use this file except in compliance with the License.
7
-# You may obtain a copy of the License at
8
-#
9
-#  http://www.apache.org/licenses/LICENSE-2.0
10
-#
11
-# Unless required by applicable law or agreed to in writing, software
12
-# distributed under the License is distributed on an "AS IS" BASIS,
13
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14
-# See the License for the specific language governing permissions and
15
-# limitations under the License.
16
-
17
-set -ex
18
-
19
-sudo add-apt-repository --yes ppa:juju/stable
20
-sudo apt-get update --yes
21
-sudo apt-get install --yes amulet \
22
-                           distro-info-data \
23
-                           python-cinderclient \
24
-                           python-distro-info \
25
-                           python-glanceclient \
26
-                           python-heatclient \
27
-                           python-keystoneclient \
28
-                           python-neutronclient \
29
-                           python-novaclient \
30
-                           python-pika \
31
-                           python-swiftclient

+ 0
- 25
tests/016-basic-trusty-juno View File

@@ -1,25 +0,0 @@
1
-#!/usr/bin/python
2
-#
3
-# Copyright 2016 Canonical Ltd
4
-#
5
-# Licensed under the Apache License, Version 2.0 (the "License");
6
-# you may not use this file except in compliance with the License.
7
-# You may obtain a copy of the License at
8
-#
9
-#  http://www.apache.org/licenses/LICENSE-2.0
10
-#
11
-# Unless required by applicable law or agreed to in writing, software
12
-# distributed under the License is distributed on an "AS IS" BASIS,
13
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14
-# See the License for the specific language governing permissions and
15
-# limitations under the License.
16
-
17
-"""Amulet tests on a basic cinder-backup deployment on trusty-juno."""
18
-
19
-from basic_deployment import CinderBackupBasicDeployment
20
-
21
-if __name__ == '__main__':
22
-    deployment = CinderBackupBasicDeployment(series='trusty',
23
-                                             openstack='cloud:trusty-juno',
24
-                                             source='cloud:trusty-updates/juno')
25
-    deployment.run_tests()

+ 0
- 68
tests/README View File

@@ -1,68 +0,0 @@
1
-This directory provides Amulet tests that focus on verification of
2
-cinder-backup deployments.
3
-
4
-test_* methods are called in lexical sort order.
5
-
6
-Test name convention to ensure desired test order:
7
-    1xx service and endpoint checks
8
-    2xx relation checks
9
-    3xx config checks
10
-    4xx functional checks
11
-    9xx restarts and other final checks
12
-
13
-Common uses of backend relations in deployments:
14
-    - [ cinder, cinder-ceph ]
15
-    - [ cinder, cinder-backup ]
16
-    - [ cinder-ceph, ceph ]
17
-    - [ cinder-backup, ceph ]
18
-
19
-In order to run tests, you'll need charm-tools installed (in addition to
20
-juju, of course):
21
-    sudo add-apt-repository ppa:juju/stable
22
-    sudo apt-get update
23
-    sudo apt-get install charm-tools
24
-
25
-If you use a web proxy server to access the web, you'll need to set the
26
-AMULET_HTTP_PROXY environment variable to the http URL of the proxy server.
27
-
28
-The following examples demonstrate different ways that tests can be executed.
29
-All examples are run from the charm's root directory.
30
-
31
-  * To run all tests (starting with 00-setup):
32
-
33
-      make test
34
-
35
-  * To run a specific test module (or modules):
36
-
37
-      juju test -v -p AMULET_HTTP_PROXY 15-basic-trusty-icehouse
38
-
39
-  * To run a specific test module (or modules), and keep the environment
40
-    deployed after a failure:
41
-
42
-      juju test --set-e -v -p AMULET_HTTP_PROXY 15-basic-trusty-icehouse
43
-
44
-  * To re-run a test module against an already deployed environment (one
45
-    that was deployed by a previous call to 'juju test --set-e'):
46
-
47
-      ./tests/15-basic-trusty-icehouse
48
-
49
-For debugging and test development purposes, all code should be idempotent.
50
-In other words, the code should have the ability to be re-run without changing
51
-the results beyond the initial run.  This enables editing and re-running of a
52
-test module against an already deployed environment, as described above.
53
-
54
-Manual debugging tips:
55
-
56
-  * Set the following env vars before using the OpenStack CLI as admin:
57
-      export OS_AUTH_URL=http://`juju-deployer -f keystone 2>&1 | tail -n 1`:5000/v2.0
58
-      export OS_TENANT_NAME=admin
59
-      export OS_USERNAME=admin
60
-      export OS_PASSWORD=openstack
61
-      export OS_REGION_NAME=RegionOne
62
-
63
-  * Set the following env vars before using the OpenStack CLI as demoUser:
64
-      export OS_AUTH_URL=http://`juju-deployer -f keystone 2>&1 | tail -n 1`:5000/v2.0
65
-      export OS_TENANT_NAME=demoTenant
66
-      export OS_USERNAME=demoUser
67
-      export OS_PASSWORD=password
68
-      export OS_REGION_NAME=RegionOne

+ 9
- 0
tests/README.md View File

@@ -0,0 +1,9 @@
1
+# Overview
2
+
3
+This directory provides Amulet tests to verify basic deployment functionality
4
+from the perspective of this charm, its requirements and its features, as
5
+exercised in a subset of the full OpenStack deployment test bundle topology.
6
+
7
+For full details on functional testing of OpenStack charms please refer to
8
+the [functional testing](http://docs.openstack.org/developer/charm-guide/testing.html#functional-testing)
9
+section of the OpenStack Charm Guide.

+ 35
- 50
tests/basic_deployment.py View File

@@ -1,4 +1,4 @@
1
-#!/usr/bin/python
1
+#!/usr/bin/env python
2 2
 #
3 3
 # Copyright 2016 Canonical Ltd
4 4
 #
@@ -45,6 +45,14 @@ class CinderBackupBasicDeployment(OpenStackAmuletDeployment):
45 45
         self._add_relations()
46 46
         self._configure_services()
47 47
         self._deploy()
48
+
49
+        u.log.info('Waiting on extended status checks...')
50
+
51
+        # XXX: cinder-backup workload status ignored until it grows support
52
+        #      https://bugs.launchpad.net/bugs/1604580
53
+        exclude_services = ['mysql', 'cinder-backup']
54
+        self._auto_wait_for_status(exclude_services=exclude_services)
55
+
48 56
         self._initialize_tests()
49 57
 
50 58
     def _add_services(self):
@@ -118,25 +126,19 @@ class CinderBackupBasicDeployment(OpenStackAmuletDeployment):
118 126
     def _initialize_tests(self):
119 127
         """Perform final initialization before tests get run."""
120 128
         # Access the sentries for inspecting service units
121
-        u.log.debug('!!!!!')
122
-        u.log.debug(dir(self.d.sentry))
123
-
124
-        self.mysql_sentry = self.d.sentry.unit['mysql/0']
125
-        self.keystone_sentry = self.d.sentry.unit['keystone/0']
126
-        self.rabbitmq_sentry = self.d.sentry.unit['rabbitmq-server/0']
127
-        self.cinder_sentry = self.d.sentry.unit['cinder/0']
128
-        self.ceph0_sentry = self.d.sentry.unit['ceph/0']
129
-        self.ceph1_sentry = self.d.sentry.unit['ceph/1']
130
-        self.ceph2_sentry = self.d.sentry.unit['ceph/2']
131
-        self.cinder_backup_sentry = self.d.sentry.unit['cinder-backup/0']
129
+        self.mysql_sentry = self.d.sentry['mysql'][0]
130
+        self.keystone_sentry = self.d.sentry['keystone'][0]
131
+        self.rabbitmq_sentry = self.d.sentry['rabbitmq-server'][0]
132
+        self.cinder_sentry = self.d.sentry['cinder'][0]
133
+        self.ceph0_sentry = self.d.sentry['ceph'][0]
134
+        self.ceph1_sentry = self.d.sentry['ceph'][1]
135
+        self.ceph2_sentry = self.d.sentry['ceph'][2]
136
+        self.cinder_backup_sentry = self.d.sentry['cinder-backup'][0]
132 137
         u.log.debug('openstack release val: {}'.format(
133 138
             self._get_openstack_release()))
134 139
         u.log.debug('openstack release str: {}'.format(
135 140
             self._get_openstack_release_string()))
136 141
 
137
-        # Let things settle a bit original moving forward
138
-        time.sleep(30)
139
-
140 142
         # Authenticate admin with keystone
141 143
         self.keystone = u.authenticate_keystone_admin(self.keystone_sentry,
142 144
                                                       user='admin',
@@ -322,7 +324,14 @@ class CinderBackupBasicDeployment(OpenStackAmuletDeployment):
322 324
 
323 325
     def get_broker_response(self):
324 326
         broker_request = self.get_broker_request()
325
-        response_key = "broker-rsp-cinder-backup-0"
327
+        u.log.debug('Broker request: {}'.format(broker_request))
328
+
329
+        response_key = "broker-rsp-{}-{}".format(
330
+            self.cinder_backup_sentry.info['service'],
331
+            self.cinder_backup_sentry.info['unit']
332
+        )
333
+        u.log.debug('Checking response_key: {}'.format(response_key))
334
+
326 335
         ceph_sentrys = [self.ceph0_sentry,
327 336
                         self.ceph1_sentry,
328 337
                         self.ceph2_sentry]
@@ -332,6 +341,7 @@ class CinderBackupBasicDeployment(OpenStackAmuletDeployment):
332 341
                 broker_response = json.loads(relation_data[response_key])
333 342
                 if (broker_request['request-id'] ==
334 343
                         broker_response['request-id']):
344
+                    u.log.debug('broker_response: {}'.format(broker_response))
335 345
                     return broker_response
336 346
 
337 347
     def test_200_cinderbackup_ceph_ceph_relation(self):
@@ -374,11 +384,6 @@ class CinderBackupBasicDeployment(OpenStackAmuletDeployment):
374 384
         if ret:
375 385
             msg = u.relation_error('cinder cinder-backup backup-backend', ret)
376 386
             amulet.raise_status(amulet.FAIL, msg=msg)
377
-        broker_response = self.get_broker_response()
378
-        if not broker_response or broker_response['exit-code'] != 0:
379
-            msg = ('Broker request invalid'
380
-                   ' or failed: {}'.format(broker_response))
381
-            amulet.raise_status(amulet.FAIL, msg=msg)
382 387
 
383 388
     def test_202_cinderbackup_cinder_backend_relation(self):
384 389
         u.log.debug('Checking cinder-backup:backup-backend to '
@@ -525,15 +530,7 @@ class CinderBackupBasicDeployment(OpenStackAmuletDeployment):
525 530
         unit = self.cinder_sentry
526 531
         conf = '/etc/cinder/cinder.conf'
527 532
         unit_mq = self.rabbitmq_sentry
528
-        unit_ks = self.keystone_sentry
529 533
         rel_mq_ci = unit_mq.relation('amqp', 'cinder:amqp')
530
-        rel_ks_ci = unit_ks.relation('identity-service',
531
-                                     'cinder:identity-service')
532
-
533
-        auth_uri = 'http://' + rel_ks_ci['auth_host'] + \
534
-                   ':' + rel_ks_ci['service_port'] + '/'
535
-        auth_url = ('http://%s:%s/' %
536
-                    (rel_ks_ci['auth_host'], rel_ks_ci['auth_port']))
537 534
 
538 535
         expected = {
539 536
             'DEFAULT': {
@@ -549,12 +546,6 @@ class CinderBackupBasicDeployment(OpenStackAmuletDeployment):
549 546
                 'backup_ceph_pool': 'cinder-backup',
550 547
                 'backup_ceph_user': 'cinder-backup'
551 548
             },
552
-            'keystone_authtoken': {
553
-                'admin_user': rel_ks_ci['service_username'],
554
-                'admin_password': rel_ks_ci['service_password'],
555
-                'admin_tenant_name': rel_ks_ci['service_tenant'],
556
-                'auth_uri': auth_uri
557
-            },
558 549
             'cinder-ceph': {
559 550
                 'volume_backend_name': 'cinder-ceph',
560 551
                 'volume_driver': 'cinder.volume.drivers.rbd.RBDDriver',
@@ -569,18 +560,6 @@ class CinderBackupBasicDeployment(OpenStackAmuletDeployment):
569 560
             'rabbit_password': rel_mq_ci['password'],
570 561
             'rabbit_host': rel_mq_ci['hostname'],
571 562
         }
572
-        if self._get_openstack_release() >= self.trusty_liberty:
573
-            expected['keystone_authtoken'] = {
574
-                'auth_uri': auth_uri.rstrip('/'),
575
-                'auth_url': auth_url.rstrip('/'),
576
-                'auth_plugin': 'password',
577
-                'project_domain_id': 'default',
578
-                'user_domain_id': 'default',
579
-                'project_name': 'services',
580
-                'username': rel_ks_ci['service_username'],
581
-                'password': rel_ks_ci['service_password'],
582
-                'signing_dir': '/var/cache/cinder'
583
-            }
584 563
 
585 564
         if self._get_openstack_release() >= self.trusty_kilo:
586 565
             # Kilo or later
@@ -588,8 +567,6 @@ class CinderBackupBasicDeployment(OpenStackAmuletDeployment):
588 567
         else:
589 568
             # Juno or earlier
590 569
             expected['DEFAULT'].update(expected_rmq)
591
-            expected['keystone_authtoken']['auth_host'] = \
592
-                rel_ks_ci['auth_host']
593 570
 
594 571
         for section, pairs in expected.iteritems():
595 572
             ret = u.validate_config_data(unit, conf, section, pairs)
@@ -631,7 +608,15 @@ class CinderBackupBasicDeployment(OpenStackAmuletDeployment):
631 608
         u.log.debug('Cinder api check (volumes.list): {}'.format(check))
632 609
         assert(check == [])
633 610
 
634
-    def test_401_create_delete_volume(self):
611
+    def test_401_check_broker_reponse(self):
612
+        u.log.debug('Checking broker response')
613
+        broker_response = self.get_broker_response()
614
+        if not broker_response or broker_response['exit-code'] != 0:
615
+            msg = ('Broker request invalid'
616
+                   ' or failed: {}'.format(broker_response))
617
+            amulet.raise_status(amulet.FAIL, msg=msg)
618
+
619
+    def test_402_create_delete_volume(self):
635 620
         """Create a cinder volume and delete it."""
636 621
         u.log.debug('Creating, checking and deleting cinder volume...')
637 622
         vol_new = u.create_cinder_volume(self.cinder)

tests/018-basic-vivid-kilo → tests/dev-basic-xenial-newton View File

@@ -1,4 +1,4 @@
1
-#!/usr/bin/python
1
+#!/usr/bin/env python
2 2
 #
3 3
 # Copyright 2016 Canonical Ltd
4 4
 #
@@ -14,10 +14,12 @@
14 14
 # See the License for the specific language governing permissions and
15 15
 # limitations under the License.
16 16
 
17
-"""Amulet tests on a basic cinder-backup deployment on vivid-kilo."""
17
+"""Amulet tests on a basic cinder backup deployment on xenial-newton."""
18 18
 
19 19
 from basic_deployment import CinderBackupBasicDeployment
20 20
 
21 21
 if __name__ == '__main__':
22
-    deployment = CinderBackupBasicDeployment(series='vivid')
22
+    deployment = CinderBackupBasicDeployment(series='xenial',
23
+                                             openstack='cloud:xenial-newton',
24
+                                             source='cloud:xenial-updates/newton')
23 25
     deployment.run_tests()

tests/019-basic-wily-liberty → tests/dev-basic-yakkety-newton View File

@@ -1,4 +1,4 @@
1
-#!/usr/bin/python
1
+#!/usr/bin/env python
2 2
 #
3 3
 # Copyright 2016 Canonical Ltd
4 4
 #
@@ -14,10 +14,10 @@
14 14
 # See the License for the specific language governing permissions and
15 15
 # limitations under the License.
16 16
 
17
-"""Amulet tests on a basic cinder-backup deployment on wily-liberty."""
17
+"""Amulet tests on a basic Cinder backup deployment on yakkety-newton."""
18 18
 
19 19
 from basic_deployment import CinderBackupBasicDeployment
20 20
 
21 21
 if __name__ == '__main__':
22
-    deployment = CinderBackupBasicDeployment(series='wily')
22
+    deployment = CinderBackupBasicDeployment(series='yakkety')
23 23
     deployment.run_tests()

tests/014-basic-precise-icehouse → tests/gate-basic-precise-icehouse View File

@@ -1,4 +1,4 @@
1
-#!/usr/bin/python
1
+#!/usr/bin/env python
2 2
 #
3 3
 # Copyright 2016 Canonical Ltd
4 4
 #

tests/015-basic-trusty-icehouse → tests/gate-basic-trusty-icehouse View File

@@ -1,4 +1,4 @@
1
-#!/usr/bin/python
1
+#!/usr/bin/env python
2 2
 #
3 3
 # Copyright 2016 Canonical Ltd
4 4
 #

tests/017-basic-trusty-kilo → tests/gate-basic-trusty-kilo View File

@@ -1,4 +1,4 @@
1
-#!/usr/bin/python
1
+#!/usr/bin/env python
2 2
 #
3 3
 # Copyright 2016 Canonical Ltd
4 4
 #

tests/021-basic-xenial-mitaka → tests/gate-basic-xenial-mitaka View File

@@ -1,4 +1,4 @@
1
-#!/usr/bin/python
1
+#!/usr/bin/env python
2 2
 #
3 3
 # Copyright 2016 Canonical Ltd
4 4
 #

+ 17
- 20
tests/tests.yaml View File

@@ -1,20 +1,17 @@
1
-bootstrap: true
2
-reset: true
3
-virtualenv: true
4
-makefile:
5
-    - lint
6
-    - test
7
-sources:
8
-    - ppa:juju/stable
9
-packages:
10
-    - amulet
11
-    - distro-info-data
12
-    - python-cinderclient
13
-    - python-distro-info
14
-    - python-glanceclient
15
-    - python-heatclient
16
-    - python-keystoneclient
17
-    - python-neutronclient
18
-    - python-novaclient
19
-    - python-pika
20
-    - python-swiftclient
1
+# Bootstrap the model if necessary.
2
+bootstrap: True
3
+# Re-use bootstrap node instead of destroying/re-bootstrapping.
4
+reset: True
5
+# Use tox/requirements to drive the venv instead of bundletester's venv feature.
6
+virtualenv: False
7
+# Leave makefile empty, otherwise unit/lint tests will rerun ahead of amulet.
8
+makefile: []
9
+# Do not specify juju PPA sources.  Juju is presumed to be pre-installed
10
+# and configured in all test runner environments.
11
+#sources:
12
+# Do not specify or rely on system packages.
13
+#packages:
14
+# Do not specify python packages here.  Use test-requirements.txt
15
+# and tox instead.  ie. The venv is constructed before bundletester
16
+# is invoked.
17
+#python-packages:

+ 47
- 1
tox.ini View File

@@ -5,6 +5,8 @@ skipsdist = True
5 5
 [testenv]
6 6
 setenv = VIRTUAL_ENV={envdir}
7 7
          PYTHONHASHSEED=0
8
+         AMULET_SETUP_TIMEOUT=2700
9
+passenv = HOME TERM AMULET_HTTP_PROXY AMULET_OS_VIP
8 10
 install_command =
9 11
   pip install --allow-unverified python-apt {opts} {packages}
10 12
 commands = ostestr {posargs}
@@ -18,12 +20,56 @@ deps = -r{toxinidir}/requirements.txt
18 20
 basepython = python2.7
19 21
 deps = -r{toxinidir}/requirements.txt
20 22
        -r{toxinidir}/test-requirements.txt
21
-commands = flake8 {posargs} hooks unit_tests tests
23
+commands = flake8 {posargs} --exclude */charmhelpers hooks unit_tests tests
22 24
            charm-proof
23 25
 
24 26
 [testenv:venv]
25 27
 commands = {posargs}
26 28
 
29
+[testenv:func27-noop]
30
+# DRY RUN - For Debug
31
+basepython = python2.7
32
+deps = -r{toxinidir}/requirements.txt
33
+       -r{toxinidir}/test-requirements.txt
34
+commands =
35
+    bundletester -vl DEBUG -r json -o func-results.json --test-pattern "gate-*" -n --no-destroy
36
+
37
+[testenv:func27]
38
+# Charm Functional Test
39
+# Run all gate tests which are +x (expected to always pass)
40
+basepython = python2.7
41
+deps = -r{toxinidir}/requirements.txt
42
+       -r{toxinidir}/test-requirements.txt
43
+commands =
44
+    bundletester -vl DEBUG -r json -o func-results.json --test-pattern "gate-*" --no-destroy
45
+
46
+[testenv:func27-smoke]
47
+# Charm Functional Test
48
+# Run a specific test as an Amulet smoke test (expected to always pass)
49
+basepython = python2.7
50
+deps = -r{toxinidir}/requirements.txt
51
+       -r{toxinidir}/test-requirements.txt
52
+commands =
53
+    bundletester -vl DEBUG -r json -o func-results.json gate-basic-xenial-mitaka --no-destroy
54
+
55
+[testenv:func27-dfs]
56
+# Charm Functional Test
57
+# Run all deploy-from-source tests which are +x (may not always pass!)
58
+basepython = python2.7
59
+deps = -r{toxinidir}/requirements.txt
60
+       -r{toxinidir}/test-requirements.txt
61
+commands =
62
+    bundletester -vl DEBUG -r json -o func-results.json --test-pattern "dfs-*" --no-destroy
63
+
64
+[testenv:func27-dev]
65
+# Charm Functional Test
66
+# Run all development test targets which are +x (may not always pass!)
67
+basepython = python2.7
68
+deps = -r{toxinidir}/requirements.txt
69
+       -r{toxinidir}/test-requirements.txt
70
+commands =
71
+    bundletester -vl DEBUG -r json -o func-results.json --test-pattern "dev-*" --no-destroy
72
+
27 73
 [flake8]
28 74
 ignore = E402,E226
29 75
 exclude = hooks/charmhelpers

Loading…
Cancel
Save