openstack
/
rally-openstack
Code Issues Proposed changes

[ci] Try fixing floating bug at functional tests 

The issue: functional job fails sometimes with the following error:

> Invalid Task: KeyError: 'platforms'

The debug gave me a next traceback:

> Traceback (most recent call last):
>   File "/usr/local/lib/python2.7/dist-packages/rally/task/engine.py", line 396, in validate
>     self._validate_config_semantic(self.config)
>   File "/usr/local/lib/python2.7/dist-packages/rally/common/logging.py", line 248, in wrapper
>     result = f(self, *args, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/rally/task/engine.py", line 369, in _validate_config_semantic
>     env_data = self.env.data
>   File "/usr/local/lib/python2.7/dist-packages/rally/env/env_mgr.py", line 180, in data
>     return self.cached_data
>   File "/usr/local/lib/python2.7/dist-packages/rally/env/env_mgr.py", line 157, in cached_data
>     for p in self._env["platforms"]:
> KeyError: 'platforms'

A possible reason: to speed up checking samples, the special workaround
was added[1]. It is designed to initialize OpenStack credentials with
OSClients once and share it between all checks for separate samples,
which resulted in decreasing auth-calls. The design of this hack includes
mocking the result of obtaining Deployment object.
After replacing Deployment component with Environment, the same
environment manager is shared across all checks. The hack was not
adopted after such change which resulted in obtaining platform data
(credentials in our case) from database[2][3][4] in each sample-check.
Since these checks are launched in parallel and the one env manager
object is used, there is a possible race condition when one thread already
"obtained" data from database and contined validation, another
check rewrote self._env obj[3] which doesn't contain 'platforms'
by-default (it is done at the next line[4]). Since switching from line[3]
to line[4] takes ms, this race condition happens not often.

This patch removes the workaround. Anyway it stopped working after
switching to Environment component.

[1] - https://github.com/openstack/rally-openstack/blob/1.2.0/tests/check_samples/test_task_samples.py#L97-L115
[2] - https://github.com/openstack/rally/blob/1.1.0/rally/task/engine.py#L369
[3] - https://github.com/openstack/rally/blob/1.1.0/rally/env/env_mgr.py#L178
[4] - https://github.com/openstack/rally/blob/1.1.0/rally/env/env_mgr.py#L179

Change-Id: I298533390fe3115c20310e7758a1548e62d5560f
This commit is contained in:
Andrey Kurilin 2018-09-02 18:27:59 +03:00
parent 467cc05559
commit 14c29790ae
1 changed files with 0 additions and 21 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

21
tests/functional/test_task_samples.py
View File

@ -16,7 +16,6 @@
import copy
import json
import mock
import os
import re
import traceback
@ -93,26 +92,6 @@ class TestTaskSamples(unittest.TestCase):
rally("deployment create --name MAIN --filename %s" % deployment_cfg,
write_report=False)
# NOTE(andreykurilin): mock building credential to share one cache of
# clients(it will allow to avoid hundreds of redundant
# authentications) between validations of different samples
deployment = rapi.deployment._get("MAIN")
original_get_credentials_for = deployment.get_credentials_for
creds_cache = {}
def get_credentials_for(platform):
if platform not in creds_cache:
creds_cache[platform] = original_get_credentials_for(
platform)
return creds_cache[platform]
deployment.get_credentials_for = get_credentials_for
deployment_patcher = mock.patch("rally.api.objects.Deployment.get")
m_deployment = deployment_patcher.start()
m_deployment.return_value = deployment
self.addCleanup(deployment_patcher.stop)
# store all failures and print them at once
failed_samples = {}