openstack
/
sahara-dashboard
Code Issues Proposed changes

Added EDP tests 

Add EDP tests to savanna dashboard tests and fixed some flows

Implements blueprint add-edp-tests-to-ui
Implements blueprint neutron-for-ui-tests

Change-Id: I1d0329423e201c8c39e51875f20e5a6da1eae608
This commit is contained in:
Vadim Rovachev 2013-11-13 19:35:40 +04:00
parent a8e780c235
commit 70f849148d
15 changed files with 657 additions and 69 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

60
savannadashboard/tests/README.rst
View File

@ -1,22 +1,54 @@
UI tests for Savanna dashboard
Savanna Dashboard Selenium Tests
=====================================
How to run
Main goal of Selenium Tests
----------
Create config file for selenium tests - `/savannadashboard/tests/configs/config.conf`.
You can take a look at the sample config file - `/savannadashboard/tests/configs/config.conf.sample`.
All values used in `/savannadashboard/tests/configs/config.py` file are
defaults, so, if they are applicable for your environment then you can skip
config file creation.
Selenium tests for Savanna Dashboard are designed to check the quality of plug-in Savanna Dashboard for the Horizon.
Install virtual framebuffer X server for X Version 11 (Xvfb):
sudo apt-get -y install xvfb
Install Firefox:
sudo add-apt-repository ppa:ubuntu-mozilla-security/ppa
sudo apt-get update
sudo apt-get install firefox libstdc++5
How to run UI tests:
----------
To run ui tests you should use the corresponding tox env: `tox -e tests`.
It's assumed that the savanna and horizon are already installed and running.
Information about installation and start of savanna and horizon can be found on the savanna site
http://docs.openstack.org/developer/savanna/#user-guide
in tabs Savanna Installation Guide and Savanna UI Installation Guide.
1. Go to savanna dashboard path.
2. Create config file for selenium tests - `savannadashboard/tests/configs/config.py`.
You can take a look at the sample config file - `savannadashboard/tests/configs/config.py.sample`.
All values used in `savannadashboard/tests/configs/parameters.py` file are
defaults, so, if they are applicable for your environment then you can skip
config file creation.
3. Install virtual framebuffer X server for X Version 11 (Xvfb):
sudo apt-get -y install xvfb
4. Install Firefox:
sudo add-apt-repository ppa:ubuntu-mozilla-security/ppa
sudo apt-get update
sudo apt-get install firefox libstdc++5
5. To run ui tests you should use the corresponding tox env: `tox -e tests`.
If need to run only one test module, use:
tox -e tests -- -a tags='<module_name>'
<module_name> may be equal 'cluster', 'cluster_template', 'image_registry', 'node_group_template', 'image_registry', 'vanilla', 'hdp'
It's full list of actual modules.
Coverage:
----------
-Clusters
-Cluster templates
-Node group templates
-Image registry
-Data sources
-Job binaries
-Jobs
-Job executions

404
savannadashboard/tests/base.py
View File

@ -13,16 +13,26 @@
# See the License for the specific language governing permissions and
# limitations under the License.
import logging
import os
import time
import traceback
import nose.plugins.attrib
import selenium.common.exceptions as selenim_except
from selenium import webdriver
import selenium.webdriver.common.by as by
from swiftclient import client as swift_client
import testtools
import time
import unittest2
import savannadashboard.tests.configs.config as cfg
logger = logging.getLogger('swiftclient')
logger.setLevel(logging.WARNING)
def attr(*args, **kwargs):
def decorator(f):
if 'type' in kwargs and isinstance(kwargs['type'], str):
@ -39,11 +49,22 @@ class UITestCase(unittest2.TestCase):
@classmethod
def setUpClass(cls):
cls.driver = webdriver.Firefox()
cls.driver.get(cfg.common.base_url + "/")
cls.find_clear_send(by.By.ID, "id_username", cfg.common.user)
cls.find_clear_send(by.By.ID, "id_password", cfg.common.password)
cls.driver.find_element_by_xpath("//button[@type='submit']").click()
try:
cls.ifFail = False
cls.driver = webdriver.Firefox()
cls.driver.get(cfg.common.base_url + "/")
cls.find_clear_send(by.By.ID, "id_username", cfg.common.user)
cls.find_clear_send(by.By.ID, "id_password", cfg.common.password)
cls.driver.find_element_by_xpath(
"//button[@type='submit']").click()
except Exception:
traceback.print_exc()
cls.ifFail = True
pass
def setUp(self):
if self.ifFail:
self.fail("setUpClass method is fail")
def image_registry(self, image_name, user_name=None, description=None,
tags_to_add=None, tags_to_remove=None, positive=True,
@ -65,10 +86,12 @@ class UITestCase(unittest2.TestCase):
close_window, message, 'Edit')
def create_node_group_template(
self, name, list_processes, plugin, flavor="m1.tiny", params=None,
self, name, list_processes, plugin, flavor=None, params=None,
storage={'type': 'Ephemeral Drive'}, description=None,
positive=True, message=None, close_window=True):
driver = self.driver
if not flavor:
flavor = cfg.common.flavor
driver.get(cfg.common.base_url + "/savanna/nodegroup_templates/")
self.await_element(by.By.ID, "nodegroup_templates__action_create")
driver.find_element_by_id("nodegroup_templates__action_create").click()
@ -84,13 +107,17 @@ class UITestCase(unittest2.TestCase):
storage['volume_per_node'])
self.find_clear_send(by.By.ID, "id_volumes_size",
storage['volume_size'])
if cfg.common.floationg_ip_pool:
self.driver.find_element_by_xpath(
"//*[@id='id_floating_ip_pool']/option[text()='%s']"
% cfg.common.floationg_ip_pool).click()
processes = []
for process in list_processes:
number_pr = self.search_id_processes(process, plugin)
driver.find_element_by_id(
"id_processes_%d" % number_pr).click()
"id_processes_%s" % str(number_pr)).click()
processes.append(driver.find_element_by_id(
"id_processes_%d" % number_pr).
"id_processes_%s" % str(number_pr)).
find_element_by_xpath('..').text)
if params:
self.config_helper(params)
@ -119,7 +146,7 @@ class UITestCase(unittest2.TestCase):
if anti_affinity_groups:
for group in anti_affinity_groups:
driver.find_element_by_id(
"id_anti_affinity_%d" % self.search_id_processes(
"id_anti_affinity_%s" % self.search_id_processes(
group, plugin)).click()
driver.find_element_by_link_text("Node Groups").click()
number_to_add = 0
@ -147,7 +174,7 @@ class UITestCase(unittest2.TestCase):
else:
self.error_helper(message)
def create_cluster(self, name, cluster_template, keypair, plugin,
def create_cluster(self, name, cluster_template, plugin, keypair=None,
close_window=True, description=None, positive=True,
await_run=True, message=None):
driver = self.driver
@ -162,8 +189,14 @@ class UITestCase(unittest2.TestCase):
driver.find_element_by_xpath("//select[@id='id_image']/option"
"[text()='%s']" %
plugin.base_image).click()
if not keypair:
keypair = cfg.common.keypair
driver.find_element_by_xpath("//select[@id='id_keypair']"
"/option[text()='%s']" % keypair).click()
if cfg.common.neutron_management_network:
driver.find_element_by_xpath(
"//select[@id='id_neutron_management_network']/option[text()="
"'%s']" % cfg.common.neutron_management_network).click()
driver.find_element_by_xpath("//input[@value='Create']").click()
if not message:
message = 'Success: Created Cluster %s' % name
@ -174,49 +207,325 @@ class UITestCase(unittest2.TestCase):
if await_run:
self.await_cluster_active(name)
def delete_node_group_templates(self, names, undelete_names=None):
def create_data_source(self, name, url, close_window=True,
description=None, positive=True, message=None):
driver = self.driver
driver.get(cfg.common.base_url + "/savanna/data_sources/")
self.await_element(by.By.ID, "data_sources__action_create data source")
driver.find_element_by_id(
"data_sources__action_create data source").click()
self.await_element(by.By.ID, "id_data_source_name")
self.find_clear_send(by.By.ID, "id_data_source_name", name)
self.find_clear_send(by.By.ID, "id_data_source_url", url)
self.find_clear_send(by.By.ID, "id_data_source_credential_user",
cfg.common.user)
self.find_clear_send(by.By.ID, "id_data_source_credential_pass",
cfg.common.password)
if description:
self.find_clear_send(by.By.ID, "id_data_source_description",
description)
driver.find_element_by_xpath("//input[@value='Create']").click()
if not message:
message = 'Success: Data source created'
if close_window:
self.check_create_object(name, positive, message)
else:
self.error_helper(message)
def create_job_binary(self, name, parameters_of_storage, description=None,
positive=True, message=None, close_window=True):
driver = self.driver
storage_type = parameters_of_storage['storage_type']
driver.get(cfg.common.base_url + "/savanna/job_binaries/")
self.await_element(by.By.ID, "job_binaries__action_create job binary")
driver.find_element_by_id(
"job_binaries__action_create job binary").click()
self.await_element(by.By.ID, "id_job_binary_name")
self.find_clear_send(by.By.ID, "id_job_binary_name", name)
driver.find_element_by_xpath("//select[@id='id_job_binary_type']/optio"
"n[text()='%s']" % storage_type).click()
if storage_type == 'Swift internal':
self.find_clear_send(by.By.ID, "id_job_binary_url",
parameters_of_storage['url'])
self.find_clear_send(by.By.ID, "id_job_binary_username",
cfg.common.user)
self.find_clear_send(by.By.ID, "id_job_binary_password",
cfg.common.password)
elif storage_type == 'Savanna internal database':
savanna_binary = parameters_of_storage['Savanna binary']
driver.find_element_by_xpath(
"//select[@id='id_job_binary_savanna_internal']/option[text()"
"='%s']" % savanna_binary).click()
if savanna_binary == '*Upload a new file':
file = '%s/tests/resources/%s' % (
os.getcwd(), parameters_of_storage['filename'])
driver.find_element_by_id('id_job_binary_file').send_keys(file)
elif savanna_binary == '*Create a script':
self.find_clear_send(by.By.ID, "id_job_binary_script_name",
parameters_of_storage['script_name'])
self.find_clear_send(by.By.ID, "id_job_binary_script",
parameters_of_storage['script_text'])
if description:
self.find_clear_send(by.By.ID, "id_job_binary_description",
description)
driver.find_element_by_xpath("//input[@value='Create']").click()
if not message:
message = 'Success: Successfully created job binary'
if close_window:
self.check_create_object(name, positive, message)
else:
self.error_helper(message)
def create_job(self, name, job_type, main=None, libs=None,
close_window=True, description=None, positive=True,
message=None):
driver = self.driver
driver.get(cfg.common.base_url + "/savanna/jobs/")
self.await_element(by.By.ID, "jobs__action_create job")
driver.find_element_by_id("jobs__action_create job").click()
self.await_element(by.By.ID, "id_job_name")
self.find_clear_send(by.By.ID, "id_job_name", name)
driver.find_element_by_xpath(
"//select[@id='id_job_type']/option[text()='%s']"
% job_type).click()
if main:
driver.find_element_by_xpath(
"//select[@id='id_main_binary']/option[text()='%s']"
% main).click()
if description:
self.find_clear_send(by.By.ID, "id_job_description", description)
if libs:
driver.find_element_by_link_text('Libs').click()
self.await_element(by.By.ID, "id_lib_binaries")
for lib in libs:
driver.find_element_by_xpath(
"//select[@id='id_lib_binaries']/option[text()='%s']"
% lib).click()
driver.find_element_by_id('add_lib_button').click()
driver.find_element_by_xpath("//input[@value='Create']").click()
if not message:
message = 'Success: Job created'
if close_window:
self.check_create_object(name, positive, message)
else:
self.error_helper(message)
def launch_job_on_existing_cluster(self, name, input, output, cluster,
configure=None, positive=True,
message=None, close_window=True,
await_launch=True):
driver = self.driver
driver.get(cfg.common.base_url + "/savanna/jobs/")
self.await_element(by.By.ID, "jobs__action_create job")
action_column = driver.find_element_by_link_text(
name).find_element_by_xpath('../../td[4]')
action_column.find_element_by_link_text('More').click()
action_column.find_element_by_link_text(
'Launch On Existing Cluster').click()
self.await_element(by.By.ID, "id_job_input")
driver.find_element_by_xpath(
"//select[@id='id_job_input']/option[text()='%s']" % input).click()
driver.find_element_by_xpath(
"//select[@id='id_job_output']/option[text()='%s']" %
output).click()
driver.find_element_by_xpath(
"//select[@id='id_cluster']/option[text()='%s']" % cluster).click()
if configure:
driver.find_element_by_link_text('Configure').click()
for config_part, values in configure.items():
config_number = 1
for config, value in values.items():
driver.find_element_by_id(
config_part).find_element_by_link_text('Add').click()
driver.find_element_by_xpath(
'//*[@id="%s"]/table/tbody/tr[%i]/td[1]/input' % (
config_part, config_number)).send_keys(config)
driver.find_element_by_xpath(
'//*[@id="%s"]/table/tbody/tr[%i]/td[2]/input' % (
config_part, config_number)).send_keys(value)
config_number += 1
driver.find_element_by_xpath("//input[@value='Launch']").click()
if not message:
message = 'Success: Job launched'
if close_window:
self.check_create_object(name, positive, message,
check_create_element=False)
if await_launch:
self.await_launch_job()
else:
self.error_helper(message)
def delete_node_group_templates(self, names, undelete_names=None,
finally_delete=False):
url = "/savanna/nodegroup_templates/"
delete_button_id = 'nodegroup_templates__action_' \
'delete_nodegroup_template'
self.delete_and_validate(url, delete_button_id, names, undelete_names)
self.delete_and_validate(url, delete_button_id, names, undelete_names,
finally_delete)
def delete_cluster_templates(self, names, undelete_names=None):
def delete_cluster_templates(self, names, undelete_names=None,
finally_delete=False):
url = "/savanna/cluster_templates/"
delete_button_id = "cluster_templates__action_delete_cluster_template"
self.delete_and_validate(url, delete_button_id, names, undelete_names)
self.delete_and_validate(url, delete_button_id, names, undelete_names,
finally_delete)
def delete_clusters(self, names, undelete_names=None):
def delete_clusters(self, names, undelete_names=None,
finally_delete=False):
url = "/savanna/"
delete_button_id = "clusters__action_delete"
msg = "Success: Deleted Cluster"
self.delete_and_validate(url, delete_button_id, names, undelete_names,
succes_msg=msg)
finally_delete, succes_msg=msg)
def unregister_images(self, names, undelete_names=[]):
def delete_data_sources(self, names, undelete_names=None,
finally_delete=False):
url = "/savanna/data_sources/"
delete_button_id = "data_sources__action_delete"
msg = "Success: Deleted Data source"
err_msg = 'Error: Unable to delete data source'
info_msg = 'Info: Deleted Data source'
self.delete_and_validate(url, delete_button_id, names, undelete_names,
finally_delete, msg, err_msg, info_msg)
def delete_job_binaries(self, names, undelete_names=None,
finally_delete=False):
url = "/savanna/job_binaries/"
delete_button_id = "job_binaries__action_delete"
msg = "Success: Deleted Job binary"
err_msg = 'Error: Unable to delete job binary'
info_msg = 'Info: Deleted Job binary'
if not undelete_names and len(names) > 1:
msg = "Success: Deleted Job binarie"
if undelete_names and len(names)-len(undelete_names) > 1:
info_msg = 'Info: Deleted Job binarie'
if undelete_names and len(undelete_names) > 1:
err_msg = 'Error: Unable to delete job binarie'
self.delete_and_validate(url, delete_button_id, names, undelete_names,
finally_delete, msg, err_msg, info_msg)
def delete_jobs(self, names, undelete_names=None, finally_delete=False):
url = "/savanna/jobs/"
delete_button_id = "jobs__action_delete"
msg = "Success: Deleted Job"
err_msg = 'Error: Unable to delete job'
info_msg = 'Info: Deleted Job'
self.delete_and_validate(url, delete_button_id, names, undelete_names,
finally_delete, msg, err_msg, info_msg)
def delete_all_job_executions(self):
driver = self.driver
driver.get(cfg.common.base_url + "/savanna/job_executions/")
delete_button_id = 'job_executions__action_delete'
self.await_element(by.By.ID, delete_button_id)
if self.does_element_present(by.By.CLASS_NAME, 'multi_select_column'):
if not driver.find_element_by_xpath(
'//*[@class=\'multi_select_column\']/input').is_selected():
driver.find_element_by_class_name(
'multi_select_column').click()
driver.find_element_by_id(delete_button_id).click()
self.await_element(by.By.LINK_TEXT, 'Delete Job executions')
driver.find_element_by_link_text('Delete Job executions').click()
self.await_element(by.By.CLASS_NAME, "alert-success")
message = 'Success: Deleted Job execution'
actual_message = self.find_alert_message(
"alert-success", first_character=2,
last_character=len(message)+2)
self.assertEqual(actual_message, message)
def unregister_images(self, names, undelete_names=[],
finally_delete=False):
url = '/savanna/image_registry/'
delete_button_id = "image_registry__action_Unregister"
msg = "Success: Unregistered Image"
self.delete_and_validate(url, delete_button_id, names, undelete_names,
succes_msg=msg)
finally_delete, succes_msg=msg,)
#-------------------------helpers_methods--------------------------------------
@staticmethod
def connect_to_swift():
return swift_client.Connection(
authurl=cfg.common.keystone_url,
user=cfg.common.user,
key=cfg.common.password,
tenant_name=cfg.common.tenant,
auth_version=2
)
@staticmethod
def delete_swift_container(swift, container):
objects = [obj['name'] for obj in swift.get_container(container)[1]]
for obj in objects:
swift.delete_object(container, obj)
swift.delete_container(container)
@classmethod
def find_clear_send(cls, by_find, find_element, send):
cls.driver.find_element(by=by_find, value=find_element).clear()
cls.driver.find_element(by=by_find, value=find_element).send_keys(send)
def delete_and_validate(self, url, delete_button_id, names, undelete_names,
finally_delete,
succes_msg='Success: Deleted Template',
error_msg='Error: Unable to delete template',
info_msg='Info: Deleted Template'):
driver = self.driver
driver.refresh()
driver.get(cfg.common.base_url + url)
self.await_element(by.By.ID, delete_button_id)
for name in names:
# choose checkbox for this element
driver.find_element_by_link_text("%s" % name).\
find_element_by_xpath("../../td[1]/input").click()
try:
driver.find_element_by_link_text("%s" % name).\
find_element_by_xpath("../../td[1]/input").click()
except selenim_except.NoSuchElementException as e:
if finally_delete:
pass
else:
print ('element with name %s not found for delete' % name)
raise e
# click deletebutton
driver.find_element_by_id(delete_button_id).click()
# wait window to confirm the deletion
@ -326,7 +635,7 @@ class UITestCase(unittest2.TestCase):
else:
#Add existing tags in the list
list_for_check_tags = driver.\
find_element(by=by.By.LINK_TEXT, value='latest-ci-image').\
find_element(by=by.By.LINK_TEXT, value=image_name).\
find_element_by_xpath('../../td[3]').text.split('\n')
# Click "Edit Tags"
driver.find_element(by=by.By.LINK_TEXT, value=image_name).\
@ -385,7 +694,7 @@ class UITestCase(unittest2.TestCase):
def choose_plugin_name(self, plugin_name, hadoop_version, name,
description, id_name):
self.await_element(by.By.XPATH, "//*[@id='modal_wrapper']"
"/div/form/div[4]/input")
"/div/form/div[3]/input")
self.driver.find_element_by_xpath(
"//select[@id='id_plugin_name']/option[text()='%s']" %
plugin_name).click()
@ -403,13 +712,13 @@ class UITestCase(unittest2.TestCase):
if description:
self.find_clear_send(by.By.ID, "id_description", description)
def check_alert(self, alert, message, list_obj, deleted=True):
def check_alert(self, alert, expected_message, list_obj, deleted=True):
self.await_element(by.By.CLASS_NAME, alert)
if self.find_alert_message(alert, first_character=2,
last_character=len(message)+2) != message:
self.fail("%s != %s" % (alert, message))
actual_message = self.find_alert_message(
alert, first_character=2, last_character=len(expected_message)+2)
self.assertEqual(actual_message, expected_message)
not_expected_objs = list(set(self.find_alert_message(
alert, first_character=len(message)+2).split(
alert, first_character=len(expected_message)+2).split(
", ")).symmetric_difference(set(list_obj)))
if not_expected_objs:
self.fail("have deleted objects: %s" % not_expected_objs)
@ -457,7 +766,7 @@ class UITestCase(unittest2.TestCase):
self.fail(message)
def check_create_object(self, name, positive, expected_message,
check_columns=None):
check_columns=None, check_create_element=True):
driver = self.driver
expected_alert = "alert-error"
unexpected_alert = "alert-success"
@ -471,13 +780,13 @@ class UITestCase(unittest2.TestCase):
fail_mesg = self.driver.find_element(
by=by.By.CLASS_NAME, value=unexpected_alert).text[2:]
self.fail("Result of creation %s is not expected: %s != %s"
% (name, expected_alert, fail_mesg))
% (name, expected_message, fail_mesg))
time.sleep(1)
else:
self.fail("alert check:%s time out" % expected_alert)
actual_message = self.driver.find_element(
by=by.By.CLASS_NAME, value=expected_alert).text[2:]
if positive:
if check_create_element and positive:
self.assertEqual(expected_message, str(actual_message))
if not self.does_element_present(by.By.LINK_TEXT, name):
self.fail("object with name:%s not found" % name)
@ -501,15 +810,48 @@ class UITestCase(unittest2.TestCase):
find_element_by_xpath("../../td[3]").text
i = 1
while str(status) != 'Active':
if i > cfg.common.cluster_creation_timeout * 6:
self.fail(
'cluster is not getting status \'Active\', '
'passed %d minutes' % cfg.common.cluster_creation_timeout)
if str(status) == 'Error':
self.fail('Cluster state == \'Error\'.')
status = driver.find_element_by_link_text("selenium-cl").\
find_element_by_xpath("../../td[3]").text
time.sleep(10)
i += 1
def await_launch_job(self):
driver = self.driver
driver.get(cfg.common.base_url + "/savanna/job_executions/")
self.await_element(by.By.ID, 'job_executions')
job_id = driver.find_element_by_id(
'job_executions').find_elements_by_class_name(
'ajax-update')[-1].get_attribute('id')
status = driver.find_element_by_xpath(
'//*[@id="%s"]/td[3]' % job_id).text
timeout = cfg.common.job_launch_timeout * 60
while str(status) != 'SUCCEEDED':
if timeout <= 0:
self.fail(
'Job did not return to \'SUCCEEDED\' status within '
'%d minute(s).' % cfg.common.job_launch_timeout)
if status == 'KILLED':
self.fail('Job status == \'KILLED\'.')
status = driver.find_element_by_xpath(
'//*[@id="%s"]/td[3]' % job_id).text
time.sleep(10)
timeout -= 10
@classmethod
def tearDownClass(cls):
cls.driver.quit()

2
savannadashboard/tests/cluster_template_tests/negative_tests.py
View File

@ -21,7 +21,7 @@ import savannadashboard.tests.configs.config as cfg
class UINegativeCreateClusterTemplateTest(base.UITestCase):
@base.attr(tags='cluster_template')
@base.attr(tags=['cluster_template', 'vanilla'])
@testtools.skip
@testtools.skipIf(cfg.vanilla.skip_plugin_tests,
'tests for vanilla plugin skipped')

2
savannadashboard/tests/cluster_template_tests/test_create_cluster_template.py
View File

@ -52,7 +52,7 @@ class UICreateClusterTemplate(base.UITestCase):
self.delete_node_group_templates(["selenium-master",
"selenium-worker"])
@base.attr('cluster_template', 'hdp')
@base.attr(tags=['cluster_template', 'hdp'])
@testtools.skipIf(cfg.hdp.skip_plugin_tests,
'tests for hdp plugin skipped')
def test_create_cluster_template_for_hdp(self):

113
savannadashboard/tests/cluster_tests/test_clusters.py
View File

@ -13,7 +13,10 @@
# See the License for the specific language governing permissions and
# limitations under the License.
import random
import string
import testtools
import traceback
from savannadashboard.tests import base
import savannadashboard.tests.configs.config as cfg
@ -21,12 +24,20 @@ import savannadashboard.tests.configs.config as cfg
class UICreateCluster(base.UITestCase):
@base.attr('cluster', 'vanilla', speed='slow')
@base.attr(tags=['cluster', 'vanilla'], speed='slow')
@testtools.skipIf(cfg.vanilla.skip_plugin_tests,
'tests for vanilla plugin skipped')
def test_create_vanilla_cluster(self):
try:
self.create_node_group_template('selenium-master', ["NN", "JT"],
processes = ["NN", "JT"]
await_run = False
if not cfg.vanilla.skip_edp_test:
processes = ["NN", "JT", "OZ"]
await_run = True
self.create_node_group_template('selenium-master', processes,
cfg.vanilla,
storage={'type': 'Cinder Volume',
"volume_per_node": 1,
@ -41,7 +52,7 @@ class UICreateCluster(base.UITestCase):
cfg.vanilla)
self.create_cluster_template("selenium-cl-tmpl",
{'selenium-master': 1,
'selenium-worker': 2}, cfg.vanilla,
'selenium-worker': 1}, cfg.vanilla,
anti_affinity_groups=["NN",
"DN", "TT"])
self.create_cluster_template("selenium-cl-tmpl2",
@ -50,8 +61,10 @@ class UICreateCluster(base.UITestCase):
cfg.vanilla,
anti_affinity_groups=
["NN", "DN", "TT", "JT"])
self.create_cluster('selenium-cl', 'selenium-cl-tmpl', 'vrovachev',
cfg.vanilla, await_run=False)
self.create_cluster('selenium-cl', 'selenium-cl-tmpl',
cfg.vanilla, await_run=await_run)
if not cfg.vanilla.skip_edp_test:
self.edp_helper()
self.delete_node_group_templates(["selenium-master",
"selenium-worker",
"selenium-del1",
@ -68,8 +81,90 @@ class UICreateCluster(base.UITestCase):
undelete_names=[
"selenium-master",
"selenium-worker"])
except Exception as e:
traceback.print_exc()
raise e
finally:
self.delete_clusters(['selenium-cl'])
self.delete_cluster_templates(['selenium-cl-tmpl'])
self.delete_node_group_templates(["selenium-master",
"selenium-worker"])
try:
self.delete_clusters(['selenium-cl'], finally_delete=True)
except Exception:
pass
try:
self.delete_cluster_templates(['selenium-cl-tmpl',
'selenium-cl-tmpl2'],
finally_delete=True)
except Exception:
pass
try:
self.delete_node_group_templates(["selenium-master",
"selenium-worker",
"selenium-del1",
"selenium-del2"],
finally_delete=True)
except Exception:
pass
def edp_helper(self):
try:
swift = self.connect_to_swift()
swift.put_container('selenium-container')
swift.put_object(
'selenium-container', 'input', ''.join(random.choice(
':' + ' ' + '\n' + string.ascii_lowercase)
for x in range(10000)))
self.create_data_source(
'input', 'selenium-container.savanna/input')
self.create_data_source(
'output', 'selenium-container.savanna/output')
parameters_of_storage = {
'storage_type': 'Savanna internal database',
'Savanna binary': '*Upload a new file',
'filename': 'edp-lib.jar'}
self.create_job_binary('edp-lib.jar', parameters_of_storage)
parameters_of_storage = {
'storage_type': 'Savanna internal database',
'Savanna binary': '*Create a script',
'script_name': 'edp-job.pig',
'script_text': open('tests/resources/edp-job.pig').read()}
self.create_job_binary('edp-job.pig', parameters_of_storage)
self.create_job(
'selenium-job', 'Pig', 'edp-job.pig', ['edp-lib.jar'])
self.launch_job_on_existing_cluster(
'selenium-job', 'input', 'output', 'selenium-cl')
except Exception as e:
raise e
finally:
try:
self.delete_swift_container(swift, 'selenium-container')
except Exception:
pass
try:
self.delete_all_job_executions()
except Exception:
pass
try:
self.delete_jobs(['selenium-job'], finally_delete=True)
except Exception:
pass
try:
self.delete_job_binaries(['edp-lib.jar', 'edp-job.pig'],
finally_delete=True)
except Exception:
pass
try:
self.delete_data_sources(['input', 'output'],
finally_delete=True)
except Exception:
pass

13
savannadashboard/tests/configs/config.conf.sample
View File

@ -2,10 +2,19 @@
base_url = "http://127.0.0.1:8080"
user = "admin"
password = "admin"
keypair = 'jenkins'
tenant = 'admin'
flavor = 'm1.minniemouse'
# uncomment this parameters if quantum in OpenStack
# neutron_management_network = ''
# floationg_ip_pool = ''
keystone_url = 'http://127.0.0.1:5000/v2.0'
# in minutes
cluster_creation_timeout = 10
# in seconds
await_element = 10
# in minutes
job_launch_timeout = 5
image_name_for_register = 'image_name'
image_name_for_edit = 'image_name'
[vanilla]
@ -13,10 +22,10 @@ skip_plugin_tests = False
plugin_name = "Vanilla Apache Hadoop"
plugin_overview_name = "vanilla"
hadoop_version = "1.2.1"
processes = {"NN": 0, "DN": 1, "SNN": 2, "OZ": 3, "TT": 4, "JT": 5}
processes = NN: 0, DN: 1, SNN: 2, OZ: 3, TT: 4, JT: 5
base_image = "image_name"
[hdp]
skip_plugin_tests = False
plugin_name = "Hortonworks Data Platform"
hadoop_version = "1.3.0"
hadoop_version = "1.3.2"
base_image = "image_name"

37
savannadashboard/tests/configs/config.py
View File

@ -28,18 +28,42 @@ CommonGroup = [
cfg.StrOpt('password',
default='pass',
help="password for keystone user"),
cfg.StrOpt('keypair',
default='public-jenkins',
help='keypair for create cluster'),
cfg.StrOpt('tenant',
default='admin',
help='keystone tenant'),
cfg.StrOpt('flavor',
default='m1.minniemouse',
help='OpenStack flavor name for image.'),
cfg.StrOpt('neutron_management_network',
default=None,
help='Private network for quantum.'
'Must be specified in create cluster tab'),
cfg.StrOpt('floationg_ip_pool',
default=None,
help='Public network for quantum.'
'Must be specified in create nodegroup template tab'),
cfg.StrOpt('keystone_url',
default='http://127.0.0.1:5000/v2.0',
help='url for keystone authentication'),
cfg.IntOpt('cluster_creation_timeout',
default=10,
help="cluster timeout in minutes"),
cfg.IntOpt('await_element',
default=10,
default=15,
help="await each web element in seconds"),
cfg.StrOpt('image_name_for_register',
default='fedora_19',
help='Image name for register to Savanna'),
cfg.StrOpt('image_name_for_edit',
default='latest-ci-image',
help='Image name for edit in image registry in Savanna')
help='Image name for edit in image registry in Savanna'),
cfg.IntOpt('job_launch_timeout',
default=5,
help='Timeout for job launch (in minutes); '
'minimal value is 1.'),
]
vanilla_group = cfg.OptGroup(name='vanilla', title="vanilla configs")
@ -51,6 +75,7 @@ VanillaGroup = [
If this variable is True then
tests for vanilla will be skipped
"""),
cfg.BoolOpt('skip_edp_test', default=True),
cfg.StrOpt('plugin_name',
default='Vanilla Apache Hadoop',
help="plugin title, default: Vanilla Apache Hadoop"),
@ -60,12 +85,12 @@ VanillaGroup = [
cfg.StrOpt('hadoop_version',
default='1.2.1',
help="hadoop version for plugin"),
cfg.ListOpt('processes',
cfg.DictOpt('processes',
default={"NN": 0, "DN": 1, "SNN": 2,
"OZ": 3, "TT": 4, "JT": 5},
help='numbers of processes for vanilla in savannabashboard'),
cfg.StrOpt('base_image',
default='latest-ci-image',
default='ubuntu_savanna_latest',
help="image name for start vanilla cluster")
]
@ -85,7 +110,7 @@ HdpGroup = [
default='hdp',
help="plugin name in overview"),
cfg.StrOpt('hadoop_version',
default='1.3.0',
default='1.3.2',
help="hadoop version for plugin"),
cfg.ListOpt('processes',
default=
@ -95,7 +120,7 @@ HdpGroup = [
"NAGIOS_SERVER": 11},
help='numbers of processes for hdp in savannabashboard'),
cfg.StrOpt('base_image',
default='latest-ci-image',
default='ib-centos-6-4-64-hdp-13',
help="image name for start hdp cluster")
]

4
savannadashboard/tests/node_group_template_tests/negative_tests.py
View File

@ -21,7 +21,7 @@ import savannadashboard.tests.configs.config as cfg
class UINegativeCreateNodeGroupTemplate(base.UITestCase):
@base.attr(tags='node_group_template')
@base.attr(tags=['node_group_template', 'vanilla'])
@testtools.skipIf(cfg.vanilla.skip_plugin_tests,
'tests for vanilla plugin skipped')
def test_create_vanilla_node_group_template_with_wrong_parameters(self):
@ -38,7 +38,7 @@ class UINegativeCreateNodeGroupTemplate(base.UITestCase):
'MapReduce Parameters:io.sort.mb:'
'Enter a whole number.')
@base.attr(tags='node_group_template')
@base.attr(tags=['node_group_template', 'vanilla'])
@testtools.skipIf(cfg.vanilla.skip_plugin_tests,
'tests for vanilla plugin skipped')
def test_create_vanilla_node_group_template_with_missing_parameters(self):

4
savannadashboard/tests/node_group_template_tests/test_create_node_group_template.py
View File

@ -21,7 +21,7 @@ import savannadashboard.tests.configs.config as cfg
class UICreateNodeGroupTemplate(base.UITestCase):
@base.attr('node_group_template', 'vanilla')
@base.attr(tags=['node_group_template', 'vanilla'])
@testtools.skipIf(cfg.vanilla.skip_plugin_tests,
'tests for vanilla plugin skipped')
def test_create_node_group_template_vanilla(self):
@ -44,7 +44,7 @@ class UICreateNodeGroupTemplate(base.UITestCase):
message=msg)
self.delete_node_group_templates(['selenium-vanilla'])
@base.attr('node_group_template', 'hdp')
@base.attr(tags=['node_group_template', 'hdp'])
@testtools.skipIf(cfg.hdp.skip_plugin_tests,
'tests for hdp plugin skipped')
def test_create_node_group_template_hdp(self):

82
savannadashboard/tests/resources/README.rst Normal file
View File

@ -0,0 +1,82 @@
README for resources
=====================================
Resources in this directory using for check EDP for Savanna.
For this purpose the cluster with oozie by process is created.
With these resources created and launched jobs.
Success of performance of jobs is checked.
Pig job
-------------------------------------
Description.
------------
Resources 'edp-job.pig' and 'edp-lib.jar' used for create oozie job
that deletes all symbols in line after ":".
Example:
--------
input file:
"""
qweqwe
qweqweqwe:
qweqweqwe:qwe
:ertertert
asd
"""
output file:
"""
qweqwe
qweqweqwe
asd
"""
Sources.
--------
Link for 'edp-job.pig':
https://github.com/apache/oozie/blob/branch-4.0/examples/src/main/apps/pig/id.pig
Source for 'edp-lib.jar':
https://github.com/apache/oozie/blob/branch-4.0/examples/src/main/java/org/apache/oozie/example/DateList.java
Jar job
-------------------------------------
Description.
------------
Resource 'edp-job.jar' used for create oozie job
which counts the characters in file and displays values ​​at end of each line.
Example:
--------
input file:
"""
qweqwe
qweqweqwe:
qweqweqwe:qwe
:ertertert
asd
"""
output file:
"""
qweqwe 6
qweqweqwe: 16
qweqweqwe:qwe 29
:ertertert 39
asd 42
"""
Sources.
--------
Sources for 'edp-job.jar':
https://github.com/apache/oozie/blob/branch-4.0/examples/src/main/java/org/apache/oozie/example/SampleMapper.java
https://github.com/apache/oozie/blob/branch-4.0/examples/src/main/java/org/apache/oozie/example/SampleReducer.java

BIN
savannadashboard/tests/resources/edp-job.jar Normal file
View File

Binary file not shown.

3
savannadashboard/tests/resources/edp-job.pig Normal file
View File

@ -0,0 +1,3 @@
A = load '$INPUT' using PigStorage(':') as (fruit: chararray);
B = foreach A generate com.hadoopbook.pig.Trim(fruit);
store B into '$OUTPUT' USING PigStorage();

BIN
savannadashboard/tests/resources/edp-lib.jar Normal file
View File

Binary file not shown.

1
test-requirements.txt
View File

@ -8,6 +8,7 @@ coverage>=3.6
mock>=1.0
nose
openstack.nose_plugin>=0.7
python-swiftclient>=1.5
pylint==0.25.2
unittest2
selenium

1
tox.ini
View File

@ -15,7 +15,6 @@ setenv =
deps =
-r{toxinidir}/requirements.txt
-r{toxinidir}/test-requirements.txt
commands = nosetests {posargs}
[testenv:tests]
sitepackages = False