From 3ec7f0bf55e0bebbe5333b0496a819c320650699 Mon Sep 17 00:00:00 2001 From: Timur Sufiev Date: Mon, 25 Jan 2016 17:50:30 +0300 Subject: [PATCH] More documentation on writing i9n tests List of topics: * go_to_page() methods and proper file structure is covered * regions facilities * customizing gate environment * running tests in plugins Change-Id: I885705b62e63b72c5ba823a8ebf7be20f9fcddbe --- doc/source/topics/testing.rst | 235 +++++++++++++++++++++++++++++++++- 1 file changed, 234 insertions(+), 1 deletion(-) diff --git a/doc/source/topics/testing.rst b/doc/source/topics/testing.rst index 1c97f5d279..fb0d891cd1 100644 --- a/doc/source/topics/testing.rst +++ b/doc/source/topics/testing.rst @@ -465,4 +465,237 @@ served uncompressed) to examine the objects of interest. If it takes long, you may also want to increase the webdriver's timeout so it will not close browser windows forcefully. Finally, sometimes it may make sense to examine the contents of `logs` directory, especially apache logs - but that is mostly the -case for the "good" failures. \ No newline at end of file +case for the "good" failures. + +Writing your first integration test +=================================== + +So, you are going to write your first integration test and looking for some +guidelines on how to do it. The first and the most comprehensive source of +knowledge is the existing codebase of integration tests. Look how other tests +are written, which Page Objects they use and learn by copying. Accurate imitation +will eventually lead to a solid understanding. Yet there are few things that may +save you some time when you know them in advance. + +File and directory layout and go_to_*page() methods +--------------------------------------------------- +Below is the filesystem structure that test helpers rely on.:: + + horizon/ + └─ openstack_dashboard/ + └─ test/ + └─ integration_tests/ + ├─ pages/ + │ ├─ admin/ + │ │ ├─ __init__.py + │ │ └─ system/ + │ │ ├─ __init__.py + │ │ └─ flavorspage.py + │ ├─ project/ + │ │ └─ compute/ + │ │ ├─ __init__.py + │ │ ├─ access_and_security/ + │ │ │ ├─ __init__.py + │ │ │ └─ keypairspage.py + │ │ └─ imagespage.py + │ └─ navigation.py + ├─ regions/ + ├─ tests/ + ├─ config.py + └─ horizon.conf + +New tests are put into integration_tests/tests, where they are grouped +by the kind of entities being tested (test_instances.py, test_networks.py, etc). +All Page Objects to be used by tests are inside pages/directory, the nested +directory structure you see within it obeys the value of `Navigation.CORE_PAGE_STRUCTURE` +you can find at pages/navigation.py module. The contents of the `CORE_PAGE_STRUCTURE` +variable should in turn mirror the structure of standard dashboard sidebar menu. +If this condition is not met, the go_to_page() methods which are generated +automatically at runtime will have problems matching the real sidebar items. How are +these go_to_*page() methods are generated? From the sidebar's point of view, dashboard +content could be at most four levels deep: Dashboard, Panel Group, Panel and Tab. +Given the mixture of these entities in existing dashboards, it was decided that: + +* When panels need to be addressed with go_to_page() methods, two components in + the method's name are enough for distinguishing the right path to go along, namely a Panel + name and a Panel Group name (or a Dashboard name, if no Panel Group exists above Panel). + For example, + + * `go_to_system_flavorspage()` method to go to Admin->System->Flavors and + + * `go_to_identity_projectspage()` method to go to Identity->Projects panel. + +* When we need to go one level deeper, i.e. go to the specific TableTab on any panel that + has several tabs, three components are enough - Panel Group, Panel and Tab names. For + example, `go_to_compute_accessandsecurity_floatingipspage()` for navigating to + Project->Compute->Access & Security->Floating IPs tab. Note that one cannot navigate + to a Panel level if that Panel has several tabs (i.e., only terminal levels could be + navigated to). + +As you might have noticed, method name components are choosen from normalized items of +the `CORE_PAGE_STRUCTURE` dictionary, where normalization means replacing spaces with `_` +symbol and `&` symbol with `and`, then downcasing all symbols. + +Once the `go_to_*page()` method's name is parsed and the proper menu item is matched in +a dashboard, it should return the proper Page Object. For that to happen a properly +named class should reside in a properly named module located in the right place of the +filesystem. More specifically and top down: + +#. Page Object class is located in: + + * //page.py file for non-tabbed pages + + * ///page.py file for tabbed pages + Values , , and are the normalized versions of + the items from the `CORE_PAGE_STRUCTURE` dictionary. + +#. Within the above module a descendant of `basepage.BaseNavigationPage` should be + defined, its name should have the form Page or Page, where and + are capitalized versions of normalized and items respectively. + +Reusable regions +---------------- + +* `TableRegion` binds to the HTML Horizon table using the `TableRegion`'s `name` + attribute. To bind to the proper table this attribute has to be the same as + the `name` attribute of a `Meta` subclass of a corresponding `tables.DataTable` + descendant in the Python code. `TableRegion` provides all the needed facilities for + solving the following table-related tasks. + + * Getting a specific row from a table matched by the column name and a target + text within that column (use `get_row()` method) or taking all the existing + rows on a current table page with `rows` property. + * Once you have a reference to a specific row, it can either be marked with + `mark()` for further batch actions or split to cells (using `cells` property + which is dictionary representing column name as a key to cell wrapper as a + value). + + * For interacting with actions `TableRegion` provides 2 decorators, namely + `@bind_table_action()` and `@bind_row_action()` which bind to the actual HTML + button widget and decorate the specific table methods. These methods in turn + should click a bound button (comes as these methods' second argument after `self`) + and usually return a new region which is most often bound to a modal form + being shown after clicking that button in real Horizon. + + * Another important part of `TableRegion` are the facilities for checking the + properties of a paged table - `assert_definition()`, `is_next_link_available()` + and `is_prev_link_available()` helpers and `turn_next_page()` / `turn_prev_page()` + which obviously cause the next / prev table page to be shown. + +* when interacting with modal and non-modal forms three flavors of form wrappers + can be used. + + * `BaseFormRegion` is used for simplest forms which are usually 'Submit' / + 'Cancel' dialogs with no fields to be filled. + + * `FormRegion` is the most used wrapper which provides interaction with the + fields within that form. Every field is backed by its own wrapper class, while + the `FormRegion` acts as a container which initializes all the field wrappers in + its `__init__()` method. Field mappings passed to `__init__()` could be + + * either a tuple of string labels, in that case the same label is used for + referencing the field in test code and for binding to the HTML input (should be + the same as `name` attribute of that widget, could be seen in Django code defining + that form in Horizon) + + * or a dictionary, where the key will be used for referencing the test field + and the value will be used for binding to the HTML input. Also it is feasible + to provide values other than strings in that dictionary - in this case they are + meant to be a Python class. This Python class will be initialized as any + BaseRegion is usually initialized and then the value's key will be used for + referencing this object. This is useful when dealing with non-standard widgets + in forms (like Membership widget in Create/​Edit Project form or Networks widget + in Launch Instance form). + + * `TabbedFormRegion` is a slight variation of `FormRegion`, it has several tabs + and thus can accept a tuple of tuples / dictionaries of field mappings, where + every tuple corresponds to a tab of a real form, binding order is that first + tuple binds to leftmost tab, which has index 0. Passing `default_tab` other than + 0 to `TabbedFormRegion.__init__` we can make the test form to be created with + the tab other than the leftmost being shown immediately. Finally the method `switch_to` + allows us to switch to any existing form's tab. + +* `MessageRegion` is a small region, but is very important for asserting that + everything goes well in Horizon under test. Technically, the `find_message_and_dismiss` + method belongs to `BasePage` class, but whenever it is called, `regions.messages` + module is imported as well to pass a `messages.SUCCESS` / `messages.ERROR` + argument into. The method returns `True` / `False` depending on if the specified + message was found and dismissed (which could be then asserted for). + +Customizing tests to a specific gate environment +------------------------------------------------ + +* Upstream gate environment is not the only possible environment where Horizon + integration tests can be run. Various downstream distributions may also + want to run them. To ease the adoption of upstream tests to possibly + different conditions of a downstream gate environment, integration tests use + a configuration machinery backed by oslo.config library. It includes the + following pieces of knowledge: + + * integration_tests/config.py file where all possible setting groups and + settings are defined along with their descriptions and defaults. If you are + going to add a new setting to Horizon integration tests, you should add it + first to this file. + + * integration_tests/horizon.conf file - where all the overrides are + actually located. For clarity its contents mirrors the default values + in config.py (although technically they could be completely commented out + with the same result). + + * To make developers' lives easier a local-only (not tracked by git) + counterpart of horizon.conf could exist at the same directory, named + 'local-horizon.conf'. It is meant solely for overriding values from + horizon.conf that a developer's environment might differ from the gate + environment (like Horizon url or admin user password). + +* When integration tests are run by openstack-infra/devstack-gate scripts they + use 2 hooks to alter the devstack gate environment, namely pre_test_hook and + post_test_hook. Contents of both hooks are defined inside the corresponding + shell scripts located at 'tools/gate/integration' at the top-level of horizon + repo. If you find yourself asking which of the hooks you need to modify - pre + or post, keep the following things in mind. + + * Pre hook is executed before the Devstack is deployed, that essentially + means that almost none of packages that are installed as OpenStack services + dependencies during Devstack deployment are going to be present in the + system. Yet all the repositories contained with `PROJECTS` variable defined + in `devstack-vm-gate-wrap.sh`_ script will be already cloned by the moment + pre hook is executed. So the natural use for it is to customize some Horizon + settings before they are used in operations like compressing statics etc. + + * Post hook is executed after Devstack is deployed, so integration tests + themselves are run inside that hook, as well as various test artifacts + collection. When you modify it, do not forget to save the exit code of + a tox integration tests run and emit at the end of the script - or you may + lose the SUCCESS/FAILURE status of the whole tests suite and tamper with the + job results! + +.. _devstack-vm-gate-wrap.sh: https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate-wrap.sh + + +Writing integration tests for Horizon plugins +--------------------------------------------- + +There are 2 possible setups when running integration tests for Horizon plugins. + +The first setup, which is suggested to be used in gate of *-dashboard plugins +is to get horizon as a dependency of a plugin and then run integration tests +using horizon.conf config file inside the plugin repo. This way the plugin augments +the location of Horizon built-in Page Objects with the location of its own +Page Objects, contained within the `plugin_page_path` option and the Horizon +built-in nav structure with its own nav structure contained within +`plugin_page_structure`. Then the plugin integration tests are run against core +Horizon augmented with just this particular plugin content. + +The second setup may be used when it is needed to run integration tests for +Horizon + several plugins. In other words, content from several plugins is +merged into core Horizon content, then the combined integration tests from core +Horizon and all the involved plugins are run against the resulting dashboards. +To make this possible both options `plugin_page_path` and +`plugin_page_structure` have MultiStrOpt type. This means that they may be +defined several times and all the specified values will be gathered in a list, +which is iterated over when running integration tests. In this setup it's easier to +run the tests from Horizon repo, using the horizon.conf file within it. + +Also keep in mind that `plugin_page_structure` needs to be a strict JSON +string, w/o trailing commas etc.