diff --git a/doc/source/index.rst b/doc/source/index.rst index e70dd95d59..ca54b128c5 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -92,8 +92,8 @@ the following topic guides. topics/workflows topics/tables topics/policy - topics/testing topics/angularjs + topics/testing topics/javascript_testing topics/styling topics/translation diff --git a/doc/source/topics/javascript_testing.rst b/doc/source/topics/javascript_testing.rst index 1fd702d34a..71b8a35767 100644 --- a/doc/source/topics/javascript_testing.rst +++ b/doc/source/topics/javascript_testing.rst @@ -39,6 +39,16 @@ Tests can be run in two ways: The code linting job can be run with ``tox -e npm -- lint``, or ``tox -e npm -- lintq`` to show errors, but not warnings. +To decipher where tests are failing it may be useful to use Jasmine in the +browser to run individual tests to see where the tests are specifically breaking. +To do this, navigate to your local horizon in the browser and add +'/jasmine' to the end of the url. e.g: 'http://localhost:8000/jasmine'. Once you +have the jasmine report you may click on the title of an individual test to +re-run just that test. From here, you can also use chrome dev tools or similar +to set breakpoints in the code by accessing the 'Sources' tab and clicking on +lines of code where you wish to break the code. This will then show you the exact +places where the code breaks. + Coverage Reports ---------------- @@ -47,9 +57,8 @@ developing, be sure to check the coverage reports on the master branch and compare your development branch; this will help identify missing tests. To generate coverage reports, run ``tox -e npm``. The coverage reports can be -found at ``horizon/coverage-karma/`` (framework tests) and -``openstack_dashboard/coverage-karma/`` (dashboard tests). Load -``/index.html`` in a browser to view the reports. +found at ``cover/horizon/`` (framework tests) and ``cover/openstack_dashboard/`` +(dashboard tests). Load ``/index.html`` in a browser to view the reports. Writing Tests ============= diff --git a/doc/source/topics/testing.rst b/doc/source/topics/testing.rst index 45153fb2d9..46e96beade 100644 --- a/doc/source/topics/testing.rst +++ b/doc/source/topics/testing.rst @@ -1,6 +1,6 @@ -=================== -Testing Topic Guide -=================== +================ +Testing Overview +================ Having good tests in place is absolutely critical for ensuring a stable, maintainable codebase. Hopefully that doesn't need any more explanation. @@ -13,6 +13,11 @@ If you already know everything about testing but are fed up with trying to debug why a specific test failed, you can skip the intro and jump straight to :ref:`debugging_unit_tests`. +.. toctree:: + :maxdepth: 1 + + Angular specific testing + An overview of testing ====================== @@ -222,6 +227,23 @@ Tips and tricks + }, 'django.db.backends': { +Coverage reports +---------------- + +It is possible for tests to fail on your patch due to the npm-run-test not +passing the minimum threshold. This is not necessarily related directly to the +functions in the patch that have failed, but more that there are not enough tests +across horizon that are related to your patch. + +The coverage reports may be found in the 'cover' directory. There's a subdirectory +for horizon and openstack_dashboard, and then under a directory for the browser +used to run the tests you should find an ``index.html``. This can then be viewed +to see the coverage details. + +In this scenario you may need to submit a secondary patch to address test coverage +for another function within horizon to ensure tests rise above the coverage +threshold and your original patch can pass the necessary tests. + Common pitfalls --------------- @@ -292,421 +314,3 @@ result of an error in the conditions of the test. Using the :meth:`~horizon.test.helpers.TestCase.assertMessageCount` will make it readily apparent what the problem is in the majority of cases. If not, then use ``pdb`` and start interrupting the code flow to see where things are getting off track. - -Integration tests in Horizon -============================ - -The integration tests currently live in the Horizon repository, see `here`_, -which also contains instructions on how to run the tests. To make integration -tests more understandable and maintainable, the Page Object pattern is used -throughout them. - -.. warning:: To enable integration tests support before running them, please - copy openstack_dashboard/local/local_settings.d/_20_integration_tests_scaffolds.py.example - to openstack_dashboard/local/local_settings.d/_20_integration_tests_scaffolds.py - and then run ./manage.py collectstatic --clear && ./manage.py compress. - -Horizon repository also provides two shell `scripts`_, which are executed in -pre_test_hook and post_test_hook respectively. Pre hook is generally used for -modifying test environment, while post hook is used for running actual -integration tests with tox and collecting test artifacts. Thanks to the -incorporating all modifications to tests into Horizon repository, one can alter -both tests and test environment and see the immediate results in Jenkins job -output. - -.. _here: https://github.com/openstack/horizon/tree/master/openstack_dashboard/test/integration_tests -.. _scripts: https://github.com/openstack/horizon/tree/master/tools/gate/integration - -Page Object pattern -------------------- - -Within any web application's user interface (UI) there are areas that the tests -interact with. A Page Object simply models these as objects within the test -code. This reduces the amount of duplicated code; if the UI changes, the fix -needs only be applied in one place. - -Page Objects can be thought of as facing in two directions simultaneously. -Facing towards the developer of a test, they represent the services offered by -a particular page. Facing away from the developer, they should be the only -thing that has a deep knowledge of the structure of the HTML of a page (or -part of a page). It is simplest to think of the methods on a Page Object as -offering the "services" that a page offers rather than exposing the details -and mechanics of the page. As an example, think of the inbox of any web-based -email system. Amongst the services that it offers are typically the ability to -compose a new email, to choose to read a single email, and to list the subject -lines of the emails in the inbox. How these are implemented should not matter -to the test. - -Writing reusable and maintainable Page Objects ----------------------------------------------- - -Because the main idea is to encourage the developer of a test to try and think -about the services that they are interacting with rather than the -implementation, Page Objects should seldom expose the underlying WebDriver -instance. To facilitate this, methods on the Page Object should return other -Page Objects. This means that we can effectively model the user's journey -through the application. - -Another important thing to mention is that a Page Object need not represent an -entire page. It may represent a section that appears many times within a site -or page, such as site navigation. The essential principle is that there is -only one place in your test suite with knowledge of the structure of the HTML -of a particular (part of a) page. With this in mind, a test developer builds -up regions that become reusable components (`example of a base form`_). These -properties can then be redefined or overridden (e.g. selectors) in the actual -pages (subclasses) (`example of a tabbed form`_). - -The page objects are read-only and define the read-only and clickable elements -of a page, which work to shield the tests. For instance, from the test -perspective, if "Logout" used to be a link but suddenly becomes an option in a -drop-down menu, there are no changes (in the test itself) because it still simply -calls the "click_on_logout" action method. - -This approach has two main aspects: - -* The classes with the actual tests should be as readable as possible -* The other parts of the testing framework should be as much about data as - possible, so that if the CSS etc. changes you only need to change that one - property. If the flow changes, only the action method should need to change. - -There is little that is Selenium-specific in the Pages, except for the -properties. There is little coupling between the tests and the pages. Writing -the tests becomes like writing out a list of steps (by using the previously -mentioned action methods). One of the key points, particularly important for -this kind of UI driven testing is to isolate the tests from what is behind -them. - -.. _example of a base form: https://github.com/openstack/horizon/blob/8.0.0/openstack_dashboard/test/integration_tests/regions/forms.py#L250 -.. _example of a tabbed form: https://github.com/openstack/horizon/blob/8.0.0/openstack_dashboard/test/integration_tests/regions/forms.py#L322 - -List of references ------------------- - -* https://wiki.openstack.org/wiki/Horizon/Testing/UI#Page_Object_Pattern_.28Selected_Approach.29 -* https://wiki.mozilla.org/QA/Execution/Web_Testing/Docs/Automation/StyleGuide#Page_Objects -* https://code.google.com/p/selenium/wiki/PageObjects - -Debugging integration tests -=========================== - -Even perfectly designed Page Objects are not a guarantee that your integration -test will not ever fail. This can happen due to different causes: - -The first and most anticipated kind of failure is the inability to perform a -testing scenario by a living person simply because some OpenStack service or -Horizon itself prevents them from doing so. This is exactly the kind that -integration tests are designed to catch. Let us call them "good" failures. - -All other kinds of failures are unwanted and could be roughly split into the -two following categories: - -#. The failures that occur due to changes in application's DOM. some CSS/ Xpath selectors no longer matching - Horizon app's DOM. The usual signature for that kind of failures is having - a DOM changing patch for which the test job fails with a message like - this `selenium.common.exceptions.NoSuchElementException: Message: Unable to - locate element: {"method":"css selector","selector":"div.modal-dialog"}`. - If you find yourself in such a situation, you should fix the Page Object - selectors according to the DOM changes you made. - -#. Unfortunately it is still quite possible to get the above error for a patch - which didn't implement any DOM changes. Among the reasons of such behavior - observed in past were: - - * Integration tests relying on relative ordering of form fields and table - actions that broke with the addition of a new field. This issue should - be fixed by now, but may reappear in future for different entities. - - * Integration tests relying on popups disappearing by the time a specific - action needs to be taken (or not existing at all). This expectation - turned out to be very fragile, since the speed of tests execution by - Jenkins workers may change independently of integration test code (hence, - popups disappear too late to free the way for the next action). The - unexpected (both too long and too short) timeouts aren't limited to just - popups, but apply to every situation when the element state transition - is not instant (like opening an external link, going to another page in - Horizon, waiting for button to become active, waiting for a table row to - change its state). Luckily, most transitions of "element becomes visible/ - emerge to existence from non-existence" kind are already bulletproofed - using `implicit_wait` parameter in `integration_tests/horizon.conf` file. - Selenium just waits for specified amount of seconds for an element to - become visible (if it's not already visible) giving up when it exceeds - (with the above error). Also it's worth mentioning `explicit_wait` parameter - which is considered when the selenium `wait_until` method is involved (and - it is used, e.g. in waiting for spinner and messages popups to disappear). - -An inconvenient thing about reading test results in the `console.html` file -attached to every `gate-horizon-dsvm-integration` finished job is that the test -failure may appear either as failure (assertion failed), or as error (expected -element didn't show up). In both cases an inquirer should suspect a legitimate -failure first (i.e., treat errors as failures). Unfortunately, no clear method -exists for the separation of "good" from "bad" failures. Each case is -unique and full of mysteries. - -The Horizon testing mechanism tries to alleviate this ambiguity by providing -several facilities to aid in failure investigation: - -* First there comes a screenshot made for every failed test (in a separate - folder, on a same level as `console.html`) - almost instant snapshot of a - screen on the moment of failure (*almost* sometimes matters, especially in - a case of popups that hang on a screen for a limited time); -* Then the patient inquirer may skim through the vast innards of - `console.html`, looking at browser log first (all javascript and css errors - should come there), -* Then looking at a full textual snapshot of a page for which test failed - (sometimes it gives a more precise picture than a screenshot), -* And finally looking at test error stacktrace (most useful) and a lengthy - output of requests/ responses with a selenium server. The last log sometimes - might tell us how long a specific web element was polled before failing (in - case of `implicit_wait` there should be a series of requests to the same - element). - -The best way to solve the cause of test failure is running and debugging the -troublesome test locally. You could use `pdb` or Python IDE of your choice to -stop test execution in arbitrary points and examining various Page Objects -attributes to understand what they missed. Looking at the real page structure -in browser developer tools also could explain why the test fails. Sometimes it -may be worth to place breakpoints in JavaScript code (provided that static is -served uncompressed) to examine the objects of interest. If it takes long, you -may also want to increase the webdriver's timeout so it will not close browser -windows forcefully. Finally, sometimes it may make sense to examine the -contents of `logs` directory, especially apache logs - but that is mostly the -case for the "good" failures. - -Writing your first integration test -=================================== - -So, you are going to write your first integration test and looking for some -guidelines on how to do it. The first and the most comprehensive source of -knowledge is the existing codebase of integration tests. Look how other tests -are written, which Page Objects they use and learn by copying. Accurate imitation -will eventually lead to a solid understanding. Yet there are few things that may -save you some time when you know them in advance. - -File and directory layout and go_to_*page() methods ---------------------------------------------------- -Below is the filesystem structure that test helpers rely on.:: - - horizon/ - └─ openstack_dashboard/ - └─ test/ - └─ integration_tests/ - ├─ pages/ - │ ├─ admin/ - │ │ ├─ __init__.py - │ │ └─ system/ - │ │ ├─ __init__.py - │ │ └─ flavorspage.py - │ ├─ project/ - │ │ └─ compute/ - │ │ ├─ __init__.py - │ │ ├─ access_and_security/ - │ │ │ ├─ __init__.py - │ │ │ └─ keypairspage.py - │ │ └─ imagespage.py - │ └─ navigation.py - ├─ regions/ - ├─ tests/ - ├─ config.py - └─ horizon.conf - -New tests are put into integration_tests/tests, where they are grouped -by the kind of entities being tested (test_instances.py, test_networks.py, etc). -All Page Objects to be used by tests are inside pages/directory, the nested -directory structure you see within it obeys the value of `Navigation.CORE_PAGE_STRUCTURE` -you can find at pages/navigation.py module. The contents of the `CORE_PAGE_STRUCTURE` -variable should in turn mirror the structure of standard dashboard sidebar menu. -If this condition is not met, the go_to_page() methods which are generated -automatically at runtime will have problems matching the real sidebar items. How are -these go_to_*page() methods are generated? From the sidebar's point of view, dashboard -content could be at most four levels deep: Dashboard, Panel Group, Panel and Tab. -Given the mixture of these entities in existing dashboards, it was decided that: - -* When panels need to be addressed with go_to_page() methods, two components in - the method's name are enough for distinguishing the right path to go along, namely a Panel - name and a Panel Group name (or a Dashboard name, if no Panel Group exists above Panel). - For example, - - * `go_to_system_flavorspage()` method to go to Admin->System->Flavors and - - * `go_to_identity_projectspage()` method to go to Identity->Projects panel. - -* When we need to go one level deeper, i.e. go to the specific TableTab on any panel that - has several tabs, three components are enough - Panel Group, Panel and Tab names. For - example, `go_to_compute_accessandsecurity_floatingipspage()` for navigating to - Project->Compute->Access & Security->Floating IPs tab. Note that one cannot navigate - to a Panel level if that Panel has several tabs (i.e., only terminal levels could be - navigated to). - -As you might have noticed, method name components are chosen from normalized items of -the `CORE_PAGE_STRUCTURE` dictionary, where normalization means replacing spaces with `_` -symbol and `&` symbol with `and`, then downcasing all symbols. - -Once the `go_to_*page()` method's name is parsed and the proper menu item is matched in -a dashboard, it should return the proper Page Object. For that to happen a properly -named class should reside in a properly named module located in the right place of the -filesystem. More specifically and top down: - -#. Page Object class is located in: - - * //page.py file for non-tabbed pages - - * ///page.py file for tabbed pages - Values , , and are the normalized versions of - the items from the `CORE_PAGE_STRUCTURE` dictionary. - -#. Within the above module a descendant of `basepage.BaseNavigationPage` should be - defined, its name should have the form Page or Page, where and - are capitalized versions of normalized and items respectively. - -Reusable regions ----------------- - -* `TableRegion` binds to the HTML Horizon table using the `TableRegion`'s `name` - attribute. To bind to the proper table this attribute has to be the same as - the `name` attribute of a `Meta` subclass of a corresponding `tables.DataTable` - descendant in the Python code. `TableRegion` provides all the needed facilities for - solving the following table-related tasks. - - * Getting a specific row from a table matched by the column name and a target - text within that column (use `get_row()` method) or taking all the existing - rows on a current table page with `rows` property. - * Once you have a reference to a specific row, it can either be marked with - `mark()` for further batch actions or split to cells (using `cells` property - which is dictionary representing column name as a key to cell wrapper as a - value). - - * For interacting with actions `TableRegion` provides 2 decorators, namely - `@bind_table_action()` and `@bind_row_action()` which bind to the actual HTML - button widget and decorate the specific table methods. These methods in turn - should click a bound button (comes as these methods' second argument after `self`) - and usually return a new region which is most often bound to a modal form - being shown after clicking that button in real Horizon. - - * Another important part of `TableRegion` are the facilities for checking the - properties of a paged table - `assert_definition()`, `is_next_link_available()` - and `is_prev_link_available()` helpers and `turn_next_page()` / `turn_prev_page()` - which obviously cause the next / prev table page to be shown. - -* when interacting with modal and non-modal forms three flavors of form wrappers - can be used. - - * `BaseFormRegion` is used for simplest forms which are usually 'Submit' / - 'Cancel' dialogs with no fields to be filled. - - * `FormRegion` is the most used wrapper which provides interaction with the - fields within that form. Every field is backed by its own wrapper class, while - the `FormRegion` acts as a container which initializes all the field wrappers in - its `__init__()` method. Field mappings passed to `__init__()` could be - - * either a tuple of string labels, in that case the same label is used for - referencing the field in test code and for binding to the HTML input (should be - the same as `name` attribute of that widget, could be seen in Django code defining - that form in Horizon) - - * or a dictionary, where the key will be used for referencing the test field - and the value will be used for binding to the HTML input. Also it is feasible - to provide values other than strings in that dictionary - in this case they are - meant to be a Python class. This Python class will be initialized as any - BaseRegion is usually initialized and then the value's key will be used for - referencing this object. This is useful when dealing with non-standard widgets - in forms (like Membership widget in Create/​Edit Project form or Networks widget - in Launch Instance form). - - * `TabbedFormRegion` is a slight variation of `FormRegion`, it has several tabs - and thus can accept a tuple of tuples / dictionaries of field mappings, where - every tuple corresponds to a tab of a real form, binding order is that first - tuple binds to leftmost tab, which has index 0. Passing `default_tab` other than - 0 to `TabbedFormRegion.__init__` we can make the test form to be created with - the tab other than the leftmost being shown immediately. Finally the method `switch_to` - allows us to switch to any existing form's tab. - -* `MessageRegion` is a small region, but is very important for asserting that - everything goes well in Horizon under test. Technically, the `find_message_and_dismiss` - method belongs to `BasePage` class, but whenever it is called, `regions.messages` - module is imported as well to pass a `messages.SUCCESS` / `messages.ERROR` - argument into. The method returns `True` / `False` depending on if the specified - message was found and dismissed (which could be then asserted for). - -Customizing tests to a specific gate environment ------------------------------------------------- - -* Upstream gate environment is not the only possible environment where Horizon - integration tests can be run. Various downstream distributions may also - want to run them. To ease the adoption of upstream tests to possibly - different conditions of a downstream gate environment, integration tests use - a configuration machinery backed by oslo.config library. It includes the - following pieces of knowledge: - - * integration_tests/config.py file where all possible setting groups and - settings are defined along with their descriptions and defaults. If you are - going to add a new setting to Horizon integration tests, you should add it - first to this file. - - * integration_tests/horizon.conf file - where all the overrides are - actually located. For clarity its contents mirrors the default values - in config.py (although technically they could be completely commented out - with the same result). - - * To make developers' lives easier a local-only (not tracked by git) - counterpart of horizon.conf could exist at the same directory, named - 'local-horizon.conf'. It is meant solely for overriding values from - horizon.conf that a developer's environment might differ from the gate - environment (like Horizon url or admin user password). - -* When integration tests are run by openstack-infra/devstack-gate scripts they - use 2 hooks to alter the devstack gate environment, namely pre_test_hook and - post_test_hook. Contents of both hooks are defined inside the corresponding - shell scripts located at 'tools/gate/integration' at the top-level of horizon - repo. If you find yourself asking which of the hooks you need to modify - pre - or post, keep the following things in mind. - - * Pre hook is executed before the Devstack is deployed, that essentially - means that almost none of packages that are installed as OpenStack services - dependencies during Devstack deployment are going to be present in the - system. Yet all the repositories contained with `PROJECTS` variable defined - in `devstack-vm-gate-wrap.sh`_ script will be already cloned by the moment - pre hook is executed. So the natural use for it is to customize some Horizon - settings before they are used in operations like compressing statics etc. - That is how it is actually used now: it sets settings variable - `INTEGRATION_TESTS_SUPPORT` to `True`, so all the support code for integration - tests is included into Horizon static assets. If this variable was set to - `False`, we would not be able to run integration tests. - - * Post hook is executed after Devstack is deployed, so integration tests - themselves are run inside that hook, as well as various test artifacts - collection. When you modify it, do not forget to save the exit code of - a tox integration tests run and emit at the end of the script - or you may - lose the SUCCESS/FAILURE status of the whole tests suite and tamper with the - job results! - -.. _devstack-vm-gate-wrap.sh: https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate-wrap.sh - - -Writing integration tests for Horizon plugins ---------------------------------------------- - -First, for more details on writing a Horizon plugin please refer to -:doc:`../tutorials/plugin`. Second, there are 2 possible setups when running -integration tests for Horizon plugins. - -The first setup, which is suggested to be used in gate of \*-dashboard plugins -is to get horizon as a dependency of a plugin and then run integration tests -using horizon.conf config file inside the plugin repo. This way the plugin augments -the location of Horizon built-in Page Objects with the location of its own -Page Objects, contained within the `plugin_page_path` option and the Horizon -built-in nav structure with its own nav structure contained within -`plugin_page_structure`. Then the plugin integration tests are run against core -Horizon augmented with just this particular plugin content. - -The second setup may be used when it is needed to run integration tests for -Horizon + several plugins. In other words, content from several plugins is -merged into core Horizon content, then the combined integration tests from core -Horizon and all the involved plugins are run against the resulting dashboards. -To make this possible both options `plugin_page_path` and -`plugin_page_structure` have MultiStrOpt type. This means that they may be -defined several times and all the specified values will be gathered in a list, -which is iterated over when running integration tests. In this setup it's easier to -run the tests from Horizon repo, using the horizon.conf file within it. - -Also keep in mind that `plugin_page_structure` needs to be a strict JSON -string, w/o trailing commas etc.