KeyValueAction previously allowed only one key=>value pair to be
supplied in the CLI per argument.
While the operator could supply multiple key=>value pairs by repeatedly
using the same argument it was unnecessarily verbose and tedious approach.
Similar behavior is implemented in the tripleoclient validator CLI
by the resulution to the rhbz#1959492 [0].
Tests were adjusted.
[0]https://bugzilla.redhat.com/show_bug.cgi?id=1959492:
Signed-off-by: Jiri Podivin <jpodivin@redhat.com>
Change-Id: I28c40bb8156d73dd95af9641acaab3cce721be2d
Python 2 is no longer supported, thus usage of six can be removed.
This change also upates setup.cfg to enforce usage of Python 3.
Change-Id: I0be22a80afbe9f16191384466fe2ba496a7ef63e
Introduces two new exceptions for Validation Framework
to be raised within appropriate context.
ValidationRunException - signifying incorrect behavior of the Run action code
ValidationShowException - signifying incorrect behavior of the Show actions code
Signed-off-by: Jiri Podivin <jpodivin@redhat.com>
Change-Id: I87f315e7bb2e28a8572982a71fbe2be4432ff2c0
Unused workdir argument of the run_validations method removed.
Host and playbook skipping simplified.
Docstrings adjusted, including some of the more important
inline code examples.
Test sensitivity to API changes was increased.
More strict assertions have been put on the calls made in CLI.
Logging statements related to the run action
in validation_actions.py were expanded.
New debug level logging statements introduced.
One of the testing constants in the fakes.py was altered
as it was not accurate representation of the real data,
and as such was breaking the new logging statements
during unit test runs.
Deprecation notice for 'log_path' and 'groups' arguments,
in both docstring and logging.
Redundant conditionals for argument types were removed.
The checks of the arguments are implemented
in the function called in the subsequent statement.
As there is no scenario in which one set of these checks
passes, while the other fails with the same values,
we can consider one of them to be redundant.
Helper function 'get_validations_details' without active references
was removed from utils submodule.
Signed-off-by: Jiri Podivin <jpodivin@redhat.com>
Change-Id: I781f6a6f9fc4bd558af56b648f0e0ee9f165dfab
As a result of a minor oversight, the test execution could
initiate a successful call to the 'open' and write in the storage.
This patch addresses the issue.
Signed-off-by: Jiri Podivin <jpodivin@redhat.com>
Change-Id: Iceb6df712b2ac0dc56aa86f465fd39cf308de2d7
The skip list feature was available only via the API, this patch
allow the user to pass a file with the list of the validations
which he wants to skip during the run, example:
check-ram: hosts: all
reason: Wrong ram value
lp: https://lp.fake.net
check-cpu: hosts: undercloud
reason: Unstable validation
lp: https://lp.fake.net
Change-Id: I04059f9339085e0dcef1f018cad1be511ee7d3c7
Introduce validation config file.
The config default location of the config file will be stored
in /etc/validation.cfg
The variables precedence will be the following:
* user's cli args
* config file
* default interval values
Change-Id: I05c54a43bc0a03878793cca3f51e23f4a8b63a23
- redundant imports in test modules were removed
- reset color code management in test simplified
Signed-off-by: Jiri Podivin <jpodivin@redhat.com>
Change-Id: Iac5e68286782c5c2fbc185650c0bdf915fe959ef
This patch adds the management of the new `products` metadata key in the
validation playbooks. We can now filter the validations by their groups,
by their categories or by their products while listing or running them.
The `list` sub command has now a new --product argument. When filtering
by groups, by categories or by products (see the example below), the
`list` sub command will return all the validation playbooks belonging to
the 'prep' group OR all the validation playbooks belonging to the 'os'
and/or 'system' categories OR all the validation playbooks to the
'tripleo' product:
$ validation list -h
$ validation list --group prep --category os,system --product tripleo
The `run` sub command has also its new --product argument. Note that
this new argument is mutually exclusive with the --validation, --group
and --category arguments:
$ validation run -h
$ validation run --product tripleo --inventory /etc/ansible/hosts
The `show parameter` sub command has the same new argument which is also
mutually exclusive with the --validation, --group and the --category
arguments:
$ validation show parameter -h
$ validation show parameter --product tripleo
Change-Id: Ief13e506c2bee18da47b31f1c2d5c0dbb1ad1ecf
Signed-off-by: Gael Chamoulaud (Strider) <gchamoul@redhat.com>
This patch adds the management of the new `categories` metadata key in
the validation playbooks. We can now filter the validations by their
groups and/or by their categories while listing or running them.
The `list` sub command has now a new --category argument. When filtering
by groups and by categories (see the example below), the `list` sub
command will return all the validation playbooks belonging to the prep
group OR all the validation playbooks belonging to the os and/or system
categories:
$ validation list -h
$ validation list --group prep --category os,system
The `run` sub command has also its new --category argument. Note that
this new argument is mutually exclusive with the --validation and
--group arguments:
$ validation run -h
$ validation run --category networking --inventory /etc/ansible/hosts
The `show parameter` sub command has the same new argument which is also
mutually exclusive with the --validation and --group arguments:
$ validation show parameter -h
$ validation show parameter --category os,system,hardware,ram
Change-Id: I4297f83355bdd209d21518fbadb17d1343fd4680
Signed-off-by: Gael Chamoulaud (Strider) <gchamoul@redhat.com>
The existing tests were using shared fakes as return value
of certain mocks. Unfortunately, while perfectly fine in most cases,
certain methods change values passed in place.
This created a race condition, which caused fluctuation in the reported
coverage of the relevant modules.
The undesirable test behavior only becomes apparent when the tests
are run in a succession and their behavior is analyzed, for example
during the run of the recently implemented 'coverchange' job.
Final statistics of the coverage measurement, displaying the totals
for number of lines, lines without coverage, branches etc., display
following pattern. Note the third column.
TOTAL 995 105 340 56 86%
TOTAL 995 105 340 56 86%
TOTAL 995 105 340 56 86%
TOTAL 995 108 340 56 86%
TOTAL 995 108 340 56 86%
TOTAL 995 105 340 56 86%
TOTAL 995 105 340 56 86%
TOTAL 995 108 340 56 86%
TOTAL 995 105 340 56 86%
Number of lines without coverage oscillates between '108' and '105' depending
on the order in which the tests were run. The culprit being one of the functions
of the 'validations_libs.cli.common' module, which edits data passed in place.
As the data processed are passed from patched object, and the data itself are
defined withing the relevant 'fakes' module, it emerges with inevitability,
that the tested code operates on data previously altered by the tests.
Furthermore the 'test_run_command_failed_validation' test was augmented
to place further assertion on the arguments passed.
Closes-Bug: #1935003
Signed-off-by: Jiri Podivin <jpodivin@redhat.com>
Change-Id: I2210fa1775691dd503eb882a24824eaf18d7bd3e
As we now have a mechanism for falling back to
a default directory for logs and artifacts, we can allow
the customers to specify their own.
This patch add the argument to the CLI along with
minimum necessary information about the usage.
Signed-off-by: Jiri Podivin <jpodivin@redhat.com>
Change-Id: Id31a7ce2dfc4d9a132fea7d600e8beb8adc0739b
(cherry picked from commit 7876cab41586157a1b5e8fd426deb16d79567a52)
Paths created trough formatted string included double '/'.
While still parsable, and as far as I know posix valid,
it is nonetheless a bit confusing.
This change merely removes one of the '/',
resulting in path that is just as valid, just easier on the eye.
Tests were adjusted to reflect the change.
Signed-off-by: Jiri Podivin <jpodivin@redhat.com>
Change-Id: Ic6bf6d0a02bead00c0e3c5a858476cf5f7c0d2a7
This patch adds the --python-interpreter argument to the run
subcommand and allows the user to specify their own python interpreter.
Change-Id: Ie76d0238f117a48b2f919cd5bdc4dfae8b47ac5e
Signed-off-by: Gael Chamoulaud (Strider) <gchamoul@redhat.com>
This patch disables the Ansible execution quiet mode when running
validations with a different Callback by passing --extra-env-vars
ANSIBLE_STDOUT_CALLBACK=default in the command line.
Change-Id: Ied50aaea240943b1eeba0c3c35b74bc7ccdb454c
Signed-off-by: Gael Chamoulaud (Strider) <gchamoul@redhat.com>
If results is empty in cli/run.py then we should raise
a RuntimeError otherwise, it will returns 0 without any
information.
Change-Id: Ic38690bd719658fe02a30691d112f3429523802a
This review adds a CLI for the Validation Framework which will become
the only entry point to run the Validation Framework.
It will deprecate the tripleo_validatior.py CLI and the
validation.py script.
This patch uses python cliff library to provide nice helpers, shell
and output formats.
Change-Id: I66800ad51cc50f4eb37efabe85fb553dce008101
