16 Commits

Author SHA1 Message Date
Slawek Kaplonski
d1d3ea7c19 Configure limit of opened files for the test runner process
For some reason, when Tobiko retries many times ssh connection
and logs a lot of data in the log file, pytest may fail with
"Too many open files" exception raised.
To avoid that lets try to configure soft ulimit for opened files
to be as high as possible (same as hard limit).

Change-Id: I6fb8977d7909798a5ab6be499044cc74ac34dfa2
2022-08-19 16:54:04 +02:00
Federico Ressi
2ac4100069 Update run_tests.py environment variable names
Change-Id: I5f6ad7369f10df04f02aa96b22ab8236a0e060ef
2022-01-07 14:06:45 +00:00
Federico Ressi
5776ac688c Set default test path for the run_tests script
Change-Id: Id001383f26233c6df1b2fc3b7d125d0b491e0e9a
2021-11-03 11:49:11 +00:00
Federico Ressi
03853a7593 Increase pytest command verbosity
This should help investigating on tests timeout sissues.

Change-Id: I91b16f5775c8ebd5f03763e8848de7efd886d7a5
2021-09-27 09:33:43 +02:00
Federico Ressi
7207175342 Update pytest reports generation
- Set workflow step name as junit test suite name
- Set HTML report title
- Render HTML report all test case rows collapsed
- Add tobiko release version and Git commit to
  reported environment metadata

Change-Id: Id827146d1f8455047dd5eedfa90171da8be02ea5
2021-02-04 15:07:23 +00:00
Federico Ressi
190451d1ed Use pytest-cov plugin for getting coverage
Change-Id: Ifb5db0e8b1e12cf7d14045c2ccc7fc5ebfeaa40b
2021-01-27 21:11:21 +00:00
Federico Ressi
db64a23d0d Use pytest timeout and log capturing
Change-Id: I4d3394274c410ff8c424fe166a1657b2540f47fd
2021-01-27 21:10:27 +00:00
Federico Ressi
4d88f7d81a Update requirements
Requires docker==4.4.1 since version 4.4.0 can't be installed
Rerun tests on integration failures
Remove redundant requirements

Co-Author: pini <pkomarov@redhat.com>
Change-Id: I0cd54b6a9361e8e3c3205da2ba939baba2c6b1ce
2021-01-04 15:20:06 +01:00
Federico Ressi
3ce3c741e5 Switch to pytest test runner
Change-Id: I8132e07ad624fe9faa01265d700b5eb37b94815f
2020-12-18 13:51:08 +00:00
Federico Ressi
71e79acb59 Fix testenv use of posargs in tox.ini
Change-Id: I0699a92d84cf67c42ccd695ade98e1a985039b58
2020-10-21 06:20:56 +02:00
Federico Ressi
3df2bb87f1 Add TOX_PYDEV_DEBUG env variable to run_tests.py
Add new env variable to handle test cases
execution when debugging test cases under a
debugger by using testools.run runner.

Change-Id: Ia6ec38e6471d93dd98c3fef17e66e9187c666c9d
2020-10-13 15:53:43 +02:00
Federico Ressi
31a9a6e8e6 Add a timeout for workflow stages
Change-Id: I9938a8bbd71ab131c8b6a2210e5d1a903ef21efd
2020-09-28 12:04:49 +02:00
Federico Ressi
7d20c11226 Skip functional test cases requiring Keystone credentials
Change-Id: Ib2143b49f8028ec6e4476dece8f8072cd9690967
2020-09-23 12:03:27 +00:00
Federico Ressi
c50ab412a0 Create Docker files for Tobiko project
- Dockerfile file is intended for running test cases
  without using tox
- infrared_plugin/Dockerfile file is intended to run
  Tobiko IR plugin
- docker-compose file is intended to emulate the tox
  experience by using docker

All images are build staring from official CentOS 8
image.

Change-Id: I2f57f6e1c9c6161cef9448d25027d784ebd37e2c
2020-06-01 10:19:21 +02:00
Alex Katz
32c91c7264 Added --ignore-test-failures CLI flag
Tox venv commands are expected to return a code that indicates
test cases final outcome:

 - 0 => any test vase has failed
 - 1 => some test case has failed
 - not in [0, 1] => any other error

This change handles the special case when some test cases have
failed. In any of below cases:
 - JENKINS_URL environment variable is defined on controller node
 - --ignore-test-failure flag is passed to IR plugin
then the plugin runs without treating it as an error.

This has been introduced because when running the plugin from
a Jenkins slave host, Jenkins itself is expected to check test
case have failed looking at produced Junit XML artifacts.

Change-Id: Ia7d7f736ae18a7561dced2271228d86ca658e892
2020-05-11 13:57:26 +00:00
Federico Ressi
92248c8506 Refactor test execution scripts
- Acreate a new tools/run_tests.py script to
  execute test cases
- Move report generation to such script and
  remove 'report' tox environment
- Running test cases generates report files
  (.log, .html, .xml) into 'report/{envname}'
  directory
- Test cases failures will returns exit code 1
  Other script test runner failures returns
  exit code 2

Big refactory to tox.ini file to try semplifying it:

- create the new 'py3' tox environment for running unit
  tests as default platform python 3 interpreter
- use same as py3 '{envdir}' for scenario, functional,
  neutron and faults tox environments

Change-Id: Id09425245cc86b84b41e6b3b1c1db759cc686f3a
2020-05-06 17:41:35 +02:00