46 Commits

Author SHA1 Message Date
Jenkins
e5f1a0ecfa Merge "Bring in a newer optional eventlet" 2014-10-18 19:38:32 +00:00
Joshua Harlow
27badfc314 Avoid usage of six.moves in local functions
Currently it appears that using six.moves in threaded code isn't
working as expected (something there in six does not appear to
be thread safe) so until this is fixed avoid using those moves
in functions in the examples and in the utility code (and instead
import the moved function at the top of the module in code to
avoid any threaded usage problems).

Upstream bug filed at:

https://bitbucket.org/gutworth/six/issue/98/

Fixes bug 1377514

Change-Id: I3fc1819df8fb42d0c3d394bbc7d047b09152af68
2014-10-04 21:41:05 -07:00
Jenkins
748a901501 Merge "Adjust docs+venv tox environments requirements/dependencies" 2014-09-28 05:42:41 +00:00
Joshua Harlow
8178e7811c Adjust docs+venv tox environments requirements/dependencies
After a prior merge these sections are no longer valid and
need to be adjusted to reflect the removal of the optional
requirements file.

Change-Id: I2a93c8e55f0d692df6d074bd86ab1a5e6d11a03f
2014-09-27 22:03:30 +00:00
Joshua Harlow
ce620c399a Use oslotest to provide our base test case class
The oslotest library has a nice openstack testing integrated
base class that can ensure we setup our base test case using
the right logging fixtures, test timeouts, and output fixtures
that better operate in the openstack ecosystem.

This will also allow us to remove some of the functionality
that we currently have in our base test case and replace it with
the equivalent (or better) functionality that oslotest now
provides.

Part of blueprint use-oslo-test

Change-Id: I1602d5180ec8649a1899185972750ddddf65990f
2014-09-27 14:53:12 -07:00
Joshua Harlow
7640b09250 Bring in a newer optional eventlet
Since the openstack requirements repo just accepted eventlet>=0.15.1
we might as well also use that version in our optional requirements
and remove a piece of code that was dealing with a bug that was fixed
in eventlet 0.15.

Change-Id: I9b4f9061c7adb7d72315315f41bb0d742b6f56b5
2014-09-27 14:50:44 -07:00
Joshua Harlow
1e216c61f1 Move some of the custom requirements out of tox.ini
This change removes the need for an optional-requirements
file which isn't being kept up to date with the rest of the
openstack ecosystem and moves most of the customizations to
test-requirements and where different versions are still
needed (aka for SQLAlchemy) we place those varations into
there needed tox environment as required.

Change-Id: I443794b83de3f6a196fa7fc29a90620fb51b7f4c
2014-09-26 12:44:05 -07:00
Jenkins
2e96516722 Merge "Add a docs virtualenv" 2014-09-21 13:36:25 +00:00
Joshua Harlow
b84b76c557 Work toward Python 3.4 support and testing
Change-Id: Icd0a426e3dd8a44c64e8ba337b36131392715c08
2014-09-12 00:05:34 -07:00
Joshua Harlow
6ffb74c7a6 Add a docs virtualenv
The various projects are standardizing on a docs venv which
will eventually be called by the infra project to build the
docs for a project.

To enable our usage of this environment add a section so that
it can be called upon.

Change-Id: I420eb0d6e6f9f6f24bc493d4342478daba27ad1e
2014-09-09 17:41:59 +00:00
Joshua Harlow
3465e0340b Reduce unused tox environments
Since the openstack-ci only tests 5 of the many tox enviroments
we had listed and from my knowledge nobody else is testing using
those other non-standard enviroments just reduce the set of applicable
environments to the ones that are being used (which also means we
can remove the usage of toxgen and just stick with a simpler file
that is easier to use, modify and adjust).

Change-Id: I4d2302594b9f9e8741f9693cb358efc1418bd45d
2014-09-09 10:36:55 -07:00
Joshua Harlow
5f0948bd12 Split requirements into py2 and py3 files
Since PBR now supports distinguishing between
a requirements file for py2 and a different requirements
file for py3 we should add the support for that separation
so that it is easier to package taskflow.

Change-Id: I3c5faa99a529bcb28aa0276b8d345a26f94b5af3
2014-08-15 12:18:26 -07:00
Christian Berendt
6132647fb7 Enabled hacking checks H305 and H307
* H305  imports not grouped correctly
* H307  like imports should be grouped together

Change-Id: If1dd9c89f65ede6959865a885777cb08c263eca0
2014-07-17 09:05:48 +02:00
Jenkins
71fbe6e76a Merge "Remove check_doc.py and use doc8" 2014-07-06 23:36:10 +00:00
Joshua Harlow
510635a913 Remove check_doc.py and use doc8
Instead of using a helper script that is internal to taskflow
use the doc8 package (which was created based on the check_doc.py
work) instead. This avoids repeating duplicated code when a
stackforge/external pypi package can provide the same
functionality.

Change-Id: Ie5a43f96b20e34f3955657ad2ef8beba05a4300e
2014-06-18 23:59:53 -07:00
Joshua Harlow
d5b835efb8 Fix E265 hacking warnings
Adjust the example code to comply with the new E265
hacking check. This check warns about the following:

* E265 block comment should start with '# '

Change-Id: I77aaf8c0bbc50bad9646f8192b1e1b80cf0afc8c
2014-06-16 19:02:28 -07:00
Joshua Harlow
c558da07b6 Upgrade hacking version and fix some of the issues
Update hacking to the new requirements version and
fix about half of the new reported issues. The other
hacking issues are for now ignored until fixed by
adjusting our tox.ini file.

This commit fixes the following new hacking errors:

H405 - multi line docstring summary not separated
       with an empty line
E265 - block comment should start with '# '
F402 - import 'endpoint' from line 21 shadowed by
       loop variable

Change-Id: I6bae61591fb988cc17fa79e21cb5f1508d22781c
2014-06-13 19:27:17 -07:00
Ivan A. Melnikov
f986a82731 Check documentation for simple style requirements
What is checked:
 - lines should not be longer than 79 characters, also
   there are few exceptions;
 - no tabulation for indentation;
 - no trailing whitespace.

Change-Id: Id5b5dff380460d1fb4ffeeb23f00864e7fc158ed
2014-05-08 12:11:18 +04:00
Ivan A. Melnikov
d9e2e202dd SQLAlchemy requirements put in order
- require SQLAlchemy>=0.7.8 if no stricter requirement present;
- allow SQLAlhchemy 0.9.x in python 3.3 and pypy test environments.

Change-Id: I8ee29c66612e8051a26a1633c0973d0dfa048fc4
2014-03-26 14:35:22 +04:00
Ivan A. Melnikov
b58aae0e9f Add tox environment for pypy
Because some of our dependencies (notably psycopg2) are not supported
by pypy we have to add special tox evnironment explicitly.

Partial-bug: 1293585
Change-Id: I6e3fd38e05dc5a388894687b51234934cae6a6b7
2014-03-17 17:46:12 +04:00
Ivan A. Melnikov
3ece5c7a9c docs: Add page describing atom arguments and results
Task Arguments and Results page from wiki converted to sphinx/rst
and edited to improve readability and make it look better. Some
examples are converted to doctest.

We also now check doctests from docs for correctness when unit
tests are run by tox.

Partially implements blueprint sphinx-docs.

Change-Id: Ief398856e1ce2d0dc1d43c4e4112872a27672fcf
2014-03-13 22:07:59 +04:00
Stanislav Kudriashev
32e8c3da61 Message-oriented worker-based flow with kombu
* Implemented Worker to be started on remote host for
  handling tasks request.
* Implemented WorkerTaskExecutor that proxies tasks
  requests to remote workers.
* Implemented Proxy that is used for consuming and
  publishing messages by Worker and Executor.
* Added worker-based engine and worker task executor.
* Added kombu dependency to requirements.
* Added worker-based flow example.
* Added unit-tests for worker-based flow components.

Implements: blueprint worker-based-engine
Change-Id: I8c6859ba4a1a56c2592e3d67cdfb8968b13ee99c
2014-02-19 14:47:34 +02:00
Stanislav Kudriashev
5acb0832df Fix command for pylint tox env
Change-Id: Ied77136bd839b5f36e4e281672a6fbfdd5edf934
2014-02-17 14:41:37 +02:00
Ivan A. Melnikov
0de3ffa891 Reduce test and optional requirements to global requirements
Currently SQLAlchemy 0.9 and pyMySQL are not available on OpenStack
PyPi mirror, which means we can't run tests with them at gating.

Related-bug: 1280008
Change-Id: I23a8b7ab31d7fb9f6ae7454c242a2ba854484cea
2014-02-15 22:25:26 +04:00
Jeremy Stanley
7f109cf517 Remove tox locale overrides
* tox.ini: The LANG, LANGUAGE and LC_ALL environment overrides were
introduced originally during the testr migration in an attempt to be
conservative about the possibility that locale settings in the
calling environment could cause consistency problems for test runs.
In actuality, this should be unnecessary and any place where it does
cause issues ought to be considered an actual bug. Also, having
these in the configuration actively causes older pip to have
problems with non-ASCII content in some package metadata files under
Python 3, so drop it now.

Change-Id: Iad912181bc00827908887d45a448f559d5083194
Closes-Bug: #1277495
2014-02-10 03:12:27 +00:00
Ivan A. Melnikov
6beebc2584 Add optional-requirements.txt
While 'optional requirements' phrase is somewhat self-contradicting,
that's what we really have: requirements of optional (pluggable) parts
of TaskFlow are optional in the sense that one can use TaskFlow
without them.

This change puts all such requirements into optional-requirements.txt to
have all them documented in one place, uses that file to list deps for
default python 2.7 test environment.

Change-Id: Ibeeb6ac68247697790cbc2a2c24d2e9f59a4479c
2014-02-04 13:16:32 +04:00
Ivan A. Melnikov
5ee077976e Move kazoo to testenv requirements
Kazoo is an optional dependency of taskflow: it is required when
zookeeper persistence layer or if kazoo_utils are used. So we should
move it to where these type of optional requirements are currently
placed.

Change-Id: I9d1605af266f4a79fd6ebfee2d68a4f4f7413022
2014-02-04 13:16:19 +04:00
Jenkins
67d8f88cd2 Merge "Test using mysql + postgres if available" 2014-01-27 09:52:45 +00:00
skudriashev
aea1f401eb Doc strings and comments clean-up
* Added missing period for doc strings
* Correct syntax errors
* Remove H402 from flake8 ignore list

Change-Id: Ia8592bf99378e3658d6cca2ceb148bf9eb0b5de8
2014-01-26 23:08:39 +02:00
Joshua Harlow
c1b9729db1 Test using mysql + postgres if available
If mysql is avaiable via the standard testing user
'openstack_citest' with password 'openstack_citest'
then attempt to setup mysql or postgres and use
those databases for the duration of the test.

Adjust the tox template to include the two main mysql
library variants [MySQL-python, PyMySQL] and the psycopg2
postgres library so that these new tests will find the
appropriate library (sqlalchemy will attempt to find
these when connecting).

Change-Id: I90e59504663277d277d94e1f2a11031119bcedd6
2014-01-25 22:56:36 -08:00
Ivan A. Melnikov
40cba47709 Put full set of requirements to py26, py27 and py33 envs
pyXY envs are considered to be default, so they must have richest set
of test requirements.

Change-Id: Ide2a8ab0500c0c6769ad7b8f0cc76f229a83db60
2014-01-23 16:24:29 +04:00
Ivan A. Melnikov
0bd6270363 Add more requirements to cover tox environment
To see how code is really covered we should run as much tests as
possible in cover environment, so we should use dependencies from
the most reach environment. Currently it's py27-sa9-ev.

Change-Id: I7af092f4789b167dad1fba20b5b267f99a741b0e
2014-01-21 13:33:21 +04:00
Ivan A. Melnikov
291e085225 Put SQLAlchemy requirements on single line
Pip seems to have 'one name, one line' requirement for requirements,
and does not merge them, raising error instead.

Closes-bug: #1271093
Change-Id: I9e573a6a4de844992eb477393df867a734cbb536
2014-01-21 13:31:02 +04:00
Joshua Harlow
28fa164fcd Use the released toxgen tool instead of our copy
Update the docs to explain how to use the toxgen.py
released on pypi and use it to regenerate the tox.ini
file.

Change-Id: I9e0d28f0e40aa0280132a644260d028190b10974
2014-01-12 08:01:50 -08:00
Joshua Harlow
3d437df749 Add envs for different sqlalchemy versions
Adjust tests to skip the sqlalchemy test if
sqlalchemy is not installed. Adjust examples
to fallback to a directory based backend if
the sqlalchemy does not load or is not available.

Include a updated tox.ini (generated from the
toxgen.py script) that includes the new venv
variations.

Change-Id: I7686f09901a9b65d7c81b4e037b5bffc24aa7ef7
2014-01-09 11:09:08 -08:00
Ivan A. Melnikov
e5d31b2b3b Enable H202 warning for flake8
Change-Id: Iaeb6d4ab1af5063558ef1a7b9a85e46c6cf0ccb7
2014-01-02 19:24:57 +02:00
Ivan A. Melnikov
63f89bc916 Coverage calculation improvements
- add .coveragerc to configure coverage calculations and omissions;
- add python 2.x specific test requirements to coverage tox env;
- update .gitignore.

Change-Id: Idc35e4f18ec297506576f09e468ab34e8704c0b1
2013-11-22 11:25:03 +04:00
Joshua Harlow
db15db8186 Fix up python 3.3 incompatabilities
Make the python 3.3 testing work by selectively
disabling & including eventlet, switch to testtools
and testrepository which has 2.6, 2.7, 3.2+ unified
testing support so that we can correctly run our
tests in all supported python versions.

Closes-Bug: #1251660
Co-authored-by: Alexander Gorodnev <agorodnev@griddynamics.com>
Change-Id: I23b6f04387cfd3bf6b5a044edffa446ca897ce3a
2013-11-21 18:38:44 +04:00
Joshua Harlow
b1f81badfd Bump requirements to the latest
Match the requirements for taskflow with
the openstack requirements (note the upstream
requirements are missing 'futures' and 'networkx')
and update the new hacking violations that were
detected due to the hacking requirement version
bump.

Change-Id: I8d1326cf2a8b1ea062f5e9aacd0c4f8261c6531a
2013-10-05 21:49:08 +00:00
Monty Taylor
9b7d463f68 Remove d2to dependency
Change-Id: I433711ba417dbb15f616fa488cec389f3138b00a
2013-09-30 10:39:23 -04:00
Ivan A. Melnikov
57c4b1bdc2 Initial implementation of action-based engine
This is first in series of changes that provides separation of flows into
engines and pattens. Patterns define structure, while engine define how
the task should be run according to said structure.

This change adds the basic patterns and a very simple engine that
is able to run flow defined with patterns by converting it to recursive
structure of so-called 'actions'.

For simplicity and ease of review certain major features like
resumption, saving and passing on task results and notifications are
left out for farther changes.

Partially implements blueprint patterns-and-engines

Co-authored-by: Anastasia Karpinska <akarpinska at griddynamics.com>
Change-Id: I68515d8a5b30d5d047bf9beb67cc3e2111175190
2013-09-03 14:46:07 -07:00
Joshua Harlow
324ea453ae Adjust a bunch of hacking violations.
- Fix new hacking violations!
- Add more comments!

Change-Id: Ida0c1b6c98a8f45a15985b33529a1f479e595db5
2013-08-01 18:46:21 -07:00
Angus Salkeld
a7482eaa18 Fix most of the hacking rules
Hacking rules: H302, H303, H304, H401, H404

Change-Id: I38e62696724a99c5ebe74d95d477999bd91a2c9a
2013-06-26 11:04:42 +10:00
Angus Salkeld
6b17ba8ab7 Fix all flake8 E* and F* errors
This was mostly unused modules or local variables.

Change-Id: I39f9598ba8b64101a10f80bae448d27272abeddd
2013-06-26 11:04:30 +10:00
Angus Salkeld
339e43d03b Fix the current flake8 errors
The test pip ordering was resulting in the pep8 rules not really
running so this fixes that up and the resulting errors so we can
get a clean run.

Change-Id: Icc27be05cd513c4f04135c7e768007dbf9d07b10
2013-06-26 11:03:23 +10:00
Joshua Harlow
6ddf738bbb Dinky change to trigger jenkins so I can cleanup.
Change-Id: I5ff975bb5ded48a84341efa4ebfdae4bf430ab60
2013-06-19 13:20:31 -07:00