173 Commits

Author SHA1 Message Date
Sergey Vilgelm
889e66eb49 Switch to the oslo_utils.fileutils
fileutils is graduated in the oslo.utils library.
Remove the openstack.common package as well.

Implements: blueprint graduate-fileutils[1]
[1] https://blueprints.launchpad.net/oslo-incubator/+spec/graduate-fileutils

Depends-On: I51ba9076e1fbc16145ee2311f47b7768c16dcb20 (requirements)
Depends-On: I661dd222da6386a7dbcf854958a63e59b13e9ba4 (oslo.utils)

Change-Id: I6ba8f492a257096d27cbe83bfd06e53b7aabfa5d
2015-07-19 23:46:28 +03:00
Robert Collins
a1e28bd54a Remove test-requirements-py3.txt.
This will unblock requirements merges.

Change-Id: I318750ec9fe471d3df57ad2f5785026dae451242
2015-06-30 23:30:16 +12:00
gordon chung
965aca8a37 remove unused notifier
there's a notifier in our code. it doesn't seem to be connected to
anything and is not accessible via setup.cfg. this appears to be
residual workaround from pre-oslo.messaging days

Change-Id: I50c8e97219da387362aef9d1ef4c53aea0c27142
2015-06-24 22:43:35 -04:00
ZhiQiang Fan
c2446a7c60 add oslo.service options
patch https://review.openstack.org/#/c/194772 try to add oslo.service
options but oslo.service has defined its options into three modules,
so the right definition should be oslo.service.service which contains
eventlet back door option.

Change-Id: I1b3efe9a2e93f8548ab0d52ca10b2acf0f07d511
ref: https://github.com/openstack/oslo.service/blob/0.1.0/setup.cfg#L33
2015-06-24 00:26:00 -07:00
Julien Danjou
c2b843036d Fix oslo.service configuration options building
Change-Id: I829b57d92df49692b56ec7a80d7406d06ffab844
2015-06-23 20:18:07 +02:00
Jenkins
17646c25b0 Merge "pip has its own download cache by default" 2015-06-22 14:22:14 +00:00
Victor Stinner
5087b255bb Port test_complex_query to Python 3
* query.py: the message attribute of Exception was removed in Python 3,
  replace e.message with e to format the new error message
* Fix test_complex_query_scenarios: convert
  sample_item["metadata"]["util"] to float; "0.5" >= 0.5 is True in
  Python 2, but it raises a TypeError on Python 3.
* tox.ini: add the following API v2 tests to Python 3.4:

  - test_complex_query_scenarios.TestQueryMetersController.test_query_with_volume_field_name_orderby

Change-Id: I112e5564a65db005cf66ce967a8ac3e52fa95267
2015-06-20 09:07:30 +02:00
Victor Stinner
54b114c428 Fix expected error message on Python 3
* Use repr() at runtime instead of using the hardcoded result:
  repr(u'abc') returns 'abc' on Python 3, not u'abc'.
* Add the following api.v2 test to tox.ini for Python 3.4

  - test_alarm_scenarios.TestAlarms.test_get_alarm_history_constrained_by_alarm_id_failed

Change-Id: Ied224b194a73e14c2c04ff3d9837964420ce766c
2015-06-20 09:07:30 +02:00
Victor Stinner
86ffa59c73 Fix usage of iterator/list on Python 3
* CombinationEvaluator: convert states (built using zip) to a list
  because states is consumed more than once. On Python 3, zip() returns
  an iterator which can only be consumed once.
* Replace dict.keys()[0] with list(dict.keys())[0]. On Python 3, keys()
  returns an iterator which is not indexable.
* test_event_scenarios: replace filter() with a list-comprehension, the
  test expects a list whereas filter() returns on iterator on Python 3.
* mongo/utils.py: replace obj.keys()[0] with list(obj.keys())[0],
  and replace obj.values()[0] with list(obj.values())[0]
* Add the following API v2 tests to tox.ini in Python 3.4:

  - test_event_scenarios.TestEventAPI.test_get_events_filter_datetime_trait

Change-Id: I28bcb65940edbc226c8dbb272f4e3c040ebc37a1
2015-06-20 09:06:52 +02:00
Jenkins
a16acb1e01 Merge "Fix unicode/bytes issues in API v2 tests" 2015-06-19 13:36:11 +00:00
Chris Dent
5f1f33cbc9 pip has its own download cache by default
We no longer need tox to tell us to use one.

Change-Id: Ib988348fe19fccf8862597e8d831d282462be105
2015-06-19 12:19:54 +00:00
Victor Stinner
14766ba719 Fix unicode/bytes issues in API v2 tests
* HTTP body type is bytes: fix tests to use bytes strings.
* test_list_resources_scenarios: on Python 3, base64.encodestring() returns
  bytes whereas the test expects bytes; decode the result from ASCII.
* tox.ini: add the following API v2 tests to Python 3.4:

  - test_alarm_scenarios.TestAlarms.test_alarms_query_with_timestamp
  - test_complex_query_scenarios.TestQueryMetersController.test_query_with_isotime
  - test_list_resources_scenarios.TestListResources.test_with_invalid_resource_id

Change-Id: I8258c2d17fbc1236f4bcda40bff93a93bddf13e2
2015-06-16 01:20:51 +02:00
Ildiko Vancsa
d496bb8ac8 Fix script name in tox.ini for Elasticsearch
Change-Id: Ifc8c2cc8a4964a3d5b00c6107f1ce9c5d6e8518a
2015-06-15 18:16:13 +02:00
Victor Stinner
852396e48d Fix publisher test_udp on Python 3
* Call msgpack.loads() with an encoding to get unicode strings, not
  byte strings
* Sort counters using their name because a counter is a dictionary
  and dictionaries are not comparable in Python 3.
* Add test_udp to tox.ini for Python 3.4.

Change-Id: Ifa20eecbc1c293da667f58843b7a6c2d7f32152c
2015-06-12 18:11:42 +02:00
Victor Stinner
f945f9b797 Fix Ceph object store tests on Python 3
* Fix keystoneclient mock on Python 3: service_catalog.url_for() must
  return a string because urllib.parse.urljoin() is now more strict on
  Python 3
* tox -e py34: add objectstore.test_rgw

Change-Id: I0a4a33a26b7984854c8561866a6f49e2b4d7ff6d
2015-06-12 18:11:42 +02:00
Victor Stinner
4a392ca17d Port IPMI to Python 3
binascii.hexlify() returns a byte string whereas the code expects a
native string, so Unicode on Python 3. On Python 3, decode the string
from ASCII to get Unicode.

Enable ipmi tests in tox.ini on Python 3.4.

Change-Id: I154723ad42efb61a06b71600975374441920a852
2015-06-12 18:11:42 +02:00
Victor Stinner
5977295daf Port middleware to Python 3
* Fix integer division: status_code/100 => status_code//100
* HTTP body type is bytes. Decode the HTTP body from UTF-8 to use it.
  Encode XML and JSON to UTF-8 to produce the HTTP body (for error
  messages).
* Factorize XML/JSON code a little bit more (ex: add the new
  content_type variable)
* Fix test_app on Python 3: decode HTTP from UTF-8 to load JSON
* tox.ini: add api.v2.test_app to Python 3.4

Change-Id: I4f916cba36306f776b01df915c55e314ade6b6ba
2015-06-12 18:11:42 +02:00
Julien Danjou
191f7bf9cc Remove deprecated Swift middleware
Change-Id: Ifd1861e3df46fad0e44ff9b5cbd58711bbc87c97
2015-06-09 18:05:19 +02:00
Jenkins
89089c2a70 Merge "Fix more tests on Python 3" 2015-06-09 14:05:11 +00:00
Victor Stinner
b47c2cd85e Fix more tests on Python 3
* Fix test_get_connection: max_bytes and backup_count parameters of the
  logging module must be int, not None.
* Fix test_query on Python 3: ast.literal_eval("1+1") now returns 2 on
  Python 3, whereas it raises a ValueError on Python 2. Use a string
  invalid in Python 2 and Python 3 based on the original bug report
  #1221736
* test_swift: fix url_for() mock, it now returns a string for the
  endpoint
* storage.test_get_connection: skip test_three_urls_no_default()
  if the happybase module is missing (it's not compatible with Python 3
  yet)

Enable more tests on Python 3.4:

*  ceilometer.tests.api.v2.test_query
*  ceilometer.tests.dispatcher.test_db
*  ceilometer.tests.dispatcher.test_file
*  ceilometer.tests.dispatcher.test_http
*  ceilometer.tests.objectstore.test_swift
*  ceilometer.tests.storage.test_get_connection

Change-Id: I1d9bdd73bd21ec5dc8ee9823352f6dfe818b8d6b
2015-06-09 11:27:53 +02:00
Ilya Tyaptin
57a9fc5bce Add running functional scripts for defined backend
This CR allows running "functional" tox job for different backends.
Short description about expected workflow:
1. Gate job run ceilometer/ceilometer/tests/functional/hooks/post_test_hook.py
with <backend> parameter.
2. This script run "tox -efunctional" command with defined variable
CEILOMETER_TEST_BACKEND
3. After in "run-functional-tests.sh" setup script for the backend
and testr are runned.

In this CR running testr with pretty_tox.sh script added because
it allows to use subunit-trace output which developed in tempest-lib
and improve useful of testr output.

Partially implements: blueprint ceilometer-functional-tests
Change-Id: Idb66aca0b46779516db2baec856df8223dbe5c13
2015-06-04 16:29:02 +03:00
Jenkins
e71e391a4f Merge "use oslo.log instead of oslo-incubator code" 2015-05-28 21:36:32 +00:00
ZhiQiang Fan
c612727ff8 use oslo.log instead of oslo-incubator code
oslo.log has graduated from oslo-incubator, and openstack.common.log
has been removed from oslo-incubator, so let's use the new one.

NOTE1:openstack.common.log registers its options at import time, but
oslo.log needs to call register_options() explicitly.

NOTE2: split unit test case in tests/objectstore/test_swift_middleware.py
to avoid duplicate cli option register exception, see NOTE1.

Change-Id: Ida30808dbe0c584919755c207ca4ee4b91963a17
2015-05-28 11:08:02 -04:00
Jenkins
68558869a6 Merge "Port test_inspector to Python 3" 2015-05-28 09:31:07 +00:00
Jenkins
b02d6835a6 Merge "Fix usage of dictionary methods on Python 3" 2015-05-28 09:29:03 +00:00
Victor Stinner
ce08e620fa Port test_inspector to Python 3
* Replace contextlib.nested() with contextlib.ExitStack
* Add contextlib2 dependency for Python 2.6 and 2.7 to get ExitStack
* TestLibvirtInspection: set libvirtError because it must be a subclass
  of Exception on Python 3

Enable ceilometer.tests.compute.virt.libvirt.test_inspector on Python 3.

Change-Id: I82f4b911c1b3ede90805150630c222b1dd5f3474
2015-05-28 09:37:22 +02:00
Victor Stinner
bfaefc9492 Fix usage of dictionary methods on Python 3
Convert the result of keys(), values() and items() dict method to list
when a list is expected. On Python 3, these methods now return an
iterator, not a list.

Instead of getting the key with .keys()[0] and then the value with
.values()[0], get both at the same time using .items()[0]. It ensures
that the key is consistent with the value, but it is also more
efficient.

Enable ceilometer.tests.storage.test_impl_sqlalchemy on Python 3.4.

Change-Id: I4bcd3dfbca3e26238cdcdf19801244497dac2729
2015-05-28 09:27:30 +02:00
Victor Stinner
7d70942132 Add oslo.vmware to Python 3 test dependencies
Enable ceilometer.tests.compute.virt.vmware.test_vsphere_operations on
Python 3.4.

Change-Id: If3fbc4c5b821cb700799db5597ca6d259167003e
2015-05-28 01:00:00 +02:00
Jenkins
c4cc3bdab4 Merge "Enable test_swift_middleware on Python 3" 2015-05-27 15:31:02 +00:00
Jenkins
35d7ff3d02 Merge "Clear useless exclude from flake8 ignore in tox" 2015-05-27 15:30:54 +00:00
Jenkins
5efc099db4 Merge "Enable more tests on Python 3" 2015-05-26 15:14:16 +00:00
Jenkins
281a8ecd94 Merge "Initial commit for functional tests" 2015-05-26 14:34:11 +00:00
Victor Stinner
ab2a1bc7ad Enable test_swift_middleware on Python 3
Enable also event_pipeline test.

Change-Id: Iffdbc770bb167f49243d7799bb50dc7c4c65a0dd
2015-05-26 15:19:40 +02:00
Victor Stinner
30159de5cd Enable more tests on Python 3
* Trait.convert_value() now explicitly decodes byte strings from UTF-8:
  it fixes a BytesWarning error when Python 3 is run using -bb command
  line option
* Replace types.NoneType with type(None)
* fileutils.write_to_tempfile() expects content as bytes: encode the
  content on Python 3
* subprocess.Popen.communicate() returns stdout and stderr as bytes:
  check for pattern using byte strings
* Fix verify_signature() on Python 3: encode new signature to ASCII
* Replace "with contextlib.nested(...):" with two nested with,
  contextlib.nested() was removed in Python 3
* Fix import in test_messaging
* tox -e py34 now runs much more tests

Change-Id: If2a16c3b8ca64ca86d8172b36ecbf6298044ec87
2015-05-26 15:19:40 +02:00
Julien Danjou
10c0be3e80 Clear useless exclude from flake8 ignore in tox
Change-Id: Ibf1354018a8140901ba44e74f5cc5d88b2578339
2015-05-26 15:18:53 +02:00
Victor Stinner
9d5dc1c482 Update Python 3 requirements
* Remove requirements-py3.txt: it's not more needed, all libraries in
  requirements.txt are compatible with Python 3
* Synchronize test-requirements-py3.txt with test-requirements.txt, add:

  - gabbi
  - psycopg2
  - pylint
  - requests-aws

* Update tox.ini to use requirements.txt instead of requirements-py3.txt

Change-Id: I7c89313663849165e985ae2dac6e05251d003f66
2015-05-15 22:31:47 +02:00
Ilya Tyaptin
afe7af8218 Initial commit for functional tests
In this patchset added a starting point for running
ceilometer functional tests.
Next points include a creating scripts for using a different env setup
for different backends, creating a new job in openstack-infra/project-config
and separating ceilometer tests to unit and functional.

Partially implements: blueprint ceilometer-functional-tests

Change-Id: I3ecadd388812afde5c48aff8214597d76faccf33
2015-05-14 18:19:37 +03:00
Chris Dent
bbab1df894 Remove py33 tox target.
Python 2.7 and 3.4 are the supported versions and we should not
indicate otherwise.

Change-Id: I732c9b0b82661546f9f3b484b85ff9e8598734ae
2015-04-28 11:23:00 +00:00
Chris Dent
823e1992b4 Stop using PYTHONHASHSEED=0 in ceilometer tests
Otherwise we have tests that rely on ordering (for example the one
fixed in this patchset).

Note that this moves the hash seed setting to only be used in docs
and venv tox targets. For some reason there is an issue in the tooling
used by the docs build that tickles a bug in those tools. Tools which
are not part of Ceilometer. To avoid tickling the problem and to move
ceilometer along on this bug, we're just going to mask that.

The venv target is used by the jenkins job that builds docs. It runs
the same build_sphinx setup.py target that the docs target does.

Change-Id: If93ec7934ad59515b821f056ca534d3676ee7e03
Closes-Bug: #1348818
2015-03-30 11:51:06 +00:00
Lianhao Lu
286d78b6b9 Switch to oslo.policy 0.3.0
Switch to the first official release of oslo.policy.

Change-Id: I72bc1450833db88ead2959d818d058fa64429c68
Closes-Bug: #1421863
2015-03-11 09:33:19 +08:00
ZhiQiang Fan
27fed841bf enable oslo namespace check for ceilometer project
Currently, oslo team starts to drop old oslo namespace, the old import
style can work now, but not any more in next development cycle. We've
already done and will continue do the migrate jobs, but we need a way
to prevent accidently import such issue again. The hacking rules cannot
satisfy the need, so let's do the local check.

Code are basically stolen from Nova project.

Note, this patch also fixes some new introduced oslo namespace issues.

Change-Id: I98c81c47a727ed71fb75820f21276cb989d47714
2015-02-11 16:09:19 +08:00
gordon chung
cc7b8b4ecf add elasticsearch events db
this patch implements an elasticsearch driver for events.

Implements: blueprint elasticsearch-driver
Change-Id: Ie579f325685c14aed78d83de7c6d7bff326bc188
2015-02-03 22:58:07 -05:00
Chris Dent
552c63849e Declarative HTTP testing for the Ceilometer API
A 'gabbi' tox target is added which runs a declarative HTTP tests
described in YAML files in 'ceilometer/tests/gabbi/gabbits' and loaded by
'ceilometer/tests/gabbi/test_gabbi.py'. These are driven by the 'gabbi'
python package (available from PyPI).

tox and testr are configured to start and run the tests efficiently:

* a mongodb server, using multiple databases, is made available
* the API wsgi application is used directly via 'wsgi-intercept', no
  web server required
* each YAML file is run as a sequence and where number of processors
  allows, in a different test process
* individual tests can be requested in the usual way:

    tox -egabbi -- <test pattern>

  If this is done, all the tests prior to the one requested, from
  its YAML file, will be run as ordered prerequisites.
* tox targets that already run the tests in ceilometer/tests will
  also discover gabbi tests. If there is no mongodb, they will
  be skipped.

A ConfigFixture does the necessary work of adjusting the configuration
and pipeline to use the mongodb database and nothing else. An
internal InterceptFixture uses wsgi-intercept to access the
ceilometer API. Each yaml file has its own intercepted host.

Fixtures are implemented as nested context managers that are declared
per YAML file, see ceilometer/gabbi/fixtures.py and fixtures-samples.yaml
for an example of how they can be used. Every yaml file uses at
least ConfigFixture.

YAML files can use a variety of strategies for formatting requests
and evaluating the correctness of response. See:
http://gabbi.readthedocs.org/en/latest/format.html

The YAML files included here test simple API features for creating and
retrieving samples. Subsequent patches can (and should) create
additional YAML files to describe more complex scenarios that cover
the entire API (for example alarms are not touched at all by this
patch).

Change-Id: I52551f88bc3beac4bf8a92afa45ac70cd97ffcec
Implements: blueprint declarative-http-tests
2015-02-03 13:03:39 +00:00
ZhiQiang Fan
b483a9ab06 remove pytidylib and netifaces from tox.ini external dependency
pytidylib is introduced on Jan 27, 2014 by
commit: 3257d83bbb3e2302638581c995e44d6e84ad44c0,
changeId: I73f3202ac7a1420aaa303d5ea2e741b2fe5673e9
https://review.openstack.org/#/c/69491/

but actually not used, nor used by packages depended by us. The commit
message, review comments and bug report don't provide details for this
change.

netifaces is introduced Jan 21 2014 by:
commit: 28f6bee013c55ac9f4b4545293fd8849da45c961
changeId: I1721804890423c242bb69697d2c42ca57d0871a4

At that time, it is required by swift, but now we have removed swift from
{test-}requirements.
Now it is only required by oslo.utils.netutils, but can be downloaded
via pypi, see:
https://github.com/openstack/oslo.utils/blob/1.2.0/requirements.txt#L11

Change-Id: I8bff587892396c32b327049b024e6e8acfff64cf
Closes-Bug: #1413455
2015-01-22 11:43:24 +08:00
Jenkins
dc656bad6c Merge "Add mysql and postgresql in tox for debug env" 2015-01-15 04:07:33 +00:00
ZhiQiang Fan
bad6844794 Add oslo.concurrency module to tox --env genconfig
We have used oslo.concurrency module, which has defined two configuration
options 'disable_process_locking' and 'lock_path', this patch adds those
two options to generated ceilometer.conf

see: https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py#L40

Change-Id: Id6dc936932f8d2703975a76405c985359dd09c02
2014-12-25 11:30:03 +08:00
Lianhao Lu
b1b6c17516 Clean up bin directory
Move ceilometer-test-event.py from directory bin to directory tools,
since it's for test purpose.

Change-Id: I3485a27ae66e255fbc460d0548be9dbb5700fa42
2014-12-18 15:00:12 +08:00
ZhiQiang Fan
28a253104a Port to graduated library oslo.i18n
log_handler is not used any more by Ceilometer project, but still
listed on openstack-common.conf, this patch fixes it.

Note: oslo-incubator.timeutils is not used by oslo-incubator modules
any more after we do code sync, this patch removes it too.

gettextutils has graduated from oslo-incubator, we should port our
project to use oslo.i18n. To use oslo.i18n conveniently, this patch
introduces a helper module ceilometer.i18n, which is stolen from Nova.

Note: gettextutils.install is deprecated, see:
http://docs.openstack.org/developer/oslo.i18n/usage.html#creating-an-integration-module
Note: _ is removed from builtins in tox.ini since we're not using
install any more.

Change-Id: I829f9faf97c825422b395bf9c01ae5c17c86d9fb
Closes-Bug: #1389546
2014-12-09 01:11:24 +08:00
ZhiQiang Fan
098dd97f6a Exclude tools/lintstack.head.py for pep8 check
Commit 062ac3313d7fd39611a6b065ffdff4bf309d8b17 has enabled pep8
check on tools directory, however, if developers have run
`tox -epylint` before, then `tox -epep8` will fail because the
automatically generated file tools/lintstack.head.py cannot pass
pep8 check.

This patch excludes tools/lintstack.head.py in tox pep8 env.

Change-Id: Idbf22821f3d44055f8a03321aa0a05416bec809f
2014-11-30 02:05:50 +08:00
Jenkins
942fb02a85 Merge "Remove Python 2.6 classifier" 2014-11-26 00:49:53 +00:00