1002 Commits

Author SHA1 Message Date
Zuul
213087869a Merge "Create 'use_all_resource_revisions' for Gnocchi collector" 2020-12-22 17:18:58 +00:00
Zuul
84ad7377e8 Merge "Increase cost fields to 28 digits precision" 2020-12-22 17:18:54 +00:00
Zuul
994edade52 Merge "Custom fields in summary get API" 2020-12-22 17:18:51 +00:00
wu.shiming
4745bfe954 Update TOX_CONSTRAINTS_FILE
UPPER_CONSTRAINTS_FILE is old name and deprecated
-https://zuul-ci.org/docs/zuul-jobs/python-roles.html#rolevar-tox.tox_constraints_file
This allows to use lower-constraints file as more
readable way instead of UPPER_CONSTRAINTS_FILE=<lower-constraints file>.

Depends-on: https://review.opendev.org/c/openstack/cloudkitty/+/706358

Change-Id: Ieddc84ca5fb13cd1936147a27e450626933f94bb
2020-12-14 18:05:09 -03:00
Andreas Jaeger
11ff7365a3 Update lower-constraints
This repo has not been testing lower-constraints at all due to
broken install_command. If you look at any lower-constraints run and
compare the install python packages with lower-constraints, you see
that those are completely different.

This change removes install_command and updates deps in tox.ini to
follow best practices (moving constraints into deps).
It also updates lower-constraints to newer versions.

Remove broken hacking test.

Co-Authored-By: Justin Ferrieu <jferrieu@objectif-libre.com>
Change-Id: I13daab9e53617266beff7053e50779d1f281802c
2020-12-14 18:03:59 -03:00
wuchunyang
11ff713042 Replace deprecated UPPER_CONSTRAINTS_FILE variable
UPPER_CONSTRAINTS_FILE is old name and deprecated
-https://zuul-ci.org/docs/zuul-jobs/python-roles.html#rolevar-tox.tox_constraints_file
This allows to use lower-constraints file as more
readable way instead of UPPER_CONSTRAINTS_FILE=<lower-constraints file>.

Change-Id: I7b88b55bfceff177b0ed0586d105353f7159b171
2020-11-27 12:40:47 +00:00
Rafael Weingärtner
abffd13426 Create 'use_all_resource_revisions' for Gnocchi collector
This option is useful when using Gnocchi with the patch introduced in
https://github.com/gnocchixyz/gnocchi/pull/1059. That patch can
cause queries to return more than one entry per granularity (
timespan), according to the revisions a resource has. This can be
problematic when using the 'mutate' option of Cloudkitty. Therefore,
we proposed this option to allow operators to discard all
datapoints returned from Gnocchi, but the last one in the
granularity queried by CloudKitty. The default behavior is
maintained, which means, CloudKitty always use all of the data
points returned.

Change-Id: I051ae1fa3ef6ace9aa417f4ccdca929dab0274b2
2020-11-20 08:36:50 -03:00
Zuul
8c4ac0098e Merge "Log the number of tenants loaded by the fetcher" 2020-10-05 15:25:23 +00:00
Rafael Weingärtner
08497f0d52 Increase cost fields to 28 digits precision
When working with some type of resources, and for some specific billing
requirements, we need to set costs that will use up to more than 8 digits on
the right side of the comma. By default, the Python object Decimal support 28
digits. Therefore, it makes sense for us to change the MySQL database schema
of CloudKitty to use 28 digits as well on the right side. This will avoid
confusion for people when using this feature.

One can argue that using the `factor` option can also be a solution for that,
but as I mentioned, for people used to Python, that can cause confusions
because the MySQL DB is using a different precision than the one supported in
Python for the data type we use to represent the `cost` field.

Change-Id: Ifbf5b2515c7eaf470b48f2695d1e45eeab708a72
2020-10-05 14:39:11 +00:00
Zuul
1cac7f9610 Merge "Add a /healthcheck URL" 2020-10-05 14:27:17 +00:00
Pierre Riteau
2ad94f808c Update sample configuration and policy files
Change-Id: I14ee38911fd1bda534d2e1baa4148dd7aa516686
2020-09-23 15:42:24 +02:00
Zuul
6f750ee55a Merge "Add Python3 wallaby unit tests" 2020-09-22 09:03:20 +00:00
Zuul
187453b708 Merge "Update master for stable/victoria" 2020-09-22 08:46:27 +00:00
3038104d3f Add Python3 wallaby unit tests
This is an automatically generated patch to ensure unit testing
is in place for all the of the tested runtimes for wallaby.

See also the PTI in governance [1].

[1]: https://governance.openstack.org/tc/reference/project-testing-interface.html

Change-Id: I3e5b2f46c60113429e3e548bda6254b81874a986
2020-09-21 22:16:38 +00:00
100a1502a9 Update master for stable/victoria
Add file to the reno documentation build to show release notes for
stable/victoria.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/victoria.

Change-Id: Ib933e0496eb684e5cb987d3292e7aa834fedb309
Sem-Ver: feature
2020-09-21 22:16:35 +00:00
Mariusz Karpiarz
4900e5d4ba Log the number of tenants loaded by the fetcher
During initial setup it's useful to know if (and how many) billable
tenants got loaded by the fetcher.

Change-Id: Ib1fbfdde5535ff83f8a657b9542c117a882f802e
2020-09-21 15:33:56 +01:00
Zuul
dbc5b7c8d4 Merge "Replace assertItemsEqual with assertCountEqual" 13.0.0.0rc1 2020-09-21 13:51:22 +00:00
Zuul
5c294c5084 Merge "bump py37 to py38 in tox.ini" 2020-09-21 13:39:12 +00:00
zhangchun
2869666c12 bump py37 to py38 in tox.ini
in 'victoria' cycle, we should test py38 by default.
Trivial change

Change-Id: I80301bac54bb8654f58d4dfc0ff3de5899f3a314
2020-09-20 14:21:04 +00:00
wangzihao
1cccdb2976 Bump hacking min version to 3.0.1
hacking 3.0.1 fix the pinning of flake8 to avoid bringing in a new
version with new checks.

bumping the min version for hacking so that any older hacking versions
which auto adopt the new checks are not used.

Change-Id: I80da860ff80bc3766d66511c70eb22467e25df01
2020-09-18 10:31:19 +08:00
wangzihao
cff987ef67 Replace assertItemsEqual with assertCountEqual
assertItemsEqual was removed from Python's unittest.TestCase in
Python 3.3 [1][2]. We have been able to use them since then, because
testtools required unittest2, which still included it. With testtools
removing Python 2.7 support [3][4], we will lose support for
assertItemsEqual, so we should switch to use assertCountEqual.

[1] - https://bugs.python.org/issue17866
[2] - https://hg.python.org/cpython/rev/d9921cb6e3cd
[3] - testing-cabal/testtools#286
[4] - testing-cabal/testtools#277

Change-Id: Ib8c2acb373db791c0b74b8bea30d424dc44325a5
2020-09-17 17:20:19 +08:00
Zuul
28b41c17e2 Merge "Make Gnocchi connection pool configurable" 2020-09-10 20:09:46 +00:00
Zuul
15e88c8c3a Merge "Fix empty metadata exception in Prometheus collector" 2020-09-10 15:45:04 +00:00
Zuul
f7a75112fa Merge "Add quantity mutation to Prometheus collector" 2020-09-10 15:40:45 +00:00
pedro
7864529c19 Make Gnocchi connection pool configurable
When CloudKitty is collecting data from Gnocchi, it opens many
connections to Gnocchi to parallelize the requests; as CloudKitty starts
to fetch the responses from Gnocchi, it releases the connections to the
pool until it starts to overflow (reaching the default limit of the
Requests connection pool). This behavior causes no problems, but
prevents the reuse of connections.

To mitigate this problem and optimize the use of connections, we propose
to add a new configuration in Gnocchi collector and fetcher to set the
connection pool maximum size when connecting to Gnocchi. The new
configuration is `http_pool_maxsize` and you can configure it in the
`cloudkitty.conf` file like:

```
[fetcher_gnocchi]
http_pool_maxsize = 50

[collector_gnocchi]
http_pool_maxsize = 50
```

The default value is defined by the `requests` library which is 10.

Change-Id: I3ee22984dc87b505924e1711bc723bc1c5f160a5
Story: 2008125
Task: 40847
2020-09-10 16:50:51 +02:00
Quentin Anglade
8962a9ab63 Add a Monasca fetcher
This adds a Monasca fetcher for CloudKitty.

Change-Id: I0bf5af21987878d1e078021b0a9fbd0cc0a37b9d
Story: 2006675
Task: 36949
2020-09-09 15:09:09 -03:00
Thomas Goirand
0217e602aa Add a /healthcheck URL
This is useful for operators to setup haproxy and know that one of
the API is alive. I've done such a patch on many other projects,
and this is unifying all.

Change-Id: I14214679849c1cd378c7487254761f1d4d73e10e
2020-09-09 13:40:05 +00:00
Ning Yao
89dfb91dd4 Fix empty metadata exception in Prometheus collector
Metadata defined in metrics.yml may be non-existent in data['metric']
from Prometheus results. Deal with the exception similarly to the
Gnocchi collector.

Change-Id: I02ee50793f3c46050250ca1c4094dd4ef0736ca6
Signed-off-by: Ning Yao <yaoning@unitedstack.com>
Story: 2008123
Task: 40845
2020-09-09 15:18:36 +02:00
Ning Yao
1eb8ecf1f4 Add quantity mutation to Prometheus collector
The Prometheus collector was ignoring the mutate definition from
metrics.yml, which may produce wrong resource ratings.

Change-Id: I662f6ac798c20ca43028b3807fa03aacfdcf252b
Signed-off-by: Ning Yao <yaoning@unitedstack.com>
Story: 2008122
Task: 40844
2020-09-09 15:10:11 +02:00
Rafael Weingärtner
18feacfa77 Custom fields in summary get API
Currently, only the quantity and price are returned in the v2 summary get API.
However, that might not satisfy all user's needs. This PR Enables custom fields
to be used in the Summary get API; thus, we start to allow users/operators/business
people to create richer reports via the summary Get API.

Change-Id: Id4dd83d0703ec0dff32510e6dd1d2dad9b181306
2020-09-08 15:05:06 +00:00
Andreas Jaeger
cbcdacac61 Switch to newer openstackdocstheme and reno versions
Switch to openstackdocstheme 2.2.1 and reno 3.1.0 versions. Using
these versions will allow especially:
* Linking from HTML to PDF document
* Allow parallel building of documents
* Fix some rendering problems

Update Sphinx version as well.

Set openstackdocs_pdf_link to link to PDF file. Note that
the link to the published document only works on docs.openstack.org
where the PDF file is placed in the top-level html directory. The
site-preview places the PDF in a pdf directory.

Disable openstackdocs_auto_name to use 'project' variable as name.

Change pygments_style to 'native' since old theme version always used
'native' and the theme now respects the setting and using 'sphinx' can
lead to some strange rendering.

openstackdocstheme renames some variables, so follow the renames
before the next release removes them. A couple of variables are also
not needed anymore, remove them.

See also
http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014971.html

Change-Id: I6be1174686cb1d8f11e8cb4be58c0e739bf0f931
2020-09-07 18:47:30 +02:00
Zuul
65552a84a2 Merge "Add Python3 victoria unit tests" 2020-09-07 14:45:57 +00:00
Zuul
ccc57b6659 Merge "Add py38 package metadata" 2020-09-07 14:31:49 +00:00
Zuul
253d3359e6 Merge "Adjust hacking tests to fix py38 support" 2020-09-07 13:18:50 +00:00
Zuul
b02b52a858 Merge "Update hacking for Python3" 2020-09-07 13:06:03 +00:00
Zuul
e829724747 Merge "Add i18n support for error message" 2020-09-07 12:51:25 +00:00
107e2d664c Add Python3 victoria unit tests
This is an automatically generated patch to ensure unit testing
is in place for all the of the tested runtimes for victoria.

See also the PTI in governance [1].

[1]: https://governance.openstack.org/tc/reference/project-testing-interface.html

Change-Id: I27d8a1a20497b15f0d33d340d9a2787e63c88fec
2020-09-07 12:33:16 +00:00
kangyufei
b776aaf98e Add py38 package metadata
Change-Id: Ibf2290599ac9a0839b01e5df9a2b9a892ec0afe7
2020-09-07 12:27:15 +00:00
Sean McGinnis
35afc4b24b Adjust hacking tests to fix py38 support
Starting in py38, some of the column calculations have changed. We had
some hacking unit tests that would assert on tuples containing those
column numbers, so things would pass fine when running on py36 or py37,
but would fail when run on py38.

Since we don't really care about the column number, only whether the
hacking check fails or doesn't, we can just match on mock.ANY for that
value.

Change-Id: I117177a43293efb389dd8c0cd5a5c0ee16a7eca9
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2020-08-24 16:41:44 -03:00
Rafael Weingärtner
b7abb9bb69 Replace tz.UTC with dateutil.tz.tzutc()
As discussed in https://review.opendev.org/#/c/742477/, this patch
replaces the use of `tz.UTC` with `dateutil.tz.tzutc()`

Also add python-dateutil to requirements.txt, using the same minimum
version as several other OpenStack projects, including nova.

Change-Id: I4da9e8854a571058e48c2f51c1d340bc135cfe2b
2020-08-21 15:58:38 +02:00
Andreas Jaeger
b014dbe64c Update hacking for Python3
The repo is Python 3 now, so update hacking to version 3.0 which
supports Python 3.

Fix problems found.

Update local hacking checks for new flake8.

Change-Id: I41a5518f1210c9a1b6d5217bbbcd99bbaa3c9b53
2020-08-18 22:27:09 -03:00
Zuul
8877fa1a37 Merge "Use unittest.mock instead of third party mock" 2020-08-17 13:52:41 +00:00
Zuul
48e667d873 Merge "Stop to use the __future__ module." 2020-08-17 13:46:52 +00:00
Zuul
5131351876 Merge "[devstack] Collector Variable" 2020-08-13 12:45:33 +00:00
Hervé Beraud
3668a76fe4 Stop to use the __future__ module.
The __future__ module [1] was used in this context to ensure compatibility
between python 2 and python 3.

We previously dropped the support of python 2.7 [2] and now we only support
python 3 so we don't need to continue to use this module and the imports
listed below.

Imports commonly used and their related PEPs:
- `division` is related to PEP 238 [3]
- `print_function` is related to PEP 3105 [4]
- `unicode_literals` is related to PEP 3112 [5]
- `with_statement` is related to PEP 343 [6]
- `absolute_import` is related to PEP 328 [7]

[1] https://docs.python.org/3/library/__future__.html
[2] https://governance.openstack.org/tc/goals/selected/ussuri/drop-py27.html
[3] https://www.python.org/dev/peps/pep-0238
[4] https://www.python.org/dev/peps/pep-3105
[5] https://www.python.org/dev/peps/pep-3112
[6] https://www.python.org/dev/peps/pep-0343
[7] https://www.python.org/dev/peps/pep-0328

Change-Id: I74f1cce5430d81aa7dad08920baa9d771271f871
2020-08-13 09:44:17 +00:00
Sean McGinnis
18a47cb24a Use unittest.mock instead of third party mock
We can now use the standard library mock library instead of the third
party one previously needed for py27 compatibility.

Change-Id: Ia8e87cd6a25621a97c876de5422d0c9d0c8dd751
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2020-08-13 09:43:50 +00:00
e8b7b3fd0b Update master for stable/ussuri
Add file to the reno documentation build to show release notes for
stable/ussuri.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/ussuri.

Change-Id: I5cf5769d6241dcc22c0665aba7ad368bf63d1abd
Sem-Ver: feature
2020-06-24 22:55:24 +00:00
Sean McGinnis
3dccfc7643
Fix docs build error due to duplicate references
Class docstrings were being pulled in to multiple locations, causing
sphinx errors due to the duplicate references. This fixes it by adding
the :noindex: option to those references.

Also temporarily disabling the PDF build as that is failing with another
issue that should be fixed in a follow up.

Change-Id: I6de2948ff49e1bb0b6a8b9a3a90f9f2ebbb8b7bb
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2020-06-24 16:56:51 -05:00
Abraham Arce
1449d76472 [devstack] Collector Variable
The collector is configurable via the CLOUDKITTY_COLLECTOR variable, no
CLOUDKITTY_GNOCCHI is available.

Change-Id: If433bb26e64747e887af3b191cfae36f31390809
Signed-off-by: Abraham Arce <abraham.arce.moreno@intel.com>
2020-03-04 14:23:27 -06:00
Andreas Jaeger
56609c46f9 [ussuri][goal] Cleanup drop python 2.7 support
OpenStack is dropping the py2.7 support in ussuri cycle.

Complete discussion & schedule can be found in
- http://lists.openstack.org/pipermail/openstack-discuss/2019-October/010142.html
- https://etherpad.openstack.org/p/drop-python2-support

Ussuri Communtiy-wide goal:
https://governance.openstack.org/tc/goals/selected/ussuri/drop-py27.html

this removes some cruft that can be removed now:
* cruft in setup.cfg and setup.py
* tox.ini: Add ignore_basepython_conflict and increase minversion
  for that
* Python 2/3 specific requirements, the repo is Python 3 only.

Change-Id: Iae68e58c82a180eb771bef00a3f44fd28abe5fae
12.0.0
2020-02-24 20:16:56 +01:00