4562 Commits

Author SHA1 Message Date
Chris Dent
2511cfb6e4 A dogpile cache of gnocchi resources
What this does is store a key value pair in oslo_cache where the key
is the resource id and the value is a hash of the frozenset of
the attributes of the resource less the defined metrics[1]. When it
is time to create or update a resource we ask the cache:

  Are the resource attributes I'm about to store the same as the
  last ones stored for this id?

If the answer is yes we don't need to store the resource. That's all
it does and that is all it needs to do because if the cache fails
to have the correct information that's the same as the cache not
existing in the first place.

To get this to work in the face of eventlet's eager beavering we
need to lock around create_resource and update_resource so that
we have a chance to write the cache before another *_resource is
called in this process. Superficial investigation shows that this
works out pretty well because when, for example, you start a new
instance the collector will all of sudden try several
_create_resources, only one of which actually needs to happen.
The lock makes sure only that one happens when there is just
one collector. Where there are several collectors that won't be
the case but _some_ of them will be stopped. And that's the point
here: better not perfect.

The cache is implemented using oslo_cache which can be configured
via oslo_config with an entry such as:

    [cache]
    backend = dogpile.cache.redis
    backend_argument = url:redis://localhost:6379
    backend_argument = db:0
    backend_argument = distributed_lock:True
    backend_argument = redis_expiration_time:600

The cache is exercised most for resource updates (as you might
expect) but does still sometimes get engaged for resource creates
(as described above).

A cache_key_mangler is used to ensure that keys generated by the
gnocchi dispatcher are in their own namespace.

[1] Metrics are not included because they are represented as
sub-dicts which are not hashable and thus cannot go in the
frozenset. Since the metrics are fairly static (coming from a yaml
file near you, soon) this shouldn't be a problem. If it is then we
can come up with a way to create a hash that can deal with
sub-dicts.

Closes-Bug: #1483634
Change-Id: I1f2da145ca87712cd2ff5b8afecf1bca0ba53788
2015-11-17 13:40:24 +00:00
Jenkins
d753c5de26 Merge "install database when collector is enabled." 2015-11-16 22:01:05 +00:00
Jenkins
7058c12f7c Merge "add reno for release notes management" 2015-11-15 06:43:45 +00:00
ZhiQiang Fan
0ba0ba7270 install database when collector is enabled.
Currently, if we run devstack for compute node, ceilometer-collector
service will be disabled on compute node, but the database is still
installed.

This patch makes sure that database is installed only when collector
is enabled.

Also add check for coordinate and hypervisor requirements.

Change-Id: Ib3614d2e698403c24ad299d95ef42c81815fa76e
Closes-Bug: #1508518
2015-11-13 19:40:14 -07:00
OpenStack Proposal Bot
0dd75dbef8 Updated from global requirements
Change-Id: I4cb014c601def629ddd543e6b97d9f4462a6e917
2015-11-13 17:09:30 +00:00
Jenkins
77ec74f3f3 Merge "Configure collector to only record meter or event" 2015-11-13 02:57:52 +00:00
Jenkins
495f6dfd7a Merge "add rohit_ to MAINTAINERS" 2015-11-11 21:20:17 +00:00
OpenStack Proposal Bot
a9949521ea Updated from global requirements
Change-Id: I42468120c57c6600f413b1a8552858f7c42952e8
2015-11-11 17:17:43 +00:00
gordon chung
2d471862e7 add reno for release notes management
Change-Id: Id36eb9d88c6a8fba635779fc9a971f015908b5de
2015-11-11 10:35:22 -05:00
OpenStack Proposal Bot
32a9d71bb4 Updated from global requirements
Change-Id: Ibc6fdd1d88c8c6e00d88ed3dbb492ef364cd4a4f
2015-11-11 04:01:49 +00:00
Jenkins
191852fbaf Merge "Add sql-expire-samples-only to option list" 2015-11-09 14:58:36 +00:00
Jenkins
ac75a23813 Merge "Fix snmp pollster to not ignore valid meters" 2015-11-09 14:01:16 +00:00
lvdongbing
43a50f9be8 Support to get hardware's cpu_util from snmp
Now, agent-central can only get cpu load, this patch add cpu_util
definition to snmp.yaml so that agent-central is able to  get
hardware's cpu_util metric.

Closes-Bug: #1513731
Change-Id: Ia43c4f103476567c607b63493261f1508dd19f5a
2015-11-06 02:46:34 -05:00
Jenkins
2bddfa28ad Merge "Factorize yaml loading of declarative stuffs" 2015-11-05 16:23:27 +00:00
gordon chung
17033f797b add rohit_ to MAINTAINERS
Rohit did a good job these last few months working on Ceilometer. Let's
bring him in.

Change-Id: Ifd6d5eb3a2623a17f116cb4ccd74bef5a756a269
2015-11-05 08:42:40 -05:00
Jenkins
2be016ec68 Merge "Factorize field definition of declarative code" 2015-11-04 22:38:56 +00:00
Jenkins
7d2306a3b5 Merge "gnocchi: set the default archive policy to None" 2015-11-04 12:49:48 +00:00
Julien Danjou
fb388c50c3 gnocchi: set the default archive policy to None
This makes sure that by default the archive policy rules created on the
Gnocchi side are used.

Closes-Bug: #1501372
Change-Id: I483a0f61d490001da66abdebe8eb0f58b8bbcb52
Depends-On: I0dca487cc9a75cf3e651cba7b0d05b66fedeee98
2015-11-04 10:01:46 +01:00
Chris Dent
94ce728f31 Mv gabbi_pipeline.yaml into test directories
Otherwise people may think that it is available for installation. It
is not, it is for testing only.

Change-Id: Iad19536fba569c6a9d43c07c64746e7f6ffde986
2015-11-03 16:30:01 +00:00
Jenkins
3198eb64d9 Merge "tox: use pretty_tox in most places" 2015-11-03 16:02:01 +00:00
Jenkins
ccac2a3274 Merge "Wrong result is returned when call events getting API" 2015-11-03 15:48:46 +00:00
Mehdi Abaakouk
72edfb6093 Factorize yaml loading of declarative stuffs
This change continues to remove code duplication by factorizing the
yaml file loading.

This also homogenizes the error handling.
The precision of error message will be the same for all.
When the yaml file is incorrectly formatted, we now raise an exception,
instead of sometimes raising and sometimes loading an empty definitions,
that can generated an unwanted behavior of collector and notification agents.

Change-Id: I9ae8f519472f1ae2a14e61028a427ba6f3b0d1f3
2015-11-03 14:38:15 +01:00
Mehdi Abaakouk
e08188f5b8 Factorize field definition of declarative code
Currently we have three differents parts of code that parse samples
and notifications. All of them do the same thing.
The event one have some additionals feature "TraitPlugin".

This change removes the code duplication and allows to use TraitPlugin
into gnocchi and meter definitions.

Change-Id: Id125de92a5893d7afa5a3d55c3f183bd2035a733
2015-11-03 14:38:15 +01:00
xiangjun li
ac3ff88c03 Wrong result is returned when call events getting API
For current implementation, events getting API only support 'eq' query
operation when query field is the one of 'event_type', 'message_id',
'start_timestamp' and 'end_timestamp'. But there is a problem, if the
query operation wasn't specified as 'eq'(e.g. 'ne'), the returned result
is still the same as the query operation was specified as 'eq'.

This patch add a check for this situation: if the query field is the one
of 'event_type', 'message_id', 'start_timestamp', 'end_timestamp', and
the query operation user specified is not 'eq', then a client side error
will be thrown.

Also, corresponding unit test case is added.

Change-Id: I4e4b127045de6e933281d9289271af891c3c80fe
Closes-Bug: #1511592
2015-11-03 09:47:36 +08:00
Julien Danjou
3e203b2585 tox: use pretty_tox in most places
Change-Id: I005a31938107177058481ead225ffc30197588b5
2015-11-02 15:23:27 +01:00
OpenStack Proposal Bot
b140be37a2 Updated from global requirements
Change-Id: Ie481f92b8c0106f8a5234b6e11b94999f575d8f2
2015-10-30 09:04:40 +00:00
Jenkins
d26ebcc74b Merge "configure Apache only when ceilometer-api is enabled" 2015-10-29 06:37:40 +00:00
Jenkins
b8527405bd Merge "api: simplify root controller" 2015-10-28 09:50:38 +00:00
Jenkins
5929a787b4 Merge "avoid unnecessary inner join in get_resources() for SQL backend" 2015-10-28 09:44:18 +00:00
Jenkins
c627f7e542 Merge "reset policy per test" 2015-10-28 04:56:43 +00:00
ZhiQiang Fan
a4f4429405 avoid unnecessary inner join in get_resources() for SQL backend
To get distinct resource ids, we do a query on resource table which
inner join sample table, and apply filters on it.

Note that when sql_expire_samples_only is enabled, there will be
some resources without any sample, in such case we must use inner
join to avoid wrong result, no matter if there is a timestamp filter
or not.

But that option is disabled by default, so when there is no timestamp
filters, the inner join is unnecessary, we should avoid it to save
some RAM/CPU cost.

Change-Id: If85dbea15d42d42c6b0be7402c06f258e278b2eb
Closes-Bug: #1509677
2015-10-26 20:29:03 -06:00
ZhiQiang Fan
7a95441d68 Add sql-expire-samples-only to option list
Change-Id: I65e9e9657304efa05d34d4771ef67eebb29d8ae4
Closes-Bug: #1510050
2015-10-26 05:39:11 -06:00
Jenkins
e8ddd5e814 Merge "avoid generate temporary table when query samples" 2015-10-26 09:31:59 +00:00
OpenStack Proposal Bot
b34865f808 Updated from global requirements
Change-Id: I698f61ef72b3359eba3011762023ad9865fbe19f
2015-10-24 00:22:10 +00:00
Jenkins
26805e623d Merge "avoid using isolation level" 2015-10-23 21:43:38 +00:00
Jenkins
289b84ba25 Merge "Use oslo_config PortOpt support" 2015-10-23 18:29:16 +00:00
Jenkins
e62079e8aa Merge "api: simplify Pecan config" 2015-10-23 15:43:55 +00:00
ZhiQiang Fan
9c3278c942 configure Apache only when ceilometer-api is enabled
Currently, even though ceilometer-api service is disabled by user,
we still try to configure ceilometer-api in Apache if CEILOMETER_USE_MOD_WSGI
is not set to false. This patch adds a constraint to ensure we only
configure when ceilometer-api is enabled.

Change-Id: I3f2bab3f646f7df57c32db3251f811cb801d93de
Partial-Bug: #1508518
2015-10-23 02:26:13 -06:00
Jenkins
6b0845fb1d Merge "Imported Translations from Zanata" 2015-10-22 12:08:07 +00:00
Jenkins
9e65978154 Merge "Correct the timestamp type when make test samples data" 2015-10-22 10:33:54 +00:00
OpenStack Proposal Bot
bd01b28599 Imported Translations from Zanata
For more information about this automatic import see:
https://wiki.openstack.org/wiki/Translations/Infrastructure

Change-Id: I0ca5098a8e55b80d6f6ebaf5dd7b83d48ac20830
2015-10-22 06:15:03 +00:00
Jenkins
d76aef533c Merge "Use gnocchiclient for integration script" 2015-10-21 21:38:23 +00:00
gordon chung
898cd3d036 avoid using isolation level
depending on sql driver, REPEATABLE READ isolation level may lock
an entire table and cause write timeouts. isolation level was set
originally to ensure consistent reads between 2 queries required to
build events. that said, we can avoid table locks by making
assumption that the 1st query is the correct base and any difference
given by 2nd query can be discarded.

Change-Id: Ic53e1addf38a4d5934b4e627c4c974c6db42517e
Closes-Bug: #1506717
2015-10-21 08:44:58 -04:00
ZhiQiang Fan
15f44c6946 specify runtime environment for scripts
I68200a23c87ceca5a237da13d9549c0aa82f1b8f has changed two scripts
to be executable, but unfortunately, it forgets to specify runtime
environment for those scripts, which will cause error when trying
to run those scripts directly.

This patch adds python environment specification for them.

Change-Id: Ibbcefb671de76146529b9a6e2debfee154a1aaa7
2015-10-20 23:43:04 +08:00
liusheng
3587ce9fdf Using oslo-config-generator to instead of generate-config-file.sh
We don't need a separate script to wrap the oslo-config-generator. Like
other projects, we can just specify a config-generator config file to
define the namespaces.

Change-Id: I9ee06658d49163f041df18a62b33fa2804f545b8
2015-10-20 17:38:57 +08:00
Mehdi Abaakouk
78ccbc6592 Use gnocchiclient for integration script
Change-Id: I3f88c7ad41518f742ac21dd3ae8e48daa4b4377c
2015-10-20 10:47:08 +02:00
Jenkins
3a3093afa6 Merge "Updated from global requirements" 2015-10-20 04:29:37 +00:00
Jenkins
d00c46848e Merge "devstack: fix debug info for Gnocchi" 2015-10-20 02:34:13 +00:00
liusheng
2c1227e09d Correct the timestamp type when make test samples data
When recording samples to database, the timestamp should be
datetime.datetime type, but in make_test_data.py, the timestamp
will be transformed to iso format with unicode type.

Change-Id: Iffb09a293684fb8eab768c7370e8967349032ae5
Closes-Bug: #1504539
2015-10-20 01:39:20 +00:00
OpenStack Proposal Bot
f64577892d Updated from global requirements
Change-Id: I3e235fda97b78f655a97752afa702f6fdf11cf2d
2015-10-20 00:48:30 +00:00