Update project ID in Gnocchi resources

When creating/updating resources in Gnocchi, the project and user ID
set on samples are managed separately to the resource metadata values
published to Gnocchi. This is to ensure the project and user ID are
only set when creating resources, and never added to resource updates.

There seems to be some kind of assumption that these values cannot
be updated once a resource created (maybe that was the case in the
past?) but with recent versions of Gnocchi this can be freely updated.

Project IDs should always be updated on resources because some types of
resources can be transferred from one project to another (for example,
Cinder volumes). This patch changes the handling for project_id so that
it gets updated with the rest of the configuration-defined metadata
attributes when changes are required.

Note that user_id is not receiving the same treatment. This is because
user_id is not always available on all samples for a given metric, and
because in some cases updating it would cause resource updates to occur
every time a new user performs a request against a resource.

This change has the side effect that it invalidates the Gnocchi
resource metadata cache, if resource caching is enabled in Ceilometer
Notification Agent.

Change-Id: I6d5bd444eab3b664dd794503ae052acdc2334992
Signed-off-by: Callum Dickinson <callum.dickinson@catalystcloud.nz>
Story: 2011706
Task: 53959

ceilometer/publisher/gnocchi.py

@@ -411,11 +411,19 @@ class GnocchiPublisher(publisher.ConfigPublisherBase):
                     gnocchi_data[resource_id] = {
                         'resource_type': rd.cfg['resource_type'],
                         'resource': {"id": resource_id,
-                                     "user_id": sample.user_id,
-                                     "project_id": sample.project_id}}
+                                     "user_id": sample.user_id}}
 
+                # NOTE(callumdickinson): project_id is added to resource_extra
+                # because the project that owns the resource can change (e.g.
+                # transferred volumes), and we want this reflected in Gnocchi.
+                # user_id is not treated this way because it is not always
+                # available on all samples for a given metric, and because in
+                # some cases updating it would cause resource updates to occur
+                # every time a new user performs a request against a resource.
                 gnocchi_data[resource_id].setdefault(
-                    "resource_extra", {}).update(rd.sample_attributes(sample))
+                    "resource_extra", {}).update({
+                        "project_id": sample.project_id,
+                        **rd.sample_attributes(sample)})
                 measures.setdefault(resource_id, {}).setdefault(
                     metric_name,
                     {"measures": [],

ceilometer/tests/unit/publisher/test_gnocchi.py

@@ -872,10 +872,13 @@ class PublisherWorkflowTest(base.BaseTestCase,
                           1, 1)
             )
 
-        if self.patchable_attributes:
+        update_attributes = {**self.postable_attributes,
+                             **self.patchable_attributes}
+        del update_attributes["user_id"]  # Only user_id is not updated.
+        if self.patchable_attributes != update_attributes:
             expected_calls.append(mock.call.resource.update(
                 self.resource_type, resource_id,
-                self.patchable_attributes))
+                update_attributes))
             if self.update_resource_fail:
                 fakeclient.resource.update.side_effect = [Exception('boom!')]
             else:
@@ -891,9 +894,9 @@ class PublisherWorkflowTest(base.BaseTestCase,
         if (self.post_measure_fail
                 or self.create_resource_fail
                 or self.retry_post_measures_fail
-                or (self.update_resource_fail and self.patchable_attributes)):
+                or (self.update_resource_fail and update_attributes)):
 
-            if self.update_resource_fail and self.patchable_attributes:
+            if self.update_resource_fail and update_attributes:
                 logger.error.assert_called_with(
                     'Unexpected exception updating resource type [%s] with '
                     'ID [%s] for resource data [%s]: [%s].', resource_type,

releasenotes/notes/gnocchi-update-project-id-c8355fc5901e7aa3.yaml

@@ -0,0 +1,13 @@
+---
+upgrade:
+  - |
+    If resource metadata caching is enabled in Ceilometer Notification Agent,
+    the Gnocchi resource metadata cache will be invalidated and refreshed upon
+    upgrading due to the fix for updating resource project IDs.
+fixes:
+  - |
+    An issue has been fixed where when publishing resource metadata changes to
+    Gnocchi, the project ID field was not updated when changed (e.g. when
+    transferring volumes from one project to another). Any existing resources
+    that have changed ownership since being added to Gnocchi will have their
+    ``project_id`` attribute updated to the correct value.