telemetry-specs/specs/juno/big-data-sql-v2.rst

'Big Data' SQL part two

https://blueprints.launchpad.net/ceilometer/+spec/bigger-data-sql

In step 1 of sql refactoring, we denormalised the data to capture and store data as close to its raw form as possible. This removed all the overhead caused by deprecated data design requirements. This blueprint further refactors the data model to organise data closer to api model.

Problem description

Currently, the Metering data model is completely denormalised. We store Samples, which are the raw data points, and Meters, which are the definition of said data points. This optimisation allows for good write performance but due to size of Sample table, can cause issues with read performance particularly with get_meter and get_resources. Specifically, joins can cause performance issues.

The current schema is as follows:

* meter - meter definition
    * id: meter id
    * name: meter name
    * type: meter type
    * unit: meter unit
* sample - the raw incoming data
    * id: sample id
    * meter_id: meter id (->meter.id)
    * user_id: user uuid
    * project_id: project uuid
    * resource_id: resource uuid
    * source_id: source id
    * resource_metadata: metadata dictionaries
    * volume: sample volume
    * timestamp: datetime
    * message_signature: message signature
    * message_id: message uuid

Proposed change

The proposed change is to re-implement a normalised model that is tailored to current API requirements. This means grouping Resource specific data in Resource table and Meter specific data in Meter Table.

the proposed schema is as follows:

* resource - resource data
    * internal_id: resource id (to handle assumption resources may not be
                   unique ie. diff user/project/source/meta per resource)
    * resource_id: resource uuid
    * user_id: user uuid
    * project_id: project uuid
    * source_id: source id
    * resource_metadata: metadata dictionary
    * metadata_hash: hash of metadata to allow comparison
* meter - meter definition
    * id: meter id
    * name: meter name
    * type: meter type
    * unit: meter unit
* sample - the raw incoming data
    * id: sample id
    * meter_id: meter id (->meter.id)
    * resource_id: resource id(->resource.internal_id)
    * volume: sample volume
    * timestamp: datetime
    * message_signature: message signature
    * message_id: message uuid

Alternatives

As a schema can be defined in quite a few ways, there are many alternatives in that sense.

Data model impact

Api model will remain unchanged. the sql backend model will change to proposal described above:

• creating a new Resource table
• moving appropriate values from sample table to resource table

These changes will require database migration.

REST API impact

We will need to adapt existing api model to interface with new backend schema but from user POV, there will be no change.

Security impact

None

Pipeline impact

None

Other end user impact

None, we will continue to store a new resource per sample

Performance/Scalability Impacts

The read performance should improve as we will not have a giant Sample table anymore but smaller, tailored Resource, Meter, and Sample tables. The write performance is not expected to degrade noticeably.

It is expected any degradation in write performance would be caught by existing tempest tests.

Use of Ilya's performance tool will be used to verify there is improved read performance and negligible write performance degradation.[1]

[1] https://github.com/ityaptin/ceilometer/blob/master/tools/sample-generator.py

Other deployer impact

None

Developer impact

None, just a new schema to learn about

Implementation

Assignee(s)

Primary assignee:

chungg

Other contributors:

None

Ongoing maintainer:

chungg

Work Items

• Migration script to add new attributes to Meter table and new Resource table
• Modify impl_sqlalchemy get_meters, get_resource, record_metering_data, expirer and any other affected methods to use new schema

Future lifecycle

Most contributors know the sql backend to some degree. The community will maintain until v3 backend is phased in.

Dependencies

None

Testing

• Existing test cases should cover change
• Tempest test cases should cover performance degradation
• Need to add test to handle data expiration

Documentation Impact

None

References

Discussion with Mike Bayer: https://etherpad.openstack.org/p/ceilometer-sqlalchemy-mike-bayer