openstack/trove-specs
Code Issues Proposed changes

Merge "Add module mangement to Trove"

Browse Source
This commit is contained in:
Jenkins
2016-02-16 19:02:53 +00:00
committed by Gerrit Code Review
parent 34ad2d86b9 6f5d350f3a
commit ac1d0a3324
1 changed files with 835 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

835
specs/mitaka/module-management.rst Normal file
View File

@@ -0,0 +1,835 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
Sections of this template were taken directly from the Nova spec
template at:
https://github.com/openstack/nova-specs/blob/master/specs/template.rst
..
This template should be in ReSTructured text. The filename in the git
repository should match the launchpad URL, for example a URL of
https://blueprints.launchpad.net/trove/+spec/awesome-thing should be named
awesome-thing.rst.
Please do not delete any of the sections in this template. If you
have nothing to say for a whole section, just write: None
Note: This comment may be removed if desired, however the license notice
above should remain.
=================
Module Management
=================
.. If section numbers are desired, unindent this
.. sectnum::
.. If a TOC is desired, unindent this
.. contents::
Historically, Trove has supported open source databases. As more datastores
were added it was inevitable that this would eventually change to include
proprietary databases as well. Starting with the Liberty release this is the
case (support for Vertica and DB2 now being available) and with this comes the
issue of managing the licenses of said databases. In addition, operators may
find it useful to include other software on their images that may require
'activation' by end-users (for example New Relic's analytical suite). A method
of activating this software is also needed.
The concept of applying a 'license' or 'activation' or 'configuration' of this
third-party software is what is referred to herein as 'module' management.
Launchpad Blueprint:
https://blueprints.launchpad.net/trove/+spec/module-management
Problem Description
===================
Users of a particular cloud may be willing to purchase a license to use a
datastore from the cloud vendor on a pay-as-you-go based model, and this is the
model assumed at the moment (as both Vertica and DB2 redstack images include a
fully-functional and licensed database). It is also desirable, however, that
users be allowed to 'bring their own license.' In this scenario the user
provides a license file that the database requires, and as such a mechanism
needs to be in place to 'activate' and/or 'renew' the license through the use
of the user provided file.
This same problem exists for any other proprietary software that an operator
may wish to include in their Trove images. These software packages also
typically require activation through the use of a license key or file (such as
New Relic [1]_) or configuration of some kind.
Proposed Change
===============
Trove's responsiblity towards module management will be restricted to the scope
of encrypting and storing the required data file (for example, a license file)
and providing a way to apply this data to a new or existing Trove instance or
cluster. A mechanism will be put in place to allow end users the ability to
manage adding, deleting, listing, viewing and updating these module data files.
Methods to apply, remove, query and retrieve the actual 'module' data file on
the Trove instance will also be provided.
A repeatable option (--module) will be added to the create and cluster-create
commands to allow adhoc module selection. In addition, modules can be set to
auto-apply, which will have the effect of the Guest Agent installing that
module on any instance created with the relevant datastore combination.
Configuration
-------------
The following configuration changes are anticipated.
A way of specifying valid module 'types' will be needed for proper validation
on module create:
.. code-block:: python
cfg.StrOpt('module_types', default=None,
help='A list of module types supported.'),
A key will be needed in order to be able to encrypt the module data file before
storing it in the database:
.. code-block:: python
cfg.StrOpt('module_aes_cbc_key', default='module_aes_cbc_key',
help='OpenSSL aes_cbc key for module encryption.'),
Database
--------
A new table (modules) will be added to the Trove schema:
================= ============ =========== ==============================
Column Type Allow Nulls Description
================= ============ =========== ==============================
id varchar(36) No ID of module (autogenerated)
type varchar(255) No Type of module. This will
correlate directly to the
required plugin (i.e. a
plugin must exist of this
'type')
tenant_id varchar(36) No ID of tenant to apply
module to. 'all' means module
applies to all tenants
datastore varchar(36) No Name of datastore to apply
module to. 'all' means module
applies to all datastores
datastore_version varchar(36) No Name of datastore version to
apply module to. 'all' means
module applies to all
datastores
name varchar(255) No Name of module
description varchar(512) Yes Description of module
auto_apply tinyint(1) No Should this module be
automatically applied during
instance/cluster create. Will
default to 'no' if not
provided
visible tinyint(1) No Should this module be
visible to non-admin users.
Will default to 'yes' if not
provided
live_update tinyint(1) No Can this module be updated
while applied-to instances
still exist. If set to 'no'
all instances must have the
corresponding module removed
before it can be updated.
Defaults to 'no'
contents blob No Encrypted module contents
md5 varchar(32) No MD5 hash of module contents
created DateTime No Created date
updated DateTime No Updated date
deleted tinyint(1) Yes Deleted flag
deleted_at DateTime Yes Deleted date
================= ============ =========== ==============================
A unique index will be created from the (datastore, datastore_version, name)
fields, to allow easy determination of the correct module to apply to a
specific instance.
An MD5 hash of the module contents will be stored in the module record as
well. This hash will be reported back when querying the module from a
running instance, as the original module record could have been modified with
new contents after it was initially applied.
On installing the module contents on a given instance, a file will be created
in a know location using <datastore>-<datastore_version>-<name>.lic as a
pattern.
Creating modules that apply to 'all' tenants or 'all' datastores and ones that
are auto-applied will require admin credentials.
Setting a module to 'not' visible is also an admin-only option. This will
allow administrators to 'hide' modules from users if they so desire. Modules
that are marked visible=False will not be returned in commands such as list or
show unless requested by an admin user. Non-admin users won't be able to apply
a non-visible module, however they will still be auto-applied if so designated.
A new table (instance_modules) will be added to the Trove schema to track which
modules have been applied to each instance:
================= ============ =========== ==============================
Column Type Allow Nulls Description
================= ============ =========== ==============================
id varchar(36) No ID of association
(autogenerated)
instance_id varchar(36) No ID of instance
module_id varchar(36) No ID of module
md5 varchar(32) No MD5 hash of module contents
created DateTime No Created date
updated DateTime No Updated date
deleted tinyint(1) Yes Deleted flag
deleted_at DateTime Yes Deleted date
================= ============ =========== ==============================
Public API
----------
New ReST API calls will be added to the Trove infrastructure. These fall into
two categories - ones to manage the maintenance of the actual modules, and
ones to handle the instance interactions.
In addition, the create and cluster-create calls will be enhanced.
Module Maintenance
..................
To retrieve a list of all modules that can be applied, the following request
would be made:
Request::
GET v1/modules
Response::
{
'modules' : [
{
'id': <id>,
'type': 'vertica_license',
'tenant': <id>,
'datastore': 'vertica',
'datastore_version': 'all',
'name': '100GB',
'description': 'Vertica license for 100GB',
'auto_apply': False,
'visible': True, # returned for admin only
'live_update': False,
'md5': <md5>,
'created': <date>,
'updated': <date>,
},
{
'id': <id>,
'type': 'new_relic_activation',
'tenant': <id>,
'datastore': 'all',
'datastore_version': 'all',
'name': 'new_relic',
'description': 'New Relic activation',
'auto_apply': True,
'visible': True, # returned for admin only
'live_update': True,
'md5': <md5>,
'created': <date>,
'updated': <date>,
},
]
}
Response Codes::
200 Success
Note that an admin user will receive the modules for all tenants, whereas
regular users will see modules for their tenant only.
To retrieve a list of valid modules that can be applied to a specific
datastore, the following request would be made:
Request::
GET v1/datastores/{datastore_id}/modules
Response::
{
'modules' : [
{
'id': <id>,
'type': 'new_relic_activation',
'tenant': <id>,
'datastore': 'all',
'datastore_version': 'all',
'name': 'new_relic',
'description': 'New Relic activation',
'auto_apply': True,
'visible': True, # returned for admin only
'live_update': True,
'md5': <md5>,
'updated': <date>,
},
]
}
Response Codes::
200 Success
To show the details of a particular module, the following request would be
made:
Request::
GET v1/modules/<id>
Response::
{
'id': <id>,
'type': 'new_relic_activation',
'tenant': <id>,
'datastore': 'all',
'datastore_version': 'all',
'name': 'new_relic',
'description': 'New Relic activation',
'auto_apply': True,
'visible': True, # returned for admin only
'live_update': True,
'md5': <md5>,
'created': <date>,
'updated': <date>,
}
Response Codes::
200 Success
404 Not Found
To create a module, the following request would be made:
Request::
POST /v1.0/modules
{
'type': 'vertica_license',
'tenant': <id>,
'datastore': 'vertica',
'datastore_version': 'all',
'name': '100GB',
'description': 'Vertica license for 100GB',
'auto_apply': False,
'visible': False, # admin-only option
'live_update': True,
'contents': <module_contents>,
}
Response::
{
"module": {
'id': <id>,
'type': 'vertica_license',
'tenant': <id>,
'datastore': 'vertica',
'datastore_version': 'all',
'name': '100GB',
'description': 'Vertica license for 100GB',
'auto_apply': False,
'visible': False, # returned for admin only
'live_update': True,
'md5': <md5>,
'created': <date>,
'updated': <date>,
}
}
Response Codes::
200 Success
400 Bad Request
To update a module, the following request would be made:
Request::
PATCH /v1.0/modules/{module_id}
{
'type': 'new_type',
'tenant': <id>,
'datastore': 'new_datastore',
'datastore_version': 'new_datastore_version',
'name': 'new_name',
'description': 'new_description',
'auto_apply': True,
'visible': False, # admin-only option
'live_update': True,
'contents': <module_contents>,
}
Response::
{
"module": {
'id': <id>,
'type': 'new_type',
'tenant': <id>,
'datastore': 'new_datastore',
'datastore_version': 'new_datastore_version',
'name': 'new_name',
'description': 'new_description',
'auto_apply': True,
'visible': False, # returned for admin only
'live_update': True,
'md5': <new_md5>,
'created': <date>,
'updated': <date>,
}
}
Response Codes::
200 Success
400 Bad Request
404 Not Found
To delete a module, the following request would be made:
Request::
DELETE /v1.0/modules/{module_id}
{
}
Response::
This operation has no response body
Response Codes::
200 Success
404 Not Found
To query which instances have a particular module applied, the following
request would be made:
Request::
GET v1/modules/{module_id}/instances
{
}
Response::
{
'instance': <id>,
'modules' : [
{
'name': '100GB',
'id': <id>,
'md5': <md5>,
'installed': <date>,
},
{
'name': 'new_relic',
'id': <id>,
'md5': <md5>,
'installed': <date>,
},
]
}
Response Codes::
200 Success
404 Not Found
Instance Interaction
....................
To apply modules to an instance, the following request would be made:
Request::
POST v1/{tenant_id}/instances/{instance_id}/modules
{
'modules' : [
{
"id": <id>,
},
]
}
Response::
{
'type': 'vertica_license',
'datastore': 'vertica',
'datastore_version': 'all',
'name': '100GB',
'md5': <md5>,
}
Response Codes::
202 Success
400 Bad Request
404 Not Found
To query an instance about installed modules, the following request would be
made:
Request::
GET v1/{tenant_id}/instances/{instance_id}/modules
{
}
Response::
{
'modules' : [
{
'type': 'vertica_license',
'datastore': 'vertica',
'datastore_version': 'all',
'name': '100GB',
'filename': 'vertica-all-100GB.lic',
'md5': <md5>,
'installed': <date>,
'status': 'OK',
'error_message': None,
},
{
'type': 'new_relic_activation',
'datastore': 'all',
'datastore_version': 'all',
'name': 'new_relic',
'filename': 'all-all-new_relic.lic',
'md5': <md5>,
'installed': <date>,
'status': 'FAILED',
'error_message': 'New Relic binaries not found',
},
]
}
Response Codes::
200 Success
404 Not Found
To retrieve a module from an instance, the following request would be made:
Request::
GET v1/{tenant_id}/instances/{instance_id}/modules/{module_id}
{
}
Response::
{
'filename': 'vertica-all-100GB.lic',
'contents': <module_contents>,
'md5': <md5>,
}
Response Codes::
200 Success
404 Not Found
To delete a module from an instance, the following request would be made:
Request::
DELETE v1/{tenant_id}/instances/{instance_id}/modules/{module_id}
{
}
Response::
This operation has no response body
Response Codes::
202 Success
404 Not Found
Creation Enhancements
.....................
The instance create API will be enhanced to include a module field, containing
a list of modules to apply. These will be sent down during the normal
'prepare' call and the appropriate plugin called once this instance has been
provisioned correctly.
.. code-block:: python
{
'modules' : [
{
"id": <id>,
},
]
}
In a similar manner, the cluster create API will also be enhanced to include
module information in the instances field, as is currently done with flavors,
AZs, etc.
Public API Security
-------------------
Since the file will be transmitted clear text across the management
network, there is a chance that the module can be intercepted if the network
is compromised.
It should be ensured that each plugin created does not 'execute' the contents
of the supplied module data file, as this would present the opportunity for a
security breach. This seems unlikely though (and will not be the case for the
proposed implementations) as most module data files will be passed to another
process for validation, and it is up to that process to ensure proper security
is maintained. Code reviews will be vital to make sure no plugin accidentally
executes this data.
Python API
----------
New methods will be added to the Python API to facilitate the licensing.
A few existing methods will need to be extended as well.
Module Maintenance
..................
.. code-block:: python
def module_list(self, datastore=None):
"""Get a list of all modules that can be applied. Return only
those that apply to the datastore if it is passed in.
"""
def module_list_instances(self, module):
"""Get a list of all instances that have a given module applied."""
def module_show(self, module):
"""Show the details of the module."""
def module_create(self, module_type, name, description, contents,
datastore, datastore_version='all', auto_apply=False,
all_tenants=False, visible=True, live_update=False):
"""Create a new module."""
def module_update(self, module, module_type=None, name=None,
description=None, contents=None, datastore=None,
datastore_version=None, auto_apply=None,
all_tenants=None, visible=None, live_update=None):
"""Update an existing module."""
def module_delete(self, module):
"""Delete a module."""
Instance Interaction
....................
.. code-block:: python
def module_apply(self, instance, modules):
"""Apply modules to an instance."""
def module_query(self, instance):
"""Query an instance about installed modules."""
def module_retrieve(self, instance, module=None, filename=None):
"""Retrieve the module data file from an instance and save it in
filename. If module is not supplied, retrieve all the modules.
If filename is not supplied, use the generated filename found
on the instance.
"""
def module_remove(self, instance, module):
"""Remove a module from an instance."""
Creation Enhancements
.....................
For instance.create, the modules field will be added to the call:
.. code-block:: python
def create(self, name, flavor_id, volume=None, databases=None, users=None,
restorePoint=None, availability_zone=None, datastore=None,
datastore_version=None, nics=None, configuration=None,
replica_of=None, slave_of=None, replica_count=None,
modules=None):
"""Create (boot) a new instance."""
For cluster.create, the modules field will be added to the
['cluster']['instances'] data structure that is already being passed in.
CLI (python-troveclient)
------------------------
The following Trove CLI commands (upon completion) will be fully functional
- module-list Displays all modules for the tenant.
- module-show Shows details for a particular module resource.
- module-create Creates a new module resource.
- module-update Updates module details for a particular module
resource.
- module-delete Delete a module resource.
- module-apply Apply the given modules to a Trove instance.
- module-query Query the given Trove instance for any installed
modules.
- module-retrieve Retrieves the current modules from a Trove instance.
- module-remove Remove a module from a Trove instance.
- create --module [--module]
Creates a new instance and applies the given modules.
- cluster-create --instance=module=<id>[,module=<id>]
Creates a new cluster and applies the given modules to
each instance.
Internal API
------------
Changes also need to be made to the internal API to include any module IDs as
a part of the message body that is sent to the task manager.
The API server will need to make calls to the Guest Agent for the instance
interaction type commands.
Guest Agent
-----------
In the Guest Agent, the modules will be managed with a plugin style
architecture based on the stevedore.driver.DriverManager paradym. Each plugin
will need to implement 'apply', 'query' and 'remove' actions. The 'query'
action will need to report the status of the module 'apply' action. This
would report (at a minimum) 'OK' or 'FAILED' plus any other state that seems
reasonable for users of the relevant software. If possible, the
'error_message' field should be filled with useful information if an error
occurs.
A simple plugin 'base class' that defines the contract will be provided. It
will also provide functionality such as placing the file contents into a
specified location and retrieving the file will be added. This can be used
as the basis for all other plugins.
The Guest Agent code will use the module 'type' to determine if a plugin exists
for the given module. If no plugin can be found, then an error will be written
to the log and processing stopped.
To provide a concrete, real-world plugin implementation, a Vertica license
module plugin will be created to allow licenses to be applied to a Vertica
datastore. A New Relic plugin will also be created to illustrate activation of
other third party software on a guest image.
Alternatives
------------
None
Dashboard Impact (UX)
=====================
A multi-dropdown will need to be added to the instance create dialog that
contains all modules for the selected datastore. These modules, along with
any auto-apply ones, will need to be sent along on the create call. The same
will be needed for the cluster create dialog.
A module detail panel will need to be created. This panel will have fields
representing the attributes of a module (see module-create command).
A 'modules' list panel will need to be created. This will have buttons for
'delete' and 'update' and will have a link to the detail page for each listed
module. This will be a high-level panel, similar to 'Instances.'
The instance list panel will need to have a new action added: 'apply module.'
This will cause a pop-up where the available modules are displayed. The
selected module will then be passed in to the module-apply command.
The instance detail panel will need to run 'module-query' and display the
results in a new section 'modules.' Alternately, a link could be placed here
that would open a module list panel with the results of the 'module-query'
call. Here, buttons for 'module-remove' and 'module-retrieve' would be
needed.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
[peterstac]
Milestones
----------
Mitaka
Work Items
----------
The work will be undertaken with the following tasks:
* Client (Python and CLI) changes
* Server (API) changes
* Guest Agent module plugin infrastructure
* Vertica/New Relic plugin implementation
Upgrade Implications
====================
Since this change is net-new, no upgrade issues are expected.
Dependencies
============
None.
Testing
=======
Generic int-tests will be written, however these will not be run under MySQL
testing as it requires no module-based handling.
Documentation Impact
====================
This is a net-new feature, and as such will require documentation.
References
==========
.. [1] nrsysmond-config --set license_key=<new_relic_key>.
Appendix
========
None