15 KiB
Upload serialized deployment facts to ConfigDB service API
https://blueprints.launchpad.net/fuel/+spec/upload-deployment-facts-to-configdb
There are multiple levels of hierarchy in Hiera used by deployment tasks on nodes. Some of those data exist only in YaML files on a node and can't be accessed by 3rd party components.
With configuration database service, we can store serialized deployment data for later use. It requires though that we can upload the data from nodes to the service as a part of deployment process.
We propose to develop a deployment task to upload the facts to external HTTP-based API and add it to a plugin that enables 3rd-party LCM application with Fuel1.
Problem description
The store for serialized deployment information (e.g. ConfigDB API extension in Nailgun2) allows 3rd party applications to access it. It also allows alternative deployment/lifecycle management solutions to synchronize their configrations with Fuel installer.
However, it doesn't solve the problem of getting the information into the service, since the extension itself is a more or less passive store.
The solution is needed to perform actual upload of required information into the configuration database. It also must keep the data up to date by synchronizing them upon every change applied to the environment.
Synchronization required in the following cases:
- Deployment settings changed in Nailgun via UI/CLI/API. In this case,
Nailgun DB will have the latest changes, and Nailgun API will respond
with properly updated serialized deployment data3.
This data can be imported into ConfigDB directly by requesting the
Nailgun API and sending result to ConfigDB API. They are accessible by
Hiera as
astute
data source4. - Deployment data changed due to changes made to the node (e.g. hardware updated, versions of packages updated, etc) outside the Fuel context. These changes are reflected in the serialized data generated and stored on the node itself, in YaML files:
/etc/hiera/globals.yaml
- global configurations calculated by deployment taskglobals
./etc/hiera/override/plugins.yaml
- plugin-specific overrides of parameters defined in data sources on higher levels (i.e.astute
andglobals
)./etc/hiera/override/configuration.yaml
- specific overrides for OpenStack configuration parameters which are not exposed by Nailgun directly./etc/hiera/override/<hostname>.yaml
- node-level configurations that override the basic parameters from other sources.
- Deployment data changed in 3rd party deployment/lifecycle management application (e.g. in Puppet Master's top-level manifests or in External Nodes Classifier application for the Puppet Master). Here we need to import data from the 3rd party application. This case is out of scope of the current proposal.
In this specification, we will focus on the use cases #1 and #2.
Proposed changes
The current deployment process implemented in Fuel installer assumes
that all actual deployment data is available to Puppet agent on a target
node locally as a set of YaML files in /etc/hiera
directory.
By the time Puppet agent starts to execute actual deployment tasks, all the configuration settings must be up to date. It means that we can import the actual set of the deployment configuration data from those files as a part of deployment process.
The following changes are proposed in scope of this specification:
- Create a deployment task that uploads the serialized deployment data
from files in
/etc/hiera
of a target node to the corresponding resources in ConfigDB API endpoint (e.g./environment/<:id>/node/<:id>/resource/<:datasource>/values
). See Orchestration section for details. - Integrate the a task into the deployment graph using plugins
mechanisms of Fuel. The task must run in the end of the deployment
process to make sure that all the tasks that impact the deployment
settings files are already executed, including:
globals
override_configuration
- any plugin task that updates/overrides basic deployment settings
- Implement a pass of auth information for ConfigDB API extension of Nailgun API to the deployment task in question in a secure way.
Deployment task details
The deployment task shall have the following ID:
upload_data_to_configdb
The task shall depend on the following tasks:
globals
upload_configuration
pre_deployment_end
The task shall run at the Fuel Master node. Auth credentials for the ConfigDB API shall be made available for it upon LCM plugin installation from the following file:
/var/www/nailgun/plugins/<lcm-plugin-name>/environment_config.yaml
That file will be created by the plugin builder and will contain metadata of the LCM application, including auth credentials in question.
The task shall perform the following operations using ConfigDB API:
- Verify if the environment
env_id
exists in the ConfigDB API.- If not, create an environment with
POST
request. It should contain a list of data sources to create for the environment. See details in ConfigDB API specification5. - The list of data sources is fetched from
astute.yaml
file, from parameterdata_sources
of Puppet Master LCM plugin's metadata.
- If not, create an environment with
- Read data from files in
/etc/hiera
directory viamcollective
client into internal variable of the deployment task. - Upload data to ConfigDB API's data sources based on the filenames from which the data was read.
- Read the effective data from ConfigDB API (i.e. values
calculated by merging uploaded data with configured overrides at all
levels or hierarchy) and write the data to files in
/etc/hiera
directory viamcollective
client.
The latter function ensures that the deployment process accounts for data overriden in ConfigDB. This is required for 2nd day operations based on the Fuel workflow, specifically for scale-out opertaions (adding a node to the cluster) and replacement of failed nodes.
Auth mechanism details
ConfigDB API extension as a part of Nailgun API uses Keystone to verify and authorize users.
Before installing an environment with the 3rd-party LCM plugin, user must add a service account for the plugin in that environment with Keystone CLI. For example:
$ keystone user-create --name=lcm-plugin --tenant=admin
While configuring environment, the user enables the LCM plugin and configures the service account credentials for the plugin via Fuel UI or API.
Plugin uses these access creates credentials to configure deployment
task upload_data_to_configdb
and the custom Hiera
backend.
Example workflow
The following example illustrates the workflow of the solution:
Assume that the User intends to use 3rd-party application to perform some tasks, for example, LCM operations, on an OpenStack environment deployed by Fuel.
- User installs the Fuel Master node with the ConfigDB extension. The extension is installed as an RPM on top of the existing system.
- User installs a plugin for LCM operations that should include
components to upload deployment data to ConfigDB API (e.g. deployment
task
upload_data_to_configdb
) and to perform lookup for certain parameters in ConfigDB API (e.g. custom Hiera backend).
User configures OpenStack environment using Fuel UI. Nailgun creates metadata for the environment and individual nodes.
- The deployment data for the environment and nodes is accessible via
Nailgun API by URIs
/cluster/<:id>/orchestrator/deployment
and/cluster/<:id>/orchestrator/deployment/default
. - Deployment data for specific node are exposed via the same URIs with
addition of parameter
?node=<:node_id>
to the URI path.
- The deployment data for the environment and nodes is accessible via
Nailgun API by URIs
User deploys the environment as usual via Fuel UI or CLI.
- Deployment task
upload_data_to_configdb
runs on every node in the environment and uploads serialized deployment data from YaML files in/etc/hiera/
directory to ConfigDB API. - Another deployment task configures the node to work with 3rd party LCM tool. This might or might not include disable of the ordinary Fuel means of deployment.
- Deployment task
Afterwards the User makes changes to the environment configuration using 3rd-party LCM tool.
- User changes or extends the deployment settings by assigning values
to parameters via ConfigDB API, for example, changes
keystone_url
parameter inglobals
data source. - ConfigDB saves the override data to an override for the data source
globals
. - User triggers 3rd party application which reads the effective data (i.e. raw uploaded values with applied overrides) from ConfigDB API and applies changes to all affected nodes.
- User changes or extends the deployment settings by assigning values
to parameters via ConfigDB API, for example, changes
In future, the User adds another node to the environment and deploys it using standard Fuel methods.
- Deployment data for the new node provided by Nailgun's standard serializers.
- When the deployment is initiated, the task
upload_data_to_configdb
synchronizesastute.yaml
file created at the node by Astute with data overrides created in ConfigDB API: it uploads the contents of the file to ConfigDB API and then downloads effective data for theasuste
data source from there. - After the pre-deployment finishes, the prepared deployment data are
synchronized to the ConfigDB API by task
upload_data_to_configdb
. This ensures that override settings from all data sources in the ConfigDB API are applied to all files in/etc/hiera
at the node. - Deployment is done by Fuel standard deployment tasks, but with settings adjusted with overrides from ConfigDB.
- After successful deployment, the LCM plugin reconfigures the node to work with 3rd-party LCM tool.
Web UI
None.
Nailgun
None.
Data model
None.
REST API
None.
Orchestration
A new deployment task shall be added to ensure that all changes to
files in /etc/hiera
directory are synchronized with the
ConfigDB.
The task shall send a series of requests to the URI of the resource in ConfigDB based on the parameters of the deployment:
<:service_uri>/environments/<:env_id>/nodes/<:node_id>/resources/<:datasource>/values
service_uri
is a endpoint from Keystone Service Catalog, defaults to/api/v1/config
.env_id
is an identifier of cluster the node belongs to. The ID of environment shall be fetched from deployment factdeployment_id
.node_id
is an identifier of the node. It should be matched in Nailgun API by the node'sFQDN
scope recieved from Puppet Master.datasource
is a name of the data source.
The task will: #. upload data to /values
of the data
source's resource and
- download effective data from
/values?effective
and write it to files.
See detailed description of the API in corresponding specification.6
RPC Protocol
None.
Fuel Client
None.
Fuel Library
None.
Alternatives
The alternative way to keep deployment data from nodes in sync with ConfigDB is to upload data to API from deployment tasks.
While it is possible to adjust globals
and
openstack_config
tasks to upload configuration data to
external service, it is generally impossible to do with all supported
plugins.
A plugin can override default values in astute.yaml
generated by the Nailgun-provided serialized data. However, this
overrides are configured by plugin tasks on a per-node basis. Override
information is not available to Nailgun or even Astute directly. So, to
ensure sync of plugins' override data we need to modify each and every
plugin, which apparently is not an option.
Another way to keep data in sync is to upload it from some bottom-level catch-all Astute post-deployment task. This would allow to keep Nailgun/ConfigDB credentials limited to the Master node and not expose them to target nodes in the deployment.
On the other hand, there was a work done on Astute to convert its
tasks into standard deployment tasks in fuel-library
. Thus,
we should net add new tasks to Astute in this proposal.
Upgrade impact
None.
Security impact
Sensitive configuration data, such as passwords and access credentials, shall be uploaded to the ConfigDB API using proposed functions. It is recommended to use encrypted HTTP protocol to transfer these data.
The approach to authentication of the plugin's application with Nailgun API assumes that the user is responsible for configuring access credentials for the plugin applications in Keystone. The user is also responsible for configuring proper credentials for the plugin when installing an environment with 3rd-party LCM application.
Notifications impact
None.
End user impact
None.
Performance impact
The deployment task proposed in this spec will take some time to upload all data to the ConfigDB API. Moreover, if many nodes trying to write to the same API endpoint at the same time, it might significantly affect the overall duration of deployment.
Deployment impact
None.
Developer impact
None.
Infrastructure impact
The deployment task is packaged as a part of 3rd-party LCM plugin.
Documentation impact
None.
Implementation
Assignee(s)
- Primary assignee:
-
<gelbuhos> Oleg S. Gelbukh
- Other contributors:
-
<sryabin> Sergey Ryabin
- Mandatory design review:
-
<rustyrobot> Evgeniy Li <ikalnitsky> Igor Kalnitsky <vsharshov> Vladimir Sharshov <vkuklin> Vladimir Kuklin
Work Items
- Develop deployment task as a part of Puppet Master LCM plugin code base7.
- Develop unit tests for the deployment task in the plugin's code base.
- Develop automated integration tests for the plugin in
openstack/fuel-qa
repository.
Dependencies
- ConfigDB API implementation as Nailgun extension8
Testing, QA
- The feature shall be tested in conjunction with ConfigDB API feature9
- Tests shall verify that contents of data sources are consistent with
contents of files in
/etc/hiera
at nodes after the deployment finishes.
Acceptance criteria
- Deployment data from nodes uploaded to corresponding data sources in ConfigDB API upon successful deployment of the OpenStack environment.
References
Puppet Master LCM plugin specification TBD↩︎
Nailgun API extension for serialized deployment facts https://review.openstack.org/#/c/284109/↩︎
Nailgun API extension for serialized deployment facts https://review.openstack.org/#/c/284109/↩︎
Nailgun API for Deployment Information https://github.com/openstack/fuel-web/blob/master/nailgun/nailgun/api/v1/handlers/orchestrator.py#L190↩︎
Nailgun API extension for serialized deployment facts https://review.openstack.org/#/c/284109/↩︎
Nailgun API extension for serialized deployment facts https://review.openstack.org/#/c/284109/↩︎
Puppet Master LCM plugin specification TBD↩︎
Nailgun API extension for serialized deployment facts https://review.openstack.org/#/c/284109/↩︎
Nailgun API extension for serialized deployment facts https://review.openstack.org/#/c/284109/↩︎