61a353cbd7
1. the pymysql is needed for the current release[0] 2. it should use keystone_database_address rather than keystone_service_ip, because they may different. [0]: https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/keystone/templates/keystone.conf.j2#L22 Change-Id: I7eb90544bd6be167df157ca352ca01e21ce781bd
318 lines
13 KiB
ReStructuredText
318 lines
13 KiB
ReStructuredText
..
|
|
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
|
License.
|
|
|
|
http://creativecommons.org/licenses/by/3.0/legalcode
|
|
|
|
==================
|
|
Multi-node Ansible
|
|
==================
|
|
|
|
This blueprint specifies an approach to automate the deployment of OpenStack
|
|
using Ansible and Docker best practices. The overriding principles used in
|
|
this specification are simplicity, flexibility and optimized deployment speed.
|
|
|
|
Problem description
|
|
===================
|
|
|
|
Kolla can be deployed multi-node currently. To do so, the environment
|
|
variables must be hand edited to define the hosts to connect to for various
|
|
services.
|
|
|
|
To implement HA in our containers, we need multi-node deployment operational
|
|
so we can validate the high availability implementation.
|
|
|
|
To meet our community approved mission, we must implement a deployment tool
|
|
and Ansible provides the most efficient development path while ensuring ease
|
|
of use.
|
|
|
|
Use cases
|
|
---------
|
|
|
|
1. Deploy Kolla's Docker containers multi-node and generate a one-hundred node
|
|
working OpenStack deployment out of the box with minimal Operator
|
|
configuration.
|
|
2. Offer a fully customizable configuration enabling an unopinionated
|
|
deployment of OpenStack.
|
|
3. Upgrade an OpenStack deployment by Operator command.
|
|
4. Offer our container content as a building block to third party downstream
|
|
deployment tools.
|
|
|
|
Proposed change
|
|
===============
|
|
|
|
The docker-compose tool is single node and does nearly the same job as Ansible
|
|
would in this specification. As a result, we recommend deprecating
|
|
docker-compose as the default deployment system for Kolla.
|
|
|
|
To replace it, we recommend Ansible as a technology choice. Ansible is easy
|
|
to learn, easy to use, and offers a base set of functionality to solve
|
|
deployment as outlined in our four use cases.
|
|
|
|
We recommend three models of configuration.
|
|
|
|
The first model is based upon internally configuring the container and having
|
|
the container take responsibility for all container configuration including
|
|
database setup, database synchronization, and keystone registration. This
|
|
model uses docker-compose and docker as dependencies. Existing containers will
|
|
be maintained but new container content will use either of the two remaining
|
|
models. James Slagle (TripleO PTL on behalf of our downstream TripleO
|
|
community) was very clear that he would prefer to see this model stay available
|
|
and maintained. As TripleO enters the world of Big Tent, they don't intend to
|
|
deploy all of the services, and as such it doesn't make sense to maintain this
|
|
legacy operational mode for new container content except on demand of our
|
|
downstreams, hopefully with their assistance. This model is called
|
|
CONFIG_INSIDE.
|
|
|
|
The second model and third model configure the containers outside of the
|
|
container. These models depend on Ansible and Docker. In the future, the
|
|
OpenStack Puppet, OpenStack Chef and TripleO communities may decide to switch
|
|
to one of these two models in which case these communities may maintain tooling
|
|
to integrate with Kolla. The major difference between these two models is that
|
|
one offers immutability and single source of truth (CONFIG_OUTSIDE_COPY_ONCE),
|
|
while the third model trades these two properties to allow an Operator to
|
|
directly modify configuration files on a system and have the configuration be
|
|
live in the container (CONFIG_OUTSIDE_COPY_ALWAYS). Because
|
|
CONFIG_OUTSIDE_COPY_ALWAYS requires direct Operator intervention on a node, and
|
|
we prefer as a community Operators interact with the tools provided by Kolla,
|
|
CONFIG_OUTSIDE_COPY_ONCE will be the default.
|
|
|
|
We do not have to further enhance two sets of container configuration, but
|
|
instead can focus our development effort on the default Ansible configuration
|
|
methods. If a defect is found in one of the containers based upon the
|
|
CONFIG_INSIDE model, the community will repair it.
|
|
|
|
Finally we will implement a complete Ansible deployment system. The details
|
|
of the implementation are covered in a later section in this specification.
|
|
We estimate this will be approximately ~1000 LOC defining ~100 Ansible tasks.
|
|
We further estimate the total code base when complete will be under 6 KLOC.
|
|
|
|
The CONFIG_INSIDE model of configuration maintains the immutable,
|
|
declarative, and idempotent nature of the Kolla containers, as defined by our
|
|
current Kolla best practices but is opinionated in configuration.
|
|
|
|
The CONFIG_OUTSIDE_COPY_ONCE model of configuration maintains the immutable and
|
|
declarative nature of the Kolla containers, as defined by our current Kolla
|
|
best practices while introducing completely customizable configuration.
|
|
|
|
The CONFIG_OUTSIDE_COPY_ALWAYS model of configuration offers the Operator
|
|
greater flexibility in managing their deployment, at greater risk of damaging
|
|
their deployment. It trades one set of best practices for another,
|
|
specifically the Kolla container best practices for flexibility.
|
|
|
|
Security impact
|
|
---------------
|
|
|
|
None.
|
|
|
|
Performance Impact
|
|
------------------
|
|
|
|
Multi-node deployment speed will be rapidly improved.
|
|
|
|
Implementation
|
|
==============
|
|
|
|
The following section uses the Keystone container as an example.
|
|
|
|
On container start, a simple shell script will be run.
|
|
|
|
Passed into the container will be the CONFIG_STRATEGY environment variable with
|
|
the following options:
|
|
|
|
CONFIG_STRATEGY="CONFIG_INSIDE"
|
|
CONFIG_STRATEGY="CONFIG_OUTSIDE_COPY_ALWAYS"
|
|
CONFIG_STRATEGY="CONFIG_OUTSIDE_COPY_ONCE"
|
|
|
|
CONFIG_INSIDE will match the current crudini.sh implementation.
|
|
|
|
The shell script will be similar in nature to the following:
|
|
|
|
case $CONFIG_STRATEGY in
|
|
CONFIG_INSIDE)
|
|
# We exec on crudini.sh to keep the same behaviour as current
|
|
exec /crudini.sh
|
|
;;
|
|
CONFIG_OUTSIDE_COPY_ONCE|CONFIG_OUTSIDE_COPY_ALWAYS)
|
|
# We source this file to allow variables to be set if needed
|
|
source /config_outside.sh
|
|
;;
|
|
*)
|
|
echo '$CONFIG_STRATEGY is not set properly'
|
|
exit 1
|
|
;;
|
|
esac
|
|
|
|
exec $CONFIG_STRATEGY_BINARY_NAME
|
|
|
|
The crudini.sh script would be almost identical to the existing start.sh script
|
|
while the config_outside.sh would copy the files to the appropriate location
|
|
and set the proper permissions on those files. The $CONFIG_STRATEGY variable
|
|
would be checked to see if the files should be copied or it should exit early.
|
|
|
|
The following bindmounts would be applied to the container in the above example
|
|
for different CONFIG_STRATEGY values:
|
|
|
|
CONFIG_INSIDE - no bind mount
|
|
CONFIG_OUTSIDE_COPY_ONCE - {{ HOST_CONFIG_DIR }}/keystone:/opt/kolla/configs/keystone:ro
|
|
CONFIG_OUTSIDE_COPY_ALWAYS - {{ HOST_CONFIG_DIR }}/keystone:/opt/kolla/configs/keystone:ro
|
|
|
|
{{ HOST_CONFIG_DIR }} would be an Ansible variable with the default of
|
|
/opt/kolla/configs. This same pattern will be used for most containers, unless
|
|
there is a compelling technical reason not to do so.
|
|
|
|
An Ansible role represents a service in OpenStack. The Ansible role contains
|
|
3 major sections. This same pattern will be used for all supported
|
|
OpenStack containers.
|
|
|
|
Each Ansible role has a set of default key/value pairs. An example key/value
|
|
file for Keystone is:
|
|
|
|
---
|
|
container: "keystone"
|
|
database_password: "{{ database_keystone_password }}"
|
|
|
|
|
|
The second major section of a Ansible role are the role tasks. The seven
|
|
tasks we will implement per role (i.e. OpenStack Service):
|
|
|
|
* bootstrap - database initialization and add roles to keystone
|
|
* pull - pulls the latest container from the registry
|
|
* main - Does the main job of orchestrating the role
|
|
* config - Joins the default configuration and the user augmented
|
|
configuration and saves the resulting file to be bind-mounted
|
|
* start - Similar in nature to a docker compose YAML file - defines the
|
|
defaults for the container start operation.
|
|
* stop - Stops the container
|
|
* upgrade - Upgrades to the latest container content
|
|
|
|
The details of how these role tasks operate is an implementation detail.
|
|
|
|
Finally each Ansible role has a default template. An example of a default
|
|
template for Keystone is:
|
|
|
|
[DEFAULT]
|
|
verbose = {{ keystone_verbose }}
|
|
debug = {{ keystone_debug }}
|
|
|
|
bind_host = {{ ansible_br_mgmt['ipv4']['address'] }}
|
|
|
|
admin_token = {{ keystone_admin_token }}
|
|
|
|
public_endpoint = http://{{ keystone_service_ip }}:{{ keystone_service_public_port }}
|
|
admin_endpoint = http://{{ keystone_service_ip }}:{{ keystone_service_admin_port }}
|
|
|
|
log_file = {{ keystone_log_file }}
|
|
log_dir = {{ keystone_log_dir }}
|
|
|
|
[database]
|
|
connection = mysql+pymysql://{{ keystone_db_user }}:{{ database_keystone_password }}@{{ keystone_database_address }}/keystone
|
|
|
|
[revoke]
|
|
driver = keystone.contrib.revoke.backends.sql.Revoke
|
|
|
|
This role default will contain sufficient mandatory configuration options to
|
|
create a working deployment. If the Operator wishes to augment the Keystone
|
|
configuration, an augmentation file can be added to the deployment. An example
|
|
augmentation file in /etc/kolla/keystone.aug is:
|
|
|
|
[DEFAULT]
|
|
public_endpoint = https://{{ keystone_service_ip }}:{{ keystone_service_public_port }}
|
|
|
|
[ipman]
|
|
life = "Two Words. Horizontal. Vertical. Make a mistake - Horizontal. Stay standing and you win."
|
|
|
|
This augmentation file will keep the original default configurations but
|
|
replace public_endpoint with an https endpoint instead of an http endpoint.
|
|
Further the [ipman] section will be added to the file placed by Ansible in
|
|
the target host's configuration directory.
|
|
|
|
The end result of the merge will be a single file on the host that is in the
|
|
appropriate format for the OpenStack service to consume containing the content
|
|
of both the Ansible default file and the augmentation file.
|
|
|
|
The final implication of these Ansible best practices is that an Operator can
|
|
deploy in 1 hour or less a one-hundred node OpenStack deployment out of the
|
|
box using Kolla containers with Ansible deployment tooling with minimal
|
|
configuration. If additional customization is required for the Operator's
|
|
environment, this can be achieved via augmentation files developed by the
|
|
Operator.
|
|
|
|
NB: to override any default key/value pair (the key is located in {{ }} above
|
|
and replaced by the value by Ansible), there is one global override file to
|
|
configure the deployment called /etc/kolla/globals.yml
|
|
|
|
We will implement a simple shell script called kolla-ansible which wraps
|
|
ansible-playbook. It will implement four commands which operate on the
|
|
OpenStack deployment globally. It will automatically load the globals.yml
|
|
overrides and an inventory file located in /etc/kolla executing the appropriate
|
|
roles for all of the deployed containers. The initial supported
|
|
commands are:
|
|
|
|
1. kolla-ansible deploy
|
|
2. kolla-ansible start
|
|
3. kolla-ansible stop
|
|
4. kolla-ansible upgrade
|
|
|
|
Ansible supports a model of deployment using an inventory file. The inventory
|
|
file specifies which nodes get assigned which roles. For an example of an
|
|
inventory file, see:
|
|
|
|
https://github.com/SamYaple/yaodu/blob/master/ansible/inventories/production
|
|
|
|
To the untrained eye, this looks like a bunch of heavy wizardy. I personally
|
|
believe we will in some way merge our globals.yml and the inventory file into
|
|
one master configuration file and generate the globals.yml and Ansible-specific
|
|
inventory file on each kolla-ansible operation. The long term goal is to get
|
|
to one configuration file with "all the things" needed to deploy OpenStack.
|
|
This would permit a GUI to simply configure the deployment.
|
|
|
|
How this is done or if it is done remains an implementation detail which may
|
|
warrant expanding this specification or a completely new specification in the
|
|
future. As we obtain more experience with what we are developing, we will
|
|
have a more complete picture of what this master configuration file format will be.
|
|
|
|
The implementation described in this section is just a sample of the
|
|
implementation details required. We intend to refactor Sam Yaple's fantastic
|
|
vision with yaodu (https://github.com/SamYaple/yaodu/) into Kolla to
|
|
implement Ansible deployment of OpenStack while retaining Kolla, Docker, and
|
|
Ansible best practices and conventions.
|
|
|
|
Assignee(s)
|
|
-----------
|
|
|
|
Primary assignees:
|
|
|
|
diga
|
|
fangfenghua
|
|
harmw
|
|
samyaple
|
|
sdake
|
|
|
|
The kolla core team will support and execute this specification through normal
|
|
workflow operations.
|
|
|
|
Work Items
|
|
----------
|
|
|
|
1. Convert all fat containers to thin containers to facilitate this work.
|
|
2. Move all start.sh scripts to crudini.sh and create the function to execute
|
|
the configuration strategy across containers.
|
|
3. Rename the kolla script to kolla-compose and create a new kolla-ansible
|
|
script to manage playbook operation.
|
|
4. Refactor the remaining portions of yaodu that are compatible with Kolla into
|
|
the Kolla code base.
|
|
5. Implement our existing crudini defaults in Ansible.
|
|
|
|
Testing
|
|
=======
|
|
|
|
Functional tests will be implemented in the OpenStack check/gating system to
|
|
automatically check that the Ansible deployment works for an AIO environment.
|
|
|
|
Documentation Impact
|
|
====================
|
|
|
|
The developer quickstart must be augmented with instructions to use the new
|
|
Ansible deployment methodology.
|