kolla/specs/ansible-multi.rst

Multi-node Ansible

This blueprint specifies an approach to automate the deployment of OpenStack using Ansible and Docker best practices. The overriding principles used in this specification are simplicity, flexibility and optimized deployment speed.

Problem description

Kolla can be deployed multi-node currently. To do so, the environment variables must be hand edited to define the hosts to connect to for various services.

To implement HA in our containers, we need multi-node deployment operational so we can validate the high availability implementation.

To meet our community approved mission, we must implement a deployment tool and Ansible provides the most efficient development path while ensuring ease of use.

Use cases

1. Deploy Kolla's Docker containers multi-node and generate a one-hundred node working OpenStack deployment out of the box with minimal Operator configuration.
2. Offer a fully customizable configuration enabling an unopinionated deployment of OpenStack.
3. Upgrade an OpenStack deployment by Operator command.
4. Offer our container content as a building block to third party downstream deployment tools.

Proposed change

The docker-compose tool is single node and does nearly the same job as Ansible would in this specification. As a result, we recommend deprecating docker-compose as the default deployment system for Kolla.

To replace it, we recommend Ansible as a technology choice. Ansible is easy to learn, easy to use, and offers a base set of functionality to solve deployment as outlined in our four use cases.

We recommend three models of configuration.

The first model is based upon internally configuring the container and having the container take responsibility for all container configuration including database setup, database synchronization, and keystone registration. This model uses docker-compose and docker as dependencies. Existing containers will be maintained but new container content will use either of the two remaining models. James Slagle (TripleO PTL on behalf of our downstream TripleO community) was very clear that he would prefer to see this model stay available and maintained. As TripleO enters the world of Big Tent, they don't intend to deploy all of the services, and as such it doesn't make sense to maintain this legacy operational mode for new container content except on demand of our downstreams, hopefully with their assistance. This model is called CONFIG_INSIDE.

The second model and third model configure the containers outside of the container. These models depend on Ansible and Docker. In the future, the OpenStack Puppet, OpenStack Chef and TripleO communities may decide to switch to one of these two models in which case these communities may maintain tooling to integrate with Kolla. The major difference between these two models is that one offers immutability and single source of truth (CONFIG_OUTSIDE_COPY_ONCE), while the third model trades these two properties to allow an Operator to directly modify configuration files on a system and have the configuration be live in the container (CONFIG_OUTSIDE_COPY_ALWAYS). Because CONFIG_OUTSIDE_COPY_ALWAYS requires direct Operator intervention on a node, and we prefer as a community Operators interact with the tools provided by Kolla, CONFIG_OUTSIDE_COPY_ONCE will be the default.

We do not have to further enhance two sets of container configuration, but instead can focus our development effort on the default Ansible configuration methods. If a defect is found in one of the containers based upon the CONFIG_INSIDE model, the community will repair it.

Finally we will implement a complete Ansible deployment system. The details of the implementation are covered in a later section in this specification. We estimate this will be approximately ~1000 LOC defining ~100 Ansible tasks. We further estimate the total code base when complete will be under 6 KLOC.

The CONFIG_INSIDE model of configuration maintains the immutable, declarative, and idempotent nature of the Kolla containers, as defined by our current Kolla best practices but is opinionated in configuration.

The CONFIG_OUTSIDE_COPY_ONCE model of configuration maintains the immutable and declarative nature of the Kolla containers, as defined by our current Kolla best practices while introducing completely customizable configuration.

The CONFIG_OUTSIDE_COPY_ALWAYS model of configuration offers the Operator greater flexibility in managing their deployment, at greater risk of damaging their deployment. It trades one set of best practices for another, specifically the Kolla container best practices for flexibility.

Security impact

None.

Performance Impact

Multi-node deployment speed will be rapidly improved.

Implementation

The following section uses the Keystone container as an example.

On container start, a simple shell script will be run.

Passed into the container will be the CONFIG_STRATEGY environment variable with the following options:

CONFIG_STRATEGY="CONFIG_INSIDE" CONFIG_STRATEGY="CONFIG_OUTSIDE_COPY_ALWAYS" CONFIG_STRATEGY="CONFIG_OUTSIDE_COPY_ONCE"

CONFIG_INSIDE will match the current crudini.sh implementation.

The shell script will be similar in nature to the following:

case $CONFIG_STRATEGY in

CONFIG_INSIDE)

# We exec on crudini.sh to keep the same behaviour as current exec /crudini.sh ;;

CONFIG_OUTSIDE_COPY_ONCE|CONFIG_OUTSIDE_COPY_ALWAYS)

# We source this file to allow variables to be set if needed source /config_outside.sh ;;

*)

echo '$CONFIG_STRATEGY is not set properly' exit 1 ;;

esac

exec $CONFIG_STRATEGY_BINARY_NAME

The crudini.sh script would be almost identical to the existing start.sh script while the config_outside.sh would copy the files to the appropriate location and set the proper permissions on those files. The $CONFIG_STRATEGY variable would be checked to see if the files should be copied or it should exit early.

The following bindmounts would be applied to the container in the above example for different CONFIG_STRATEGY values:

CONFIG_INSIDE - no bind mount CONFIG_OUTSIDE_COPY_ONCE - {{ HOST_CONFIG_DIR }}/keystone:/opt/kolla/configs/keystone:ro CONFIG_OUTSIDE_COPY_ALWAYS - {{ HOST_CONFIG_DIR }}/keystone:/opt/kolla/configs/keystone:ro

{{ HOST_CONFIG_DIR }} would be an Ansible variable with the default of /opt/kolla/configs. This same pattern will be used for most containers, unless there is a compelling technical reason not to do so.

An Ansible role represents a service in OpenStack. The Ansible role contains 3 major sections. This same pattern will be used for all supported OpenStack containers.

Each Ansible role has a set of default key/value pairs. An example key/value file for Keystone is:

---container: "keystone" database_password: "{{ database_keystone_password }}"

The second major section of a Ansible role are the role tasks. The seven tasks we will implement per role (i.e. OpenStack Service):

• bootstrap - database initialization and add roles to keystone
• pull - pulls the latest container from the registry
• main - Does the main job of orchestrating the role
• config - Joins the default configuration and the user augmented configuration and saves the resulting file to be bind-mounted
• start - Similar in nature to a docker compose YAML file - defines the defaults for the container start operation.
• stop - Stops the container
• upgrade - Upgrades to the latest container content

The details of how these role tasks operate is an implementation detail.

Finally each Ansible role has a default template. An example of a default template for Keystone is:

[DEFAULT] verbose = {{ keystone_verbose }} debug = {{ keystone_debug }}

bind_host = {{ ansible_br_mgmt['ipv4']['address'] }}

admin_token = {{ keystone_admin_token }}

public_endpoint = http://{{ keystone_service_ip }}:{{ keystone_service_public_port }} admin_endpoint = http://{{ keystone_service_ip }}:{{ keystone_service_admin_port }}

log_file = {{ keystone_log_file }} log_dir = {{ keystone_log_dir }}

[database] connection = mysql://{{ keystone_db_user }}:{{ database_keystone_password }}@{{ keystone_service_ip }}/keystone

[revoke] driver = keystone.contrib.revoke.backends.sql.Revoke

This role default will contain sufficient mandatory configuration options to create a working deployment. If the Operator wishes to augment the Keystone configuration, an augmentation file can be added to the deployment. An example augmentation file in /etc/kolla/keystone.aug is:

[DEFAULT] public_endpoint = https://{{ keystone_service_ip }}:{{ keystone_service_public_port }}

[ipman] life = "Two Words. Horizontal. Vertical. Make a mistake - Horizontal. Stay standing and you win."

This augmentation file will keep the original default configurations but replace public_endpoint with an https endpoint instead of an http endpoint. Further the [ipman] section will be added to the file placed by Ansible in the target host's configuration directory.

The end result of the merge will be a single file on the host that is in the appropriate format for the Openstack service to consume containing the content of both the Ansible default file and the augmentation file.

The final implication of these Ansible best practices is that an Operator can deploy in 1 hour or less a one-hundred node OpenStack deployment out of the box using Kolla containers with Ansible deployment tooling with minimal configuration. If additional customization is required for the Operator's environment, this can be achieved via augmentation files developed by the Operator.

NB: to override any default key/value pair (the key is located in {{ }} above and replaced by the value by Ansible), there is one global override file to configure the deployment called /etc/kolla/globals.yml

We will implement a simple shell script called kolla-ansible which wraps ansible-playbook. It will implement four commands which operate on the OpenStack deployment globally. It will automatically load the globals.yml overrides and an invetory file located in /etc/kolla executing the appropriate roles for all of the deployed containers. The initial supported commands are:

1. kolla-ansible deploy
2. kolla-ansible start
3. kolla-ansible stop
4. kolla-ansible upgrade

Ansible supports a model of deployment using an inventory file. The inventory file specifies which nodes get assigned which roles. For an example of an inventory file, see:

https://github.com/SamYaple/yaodu/blob/master/ansible/inventories/production

To the untrained eye, this looks like a bunch of heavy wizardy. I personally believe we will in some way merge our globals.yml and the inventory file into one master configuration file and generate the globals.yml and Ansible-specific inventory file on each kolla-ansible operation. The long term goal is to get to one configuration file with "all the things" needed to deploy OpenStack. This would permit a GUI to simply configure the deployment.

How this is done or if it is done remains an implementation detail which may warrant expanding this specification or a completely new specification in the future. As we obtain more experience with what we are developing, we will have a more complete picture of what this master configuration file format will be.

The implementation described in this section is just a sample of the implementation details required. We intend to refactor Sam Yaple's fantastic vision with yaodu (https://github.com/SamYaple/yaodu/) into Kolla to implement Ansible deployment of OpenStack while retaining Kolla, Docker, and Ansible best practices and conventions.

Assignee(s)

Primary assignees:

diga fangfenghua harmw samyaple sdake

The kolla core team will support and execute this specification through normal workflow operations.

Work Items

1. Convert all fat containers to thin containers to facilitate this work.
2. Move all start.sh scripts to crudini.sh and create the function to execute the configuration strategy across containers.
3. Rename the kolla script to kolla-compose and create a new kolla-ansible script to manage playbook operation.
4. Refactor the remaining portions of yaodu that are compatible with Kolla into the Kolla code base.
5. Implement our existing crudini defaults in Ansible.

Testing

Functional tests will be implemented in the OpenStack check/gating system to automatically check that the Ansible deployment works for an AIO environment.

Documentation Impact

The developer quickstart must be augmented with instructions to use the new Ansible deployment methodology.