Change structure of CCP documentation

- Change `Application definition contribution guide` by adding sections
   with description structure of component repository and links on other
   sections.
 - Make `Application definition contribution guide` upper level doc for
   description of application development.
 - Move DSL description to doc with description of DSL for application
   service.
 - Merge `Understanding globals and defaults config` and `Config Files`
   sections. It's necessary to have all configs related stuff in one
   place.
 - Move description of "Global Config" to separate doc.
 - Move and Merge files with description of config key types.

Change-Id: Icb3fc0a807b3f754f81752ee80702d8e620d1a7b
This commit is contained in:
Sergey Kraynev 2016-12-02 16:23:31 +03:00
parent b21f43005f
commit 9c2923f5a5
6 changed files with 417 additions and 367 deletions

View File

@ -4,329 +4,59 @@
Application definition contribution guide
=========================================
This guide covers CCP specific DSL, which is used by the CCP CLI to populate
k8s objects.
This document gives high overview of component repository structure.
Application definition files location
=====================================
Overview
~~~~~~~~
All application definition files should be located in the ``service/``
directory, as a ``component_name.yaml`` file, for example:
CCP provides wide spectrum of operations for microservices manipulations on
Kubernetes cluster. Each microservice is an independent component with
common architecture. Whole data releated to component can be found in the
corresponding repository. The full list of the related components can be found
by `link`_, where each repository has prefix ``fuel-ccp-*``.
::
Structure
~~~~~~~~~
Component repositories have common structure:
1. Docker image related data
----------------------------
``docker`` folder with Docker files, which will be used for building docker
images. Each subfolder will be processed as a separate image for building.
See detailed instructions are available in the :doc:`docker`.
2. Application definition files
-------------------------------
All application definition files should be located in the ``service/``
directory, as a ``component_name.yaml`` file, for example:
::
service/keystone.yaml
All templates, such as configs, scripts, etc, which will be used for this
service, should be located in ``service/<component_name>/files``, for example:
Please refer to :doc:`dsl` for detailed description of CCP DSL syntax.
::
3. Application related scripts and configs
------------------------------------------
All templates, such as configs, scripts, etc, which will be used for this
service, should be located in ``service/<component_name>/files``, for example:
::
service/files/keystone.conf.j2
All files inside this directory are Jinja2 templates. Default variables for
these templates should be located in
``service/component_name/files/defaults.yaml`` inside the ``configs`` key.
All files inside this directory are Jinja2 templates, except the file with
default variables. Default variables for these templates should be located
in ``service/files/defaults.yaml`` inside the following section.
Understanding globals and defaults config
=========================================
There are three config locations, which the CCP CLI uses:
#. ``Global defaults`` - fuel_ccp/resources/defaults.yaml in ``fuel-ccp`` repo.
#. ``Component defaults`` - service/files/defaults.yaml in each component repo.
#. ``Global config`` - Optional. Set path to this config via
"--config-file /path" CCP CLI arg. Otherwise ``fuel-ccp`` will try to find
config in next locations: ~/.ccp.yaml, ~/.ccp/ccp.yaml, /etc/ccp.yaml,
/etc/ccp/ccp.yaml
Before deployment, CCP will merge all these files into one dict, using the
order above, so "component defaults" will override "global defaults" and
"global config" will override everything.
For example, one of the common situations is to specify custom options for
networking. To achieve it user may overwrite options defined in
``Global defaults`` and ``Component defaults`` by setting new values in
``Global config``.
File ``fuel_ccp/resources/defaults.yaml`` has the following settings:
::
configs:
private_interface: eth0
public_interface: eth1
...
And part of the ``fuel-ccp-neutron/service/files/defaults.yaml`` looks like:
::
configs:
neutron:
...
bootstrap:
internal:
net_name: int-net
subnet_name: int-subnet
network: 10.0.1.0/24
gateway: 10.0.1.1
...
User may overwrite these sections by defining the following content in the
~/.ccp.yaml:
::
debug: true
configs:
private_interface: ens10
neutron:
bootstrap:
internal:
network: 22.0.1.0/24
gateway: 22.0.1.1
To validate these changes user needs to execute command ``ccp config dump``.
It will return final config file with changes, which user did. So output should
contain the following changes:
::
debug: true
...
configs:
private_interface: ens10 <----- it was changed
public_interface: eth1 <----- it wasn't changed
neutron:
bootstrap:
internal:
net_name: int-net <--- it wasn't changed
subnet_name: int-subnet <--- it wasn't changed
network: 22.0.1.0/24 <----- it was changed
gateway: 22.0.1.1 <----- it was changed
Global defaults
---------------
This is project wide defaults, CCP keeps it inside fuel-ccp repository in
``fuel_ccp/resources/defaults.yaml`` file. This file defines global variables,
that is variables that are not specific to any component, like eth interface
names.
Component defaults
------------------
Each component repository could contain a ``service/files/defaults.yaml`` file
with default config for this component only.
Global config
-------------
Optional config with global overrides for all services. Use it only if you need
to override some defaults.
Config keys types
=================
Each config could contain 3 keys:
- ``configs``
- ``versions``
- ``sources``
- ``nodes``
Each key has its own purpose and isolation, so you have to add your variable
to the right key to make it work.
configs key
-----------
Isolation:
- Used in service templates files (service/files/).
- Used in application definition file service/component_name.yaml.
Allowed content:
- Any types of variables allowed.
Example:
::
::
configs:
keystone_debug: false
<service_name>:
So you could add "{{ keystone_debug }}" variable to you templates, which will
be rendered into "false" in this case.
Description of available values can be found in the following guide
:doc:`config/index`.
versions key
------------
Isolation:
- Used in Dockerfile.j2 only.
Allowed content:
- Only versions of different software should be kept here.
For example:
::
versions:
influxdb_version: "0.13.0"
So you could add this to influxdb Dockerfile.j2:
::
curl https://dl.influxdata.com/influxdb/releases/influxdb_{{ influxdb_version }}_amd64.deb
sources key
-----------
Isolation:
- Used in Dockerfile.j2 only.
Allowed content:
- This key has a restricted format, examples below.
Remote git repository example:
::
sources:
openstack/keystone:
git_url: https://github.com/openstack/keystone.git
git_ref: master
Local git repository exaple:
::
sources:
openstack/keystone:
source_dir: /tmp/keystone
So you could add this to Dockerfile.j2:
::
{{ copy_sources("openstack/keystone", "/keystone") }}
CCP will use the chosen configuration, to copy git repository into Docker
container, so you could use it latter.
network_topology key
--------------------
Isolation:
- Used in service templates files (service/files/).
Allowed content:
- This key is auto-created by entrypoint script and populated with container
network topology, based on the following variables: ``private_interface`` and
``public_interface``.
You could use it to get the private and public eth IP address. For example:
::
bind = network_topology["private"]["address"]
listen = network_topology["public"]["address"]
nodes and roles key
-------------------
Isolation:
- Not used in any template file, only used by the CCP CLI to create a cluster
topology.
Allowed content:
- This key has a restricted format, example of this format can be found in
``fuel-ccp`` git repository in ``etc/topology-example.yaml`` file.
"CCP_*" env variables
---------------------
Isolation:
- Used in service templates files (service/files/).
Allowed content:
- This variables are created from the application definition ``env`` key.
Only env keys which start with "CCP\_" will be passed to config hash.
This is mainly used to pass some k8s related information to container, for
example, you could use it to pass k8s node hostname to container via this
variable:
Create env key:
::
env:
- name: CCP_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
Use this variable in some config:
::
{{ CCP_NODE_NAME }}
Application definition language
===============================
Please refer to :doc:`dsl` for detailed description of CCP DSL syntax.
DSL versioning
==============
Some changes in CCP framework are backward compatible and some of them are not.
To prevent situations when service definitions are being processed by
incompatible version of CCP framework, DSL versioning has been implemented.
DSL versioning is based on Semantic Versioning model. Version has a format
``MAJOR.MINOR.PATCH`` and is being defined in ``dsl_version`` field of
:file:`fuel-ccp/__init__.py` module. Each service definition contains
``dsl_version`` field with the version of DSL it was implemented/updated for.
During the validation phase of :command:`ccp deploy` those versions will be
compared according to the following rules:
#. if DSL version of ``fuel-ccp`` is less than service's DSL version -
they are incompatible - error will be printed, deployment will be
aborted;
#. if ``MAJOR`` parts of these versions are different - they are incompatible
- error will be printed, deployment will be aborted;
#. otherwise they are compatible and deployment can be continued.
For ``dsl_version`` in ``fuel-ccp`` repository you should increment:
#. MAJOR version when you make incompatible changes in DSL;
#. MINOR version when you make backward-compatible changes in DSL;
#. PATCH version when you make fixes that do not change DSL, but affect
processing flow.
If you made a change in service definition that is not supposed to work with
the current ```dsl_version```, you should bump it to the minimal appropriate
number.
.. _link: https://github.com/openstack?q=fuel-ccp-

View File

@ -0,0 +1,84 @@
.. _config_global_config:
=============
Global config
=============
Optional config with global overrides for all services. Use it only if you need
to override some defaults.
File location
~~~~~~~~~~~~~
You can provide a path to file that you want to use as ``--config-file``
argument to ccp tool, otherwise it will be taken from the first existing
location out of the following ones:
* :file:`~/.ccp.yaml`
* :file:`~/.ccp/ccp.yaml`
* :file:`/etc/ccp.yaml`
* :file:`/etc/ccp/ccp.yaml`
Note that you can use only one config file, if you want to split your file into
several, you should use :ref:`includes <includes>`.
Format
~~~~~~
Every config file is a simple YAML file with any number of YAML documents
separated with ``---`` line::
config_a: 1
config_b:
config_c: 2
---
config_b:
config_d: 3
All documents are deeply merged into one (only dicts are deeply merged, not
lists or other structures). So above config will be equivalent to::
config_a: 1
config_b:
config_c: 2
config_d: 3
.. _includes:
Includes
~~~~~~~~
If you want to split your config over several files (e.g. keep sentitive config
arguments separately or have a general config file part for several
deployments) you can use includes. An include is a separate YAML document with
``!include`` tag and a list of files to be included in its place::
!include
- file_a
- file_b
If files are specified with relative paths, they are considered to be relative
to file with includes. Absolute paths are taken as is.
All documents from files in include list are substituted in place of an include
in order of appearance, so values from the latest file take precedence over
values in former ones.
Note that include is just another YAML document in config file, so you can
override values from include in following documents::
basic_value: 1
---
!include
- override_basic_value
---
override_value: from_include
Configuration file sections
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Here you can find description of configuration parameters in these sections:
.. toctree::
types

View File

@ -6,78 +6,99 @@ Configuration files
This section will describe configuration format used in CCP.
File location
=============
Understanding global and default configs
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can provide a path to file that you want to use as ``--config-file``
argument to ccp tool, otherwise it will be taken from the first existing
location out of the following ones:
There are three config locations, which the CCP CLI uses:
* :file:`~/.ccp.yaml`
* :file:`~/.ccp/ccp.yaml`
* :file:`/etc/ccp.yaml`
* :file:`/etc/ccp/ccp.yaml`
#. ``Global defaults`` - fuel_ccp/resources/defaults.yaml in ``fuel-ccp`` repo.
#. ``Component defaults`` - service/files/defaults.yaml in each component repo.
#. ``Global config`` - Optional. For more information read the
:doc:`global_config`.
Note that you can use only one config file, if you want to split your file into
several, you should use :ref:`includes <includes>`.
Before deployment, CCP will merge all these files into one dict, using the
order above, so "component defaults" will override "global defaults" and
"global config" will override everything.
Format
======
For example, one of common situations is to specify custom options for
networking. To achieve user may overwrite options defined in
``Global defaults`` and ``Component defaults`` by setting new values in
``Global config``.
Every config file is a simple YAML file with any number of YAML documents
separated with ``---`` line::
File ``fuel_ccp/resources/defaults.yaml`` has follow settings:
config_a: 1
config_b:
config_c: 2
---
config_b:
config_d: 3
::
All documents are deeply merged into one (only dicts are deeply merged, not
lists or other structures). So above config will be equivalent to::
configs:
private_interface: eth0
public_interface: eth1
...
config_a: 1
config_b:
config_c: 2
config_d: 3
And part of the ``fuel-ccp-neutron/service/files/defaults.yaml`` looks like:
.. _includes:
::
Includes
========
configs:
neutron:
...
bootstrap:
internal:
net_name: int-net
subnet_name: int-subnet
network: 10.0.1.0/24
gateway: 10.0.1.1
...
If you want to split your config over several files (e.g. keep sentitive config
arguments separately or have a general config file part for several
deployments) you can use includes. An include is a separate YAML document with
``!include`` tag and a list of files to be included in its place::
User may overwrite these sections by defining the following content in the
~/.ccp.yaml:
!include
- file_a
- file_b
::
If files are specified with relative paths, they are considered to be relative
to file with includes. Absolute paths are taken as is.
debug: true
configs:
private_interface: ens10
neutron:
bootstrap:
internal:
network: 22.0.1.0/24
gateway: 22.0.1.1
All documents from files in include list are substituted in place of an include
in order of appearance, so values from the latest file take precedence over
values in former ones.
To validate these changes user needs to execute command ``ccp config dump``.
It will return final config file with changes, which user did. So output should
contain the following changes:
Note that include is just another YAML document in config file, so you can
override values from include in following documents::
::
basic_value: 1
---
!include
- override_basic_value
---
override_value: from_include
debug: true
...
configs:
private_interface: ens10 <----- it was changed
public_interface: eth1 <----- it wasn't changed
neutron:
bootstrap:
internal:
net_name: int-net <--- it wasn't changed
subnet_name: int-subnet <--- it wasn't changed
network: 22.0.1.0/24 <----- it was changed
gateway: 22.0.1.1 <----- it was changed
Configuration file sections
===========================
Here you can find description of configuration parameters in these sections:
Global defaults
---------------
.. toctree::
This is project wide defaults, CCP keeps it inside fuel-ccp repository in
``fuel_ccp/resources/defaults.yaml`` file. This file defines global variables,
that is variables that are not specific to any component, like interface names.
repositories
Component defaults
------------------
Each component repository could contain a ``service/files/defaults.yaml`` file
with default config for this component only.
.. _section:
Global config
-------------
See description in :doc:`global_config`.

180
doc/source/config/types.rst Normal file
View File

@ -0,0 +1,180 @@
.. _config_types:
=======================
Configuration key types
=======================
List of keys
~~~~~~~~~~~~
Each config could contain several keys:
- ``configs``
- ``versions``
- ``sources``
- ``nodes``
- ``network_topology``
- ``repositories``
Each key has its own purpose and isolation, so you have to add your variable
to the right key to make it work.
configs key
-----------
Isolation:
- Used in service templates files (service/files/).
- Used in application definition file service/component_name.yaml.
Allowed content:
- Any types of variables allowed.
Example:
::
configs:
keystone_debug: false
So you could add "{{ keystone_debug }}" variable to you templates, which will
be rendered into "false" in this case.
versions key
------------
Isolation:
- Used in Dockerfile.j2 only.
Allowed content:
- Only versions of different software should be kept here.
For example:
::
versions:
influxdb_version: "0.13.0"
So you could add this to influxdb Dockerfile.j2:
::
curl https://dl.influxdata.com/influxdb/releases/influxdb_{{ influxdb_version }}_amd64.deb
sources key
-----------
Isolation:
- Used in Dockerfile.j2 only.
Allowed content:
- This key has a restricted format, examples below.
Remote git repository example:
::
sources:
openstack/keystone:
git_url: https://github.com/openstack/keystone.git
git_ref: master
Local git repository exaple:
::
sources:
openstack/keystone:
source_dir: /tmp/keystone
So you could add this to Dockerfile.j2:
::
{{ copy_sources("openstack/keystone", "/keystone") }}
CCP will use the chosen configuration, to copy git repository into Docker
container, so you could use it latter.
network_topology key
--------------------
Isolation:
- Used in service templates files (service/files/).
Allowed content:
- This key is auto-created by entrypoint script and populated with container
network topology, based on the following variables: ``private_interface`` and
``public_interface``.
You could use it to get the private and public eth IP address. For example:
::
bind = network_topology["private"]["address"]
listen = network_topology["public"]["address"]
nodes and roles key
-------------------
Isolation:
- Not used in any template file, only used by the CCP CLI to create a cluster
topology.
Allowed content:
- This key has a restricted format, example of this format can be found in
``fuel-ccp`` git repository in ``etc/topology-example.yaml`` file.
"CCP_*" env variables
---------------------
Isolation:
- Used in service templates files (service/files/).
Allowed content:
- This variables are created from the application definition ``env`` key.
Only env keys which start with "CCP\_" will be passed to config hash.
This is mainly used to pass some k8s related information to container, for
example, you could use it to pass k8s node hostname to container via this
variable:
Create env key:
::
env:
- name: CCP_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
Use this variable in some config:
::
{{ CCP_NODE_NAME }}
repositories key
----------------
:doc:`repositories`

View File

@ -230,3 +230,37 @@ file
+---------+------------------------------------------------+----------+--------+---------+
| user | | false | string | |
+---------+------------------------------------------------+----------+--------+---------+
DSL versioning
~~~~~~~~~~~~~~
Some changes in CCP framework are backward compatible and some of them are not.
To prevent situations when service definitions are being processed by
incompatible version of CCP framework, DSL versioning has been implemented.
DSL versioning is based on Semantic Versioning model. Version has a format
``MAJOR.MINOR.PATCH`` and is being defined in ``dsl_version`` field of
:file:`fuel_ccp/__init__.py` module. Each service definition contains
``dsl_version`` field with the version of DSL it was implemented/updated for.
During the validation phase of :command:`ccp deploy` those versions will be
compared according to the following rules:
#. if DSL version of ``fuel-ccp`` is less than service's DSL version -
they are incompatible - error will be printed, deployment will be
aborted;
#. if ``MAJOR`` parts of these versions are different - they are incompatible
- error will be printed, deployment will be aborted;
#. otherwise they are compatible and deployment can be continued.
For ``dsl_version`` in ``fuel-ccp`` repository you should increment:
#. MAJOR version when you make incompatible changes in DSL;
#. MINOR version when you make backward-compatible changes in DSL;
#. PATCH version when you make fixes that do not change DSL, but affect
processing flow.
If you made a change in service definition that is not supposed to work with
the current ```dsl_version```, you should bump it to the minimal appropriate
number.

View File

@ -15,6 +15,7 @@ User docs
quickstart
monitoring_and_logging
config/index
config/types
Advanced topics
---------------
@ -34,8 +35,8 @@ Developer docs
:maxdepth: 1
contributing
docker
app_def_guide
docker
dsl
Design docs