Add sphinx extension to document policies/profiles

This change adds a sphinx extention to build sections of the policy and
profile documentation from source code.

- Add new extension
- Modify contributor documentation for policies to show spec fields
  from source code
- Modify user documentation for policies and profiles to show properties
  from source code
- Modify user documentation for policies and profiles to include
  samples extracted from examples directory
- Update tox to run pep8 against sphinx extension

Change-Id: I32f68867d14c3288385dbdf3b4df14e8ca2fd01f
This commit is contained in:
Duc Truong 2018-10-17 21:26:55 +00:00
parent 042ce37bde
commit d8df1f8c51
25 changed files with 511 additions and 160 deletions

View File

@ -18,6 +18,13 @@ import sys
from senlin.version import version_info as senlin_version
sys.path.insert(0, os.path.abspath('../..'))
BASE_DIR = os.path.dirname(os.path.abspath(__file__))
ROOT = os.path.abspath(os.path.join(BASE_DIR, "..", ".."))
sys.path.insert(0, ROOT)
sys.path.insert(0, BASE_DIR)
# -- General configuration ----------------------------------------------------
# Add any Sphinx extension module names here, as strings. They can be
@ -31,6 +38,7 @@ extensions = [
'oslo_config.sphinxext',
'oslo_policy.sphinxext',
'oslo_policy.sphinxpolicygen',
'ext.resources'
]
# openstackdocstheme options

View File

@ -21,12 +21,8 @@ exposed by the Nova compute service. The basic policy has been extended to
work with vSphere hypervisor when VMware DRS feature is enabled. However, such
an extension is only applicable to *admin* owned server clusters.
Applicable Profiles
~~~~~~~~~~~~~~~~~~~
The policy is designed to handle only Nova server profile type, e.g.
``os.nova.server-1.0``.
.. schemaspec::
:package: senlin.policies.affinity_policy.AffinityPolicy
Actions Handled

View File

@ -19,11 +19,8 @@ Deletion Policy V1.1
The deletion policy is designed to be enforced when a cluster's size is to be
shrunk.
Applicable Profiles
~~~~~~~~~~~~~~~~~~~
The policy is designed to handle any (``ANY``) profile types.
.. schemaspec::
:package: senlin.policies.deletion_policy.DeletionPolicy
Actions Handled

View File

@ -19,12 +19,8 @@ Health Policy V1.0
The health policy is designed to automate the failure detection and recovery
process for a cluster.
Applicable Profile Types
~~~~~~~~~~~~~~~~~~~~~~~~
The policy is designed to handle both ``os.nova.server`` and ``os.heat.stack``
profile types.
.. schemaspec::
:package: senlin.policies.health_policy.HealthPolicy
Actions Handled

View File

@ -20,12 +20,8 @@ This policy is designed to enable senlin clusters to leverage the Neutron
LBaaS V2 features so that workloads can be distributed across nodes in a
reasonable manner.
Applicable Profiles
~~~~~~~~~~~~~~~~~~~
The policy is designed to handle only Nova server clusters, i.e. clusters
whose profile is a type of "``os.nova.server-1.0``".
.. schemaspec::
:package: senlin.policies.lb_policy.LoadBalancingPolicy
Actions Handled

View File

@ -19,11 +19,8 @@ Region Placement Policy V1.0
This policy is designed to make sure the nodes in a cluster are distributed
across multiple regions according to a specified scheme.
Applicable Profiles
~~~~~~~~~~~~~~~~~~~
The policy is designed to handle any profile types.
.. schemaspec::
:package: senlin.policies.region_placement.RegionPlacementPolicy
Actions Handled

View File

@ -28,11 +28,8 @@ parameters for such a request.
Note that when calculating the target capacity of the cluster, Senlin only
considers the **ACTIVE** nodes.
Applicable Profiles
~~~~~~~~~~~~~~~~~~~
The policy is designed to handle any (``ANY``) profile types.
.. schemaspec::
:package: senlin.policies.scaling_policy.ScalingPolicy
Actions Handled

View File

@ -19,12 +19,8 @@ Zone Placement Policy V1.0
This policy is designed to make sure the nodes in a cluster are distributed
across multiple availability zones according to a specified scheme.
Applicable Profiles
~~~~~~~~~~~~~~~~~~~
The policy is designed to handle Nova server clusters only, i.e. clusters with
a profile of type ``os.nova.server-1.0`` for example.
.. schemaspec::
:package: senlin.policies.zone_placement.ZonePlacementPolicy
Actions Handled

View File

291
doc/source/ext/resources.py Normal file
View File

@ -0,0 +1,291 @@
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
# -*- coding: utf-8 -*-
from docutils import nodes
from docutils.parsers import rst
from docutils.parsers.rst import directives
from functools import cmp_to_key
from oslo_utils import importutils
from sphinx.util import logging
from senlin.common import schema
LOG = logging.getLogger(__name__)
class SchemaDirective(rst.Directive):
required_arguments = 0
optional_arguments = 0
final_argument_whitespace = True
option_spec = {'package': directives.unchanged}
has_content = False
add_index = True
section_title = 'Spec'
properties_only = False
def run(self):
"""Build doctree nodes consisting for the specified schema class
:returns: doctree node list
"""
# gives you access to the options of the directive
options = self.options
content = []
# read in package class
obj = importutils.import_class(options['package'])
# skip other spec properties if properties_only is True
if not self.properties_only:
section = self._create_section(content, 'spec',
title=self.section_title)
# create version section
version_section = self._create_section(section, 'version',
title='Latest Version')
field = nodes.line('', obj.VERSION)
version_section.append(field)
# build versions table
version_tbody = self._build_table(
section, 'Available Versions',
['Version', 'Status', 'Supported Since'])
sorted_versions = sorted(obj.VERSIONS.items())
for version, support_status in sorted_versions:
for support in support_status:
cells = [version]
sorted_support = sorted(support.items(), reverse=True)
cells += [x[1] for x in sorted_support]
self._create_table_row(cells, version_tbody)
# create applicable profile types
profile_type_description = ('This policy is designed to handle '
'the following profile types:')
profile_type_section = self._create_section(
section, 'profile_types', title='Applicable Profile Types')
field = nodes.line('', profile_type_description)
profile_type_section.append(field)
for profile_type in obj.PROFILE_TYPE:
profile_type_section += self._create_list_item(profile_type)
# create actions handled
policy_trigger_description = ('This policy is triggered by the '
'following actions during the '
'respective phases:')
target_tbody = self._build_table(
section, 'Policy Triggers',
['Action', 'Phase'],
policy_trigger_description
)
sorted_targets = sorted(obj.TARGET, key=lambda tup: tup[1])
for phase, action in sorted_targets:
cells = [action, phase]
self._create_table_row(cells, target_tbody)
# build properties
properties_section = self._create_section(section, 'properties',
title='Properties')
else:
properties_section = content
sorted_schema = sorted(obj.properties_schema.items(),
key=cmp_to_key(self._sort_by_type))
for k, v in sorted_schema:
self._build_properties(k, v, properties_section)
# we return the result
return content
def _create_section(self, parent, sectionid, title=None, term=None):
"""Create a new section
:returns: If term is specified, returns a definition node contained
within the newly created section. Otherwise return the newly created
section node.
"""
idb = nodes.make_id(sectionid)
section = nodes.section(ids=[idb])
parent.append(section)
if term:
if term != '**':
section.append(nodes.term('', term))
definition = nodes.definition()
section.append(definition)
return definition
if title:
section.append(nodes.title('', title))
return section
def _create_list_item(self, str):
"""Creates a new list item
:returns: List item node
"""
para = nodes.paragraph()
para += nodes.strong('', str)
item = nodes.list_item()
item += para
return item
def _create_def_list(self, parent):
"""Creates a definition list
:returns: Definition list node
"""
definition_list = nodes.definition_list()
parent.append(definition_list)
return definition_list
def _sort_by_type(self, x, y):
"""Sort two keys so that map and list types are ordered last."""
x_key, x_value = x
y_key, y_value = y
# if both values are map or list, sort by their keys
if ((isinstance(x_value, schema.Map) or
isinstance(x_value, schema.List)) and
(isinstance(y_value, schema.Map) or
isinstance(y_value, schema.List))):
return (x_key > y_key) - (x_key < y_key)
# show simple types before maps or list
if (isinstance(x_value, schema.Map) or
isinstance(x_value, schema.List)):
return 1
if (isinstance(y_value, schema.Map) or
isinstance(y_value, schema.List)):
return -1
return (x_key > y_key) - (x_key < y_key)
def _create_table_row(self, cells, parent):
"""Creates a table row for cell in cells
:returns: Row node
"""
row = nodes.row()
parent.append(row)
for c in cells:
entry = nodes.entry()
row += entry
entry += nodes.literal(text=c)
return row
def _build_table(self, section, title, headers, description=None):
"""Creates a table with given title, headers and description
:returns: Table body node
"""
table_section = self._create_section(section, title, title=title)
if description:
field = nodes.line('', description)
table_section.append(field)
table = nodes.table()
tgroup = nodes.tgroup(len(headers))
table += tgroup
table_section.append(table)
for _ in headers:
tgroup.append(nodes.colspec(colwidth=1))
# create header
thead = nodes.thead()
tgroup += thead
self._create_table_row(headers, thead)
tbody = nodes.tbody()
tgroup += tbody
# create body consisting of targets
tbody = nodes.tbody()
tgroup += tbody
return tbody
def _build_properties(self, k, v, definition):
"""Build schema property documentation
:returns: None
"""
if isinstance(v, schema.Map):
newdef = self._create_section(definition, k, term=k)
if v.schema is None:
# if it's a map for arbritary values, only include description
field = nodes.line('', v.description)
newdef.append(field)
return
newdeflist = self._create_def_list(newdef)
sorted_schema = sorted(v.schema.items(),
key=cmp_to_key(self._sort_by_type))
for key, value in sorted_schema:
self._build_properties(key, value, newdeflist)
elif isinstance(v, schema.List):
newdef = self._create_section(definition, k, term=k)
# identify next section as list properties
field = nodes.line()
emph = nodes.emphasis('', 'List properties:')
field.append(emph)
newdef.append(field)
newdeflist = self._create_def_list(newdef)
self._build_properties('**', v.schema['*'], newdeflist)
else:
newdef = self._create_section(definition, k, term=k)
if 'description' in v:
field = nodes.line('', v['description'])
newdef.append(field)
else:
field = nodes.line('', '++')
newdef.append(field)
class SchemaProperties(SchemaDirective):
properties_only = True
class SchemaSpec(SchemaDirective):
section_title = 'Spec'
properties_only = False
def setup(app):
app.add_directive('schemaprops', SchemaProperties)
app.add_directive('schemaspec', SchemaSpec)

View File

@ -90,6 +90,22 @@ The following is a list of builtin policy types:
user/policy_types/region_placement
user/policy_types/zone_placement
3.3 Built-in Profile Types
--------------------------
The senlin service is released with some built-in profile types that target
some common use cases. You can develop and deploy your own profile types by
following the instructions in the :ref:`developer-guide` section.
The following is a list of builtin profile types:
.. toctree::
:maxdepth: 1
user/profile_types/nova
user/profile_types/stack
user/profile_types/docker
4 Usage Scenarios
~~~~~~~~~~~~~~~~~

View File

@ -234,5 +234,13 @@ The list below provides links to documents related to the creation and usage
of policy objects.
* :doc:`Working with Policy Types <policy_types>`
* :ref:`Affinity Policy <ref-affinity-policy>`
* :ref:`Batch Policy <ref-batch-policy>`
* :ref:`Deletion Policy <ref-deletion-policy>`
* :ref:`Health Policy <ref-health-policy>`
* :ref:`Load-Balancing Policy <ref-lb-policy>`
* :ref:`Region Placement Policy <ref-region-policy>`
* :ref:`Scaling Policy <ref-scaling-policy>`
* :ref:`Zone Placement Policy <ref-zone-policy>`
* :doc:`Managing the Bindings between Clusters and Policies <bindings>`
* :doc:`Browsing Events <events>`

View File

@ -11,6 +11,7 @@
License for the specific language governing permissions and limitations
under the License.
.. _ref-affinity-policy:
===============
Affinity Policy
@ -28,19 +29,16 @@ words, the type name of the cluster's profile has to be ``os.nova.server``.
Properties
~~~~~~~~~~
.. schemaprops::
:package: senlin.policies.affinity_policy.AffinityPolicy
Sample
~~~~~~
A typical spec for an affinity policy looks like the following example:
.. code-block:: yaml
type: senlin.policy.affinity
version: 1.0
properties:
servergroup:
name: my_server_group
policies: affinity
availability_zone: nova
enable_drs_extension: false
.. literalinclude :: /../../examples/policies/affinity_policy.yaml
:language: yaml
The affinity policy has the following properties:

View File

@ -30,17 +30,16 @@ for an elegant solution that can regulate the resource creation requests.
Properties
~~~~~~~~~~
.. schemaprops::
:package: senlin.policies.batch_policy.BatchPolicy
Sample
~~~~~~
Below is a typical spec for a batch policy:
.. code-block:: yaml
type: senlin.policy.batch
version: 1.0
properties:
min_in_service: 8
max_batch_size: 3
pause_time: 30
.. literalinclude :: /../../examples/policies/batch_policy.yaml
:language: yaml
The ``min_in_service`` property specifies the minimum number of nodes to be
kept in ACTIVE status. This is mainly for cluster update use cases. The

View File

@ -27,17 +27,16 @@ is enforced when the cluster's size is about to be reduced.
Properties
~~~~~~~~~~
.. schemaprops::
:package: senlin.policies.deletion_policy.DeletionPolicy
Sample
~~~~~~
Below is a typical spec for a deletion policy:
.. code-block:: yaml
type: senlin.policy.deletion
version: 1.1
properties:
criteria: OLDEST_FIRST
destroy_after_deletion: false
grace_period: 30
reduce_desired_capacity: true
.. literalinclude :: /../../examples/policies/deletion_policy.yaml
:language: yaml
The valid values for the "``criteria`` property include:

View File

@ -27,37 +27,19 @@ most deployment scenarios.
The policy type is currently applicable to clusters whose profile type is one
of ``os.nova.server`` or ``os.heat.stack``. This could be extended in future.
.. note::
The health policy is still under rapid development. More features are being
designed, implemented and verified. Its support status is still
``EXPERIMENTAL``, which means there could be changes at the discretion of
the development team before it is formally supported.
Properties
~~~~~~~~~~
.. schemaprops::
:package: senlin.policies.health_policy.HealthPolicy
Sample
~~~~~~
A typical spec for a health policy looks like the following example:
.. code-block:: yaml
type: senlin.policy.health
version: 1.0
properties:
detection:
type: NODE_STATUS_POLLING
options:
interval: 60
recovery:
actions:
- name: REBOOT
params:
type: soft
fencing:
- compute
.. literalinclude :: /../../examples/policies/health_policy_poll.yaml
:language: yaml
There are two groups of properties (``detection`` and ``recovery``), each of
which provides information related to the failure detection and the failure

View File

@ -35,40 +35,17 @@ installed and configured properly.
Properties
~~~~~~~~~~
.. schemaprops::
:package: senlin.policies.lb_policy.LoadBalancingPolicy
Sample
~~~~~~
The design of the load-balancing policy faithfully follows the interface and
properties exposed by the LBaaS v2 service. A sample spec is shown below:
.. code-block:: yaml
type: senlin.policy.loadbalance
version: 1.1
properties:
pool:
protocol: HTTP
protocol_port: 80
subnet: private_subnet
lb_method: ROUND_ROBIN
admin_state_up: true
session_persistence:
type: HTTP_COOKIE
cookie_name: my_cookie
vip:
subnet: public_subnet
address: 12.34.56.78
connection_limit: 5000
protocol: HTTP
protocol_port: 80
admin_state_up: true
health_monitor:
type: HTTP
delay: 20
timeout: 5
max_retries: 3
admin_state_up: true
http_method: GET
url_path: /health
expected_codes: 200
lb_status_timeout: 300
.. literalinclude :: /../../examples/policies/lb_policy.yaml
:language: yaml
As you can see, there are many properties related to the policy. The good news
is that for most of them, there are reasonable default values. All properties

View File

@ -28,20 +28,16 @@ The policy is designed to work with clusters of any profile types.
Properties
~~~~~~~~~~
.. schemaprops::
:package: senlin.policies.region_placement.RegionPlacementPolicy
Sample
~~~~~~
A typical spec for a region placement policy is shown in the following sample:
.. code-block:: yaml
type: senlin.policy.region_placement
version: 1.0
properties:
regions:
- name: region_1
weight: 100
cap: 50
- name: region_2
weight: 200
cap: 100
.. literalinclude :: /../../examples/policies/placement_region.yaml
:language: yaml
In this sample spec, two regions are provided, namely "``region_1``" and
"``region_2``". There are "weight" and "cap" attributes associated with them,

View File

@ -25,20 +25,16 @@ expected to be applicable on clusters of all profile types.
Properties
~~~~~~~~~~
.. schemaprops::
:package: senlin.policies.scaling_policy.ScalingPolicy
Sample
~~~~~~
A typical spec for a scaling policy is shown below:
.. code-block:: yaml
type: senlin.policy.scaling
version: 1.0
properties:
event: CLUSTER_SCALE_IN
adjustment:
type: CHANGE_IN_PERCENTAGE
number: 10
min_step: 1
best_effort: true
cooldown: 30
.. literalinclude :: /../../examples/policies/scaling_policy.yaml
:language: yaml
You should pay special attentions to the ``event`` property, whose valid
values include "``CLUSTER_SCALE_IN``" and "``CLUSTER_SCALE_OUT``". One

View File

@ -31,19 +31,17 @@ Nova virtual machines only.
Properties
~~~~~~~~~~
.. schemaprops::
:package: senlin.policies.zone_placement.ZonePlacementPolicy
Sample
~~~~~~
A typical spec for a zone placement policy is exemplified in the following
sample:
.. code-block:: yaml
type: senlin.policy.zone_placement
version: 1.0
properties:
regions:
- name: az_1
weight: 100
- name: az_2
weight: 200
.. literalinclude :: /../../examples/policies/placement_zone.yaml
:language: yaml
In this sample spec, two availability zones are provided, namely "``az_1``" and
"``az_2``". Each availability zone can have an optional "``weight``" attribute

View File

@ -0,0 +1,35 @@
..
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
.. _ref-docker-profile:
==============
Docker Profile
==============
The docker profile instantiates nodes that are associated with docker container
instances.
Properties
~~~~~~~~~~
.. schemaprops::
:package: senlin.profiles.container.docker.DockerProfile
Sample
~~~~~~
Below is a typical spec for a docker profile:
.. literalinclude :: /../../examples/profiles/docker_container/docker_basic.yaml
:language: yaml

View File

@ -0,0 +1,35 @@
..
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
.. _ref-nova-profile:
============
Nova Profile
============
The nova profile instantiates nodes that are associated with nova server
instances.
Properties
~~~~~~~~~~
.. schemaprops::
:package: senlin.profiles.os.nova.server.ServerProfile
Sample
~~~~~~
Below is a typical spec for a nova profile:
.. literalinclude :: /../../examples/profiles/nova_server/cirros_basic.yaml
:language: yaml

View File

@ -0,0 +1,35 @@
..
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
.. _ref-stack-profile:
=============
Stack Profile
=============
The stack profile instantiates nodes that are associated with heat stack
instances.
Properties
~~~~~~~~~~
.. schemaprops::
:package: senlin.profiles.os.heat.stack.StackProfile
Sample
~~~~~~
Below is a typical spec for a stack profile:
.. literalinclude :: /../../examples/profiles/heat_stack/nova_server/heat_stack_nova_server.yaml
:language: yaml

View File

@ -416,6 +416,9 @@ The following is a list of the links to documents related to profile's
creation and usage:
- :doc:`Working with Profile Types <profile_types>`
- :ref:`Nova Profile <ref-nova-profile>`
- :ref:`Stack Profile <ref-stack-profile>`
- :ref:`Docker Profile <ref-docker-profile>`
- :doc:`Creating and Managing Clusters <clusters>`
- :doc:`Creating and Managing Nodes <nodes>`
- :doc:`Managing Cluster Membership <membership>`

View File

@ -38,7 +38,7 @@ commands = oslo_debug_helper -t senlin/tests/unit {posargs}
[testenv:pep8]
basepython = python3
commands =
flake8 senlin
flake8 senlin doc/source/ext
[testenv:genconfig]
basepython = python3