openstack/tripleo-docs
Code Issues Proposed changes

Remove mistral related developer documentation

Browse Source 
Change-Id: I301489c666545418d1e23c58ba75bc30e3db21b9
This commit is contained in:
Rabi Mishra 2020-05-05 08:15:44 +05:30
parent f5a668bad5
commit b01aaf15a3
12 changed files with 0 additions and 470 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
doc/source/developer/index.rst
View File

@ -9,5 +9,4 @@ Documentation of developer-specific options in |project|.
tht_walkthrough/tht_walkthrough
release
tripleoclient_primer
mistral_workflows/00-index
../upgrade/developer/upgrades/upgrades

25
doc/source/developer/mistral_workflows/00-index.rst
View File

@ -1,25 +0,0 @@
Mistral introduction for creating workbooks, workflows and actions
==================================================================
.. include:: ../../links.rst
The goal of this documentation section is to give developers an overview about
how to integrate new Mistral workbooks, workflows and actions into the Python
TripleO client, this will be described using as a point of reference the
the Undercloud backups automation.
This tutorial will be divided into several sections:
.. toctree::
:maxdepth: 2
01-introduction
02-undercloud-backups
03-creating-new-cli-option
04-creating-mistral-workflows
05-tripleo-environment-variables
06-test-local-changes
07-give-elevated-privileges-to-Mistral-actions
08-debugging-actions
09-unit-tests
10-summary

27
doc/source/developer/mistral_workflows/01-introduction.rst
View File

@ -1,27 +0,0 @@
Introduction and prerequisites
------------------------------
.. include:: ../../links.rst
Let's assume you have a TripleO development environment healthy and working
properly. All the commands and customization we are going to run will run in
the Undercloud, as usual logged in as the stack user and having sourced the
stackrc file.
Then let's proceed by cloning the repositories we are going to work within a
temporary folder:
::
mkdir dev-docs
cd dev-docs
git clone https://github.com/openstack/python-tripleoclient
git clone https://github.com/openstack/tripleo-common
git clone https://github.com/openstack/instack-undercloud
- **python-tripleoclient:** Will define the OpenStack CLI commands.
- **tripleo-common:** Will have the Mistral logic.
- **instack-undercloud:** Allows to update and create mistral
environments to store configuration details
needed when executing Mistral workflows.

26
doc/source/developer/mistral_workflows/02-undercloud-backups.rst
View File

@ -1,26 +0,0 @@
Undercloud backups
------------------
Most of the Undercloud backup procedure is available in the `TripleO
official documentation site`_.
We will focus on the automation of backing up the resources required to
restore the Undercloud in case of a failed upgrade.
- All MariaDB databases on the undercloud node
- MariaDB configuration file on undercloud (so we can restore databases
accurately)
- All glance image data in /var/lib/glance/images
- All swift data in /srv/node
- All data in stack user home directory
For doing this we need to be able to:
- Connect to the database server as root.
- Dump all databases to file.
- Create a filesystem backup of several folders (and be able to access
folders with restricted access).
- Upload this backup to a swift container to be able to get it from the
TripleO web UI.
.. _TripleO official documentation site: https://docs.openstack.org/tripleo-docs/latest/install/post_deployment/backup_restore_undercloud.html

84
doc/source/developer/mistral_workflows/03-creating-new-cli-option.rst
View File

@ -1,84 +0,0 @@
Creating a new OpenStack CLI command in python-tripleoclient (openstack undercloud backup)
------------------------------------------------------------------------------------------
The first action needed is to be able to create a new CLI command for
the OpenStack client. In this case, we are going to implement the
**openstack undercloud backup** command.
::
cd dev-docs
cd python-tripleoclient
Let’s list the files inside this folder:
::
[stack@undercloud python-tripleoclient]$ ls
AUTHORS doc setup.py
babel.cfg LICENSE test-requirements.txt
bindep.txt zuul.d tools
build README.rst tox.ini
ChangeLog releasenotes tripleoclient
config-generator requirements.txt
CONTRIBUTING.rst setup.cfg
Once inside the **python-tripleoclient** folder we need to check the
**setup.cfg** file.
This file defines all the CLI commands for the Python
TripleO client.
Specifically, we will need at the end of this file our
new command definition:
::
undercloud_backup = tripleoclient.v1.undercloud_backup:BackupUndercloud
This means that we have a new command defined as **undercloud backup**
that will instantiate the **BackupUndercloud** class defined in the file
**tripleoclient/v1/undercloud_backup.py**
For further details related to this class definition please go to the
`gerrit review`_.
Now, having our class defined we can call other methods to invoke
Mistral in this way:
::
clients = self.app.client_manager
files_to_backup = ','.join(list(set(parsed_args.add_files_to_backup)))
workflow_input = {
"sources_path": files_to_backup
}
output = undercloud_backup.prepare(clients, workflow_input)
So forth, we will call the **undercloud_backup.prepare** method defined
in the file **tripleoclient/workflows/undercloud_backup.py**
which will call the **tripleo.undercloud_backup.v1.prepare_environment**
Mistral workflow we are about to create:
::
def prepare(clients, workflow_input):
workflow_client = clients.workflow_engine
tripleoclients = clients.tripleoclient
with tripleoclients.messaging_websocket() as ws:
execution = base.start_workflow(
workflow_client,
'tripleo.undercloud_backup.v1.prepare_environment',
workflow_input=workflow_input
)
for payload in base.wait_for_messages(workflow_client, ws, execution):
if 'message' in payload:
return payload['message']
In this case, we will create a loop within the tripleoclient and wait
until we receive a message from the Mistral workflow
**tripleo.undercloud_backup.v1.prepare_environment** that indicates if
the invoked workflow ended correctly.
.. _gerrit review: https://review.opendev.org/#/c/466213

80
doc/source/developer/mistral_workflows/04-creating-mistral-workflows.rst
View File

@ -1,80 +0,0 @@
Creating Mistral workflows for the new python-tripleoclient CLI command
-----------------------------------------------------------------------
The next step is to define the
**tripleo.undercloud_backup.v1.prepare_environment** Mistral workflow,
all the Mistral workbooks, workflows and actions will be defined in the
**tripleo-common** repository.
Let’s go inside **tripleo-common**
::
cd dev-docs
cd tripleo-common
And see it’s content:
::
[stack@undercloud tripleo-common]$ ls
AUTHORS doc README.rst test-requirements.txt
babel.cfg HACKING.rst releasenotes tools
build healthcheck requirements.txt tox.ini
ChangeLog heat_docker_agent scripts tripleo_common
container-images image-yaml setup.cfg undercloud_heat_plugins
contrib LICENSE setup.py workbooks
CONTRIBUTING.rst playbooks sudoers zuul.d
Again, we need to check the **setup.cfg** file. This file defines all the
Mistral actions we can call.
Specifically, we will need at the end of this file our new actions:
::
tripleo.undercloud.get_free_space = tripleo_common.actions.undercloud:GetFreeSpace
tripleo.undercloud.create_backup_dir = tripleo_common.actions.undercloud:CreateBackupDir
tripleo.undercloud.create_database_backup = tripleo_common.actions.undercloud:CreateDatabaseBackup
tripleo.undercloud.create_file_system_backup = tripleo_common.actions.undercloud:CreateFileSystemBackup
tripleo.undercloud.upload_backup_to_swift = tripleo_common.actions.undercloud:UploadUndercloudBackupToSwift
4.1. Action definition
~~~~~~~~~~~~~~~~~~~~~~
Let’s take the first action to describe it’s definition,
**tripleo.undercloud.get_free_space =
tripleo_common.actions.undercloud:GetFreeSpace**
We have defined the action named as
**tripleo.undercloud.get_free_space** which will instantiate the class
**GetFreeSpace** defined in the file
**tripleo_common/actions/undercloud.py** file.
If we open **tripleo_common/actions/undercloud.py** we can see the class
definition as:
::
class GetFreeSpace(base.Action):
""""
Get the Undercloud free space for the backup.
The default path to check will be /var/tmp and the
default minimum size will be 10240 MB (10GB).
""""
def __init__(self, min_space=10240):
self.min_space = min_space
def run(self, context):
temp_path = tempfile.gettempdir()
min_space = self.min_space
while not os.path.isdir(temp_path):
head, tail = os.path.split(temp_path)
temp_path = head
available_space = (
(os.statvfs(temp_path).f_frsize * os.statvfs(temp_path).f_bavail) /
(1024 * 1024))
if (available_space < min_space):
msg = "There is no enough space, avail. - %s MB" \
% str(available_space)
return actions.Result(error={'msg': msg})
else:
msg = "There is enough space

25
doc/source/developer/mistral_workflows/05-tripleo-environment-variables.rst
View File

@ -1,25 +0,0 @@
Give support for new Mistral environment variables when installing the undercloud.
----------------------------------------------------------------------------------
Sometimes developers will need to use additional values inside Mistral tasks. For
example, if we need to create a dump of a database we might need another one
than the Mistral user credentials for authentication purposes.
Initially when the Undercloud is installed it’s created a Mistral
environment called **tripleo.undercloud-config**. This environment
variable will have all required configuration details that we can get
from Mistral. This is defined in the **instack-undercloud** repository.
Let’s get into the repository and check the content of the file
`instack_undercloud/undercloud.py`_.
This file defines a set of methods to interact with the Undercloud,
specifically the method called **_create_mistral_config_environment**
allows to configure additional environment variables when installing the
Undercloud.
For additional testing, you can use the `Python snippet`_ to call
Mistral client from the Undercloud node available in gist.github.com.
.. _Python snippet: https://gist.github.com/ccamacho/354f798102710d165c1f6167eb533caf#file-mistral_client_snippet-py
.. _instack_undercloud/undercloud.py: https://opendev.org/openstack/instack-undercloud/src/branch/master/instack_undercloud/undercloud.py

56
doc/source/developer/mistral_workflows/06-test-local-changes.rst
View File

@ -1,56 +0,0 @@
Show how to test locally the changes in python-tripleoclient and tripleo-common
-------------------------------------------------------------------------------
The following procedures will allow you to test a
change in python-tripleoclient or tripleo-common locally.
For a change in **python-tripleoclient**, assuming you already have
downloaded the change you want to test, execute:
::
cd python-tripleoclient
sudo rm -Rf /usr/lib/python2.7/site-packages/tripleoclient*
sudo rm -Rf /usr/lib/python2.7/site-packages/python_tripleoclient*
sudo python setup.py clean --all install
For a change in **tripleo-common**, assuming you already have downloaded
the change you want to test, execute:
::
cd tripleo-common
sudo rm -Rf /usr/lib/python2.7/site-packages/tripleo_common*
sudo python setup.py clean --all install
sudo cp /usr/share/tripleo-common/sudoers /etc/sudoers.d/tripleo-common
# this loads the actions via entrypoints
sudo mistral-db-manage --config-file /etc/mistral/mistral.conf populate
# make sure the new actions got loaded
mistral action-list | grep tripleo
for workbook in workbooks/*.yaml; do
mistral workbook-create $workbook
done
for workbook in workbooks/*.yaml; do
mistral workbook-update $workbook
done
sudo systemctl restart openstack-mistral-executor
sudo systemctl restart openstack-mistral-engine
If we want to execute a Mistral action or a Mistral workflow you can
execute:
Examples about how to test Mistral actions independently:
::
mistral run-action tripleo.undercloud.get_free_space #Without parameters
mistral run-action tripleo.undercloud.get_free_space '{"path": "/etc/"}' # With parameters
mistral run-action tripleo.undercloud.create_file_system_backup '{"sources_path": "/tmp/asdf.txt,/tmp/asdf", "destination_path": "/tmp/"}'
Examples about how to test a Mistral workflow independently:
::
mistral execution-create tripleo.undercloud_backup.v1.prepare_environment # No parameters
mistral execution-create tripleo.undercloud_backup.v1.filesystem_backup '{"sources_path": "/tmp/asdf.txt,/tmp/asdf", "destination_path": "/tmp/"}' # With parameters

33
doc/source/developer/mistral_workflows/07-give-elevated-privileges-to-Mistral-actions.rst
View File

@ -1,33 +0,0 @@
Give elevated privileges to specific Mistral actions that need to run with elevated privileges.
-----------------------------------------------------------------------------------------------
Sometimes it is not possible to execute some restricted actions from
the mistral user, for example, when creating the Undercloud backup we
won’t be able to access the **/home/stack/** folder to create a tarball
of it. For this cases it’s possible to execute elevates actions from the
mistral user:
This is the content of the **sudoers** file in the root of the
**tripleo-common** `repository`_
at the time of the creation of this guide.
::
Defaults!/usr/bin/run-validation !requiretty
Defaults:validations !requiretty
Defaults:mistral !requiretty
mistral ALL = (validations) NOPASSWD:SETENV: /usr/bin/run-validation
mistral ALL = NOPASSWD: /usr/bin/chown -h validations\: /tmp/validations_identity_[A-Za-z0-9_][A-Za-z0-9_][A-Za-z0-9_][A-Za-z0-9_][A-Za-z0-9_][A-Za-z0-9_], \
/usr/bin/chown validations\: /tmp/validations_identity_[A-Za-z0-9_][A-Za-z0-9_][A-Za-z0-9_][A-Za-z0-9_][A-Za-z0-9_][A-Za-z0-9_], \
!/usr/bin/chown /tmp/validations_identity_* *, !/usr/bin/chown /tmp/validations_identity_*..*
mistral ALL = NOPASSWD: /usr/bin/rm -f /tmp/validations_identity_[A-Za-z0-9_][A-Za-z0-9_][A-Za-z0-9_][A-Za-z0-9_][A-Za-z0-9_][A-Za-z0-9_], \
!/usr/bin/rm /tmp/validations_identity_* *, !/usr/bin/rm /tmp/validations_identity_*..*
mistral ALL = NOPASSWD: /bin/nova-manage cell_v2 discover_hosts *
mistral ALL = NOPASSWD: /usr/bin/tar --ignore-failed-read -C / -cf /tmp/undercloud-backup-*.tar *
mistral ALL = NOPASSWD: /usr/bin/chown mistral. /tmp/undercloud-backup-*/filesystem-*.tar
validations ALL = NOPASSWD: ALL
Here you can grant permissions for specific tasks in when executing
Mistral workflows from **tripleo-common**
.. _repository: https://github.com/openstack/tripleo-common/blob/63ab54411e56ad0e70e5e145fcb0ce60a55eb3f8/sudoers

42
doc/source/developer/mistral_workflows/08-debugging-actions.rst
View File

@ -1,42 +0,0 @@
Debugging actions
-----------------
Let’s assume the action is written, added to setup.cfg but
is still not being invoked as expected in the CLI.
Firstly, check if the action was added by
``sudo mistral-db-manage populate``. Run
::
mistral action-list -f value -c Name | grep -e '^tripleo.undercloud'
If you don’t see your actions check output of
``sudo mistral-db-manage populate`` as
::
sudo mistral-db-manage populate 2>&1| grep ERROR | less
The following output may indicate issues in code. Simply fix code.
::
2018-01-01:00:59.730 7218 ERROR stevedore.extension [-] Could not load 'tripleo.undercloud.get_free_space': unexpected indent (undercloud.py, line 40): File "/usr/lib/python2.7/site-packages/tripleo_common/actions/undercloud.py", line 40
Execute a single action or execute a workflow from an existing
workbook to make sure it works as expected, for example:
Run a single Mistral action.
::
mistral run-action tripleo.undercloud.get_free_space '{"path": "/etc/"}'
mistral run-action tripleo.undercloud.create_file_system_backup '{"sources_path": "/tmp/asdf.txt,/tmp/asdf", "destination_path": "/tmp/"}'
Run a single Mistral workflow.
::
mistral execution-create tripleo.undercloud_backup.v1.filesystem_backup '{"sources_path": "/tmp/asdf.txt,/tmp/asdf", "destination_path": "/tmp/"}'

62
doc/source/developer/mistral_workflows/09-unit-tests.rst
View File

@ -1,62 +0,0 @@
Unit tests
----------
Now, let’s continue writing unit tests for the Mistral action we
wrote before. Let’s add the
**tripleo_common/tests/actions/test_undercloud.py** file with the
following content in the **tripleo-common** repository.
::
import mock
from tripleo_common.actions import undercloud
from tripleo_common.tests import base
class GetFreeSpaceTest(base.TestCase):
def setUp(self):
super(GetFreeSpaceTest, self).setUp()
self.temp_dir = "/tmp"
@mock.patch('tempfile.gettempdir')
@mock.patch("os.path.isdir")
@mock.patch("os.statvfs")
def test_run_false(self, mock_statvfs, mock_isdir, mock_gettempdir):
mock_gettempdir.return_value = self.temp_dir
mock_isdir.return_value = True
mock_statvfs.return_value = mock.MagicMock(
spec_set=['f_frsize', 'f_bavail'],
f_frsize=4096, f_bavail=1024)
action = undercloud.GetFreeSpace()
action_result = action.run(context={})
mock_gettempdir.assert_called()
mock_isdir.assert_called()
mock_statvfs.assert_called()
self.assertEqual("There is no enough space, avail. - 4 MB",
action_result.error['msg'])
@mock.patch('tempfile.gettempdir')
@mock.patch("os.path.isdir")
@mock.patch("os.statvfs")
def test_run_true(self, mock_statvfs, mock_isdir, mock_gettempdir):
mock_gettempdir.return_value = self.temp_dir
mock_isdir.return_value = True
mock_statvfs.return_value = mock.MagicMock(
spec_set=['f_frsize', 'f_bavail'],
f_frsize=4096, f_bavail=10240000)
action = undercloud.GetFreeSpace()
action_result = action.run(context={})
mock_gettempdir.assert_called()
mock_isdir.assert_called()
mock_statvfs.assert_called()
self.assertEqual("There is enough space, avail. - 40000 MB",
action_result.data['msg'])
Run
::
tox -epy27
to see any unit tests errors.

9
doc/source/developer/mistral_workflows/10-summary.rst
View File

@ -1,9 +0,0 @@
Summary
-------
- https://www.anstack.com/blog/2017/12/18/automating-the-undercloud-backup-and-mistral-workflows-intro.html
- http://www.dougalmatthews.com/2016/Sep/21/debugging-mistral-in-tripleo/
- http://blog.johnlikesopenstack.com/2017/06/accessing-mistral-environment-in-cli.html
- http://hardysteven.blogspot.com.es/2017/03/developing-mistral-workflows-for-tripleo.html