This patch updates our Popen calls to enable universal newlines for
calls that we parse or consume the output for. Without
univeral_newlines=True, the output is treated as bytes under python3
which leads to issues later where we are using it as strings.
See https://docs.python.org/3/glossary.html#term-universal-newlines
Change-Id: Id0060a3abbcda8edb6124eb096cb824aaea48396
This is a simple patch to display properly the
warning text for operators when running the FFWD.
Also, rephrases the warning message as FFWD are
now fully tested.
Resolves: rhbz#1566294
Closes-Bug: 1785829
Change-Id: I0174a075e82d41ee9eacb4bdf0b4ba70cfae1007
Currently the container image registry references are passed with
--container-registry-file in prepare but as -e in converge for
the upgrade/update/ffwd-upgrade/ceph-upgrade clis.
There is no special handling of the file, and it is ultimately
included as any other environment file [1].
This removes the --container-registry-file parameter from
openstack overcloud [upgrade|update|ffwd-upgrade|ceph-upgrade]
prepare.
The related tripleo-common review is for removal of the parameter
from the mistral workflow and action
The related tripleo-docs patch is for updating the operator
instructions to include the container image registry references
using -e as any other environment file.
[1] 2c83dc964b/tripleo_common/actions/package_update.py (L60)
Related: https://review.openstack.org/570903 tripleo-docs
Partial-Bug: #1785825
Co-Authored-By: Jiri Stransky <jistr@jistr.com>
Depends-On: https://review.openstack.org/571186
Change-Id: Id2811dbef59d1be2a35cea062eb7116648f52145
This new function allows to run ansible playbooks
in a convenient way, taking advantage of already existing
run_command_and_log - that latter method has been modified so
that we can get the process full state instead of just the exit
code.
It implements some checks in order to ensure the playbook exists,
and allows to set a temporary ansible configuration file in order
to avoid unwanted behaviors (like retry file creation, callback for
output, and so on).
Change-Id: I5de4d68cc2669d37e7a667209423b9ac842136ff
New `openstack overcloud external-update run` and `openstack overcloud
external-upgrade run` commands are defined. These are meant to perform
updates and upgrades for services deployed via
external_deploy_tasks. A separate command is used because external
installers don't fit well within the --nodes and --roles selection
pattern we've established for the normal `update run` and `upgrade
run` commands.
Partial-Bug: #1783949
Depends-On: Ib2474e8f69711cd6610a78884d5032ffd19ad249
Depends-On: I982032a0eadfbfb7f1eadee9cae26c8cd5fcdbba
Change-Id: Ib2f32ae8ac234b0c25b0e1ff1f8f8d8e041185e0
Extend the tripleo client Command class to fetch heat roles data
from a role file. That new class method is shared by many derived
classes afterwards and used for containers images preparations,
containerized overcloud, undercloud and standalone deployments.
Apply normalization to the roles_file in plan management and
preflight checks as well.
Change-Id: I7b35e117b9d12f1e5a51e2ee0465244692d33e33
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
The prior changes in this series enabled uploading of images tagged in
such a way that they could be used with multiple architectures, this
change uses the data from instckenv.json to select appropriate deploy
images for each node as it's registered to ironic.
Blueprint: multiarch-support
Depends-On: Id82de41e7a49c2d8124fc74125ed51031579aa80
Depends-On: Idaf05b8efce28cd0cbf339cf693db4f55a693d9b
Depends-On: I41dce6e25766562db4366021309b8c2b74a8ab80
Change-Id: I7c84f3035853d8ee7b8d45895e7acce8e9dd3d13
Adding --platform when uploading images will set a tripleo_platform
image meta data property and prefix all images with the platform (and
arch) to make them stand out in an openstack image list.
Support is added in such a way that that image names are not altered if
no --platform is specified. As platform really only makes sense
*within* and architecture we ensure that both options are supplied.
Blueprint: multiarch-support
Change-Id: Ia56d22b244cfa5dddb5d110b54a525dfd80a7aaf
Adding --architecture when uploading images will set a hw_architecture image
meta data property and prefix all images with the arch to make them
stand out in an openstack image list
While tripleo doesn't yet take advantage of using the image meta data
it's still helpful for manual testing to ensure that image selection
implies appropriate hardware.
Support is added in such a way that that image names are not altered if
no --architecture is specified.
Blueprint: multiarch-support
Change-Id: I352759014ce4c249d8391b03317012000221f96f
This is a trivial re-factor but subsequent patches will increase
complexity of these helpers without clogging up callers with this logic.
Blueprint: multiarch-support
Change-Id: If9b115880e9755f2753a2e53a5869e12a17722a3
Undercloud deployment starts logging into its log file
too late, omitting log records from undercloud_config,
preflight checks. This is a problem as we want to log
at least the commands used for undercloud install/upgrade.
Ideally, messages in stdout/stderr should not miss the
log file.
Fix this via the added custom undercloud_log_file config
(use --log-file for standalone CLI). Use those to
configure logging to a file.
Fix missing formatting, like timestamps and source, for
undercloud config and preflight checks.
Write preflight/install/upgrade logs into the same
logfile.
Move the load oslo config method into shared utilities.
Add the configure logging shared utility for the classless
modules (w/o oslo_log supported). Such modules can
mimic oslo_log behavior defined for the main deployment
modules derived from openstack client classes, which
support logging to files natively and are not affected
by the subject issue. With an exception made for those
classless modules allowing them to log INFO+ be default.
So operators will have the pre-flight check
notes logged, for example.
Change-Id: I340be11bc9471df22f038629679634c3542d34d6
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
This enables passing a different plan environment file, and also
avoids hard-coded references to some of the environment files
such as the overcloud-resource-registry-puppet.yaml which we may
want to rename to enable plans per application e.g openstack,
openshift, ceph standalone etc in future.
This also aligns better with the overcloud deploy command.
Change-Id: I61763f62c5d44f098ad60f9426871caef16cd6de
In some cases, we may not want to always assume the ctlplane as the
network to use for both ssh access to the overcloud nodes, and for
generating the ansible inventory. This patches adds a new cli argument,
--overcloud-ssh-network that can be used to set the network name to be
used. Defaults to ctlplane.
Depends-On: I1e3f9a5142bff5e4f77e5e40df164089d9144652
Change-Id: I767984623c7d2402b10bae3d30b695783166f6bf
Running:
$ openstack tripleo container image prepare default \
--output-env-file prepare-default.yaml \
--local-push-destination
$ openstack tripleo container image prepare -e prepare-default.yaml
Will read a Heat template with ContainerImagePrepare, prepare containers
and upload to a registry if needed.
The idea is to replace the other commands used by the overcloud for any
use case: undercloud, overcloud or any cloud.
One of the goals here is to execute this process before starting the
containers while deploying OpenStack on any cloud.
Change-Id: Ie4b7951147f5a1aec654982e21296a749fdd865c
Blueprint: container-prepare-workflow
We need to be able to define the stack name if it
is different than 'overcloud', for the ansible inventory
used with the upgrade playbooks.
Change-Id: I0499eabc5c6f15cb32b2471c86245578c71a2a60
Closes-Bug: 1767379
Adds for all invocations of the ffwd-upgrade prepare, run and
converge a warning to the operator that ffwd-upgrade is still
under development and to be used with caution. The operator
must type 'yes' in order to proceed, or set the new --yes
parameter added here.
We can revisit this warning as development
and testing further mature the ffwd upgrade workflow.
Change-Id: Ie2279cba2f710747e93a9784d474f81ae199636e
The run command in the deploy can be reused. It would be better in utils
rather than buried in the tripleo deploy command.
Change-Id: Ia40660c4d2cfbc8a5c7ff3a2a21e82abd8e3ecd3
After refactoring the deployment command, the tests should be migrated
to the tripleo deploy command tests. Additionally some of the utility
functions have been moved to tripleoclient.utils.
Change-Id: I48b77a4c91412a0fba9a78bc5fe5f81d8962ac18
Related-Blueprint: all-in-one
Help undercloud_parameters.yaml to find way out
of /tmp box.
Move write_env_file into tripleoclient utils for
better unit testing.
Change-Id: Ibe4e1d8f683a02d55fee86bbf0f057f41f93b72e
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
Working with the sequence described in [0] with this patch and
the tripleo-common review in dependency below the operator can do:
1 openstack overcloud ffwd-upgrade prepare ... $params
# ^^^ added here, stack update for playbook generation.
2 openstack overcloud ffwd-upgrade run ... $params
# ^^^ added here. run fast_forward_upgrade_playbook.yaml
# against the overcloud (don't expost limit_hosts)
3 openstack overcloud upgrade run --roles Controller
# ^^^ exists... runs the upgrade_steps_playbook.yaml and
# deploy_steps_playbook.yaml and
# post_upgrade_steps_playbook.yaml. Repeat 3. with all --roles
4 openstack overcloud ffwd-upgrade converge
# ^^^ added here. no stack update, just unset noop from plan
Co-Authored-By: Lukas Bezdicka <social@v3.sk>
Co-Authored-By: Jiri Stransky <jistr@redhat.com>
[0] https://bugzilla.redhat.com/show_bug.cgi?id=1561178#c0
Depends-On: Ib9647b899fb9d49203c4d35c1abfe9c36e0babbe
Change-Id: Ie7823e121a4fa57feb1c56bd4d58f3183e41b8d7
This explicitly adds the upgrade-prepare.yaml environment file
or the upgrade-converge.yaml environment file, into the stored
swift plan, before continuing with the previous workflow (here
heat stack update in both cases). See dependency below in tht
and this tie-in is the biggest downside I can see.
The idea is to stop doing it in the tripleo-common like [1][2]
and move all the upgrade|update|ffwd-upgrade cli to use the
same and include the expected environment file.
Co-Authored-By: Lukas Bezdicka <social@v3.sk>
Co-Authored-By: Jiri Stransky <jistr@redhat.com>
[1] 6090d32b51/tripleo_common/actions/package_update.py (L62-L76)
[2] 6090d32b51/tripleo_common/actions/plan.py (L492-L502)
Depends-On: Icfe494e3219d6d6cd3251f75bb4329fc4d793c3c
Change-Id: I1288fe68ae8af02a5d77390d237ec467d88e43d2
When config-download becomes our only way of deploying, we'll be able
to assume that tripleo-admin user always exists. Until then it's not a
given, so we need to allow using custom user names for running updates
and upgrades playbooks. We default to heat-admin for now, as that's
expected to work on majority of current production environments.
Change-Id: I0df57002b2305c1e2504c9f7a7d0c326d8ffcaf7
Closes-Bug: #1759845
This is already the default when we generate the inventory via Mistral
workflow [1]. By using heat-admin we cannot connect to Deployed Server
environments, which e.g. affects CI, so we should use the same default
that the Mistral workflow uses nowadays.
[1] https://git.openstack.org/cgit/openstack/tripleo-common/tree/tripleo_common/actions/ansible.py#n526
Partial-Bug: #1757090
Change-Id: Idc0dba7a119efedae7f762d3a351419c22584b61
This exposes the --skip-tags which is useful for skipping the
service validations (--skip-tags validation). The expected
format is a comma-separated string for multiple values,
though 'validation' and 'pre-upgrade' are the only currently
supported tags for --skip-tags in the upgrade playbooks. The
full list of tags currently used in tripleo-heat-templates
upgrade_tasks is at [0]
openstack overcloud upgrade run --nodes foo
--skip-tags validation
[0]: 3eb0c62e47/tools/yaml-validate.py (L167)
Change-Id: Ie7fb8d9a388c6d53a31800406b03ddb8ed426401
Depends-On: I8544de64d3307e3dc925c1cecf2d9156e31d25a8
The issue here is that the undercloud heat installer
default uses template in /usr/share and renders the
result of the process-templates.py in place.
Process templates in a temporary work dir created
from the source --templates dir under the --output-dir.
Delete generated templates and temp files after the
undercloud deployment done or failed, when --cleanup
is specified.
Ensure the --output-dir exists for all of the cases.
The --output-dir and --cleanup may be passed
either as the 'undercloud deploy' CLI args, or via
the undercloud.conf file, as 'output_dir' and 'cleanup'.
The latter way only works for 'undercloud install --use-heat'.
Also, share process_multiple_environments in utils lib.
Reuse the shared process_multiple_environments for
undercloud and overcloud heat installers and add unit tests.
Additionally, fix the shared process_multiple_environments
redirect/rewrite: only redirect and rewrite paths, if fully
matched.
Closes-Bug: #1752272
Closes-Bug: #1749683
Change-Id: Ib0c2e3ffd81742441400d27857afae457d71a424
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
Adds a new run_update_ansible_action function in utils to be used by
the upgrade and update to avoid repetition. The package_update action is
passed into the utils function to avoid circular dependency.
Change-Id: Idbe0a98fa8f2e8dfe220796c02593a805942b0da
The undercloud deploy command reimplemented poorly the poll_for_events
function, calling get_stack uselessly and resolving outputs. This fixes
it.
Change-Id: I9a1c64d5ee7336d9943aeb100f7dc4a6e4cc3402
This further refactors the update/upgrades cli and separates
the update and upgrade code. This adds the overcloud_upgrade.py
which defines the UpgradePrepare and UpgradeRun. The entry
points are now:
openstack overcloud upgrade prepare --container-registry-file ...
For the no-op heat stack update to refresh stack outputs and
openstack overcloud upgrade run --nodes foo --playbooks all
For running all the upgrade ansible playbooks. A corresponding
converge sub-command will be introduced in a subsequent patch.
Change-Id: I1880e8f546df8d509871ba3b4f02877e95c611c8
deployment_user is a new parameter, default to getpass.getuser(),
will feed DeploymentUser parameter in THT, primarly used to add the
user to the 'docker' group, so our operators can run the `overcloud
container` commands when the undercloud is containerized.
Right now, the docker group was modified in puppet with
instack-undercloud but with the heat-installer we want to do it in
tripleoclient. DeploymentUser was chosen to be "standard" and can be
used for both undercloud & overclouds notions.
Change-Id: Ia5cc7b34ebee8cf2f49300ce23050370d5f1038a
This reverts commit 977bc6f1b8.
The original change is backwards incompatible since it removes the
overcloudrc v3 file without any deprecation.
Related-Bug: #1733640
Change-Id: I6b62e014312f7502a693f2990556d58696b1df11
The default file is now v3, so these will always be the same.
Related-Bug: #1733640
Depends-On: I5e779f48a02c5389cebf84e9d3899250d63cace8
Change-Id: I3612c58c356f8955bd44655cf5aacc20be532cbc
As the rc file is based on the stack name, it isn't always obvious where
it will be. This change adds to the output at the end of a deploy to let
the user know where it is located.
Change-Id: Ib6723571aa38aeec053bb2e3e7a11b37f725fa80
Closes-Bug: #1733593
The second part of the message with the error details would never get
displayed.
Change-Id: Id1725d6d0028aab907263c1d867d64bb15df77f1
Related-Bug: #1704380
Related-Bug: #1732751
Add a new cli option to tripleoclient called --config-download that will
trigger using the config download mechanism to apply the overcloud
configuration via ansible.
When using --config-download, an additional workflow called
tripleo.deployment.v1.config_download_deploy is run after the
deploy_play workflow. This new workflow does the necessary setup and
then executes the software configuration steps via ansible.
Change-Id: If3114a6fda75ee502d80e4ba27b5f2e81ba44085
implements: blueprint ansible-config-download
Depends-On: I738c78b32590c58ae8b721289ad8103ee09e9956
Implement the '--environment-directory' option to the prepare command
to match what the deploy command does.
Change-Id: I9192492580ec95c7d5c9c2ab7b06b79a73025859
Closes-Bug: #1723057
Probably the most common format for documenting arguments is reST field
lists [1]. This change updates some docstrings to comply with the field
lists syntax.
[1] http://sphinx-doc.org/domains.html#info-field-lists
Change-Id: Idb8322217bbad068f724239eb4580765924d24e0
Alex Schultz