344 Commits

Author SHA1 Message Date
Lance Bragstad
c9635bf92e Implement a tool for converting policy.yaml files to heat templates
This commit adds a tool that parses a directory of service.yaml policy
files and then converts them to the appropriate THT structure, using the
necessary service variables and templating.

The enable-secure-rbac.yaml is simply the current defaults generated
from code. First, generate all the policy files for each OpenStack
service:

  $ oslopolicy-sample-generator --namespace $SERVICE --output-file $DEST/$SERVICE.yaml

Next, uncomment all the default policies as a starting point for making
policy changes:

  $ sed -i 's/^#"/"/g' $DEST/$SERVICE.yaml

Next you can make changes to the policy files to reflect the changes you
want in your deployment.

Finally, you can generate the necessary heat template:

  $ ./convert_policy_yaml_to_heat_template.py -d $DEST

The tool outputs to stdout. It's up to the user to redirect to a file if
they wish to save results.

The enable-secure-rbac.yaml environment will be updated in subsequent
patches to implement project personas.

Change-Id: I9957243d307758f56b84cde3a408006d8161fa41
2021-09-28 13:33:53 +00:00
David Vallee Delisle
dcb33e8c5a Adding json output to yaml-diff
It's easier to read than pformat.

Change-Id: I92c345130928028207738d788f5d703372b4c05a
2021-09-06 12:53:41 +00:00
Zuul
a332241b79 Merge "Use yaml.safe_load to load YAML files" 2021-07-28 15:48:14 +00:00
Takashi Kajinami
53040573ab Use yaml.safe_load to load YAML files
Since PyYAML 5.1, yaml.load without specifying the Loader option is
deprecated and shows the following warning.

YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated,
as the default Loader is unsafe.
Please read https://msg.pyyaml.org/load for full details.

This change replaces yaml.load by yaml.safe_load (which is effectively
same as adding Loader=yaml.SafeLoader) to get rid of that warning
message. Also, existing all usage of yaml.load with the Loader option
are also replaced so that we to make all implementation to load yaml
files consistent.

Change-Id: Id44fa2354429b944fbc0809f63db558bb7de23f7
2021-07-24 12:08:22 +09:00
Zuul
200f0522ed Merge "Add tool to convert net-data from v1 to v2" 2021-07-15 06:47:19 +00:00
Zuul
7def8038a3 Merge "yaml-diff.py: Making it a bit better" 2021-07-14 05:34:32 +00:00
Zuul
0a60aad3aa Merge "Introducing the ComputeVdpa role" 2021-07-13 19:06:30 +00:00
Kevin Carter
a191377b34
Add tool to convert net-data from v1 to v2
This new tool will convert the old v1 net-data format to v2. This tool
will create a backup of the older net-data file so that its possible to
recover data should anything go wrong in the conversion. Upon completion
the tool will create the v2 file using the existing file format name.

Change-Id: I4b4d79b0f68287921fedb3ad7b29b50df5ae3f02
Signed-off-by: Kevin Carter <kecarter@redhat.com>
2021-07-12 10:40:16 -05:00
David Vallee Delisle
7010a27415 Introducing the ComputeVdpa role
Now that nova supports libvirt's vDPA implementation [1][2], we need
deployment code to enable such a feature.

The ComputeVdpa role is basically a copy of ComputeSriov with the
addition of the ExtraKernelModules. Beside these modules, operators
should define to their ``NovaPCIPassthrough`` as they normally do
for standard SR-IOV as a role parameter.

When having the right modules loaded, at least in RHEL8.5, the kernel
will automatically create /dev/vhost-vdpa-X devices for each one of the
VFs.

[1] https://opendev.org/openstack/nova-specs/src/branch/master/specs/wallaby/implemented/libvirt-vdpa-support.rst
[2] https://bugzilla.redhat.com/1873409

Depends-On: https://review.opendev.org/c/openstack/os-net-config/+/798926
Related: https://bugzilla.redhat.com/1956297
Change-Id: Ifda3ac08c596f65bc75d869097a0f72e816ba783
2021-07-07 10:29:01 -04:00
David Vallee Delisle
ec1f6102e0 yaml-diff.py: Making it a bit better
Adding a --details switch to have a bit more information about the
differences between yaml files.

This is pretty much leveraging the difflib to get at most 2 levels of
differences in the yaml structure.

Change-Id: If300a5f5d351404dbcf73d02822cdccf8e9f68df
2021-07-05 18:11:28 -04:00
Francesco Pantano
dd5a5fc240
Remove from master the ceph-ansible references
ceph-ansible is not used anymore in master and both ceph
deployment and day2 ops are managed by the new cephadm tool.
This change aligns the tripleo-heat-templates tree to make sure
only cephadm is used to deploy Ceph (see [1]).

[1] https://blueprints.launchpad.net/tripleo/+spec/tripleo-ceph

Change-Id: Ib87615112264bd65e38ed7fb4440cca62f067de5
2021-06-29 15:55:24 +02:00
Zuul
2c4363561e Merge "Add network-v2 default files + vip data examples" 2021-06-18 22:33:24 +00:00
Zuul
e9d053b46d Merge "Use merge strategy for EndpointMap" 2021-06-16 01:41:17 +00:00
Zuul
9fbe318125 Merge "Replace unused var 'net' with _" 2021-06-16 00:11:31 +00:00
ramishra
a263da138f Use merge strategy for EndpointMap
Now someone can override individual endpoints like
below in an environment:

parameter_merge_strategies:
  EndpointMap: merge

parameter_defaults:
  EndpointMap:
    AodhPublic:
      protocol: http
      port: 9999
      host: 121.21.2.1

Closes-Bug: #1897592
Change-Id: Ibf2e0d183dd51421c4feb7467c3c01fb416d2965
2021-06-15 09:15:41 +05:30
Harald Jensås
be0b91ef56 Add network-v2 default files + vip data examples
Add a default file for network data and vip data to
use with network-data-v2.

The network data is an empty list, i.e no network
isolation is the default.

The default vip data file keeps one entry for the
ctlplane network, which is the only Virtual IP in
the no network isolation scenarios.

Related-Blueprint: blueprint network-data-v2-ports
Change-Id: Ia50435c0560ed76791d1859612f625a1b776a8b2
2021-06-07 13:22:40 +02:00
Alex Schultz
45fca9e029 Ignore puppet_config is an empty dict
Previously this was required which lead to folks defining invalid empty
data that actually gets ignored later in the process. Since you don't
have to have a puppet_config, let's say it needs to be defined but skip
validation if it is an empty.

Change-Id: I7310c340a1e2b48cdbc378ab4fe3944bd7e89112
2021-06-03 11:12:40 -06:00
Harald Jensås
78e870712a Replace unused var 'net' with _
Replace the unused variable 'net' with _.

Change-Id: I9e0ffac2d827feeb5e306d79e8ebdded1c350909
2021-05-20 23:49:15 +02:00
Brent Eagles
7de18e52cd Generate bind pool and bind configuration using deployment info
The bind pool information is now automatically generated and the
variables and sample config files are no longer needed. Matching bind9
and rndc key configuration is also generated.

Note: this patch also removes the use of puppet-dns which is problematic
when bind and the worker aren't on the same host and is awkward to use
with respect to rndc keys. It also modifies yaml-validate.py to correct
a rule changed with respect rndc_allowed_addresses.

Depends-On: Ib121888061b8bfcc4155528a8a209c7e274fafcb
Depends-On: I3383c19f80e70553ae71e644a01dda0f250d19da
Depends-On: I1b6674acbd6f999474cd66cb44357cf6b756a7d0

Change-Id: Ib89bcafe9f65431aee5756a32b2a82adc3d384dc
2021-04-09 15:04:22 -02:30
ramishra
c9991c2e31 Use 'wallaby' heat_template_version
With I57047682cfa82ba6ca4affff54fab5216e9ba51c Heat has added
a new template version for wallaby. This would allow us to use
2-argument variant of the ``if`` function that would allow for
 e.g. conditional definition of resource properties and help
cleanup templates. If only two arguments are passed to ``if``
function, the entire enclosing item is removed when the condition
is false.

Change-Id: I25f981b60c6a66b39919adc38c02a051b6c51269
2021-03-31 17:35:12 +05:30
Sorin Sbarnea
27788212cc Remove duplicate keys from yaml files
- removes duplicate keys from yaml files by assuming that the last
  one was the desired one (matches current loader behavior)
- prevent regressions by activating yaml lint rule that detects them
  (yaml skip was silencing all yaml checks, so the long list seen
  is in fact shorter than just 'yaml')
- includes sorting of some of the keys, was needed in order to spot
  the duplicates.

Change-Id: Idf5c0041a0c6d3ed7d5d49fb68be856719916663
2021-03-29 13:56:31 +00:00
Harald Jensås
fc50cfd2e4 Close if block in dual bonds
Fix the same issue found in:
https://review.opendev.org/c/openstack/tripleo-ansible/+/781102

Change-Id: Ic1a3bc39fe0124f88785834a8004f08413adb367
2021-03-17 23:50:36 +01:00
Zuul
d18c6abac7 Merge "Add deployed_vip_port templates" 2021-03-15 15:38:52 +00:00
Francesco Pantano
4b01172be2 Add cephadm deployment and environments branch
During the overcloud deployment phase, some operations should still be
performed on the provisioned Ceph cluster.
When Ceph is TripleO deployed, cephadm doesn't provide any firewall
configuration (–-skip-firewalld is provided), as well as any HA
configuration for the Ceph Dashboard.
This change introduces a new cephadm/ branch containing all the services
that should be configured in the TripleO context.

Co-Authored-By: Giulio Fidente <gfidente@redhat.com>
Co-Authored-By: John Fulton <fulton@redhat.com>

Depends-On: I35e57abddc64310a6422174fe191bd328588d7cd
Change-Id: Ie9537471924d1d60f7642699e58e734511b91a2b
2021-03-13 07:41:31 +00:00
Grzegorz Grasza
e329ca915e Generate certificates using ansible role
This is using linux-system-roles.certificate ansible role,
which replaces puppet-certmonger for submitting certificate
requests to certmonger. Each service is configured through
it's heat template.

Partial-Implements: blueprint ansible-certmonger
Depends-On: https://review.rdoproject.org/r/31713
Change-Id: Ib868465c20d97c62cbcb214bfc62d949bd6efc62
2021-03-10 16:28:22 +01:00
Harald Jensås
d498ebe872 Add deployed_vip_port templates
Add a new deployed_vip_port template which will replace
the current neutron port templates when vip ports are
managed outside of the heat stack.

Change-Id: Ia1f7e344ccebb2291f1574fdb69c05f00f4c3dc9
Partial-Implements: blueprint network-data-v2-ports
2021-03-10 01:30:25 +01:00
Zuul
a0ad81b0b6 Merge "Remove DefaultPasswords interface" 2021-02-16 08:00:59 +00:00
ramishra
7f195ff9a8 Remove DefaultPasswords interface
This was mainly there as an legacy interface which was
for internal use. Now that we pull the passwords from
the existing environment and don't use it, we can drop
this.

Reduces a number of heat resources.

Change-Id: If83d0f3d72a229d737a45b2fd37507dc11a04649
2021-02-12 11:38:44 +05:30
Zuul
c72909b35a Merge "tool: convert heat network-config to ansible j2" 2021-02-04 03:28:07 +00:00
Harald Jensås
2d0125ed40 Add an index tag on neutron network resources
When using the multiple nic-config templates in THT
and in tripleo-ansible the ordering of networks in
the network_data file matters.

Add a a tag 'tripleo_net_idx' to neutron resources
to keep track of the ordering.

Depends-On: https://review.opendev.org/772795
Partial-Implements: blueprint network-data-v2-ports
Change-Id: I0d39ea713ce1201b60f3def69f453792e402eb33
2021-01-28 02:04:57 +01:00
Francesco Pantano
79686663e8 Configure Ceph clients via tripleo-ceph-client (not ceph-ansible)
Default CephAnsibleSkipClient to True and CephConfigPath to
/var/lib/tripleo-config/ceph (instead of /etc/ceph) and set
these paramters explicitly in scenario00{1,4}. This will
result in all Ceph client configuration being done not by
ceph-ansible but by the new tripleo-ceph-client role from
tripleo-ansible.

Add the CephClient service to all Controller* roles which will
use Ceph. The service could have always been there as there are
Ceph clients on the these controllers, but it was not because
ceph-ansible configured clients as a side effect. With new
CephConfigPath default they no longer overlap so the service
is required.

Add support for CephExternalMultiConfig via tripleo-ceph-client
by looping on the contents of the CephExternalMultiConfig list
and passing each map as the dcn variable while including the
tripleo-ceph-client role each time.

Related-Bug: #1708302
Depends-On: I938ab604859fda88f3491399444841a3a373d162
Change-Id: I784e6a476752ed701192b3a0155c42edd4836d97
2021-01-04 15:16:11 +00:00
Harald Jensås
7de39925d0 tool: convert heat network-config to ansible j2
Add a script that does a best effort conversion of
a heat network-config tempalte to an ansible j2
network-config template.

The script uses a networks data file as input to
genereta a map of Heat Parameters to ansible vars.
For parameters not in the generated map the stack
environment is used, parameter values from the
stack environment is hard-coded in the j2 template.

A j2 comment is added whenever a value was hard coded,
in a header comment and also an inline comment if
possible.

NOTE: The j2 reference files in the unit tests was
      created by converting heat templates.

Change-Id: I8165a077b87307ca3c2ebee54703a939517dc9bf
2020-12-18 07:26:01 +01:00
James Slagle
5b60ef1ee8 Don't try and clean non-existing nic config dir
tools/process-templates.py -c was failing with a traceback:
 FileNotFoundError: [Errno 2] No such file or directory:
 './network/config'

That directory was removed in commit
3c246d15d87921ae423f3f012304368b127ecfbe, so we can remove trying to
clean it from process-templates.py

Closes-Bug: #1907268
Change-Id: I9e07d82240dee7d066634b1cade1390fe62e8341
Signed-off-by: James Slagle <jslagle@redhat.com>
2020-12-08 09:46:34 -05:00
Zuul
cfe1e59f78 Merge "Allow to configure vxlan network type for OVN" 2020-11-20 13:17:10 +00:00
Zuul
655f8fd73f Merge "Move cell_v2 discovery off compute hosts" 2020-11-19 20:55:11 +00:00
Ihar Hrachyshka
b1204893eb Allow to configure vxlan network type for OVN
Since 20.09, OVN supports VXLAN type for inter-chassis communication.

This patch also gets rid of no longer needed override for
NeutronNetworkType for OVN, moving the constraints into generic ML2
definition list. The constraints list is extended to include vxlan.

Depends-On: I81c016ba9c91282d1bebb40a282077e14ce4bd6b
Change-Id: I447458c344a8817f3cfacba06f3410d500ed1f59
2020-11-19 10:52:05 +01:00
Oliver Walsh
629485dde5 Move cell_v2 discovery off compute hosts
In I12a02f636f31985bc1b71bff5b744d346286a95f cell_v2 discovery was
originally moved from the nova-api container to the
nova-compute|nova-ironic containers in order to run cell
discovery during a scale up where the controllers are omitted
(e.g to exclude the controllers from a maintenance window).

This requires api database credentials on the compute node, which is
forbidden, so it must move back to a nova-api host as a pre-requisite
for removing these credentials in a follow-up patch.

Scale-up while omitting the controllers will no longer work out of the
box. Either a manual cell_v2 discovery can be run after scale up, or an
additional node can be deployed using the NovaManager tripleo role.

Related-bug: #1786961
Related-bug: #1871482
Change-Id: I47b95ad46e2d4e5b1f370a2f840826e87da2d703
2020-11-18 12:21:59 +00:00
Rabi Mishra
ddec6c8b65 Fix convert_nic_config.py script
Use 'value' property of the OS::Heat::Value resource instead.

Change-Id: I7acaa9e6b9d95b067b9fb1be4f5890aeee68742c
2020-11-18 10:45:30 +05:30
Zuul
0519978254 Merge "Remove Sahara support" 2020-11-06 03:12:06 +00:00
Takashi Kajinami
4a7d56947a Remove Sahara support
Sahara support was deprecated during previous Ussuri cycle[1], so we
can remove it completely now.

[1] f1d9b15c85fd1ed2250d40cea8184a18f458234f
Change-Id: Id047221cb912c09984cc3bf864196a26fd36736f
2020-10-19 09:39:36 +09:00
Harald Jensås
ff4d1fbb66 Use tags instead of role names
Role names can be customized, yet in THT jinja2 we
have several places where conditions are based on
the role name. By using tag's such as 'storage',
'ceph' and 'ovsdpdk' we the role names become truly
customizable.

The depends-on change in TripleO common will
dynamically add tag's to role's based on role.name
for backward compatibility during deprecation
period.

Depends-On: https://review.opendev.org/758124
Change-Id: I5ab4e4a220294245f95d328391bfffec87781a09
2020-10-15 12:57:40 +05:30
Rabi Mishra
eb8691c239 Add script to covert nic configs to use OS::Heat::Value
This adds a new script to convert jinja rendered nic config
templates to use OS::Heat::Value.

There is lots of duplication of code as tht is not installed
as a python package and can also be extended later to handle
multiple files if required.

Change-Id: I06ef54f70ffcd4fc1e501bf9c8395bbede1c6dda
2020-09-26 13:34:19 +05:30
Rabi Mishra
a4622f1382 Use OS::Heat::Value for NetworkConfig
This removes the run-os-net-config.sh script and uses
OS::Heat::Value for the NetworkConfig resources.

Depends-On: https://review.opendev.org/#/c/751713/
Change-Id: Ic3a0234d36525cdd6f415c77733d05a39bbeb3c2
2020-09-22 08:18:22 +05:30
Michele Baldessari
87b365afd3 Fix Flakes and lower-constraints errors
With the switch to Ubuntu Focal for tox jobs via https://review.opendev.org/#/c/738322/
our 1.1.0 version of hacking pulls in old modules that are not compatible
with python3.8:
https://github.com/openstack/hacking/blob/1.1.0/requirements.txt#L6

Let's upgrade hacking to >= 3.0.1 and < 3.1.0 so that it supports python3.8
correctly. The newer hacking also triggered new errors which are
fixed in this review as well:
./tools/render-ansible-tasks.py:113:25: F841 local variable 'e' is assigned to but never used
./tools/yaml-validate.py:541:19: F999 '...'.format(...) has unused arguments at position(s): 2
./tools/render-ansible-tasks.py:126:1: E305 expected 2 blank lines after class or function definition, found 1
./tools/yaml-validate.py:33:1: E305 expected 2 blank lines after class or function definition, found 1
./container_config_scripts/tests/test_nova_statedir_ownership.py:35:1: E305 expected 2 blank lines after class or function definition, found 0

Also make sure we exclude .tox and __pycache__ from flake8 as well

We also need to change the lower-constraint requirements to make them
py3.8 compatible. See https://bugs.launchpad.net/nova/+bug/1886298
cffi==1.14.0
greenlet==0.4.15
MarkupSafe==1.1.0
paramiko==2.7.1

Suggested-By: Yatin Karel <ykarel@redhat.com>

Change-Id: Ic280ce9a51f26d165d4e93ba0dc0c47cdf8d7961
Closes-Bug: #1895093
2020-09-10 11:10:54 +02:00
Rabi Mishra
dc171d2493 Remove all broken references to ./puppet/services
We've long moved these service definitions under ./deployment.

Change-Id: I46609facb862cb6451afd8920ae22ae054894178
2020-08-06 08:50:51 +05:30
Jose Luis Franco Arza
8783ec9c45 Remove ffwd-upgrade leftovers from THT.
Now that the FFU process relies on the upgrade_tasks and deployment
tasts there is no need to keep the old fast_forward_upgrade_tasks.

This patch removes all the fast_forward_upgrade_tasks section from
the services, as well as from the common structures.

Change-Id: I39b8a846145fdc2fb3d0f6853df541c773ee455e
2020-07-23 15:33:25 +00:00
Damien Ciabrini
b91a1a09cb Ensure redis_tls_proxy starts after all redis instances
When converting a HA control plane to TLS-e, 1) the bootstrap node
tells pacemaker to restart all redis instances to take into
account the new TLS-e config; 2) a new container redis_tls_proxy
is started on every controller to encapsulate redis traffic in TLS
tunnels. This happens during step 2.

Redis servers have to be restarted everywhere for redis_tls_proxy
to be able to start tunnels properly. Since we can't guarantee that
across several nodes during the same step, tweak the startup of
redis_tls_proxy instead; make sure to only create the tunnels once
the targeted host:port can be bound (i.e. redis was restarted).

Change-Id: I70560f80775dacddd82262e8079c13f86b0eb0e6
Closes-Bug: #1883096
2020-07-07 05:36:43 +00:00
Damien Ciabrini
832f71096d Fix regexes in yaml-validate.py with python 3.8
In Python 3.8, the re module no longer escapes various
characters (such as ; or /), as opposed to python 3.6.

Update various regexes in yaml-validate.py to match
patterns as expected for all python versions.

Change-Id: Ica7a01b10bc9483acdf7468bec2f8840ee3f7170
Closes-Bug: #1883722
2020-06-16 18:22:46 +02:00
Harald Jensås
a03f33a7d4 Deprecate service ipv6 params
Detect the IP version used instead of relying on the
user specifying the IP version for services.

The IP version is stored in a net_ip_version_map which
is passed to all services in ServiceData.

Deprecates the following parameters:
 CephIPv6, CorosyncIPv6, RabbitIPv6, MemcachedIPv6,
 MysqlIPv6, RedisIPv6 and NeutronOverlayIPVersion.

Change-Id: Iebfd8ef686381be2f45d0d4c45dfd6bf654d1ac6
2020-05-12 07:25:11 +00:00
Harald Jensås
a73c2281e6 Use python3 in tools scripts
The scripts in tools should be updated to use python3
by default. Support for python2 has been dropped.

Closes-Bug: #1875685
Change-Id: Ia13a8a43d767d93263d5c717f273184bf78d23c9
2020-04-29 09:23:36 +02:00