OVN DBs clustering

We introduce support for running both the NB and SB OVN databases in
clustered mode. This OVN DBs clustered mode is based on OVNs own
clustering protocol and does not rely on pacemaker.

Clustering the two OVN databases increases reliability. The cluster
works in active-active mode and has the potential to be more
resilient and performant.

See
https://docs.openvswitch.org/en/latest/ref/ovsdb.7/#clustered-database-service-model
for more information.

For backport simplicity we also add I50cf3b7d79d8cd139ae514438e147df73901a366
("Fix typo in ovn-dbs-cluster northd kolla config file") which
is a cherry-pick + squash of commit 1115698c14,
so we avoid ovn_northd connecting only to the local db via unix socket.

Co-Authored-By: Michele Baldessari <michele@acksyn.org>

Related-Bug: #1931133
Depends-On: https://review.opendev.org/c/openstack/tripleo-ansible/+/804435
Depends-On: https://review.opendev.org/c/openstack/puppet-tripleo/+/795763

NB: Some changes for this ussuri (just like for victoria) backport are needed:
- Readd DefaultPasswords:
- go back to the old way of doing TLS (i.e. no system-roles.certificate)
- template version back to rocky
- Set tripleo_container_manage_systemd_order: false (I believe it was a
  mistake to start with and we removed it in master as well)

Change-Id: I59bfe69dbb5f3d525ac6f6d655577d24036328c0
(cherry picked from commit baf4a16149)
(cherry picked from commit 15433f131c)
(cherry picked from commit 40da6ed234)

deployment/ovn/ovn-dbs-cluster-ansible.yaml

@@ -0,0 +1,312 @@
+heat_template_version: rocky
+
+description: >
+  OpenStack containerized OVN DBs service in cluster mode
+
+parameters:
+  ContainerOvnNbDbImage:
+    description: image
+    type: string
+  ContainerOvnSbDbImage:
+    description: image
+    type: string
+  ContainerOvnNorthdImage:
+    description: image
+    type: string
+  EndpointMap:
+    default: {}
+    description: Mapping of service endpoint -> protocol. Typically set
+                 via parameter_defaults in the resource registry.
+    type: json
+  ServiceData:
+    default: {}
+    description: Dictionary packing service data
+    type: json
+  ServiceNetMap:
+    default: {}
+    description: Mapping of service_name -> network name. Typically set
+                 via parameter_defaults in the resource registry.  This
+                 mapping overrides those in ServiceNetMapDefaults.
+    type: json
+  DefaultPasswords:
+    default: {}
+    type: json
+  RoleName:
+    default: ''
+    description: Role name on which the service is applied
+    type: string
+  RoleParameters:
+    default: {}
+    description: Parameters specific to the role
+    type: json
+  OVNNorthboundServerPort:
+    description: Port of the OVN Northbound DB server
+    type: number
+    default: 6641
+  OVNSouthboundServerPort:
+    description: Port of the OVN Southbound DB server
+    type: number
+    default: 6642
+  OVNNorthboundClusterPort:
+    description: Cluster port of the OVN Northbound DB server
+    type: number
+    default: 6643
+  OVNSouthboundClusterPort:
+    description: Cluster port of the OVN Southbound DB server
+    type: number
+    default: 6644
+  EnableInternalTLS:
+    type: boolean
+    default: false
+  InternalTLSCAFile:
+    default: '/etc/ipa/ca.crt'
+    type: string
+    description: Specifies the default CA cert to use if TLS is used for
+                 services in the internal network.
+  CertificateKeySize:
+    type: string
+    default: '2048'
+    description: Specifies the private key size used when creating the
+                 certificate.
+  OvnDBSCertificateKeySize:
+    type: string
+    default: ''
+    description: Override the private key size used when creating the
+                 certificate for this service
+
+conditions:
+  key_size_override_unset: {equals: [{get_param: OvnDBSCertificateKeySize}, '']}
+  internal_tls_enabled: {equals: [{get_param: EnableInternalTLS}, true]}
+
+resources:
+
+  ContainersCommon:
+    type: ../containers-common.yaml
+
+outputs:
+  role_data:
+    description: Role data for the OVN multi-active cluster role.
+    value:
+      service_name: ovn_dbs
+      firewall_rules:
+        '121 OVN DB server and cluster ports':
+          proto: 'tcp'
+          dport:
+            - {get_param: OVNNorthboundServerPort}
+            - {get_param: OVNSouthboundServerPort}
+            - {get_param: OVNNorthboundClusterPort}
+            - {get_param: OVNSouthboundClusterPort}
+      config_settings:
+        map_merge:
+          - if:
+            - internal_tls_enabled
+            - generate_service_certificates: true
+              ovn_dbs_certificate_specs:
+                service_certificate: '/etc/pki/tls/certs/ovn_dbs.crt'
+                service_key: '/etc/pki/tls/private/ovn_dbs.key'
+                hostname:
+                  str_replace:
+                    template: "%{hiera('fqdn_NETWORK')}"
+                    params:
+                      NETWORK: {get_param: [ServiceNetMap, OvnDbsNetwork]}
+                principal:
+                  str_replace:
+                    template: "ovn_dbs/%{hiera('fqdn_NETWORK')}"
+                    params:
+                      NETWORK: {get_param: [ServiceNetMap, OvnDbsNetwork]}
+                key_size:
+                  if:
+                    - key_size_override_unset
+                    - {get_param: CertificateKeySize}
+                    - {get_param: OvnDBSCertificateKeySize}
+            - {}
+      kolla_config:
+        /var/lib/kolla/config_files/ovn_cluster_north_db_server.json:
+          command: bash -c $* -- eval source /etc/sysconfig/ovn_cluster; exec /usr/local/bin/start-nb-db-server ${OVN_NB_DB_OPTS}
+          config_files: &ovn_dbs_kolla_config_files
+            - source: "/var/lib/kolla/config_files/src/*"
+              dest: "/"
+              merge: true
+              preserve_properties: true
+            - source: "/var/lib/kolla/config_files/src-tls/*"
+              dest: "/"
+              merge: true
+              preserve_properties: true
+              optional: true
+          permissions: &ovn_dbs_kolla_permissions
+            - path: /var/log/openvswitch
+              owner: root:root
+              recurse: true
+            - path: /var/log/ovn
+              owner: root:root
+              recurse: true
+        /var/lib/kolla/config_files/ovn_cluster_south_db_server.json:
+          command: bash -c $* -- eval source /etc/sysconfig/ovn_cluster; exec /usr/local/bin/start-sb-db-server ${OVN_SB_DB_OPTS}
+          config_files: *ovn_dbs_kolla_config_files
+          permissions: *ovn_dbs_kolla_permissions
+        /var/lib/kolla/config_files/ovn_cluster_northd.json:
+          command: bash -c $* -- eval source /etc/sysconfig/ovn_cluster; exec /usr/bin/ovn-northd ${OVN_NORTHD_OPTS}
+          config_files: *ovn_dbs_kolla_config_files
+          permissions: *ovn_dbs_kolla_permissions
+      docker_config:
+        step_0:
+          ovn_cluster_north_db_server:
+            start_order: 0
+            image: {get_param: ContainerOvnNbDbImage}
+            net: host
+            privileged: false
+            restart: always
+            volumes:
+              list_concat:
+                - {get_attr: [ContainersCommon, volumes]}
+                -
+                  - /var/lib/kolla/config_files/ovn_cluster_north_db_server.json:/var/lib/kolla/config_files/config.json:ro
+                  - /lib/modules:/lib/modules:ro
+                  - /var/lib/openvswitch/ovn:/var/lib/openvswitch:shared,z
+                  - /var/lib/openvswitch/ovn:/run/openvswitch:shared,z
+                  - /var/log/containers/openvswitch:/var/log/openvswitch:z
+                  - /var/lib/openvswitch/ovn:/var/lib/ovn:shared,z
+                  - /var/lib/openvswitch/ovn:/etc/openvswitch:shared,z
+                  - /var/lib/openvswitch/ovn:/etc/ovn:shared,z
+                  - /var/lib/openvswitch/ovn:/run/ovn:shared,z
+                  - /var/log/containers/openvswitch:/var/log/ovn:z
+                  - /var/lib/config-data/ansible-generated/ovn:/var/lib/kolla/config_files/src:ro
+                - if:
+                  - {get_param: EnableInternalTLS}
+                  -
+                    - /etc/pki/tls/private/ovn_dbs.key:/var/lib/kolla/config_files/src-tls/etc/pki/tls/private/ovn_dbs.key:ro
+                    - /etc/pki/tls/certs/ovn_dbs.crt:/var/lib/kolla/config_files/src-tls/etc/pki/tls/certs/ovn_dbs.crt:ro
+                  - null
+            environment:
+              KOLLA_CONFIG_STRATEGY: COPY_ALWAYS
+          ovn_cluster_south_db_server:
+            start_order: 0
+            image: {get_param: ContainerOvnSbDbImage}
+            net: host
+            privileged: false
+            restart: always
+            volumes:
+              list_concat:
+                - {get_attr: [ContainersCommon, volumes]}
+                -
+                  - /var/lib/kolla/config_files/ovn_cluster_south_db_server.json:/var/lib/kolla/config_files/config.json:ro
+                  - /lib/modules:/lib/modules:ro
+                  - /var/lib/openvswitch/ovn:/var/lib/openvswitch:shared,z
+                  - /var/lib/openvswitch/ovn:/run/openvswitch:shared,z
+                  - /var/log/containers/openvswitch:/var/log/openvswitch:z
+                  - /var/lib/openvswitch/ovn:/var/lib/ovn:shared,z
+                  - /var/lib/openvswitch/ovn:/etc/openvswitch:shared,z
+                  - /var/lib/openvswitch/ovn:/etc/ovn:shared,z
+                  - /var/lib/openvswitch/ovn:/run/ovn:shared,z
+                  - /var/log/containers/openvswitch:/var/log/ovn:z
+                  - /var/lib/config-data/ansible-generated/ovn:/var/lib/kolla/config_files/src:ro
+                - if:
+                  - {get_param: EnableInternalTLS}
+                  -
+                    - /etc/pki/tls/private/ovn_dbs.key:/var/lib/kolla/config_files/src-tls/etc/pki/tls/private/ovn_dbs.key:ro
+                    - /etc/pki/tls/certs/ovn_dbs.crt:/var/lib/kolla/config_files/src-tls/etc/pki/tls/certs/ovn_dbs.crt:ro
+                  - null
+            environment:
+              KOLLA_CONFIG_STRATEGY: COPY_ALWAYS
+          ovn_cluster_northd:
+            start_order: 2
+            image: {get_param: ContainerOvnNorthdImage}
+            net: host
+            privileged: false
+            restart: always
+            healthcheck:
+              test: /openstack/healthcheck
+            volumes:
+              list_concat:
+                - {get_attr: [ContainersCommon, volumes]}
+                -
+                  - /var/lib/kolla/config_files/ovn_cluster_northd.json:/var/lib/kolla/config_files/config.json:ro
+                  - /lib/modules:/lib/modules:ro
+                  - /var/lib/openvswitch/ovn:/run/openvswitch:shared,z
+                  - /var/log/containers/openvswitch:/var/log/openvswitch:z
+                  - /var/lib/openvswitch/ovn:/run/ovn:shared,z
+                  - /var/log/containers/openvswitch:/var/log/ovn:z
+                  - /var/lib/config-data/ansible-generated/ovn:/var/lib/kolla/config_files/src:ro
+                - if:
+                  - {get_param: EnableInternalTLS}
+                  -
+                    - /etc/pki/tls/private/ovn_dbs.key:/var/lib/kolla/config_files/src-tls/etc/pki/tls/private/ovn_dbs.key:ro
+                    - /etc/pki/tls/certs/ovn_dbs.crt:/var/lib/kolla/config_files/src-tls/etc/pki/tls/certs/ovn_dbs.crt:ro
+                  - null
+            environment:
+              KOLLA_CONFIG_STRATEGY: COPY_ALWAYS
+      global_config_settings:
+        ovn_db_clustered: true
+      metadata_settings:
+        if:
+          - internal_tls_enabled
+          - - service: ovn_dbs
+              network: {get_param: [ServiceNetMap, OvnDbsNetwork]}
+              type: node
+          - null
+      host_prep_tasks:
+        - name: create persistent directories
+          file:
+            path: "{{ item.path }}"
+            state: directory
+            setype: "{{ item.setype }}"
+            mode: "{{ item.mode|default(omit) }}"
+          loop:
+            - { 'path': /var/log/containers/openvswitch, 'setype': container_file_t, 'mode': '0750' }
+            - { 'path': /var/lib/openvswitch/ovn, 'setype': container_file_t }
+      deploy_steps_tasks:
+        - name: Prepare OVN cluster
+          when: step|int == 1
+          block:
+            - name: set is_ovn_dbs_bootstrap_node fact
+              set_fact: is_ovn_dbs_bootstrap_node={{ovn_dbs_short_bootstrap_node_name|lower == ansible_facts['hostname']|lower}}
+            - name: Configure OVN DBs and northd
+              include_role:
+                name: tripleo_ovn_cluster
+              vars:
+                tripleo_ovn_cluster_dbs_protocol: "{{ enable_internal_tls | ternary('ssl', 'tcp', 'tcp') }}"
+                tripleo_ovn_cluster_network: {get_param: [ServiceNetMap, OvnDbsNetwork]}
+                tripleo_ovn_cluster_nb_db_port: {get_param: OVNNorthboundServerPort}
+                tripleo_ovn_cluster_sb_db_port: {get_param: OVNSouthboundServerPort}
+                tripleo_ovn_cluster_nb_local_port: {get_param: OVNNorthboundClusterPort}
+                tripleo_ovn_cluster_nb_remote_port: {get_param: OVNNorthboundClusterPort}
+                tripleo_ovn_cluster_sb_local_port: {get_param: OVNSouthboundClusterPort}
+                tripleo_ovn_cluster_sb_remote_port: {get_param: OVNSouthboundClusterPort}
+        - name: Start OVN DBs and northd containers (bootstrap node)
+          when:
+            - step|int == 3
+            - is_ovn_dbs_bootstrap_node | bool
+          block: &ovn_dbs_start_containers
+            - name: Start OVN container
+              include_role:
+                name: tripleo_container_manage
+              vars:
+                tripleo_container_manage_config: "/var/lib/tripleo-config/container-startup-config/step_0"
+                tripleo_container_manage_config_id: "{{ ovn_container }}"
+                tripleo_container_manage_config_patterns: "{{ ovn_container }}.json"
+                tripleo_container_manage_systemd_order: false
+              loop:
+                - ovn_cluster_north_db_server
+                - ovn_cluster_south_db_server
+                - ovn_cluster_northd
+              loop_control:
+                loop_var: ovn_container
+            - name: Set connection # FIXME workaround until RHBZ #1952038 is fixed
+              become: yes
+              shell: |
+                podman exec ovn_cluster_north_db_server bash -c "ovn-nbctl -p /etc/pki/tls/private/ovn_dbs.key -c /etc/pki/tls/certs/ovn_dbs.crt -C /etc/ipa/ca.crt set-connection pssl:{{ tripleo_ovn_cluster_nb_db_port }}"
+                podman exec ovn_cluster_south_db_server bash -c "ovn-sbctl -p /etc/pki/tls/private/ovn_dbs.key -c /etc/pki/tls/certs/ovn_dbs.crt -C /etc/ipa/ca.crt set-connection pssl:{{ tripleo_ovn_cluster_sb_db_port }}"
+              when:
+                - enable_internal_tls | bool
+                - is_ovn_dbs_bootstrap_node | bool
+              vars:
+                tripleo_ovn_cluster_network: {get_param: [ServiceNetMap, OvnDbsNetwork]}
+                tripleo_ovn_cluster_nb_db_port: {get_param: OVNNorthboundServerPort}
+                tripleo_ovn_cluster_sb_db_port: {get_param: OVNSouthboundServerPort}
+        - name: Start OVN DBs and northd containers (non-bootstrap nodes)
+          when:
+            - step|int == 4
+            - not is_ovn_dbs_bootstrap_node | bool
+          block: *ovn_dbs_start_containers
+      update_tasks: []
+      upgrade_tasks: []

releasenotes/notes/add-ovn-dbs-cluster-support-6193cba5be432865.yaml

@@ -0,0 +1,14 @@
+---
+features:
+  - |
+    Added OVN DBs clustering support. In this service model, a clustered
+    database runs across multiple hosts in multi-active mode.
+upgrade:
+  - |
+    Upgrades from OVN non-HA and OVN DBs pacemaker to OVN DBs clustered are
+    currently not supported.
+security:
+  - |
+    The OVN database servers in an OVN DBs clustering and TLS-everywhere
+    deployment will listen on all IP addresses (0.0.0.0). This is a caveat that
+    can only be addressed once RHBZ 1952038 is fixed.