[doc] Add documentation on how to rotate passwords

Change-Id: I59675353a1c6afab3cf0ece10d9ae09da15f568c
Signed-off-by: Amy Marrich (spotz) <amy@demarco.com>

deploy-guide/source/configure.rst

@@ -364,8 +364,5 @@ values for the variables in each file that contains service credentials:
 
 To regenerate existing passwords, add the ``--regen`` flag.
 
-.. warning::
-
-   The playbooks do not currently manage changing passwords in an existing
-   environment. Changing passwords and rerunning the playbooks will fail
-   and might break your OpenStack environment.
+For information on how to rotate passwords, please refer to the `Password Rotation <https://docs.openstack.org/openstack-ansible/latest/user/security/password-rotation.html>`_
+documentation.

doc/source/user/security/index.rst

@@ -18,3 +18,4 @@ For understanding security design, please see
    security-txt
    hardening
    non-root
+   password-rotation

doc/source/user/security/password-rotation.rst

@@ -0,0 +1,106 @@
+Password Rotation
+=================
+
+.. warning::
+
+   The playbooks do not guarantee password rotation with zero downtime in an
+   existing environment. Downtime for the service is expected between
+   password is reset on backend and services are restarted to apply new value.
+
+All service passwords are defined and stored as Ansible variables in
+OpenStack-Ansible.
+This allows the operator to store passwords in an encrypted format using
+`Ansible Vault <https://docs.ansible.com/ansible/latest/vault_guide/index.html>`_
+or define them as a lookup to `SOPS <https://getsops.io/>`_ or `OpenBao <https://openbao.org/>`_
+
+Typical password change processes include the following steps:
+
+#. Define a new password in Ansible variables (or where defined lookup is pointing to).
+#. Change password on an infrastructure backend (i.e. MariaDB, RabbitMQ, etc.).
+#. Update service configuration file with the new password.
+#. Restart the service to apply new configuration.
+
+Due to the variety of methods which can be used for storing and defining
+Ansible variables, we will leave the process of changing password definitions out
+of scope of this article, and will focus solely on the process of applying
+new passwords to the environment.
+
+
+Applying new passwords to the service
+-------------------------------------
+
+A typical service has a set of passwords for authentication in "infra" backends,
+which include, but not limited to:
+
+* Keystone
+* MariaDB
+* RabbitMQ
+
+As a service downtime is expected after a password is changed (which generally happens
+at the very beginning of each role) until the service is restarted, it is important
+to ensure that playbook will execute as fast as possible. For that, we will use a set
+of specific tags and variables for each "backend" individually.
+
+For following examples, we will take Nova as a sample service, as Nova may struggle
+from playbook runtime the most due to amount of hosts which needs to be updated.
+
+As a common technique, we will disable ``serial`` execution, which is enabled by
+default, as the password for the backend will be reset during the runtime for
+the first host, so rest will not be able to communicate normally regardless of
+used ``serial``.
+
+Another common thing among all sections below is usage of ``post-install`` tag.
+It has been introduced in 2024.1 (Caracal) release and is applied only to
+``<service>_post_install.yml`` tasks, where templating of config files is
+happening.
+
+Changing Keystone password for the service
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In order to trigger a password update in Keystone for the service, we
+need to supply ``service_update_password`` variable.
+To execute the rotation of the Keystone password for a service like Nova,
+you will need to execute playbook like this:
+
+.. code-block:: shell-session
+
+    openstack-ansible openstack.osa.nova -e nova_conductor_serial=100% -e nova_compute_serial=100% \
+        -e service_update_password=true --tags common-service,post-install
+
+Changing MariaDB password for the service
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To execute the rotation of the MariaDB password for a service like Nova,
+you will need to execute playbook like this:
+
+.. code-block:: shell-session
+
+    openstack-ansible openstack.osa.nova -e nova_conductor_serial=100% -e nova_compute_serial=100% \
+        --tags common-db,post-install
+
+Changing RabbitMQ password for the service
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To execute the rotation of the MariaDB password for a service like Nova,
+you will need to execute playbook like this:
+
+.. code-block:: shell-session
+
+    openstack-ansible openstack.osa.nova -e nova_conductor_serial=100% -e nova_compute_serial=100% \
+        --tags common-mq,post-install
+
+Changing all passwords for the service at once
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+It is worth to mention, that operator can combine them together to perform
+rotation of passwords for all backends at the same time. While it will increase
+playbook runtime (and thus a downtime), it will still be more efficient when
+all passwords need to be changed anyway.
+
+To update all passwords mentioned in previous sections, we will simply
+combine all used tags and variables:
+
+.. code-block:: shell-session
+
+    openstack-ansible openstack.osa.nova -e nova_conductor_serial=100% -e nova_compute_serial=100% \
+        -e service_update_password=true --tags common-service,common-db,common-mq,post-install