Extend scale back hacluster operation - retry
Extend the scale back hacluster cloud operation from using Vault as an example to replacing a Vault cluster node. This consists essentially of adding a section on scaling out the application. Improve the Unseal Vault operation by linking to the Vault TLS cloud operation. Change-Id: Id2e146e7e5dbad8f1df4acb85f60728257ba526d
This commit is contained in:
parent
1fa01acd73
commit
ab54e97433
|
@ -15,7 +15,7 @@ General cloud operations:
|
||||||
ops-unseal-vault
|
ops-unseal-vault
|
||||||
ops-config-tls-vault-api
|
ops-config-tls-vault-api
|
||||||
ops-live-migrate-vms
|
ops-live-migrate-vms
|
||||||
ops-scale-back-with-hacluster
|
ops-replace-vault-node
|
||||||
ops-scale-out-nova-compute
|
ops-scale-out-nova-compute
|
||||||
ops-start-innodb-from-outage
|
ops-start-innodb-from-outage
|
||||||
ops-auto-glance-image-updates
|
ops-auto-glance-image-updates
|
||||||
|
|
|
@ -0,0 +1,175 @@
|
||||||
|
:orphan:
|
||||||
|
|
||||||
|
==================================================
|
||||||
|
Scale back an application with the hacluster charm
|
||||||
|
==================================================
|
||||||
|
|
||||||
|
Introduction
|
||||||
|
------------
|
||||||
|
|
||||||
|
This article shows how to replace a Vault node in a cluster made highly
|
||||||
|
available by means of the subordinate hacluster charm. It implies the removal
|
||||||
|
and then the addition of a vault unit. This is done with generic Juju commands
|
||||||
|
and actions available to the hacluster charm.
|
||||||
|
|
||||||
|
.. important::
|
||||||
|
|
||||||
|
This procedure will not result in cloud downtime providing that there is at
|
||||||
|
least one functional Vault node present at all times.
|
||||||
|
|
||||||
|
.. warning::
|
||||||
|
|
||||||
|
This procedure will involve a sealed Vault instance. Please ensure that the
|
||||||
|
requisite number of unseal keys are available before continuing.
|
||||||
|
|
||||||
|
Procedure
|
||||||
|
---------
|
||||||
|
|
||||||
|
If the unit being removed is in a 'lost' state (as seen in :command:`juju
|
||||||
|
status`) please first see the `Notes`_ section.
|
||||||
|
|
||||||
|
List the application units
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Display the units, in this case for the vault application:
|
||||||
|
|
||||||
|
.. code-block:: none
|
||||||
|
|
||||||
|
juju status vault
|
||||||
|
|
||||||
|
This article will be based on the following (partial) output:
|
||||||
|
|
||||||
|
.. code-block:: console
|
||||||
|
|
||||||
|
Unit Workload Agent Machine Public address Ports Message
|
||||||
|
vault/0* active idle 1/lxd/4 10.246.114.76 8200/tcp Unit is ready (active: false, mlock: disabled)
|
||||||
|
vault-hacluster/1 active idle 10.246.114.76 Unit is ready and clustered
|
||||||
|
vault-mysql-router/0* active idle 10.246.114.76 Unit is ready
|
||||||
|
vault/3 active idle 0/lxd/8 10.246.114.83 8200/tcp Unit is ready (active: true, mlock: disabled)
|
||||||
|
vault-hacluster/2 active idle 10.246.114.83 Unit is ready and clustered
|
||||||
|
vault-mysql-router/25 active idle 10.246.114.83 Unit is ready
|
||||||
|
vault/4 active idle 2/lxd/9 10.246.114.84 8200/tcp Unit is ready (active: false, mlock: disabled)
|
||||||
|
vault-hacluster/0* active idle 10.246.114.84 Unit is ready and clustered
|
||||||
|
vault-mysql-router/24 active idle 10.246.114.84 Unit is ready
|
||||||
|
|
||||||
|
In this example, unit ``vault/3`` will be removed.
|
||||||
|
|
||||||
|
Pause the subordinate hacluster unit
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Pause the hacluster unit that corresponds to the principal application unit
|
||||||
|
being removed. Here, unit ``vault-hacluster/2`` corresponds to unit
|
||||||
|
``vault/3``:
|
||||||
|
|
||||||
|
.. code-block:: none
|
||||||
|
|
||||||
|
juju run-action --wait vault-hacluster/2 pause
|
||||||
|
|
||||||
|
Remove the principal application unit
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Remove the principal application unit:
|
||||||
|
|
||||||
|
.. code-block:: none
|
||||||
|
|
||||||
|
juju remove-unit vault/3
|
||||||
|
|
||||||
|
This will also remove the hacluster subordinate unit (and any other subordinate
|
||||||
|
units).
|
||||||
|
|
||||||
|
Add a principal application unit
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Add a principal application unit. We accomplish this by scaling out the
|
||||||
|
existing vault application and placing the new (containerised) unit on the same
|
||||||
|
host that the removed unit was on (machine 0):
|
||||||
|
|
||||||
|
.. code-block:: none
|
||||||
|
|
||||||
|
juju add-unit --to lxd:0 vault
|
||||||
|
|
||||||
|
.. caution::
|
||||||
|
|
||||||
|
If network spaces are in use the above command will not succeed. See Juju
|
||||||
|
issue `LP #1969523`_ for a workaround.
|
||||||
|
|
||||||
|
The new :command:`juju status` output now contains:
|
||||||
|
|
||||||
|
.. code-block:: console
|
||||||
|
|
||||||
|
Unit Workload Agent Machine Public address Ports Message
|
||||||
|
vault/0* active idle 1/lxd/4 10.246.114.76 8200/tcp Unit is ready (active: false, mlock: disabled)
|
||||||
|
vault-hacluster/1 active idle 10.246.114.76 Unit is ready and clustered
|
||||||
|
vault-mysql-router/0* active idle 10.246.114.76 Unit is ready
|
||||||
|
vault/4 active idle 2/lxd/9 10.246.114.84 8200/tcp Unit is ready (active: true, mlock: disabled)
|
||||||
|
vault-hacluster/0* active idle 10.246.114.84 Unit is ready and clustered
|
||||||
|
vault-mysql-router/24 active idle 10.246.114.84 Unit is ready
|
||||||
|
vault/6 blocked idle 0/lxd/9 10.246.114.83 8200/tcp Unit is sealed
|
||||||
|
vault-hacluster/28 active idle 10.246.114.83 Unit is ready and clustered
|
||||||
|
vault-mysql-router/40 active idle 10.246.114.83 Unit is ready
|
||||||
|
|
||||||
|
Notice that the new vault unit (``vault/6``) is sealed.
|
||||||
|
|
||||||
|
Unseal the new Vault instance
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Here we will assume that the original Vault deploy was initialised with a
|
||||||
|
requirement of three unseal keys.
|
||||||
|
|
||||||
|
Set an environment variable based on the address of the newly-introduced unit,
|
||||||
|
and unseal the instance:
|
||||||
|
|
||||||
|
.. code-block:: none
|
||||||
|
|
||||||
|
export VAULT_ADDR="http://10.246.114.83:8200"
|
||||||
|
|
||||||
|
vault operator unseal
|
||||||
|
vault operator unseal
|
||||||
|
vault operator unseal
|
||||||
|
|
||||||
|
For more information on unsealing Vault see cloud operation :doc:`Unseal Vault
|
||||||
|
<ops-unseal-vault>`.
|
||||||
|
|
||||||
|
Verify cloud services
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
The final :command:`juju status vault` (partial) output is:
|
||||||
|
|
||||||
|
.. code-block:: console
|
||||||
|
|
||||||
|
Unit Workload Agent Machine Public address Ports Message
|
||||||
|
vault/0* active idle 1/lxd/4 10.246.114.76 8200/tcp Unit is ready (active: false, mlock: disabled)
|
||||||
|
vault-hacluster/1 active idle 10.246.114.76 Unit is ready and clustered
|
||||||
|
vault-mysql-router/0* active idle 10.246.114.76 Unit is ready
|
||||||
|
vault/4 active idle 2/lxd/9 10.246.114.84 8200/tcp Unit is ready (active: true, mlock: disabled)
|
||||||
|
vault-hacluster/0* active idle 10.246.114.84 Unit is ready and clustered
|
||||||
|
vault-mysql-router/24 active idle 10.246.114.84 Unit is ready
|
||||||
|
vault/6 active idle 0/lxd/9 10.246.114.83 8200/tcp Unit is ready (active: false, mlock: disabled)
|
||||||
|
vault-hacluster/28 active idle 10.246.114.83 Unit is ready and clustered
|
||||||
|
vault-mysql-router/40 active idle 10.246.114.83 Unit is ready
|
||||||
|
|
||||||
|
Ensure that all cloud services are working as expected.
|
||||||
|
|
||||||
|
Notes
|
||||||
|
-----
|
||||||
|
|
||||||
|
Pre-removal, in the case where the principal application unit has transitioned
|
||||||
|
to a 'lost' state (e.g. dropped off the network due to a hardware failure),
|
||||||
|
|
||||||
|
#. the first step (pause the hacluster unit) can be skipped
|
||||||
|
#. the second step (remove the principal unit) can be replaced by:
|
||||||
|
|
||||||
|
.. code-block:: none
|
||||||
|
|
||||||
|
juju remove-machine N --force
|
||||||
|
|
||||||
|
N is the Juju machine ID (see the :command:`juju status` command) where the
|
||||||
|
unit to be removed is running.
|
||||||
|
|
||||||
|
.. warning::
|
||||||
|
|
||||||
|
Removing the machine by force will naturally remove any other units that
|
||||||
|
may be present, including those from an entirely different application.
|
||||||
|
|
||||||
|
.. LINKS
|
||||||
|
.. _LP #1969523: https://bugs.launchpad.net/juju/+bug/1969523
|
|
@ -1,159 +0,0 @@
|
||||||
:orphan:
|
|
||||||
|
|
||||||
==================================================
|
|
||||||
Scale back an application with the hacluster charm
|
|
||||||
==================================================
|
|
||||||
|
|
||||||
Introduction
|
|
||||||
------------
|
|
||||||
|
|
||||||
This article shows how to scale back an application that is made highly
|
|
||||||
available by means of the subordinate hacluster charm. It implies the removal
|
|
||||||
of one or more of the principal application's units. This is easily done with
|
|
||||||
generic Juju commands and actions available to the hacluster charm.
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
Since the application being scaled back is already in HA mode the removal of
|
|
||||||
one of its cluster members should not cause any immediate interruption of
|
|
||||||
cloud services.
|
|
||||||
|
|
||||||
Scaling back an application will also remove its associated hacluster unit.
|
|
||||||
It is best practice to have at least three hacluster units per application
|
|
||||||
at all times. An odd number is also recommended.
|
|
||||||
|
|
||||||
Procedure
|
|
||||||
---------
|
|
||||||
|
|
||||||
If the unit being removed is in a 'lost' state (as seen in :command:`juju
|
|
||||||
status`) please first see the `Notes`_ section.
|
|
||||||
|
|
||||||
List the application units
|
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
||||||
|
|
||||||
Display the units, in this case for the vault application:
|
|
||||||
|
|
||||||
.. code-block:: none
|
|
||||||
|
|
||||||
juju status vault
|
|
||||||
|
|
||||||
This article will be based on the following output:
|
|
||||||
|
|
||||||
.. code-block:: console
|
|
||||||
|
|
||||||
Unit Workload Agent Machine Public address Ports Message
|
|
||||||
vault/0* active idle 0/lxd/5 10.0.0.227 8200/tcp Unit is ready (active: true, mlock: disabled)
|
|
||||||
vault-hacluster/0* active idle 10.0.0.227 Unit is ready and clustered
|
|
||||||
vault-mysql-router/0* active idle 10.0.0.227 Unit is ready
|
|
||||||
vault/1 active idle 1/lxd/5 10.0.0.234 8200/tcp Unit is ready (active: true, mlock: disabled)
|
|
||||||
vault-hacluster/1 active idle 10.0.0.234 Unit is ready and clustered
|
|
||||||
vault-mysql-router/1 active idle 10.0.0.234 Unit is ready
|
|
||||||
vault/2 active idle 2/lxd/6 10.0.0.233 8200/tcp Unit is ready (active: true, mlock: disabled)
|
|
||||||
vault-hacluster/2 active idle 10.0.0.233 Unit is ready and clustered
|
|
||||||
vault-mysql-router/2 active idle 10.0.0.233 Unit is ready
|
|
||||||
|
|
||||||
In the below example, unit ``vault/1`` will be removed.
|
|
||||||
|
|
||||||
Pause the subordinate hacluster unit
|
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
||||||
|
|
||||||
Pause the hacluster unit that corresponds to the principle application unit
|
|
||||||
being removed. Here, unit ``vault-hacluster/1`` corresponds to unit
|
|
||||||
``vault/1``:
|
|
||||||
|
|
||||||
.. code-block:: none
|
|
||||||
|
|
||||||
juju run-action --wait vault-hacluster/1 pause
|
|
||||||
|
|
||||||
.. caution::
|
|
||||||
|
|
||||||
Unit numbers for a subordinate unit and its corresponding principal unit are
|
|
||||||
not necessarily the same (e.g. it is possible to have ``vault-hacluster/2``
|
|
||||||
correspond to ``vault/1``).
|
|
||||||
|
|
||||||
Remove the principal application unit
|
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
||||||
|
|
||||||
Remove the principal application unit:
|
|
||||||
|
|
||||||
.. code-block:: none
|
|
||||||
|
|
||||||
juju remove-unit vault/1
|
|
||||||
|
|
||||||
This will also remove the hacluster subordinate unit (and any other subordinate
|
|
||||||
units).
|
|
||||||
|
|
||||||
Update the ``cluster_count`` value
|
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
||||||
|
|
||||||
Inform the hacluster charm about the new number of hacluster units, two here:
|
|
||||||
|
|
||||||
.. code-block:: none
|
|
||||||
|
|
||||||
juju config vault-hacluster cluster_count=2
|
|
||||||
|
|
||||||
In this example a count of two (less than three) removes quorum functionality
|
|
||||||
and enables a two-node cluster. This is a sub-optimal state and is shown as an
|
|
||||||
example only.
|
|
||||||
|
|
||||||
Update Corosync
|
|
||||||
~~~~~~~~~~~~~~~
|
|
||||||
|
|
||||||
Remove Corosync nodes from its ring and update ``corosync.conf`` to reflect the
|
|
||||||
new number of nodes (``min_quorum`` is recalculated):
|
|
||||||
|
|
||||||
.. code-block:: none
|
|
||||||
|
|
||||||
juju run-action --wait vault-hacluster/leader update-ring i-really-mean-it=true
|
|
||||||
|
|
||||||
Check the status of the Corosync cluster by querying a remaining hacluster
|
|
||||||
unit:
|
|
||||||
|
|
||||||
.. code-block:: none
|
|
||||||
|
|
||||||
juju ssh vault-hacluster/leader sudo crm status
|
|
||||||
|
|
||||||
There should not be any node listed as OFFLINE.
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
With Juju client < 2.9 a subordinate leader unit must be referenced via its
|
|
||||||
machine ID (e.g. 0/lxd/5) when using the :command:`juju ssh` command.
|
|
||||||
|
|
||||||
Verify cloud services
|
|
||||||
~~~~~~~~~~~~~~~~~~~~~
|
|
||||||
|
|
||||||
For this example, the final :command:`juju status vault` output is:
|
|
||||||
|
|
||||||
.. code-block:: console
|
|
||||||
|
|
||||||
Unit Workload Agent Machine Public address Ports Message
|
|
||||||
vault/0* active idle 0/lxd/5 10.0.0.227 8200/tcp Unit is ready (active: true, mlock: disabled)
|
|
||||||
vault-hacluster/0* active idle 10.0.0.227 Unit is ready and clustered
|
|
||||||
vault-mysql-router/0* active idle 10.0.0.227 Unit is ready
|
|
||||||
vault/2 active idle 2/lxd/6 10.0.0.233 8200/tcp Unit is ready (active: true, mlock: disabled)
|
|
||||||
vault-hacluster/2 active idle 10.0.0.233 Unit is ready and clustered
|
|
||||||
vault-mysql-router/2 active idle 10.0.0.233 Unit is ready
|
|
||||||
|
|
||||||
Ensure that all cloud services are working as expected.
|
|
||||||
|
|
||||||
Notes
|
|
||||||
-----
|
|
||||||
|
|
||||||
Pre-removal, in the case where the principal application unit has transitioned
|
|
||||||
to a 'lost' state (e.g. dropped off the network due to a hardware failure),
|
|
||||||
|
|
||||||
#. the first step (pause the hacluster unit) can be skipped
|
|
||||||
#. the second step (remove the principal unit) can be replaced by:
|
|
||||||
|
|
||||||
.. code-block:: none
|
|
||||||
|
|
||||||
juju remove-machine N --force
|
|
||||||
|
|
||||||
N is the Juju machine ID (see the :command:`juju status` command) where the
|
|
||||||
unit to be removed is running.
|
|
||||||
|
|
||||||
.. warning::
|
|
||||||
|
|
||||||
Removing the machine by force will naturally remove any other units that
|
|
||||||
may be present, including those from an entirely different application.
|
|
|
@ -51,6 +51,13 @@ For a single unit requiring three keys (``vault/0`` with IP address
|
||||||
vault operator unseal
|
vault operator unseal
|
||||||
vault operator unseal
|
vault operator unseal
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
|
||||||
|
If the Vault API is encrypted you will need to inform your Vault client of
|
||||||
|
the associated CA certificate via an additional variable (VAULT_CACERT). See
|
||||||
|
cloud operation :doc:`Configure TLS for the Vault API
|
||||||
|
<ops-config-tls-vault-api>`.
|
||||||
|
|
||||||
You will be prompted for the unseal keys. The information will not be echoed
|
You will be prompted for the unseal keys. The information will not be echoed
|
||||||
back to the screen nor captured in the shell's history.
|
back to the screen nor captured in the shell's history.
|
||||||
|
|
||||||
|
|
Loading…
Reference in New Issue