You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

123 lines
4.3 KiB

Self-healing and auto-scaling with Heat, Mistral, Zaqar, Aodh
This use case describes an existing scenario for auto-scaling a
self-healing cluster.
OpenStack projects used
* Heat
* Mistral
* Zaqar
* Telemetry (Aodh, Pinko, Gnocchi)
Remediation class
Fault detection
The self-healing scenario detect failure like when any instance in
cluster been delete, or updated out side of this automatic flow.
Inputs and decision-making
* A cluster builder (Heat):
You can use Heat to build cluster, managing resource dependancies,
plan some flexibility with batch action, and also reproducable to
another OpenStack environment with same template. Whatever you're
using for building cluster, you need to make sure it allow to your
flow control service (Mistral) to trigger update and replace
unhealthy node in cluster.
* A monitoring service (Aodh):
There are multiple choices for self-healing monitors to monitor that
cluster and notify queue service; you can absolutely replace any
monitoring service in this showcase and use your own existing
services. Just make sure you're able to monitoring nodes in cluster
and able to send notification out when alarm triggered.
* A queue service (Zaqar):
A queue or any services that accept notification from alarm service
and capable to trigger healing flow.
* A triggerable flow control service (Mistral):
A triggerable flow control service to run healing flow. Make sure
your flow contains most cases. So we can guarantee that cluster is
healed. Covering most successes and failure cases (as many as you
can) is always a good idea.
We're using Heat, Aodh, Zaqar, and Mistral in this example. You can
always replace any service with your own services. Just make sure
they're fully integrated to each other for your healing scenario.
In our example, for each unit in the cluster, we set up a Nova
instance to run application on, binding that instance to a
load-balancer so the application can use lbaas in its infrastructure.
For healing, we create multiple event alarms monitoring the status of
that instance, a Zaqar queue for alarms to send notifications on, and
a Zaqar MistralTrigger (a resource type in Heat to use Zaqar
subscription to trigger Mistral workflow) to listen on that queue and
trigger Mistral workflow to mark this unit as unhealthy and then use
Heat stack update to replace unhealthy unit in cluster.
At this point you already got your structure for self-healing. As you
can see in the example, we are monitoring events like
``compute.instance.delete.*``. If you also use Aodh, please make sure
the event you're monitoring is enabled from services and ceilometer
(be aware that some events are disabled by default).
The event alarm going to send a ``POST`` request to Zaqar queue if the
instance status changed to delete. Again, it's possible to use your
own alarm services.
Once Zaqar queue gets that information, it will trigger a Mistral
workflow. That workflow will mark that unit (in this case it's a
server with nginx) as unhealthy in Heat and trigger a stack
update. Heat will replace all unhealthy unit and regenerate one for
your cluster.
Existing implementation(s)
With ability to self-heal, you can (as we show in this case) combine
with autoscaling ability. Also as you can notice, we use Heat
container agent, so your image only need to support ability to run
container (instead of pre-install services with
diskimage-builder). Here is the example Heat template :doc:`sample
files <heat-mistral-aodh>`. Also here's link for relevant `slides
(with video link)
Future work
We need better support for scaling with some resources require trust.
All further work is tracked in `the story "Improve Autoscaling
integration with Self-healing"
<!/story/2003690>`_. Also
integration with Vitrage is work in progress tracked in `the story
"Achieve Self healing with Vitrage Mistral And Heat"