fuel-specs/specs/6.1/controller-maintenance-mode.rst

Support for maintenance mode on controllers

https://blueprints.launchpad.net/fuel/+spec/controller-maintenance-mode

Support for maintenance mode on controllers

Problem description

First

there is no unified way to obtain needed state across all operation systems (OS) used within fuel. There are different ways for each of them:

• for Ubuntu it is recovery mode
• for Centos 6 it is single mode with modification
• for Centos 7 it is rescue runlevel.

So we have to use different algorithms which depend on OS. Common for all mentioned operating systems single mode does not provide network access and has different behavior.

Second

when we stop services or use switching runlevels for obtaining maintenance mode (MM), very often we have “garbage things” (zombie processes, locks, memory leaks, etc). It doesn’t let to do service staff properly.

Third

dependency and interaction between services. If we want to stop some service manually, we have to keep in mind all its dependencies and take care of them as well.

Fourth

similar mode in other OS, for example “Windows safe mode”, has mechanism for automatic enforcing MM mode, if we have unexpected emergency reboot. So we need the same, but because we usually don't have access to console in one hand and in other hand automatic enforce of MM should accept some “emergency” reboots.

Fifth

some services have own “maintenance mode” such as corosync which let us do the same things. But they may do it in a different way than required by us, they may be absent in current cloud configuration.

Sixth

HA services may look node in MM like node in “fail state” because services on it don’t stop own work properly.

Proposed change

Proposed features

• Solution should be compatible with Ubuntu and CentOS, and have unified OS independent interface.
• Ability to enter maintenance mode manually.
• Ability to enter maintenance mode automatically when operating system keeps rebooting unexpectedly. If we have more than REBOOT_COUNT “unclean” reboots MM will be enforced. “unclean” means unexpected reboot by COUNTER_RESET_TIME sec from boot.
• Maintenance mode should stop all services except really crucial ones like networking, ssh, etc.
• Add ability to boot node in maintenance mode.
• There should be configuration parameter to switch off MM functionality.
• Allow schedule command which will be execute after switching into MM.
• Allow specify custom command which will be execute before switching into and from MM.

Implementation model

General overview

We will create common procedure and unified interface for all operating system which are used by fuel. It let us enforce MM state and return into operational mode in a unified way for all operating systems. Under the hood it will be based on boot scripts and mechanisms which are specific for each operating system. It is possible that we will introduce some changes to these mechanisms to obtain proper set of services running in MM.

This procedure is not a service of openstack, but unification of recovery procedures across all OSs. It will give us the same user interface across all used systems.

Interface description

It is suggested to create “umm” utility which will enforce maintenance mode on the system and resume normal operation.

Usage:

umm status                               - check mm status

umm on [command to execute in mm mode]   - enforce MM mode [and execute
                                           command when MM is reached]

umm off [reboot]                         - continue boot process [or reboot]
                                           into operational mode.

umm enable                               - enable mm functionality

umm disable                              - disable mm functionality

Architectural design

To avoid “garbage things” described in the second problem, maintenance mode will be obtained only by reboot and subsequent pausing of the boot process on apropriate state and resuming it when we want switch back into operational state.

It lets us:

• be sure that only needed processes work on the node
• don’t process start and stop dependencies manually
• expect that all services will be properly notified and all roles will be transferred to other nodes. So in case HA configuration - cluster will still work properly if we have enough working controllers.

For that we will modify boot-shutdown mechanism and create state in which only network, ssh daemon and services which are needed for them are run.

We will modify current boot process for automatically enforce MM if system has some “unexpected” reboot during established time.

Delivery details

Full set of needed files will be provided as a single puppet class. Also we will provide UMM task for granular deployment in 6.1.

Alternatives

Corosync MM

OS native recovery tools - for example friendly-recovery Ubuntu package

Data model impact

None

REST API impact

None

Upgrade impact

None

Security impact

None

Notifications impact

None

Other end user impact

• Potentially proposed changes could introduce additional tool to manage Maintenance mode.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee

Peter Zhurba

QA

Veronika Krayneva

Documentation

Peter Zhurba, Dmitry Klenov

Reviewer

Vladimir Kuklin

Work Items

1. Create common interface for all operating systems.
2. Create Ubuntu solution. Modification upstart configs and grub loader.
3. Create Centos solution. Modification initscript configs and grub loader.
4. Create Centos solution. Modification SystemD configs and grub loader.
5. Modify controller deploying procedure.

Dependencies

None

Testing

Boot in maintenance mode on one node

Preconditions

All actions are performed on the same controller. Once finished with these actions, move on to another controller

Actions

1. Enter maintenance mode
2. Wait when maintenance mode is reached
3. Leave maintenance mode

Expected result

1. Maintenance mode is enabled
2. Maintenance mode started successfully
3. All services start successfully when maintenance mode is switched off

Auto maintenance mode on one node

Preconditions

All actions are performed on the same controller. Once finished with all actions, move on to another controller

Actions

1. X or more unexpected reboots per Y min
2. Wait when maintenance mode starts
3. Disable maintenance mode
4. X or more unexpected reboots per Y min

Expected result

1. Reboot finished successfully
2. Maintenance mode started successfully
3. MM is disabled
4. After MM is disabled, MM shouldn't be reached after unexpected reboot

Documentation Impact

Operations Guide      -> “Maintenance Mode” will be added.

Terminology Reference -> “Maintenance Mode” will be added.

References

• https://blueprints.launchpad.net/fuel/+spec/controller-maintenance-mode