openstack-manuals/doc/arch-design-draft/source/operator-requirements.rst

Operator requirements

Introduction

Several operational factors affect the design choices for a general purpose cloud. Operations staff receive tasks regarding the maintenance of cloud environments, including:

Maintenance tasks

Operating system patching, hardware/firmware upgrades, and datacenter related changes, as well as minor and release upgrades to OpenStack components are all ongoing operational tasks. In particular, the six monthly release cycle of the OpenStack projects needs to be considered as part of the cost of ongoing maintenance. The solution should take into account storage and network maintenance and the impact on underlying workloads.

Reliability and availability

Reliability and availability depend on the many supporting components' availability and on the level of precautions taken by the service provider. This includes network, storage systems, datacenter, and operating systems.

In order to run efficiently, automate as many of the operational processes as possible. Automation includes the configuration of provisioning, monitoring and alerting systems. Part of the automation process includes the capability to determine when human intervention is required and who should act. The objective is to increase the ratio of operational staff to running systems as much as possible in order to reduce maintenance costs. In a massively scaled environment, it is very difficult for staff to give each system individual care.

Configuration management tools such as Ansible, Puppet, and Chef enable operations staff to categorize systems into groups based on their roles and thus create configurations and system states that the provisioning system enforces. Systems that fall out of the defined state due to errors or failures are quickly removed from the pool of active nodes and replaced.

At large scale, the resource cost of diagnosing failed individual systems is far greater than the cost of replacement. It is more economical to replace the failed system with a new system, provisioning and configuring it automatically and adding it to the pool of active nodes. By automating tasks that are labor-intensive, repetitive, and critical to operations, cloud operations teams can work more efficiently because fewer resources are required for these common tasks. Administrators are then free to tackle tasks that are not easy to automate and that have longer-term impacts on the business, for example, capacity planning.

SLA Considerations

Service-level agreements (SLAs) are contractual obligations that ensure the availability of a service. When designing an OpenStack cloud, factoring in promises of availability implies a certain level of redundancy and resiliency.

Expectations set by the Service Level Agreements (SLAs) directly affect knowing when and where you should implement redundancy and high availability. SLAs are contractual obligations that provide assurances for service availability. They define the levels of availability that drive the technical design, often with penalties for not meeting contractual obligations.

SLA terms that affect design include:

• API availability guarantees implying multiple infrastructure services and highly available load balancers.
• Network uptime guarantees affecting switch design, which might require redundant switching and power.
• Factor in networking security policy requirements in to your deployments.

In any environment larger than just a few hosts, it is important to note that there are two separate areas that might be subject to an SLA. Firstly, the services that provide actual virtualization, networking and storage, are subject to an SLA that customers of the environment are most likely to want to be continuously available. This is often referred to as the 'Data Plane'.

Secondly, there are the ancillary services such as API endpoints, and the various services that control CRUD operations. These are often referred to as the 'Control Plane'. The services in this category are usually subject to a different SLA expectation and therefore may be better suited on separate hardware or at least containers from the Data Plane services.

To effectively run cloud installations, initial downtime planning includes creating processes and architectures that support the following:

• Planned (maintenance)
• Unplanned (system faults)

It is important to determine as part of the SLA negotiation which party is responsible for monitoring and starting up Compute service Instances should an outage occur which shuts them down.

Resiliency of overall system and individual components are going to be dictated by the requirements of the SLA, meaning designing for high availability (HA) can have cost ramifications.

When upgrading, patching and changing configuration items this may require downtime for some services. In these cases, stopping services that form the Control Plane may leave the Data Plane unaffected, while actions such as live-migration of Compute instances may be required in order to perform any actions that require downtime to Data Plane components whilst still meeting SLA expectations.

Note that there are many services that are outside the realms of pure OpenStack code which affects the ability of any given design to meet SLA, including:

• Database services, such as MySQL or PostgreSQL.
• Services providing RPC, such as RabbitMQ.
• External network attachments.
• Physical constraints such as power, rack space, network cabling, etc.
• Shared storage including SAN based arrays, storage clusters such as Ceph, and/or NFS services.

Depending on the design, some Network service functions may fall into both the Control and Data Plane categories. E.g. the neutron L3 Agent service may be considered a Control Plane component, but the routers themselves would be Data Plane.

It may be that a given set of requirements could dictate an SLA that suggests some services need HA, and some do not.

In a design with multiple regions, the SLA would also need to take into consideration the use of shared services such as the Identity service, Dashboard, and so on.

Any SLA negotiation must also take into account the reliance on 3rd parties for critical aspects of the design - for example, if there is an existing SLA on a component such as a storage system, the cloud SLA must take into account this limitation. If the required SLA for the cloud exceeds the agreed uptime levels of the components comprising that cloud, additional redundancy would be required. This consideration is critical to review in a hybrid cloud design, where there are multiple 3rd parties involved.

Logging and Monitoring

OpenStack clouds require appropriate monitoring platforms to catch and manage errors.

Note

We recommend leveraging existing monitoring systems to see if they are able to effectively monitor an OpenStack environment.

Specific meters that are critically important to capture include:

• Image disk utilization
• Response time to the Compute API

Logging and monitoring does not significantly differ for a multi-site OpenStack cloud. The tools described in the Logging and monitoring chapter of the Operations Guide remain applicable. Logging and monitoring can be provided on a per-site basis, and in a common centralized location.

When attempting to deploy logging and monitoring facilities to a centralized location, care must be taken with the load placed on the inter-site networking links.

Network

The network design for an OpenStack cluster includes decisions regarding the interconnect needs within the cluster, plus the need to allow clients to access their resources, and for operators to access the cluster for maintenance. The bandwidth, latency, and reliability of these networks needs consideration.

Make additional design decisions about monitoring and alarming. This can be an internal responsibility or the responsibility of the external provider. In the case of using an external provider, service level agreements (SLAs) likely apply. In addition, other operational considerations such as bandwidth, latency, and jitter can be part of an SLA.

Consider the ability to upgrade the infrastructure. As demand for network resources increase, operators add additional IP address blocks and add additional bandwidth capacity. In addition, consider managing hardware and software life cycle events, for example upgrades, decommissioning, and outages, while avoiding service interruptions for tenants.

Factor maintainability into the overall network design. This includes the ability to manage and maintain IP addresses as well as the use of overlay identifiers including VLAN tag IDs, GRE tunnel IDs, and MPLS tags. As an example, if you may need to change all of the IP addresses on a network, a process known as renumbering, then the design must support this function.

Address network-focused applications when considering certain operational realities. For example, consider the impending exhaustion of IPv4 addresses, the migration to IPv6, and the use of private networks to segregate different types of traffic that an application receives or generates. In the case of IPv4 to IPv6 migrations, applications should follow best practices for storing IP addresses. We recommend you avoid relying on IPv4 features that did not carry over to the IPv6 protocol or have differences in implementation.

To segregate traffic, allow applications to create a private tenant network for database and storage network traffic. Use a public network for services that require direct client access from the internet. Upon segregating the traffic, consider quality of service (QoS) and security to ensure each network has the required level of service.

Finally, consider the routing of network traffic. For some applications, develop a complex policy framework for routing. To create a routing policy that satisfies business requirements, consider the economic cost of transmitting traffic over expensive links versus cheaper links, in addition to bandwidth, latency, and jitter requirements.

Additionally, consider how to respond to network events. As an example, how load transfers from one link to another during a failure scenario could be a factor in the design. If you do not plan network capacity correctly, failover traffic could overwhelm other ports or network links and create a cascading failure scenario. In this case, traffic that fails over to one link overwhelms that link and then moves to the subsequent links until all network traffic stops.

Licensing

The many different forms of license agreements for software are often written with the use of dedicated hardware in mind. This model is relevant for the cloud platform itself, including the hypervisor operating system, supporting software for items such as database, RPC, backup, and so on. Consideration must be made when offering Compute service instances and applications to end users of the cloud, since the license terms for that software may need some adjustment to be able to operate economically in the cloud.

Multi-site OpenStack deployments present additional licensing considerations over and above regular OpenStack clouds, particularly where site licenses are in use to provide cost efficient access to software licenses. The licensing for host operating systems, guest operating systems, OpenStack distributions (if applicable), software-defined infrastructure including network controllers and storage systems, and even individual applications need to be evaluated.

Topics to consider include:

• The definition of what constitutes a site in the relevant licenses, as the term does not necessarily denote a geographic or otherwise physically isolated location.
• Differentiations between "hot" (active) and "cold" (inactive) sites, where significant savings may be made in situations where one site is a cold standby for disaster recovery purposes only.
• Certain locations might require local vendors to provide support and services for each site which may vary with the licensing agreement in place.

Support and maintainability

To be able to support and maintain an installation, OpenStack cloud management requires operations staff to understand and comprehend design architecture content. The operations and engineering staff skill level, and level of separation, are dependent on size and purpose of the installation. Large cloud service providers, or telecom providers, are more likely to be managed by specially trained, dedicated operations organizations. Smaller implementations are more likely to rely on support staff that need to take on combined engineering, design and operations functions.

Maintaining OpenStack installations requires a variety of technical skills. You may want to consider using a third-party management company with special expertise in managing OpenStack deployment.

Operator access to systems

As more and more applications are migrated into a Cloud based environment, we get to a position where systems that are critical for Cloud operations are hosted within the cloud that is being operated. Consideration must be given to the ability for operators to be able to access the systems and tools required in order to resolve a major incident.

If a significant portion of the cloud is on externally managed systems, prepare for situations where it may not be possible to make changes. Additionally, providers may differ on how infrastructure must be managed and exposed. This can lead to delays in root cause analysis where each insists the blame lies with the other provider.

Ensure that the network structure connects all clouds to form an integrated system, keeping in mind the state of handoffs. These handoffs must both be as reliable as possible and include as little latency as possible to ensure the best performance of the overall system.

Capacity planning

An important consideration in running a cloud over time is projecting growth and utilization trends in order to plan capital expenditures for the short and long term. Gather utilization meters for compute, network, and storage, along with historical records of these meters. While securing major anchor tenants can lead to rapid jumps in the utilization rates of all resources, the steady adoption of the cloud inside an organization or by consumers in a public offering also creates a steady trend of increased utilization.

Capacity constraints for a general purpose cloud environment include:

• Compute limits
• Storage limits

A relationship exists between the size of the compute environment and the supporting OpenStack infrastructure controller nodes requiring support.

Increasing the size of the supporting compute environment increases the network traffic and messages, adding load to the controller or networking nodes. Effective monitoring of the environment will help with capacity decisions on scaling.

Compute nodes automatically attach to OpenStack clouds, resulting in a horizontally scaling process when adding extra compute capacity to an OpenStack cloud. Additional processes are required to place nodes into appropriate availability zones and host aggregates. When adding additional compute nodes to environments, ensure identical or functional compatible CPUs are used, otherwise live migration features will break. It is necessary to add rack capacity or network switches as scaling out compute hosts directly affects network and datacenter resources.

Compute host components can also be upgraded to account for increases in demand; this is known as vertical scaling. Upgrading CPUs with more cores, or increasing the overall server memory, can add extra needed capacity depending on whether the running applications are more CPU intensive or memory intensive.

Another option is to assess the average workloads and increase the number of instances that can run within the compute environment by adjusting the overcommit ratio.

Note

It is important to remember that changing the CPU overcommit ratio can have a detrimental effect and cause a potential increase in a noisy neighbor.

Insufficient disk capacity could also have a negative effect on overall performance including CPU and memory usage. Depending on the back-end architecture of the OpenStack Block Storage layer, capacity includes adding disk shelves to enterprise storage systems or installing additional block storage nodes. Upgrading directly attached storage installed in compute hosts, and adding capacity to the shared storage for additional ephemeral storage to instances, may be necessary.

For a deeper discussion on many of these topics, refer to the OpenStack Operations Guide.

Quota management

Quotas are used to set operational limits to prevent system capacities from being exhausted without notification. They are currently enforced at the tenant (or project) level rather than at the user level.

Quotas are defined on a per-region basis. Operators can define identical quotas for tenants in each region of the cloud to provide a consistent experience, or even create a process for synchronizing allocated quotas across regions. It is important to note that only the operational limits imposed by the quotas will be aligned consumption of quotas by users will not be reflected between regions.

For example, given a cloud with two regions, if the operator grants a user a quota of 25 instances in each region then that user may launch a total of 50 instances spread across both regions. They may not, however, launch more than 25 instances in any single region.

For more information on managing quotas refer to the Managing projects and users chapter of the OpenStack Operators Guide.

Policy management

OpenStack provides a default set of Role Based Access Control (RBAC) policies, defined in a policy.json file, for each service. Operators edit these files to customize the policies for their OpenStack installation. If the application of consistent RBAC policies across sites is a requirement, then it is necessary to ensure proper synchronization of the policy.json files to all installations.

This must be done using system administration tools such as rsync as functionality for synchronizing policies across regions is not currently provided within OpenStack.

Selecting Hardware

Integration with external IDP

Upgrades

Running OpenStack with a focus on availability requires striking a balance between stability and features. For example, it might be tempting to run an older stable release branch of OpenStack to make deployments easier. However known issues that may be of some concern or only have minimal impact in smaller deployments could become pain points as scale increases. Recent releases may address well known issues. The OpenStack community can help resolve reported issues by applying the collective expertise of the OpenStack developers.

In multi-site OpenStack clouds deployed using regions, sites are independent OpenStack installations which are linked together using shared centralized services such as OpenStack Identity. At a high level the recommended order of operations to upgrade an individual OpenStack environment is (see the Upgrades chapter of the Operations Guide for details):

1. Upgrade the OpenStack Identity service (keystone).
2. Upgrade the OpenStack Image service (glance).
3. Upgrade OpenStack Compute (nova), including networking components.
4. Upgrade OpenStack Block Storage (cinder).
5. Upgrade the OpenStack dashboard (horizon).

The process for upgrading a multi-site environment is not significantly different:

1. Upgrade the shared OpenStack Identity service (keystone) deployment.
2. Upgrade the OpenStack Image service (glance) at each site.
3. Upgrade OpenStack Compute (nova), including networking components, at each site.
4. Upgrade OpenStack Block Storage (cinder) at each site.
5. Upgrade the OpenStack dashboard (horizon), at each site or in the single central location if it is shared.

Compute upgrades within each site can also be performed in a rolling fashion. Compute controller services (API, Scheduler, and Conductor) can be upgraded prior to upgrading of individual compute nodes. This allows operations staff to keep a site operational for users of Compute services while performing an upgrade.

The bleeding edge

The number of organizations running at massive scales is a small proportion of the OpenStack community, therefore it is important to share related issues with the community and be a vocal advocate for resolving them. Some issues only manifest when operating at large scale, and the number of organizations able to duplicate and validate an issue is small, so it is important to document and dedicate resources to their resolution.

In some cases, the resolution to the problem is ultimately to deploy a more recent version of OpenStack. Alternatively, when you must resolve an issue in a production environment where rebuilding the entire environment is not an option, it is sometimes possible to deploy updates to specific underlying components in order to resolve issues or gain significant performance improvements. Although this may appear to expose the deployment to increased risk and instability, in many cases it could be an undiscovered issue.

We recommend building a development and operations organization that is responsible for creating desired features, diagnosing and resolving issues, and building the infrastructure for large scale continuous integration tests and continuous deployment. This helps catch bugs early and makes deployments faster and easier. In addition to development resources, we also recommend the recruitment of experts in the fields of message queues, databases, distributed systems, networking, cloud, and storage.

Skills and training

Projecting growth for storage, networking, and compute is only one aspect of a growth plan for running OpenStack at massive scale. Growing and nurturing development and operational staff is an additional consideration. Sending team members to OpenStack conferences, meetup events, and encouraging active participation in the mailing lists and committees is a very important way to maintain skills and forge relationships in the community. For a list of OpenStack training providers in the marketplace, see: http://www.openstack.org/marketplace/training/.