openstack-manuals/doc/arch-design/source/storage-focus-operational-considerations.rst

Operational Considerations

Several operational factors affect the design choices for a general purpose cloud. Operations staff receive tasks regarding the maintenance of cloud environments for larger installations, including:

Maintenance tasks

The storage solution should take into account storage maintenance and the impact on underlying workloads.

Reliability and availability

Reliability and availability depend on wide area network availability and on the level of precautions taken by the service provider.

Flexibility

Organizations need to have the flexibility to choose between off-premise and on-premise cloud storage options. This relies on relevant decision criteria with potential cost savings. For example, continuity of operations, disaster recovery, security, records retention laws, regulations, and policies.

Monitoring and alerting services are vital in cloud environments with high demands on storage resources. These services provide a real-time view into the health and performance of the storage systems. An integrated management console, or other dashboards capable of visualizing SNMP data, is helpful when discovering and resolving issues that arise within the storage cluster.

A storage-focused cloud design should include:

• Monitoring of physical hardware resources.
• Monitoring of environmental resources such as temperature and humidity.
• Monitoring of storage resources such as available storage, memory, and CPU.
• Monitoring of advanced storage performance data to ensure that storage systems are performing as expected.
• Monitoring of network resources for service disruptions which would affect access to storage.
• Centralized log collection.
• Log analytics capabilities.
• Ticketing system (or integration with a ticketing system) to track issues.
• Alerting and notification of responsible teams or automated systems which remediate problems with storage as they arise.
• Network Operations Center (NOC) staffed and always available to resolve issues.

Application awareness

Well-designed applications should be aware of underlying storage subsystems in order to use cloud storage solutions effectively.

If natively available replication is not available, operations personnel must be able to modify the application so that they can provide their own replication service. In the event that replication is unavailable, operations personnel can design applications to react such that they can provide their own replication services. An application designed to detect underlying storage systems can function in a wide variety of infrastructures, and still have the same basic behavior regardless of the differences in the underlying infrastructure.

Fault tolerance and availability

Designing for fault tolerance and availability of storage systems in an OpenStack cloud is vastly different when comparing the Block Storage and Object Storage services.

Block Storage fault tolerance and availability

Configure Block Storage resource nodes with advanced RAID controllers and high performance disks to provide fault tolerance at the hardware level.

Deploy high performing storage solutions such as SSD disk drives or flash storage systems for applications requiring extreme performance out of Block Storage devices.

In environments that place extreme demands on Block Storage, we recommend using multiple storage pools. In this case, each pool of devices should have a similar hardware design and disk configuration across all hardware nodes in that pool. This allows for a design that provides applications with access to a wide variety of Block Storage pools, each with their own redundancy, availability, and performance characteristics. When deploying multiple pools of storage it is also important to consider the impact on the Block Storage scheduler which is responsible for provisioning storage across resource nodes. Ensuring that applications can schedule volumes in multiple regions, each with their own network, power, and cooling infrastructure, can give projects the ability to build fault tolerant applications that are distributed across multiple availability zones.

In addition to the Block Storage resource nodes, it is important to design for high availability and redundancy of the APIs, and related services that are responsible for provisioning and providing access to storage. We recommend designing a layer of hardware or software load balancers in order to achieve high availability of the appropriate REST API services to provide uninterrupted service. In some cases, it may also be necessary to deploy an additional layer of load balancing to provide access to back-end database services responsible for servicing and storing the state of Block Storage volumes. We also recommend designing a highly available database solution to store the Block Storage databases. Leverage highly available database solutions such as Galera and MariaDB to help keep database services online for uninterrupted access, so that projects can manage Block Storage volumes.

In a cloud with extreme demands on Block Storage, the network architecture should take into account the amount of East-West bandwidth required for instances to make use of the available storage resources. The selected network devices should support jumbo frames for transferring large blocks of data. In some cases, it may be necessary to create an additional back-end storage network dedicated to providing connectivity between instances and Block Storage resources so that there is no contention of network resources.

Object Storage fault tolerance and availability

While consistency and partition tolerance are both inherent features of the Object Storage service, it is important to design the overall storage architecture to ensure that the implemented system meets those goals. The OpenStack Object Storage service places a specific number of data replicas as objects on resource nodes. These replicas are distributed throughout the cluster based on a consistent hash ring which exists on all nodes in the cluster.

Design the Object Storage system with a sufficient number of zones to provide quorum for the number of replicas defined. For example, with three replicas configured in the Swift cluster, the recommended number of zones to configure within the Object Storage cluster in order to achieve quorum is five. While it is possible to deploy a solution with fewer zones, the implied risk of doing so is that some data may not be available and API requests to certain objects stored in the cluster might fail. For this reason, ensure you properly account for the number of zones in the Object Storage cluster.

Each Object Storage zone should be self-contained within its own availability zone. Each availability zone should have independent access to network, power and cooling infrastructure to ensure uninterrupted access to data. In addition, a pool of Object Storage proxy servers providing access to data stored on the object nodes should service each availability zone. Object proxies in each region should leverage local read and write affinity so that local storage resources facilitate access to objects wherever possible. We recommend deploying upstream load balancing to ensure that proxy services are distributed across the multiple zones and, in some cases, it may be necessary to make use of third-party solutions to aid with geographical distribution of services.

A zone within an Object Storage cluster is a logical division. Any of the following may represent a zone:

• A disk within a single node
• One zone per node
• Zone per collection of nodes
• Multiple racks
• Multiple DCs

Selecting the proper zone design is crucial for allowing the Object Storage cluster to scale while providing an available and redundant storage system. It may be necessary to configure storage policies that have different requirements with regards to replicas, retention and other factors that could heavily affect the design of storage in a specific zone.

Scaling storage services

Adding storage capacity and bandwidth is a very different process when comparing the Block and Object Storage services. While adding Block Storage capacity is a relatively simple process, adding capacity and bandwidth to the Object Storage systems is a complex task that requires careful planning and consideration during the design phase.

Scaling Block Storage

You can upgrade Block Storage pools to add storage capacity without interrupting the overall Block Storage service. Add nodes to the pool by installing and configuring the appropriate hardware and software and then allowing that node to report in to the proper storage pool via the message bus. This is because Block Storage nodes report into the scheduler service advertising their availability. After the node is online and available, projects can make use of those storage resources instantly.

In some cases, the demand on Block Storage from instances may exhaust the available network bandwidth. As a result, design network infrastructure that services Block Storage resources in such a way that you can add capacity and bandwidth easily. This often involves the use of dynamic routing protocols or advanced networking solutions to add capacity to downstream devices easily. Both the front-end and back-end storage network designs should encompass the ability to quickly and easily add capacity and bandwidth.

Scaling Object Storage

Adding back-end storage capacity to an Object Storage cluster requires careful planning and consideration. In the design phase, it is important to determine the maximum partition power required by the Object Storage service, which determines the maximum number of partitions which can exist. Object Storage distributes data among all available storage, but a partition cannot span more than one disk, so the maximum number of partitions can only be as high as the number of disks.

For example, a system that starts with a single disk and a partition power of 3 can have 8 (2^3) partitions. Adding a second disk means that each has 4 partitions. The one-disk-per-partition limit means that this system can never have more than 8 disks, limiting its scalability. However, a system that starts with a single disk and a partition power of 10 can have up to 1024 (2^10) disks.

As you add back-end storage capacity to the system, the partition maps redistribute data amongst the storage nodes. In some cases, this replication consists of extremely large data sets. In these cases, we recommend using back-end replication links that do not contend with projects' access to data.

As more projects begin to access data within the cluster and their data sets grow, it is necessary to add front-end bandwidth to service data access requests. Adding front-end bandwidth to an Object Storage cluster requires careful planning and design of the Object Storage proxies that projects use to gain access to the data, along with the high availability solutions that enable easy scaling of the proxy layer. We recommend designing a front-end load balancing layer that projects and consumers use to gain access to data stored within the cluster. This load balancing layer may be distributed across zones, regions or even across geographic boundaries, which may also require that the design encompass geo-location solutions.

In some cases, you must add bandwidth and capacity to the network resources servicing requests between proxy servers and storage nodes. For this reason, the network architecture used for access to storage nodes and proxy servers should make use of a design which is scalable.