openstack
/
openstack-manuals
Code Issues Proposed changes

Remove arch-design docs 

The arch-design docs have not been maintained and the Ops Docs SIG plans
to take ownership and maintain it out of its own repo.

To avoid jobs overwriting the published content, this removes the docs
from openstack-manuals.

Depends-on: https://review.openstack.org/621012

Change-Id: I58acb6a5d25d8e0b02e5f3b068aebb4ec144bf1a
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
This commit is contained in:
Sean McGinnis 2018-11-29 14:39:26 -06:00
parent da06d71409
commit aee2cd3da8
66 changed files with 2 additions and 12706 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

27
doc/arch-design/setup.cfg
View File

@ -1,27 +0,0 @@
[metadata]
name = architecturedesignguide
summary = OpenStack Architecture Design Guide
author = OpenStack
author-email = openstack-dev@lists.openstack.org
home-page = https://docs.openstack.org/
classifier =
Environment :: OpenStack
Intended Audience :: Information Technology
Intended Audience :: Cloud Architects
License :: OSI Approved :: Apache Software License
Operating System :: POSIX :: Linux
Topic :: Documentation
[global]
setup-hooks =
pbr.hooks.setup_hook
[files]
[build_sphinx]
warning-is-error = 1
build-dir = build
source-dir = source
[wheel]
universal = 1

30
doc/arch-design/setup.py
View File

@ -1,30 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2013 Hewlett-Packard Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# THIS FILE IS MANAGED BY THE GLOBAL REQUIREMENTS REPO - DO NOT EDIT
import setuptools
# In python < 2.7.4, a lazy loading of package `pbr` will break
# setuptools if some other modules registered functions in `atexit`.
# solution from: http://bugs.python.org/issue15881#msg170215
try:
import multiprocessing # noqa
except ImportError:
pass
setuptools.setup(
setup_requires=['pbr'],
pbr=True)

13
doc/arch-design/source/arch-requirements.rst
View File

@ -1,13 +0,0 @@
=========================
Architecture requirements
=========================
This chapter describes the enterprise and operational factors that impacts the
design of an OpenStack cloud.
.. toctree::
:maxdepth: 2
arch-requirements/arch-requirements-enterprise
arch-requirements/arch-requirements-operations
arch-requirements/arch-requirements-ha

433
doc/arch-design/source/arch-requirements/arch-requirements-enterprise.rst
View File

@ -1,433 +0,0 @@
=======================
Enterprise requirements
=======================
The following sections describe business, usage, and performance
considerations for customers which will impact cloud architecture design.
Cost
~~~~
Financial factors are a primary concern for any organization. Cost
considerations may influence the type of cloud that you build.
For example, a general purpose cloud is unlikely to be the most
cost-effective environment for specialized applications.
Unless business needs dictate that cost is a critical factor,
cost should not be the sole consideration when choosing or designing a cloud.
As a general guideline, increasing the complexity of a cloud architecture
increases the cost of building and maintaining it. For example, a hybrid or
multi-site cloud architecture involving multiple vendors and technical
architectures may require higher setup and operational costs because of the
need for more sophisticated orchestration and brokerage tools than in other
architectures. However, overall operational costs might be lower by virtue of
using a cloud brokerage tool to deploy the workloads to the most cost effective
platform.
.. TODO Replace examples with the proposed example use cases in this guide.
Consider the following costs categories when designing a cloud:
* Compute resources
* Networking resources
* Replication
* Storage
* Management
* Operational costs
It is also important to consider how costs will increase as your cloud scales.
Choices that have a negligible impact in small systems may considerably
increase costs in large systems. In these cases, it is important to minimize
capital expenditure (CapEx) at all layers of the stack. Operators of massively
scalable OpenStack clouds require the use of dependable commodity hardware and
freely available open source software components to reduce deployment costs and
operational expenses. Initiatives like Open Compute (more information available
in the `Open Compute Project <http://www.opencompute.org>`_) provide additional
information.
Time-to-market
~~~~~~~~~~~~~~
The ability to deliver services or products within a flexible time
frame is a common business factor when building a cloud. Allowing users to
self-provision and gain access to compute, network, and
storage resources on-demand may decrease time-to-market for new products
and applications.
You must balance the time required to build a new cloud platform against the
time saved by migrating users away from legacy platforms. In some cases,
existing infrastructure may influence your architecture choices. For example,
using multiple cloud platforms may be a good option when there is an existing
investment in several applications, as it could be faster to tie the
investments together rather than migrating the components and refactoring them
to a single platform.
Revenue opportunity
~~~~~~~~~~~~~~~~~~~
Revenue opportunities vary based on the intent and use case of the cloud.
The requirements of a commercial, customer-facing product are often very
different from an internal, private cloud. You must consider what features
make your design most attractive to your users.
Capacity planning and scalability
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Capacity and the placement of workloads are key design considerations
for clouds. A long-term capacity plan for these designs must
incorporate growth over time to prevent permanent consumption of more
expensive external clouds. To avoid this scenario, account for future
applications' capacity requirements and plan growth appropriately.
It is difficult to predict the amount of load a particular
application might incur if the number of users fluctuates, or the
application experiences an unexpected increase in use.
It is possible to define application requirements in terms of
vCPU, RAM, bandwidth, or other resources and plan appropriately.
However, other clouds might not use the same meter or even the same
oversubscription rates.
Oversubscription is a method to emulate more capacity than
may physically be present. For example, a physical hypervisor node with 32 GB
RAM may host 24 instances, each provisioned with 2 GB RAM.
As long as all 24 instances do not concurrently use 2 full
gigabytes, this arrangement works well.
However, some hosts take oversubscription to extremes and,
as a result, performance can be inconsistent.
If at all possible, determine what the oversubscription rates
of each host are and plan capacity accordingly.
.. TODO Considerations when building your cloud, racks, CPUs, compute node
density. For ongoing capacity planning refer to the Ops Guide.
Performance
~~~~~~~~~~~
Performance is a critical consideration when designing any cloud, and becomes
increasingly important as size and complexity grow. While single-site, private
clouds can be closely controlled, multi-site and hybrid deployments require
more careful planning to reduce problems such as network latency between sites.
For example, you should consider the time required to
run a workload in different clouds and methods for reducing this time.
This may require moving data closer to applications or applications
closer to the data they process, and grouping functionality so that
connections that require low latency take place over a single cloud
rather than spanning clouds.
This may also require a CMP that can determine which cloud can most
efficiently run which types of workloads.
Using native OpenStack tools can help improve performance.
For example, you can use Telemetry to measure performance and the
Orchestration service (heat) to react to changes in demand.
.. note::
Orchestration requires special client configurations to integrate
with Amazon Web Services. For other types of clouds, use CMP features.
Cloud resource deployment
The cloud user expects repeatable, dependable, and deterministic processes
for launching and deploying cloud resources. You could deliver this through
a web-based interface or publicly available API endpoints. All appropriate
options for requesting cloud resources must be available through some type
of user interface, a command-line interface (CLI), or API endpoints.
Consumption model
Cloud users expect a fully self-service and on-demand consumption model.
When an OpenStack cloud reaches the massively scalable size, expect
consumption as a service in each and every way.
* Everything must be capable of automation. For example, everything from
compute hardware, storage hardware, networking hardware, to the installation
and configuration of the supporting software. Manual processes are
impractical in a massively scalable OpenStack design architecture.
* Massively scalable OpenStack clouds require extensive metering and
monitoring functionality to maximize the operational efficiency by keeping
the operator informed about the status and state of the infrastructure. This
includes full scale metering of the hardware and software status. A
corresponding framework of logging and alerting is also required to store
and enable operations to act on the meters provided by the metering and
monitoring solutions. The cloud operator also needs a solution that uses the
data provided by the metering and monitoring solution to provide capacity
planning and capacity trending analysis.
Location
For many use cases the proximity of the user to their workloads has a
direct influence on the performance of the application and therefore
should be taken into consideration in the design. Certain applications
require zero to minimal latency that can only be achieved by deploying
the cloud in multiple locations. These locations could be in different
data centers, cities, countries or geographical regions, depending on
the user requirement and location of the users.
Input-Output requirements
Input-Output performance requirements require researching and
modeling before deciding on a final storage framework. Running
benchmarks for Input-Output performance provides a baseline for
expected performance levels. If these tests include details, then
the resulting data can help model behavior and results during
different workloads. Running scripted smaller benchmarks during the
lifecycle of the architecture helps record the system health at
different points in time. The data from these scripted benchmarks
assist in future scoping and gaining a deeper understanding of an
organization's needs.
Scale
Scaling storage solutions in a storage-focused OpenStack
architecture design is driven by initial requirements, including
:term:`IOPS <Input/output Operations Per Second (IOPS)>`, capacity,
bandwidth, and future needs. Planning capacity based on projected needs
over the course of a budget cycle is important for a design. The
architecture should balance cost and capacity, while also allowing
flexibility to implement new technologies and methods as they become
available.
Network
~~~~~~~
It is important to consider the functionality, security, scalability,
availability, and testability of the network when choosing a CMP and cloud
provider.
* Decide on a network framework and design minimum functionality tests.
This ensures testing and functionality persists during and after
upgrades.
* Scalability across multiple cloud providers may dictate which underlying
network framework you choose in different cloud providers.
It is important to present the network API functions and to verify
that functionality persists across all cloud endpoints chosen.
* High availability implementations vary in functionality and design.
Examples of some common methods are active-hot-standby, active-passive,
and active-active.
Development of high availability and test frameworks is necessary to
insure understanding of functionality and limitations.
* Consider the security of data between the client and the endpoint,
and of traffic that traverses the multiple clouds.
For example, degraded video streams and low quality VoIP sessions negatively
impact user experience and may lead to productivity and economic loss.
Network misconfigurations
Configuring incorrect IP addresses, VLANs, and routers can cause
outages to areas of the network or, in the worst-case scenario, the
entire cloud infrastructure. Automate network configurations to
minimize the opportunity for operator error as it can cause
disruptive problems.
Capacity planning
Cloud networks require management for capacity and growth over time.
Capacity planning includes the purchase of network circuits and
hardware that can potentially have lead times measured in months or
years.
Network tuning
Configure cloud networks to minimize link loss, packet loss, packet
storms, broadcast storms, and loops.
Single Point Of Failure (SPOF)
Consider high availability at the physical and environmental layers.
If there is a single point of failure due to only one upstream link,
or only one power supply, an outage can become unavoidable.
Complexity
An overly complex network design can be difficult to maintain and
troubleshoot. While device-level configuration can ease maintenance
concerns and automated tools can handle overlay networks, avoid or
document non-traditional interconnects between functions and
specialized hardware to prevent outages.
Non-standard features
There are additional risks that arise from configuring the cloud
network to take advantage of vendor specific features. One example
is multi-link aggregation (MLAG) used to provide redundancy at the
aggregator switch level of the network. MLAG is not a standard and,
as a result, each vendor has their own proprietary implementation of
the feature. MLAG architectures are not interoperable across switch
vendors, which leads to vendor lock-in, and can cause delays or
inability when upgrading components.
Dynamic resource expansion or bursting
An application that requires additional resources may suit a multiple
cloud architecture. For example, a retailer needs additional resources
during the holiday season, but does not want to add private cloud
resources to meet the peak demand.
The user can accommodate the increased load by bursting to
a public cloud for these peak load periods. These bursts could be
for long or short cycles ranging from hourly to yearly.
Compliance and geo-location
~~~~~~~~~~~~~~~~~~~~~~~~~~~
An organization may have certain legal obligations and regulatory
compliance measures which could require certain workloads or data to not
be located in certain regions.
Compliance considerations are particularly important for multi-site clouds.
Considerations include:
- federal legal requirements
- local jurisdictional legal and compliance requirements
- image consistency and availability
- storage replication and availability (both block and file/object storage)
- authentication, authorization, and auditing (AAA)
Geographical considerations may also impact the cost of building or leasing
data centers. Considerations include:
- floor space
- floor weight
- rack height and type
- environmental considerations
- power usage and power usage efficiency (PUE)
- physical security
Auditing
~~~~~~~~
A well-considered auditing plan is essential for quickly finding issues.
Keeping track of changes made to security groups and tenant changes can be
useful in rolling back the changes if they affect production. For example,
if all security group rules for a tenant disappeared, the ability to quickly
track down the issue would be important for operational and legal reasons.
For more details on auditing, see the `Compliance chapter
<https://docs.openstack.org/security-guide/compliance.html>`_ in the OpenStack
Security Guide.
Security
~~~~~~~~
The importance of security varies based on the type of organization using
a cloud. For example, government and financial institutions often have
very high security requirements. Security should be implemented according to
asset, threat, and vulnerability risk assessment matrices.
See `security-requirements`.
Service level agreements
~~~~~~~~~~~~~~~~~~~~~~~~
Service level agreements (SLA) must be developed in conjunction with business,
technical, and legal input. Small, private clouds may operate under an informal
SLA, but hybrid or public clouds generally require more formal agreements with
their users.
For a user of a massively scalable OpenStack public cloud, there are no
expectations for control over security, performance, or availability. Users
expect only SLAs related to uptime of API services, and very basic SLAs for
services offered. It is the user's responsibility to address these issues on
their own. The exception to this expectation is the rare case of a massively
scalable cloud infrastructure built for a private or government organization
that has specific requirements.
High performance systems have SLA requirements for a minimum quality of service
with regard to guaranteed uptime, latency, and bandwidth. The level of the
SLA can have a significant impact on the network architecture and
requirements for redundancy in the systems.
Hybrid cloud designs must accommodate differences in SLAs between providers,
and consider their enforceability.
Application readiness
~~~~~~~~~~~~~~~~~~~~~
Some applications are tolerant of a lack of synchronized object
storage, while others may need those objects to be replicated and
available across regions. Understanding how the cloud implementation
impacts new and existing applications is important for risk mitigation,
and the overall success of a cloud project. Applications may have to be
written or rewritten for an infrastructure with little to no redundancy,
or with the cloud in mind.
Application momentum
Businesses with existing applications may find that it is
more cost effective to integrate applications on multiple
cloud platforms than migrating them to a single platform.
No predefined usage model
The lack of a pre-defined usage model enables the user to run a wide
variety of applications without having to know the application
requirements in advance. This provides a degree of independence and
flexibility that no other cloud scenarios are able to provide.
On-demand and self-service application
By definition, a cloud provides end users with the ability to
self-provision computing power, storage, networks, and software in a
simple and flexible way. The user must be able to scale their
resources up to a substantial level without disrupting the
underlying host operations. One of the benefits of using a general
purpose cloud architecture is the ability to start with limited
resources and increase them over time as the user demand grows.
Authentication
~~~~~~~~~~~~~~
It is recommended to have a single authentication domain rather than a
separate implementation for each and every site. This requires an
authentication mechanism that is highly available and distributed to
ensure continuous operation. Authentication server locality might be
required and should be planned for.
Migration, availability, site loss and recovery
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Outages can cause partial or full loss of site functionality. Strategies
should be implemented to understand and plan for recovery scenarios.
* The deployed applications need to continue to function and, more
importantly, you must consider the impact on the performance and
reliability of the application when a site is unavailable.
* It is important to understand what happens to the replication of
objects and data between the sites when a site goes down. If this
causes queues to start building up, consider how long these queues
can safely exist until an error occurs.
* After an outage, ensure the method for resuming proper operations of
a site is implemented when it comes back online. We recommend you
architect the recovery to avoid race conditions.
Disaster recovery and business continuity
Cheaper storage makes the public cloud suitable for maintaining
backup applications.
Migration scenarios
Hybrid cloud architecture enables the migration of
applications between different clouds.
Provider availability or implementation details
Business changes can affect provider availability.
Likewise, changes in a provider's service can disrupt
a hybrid cloud environment or increase costs.
Provider API changes
Consumers of external clouds rarely have control over provider
changes to APIs, and changes can break compatibility.
Using only the most common and basic APIs can minimize potential conflicts.
Image portability
As of the Kilo release, there is no common image format that is
usable by all clouds. Conversion or recreation of images is necessary
if migrating between clouds. To simplify deployment, use the smallest
and simplest images feasible, install only what is necessary, and
use a deployment manager such as Chef or Puppet. Do not use golden
images to speed up the process unless you repeatedly deploy the same
images on the same cloud.
API differences
Avoid using a hybrid cloud deployment with more than just
OpenStack (or with different versions of OpenStack) as API changes
can cause compatibility issues.
Business or technical diversity
Organizations leveraging cloud-based services can embrace business
diversity and utilize a hybrid cloud design to spread their
workloads across multiple cloud providers. This ensures that
no single cloud provider is the sole host for an application.

182
doc/arch-design/source/arch-requirements/arch-requirements-ha.rst
View File

@ -1,182 +0,0 @@
.. _high-availability:
=================
High availability
=================
Data plane and control plane
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When designing an OpenStack cloud, it is important to consider the needs
dictated by the :term:`Service Level Agreement (SLA)`. This includes the core
services required to maintain availability of running Compute service
instances, networks, storage, and additional services running on top of those
resources. These services are often referred to as the Data Plane services,
and are generally expected to be available all the time.
The remaining services, responsible for create, read, update and delete (CRUD)
operations, metering, monitoring, and so on, are often referred to as the
Control Plane. The SLA is likely to dictate a lower uptime requirement for
these services.
The services comprising an OpenStack cloud have a number of requirements that
you need to understand in order to be able to meet SLA terms. For example, in
order to provide the Compute service a minimum of storage, message queueing and
database services are necessary as well as the networking between
them.
Ongoing maintenance operations are made much simpler if there is logical and
physical separation of Data Plane and Control Plane systems. It then becomes
possible to, for example, reboot a controller without affecting customers.
If one service failure affects the operation of an entire server (``noisy
neighbor``), the separation between Control and Data Planes enables rapid
maintenance with a limited effect on customer operations.
Eliminating single points of failure within each site
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
OpenStack lends itself to deployment in a highly available manner where it is
expected that at least 2 servers be utilized. These can run all the services
involved from the message queuing service, for example ``RabbitMQ`` or
``QPID``, and an appropriately deployed database service such as ``MySQL`` or
``MariaDB``. As services in the cloud are scaled out, back-end services will
need to scale too. Monitoring and reporting on server utilization and response
times, as well as load testing your systems, will help determine scale out
decisions.
The OpenStack services themselves should be deployed across multiple servers
that do not represent a single point of failure. Ensuring availability can
be achieved by placing these services behind highly available load balancers
that have multiple OpenStack servers as members.
There are a small number of OpenStack services which are intended to only run
in one place at a time (for example, the ``ceilometer-agent-central`` service)
. In order to prevent these services from becoming a single point of failure,
they can be controlled by clustering software such as ``Pacemaker``.
In OpenStack, the infrastructure is integral to providing services and should
always be available, especially when operating with SLAs. Ensuring network
availability is accomplished by designing the network architecture so that no
single point of failure exists. A consideration of the number of switches,
routes and redundancies of power should be factored into core infrastructure,
as well as the associated bonding of networks to provide diverse routes to your
highly available switch infrastructure.
Care must be taken when deciding network functionality. Currently, OpenStack
supports both the legacy networking (nova-network) system and the newer,
extensible OpenStack Networking (neutron). OpenStack Networking and legacy
networking both have their advantages and disadvantages. They are both valid
and supported options that fit different network deployment models described in
the `OpenStack Operations Guide
<https://docs.openstack.org/ops-guide/arch_network_design.html#network-topology>`_.
When using the Networking service, the OpenStack controller servers or separate
Networking hosts handle routing unless the dynamic virtual routers pattern for
routing is selected. Running routing directly on the controller servers mixes
the Data and Control Planes and can cause complex issues with performance and
troubleshooting. It is possible to use third party software and external
appliances that help maintain highly available layer three routes. Doing so
allows for common application endpoints to control network hardware, or to
provide complex multi-tier web applications in a secure manner. It is also
possible to completely remove routing from Networking, and instead rely on
hardware routing capabilities. In this case, the switching infrastructure must
support layer three routing.
Application design must also be factored into the capabilities of the
underlying cloud infrastructure. If the compute hosts do not provide a seamless
live migration capability, then it must be expected that if a compute host
fails, that instance and any data local to that instance will be deleted.
However, when providing an expectation to users that instances have a
high-level of uptime guaranteed, the infrastructure must be deployed in a way
that eliminates any single point of failure if a compute host disappears.
This may include utilizing shared file systems on enterprise storage or
OpenStack Block storage to provide a level of guarantee to match service
features.
If using a storage design that includes shared access to centralized storage,
ensure that this is also designed without single points of failure and the SLA
for the solution matches or exceeds the expected SLA for the Data Plane.
Eliminating single points of failure in a multi-region design
-------------------------------------------------------------
Some services are commonly shared between multiple regions, including the
Identity service and the Dashboard. In this case, it is necessary to ensure
that the databases backing the services are replicated, and that access to
multiple workers across each site can be maintained in the event of losing a
single region.
Multiple network links should be deployed between sites to provide redundancy
for all components. This includes storage replication, which should be isolated
to a dedicated network or VLAN with the ability to assign QoS to control the
replication traffic or provide priority for this traffic.
.. note::
If the data store is highly changeable, the network requirements could have
a significant effect on the operational cost of maintaining the sites.
If the design incorporates more than one site, the ability to maintain object
availability in both sites has significant implications on the Object Storage
design and implementation. It also has a significant impact on the WAN network
design between the sites.
If applications running in a cloud are not cloud-aware, there should be clear
measures and expectations to define what the infrastructure can and cannot
support. An example would be shared storage between sites. It is possible,
however such a solution is not native to OpenStack and requires a third-party
hardware vendor to fulfill such a requirement. Another example can be seen in
applications that are able to consume resources in object storage directly.
Connecting more than two sites increases the challenges and adds more
complexity to the design considerations. Multi-site implementations require
planning to address the additional topology used for internal and external
connectivity. Some options include full mesh topology, hub spoke, spine leaf,
and 3D Torus.
For more information on high availability in OpenStack, see the `OpenStack High
Availability Guide <https://docs.openstack.org/ha-guide/>`_.
Site loss and recovery
~~~~~~~~~~~~~~~~~~~~~~
Outages can cause partial or full loss of site functionality. Strategies
should be implemented to understand and plan for recovery scenarios.
* The deployed applications need to continue to function and, more
importantly, you must consider the impact on the performance and
reliability of the application if a site is unavailable.
* It is important to understand what happens to the replication of
objects and data between the sites when a site goes down. If this
causes queues to start building up, consider how long these queues
can safely exist until an error occurs.
* After an outage, ensure that operations of a site are resumed when it
comes back online. We recommend that you architect the recovery to
avoid race conditions.
Replicating inter-site data
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Traditionally, replication has been the best method of protecting object store
implementations. A variety of replication methods exist in storage
architectures, for example synchronous and asynchronous mirroring. Most object
stores and back-end storage systems implement methods for replication at the
storage subsystem layer. Object stores also tailor replication techniques to
fit a cloud's requirements.
Organizations must find the right balance between data integrity and data
availability. Replication strategy may also influence disaster recovery
methods.
Replication across different racks, data centers, and geographical regions
increases focus on determining and ensuring data locality. The ability to
guarantee data is accessed from the nearest or fastest storage can be necessary
for applications to perform well.
.. note::
When running embedded object store methods, ensure that you do not
instigate extra data replication as this may cause performance issues.

259
doc/arch-design/source/arch-requirements/arch-requirements-operations.rst
View File

@ -1,259 +0,0 @@
========================
Operational requirements
========================
This section describes operational factors affecting the design of an
OpenStack cloud.
Network design
~~~~~~~~~~~~~~
The network design for an OpenStack cluster includes decisions regarding
the interconnect needs within the cluster, the need to allow clients to
access their resources, and the access requirements for operators to
administrate the cluster. You should consider the bandwidth, latency,
and reliability of these networks.
Consider additional design decisions about monitoring and alarming.
If you are using an external provider, service level agreements (SLAs)
are typically defined in your contract. Operational considerations such
as bandwidth, latency, and jitter can be part of the SLA.
As demand for network resources increase, make sure your network design
accommodates expansion and upgrades. Operators add additional IP address
blocks and add additional bandwidth capacity. In addition, consider
managing hardware and software lifecycle events, for example upgrades,
decommissioning, and outages, while avoiding service interruptions for
tenants.
Factor maintainability into the overall network design. This includes
the ability to manage and maintain IP addresses as well as the use of
overlay identifiers including VLAN tag IDs, GRE tunnel IDs, and MPLS
tags. As an example, if you may need to change all of the IP addresses
on a network, a process known as renumbering, then the design must
support this function.
Address network-focused applications when considering certain
operational realities. For example, consider the impending exhaustion of
IPv4 addresses, the migration to IPv6, and the use of private networks
to segregate different types of traffic that an application receives or
generates. In the case of IPv4 to IPv6 migrations, applications should
follow best practices for storing IP addresses. We recommend you avoid
relying on IPv4 features that did not carry over to the IPv6 protocol or
have differences in implementation.
To segregate traffic, allow applications to create a private tenant
network for database and storage network traffic. Use a public network
for services that require direct client access from the Internet. Upon
segregating the traffic, consider :term:`quality of service (QoS)` and
security to ensure each network has the required level of service.
Also consider the routing of network traffic. For some applications,
develop a complex policy framework for routing. To create a routing
policy that satisfies business requirements, consider the economic cost
of transmitting traffic over expensive links versus cheaper links, in
addition to bandwidth, latency, and jitter requirements.
Finally, consider how to respond to network events. How load
transfers from one link to another during a failure scenario could be
a factor in the design. If you do not plan network capacity
correctly, failover traffic could overwhelm other ports or network
links and create a cascading failure scenario. In this case,
traffic that fails over to one link overwhelms that link and then
moves to the subsequent links until all network traffic stops.
SLA considerations
~~~~~~~~~~~~~~~~~~
Service-level agreements (SLAs) define the levels of availability that will
impact the design of an OpenStack cloud to provide redundancy and high
availability.
SLA terms that affect the design include:
* API availability guarantees implying multiple infrastructure services
and highly available load balancers.
* Network uptime guarantees affecting switch design, which might
require redundant switching and power.
* Networking security policy requirements.
In any environment larger than just a few hosts, there are two areas
that might be subject to a SLA:
* Data Plane - services that provide virtualization, networking, and
storage. Customers usually require these services to be continuously
available.
* Control Plane - ancillary services such as API endpoints, and services that
control CRUD operations. The services in this category are usually subject to
a different SLA expectation and may be better suited on separate
hardware or containers from the Data Plane services.
To effectively run cloud installations, initial downtime planning includes
creating processes and architectures that support planned maintenance
and unplanned system faults.
It is important to determine as part of the SLA negotiation which party is
responsible for monitoring and starting up the Compute service instances if an
outage occurs.
Upgrading, patching, and changing configuration items may require
downtime for some services. Stopping services that form the Control Plane may
not impact the Data Plane. Live-migration of Compute instances may be required
to perform any actions that require downtime to Data Plane components.
There are many services outside the realms of pure OpenStack
code which affects the ability of a cloud design to meet SLAs, including:
* Database services, such as ``MySQL`` or ``PostgreSQL``.
* Services providing RPC, such as ``RabbitMQ``.
* External network attachments.
* Physical constraints such as power, rack space, network cabling, etc.
* Shared storage including SAN based arrays, storage clusters such as ``Ceph``,
and/or NFS services.
Depending on the design, some network service functions may fall into both the
Control and Data Plane categories. For example, the neutron L3 Agent service
may be considered a Control Plane component, but the routers themselves would
be a Data Plane component.
In a design with multiple regions, the SLA would also need to take into
consideration the use of shared services such as the Identity service
and Dashboard.
Any SLA negotiation must also take into account the reliance on third parties
for critical aspects of the design. For example, if there is an existing SLA
on a component such as a storage system, the SLA must take into account this
limitation. If the required SLA for the cloud exceeds the agreed uptime levels
of the cloud components, additional redundancy would be required. This
consideration is critical in a hybrid cloud design, where there are multiple
third parties involved.
Support and maintenance
~~~~~~~~~~~~~~~~~~~~~~~
An operations staff supports, manages, and maintains an OpenStack environment.
Their skills may be specialized or varied depending on the size and purpose of
the installation.
The maintenance function of an operator should be taken into consideration:
Maintenance tasks
Operating system patching, hardware/firmware upgrades, and datacenter
related changes, as well as minor and release upgrades to OpenStack
components are all ongoing operational tasks. The six monthly release
cycle of the OpenStack projects needs to be considered as part of the
cost of ongoing maintenance. The solution should take into account
storage and network maintenance and the impact on underlying
workloads.
Reliability and availability
Reliability and availability depend on the many supporting components'
availability and on the level of precautions taken by the service provider.
This includes network, storage systems, datacenter, and operating systems.
For more information on
managing and maintaining your OpenStack environment, see the
`OpenStack Operations Guide <https://docs.openstack.org/operations-guide/>`_.
Logging and monitoring
----------------------
OpenStack clouds require appropriate monitoring platforms to identify and
manage errors.
.. note::
We recommend leveraging existing monitoring systems to see if they
are able to effectively monitor an OpenStack environment.
Specific meters that are critically important to capture include:
* Image disk utilization
* Response time to the Compute API
Logging and monitoring does not significantly differ for a multi-site OpenStack
cloud. The tools described in the `Logging and monitoring
<https://docs.openstack.org/operations-guide/ops-logging-monitoring.html>`__ in
the Operations Guide remain applicable. Logging and monitoring can be provided
on a per-site basis, and in a common centralized location.
When attempting to deploy logging and monitoring facilities to a centralized
location, care must be taken with the load placed on the inter-site networking
links
Management software
-------------------
Management software providing clustering, logging, monitoring, and alerting
details for a cloud environment is often used. This impacts and affects the
overall OpenStack cloud design, and must account for the additional resource
consumption such as CPU, RAM, storage, and network
bandwidth.
The inclusion of clustering software, such as Corosync or Pacemaker, is
primarily determined by the availability of the cloud infrastructure and
the complexity of supporting the configuration after it is deployed. The
`OpenStack High Availability Guide <https://docs.openstack.org/ha-guide/>`_
provides more details on the installation and configuration of Corosync
and Pacemaker, should these packages need to be included in the design.
Some other potential design impacts include:
* OS-hypervisor combination
Ensure that the selected logging, monitoring, or alerting tools support
the proposed OS-hypervisor combination.
* Network hardware
The network hardware selection needs to be supported by the logging,
monitoring, and alerting software.
Database software
-----------------
Most OpenStack components require access to back-end database services
to store state and configuration information. Choose an appropriate
back-end database which satisfies the availability and fault tolerance
requirements of the OpenStack services.
MySQL is the default database for OpenStack, but other compatible
databases are available.
.. note::
Telemetry uses MongoDB.
The chosen high availability database solution changes according to the
selected database. MySQL, for example, provides several options. Use a
replication technology such as Galera for active-active clustering. For
active-passive use some form of shared storage. Each of these potential
solutions has an impact on the design:
* Solutions that employ Galera/MariaDB require at least three MySQL
nodes.
* MongoDB has its own design considerations for high availability.
* OpenStack design, generally, does not include shared storage.
However, for some high availability designs, certain components might
require it depending on the specific implementation.
Operator access to systems
~~~~~~~~~~~~~~~~~~~~~~~~~~
There is a trend for cloud operations systems being hosted within the cloud
environment. Operators require access to these systems to resolve a major
incident.
Ensure that the network structure connects all clouds to form an integrated
system. Also consider the state of handoffs which must be reliable and have
minimal latency for optimal performance of the system.
If a significant portion of the cloud is on externally managed systems,
prepare for situations where it may not be possible to make changes.
Additionally, cloud providers may differ on how infrastructure must be managed
and exposed. This can lead to delays in root cause analysis where a provider
insists the blame lies with the other provider.

1
doc/arch-design/source/common
View File

@ -1 +0,0 @@
../../common

307
doc/arch-design/source/conf.py
View File

@ -1,307 +0,0 @@
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# This file is execfile()d with the current directory set to its
# containing dir.
#
# Note that not all possible configuration values are present in this
# autogenerated file.
#
# All configuration values have a default; values that are commented out
# serve to show the default.
import os
# import sys
import openstackdocstheme
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
# sys.path.insert(0, os.path.abspath('.'))
# -- General configuration ------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
# needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = ['openstackdocstheme']
# Add any paths that contain templates here, relative to this directory.
# templates_path = ['_templates']
# The suffix of source filenames.
source_suffix = '.rst'
# The encoding of source files.
# source_encoding = 'utf-8-sig'
# The master toctree document.
master_doc = 'index'
# General information about the project.
repository_name = "openstack/openstack-manuals"
bug_project = 'openstack-manuals'
project = u'Architecture Design Guide'
bug_tag = u'arch-design'
copyright = u'2015-2018, OpenStack contributors'
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
version = ''
# The full version, including alpha/beta/rc tags.
release = ''
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
# language = None
# There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used:
# today = ''
# Else, today_fmt is used as the format for a strftime call.
# today_fmt = '%B %d, %Y'
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
exclude_patterns = ['common/cli*', 'common/nova*', 'common/get-started-*']
# The reST default role (used for this markup: `text`) to use for all
# documents.
# default_role = None
# If true, '()' will be appended to :func: etc. cross-reference text.
# add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
# add_module_names = True
# If true, sectionauthor and moduleauthor directives will be shown in the
# output. They are ignored by default.
# show_authors = False
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# A list of ignored prefixes for module index sorting.
# modindex_common_prefix = []
# If true, keep warnings as "system message" paragraphs in the built documents.
# keep_warnings = False
# -- Options for HTML output ----------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
html_theme = 'openstackdocs'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
html_theme_options = {
'display_badge': False
}
# Add any paths that contain custom themes here, relative to this directory.
# html_theme_path = [openstackdocstheme.get_html_theme_path()]
# The name for this set of Sphinx documents. If None, it defaults to
# "<project> v<release> documentation".
# html_title = None
# A shorter title for the navigation bar. Default is the same as html_title.
# html_short_title = None
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
# html_logo = None
# The name of an image file (within the static path) to use as favicon of the
# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
# html_favicon = None
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
# html_static_path = []
# Add any extra paths that contain custom files (such as robots.txt or
# .htaccess) here, relative to this directory. These files are copied
# directly to the root of the documentation.
# html_extra_path = []
# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.
# So that we can enable "log-a-bug" links from each output HTML page, this
# variable must be set to a format that includes year, month, day, hours and
# minutes.
html_last_updated_fmt = '%Y-%m-%d %H:%M'
# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.
# html_use_smartypants = True
# Custom sidebar templates, maps document names to template names.
# html_sidebars = {}
# Additional templates that should be rendered to pages, maps page names to
# template names.
# html_additional_pages = {}
# If false, no module index is generated.
# html_domain_indices = True
# If false, no index is generated.
html_use_index = False
# If true, the index is split into individual pages for each letter.
# html_split_index = False
# If true, links to the reST sources are added to the pages.
html_show_sourcelink = False
# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
# html_show_sphinx = True
# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
# html_show_copyright = True
# If true, an OpenSearch description file will be output, and all pages will
# contain a <link> tag referring to it. The value of this option must be the
# base URL from which the finished HTML is served.
# html_use_opensearch = ''
# This is the file name suffix for HTML files (e.g. ".xhtml").
# html_file_suffix = None
# Output file base name for HTML help builder.
htmlhelp_basename = 'arch-design'
# If true, publish source files
html_copy_source = False
# -- Options for LaTeX output ---------------------------------------------
pdf_theme_path = openstackdocstheme.get_pdf_theme_path()
openstack_logo = openstackdocstheme.get_openstack_logo_path()
latex_custom_template = r"""
\newcommand{\openstacklogo}{%s}
\usepackage{%s}
""" % (openstack_logo, pdf_theme_path)
latex_engine = 'xelatex'
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
'papersize': 'a4paper',
# The font size ('10pt', '11pt' or '12pt').
'pointsize': '11pt',
#Default figure align
'figure_align': 'H',
# Not to generate blank page after chapter
'classoptions': ',openany',
# Additional stuff for the LaTeX preamble.
'preamble': latex_custom_template,
}
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
('index', 'ArchGuide.tex', u'Architecture Design Guide',
u'OpenStack contributors', 'manual'),
]
# The name of an image file (relative to this directory) to place at the top of
# the title page.
# latex_logo = None
# For "manual" documents, if this is true, then toplevel headings are parts,
# not chapters.
# latex_use_parts = False
# If true, show page references after internal links.
# latex_show_pagerefs = False
# If true, show URL addresses after external links.
# latex_show_urls = False
# Documents to append as an appendix to all manuals.
# latex_appendices = []
# If false, no module index is generated.
# latex_domain_indices = True
# -- Options for manual page output ---------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
('index', 'ArchDesign', u'Architecture Design Guide',
[u'OpenStack contributors'], 1)
]
# If true, show URL addresses after external links.
# man_show_urls = False
# -- Options for Texinfo output -------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
('index', 'ArchDesign', u'Architecture Design Guide',
u'OpenStack contributors', 'ArchDesign',
'To reap the benefits of OpenStack, you should plan, design,'
'and architect your cloud properly, taking user needs into'
'account and understanding the use cases.'
'commands.', 'Miscellaneous'),
]
# Documents to append as an appendix to all manuals.
# texinfo_appendices = []
# If false, no module index is generated.
# texinfo_domain_indices = True
# How to display URL addresses: 'footnote', 'no', or 'inline'.
# texinfo_show_urls = 'footnote'
# If true, do not generate a @detailmenu in the "Top" node's menu.
# texinfo_no_detailmenu = False
# -- Options for Internationalization output ------------------------------
locale_dirs = ['locale/']
# -- Options for PDF output --------------------------------------------------
pdf_documents = [
('index', u'ArchDesignGuide', u'Architecture Design Guide',
u'OpenStack contributors')
]

49
doc/arch-design/source/design-cmp-tools.rst
View File

@ -1,49 +0,0 @@
=============================
Cloud management architecture
=============================
Complex clouds, in particular hybrid clouds, may require tools to
facilitate working across multiple clouds.
Broker between clouds
Brokering software evaluates relative costs between different
cloud platforms. Cloud Management Platforms (CMP)
allow the designer to determine the right location for the
workload based on predetermined criteria.
Facilitate orchestration across the clouds
CMPs simplify the migration of application workloads between
public, private, and hybrid cloud platforms.
We recommend using cloud orchestration tools for managing a diverse
portfolio of systems and applications across multiple cloud platforms.
Technical details
~~~~~~~~~~~~~~~~~
.. TODO
Capacity and scale
~~~~~~~~~~~~~~~~~~
.. TODO
High availability
~~~~~~~~~~~~~~~~~
.. TODO
Operator requirements
~~~~~~~~~~~~~~~~~~~~~
.. TODO
Deployment considerations
~~~~~~~~~~~~~~~~~~~~~~~~~
.. TODO
Maintenance considerations
~~~~~~~~~~~~~~~~~~~~~~~~~~
.. TODO

20
doc/arch-design/source/design-compute.rst
View File

@ -1,20 +0,0 @@
====================
Compute architecture
====================
.. toctree::
:maxdepth: 3
design-compute/design-compute-arch
design-compute/design-compute-cpu
design-compute/design-compute-hypervisor
design-compute/design-compute-hardware
design-compute/design-compute-overcommit
design-compute/design-compute-storage
design-compute/design-compute-networking
design-compute/design-compute-logging
This section describes some of the choices you need to consider
when designing and building your compute nodes. Compute nodes form the
resource core of the OpenStack Compute cloud, providing the processing, memory,
network and storage resources to run instances.

104
doc/arch-design/source/design-compute/design-compute-arch.rst
View File

@ -1,104 +0,0 @@
====================================
Compute server architecture overview
====================================
When designing compute resource pools, consider the number of processors,
amount of memory, network requirements, the quantity of storage required for
each hypervisor, and any requirements for bare metal hosts provisioned
through ironic.
When architecting an OpenStack cloud, as part of the planning process, you
must not only determine what hardware to utilize but whether compute
resources will be provided in a single pool or in multiple pools or
availability zones. You should consider if the cloud will provide distinctly
different profiles for compute.
For example, CPU, memory or local storage based compute nodes. For NFV
or HPC based clouds, there may even be specific network configurations that
should be reserved for those specific workloads on specific compute nodes. This
method of designing specific resources into groups or zones of compute can be
referred to as bin packing.
.. note::
In a bin packing design, each independent resource pool provides service for
specific flavors. Since instances are scheduled onto compute hypervisors,
each independent node's resources will be allocated to efficiently use the
available hardware. While bin packing can separate workload specific
resources onto individual servers, bin packing also requires a common
hardware design, with all hardware nodes within a compute resource pool
sharing a common processor, memory, and storage layout. This makes it easier
to deploy, support, and maintain nodes throughout their lifecycle.
Increasing the size of the supporting compute environment increases the network
traffic and messages, adding load to the controllers and administrative
services used to support the OpenStack cloud or networking nodes. When
considering hardware for controller nodes, whether using the monolithic
controller design, where all of the controller services live on one or more
physical hardware nodes, or in any of the newer shared nothing control plane
models, adequate resources must be allocated and scaled to meet scale
requirements. Effective monitoring of the environment will help with capacity
decisions on scaling. Proper planning will help avoid bottlenecks and network
oversubscription as the cloud scales.
Compute nodes automatically attach to OpenStack clouds, resulting in a
horizontally scaling process when adding extra compute capacity to an
OpenStack cloud. To further group compute nodes and place nodes into
appropriate availability zones and host aggregates, additional work is
required. It is necessary to plan rack capacity and network switches as scaling
out compute hosts directly affects data center infrastructure resources as
would any other infrastructure expansion.
While not as common in large enterprises, compute host components can also be
upgraded to account for increases in
demand, known as vertical scaling. Upgrading CPUs with more
cores, or increasing the overall server memory, can add extra needed
capacity depending on whether the running applications are more CPU
intensive or memory intensive. We recommend a rolling upgrade of compute
nodes for redundancy and availability.
After the upgrade, when compute nodes return to the OpenStack cluster, they
will be re-scanned and the new resources will be discovered adjusted in the
OpenStack database.
When selecting a processor, compare features and performance
characteristics. Some processors include features specific to
virtualized compute hosts, such as hardware-assisted virtualization, and
technology related to memory paging (also known as EPT shadowing). These
types of features can have a significant impact on the performance of
your virtual machine.
The number of processor cores and threads impacts the number of worker
threads which can be run on a resource node. Design decisions must
relate directly to the service being run on it, as well as provide a
balanced infrastructure for all services.
Another option is to assess the average workloads and increase the
number of instances that can run within the compute environment by
adjusting the overcommit ratio. This ratio is configurable for CPU and
memory. The default CPU overcommit ratio is 16:1, and the default memory
overcommit ratio is 1.5:1. Determining the tuning of the overcommit
ratios during the design phase is important as it has a direct impact on
the hardware layout of your compute nodes.
.. note::
Changing the CPU overcommit ratio can have a detrimental effect
and cause a potential increase in a noisy neighbor.
Insufficient disk capacity could also have a negative effect on overall
performance including CPU and memory usage. Depending on the back end
architecture of the OpenStack Block Storage layer, capacity includes
adding disk shelves to enterprise storage systems or installing
additional Block Storage nodes. Upgrading directly attached storage
installed in Compute hosts, and adding capacity to the shared storage
for additional ephemeral storage to instances, may be necessary.
Consider the Compute requirements of non-hypervisor nodes (also referred to as
resource nodes). This includes controller, Object Storage nodes, Block Storage
nodes, and networking services.
The ability to create pools or availability zones for unpredictable workloads
should be considered. In some cases, the demand for certain instance types or
flavors may not justify individual hardware design. Allocate hardware designs
that are capable of servicing the most common instance requests. Adding
hardware to the overall architecture can be done later.

85
doc/arch-design/source/design-compute/design-compute-cpu.rst
View File

@ -1,85 +0,0 @@
.. _choosing-a-cpu:
==============
Choosing a CPU
==============
The type of CPU in your compute node is a very important decision. You must
ensure that the CPU supports virtualization by way of *VT-x* for Intel chips
and *AMD-v* for AMD chips.
.. tip::
Consult the vendor documentation to check for virtualization support. For
Intel CPUs, see
`Does my processor support Intel® Virtualization Technology?
<https://www.intel.com/content/www/us/en/support/processors/000005486.html>`_. For AMD CPUs,
see `AMD Virtualization
<https://www.amd.com/en-us/innovations/software-technologies/server-solution/virtualization>`_.
Your CPU may support virtualization but it may be disabled. Consult your
BIOS documentation for how to enable CPU features.
The number of cores that the CPU has also affects your decision. It is
common for current CPUs to have up to 24 cores. Additionally, if an Intel CPU
supports hyper-threading, those 24 cores are doubled to 48 cores. If you
purchase a server that supports multiple CPUs, the number of cores is further
multiplied.
As of the Kilo release, key enhancements have been added to the
OpenStack code to improve guest performance. These improvements allow the
Compute service to take advantage of greater insight into a compute host's
physical layout and therefore make smarter decisions regarding workload
placement. Administrators can use this functionality to enable smarter planning
choices for use cases like NFV (Network Function Virtualization) and HPC (High
Performance Computing).
Considering non-uniform memory access (NUMA) is important when selecting CPU
sizes and types, as there are use cases that use NUMA pinning to reserve host
cores for operating system processes. These reduce the available CPU for
workloads and protects the operating system.
.. tip::
When CPU pinning is requested for a guest, it is assumed
there is no overcommit (or, an overcommit ratio of 1.0). When dedicated
resourcing is not requested for a workload, the normal overcommit ratios
are applied.
Therefore, we recommend that host aggregates are used to separate not
only bare metal hosts, but hosts that will provide resources for workloads
that require dedicated resources. This said, when workloads are provisioned
to NUMA host aggregates, NUMA nodes are chosen at random and vCPUs can float
across NUMA nodes on a host. If workloads require SR-IOV or DPDK, they should
be assigned to a NUMA node aggregate with hosts that supply the
functionality. More importantly, the workload or vCPUs that are executing
processes for a workload should be on the same NUMA node due to the limited
amount of cross-node memory bandwidth. In all cases, the ``NUMATopologyFilter``
must be enabled for ``nova-scheduler``.
Additionally, CPU selection may not be one-size-fits-all across enterprises,
but more of a list of SKUs that are tuned for the enterprise workloads.
For more information about NUMA, see `CPU topologies
<https://docs.openstack.org/admin-guide/compute-cpu-topologies.html>`_ in
the Administrator Guide.
In order to take advantage of these new enhancements in the Compute service,
compute hosts must be using NUMA capable CPUs.
.. tip::
**Multithread Considerations**
Hyper-Threading is Intel's proprietary simultaneous multithreading
implementation used to improve parallelization on their CPUs. You might
consider enabling Hyper-Threading to improve the performance of
multithreaded applications.
Whether you should enable Hyper-Threading on your CPUs depends upon your use
case. For example, disabling Hyper-Threading can be beneficial in intense
computing environments. We recommend performance testing with your local
workload with both Hyper-Threading on and off to determine what is more
appropriate in your case.
In most cases, hyper-threading CPUs can provide a 1.3x to 2.0x performance
benefit over non-hyper-threaded CPUs depending on types of workload.

165
doc/arch-design/source/design-compute/design-compute-hardware.rst
View File

@ -1,165 +0,0 @@
========================
Choosing server hardware
========================
Consider the following factors when selecting compute server hardware:
* Server density
A measure of how many servers can fit into a given measure of
physical space, such as a rack unit [U].
* Resource capacity
The number of CPU cores, how much RAM, or how much storage a given
server delivers.
* Expandability
The number of additional resources you can add to a server before it
reaches capacity.
* Cost
The relative cost of the hardware weighed against the total amount of
capacity available on the hardware based on predetermined requirements.
Weigh these considerations against each other to determine the best design for
the desired purpose. For example, increasing server density means sacrificing
resource capacity or expandability. It also can decrease availability and
increase the chance of noisy neighbor issues. Increasing resource capacity and
expandability can increase cost but decrease server density. Decreasing cost
often means decreasing supportability, availability, server density, resource
capacity, and expandability.
Determine the requirements for the cloud prior to constructing the cloud,
and plan for hardware lifecycles, and expansion and new features that may
require different hardware.
If the cloud is initially built with near end of life, but cost effective
hardware, then the performance and capacity demand of new workloads will drive
the purchase of more modern hardware. With individual hardware components
changing over time, you may prefer to manage configurations as stock keeping
units (SKU)s. This method provides an enterprise with a standard
configuration unit of compute (server) that can be placed in any IT service
manager or vendor supplied ordering system that can be triggered manually or
through advanced operational automations. This simplifies ordering,
provisioning, and activating additional compute resources. For example, there
are plug-ins for several commercial service management tools that enable
integration with hardware APIs. These configure and activate new compute
resources from standby hardware based on a standard configurations. Using this
methodology, spare hardware can be ordered for a datacenter and provisioned
based on capacity data derived from OpenStack Telemetry.
Compute capacity (CPU cores and RAM capacity) is a secondary consideration for
selecting server hardware. The required server hardware must supply adequate
CPU sockets, additional CPU cores, and adequate RA. For more information, see
:ref:`choosing-a-cpu`.
In compute server architecture design, you must also consider network and
storage requirements. For more information on network considerations, see
:ref:`network-design`.
Considerations when choosing hardware
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Here are some other factors to consider when selecting hardware for your
compute servers.
Instance density
----------------
More hosts are required to support the anticipated scale
if the design architecture uses dual-socket hardware designs.
For a general purpose OpenStack cloud, sizing is an important consideration.
The expected or anticipated number of instances that each hypervisor can
host is a common meter used in sizing the deployment. The selected server
hardware needs to support the expected or anticipated instance density.
Host density
------------
Another option to address the higher host count is to use a
quad-socket platform. Taking this approach decreases host density
which also increases rack count. This configuration affects the
number of power connections and also impacts network and cooling
requirements.
Physical data centers have limited physical space, power, and
cooling. The number of hosts (or hypervisors) that can be fitted
into a given metric (rack, rack unit, or floor tile) is another
important method of sizing. Floor weight is an often overlooked
consideration.
The data center floor must be able to support the weight of the proposed number
of hosts within a rack or set of racks. These factors need to be applied as
part of the host density calculation and server hardware selection.
Power and cooling density
-------------------------
The power and cooling density requirements might be lower than with
blade, sled, or 1U server designs due to lower host density (by
using 2U, 3U or even 4U server designs). For data centers with older
infrastructure, this might be a desirable feature.
Data centers have a specified amount of power fed to a given rack or
set of racks. Older data centers may have power densities as low as 20A per
rack, and current data centers can be designed to support power densities as
high as 120A per rack. The selected server hardware must take power density
into account.
Selecting hardware form factor
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Consider the following in selecting server hardware form factor suited for
your OpenStack design architecture:
* Most blade servers can support dual-socket multi-core CPUs. To avoid
this CPU limit, select ``full width`` or ``full height`` blades. Be
aware, however, that this also decreases server density. For example,
high density blade servers such as HP BladeSystem or Dell PowerEdge
M1000e support up to 16 servers in only ten rack units. Using
half-height blades is twice as dense as using full-height blades,
which results in only eight servers per ten rack units.
* 1U rack-mounted servers have the ability to offer greater server density
than a blade server solution, but are often limited to dual-socket,
multi-core CPU configurations. It is possible to place forty 1U servers
in a rack, providing space for the top of rack (ToR) switches, compared
to 32 full width blade servers.
To obtain greater than dual-socket support in a 1U rack-mount form
factor, customers need to buy their systems from Original Design
Manufacturers (ODMs) or second-tier manufacturers.
.. warning::
This may cause issues for organizations that have preferred
vendor policies or concerns with support and hardware warranties
of non-tier 1 vendors.
* 2U rack-mounted servers provide quad-socket, multi-core CPU support,
but with a corresponding decrease in server density (half the density
that 1U rack-mounted servers offer).
* Larger rack-mounted servers, such as 4U servers, often provide even
greater CPU capacity, commonly supporting four or even eight CPU
sockets. These servers have greater expandability, but such servers
have much lower server density and are often more expensive.
* ``Sled servers`` are rack-mounted servers that support multiple
independent servers in a single 2U or 3U enclosure. These deliver
higher density as compared to typical 1U or 2U rack-mounted servers.
For example, many sled servers offer four independent dual-socket
nodes in 2U for a total of eight CPU sockets in 2U.
Scaling your cloud
~~~~~~~~~~~~~~~~~~
When designing a OpenStack cloud compute server architecture, you must
decide whether you intend to scale up or scale out. Selecting a
smaller number of larger hosts, or a larger number of smaller hosts,
depends on a combination of factors: cost, power, cooling, physical rack
and floor space, support-warranty, and manageability. Typically, the scale out
model has been popular for OpenStack because it reduces the number of possible
failure domains by spreading workloads across more infrastructure.
However, the downside is the cost of additional servers and the datacenter
resources needed to power, network, and cool the servers.

46
doc/arch-design/source/design-compute/design-compute-hypervisor.rst
View File

@ -1,46 +0,0 @@
======================
Choosing a hypervisor
======================
A hypervisor provides software to manage virtual machine access to the
underlying hardware. The hypervisor creates, manages, and monitors
virtual machines. OpenStack Compute (nova) supports many hypervisors to various
degrees, including:
* `Ironic <https://docs.openstack.org/ironic/latest/>`_
* `KVM <https://www.linux-kvm.org/page/Main_Page>`_
* `LXC <https://linuxcontainers.org/>`_
* `QEMU <https://wiki.qemu.org/Main_Page>`_
* `VMware ESX/ESXi <https://www.vmware.com/support/vsphere-hypervisor.html>`_
* `Xen (using libvirt) <https://www.xenproject.org>`_
* `XenServer <https://xenserver.org>`_
* `Hyper-V
<https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/hyper-v-technology-overview>`_
* `PowerVM <https://www.ibm.com/us-en/marketplace/ibm-powervm>`_
* `UML <http://user-mode-linux.sourceforge.net>`_
* `Virtuozzo <https://www.virtuozzo.com/products/vz7.html>`_
* `zVM <https://www.ibm.com/it-infrastructure/z/zvm>`_
An important factor in your choice of hypervisor is your current organization's
hypervisor usage or experience. Also important is the hypervisor's feature
parity, documentation, and the level of community experience.
As per the recent OpenStack user survey, KVM is the most widely adopted
hypervisor in the OpenStack community. Besides KVM, there are many deployments
that run other hypervisors such as LXC, VMware, Xen, and Hyper-V. However,
these hypervisors are either less used, are niche hypervisors, or have limited
functionality compared to more commonly used hypervisors.
.. note::
It is also possible to run multiple hypervisors in a single
deployment using host aggregates or cells. However, an individual
compute node can run only a single hypervisor at a time.
For more information about feature support for
hypervisors as well as ironic and Virtuozzo (formerly Parallels), see
`Hypervisor Support Matrix
<https://docs.openstack.org/nova/latest/user/support-matrix.html>`_
and `Hypervisors
<https://docs.openstack.org/ocata/config-reference/compute/hypervisors.html>`_
in the Configuration Reference.

105
doc/arch-design/source/design-compute/design-compute-logging.rst
View File

@ -1,105 +0,0 @@
======================
Compute server logging
======================
The logs on the compute nodes, or any server running nova-compute (for example
in a hyperconverged architecture), are the primary points for troubleshooting
issues with the hypervisor and compute services. Additionally, operating system
logs can also provide useful information.
As the cloud environment grows, the amount of log data increases exponentially.
Enabling debugging on either the OpenStack services or the operating system
further compounds the data issues.
Logging is described in more detail in the `Logging and Monitoring
<https://docs.openstack.org/operations-guide/ops-logging-monitoring.html>`_.
However, it is an important design consideration to take into account before
commencing operations of your cloud.
OpenStack produces a great deal of useful logging information, but for
the information to be useful for operations purposes, you should consider
having a central logging server to send logs to, and a log parsing/analysis
system such as Elastic Stack [formerly known as ELK].
Elastic Stack consists of mainly three components: Elasticsearch (log search
and analysis), Logstash (log intake, processing and output) and Kibana (log
dashboard service).
.. figure:: ../figures/ELKbasicArch.png
:align: center
:alt: Elastic Search Basic Architecture
Due to the amount of logs being sent from servers in the OpenStack environment,
an optional in-memory data structure store can be used. Common examples are
Redis and Memcached. In newer versions of Elastic Stack, a file buffer called
`Filebeat <https://www.elastic.co/products/beats/filebeat>`_ is used for a
similar purpose but adds a "backpressure-sensitive" protocol when sending data
to Logstash or Elasticsearch.
Log analysis often requires disparate logs of differing formats. Elastic
Stack (namely Logstash) was created to take many different log inputs and
transform them into a consistent format that Elasticsearch can catalog and
analyze. As seen in the image above, the process of ingestion starts on the
servers by Logstash, is forwarded to the Elasticsearch server for storage and
searching, and then displayed through Kibana for visual analysis and
interaction.
For instructions on installing Logstash, Elasticsearch and Kibana, see the
`Elasticsearch reference
<https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html>`_.
There are some specific configuration parameters that are needed to
configure Logstash for OpenStack. For example, in order to get Logstash to
collect, parse, and send the correct portions of log files to the Elasticsearch
server, you need to format the configuration file properly. There
are input, output and filter configurations. Input configurations tell Logstash
where to receive data from (log files/forwarders/filebeats/StdIn/Eventlog),
output configurations specify where to put the data, and filter configurations
define the input contents to forward to the output.
The Logstash filter performs intermediary processing on each event. Conditional
filters are applied based on the characteristics of the input and the event.
Some examples of filtering are:
* grok
* date
* csv
* json
There are also output filters available that send event data to many different
destinations. Some examples are:
* csv
* redis
* elasticsearch
* file
* jira
* nagios
* pagerduty
* stdout
Additionally there are several codecs that can be used to change the data
representation of events such as:
* collectd
* graphite
* json
* plan
* rubydebug
These input, output and filter configurations are typically stored in
:file:`/etc/logstash/conf.d` but may vary by linux distribution. Separate
configuration files should be created for different logging systems such as
syslog, Apache, and OpenStack.
General examples and configuration guides can be found on the Elastic `Logstash
Configuration page
<https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html>`_.
OpenStack input, output and filter examples can be found at
`sorantis/elkstack
<https://github.com/sorantis/elkstack/tree/master/elk/logstash>`_.
Once a configuration is complete, Kibana can be used as a visualization tool
for OpenStack and system logging. This will allow operators to configure custom
dashboards for performance, monitoring and security.

51
doc/arch-design/source/design-compute/design-compute-networking.rst
View File

@ -1,51 +0,0 @@
====================
Network connectivity
====================
The selected server hardware must have the appropriate number of network
connections, as well as the right type of network connections, in order to
support the proposed architecture. Ensure that, at a minimum, there are at
least two diverse network connections coming into each rack.
The selection of form factors or architectures affects the selection of server
hardware. Ensure that the selected server hardware is configured to support
enough storage capacity (or storage expandability) to match the requirements of
selected scale-out storage solution. Similarly, the network architecture
impacts the server hardware selection and vice versa.
While each enterprise install is different, the following networks with their
proposed bandwidth is highly recommended for a basic production OpenStack
install.
**Install or OOB network** - Typically used by most distributions and
provisioning tools as the network for deploying base software to the
OpenStack compute nodes. This network should be connected at a minimum of 1Gb
and no routing is usually needed.
**Internal or Management network** - Used as the internal communication network
between OpenStack compute and control nodes. Can also be used as a network
for iSCSI communication between the compute and iSCSI storage nodes. Again,
this should be a minimum of a 1Gb NIC and should be a non-routed network. This
interface should be redundant for high availability (HA).
**Tenant network** - A private network that enables communication between each
tenant's instances. If using flat networking and provider networks, this
network is optional. This network should also be isolated from all other
networks for security compliance. A 1Gb interface should be sufficient and
redundant for HA.
**Storage network** - A private network which could be connected to the Ceph
frontend or other shared storage. For HA purposes this should be a redundant
configuration with suggested 10Gb NICs. This network isolates the storage for
the instances away from other networks. Under load, this storage traffic
could overwhelm other networks and cause outages on other OpenStack services.
**(Optional) External or Public network** - This network is used to communicate
externally from the VMs to the public network space. These addresses are
typically handled by the neutron agent on the controller nodes and can also
be handled by a SDN other than neutron. However, when using neutron DVR with
OVS, this network must be present on the compute node since north and south
traffic will not be handled by the controller nodes, but by the compute node
itself. For more information on DVR with OVS and compute nodes, see
`Open vSwitch: High availability using DVR
<https://docs.openstack.org/ocata/networking-guide/deploy-ovs-ha-dvr.html>`_

48
doc/arch-design/source/design-compute/design-compute-overcommit.rst
View File

@ -1,48 +0,0 @@
==========================
Overcommitting CPU and RAM
==========================
OpenStack allows you to overcommit CPU and RAM on compute nodes. This
allows you to increase the number of instances running on your cloud at the
cost of reducing the performance of the instances. The Compute service uses the
following ratios by default:
* CPU allocation ratio: 16:1
* RAM allocation ratio: 1.5:1
The default CPU allocation ratio of 16:1 means that the scheduler
allocates up to 16 virtual cores per physical core. For example, if a
physical node has 12 cores, the scheduler sees 192 available virtual
cores. With typical flavor definitions of 4 virtual cores per instance,
this ratio would provide 48 instances on a physical node.
The formula for the number of virtual instances on a compute node is
``(OR*PC)/VC``, where:
OR
CPU overcommit ratio (virtual cores per physical core)
PC
Number of physical cores
VC
Number of virtual cores per instance
Similarly, the default RAM allocation ratio of 1.5:1 means that the
scheduler allocates instances to a physical node as long as the total
amount of RAM associated with the instances is less than 1.5 times the
amount of RAM available on the physical node.
For example, if a physical node has 48 GB of RAM, the scheduler
allocates instances to that node until the sum of the RAM associated
with the instances reaches 72 GB (such as nine instances, in the case
where each instance has 8 GB of RAM).
.. note::
Regardless of the overcommit ratio, an instance can not be placed
on any physical node with fewer raw (pre-overcommit) resources than
the instance flavor requires.
You must select the appropriate CPU and RAM allocation ratio for your
particular use case.

154
doc/arch-design/source/design-compute/design-compute-storage.rst
View File

@ -1,154 +0,0 @@
==========================
Instance storage solutions
==========================
As part of the architecture design for a compute cluster, you must specify
storage for the disk on which the instantiated instance runs. There are three
main approaches to providing temporary storage:
* Off compute node storage—shared file system
* On compute node storage—shared file system
* On compute node storage—nonshared file system
In general, the questions you should ask when selecting storage are as
follows:
* What are my workloads?
* Do my workloads have IOPS requirements?
* Are there read, write, or random access performance requirements?
* What is my forecast for the scaling of storage for compute?
* What storage is my enterprise currently using? Can it be re-purposed?
* How do I manage the storage operationally?
Many operators use separate compute and storage hosts instead of a
hyperconverged solution. Compute services and storage services have different
requirements, and compute hosts typically require more CPU and RAM than storage
hosts. Therefore, for a fixed budget, it makes sense to have different
configurations for your compute nodes and your storage nodes. Compute nodes
will be invested in CPU and RAM, and storage nodes will be invested in block
storage.
However, if you are more restricted in the number of physical hosts you have
available for creating your cloud and you want to be able to dedicate as many
of your hosts as possible to running instances, it makes sense to run compute
and storage on the same machines or use an existing storage array that is
available.
The three main approaches to instance storage are provided in the next
few sections.
Non-compute node based shared file system
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In this option, the disks storing the running instances are hosted in
servers outside of the compute nodes.
If you use separate compute and storage hosts, you can treat your
compute hosts as "stateless". As long as you do not have any instances
currently running on a compute host, you can take it offline or wipe it
completely without having any effect on the rest of your cloud. This
simplifies maintenance for the compute hosts.
There are several advantages to this approach:
* If a compute node fails, instances are usually easily recoverable.
* Running a dedicated storage system can be operationally simpler.
* You can scale to any number of spindles.
* It may be possible to share the external storage for other purposes.
The main disadvantages to this approach are:
* Depending on design, heavy I/O usage from some instances can affect
unrelated instances.
* Use of the network can decrease performance.
* Scalability can be affected by network architecture.
On compute node storage—shared file system
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In this option, each compute node is specified with a significant amount
of disk space, but a distributed file system ties the disks from each
compute node into a single mount.
The main advantage of this option is that it scales to external storage
when you require additional storage.
However, this option has several disadvantages:
* Running a distributed file system can make you lose your data
locality compared with nonshared storage.
* Recovery of instances is complicated by depending on multiple hosts.
* The chassis size of the compute node can limit the number of spindles
able to be used in a compute node.
* Use of the network can decrease performance.
* Loss of compute nodes decreases storage availability for all hosts.
On compute node storage—nonshared file system
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In this option, each compute node is specified with enough disks to store the
instances it hosts.
There are two main advantages:
* Heavy I/O usage on one compute node does not affect instances on other
compute nodes. Direct I/O access can increase performance.
* Each host can have different storage profiles for hosts aggregation and
availability zones.
There are several disadvantages:
* If a compute node fails, the data associated with the instances running on
that node is lost.
* The chassis size of the compute node can limit the number of spindles
able to be used in a compute node.
* Migrations of instances from one node to another are more complicated
and rely on features that may not continue to be developed.
* If additional storage is required, this option does not scale.
Running a shared file system on a storage system apart from the compute nodes
is ideal for clouds where reliability and scalability are the most important
factors. Running a shared file system on the compute nodes themselves may be
best in a scenario where you have to deploy to pre-existing servers for which
you have little to no control over their specifications or have specific
storage performance needs but do not have a need for persistent storage.
Issues with live migration
--------------------------
Live migration is an integral part of the operations of the
cloud. This feature provides the ability to seamlessly move instances
from one physical host to another, a necessity for performing upgrades
that require reboots of the compute hosts, but only works well with
shared storage.
Live migration can also be done with non-shared storage, using a feature
known as *KVM live block migration*. While an earlier implementation of
block-based migration in KVM and QEMU was considered unreliable, there
is a newer, more reliable implementation of block-based live migration
as of the Mitaka release.
Live migration and block migration still have some issues:
* Error reporting has received some attention in Mitaka and Newton but there
are improvements needed.
* Live migration resource tracking issues.
* Live migration of rescued images.
Choice of file system
---------------------
If you want to support shared-storage live migration, you need to
configure a distributed file system.
Possible options include:
* NFS (default for Linux)
* Ceph
* GlusterFS
* MooseFS
* Lustre
We recommend that you choose the option operators are most familiar with.
NFS is the easiest to set up and there is extensive community knowledge
about it.

413
doc/arch-design/source/design-control-plane.rst
View File

@ -1,413 +0,0 @@
==========================
Control plane architecture
==========================
.. From Ops Guide chapter: Designing for Cloud Controllers and Cloud
Management
OpenStack is designed to be massively horizontally scalable, which
allows all services to be distributed widely. However, to simplify this
guide, we have decided to discuss services of a more central nature,
using the concept of a *cloud controller*. A cloud controller is a
conceptual simplification. In the real world, you design an architecture
for your cloud controller that enables high availability so that if any
node fails, another can take over the required tasks. In reality, cloud
controller tasks are spread out across more than a single node.
The cloud controller provides the central management system for
OpenStack deployments. Typically, the cloud controller manages
authentication and sends messaging to all the systems through a message
queue.
For many deployments, the cloud controller is a single node. However, to
have high availability, you have to take a few considerations into
account, which we'll cover in this chapter.
The cloud controller manages the following services for the cloud:
Databases
Tracks current information about users and instances, for example,
in a database, typically one database instance managed per service
Message queue services
All :term:`Advanced Message Queuing Protocol (AMQP)` messages for
services are received and sent according to the queue broker
Conductor services
Proxy requests to a database
Authentication and authorization for identity management
Indicates which users can do what actions on certain cloud
resources; quota management is spread out among services,
howeverauthentication
Image-management services
Stores and serves images with metadata on each, for launching in the
cloud
Scheduling services
Indicates which resources to use first; for example, spreading out
where instances are launched based on an algorithm
User dashboard
Provides a web-based front end for users to consume OpenStack cloud
services
API endpoints
Offers each service's REST API access, where the API endpoint
catalog is managed by the Identity service
For our example, the cloud controller has a collection of ``nova-*``
components that represent the global state of the cloud; talks to
services such as authentication; maintains information about the cloud
in a database; communicates to all compute nodes and storage
:term:`workers <worker>` through a queue; and provides API access.
Each service running on a designated cloud controller may be broken out
into separate nodes for scalability or availability.
As another example, you could use pairs of servers for a collective
cloud controller—one active, one standby—for redundant nodes providing a
given set of related services, such as:
- Front end web for API requests, the scheduler for choosing which
compute node to boot an instance on, Identity services, and the
dashboard
- Database and message queue server (such as MySQL, RabbitMQ)
- Image service for the image management
Now that you see the myriad designs for controlling your cloud, read
more about the further considerations to help with your design
decisions.
Hardware Considerations
~~~~~~~~~~~~~~~~~~~~~~~
A cloud controller's hardware can be the same as a compute node, though
you may want to further specify based on the size and type of cloud that
you run.
It's also possible to use virtual machines for all or some of the
services that the cloud controller manages, such as the message queuing.
In this guide, we assume that all services are running directly on the
cloud controller.
:ref:`table_controller_hardware` contains common considerations to
review when sizing hardware for the cloud controller design.
.. _table_controller_hardware:
.. list-table:: Table. Cloud controller hardware sizing considerations
:widths: 25 75
:header-rows: 1
* - Consideration
- Ramification
* - How many instances will run at once?
- Size your database server accordingly, and scale out beyond one cloud
controller if many instances will report status at the same time and
scheduling where a new instance starts up needs computing power.
* - How many compute nodes will run at once?
- Ensure that your messaging queue handles requests successfully and size
accordingly.
* - How many users will access the API?
- If many users will make multiple requests, make sure that the CPU load
for the cloud controller can handle it.
* - How many users will access the dashboard versus the REST API directly?
- The dashboard makes many requests, even more than the API access, so
add even more CPU if your dashboard is the main interface for your users.
* - How many ``nova-api`` services do you run at once for your cloud?
- You need to size the controller with a core per service.
* - How long does a single instance run?
- Starting instances and deleting instances is demanding on the compute
node but also demanding on the controller node because of all the API
queries and scheduling needs.
* - Does your authentication system also verify externally?
- External systems such as :term:`LDAP <Lightweight Directory Access
Protocol (LDAP)>` or :term:`Active Directory` require network
connectivity between the cloud controller and an external authentication
system. Also ensure that the cloud controller has the CPU power to keep
up with requests.
Separation of Services
~~~~~~~~~~~~~~~~~~~~~~
While our example contains all central services in a single location, it
is possible and indeed often a good idea to separate services onto
different physical servers. :ref:`table_deployment_scenarios` is a list
of deployment scenarios we've seen and their justifications.
.. _table_deployment_scenarios:
.. list-table:: Table. Deployment scenarios
:widths: 25 75
:header-rows: 1
* - Scenario
- Justification
* - Run ``glance-*`` servers on the ``swift-proxy`` server.
- This deployment felt that the spare I/O on the Object Storage proxy
server was sufficient and that the Image Delivery portion of glance
benefited from being on physical hardware and having good connectivity
to the Object Storage back end it was using.
* - Run a central dedicated database server.
- This deployment used a central dedicated server to provide the databases
for all services. This approach simplified operations by isolating
database server updates and allowed for the simple creation of slave
database servers for failover.
* - Run one VM per service.
- This deployment ran central services on a set of servers running KVM.
A dedicated VM was created for each service (``nova-scheduler``,
rabbitmq, database, etc). This assisted the deployment with scaling
because administrators could tune the resources given to each virtual
machine based on the load it received (something that was not well
understood during installation).
* - Use an external load balancer.
- This deployment had an expensive hardware load balancer in its
organization. It ran multiple ``nova-api`` and ``swift-proxy``
servers on different physical servers and used the load balancer
to switch between them.
One choice that always comes up is whether to virtualize. Some services,
such as ``nova-compute``, ``swift-proxy`` and ``swift-object`` servers,
should not be virtualized. However, control servers can often be happily
virtualized—the performance penalty can usually be offset by simply
running more of the service.
Database
~~~~~~~~
OpenStack Compute uses an SQL database to store and retrieve stateful
information. MySQL is the popular database choice in the OpenStack
community.
Loss of the database leads to errors. As a result, we recommend that you
cluster your database to make it failure tolerant. Configuring and
maintaining a database cluster is done outside OpenStack and is
determined by the database software you choose to use in your cloud
environment. MySQL/Galera is a popular option for MySQL-based databases.
Message Queue
~~~~~~~~~~~~~
Most OpenStack services communicate with each other using the *message
queue*. For example, Compute communicates to block storage services and
networking services through the message queue. Also, you can optionally
enable notifications for any service. RabbitMQ, Qpid, and Zeromq are all
popular choices for a message-queue service. In general, if the message
queue fails or becomes inaccessible, the cluster grinds to a halt and
ends up in a read-only state, with information stuck at the point where
the last message was sent. Accordingly, we recommend that you cluster
the message queue. Be aware that clustered message queues can be a pain
point for many OpenStack deployments. While RabbitMQ has native
clustering support, there have been reports of issues when running it at
a large scale. While other queuing solutions are available, such as Zeromq
and Qpid, Zeromq does not offer stateful queues. Qpid is the messaging
system of choice for Red Hat and its derivatives. Qpid does not have
native clustering capabilities and requires a supplemental service, such
as Pacemaker or Corsync. For your message queue, you need to determine
what level of data loss you are comfortable with and whether to use an
OpenStack project's ability to retry multiple MQ hosts in the event of a
failure, such as using Compute's ability to do so.
Conductor Services
~~~~~~~~~~~~~~~~~~
In the previous version of OpenStack, all ``nova-compute`` services
required direct access to the database hosted on the cloud controller.
This was problematic for two reasons: security and performance. With
regard to security, if a compute node is compromised, the attacker
inherently has access to the database. With regard to performance,
``nova-compute`` calls to the database are single-threaded and blocking.
This creates a performance bottleneck because database requests are
fulfilled serially rather than in parallel.
The conductor service resolves both of these issues by acting as a proxy
for the ``nova-compute`` service. Now, instead of ``nova-compute``
directly accessing the database, it contacts the ``nova-conductor``
service, and ``nova-conductor`` accesses the database on
``nova-compute``'s behalf. Since ``nova-compute`` no longer has direct
access to the database, the security issue is resolved. Additionally,
``nova-conductor`` is a nonblocking service, so requests from all
compute nodes are fulfilled in parallel.
.. note::
If you are using ``nova-network`` and multi-host networking in your
cloud environment, ``nova-compute`` still requires direct access to
the database.
The ``nova-conductor`` service is horizontally scalable. To make
``nova-conductor`` highly available and fault tolerant, just launch more
instances of the ``nova-conductor`` process, either on the same server
or across multiple servers.
Application Programming Interface (API)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
All public access, whether direct, through a command-line client, or
through the web-based dashboard, uses the API service. Find the API
reference at `Development resources for OpenStack clouds
<https://developer.openstack.org/>`_.
You must choose whether you want to support the Amazon EC2 compatibility
APIs, or just the OpenStack APIs. One issue you might encounter when
running both APIs is an inconsistent experience when referring to images
and instances.
For example, the EC2 API refers to instances using IDs that contain
hexadecimal, whereas the OpenStack API uses names and digits. Similarly,
the EC2 API tends to rely on DNS aliases for contacting virtual
machines, as opposed to OpenStack, which typically lists IP
addresses.
If OpenStack is not set up in the right way, it is simple to have
scenarios in which users are unable to contact their instances due to
having only an incorrect DNS alias. Despite this, EC2 compatibility can
assist users migrating to your cloud.
As with databases and message queues, having more than one :term:`API server`
is a good thing. Traditional HTTP load-balancing techniques can be used to
achieve a highly available ``nova-api`` service.
Extensions
~~~~~~~~~~
The `API
Specifications <https://developer.openstack.org/api-guide/quick-start/index.html>`_ define
the core actions, capabilities, and mediatypes of the OpenStack API. A
client can always depend on the availability of this core API, and
implementers are always required to support it in its entirety.
Requiring strict adherence to the core API allows clients to rely upon a
minimal level of functionality when interacting with multiple
implementations of the same API.
The OpenStack Compute API is extensible. An extension adds capabilities
to an API beyond those defined in the core. The introduction of new
features, MIME types, actions, states, headers, parameters, and
resources can all be accomplished by means of extensions to the core
API. This allows the introduction of new features in the API without
requiring a version change and allows the introduction of
vendor-specific niche functionality.
Scheduling
~~~~~~~~~~
The scheduling services are responsible for determining the compute or
storage node where a virtual machine or block storage volume should be
created. The scheduling services receive creation requests for these
resources from the message queue and then begin the process of
determining the appropriate node where the resource should reside. This
process is done by applying a series of user-configurable filters
against the available collection of nodes.
There are currently two schedulers: ``nova-scheduler`` for virtual
machines and ``cinder-scheduler`` for block storage volumes. Both
schedulers are able to scale horizontally, so for high-availability
purposes, or for very large or high-schedule-frequency installations,
you should consider running multiple instances of each scheduler. The
schedulers all listen to the shared message queue, so no special load
balancing is required.
Images
~~~~~~
The OpenStack Image service consists of two parts: ``glance-api`` and
``glance-registry``. The former is responsible for the delivery of
images; the compute node uses it to download images from the back end.
The latter maintains the metadata information associated with virtual
machine images and requires a database.
The ``glance-api`` part is an abstraction layer that allows a choice of
back end. Currently, it supports:
OpenStack Object Storage
Allows you to store images as objects.
File system
Uses any traditional file system to store the images as files.
S3
Allows you to fetch images from Amazon S3.
HTTP
Allows you to fetch images from a web server. You cannot write
images by using this mode.
If you have an OpenStack Object Storage service, we recommend using this
as a scalable place to store your images. You can also use a file system
with sufficient performance or Amazon S3—unless you do not need the
ability to upload new images through OpenStack.
Dashboard
~~~~~~~~~
The OpenStack dashboard (horizon) provides a web-based user interface to
the various OpenStack components. The dashboard includes an end-user
area for users to manage their virtual infrastructure and an admin area
for cloud operators to manage the OpenStack environment as a
whole.
The dashboard is implemented as a Python web application that normally
runs in :term:`Apache` ``httpd``. Therefore, you may treat it the same as any
other web application, provided it can reach the API servers (including
their admin endpoints) over the network.
Authentication and Authorization
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The concepts supporting OpenStack's authentication and authorization are
derived from well-understood and widely used systems of a similar
nature. Users have credentials they can use to authenticate, and they
can be a member of one or more groups (known as projects or tenants,
interchangeably).
For example, a cloud administrator might be able to list all instances
in the cloud, whereas a user can see only those in his current group.
Resources quotas, such as the number of cores that can be used, disk
space, and so on, are associated with a project.
OpenStack Identity provides authentication decisions and user attribute
information, which is then used by the other OpenStack services to
perform authorization. The policy is set in the ``policy.json`` file.
For information on how to configure these, see `Managing Projects and Users
<https://docs.openstack.org/operations-guide/ops-projects-users.html>`_ in the
OpenStack Operations Guide.
OpenStack Identity supports different plug-ins for authentication
decisions and identity storage. Examples of these plug-ins include:
- In-memory key-value Store (a simplified internal storage structure)
- SQL database (such as MySQL or PostgreSQL)
- Memcached (a distributed memory object caching system)
- LDAP (such as OpenLDAP or Microsoft's Active Directory)
Many deployments use the SQL database; however, LDAP is also a popular
choice for those with existing authentication infrastructure that needs
to be integrated.
Network Considerations
~~~~~~~~~~~~~~~~~~~~~~
Because the cloud controller handles so many different services, it must
be able to handle the amount of traffic that hits it. For example, if
you choose to host the OpenStack Image service on the cloud controller,
the cloud controller should be able to support the transferring of the
images at an acceptable speed.
As another example, if you choose to use single-host networking where
the cloud controller is the network gateway for all instances, then the
cloud controller must support the total amount of traffic that travels
between your cloud and the public Internet.
We recommend that you use a fast NIC, such as 10 GB. You can also choose
to use two 10 GB NICs and bond them together. While you might not be
able to get a full bonded 20 GB speed, different transmission streams
use different NICs. For example, if the cloud controller transfers two
images, each image uses a different NIC and gets a full 10 GB of
bandwidth.

3
doc/arch-design/source/design-identity.rst
View File

@ -1,3 +0,0 @@
=====================
Identity architecture
=====================

3
doc/arch-design/source/design-images.rst
View File

@ -1,3 +0,0 @@
==========================
Image Service architecture
==========================

31
doc/arch-design/source/design-networking.rst
View File

@ -1,31 +0,0 @@
.. _network-design:
====================
Network architecture
====================
.. toctree::
:maxdepth: 2
design-networking/design-networking-concepts
design-networking/design-networking-design
design-networking/design-networking-services
OpenStack provides a rich networking environment. This chapter
details the requirements and options to consider when designing your
cloud. This includes examples of network implementations to
consider, information about some OpenStack network layouts and networking
services that are essential for stable operation.
.. warning::
If this is the first time you are deploying a cloud infrastructure
in your organization, your first conversations should be with your
networking team. Network usage in a running cloud is vastly different
from traditional network deployments and has the potential to be
disruptive at both a connectivity and a policy level.
For example, you must plan the number of IP addresses that you need for
both your guest instances as well as management infrastructure.
Additionally, you must research and discuss cloud network connectivity
through proxy servers and firewalls.

218
doc/arch-design/source/design-networking/design-networking-concepts.rst
View File

@ -1,218 +0,0 @@
===================
Networking concepts
===================
A cloud environment fundamentally changes the ways that networking is provided
and consumed. Understanding the following concepts and decisions is imperative
when making architectural decisions. For detailed information on networking
concepts, see the `OpenStack Networking Guide
<https://docs.openstack.org/ocata/networking-guide/>`_.
Network zones
~~~~~~~~~~~~~
The cloud networks are divided into a number of logical zones that support the
network traffic flow requirements. We recommend defining at the least four
distinct network zones.
Underlay
--------
The underlay zone is defined as the physical network switching infrastructure
that connects the storage, compute and control platforms. There are a large
number of potential underlay options available.
Overlay
-------
The overlay zone is defined as any L3 connectivity between the cloud components
and could take the form of SDN solutions such as the neutron overlay solution
or 3rd Party SDN solutions.
Edge
----
The edge zone is where network traffic transitions from the cloud overlay or
SDN networks into the traditional network environments.
External
--------
The external network is defined as the configuration and components that are
required to provide access to cloud resources and workloads, the external
network is defined as all the components outside of the cloud edge gateways.
Traffic flow
~~~~~~~~~~~~
There are two primary types of traffic flow within a cloud infrastructure, the
choice of networking technologies is influenced by the expected loads.
East/West - The internal traffic flow between workload within the cloud as well
as the traffic flow between the compute nodes and storage nodes falls into the
East/West category. Generally this is the heaviest traffic flow and due to the
need to cater for storage access needs to cater for a minimum of hops and low
latency.
North/South - The flow of traffic between the workload and all external
networks, including clients and remote services. This traffic flow is highly
dependant on the workload within the cloud and the type of network services
being offered.
Layer networking choices
~~~~~~~~~~~~~~~~~~~~~~~~
There are several factors to take into consideration when deciding on whether
to use Layer 2 networking architecture or a layer 3 networking architecture.
For more information about OpenStack networking concepts, see the
`OpenStack Networking <https://docs.openstack.org/ocata/networking-guide/intro-os-networking.html#>`_
section in the OpenStack Networking Guide.
Benefits using a Layer-2 network
--------------------------------
There are several reasons a network designed on layer-2 protocols is selected
over a network designed on layer-3 protocols. In spite of the difficulties of
using a bridge to perform the network role of a router, many vendors,
customers, and service providers choose to use Ethernet in as many parts of
their networks as possible. The benefits of selecting a layer-2 design are:
* Ethernet frames contain all the essentials for networking. These include, but
are not limited to, globally unique source addresses, globally unique
destination addresses, and error control.
* Ethernet frames can carry any kind of packet. Networking at layer-2 is
independent of the layer-3 protocol.
* Adding more layers to the Ethernet frame only slows the networking process
down. This is known as nodal processing delay.
* You can add adjunct networking features, for example class of service (CoS)
or multicasting, to Ethernet as readily as IP networks.
* VLANs are an easy mechanism for isolating networks.
Most information starts and ends inside Ethernet frames. Today this applies
to data, voice, and video. The concept is that the network will benefit more
from the advantages of Ethernet if the transfer of information from a source
to a destination is in the form of Ethernet frames.
Although it is not a substitute for IP networking, networking at layer-2 can
be a powerful adjunct to IP networking.
Layer-2 Ethernet usage has additional benefits over layer-3 IP network usage:
* Speed
* Reduced overhead of the IP hierarchy.
* No need to keep track of address configuration as systems move around.
Whereas the simplicity of layer-2 protocols might work well in a data center
with hundreds of physical machines, cloud data centers have the additional
burden of needing to keep track of all virtual machine addresses and
networks. In these data centers, it is not uncommon for one physical node
to support 30-40 instances.
.. Important::
Networking at the frame level says nothing about the presence or
absence of IP addresses at the packet level. Almost all ports, links, and
devices on a network of LAN switches still have IP addresses, as do all the
source and destination hosts. There are many reasons for the continued need
for IP addressing. The largest one is the need to manage the network. A
device or link without an IP address is usually invisible to most
management applications. Utilities including remote access for diagnostics,
file transfer of configurations and software, and similar applications
cannot run without IP addresses as well as MAC addresses.
Layer-2 architecture limitations
--------------------------------
Layer-2 network architectures have some limitations that become noticeable when
used outside of traditional data centers.
* Number of VLANs is limited to 4096.
* The number of MACs stored in switch tables is limited.
* You must accommodate the need to maintain a set of layer-4 devices to handle
traffic control.
* MLAG, often used for switch redundancy, is a proprietary solution that does
not scale beyond two devices and forces vendor lock-in.
* It can be difficult to troubleshoot a network without IP addresses and ICMP.
* Configuring ARP can be complicated on a large layer-2 networks.
* All network devices need to be aware of all MACs, even instance MACs, so
there is constant churn in MAC tables and network state changes as instances
start and stop.
* Migrating MACs (instance migration) to different physical locations are a
potential problem if you do not set ARP table timeouts properly.
It is important to know that layer-2 has a very limited set of network
management tools. It is difficult to control traffic as it does not have
mechanisms to manage the network or shape the traffic. Network
troubleshooting is also troublesome, in part because network devices have
no IP addresses. As a result, there is no reasonable way to check network
delay.
In a layer-2 network all devices are aware of all MACs, even those that belong
to instances. The network state information in the backbone changes whenever an
instance starts or stops. Because of this, there is far too much churn in the
MAC tables on the backbone switches.
Furthermore, on large layer-2 networks, configuring ARP learning can be
complicated. The setting for the MAC address timer on switches is critical
and, if set incorrectly, can cause significant performance problems. So when
migrating MACs to different physical locations to support instance migration,
problems may arise. As an example, the Cisco default MAC address timer is
extremely long. As such, the network information maintained in the switches
could be out of sync with the new location of the instance.
Benefits using a Layer-3 network
--------------------------------
In layer-3 networking, routing takes instance MAC and IP addresses out of the
network core, reducing state churn. The only time there would be a routing
state change is in the case of a Top of Rack (ToR) switch failure or a link
failure in the backbone itself. Other advantages of using a layer-3
architecture include:
* Layer-3 networks provide the same level of resiliency and scalability
as the Internet.
* Controlling traffic with routing metrics is straightforward.
* You can configure layer-3 to use Border Gateway Protocol (BGP) confederation
for scalability. This way core routers have state proportional to the number
of racks, not to the number of servers or instances.
* There are a variety of well tested tools, such as Internet Control Message
Protocol (ICMP) to monitor and manage traffic.
* Layer-3 architectures enable the use of :term:`quality of service (QoS)` to
manage network performance.
Layer-3 architecture limitations
--------------------------------
The main limitation of layer-3 networking is that there is no built-in
isolation mechanism comparable to the VLANs in layer-2 networks. Furthermore,
the hierarchical nature of IP addresses means that an instance is on the same
subnet as its physical host, making migration out of the subnet difficult. For
these reasons, network virtualization needs to use IP encapsulation and
software at the end hosts. This is for isolation and the separation of the
addressing in the virtual layer from the addressing in the physical layer.
Other potential disadvantages of layer-3 networking include the need to design
an IP addressing scheme rather than relying on the switches to keep track of
the MAC addresses automatically, and to configure the interior gateway routing
protocol in the switches.
Networking service (neutron)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
OpenStack Networking (neutron) is the component of OpenStack that provides
the Networking service API and a reference architecture that implements a
Software Defined Network (SDN) solution.
The Networking service provides full control over creation of virtual network
resources to tenants. This is often accomplished in the form of tunneling
protocols that establish encapsulated communication paths over existing
network infrastructure in order to segment tenant traffic. This method varies
depending on the specific implementation, but some of the more common methods
include tunneling over GRE, encapsulating with VXLAN, and VLAN tags.

281
doc/arch-design/source/design-networking/design-networking-design.rst
View File

@ -1,281 +0,0 @@
==============================
Designing an OpenStack network
==============================
There are many reasons an OpenStack network has complex requirements. One main
factor is that many components interact at different levels of the system
stack. Data flows are also complex.
Data in an OpenStack cloud moves between instances across the network
(known as east-west traffic), as well as in and out of the system (known
as north-south traffic). Physical server nodes have network requirements that
are independent of instance network requirements and must be isolated to
account for scalability. We recommend separating the networks for security
purposes and tuning performance through traffic shaping.
You must consider a number of important technical and business requirements
when planning and designing an OpenStack network:
* Avoid hardware or software vendor lock-in. The design should not rely on
specific features of a vendor's network router or switch.
* Massively scale the ecosystem to support millions of end users.
* Support an indeterminate variety of platforms and applications.
* Design for cost efficient operations to take advantage of massive scale.
* Ensure that there is no single point of failure in the cloud ecosystem.
* High availability architecture to meet customer SLA requirements.
* Tolerant to rack level failure.
* Maximize flexibility to architect future production environments.
Considering these requirements, we recommend the following:
* Design a Layer-3 network architecture rather than a layer-2 network
architecture.
* Design a dense multi-path network core to support multi-directional
scaling and flexibility.
* Use hierarchical addressing because it is the only viable option to scale
a network ecosystem.
* Use virtual networking to isolate instance service network traffic from the
management and internal network traffic.
* Isolate virtual networks using encapsulation technologies.
* Use traffic shaping for performance tuning.
* Use External Border Gateway Protocol (eBGP) to connect to the Internet
up-link.
* Use Internal Border Gateway Protocol (iBGP) to flatten the internal traffic
on the layer-3 mesh.
* Determine the most effective configuration for block storage network.
Additional network design considerations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
There are several other considerations when designing a network-focused
OpenStack cloud.
Redundant networking
--------------------
You should conduct a high availability risk analysis to determine whether to
use redundant switches such as Top of Rack (ToR) switches. In most cases, it
is much more economical to use single switches with a small pool of spare
switches to replace failed units than it is to outfit an entire data center
with redundant switches. Applications should tolerate rack level outages
without affecting normal operations since network and compute resources are
easily provisioned and plentiful.
Research indicates the mean time between failures (MTBF) on switches is
between 100,000 and 200,000 hours. This number is dependent on the ambient
temperature of the switch in the data center. When properly cooled and
maintained, this translates to between 11 and 22 years before failure. Even
in the worst case of poor ventilation and high ambient temperatures in the data
center, the MTBF is still 2-3 years.
.. Link to research findings?
.. TODO Legacy networking (nova-network)
.. TODO OpenStack Networking
.. TODO Simple, single agent
.. TODO Complex, multiple agents
.. TODO Flat or VLAN
.. TODO Flat, VLAN, Overlays, L2-L3, SDN
.. TODO No plug-in support
.. TODO Plug-in support for 3rd parties
.. TODO No multi-tier topologies
.. TODO Multi-tier topologies
.. What about network security? (DC)
Providing IPv6 support
----------------------
One of the most important networking topics today is the exhaustion of
IPv4 addresses. As of late 2015, ICANN announced that the final
IPv4 address blocks have been fully assigned. Because of this, IPv6
protocol has become the future of network focused applications. IPv6
increases the address space significantly, fixes long standing issues
in the IPv4 protocol, and will become essential for network focused
applications in the future.
OpenStack Networking, when configured for it, supports IPv6. To enable
IPv6, create an IPv6 subnet in Networking and use IPv6 prefixes when
creating security groups.
Supporting asymmetric links
---------------------------
When designing a network architecture, the traffic patterns of an
application heavily influence the allocation of total bandwidth and
the number of links that you use to send and receive traffic. Applications
that provide file storage for customers allocate bandwidth and links to
favor incoming traffic; whereas video streaming applications allocate
bandwidth and links to favor outgoing traffic.
Optimizing network performance
------------------------------
It is important to analyze the applications tolerance for latency and
jitter when designing an environment to support network focused
applications. Certain applications, for example VoIP, are less tolerant
of latency and jitter. When latency and jitter are issues, certain
applications may require tuning of QoS parameters and network device
queues to ensure that they immediately queue for transmitting or guarantee
minimum bandwidth. Since OpenStack currently does not support these functions,
consider carefully your selected network plug-in.
The location of a service may also impact the application or consumer
experience. If an application serves differing content to different users,
it must properly direct connections to those specific locations. Where
appropriate, use a multi-site installation for these situations.
You can implement networking in two separate ways. Legacy networking
(nova-network) provides a flat DHCP network with a single broadcast domain.
This implementation does not support tenant isolation networks or advanced
plug-ins, but it is currently the only way to implement a distributed
layer-3 (L3) agent using the multi-host configuration. The Networking service
(neutron) is the official networking implementation and provides a pluggable
architecture that supports a large variety of network methods. Some of these
include a layer-2 only provider network model, external device plug-ins, or
even OpenFlow controllers.
Networking at large scales becomes a set of boundary questions. The
determination of how large a layer-2 domain must be is based on the
number of nodes within the domain and the amount of broadcast traffic
that passes between instances. Breaking layer-2 boundaries may require
the implementation of overlay networks and tunnels. This decision is a
balancing act between the need for a smaller overhead or a need for a smaller
domain.
When selecting network devices, be aware that making a decision based on the
greatest port density often comes with a drawback. Aggregation switches and
routers have not all kept pace with ToR switches and may induce
bottlenecks on north-south traffic. As a result, it may be possible for
massive amounts of downstream network utilization to impact upstream network
devices, impacting service to the cloud. Since OpenStack does not currently
provide a mechanism for traffic shaping or rate limiting, it is necessary to
implement these features at the network hardware level.
Using tunable networking components
-----------------------------------
Consider configurable networking components related to an OpenStack
architecture design when designing for network intensive workloads
that include MTU and QoS. Some workloads require a larger MTU than normal
due to the transfer of large blocks of data. When providing network
service for applications such as video streaming or storage replication,
we recommend that you configure both OpenStack hardware nodes and the
supporting network equipment for jumbo frames where possible. This
allows for better use of available bandwidth. Configure jumbo frames across the
complete path the packets traverse. If one network component is not capable of
handling jumbo frames then the entire path reverts to the default MTU.
:term:`Quality of Service (QoS)` also has a great impact on network intensive
workloads as it provides instant service to packets which have a higher
priority due to the impact of poor network performance. In applications such as
Voice over IP (VoIP), differentiated services code points are a near
requirement for proper operation. You can also use QoS in the opposite
direction for mixed workloads to prevent low priority but high bandwidth
applications, for example backup services, video conferencing, or file sharing,
from blocking bandwidth that is needed for the proper operation of other
workloads. It is possible to tag file storage traffic as a lower class, such as
best effort or scavenger, to allow the higher priority traffic through. In
cases where regions within a cloud might be geographically distributed it may
also be necessary to plan accordingly to implement WAN optimization to combat
latency or packet loss.
Choosing network hardware
~~~~~~~~~~~~~~~~~~~~~~~~~
The network architecture determines which network hardware will be
used. Networking software is determined by the selected networking
hardware.
There are more subtle design impacts that need to be considered. The
selection of certain networking hardware (and the networking software)
affects the management tools that can be used. There are exceptions to
this; the rise of *open* networking software that supports a range of
networking hardware means there are instances where the relationship
between networking hardware and networking software are not as tightly
defined.
Some of the key considerations in the selection of networking hardware
include:
Port count
The design will require networking hardware that has the requisite
port count.
Port density
The network design will be affected by the physical space that is
required to provide the requisite port count. A higher port density
is preferred, as it leaves more rack space for compute or storage
components. This can also lead into considerations about fault domains
and power density. Higher density switches are more expensive, therefore
it is important not to over design the network.
Port speed
The networking hardware must support the proposed network speed, for
example: 1 GbE, 10 GbE, or 40 GbE (or even 100 GbE).
Redundancy
User requirements for high availability and cost considerations
influence the level of network hardware redundancy. Network redundancy
can be achieved by adding redundant power supplies or paired switches.
.. note::
Hardware must support network redundancy.
Power requirements
Ensure that the physical data center provides the necessary power
for the selected network hardware.
.. note::
This is not an issue for top of rack (ToR) switches. This may be an issue
for spine switches in a leaf and spine fabric, or end of row (EoR)
switches.
Protocol support
It is possible to gain more performance out of a single storage
system by using specialized network technologies such as RDMA, SRP,
iSER and SCST. The specifics of using these technologies is beyond
the scope of this book.
There is no single best practice architecture for the networking
hardware supporting an OpenStack cloud. Some of the key factors that will
have a major influence on selection of networking hardware include:
Connectivity
All nodes within an OpenStack cloud require network connectivity. In
some cases, nodes require access to more than one network segment.
The design must encompass sufficient network capacity and bandwidth
to ensure that all communications within the cloud, both north-south
and east-west traffic, have sufficient resources available.
Scalability
The network design should encompass a physical and logical network
design that can be easily expanded upon. Network hardware should
offer the appropriate types of interfaces and speeds that are
required by the hardware nodes.
Availability
To ensure access to nodes within the cloud is not interrupted,
we recommend that the network architecture identifies any single
points of failure and provides some level of redundancy or fault
tolerance. The network infrastructure often involves use of
networking protocols such as LACP, VRRP or others to achieve a highly
available network connection. It is also important to consider the
networking implications on API availability. We recommend a load balancing
solution is designed within the network architecture to ensure that the APIs
and potentially other services in the cloud are highly available.
Choosing networking software
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
OpenStack Networking (neutron) provides a wide variety of networking
services for instances. There are many additional networking software
packages that can be useful when managing OpenStack components. Some
examples include:
- Software to provide load balancing
- Network redundancy protocols
- Routing daemons.
.. TODO Provide software examples

70
doc/arch-design/source/design-networking/design-networking-services.rst
View File

@ -1,70 +0,0 @@
==============================
Additional networking services
==============================
OpenStack, like any network application, has a number of standard
services to consider, such as NTP and DNS.
NTP
~~~
Time synchronization is a critical element to ensure continued operation
of OpenStack components. Ensuring that all components have the correct
time is necessary to avoid errors in instance scheduling, replication of
objects in the object store, and matching log timestamps for debugging.
All servers running OpenStack components should be able to access an
appropriate NTP server. You may decide to set up one locally or use the
public pools available from the `Network Time Protocol
project <http://www.pool.ntp.org/>`_.
DNS
~~~
Designate is a multi-tenant DNSaaS service for OpenStack. It provides a REST
API with integrated keystone authentication. It can be configured to
auto-generate records based on nova and neutron actions. Designate supports a
variety of DNS servers including Bind9 and PowerDNS.
The DNS service provides DNS Zone and RecordSet management for OpenStack
clouds. The DNS Service includes a REST API, a command-line client, and a
horizon Dashboard plugin.
For more information, see the `Designate project <https://www.openstack.org/software/releases/ocata/components/designate>`_
web page.
.. note::
The Designate service does not provide DNS service for the OpenStack
infrastructure upon install. We recommend working with your service
provider when installing OpenStack in order to properly name your
servers and other infrastructure hardware.
DHCP
~~~~
OpenStack neutron deploys various agents when a network is created within
OpenStack. One of these agents is a DHCP agent. This DHCP agent uses the linux
binary, dnsmasq as the delivery agent for DHCP. This agent manages the network
namespaces that are spawned for each project subnet to act as a DHCP server.
The dnsmasq process is capable of allocating IP addresses to all virtual
machines running on a network. When a network is created through OpenStack and
the DHCP agent is enabled for that network, DHCP services are enabled by
default.
LBaaS
~~~~~
OpenStack neutron has the ability to distribute incoming requests between
designated instances. Using neutron networking and OVS, Load
Balancing-as-a-Service (LBaaS) can be created. The load balancing of workloads
is used to distribute incoming application requests evenly between designated
instances. This operation ensures that a workload is shared predictably among
defined instances and allows a more effective use of underlying resources.
OpenStack LBaaS can distribute load in the following methods:
* Round robin - Even rotation between multiple defined instances.
* Source IP - Requests from specific IPs are consistently directed to the same
instance.
* Least connections - Sends requests to the instance with the least number of
active connections.

13
doc/arch-design/source/design-storage.rst
View File

@ -1,13 +0,0 @@
====================
Storage architecture
====================
Storage is found in many parts of the OpenStack cloud environment. This
chapter describes storage type, design considerations and options when
selecting persistent storage options for your cloud environment.
.. toctree::
:maxdepth: 2
design-storage/design-storage-concepts
design-storage/design-storage-arch

546
doc/arch-design/source/design-storage/design-storage-arch.rst
View File

@ -1,546 +0,0 @@
====================
Storage architecture
====================
There are many different storage architectures available when designing an
OpenStack cloud. The convergence of orchestration and automation within the
OpenStack platform enables rapid storage provisioning without the hassle of
the traditional manual processes like volume creation and
attachment.
However, before choosing a storage architecture, a few generic questions should
be answered:
* Will the storage architecture scale linearly as the cloud grows and what are
its limits?
* What is the desired attachment method: NFS, iSCSI, FC, or other?
* Is the storage proven with the OpenStack platform?
* What is the level of support provided by the vendor within the community?
* What OpenStack features and enhancements does the cinder driver enable?
* Does it include tools to help troubleshoot and resolve performance issues?
* Is it interoperable with all of the projects you are planning on using
in your cloud?
Choosing storage back ends
~~~~~~~~~~~~~~~~~~~~~~~~~~
Users will indicate different needs for their cloud architecture. Some may
need fast access to many objects that do not change often, or want to
set a time-to-live (TTL) value on a file. Others may access only storage
that is mounted with the file system itself, but want it to be
replicated instantly when starting a new instance. For other systems,
ephemeral storage is the preferred choice. When you select
:term:`storage back ends <storage back end>`,
consider the following questions from user's perspective:
First and foremost:
* Do I need block storage?
* Do I need object storage?
* Do I need file-based storage?
Next answer the following:
* Do I need to support live migration?
* Should my persistent storage drives be contained in my compute nodes,
or should I use external storage?
* What type of performance do I need in regards to IOPS? Total IOPS and IOPS
per instance? Do I have applications with IOPS SLAs?
* Are my storage needs mostly read, or write, or mixed?
* Which storage choices result in the best cost-performance scenario I am
aiming for?
* How do I manage the storage operationally?
* How redundant and distributed is the storage? What happens if a
storage node fails? To what extent can it mitigate my data-loss disaster
scenarios?
* What is my company currently using and can I use it with OpenStack?
* Do I need more than one storage choice? Do I need tiered performance storage?
While this is not a definitive list of all the questions possible, the list
above will hopefully help narrow the list of possible storage choices down.
A wide variety of use case requirements dictate the nature of the storage
back end. Examples of such requirements are as follows:
* Public, private, or a hybrid cloud (performance profiles, shared storage,
replication options)
* Storage-intensive use cases like HPC and Big Data clouds
* Web-scale or development clouds where storage is typically ephemeral in
nature
Data security recommendations:
* We recommend that data be encrypted both in transit and at-rest.
To this end, carefully select disks, appliances, and software.
Do not assume these features are included with all storage solutions.
* Determine the security policy of your organization and understand
the data sovereignty of your cloud geography and plan accordingly.
If you plan to use live migration, we highly recommend a shared storage
configuration. This allows the operating system and application volumes
for instances to reside outside of the compute nodes and adds significant
performance increases when live migrating.
To deploy your storage by using only commodity hardware, you can use a number
of open-source packages, as described in :ref:`table_persistent_file_storage`.
.. _table_persistent_file_storage:
.. list-table:: Persistent file-based storage support
:widths: 25 25 25 25
:header-rows: 1
* -
- Object
- Block
- File-level
* - Swift
- .. image:: /figures/Check_mark_23x20_02.png
:width: 30%
-
-
* - LVM
-
- .. image:: /figures/Check_mark_23x20_02.png
:width: 30%
-
* - Ceph
- .. image:: /figures/Check_mark_23x20_02.png
:width: 30%
- .. image:: /figures/Check_mark_23x20_02.png
:width: 30%
- Experimental
* - Gluster
- .. image:: /figures/Check_mark_23x20_02.png
:width: 30%
- .. image:: /figures/Check_mark_23x20_02.png
:width: 30%
- .. image:: /figures/Check_mark_23x20_02.png
:width: 30%
* - NFS
-
- .. image:: /figures/Check_mark_23x20_02.png
:width: 30%
- .. image:: /figures/Check_mark_23x20_02.png
:width: 30%
* - ZFS
-
- .. image:: /figures/Check_mark_23x20_02.png
:width: 30%
-
* - Sheepdog
- .. image:: /figures/Check_mark_23x20_02.png
:width: 30%
- .. image:: /figures/Check_mark_23x20_02.png
:width: 30%
-
This list of open source file-level shared storage solutions is not
exhaustive. Your organization may already have deployed a file-level shared
storage solution that you can use.
.. note::
**Storage driver support**
In addition to the open source technologies, there are a number of
proprietary solutions that are officially supported by OpenStack Block
Storage. You can find a matrix of the functionality provided by all of the
supported Block Storage drivers on the `CinderSupportMatrix
wiki <https://wiki.openstack.org/wiki/CinderSupportMatrix>`_.
Also, you need to decide whether you want to support object storage in
your cloud. The two common use cases for providing object storage in a
compute cloud are to provide:
* Users with a persistent storage mechanism for objects like images and video.
* A scalable, reliable data store for OpenStack virtual machine images.
* An API driven S3 compatible object store for application use.
Selecting storage hardware
~~~~~~~~~~~~~~~~~~~~~~~~~~
Storage hardware architecture is determined by selecting specific storage
architecture. Determine the selection of storage architecture by
evaluating possible solutions against the critical factors, the user
requirements, technical considerations, and operational considerations.
Consider the following factors when selecting storage hardware:
Cost
Storage can be a significant portion of the overall system cost. For
an organization that is concerned with vendor support, a commercial
storage solution is advisable, although it comes with a higher price
tag. If initial capital expenditure requires minimization, designing
a system based on commodity hardware would apply. The trade-off is
potentially higher support costs and a greater risk of
incompatibility and interoperability issues.
Performance
Performance of block based storage is typically measured in the maximum read
and write operations to non-contiguous storage locations per second. This
measurement typically applies to SAN, hard drives, and solid state drives.
While IOPS can be broadly measured and is not an official benchmark, many
vectors like to be used by vendors to communicate performance levels. Since
there are no real standards for measuring IOPS, vendor test results may vary,
sometimes wildly. However, along with transfer rate which measures the speed
that data can be transferred to contiguous storage locations, IOPS can be
used in a performance evaluation. Typically, transfer rate is represented by
a bytes per second calculation but IOPS is measured by an integer.
To calculate IOPS for a single drive you could use:
IOPS = 1 / (AverageLatency + AverageSeekTime)
For example:
Average Latency for Single Disk = 2.99ms or .00299 seconds
Average Seek Time for Single Disk = 4.7ms or .0047 seconds
IOPS = 1/(.00299 + .0047)
IOPS = 130
To calculate maximum IOPS for a disk array:
Maximum Read IOPS:
In order to accurately calculate maximum read IOPS for a disk array,
multiply the IOPS for each disk by the maximum read or write IOPS per disk.
maxReadIOPS = nDisks * diskMaxIOPS
For example, 15 10K Spinning Disks would be measured the following way:
maxReadIOPS = 15 * 130 maxReadIOPS = 1950
Maximum write IOPS per array:
Determining the maximum *write* IOPS is a little different because most
administrators configure disk replication using RAID and since the RAID
controller requires IOPS itself, there is a write penalty. The severity of
the write penalty is determined by the type of RAID used.
=========== ==========
Raid Type Penalty
----------- ----------
1 2
5 4
10 2
=========== ==========
.. note::
Raid 5 has the worst penalty (has the most cross disk writes.)
Therefore, when using the above examples, a 15 disk array using RAID 5 is
capable of 1950 read IOPS however, we need to add the penalty when
determining the *write* IOPS:
.. code-block:: none
maxWriteIOPS = 1950 / 4
maxWriteIOPS = 487.5
A RAID 5 array only has 25% of the write IOPS of the read IOPS while a RAID
1 array in this case would produce a maximum of 975 IOPS.
What about SSD? DRAM SSD?
In an HDD, data transfer is sequential. The actual read/write head "seeks" a
point in the hard drive to execute the operation. Seek time is significant.
Transfer rate can also be influenced by file system fragmentation and the
layout. Finally, the mechanical nature of hard disks also has certain
performance limitations.
In an SSD, data transfer is *not* sequential; it is random so it is faster.
There is consistent read performance because the physical location of data is
irrelevant because SSDs have no read/write heads and thus no delays due to
head motion (seeking).
.. note::
Some basic benchmarks for small read/writes:
- **HDDs**: Small reads – 175 IOPs, Small writes – 280 IOPs
- **Flash SSDs**: Small reads – 1075 IOPs (6x), Small writes – 21 IOPs (0.1x)
- **DRAM SSDs**: Small reads – 4091 IOPs (23x), Small writes – 4184 IOPs
(14x)
Scalability
Scalability, along with expandability, is a major consideration in
a general purpose OpenStack cloud. It might be difficult to predict the final
intended size of the implementation as there are no established usage patterns
for a general purpose cloud. It might become necessary to expand the initial
deployment in order to accommodate growth and user demand. Many vendors have
implemented their own solutions to this problem. Some use clustered file
systems that span multiple appliances, while others have similar technologies
to allow block storage to scale past a fixed capacity. Ceph, a distributed
storage solution that offers block storage, was designed to solve this scale
issue and does not have the same limitations on domains, clusters, or scale
issues of other appliance driven models.
Expandability
Expandability is a major architecture factor for storage solutions
with general purpose OpenStack cloud. A storage solution that
expands to 50 PB is considered more expandable than a solution that
only scales to 10 PB. This meter is related to scalability, which is
the measure of a solution's performance as it expands.
Implementing Block Storage
--------------------------
Configure Block Storage resource nodes with advanced RAID controllers
and high-performance disks to provide fault tolerance at the hardware
level.
We recommend deploying high performing storage solutions such as SSD
drives or flash storage systems for applications requiring additional
performance out of Block Storage devices.
In environments that place substantial demands on Block Storage, we
recommend using multiple storage pools. In this case, each pool of
devices should have a similar hardware design and disk configuration
across all hardware nodes in that pool. This allows for a design that
provides applications with access to a wide variety of Block Storage pools,
each with their own redundancy, availability, and performance
characteristics. When deploying multiple pools of storage, it is also
important to consider the impact on the Block Storage scheduler which is
responsible for provisioning storage across resource nodes. Ideally,
ensure that applications can schedule volumes in multiple regions, each with
their own network, power, and cooling infrastructure. This will give tenants
the option of building fault-tolerant applications that are distributed
across multiple availability zones.
In addition to the Block Storage resource nodes, it is important to
design for high availability and redundancy of the APIs, and related
services that are responsible for provisioning and providing access to
storage. We recommend designing a layer of hardware or software load
balancers in order to achieve high availability of the appropriate REST
API services to provide uninterrupted service. In some cases, it may
also be necessary to deploy an additional layer of load balancing to
provide access to back-end database services responsible for servicing
and storing the state of Block Storage volumes. It is imperative that a
highly available database cluster is used to store the Block Storage metadata.
In a cloud with significant demands on Block Storage, the network
architecture should take into account the amount of East-West bandwidth
required for instances to make use of the available storage resources.
The selected network devices should support jumbo frames for
transferring large blocks of data, and utilize a dedicated network for
providing connectivity between instances and Block Storage.
Implementing Object Storage
~~~~~~~~~~~~~~~~~~~~~~~~~~~
While consistency and partition tolerance are both inherent features of
the Object Storage service, it is important to design the overall
storage architecture to ensure that the implemented system meets those goals.
The OpenStack Object Storage service places a specific number of
data replicas as objects on resource nodes. Replicas are distributed
throughout the cluster, based on a consistent hash ring also stored on
each node in the cluster.
When designing your cluster, you must consider durability and
availability which is dependent on the spread and placement of your data,
rather than the reliability of the hardware.
Consider the default value of the number of replicas, which is three. This
means that before an object is marked as having been written, at least two
copies exist in case a single server fails to write, the third copy may or
may not yet exist when the write operation initially returns. Altering this
number increases the robustness of your data, but reduces the amount of
storage you have available. Look at the placement of your servers. Consider
spreading them widely throughout your data center's network and power-failure
zones. Is a zone a rack, a server, or a disk?
Consider these main traffic flows for an Object Storage network:
* Among :term:`object`, :term:`container`, and
:term:`account servers <account server>`
* Between servers and the proxies
* Between the proxies and your users
Object Storage frequent communicates among servers hosting data. Even a small
cluster generates megabytes per second of traffic.
Consider the scenario where an entire server fails and 24 TB of data
needs to be transferred "immediately" to remain at three copies — this can
put significant load on the network.
Another consideration is when a new file is being uploaded, the proxy server
must write out as many streams as there are replicas, multiplying network
traffic. For a three-replica cluster, 10 Gbps in means 30 Gbps out. Combining
this with the previous high bandwidth bandwidth private versus public network
recommendations demands of replication is what results in the recommendation
that your private network be of significantly higher bandwidth than your public
network requires. OpenStack Object Storage communicates internally with
unencrypted, unauthenticated rsync for performance, so the private
network is required.
The remaining point on bandwidth is the public-facing portion. The
``swift-proxy`` service is stateless, which means that you can easily
add more and use HTTP load-balancing methods to share bandwidth and
availability between them. More proxies means more bandwidth.
You should consider designing the Object Storage system with a sufficient
number of zones to provide quorum for the number of replicas defined. For
example, with three replicas configured in the swift cluster, the recommended
number of zones to configure within the Object Storage cluster in order to
achieve quorum is five. While it is possible to deploy a solution with
fewer zones, the implied risk of doing so is that some data may not be
available and API requests to certain objects stored in the cluster
might fail. For this reason, ensure you properly account for the number
of zones in the Object Storage cluster.
Each Object Storage zone should be self-contained within its own
availability zone. Each availability zone should have independent access
to network, power, and cooling infrastructure to ensure uninterrupted
access to data. In addition, a pool of Object Storage proxy servers
providing access to data stored on the object nodes should service each
availability zone. Object proxies in each region should leverage local
read and write affinity so that local storage resources facilitate
access to objects wherever possible. We recommend deploying upstream
load balancing to ensure that proxy services are distributed across the
multiple zones and, in some cases, it may be necessary to make use of
third-party solutions to aid with geographical distribution of services.
A zone within an Object Storage cluster is a logical division. Any of
the following may represent a zone:
* A disk within a single node
* One zone per node
* Zone per collection of nodes
* Multiple racks
* Multiple data centers
Selecting the proper zone design is crucial for allowing the Object
Storage cluster to scale while providing an available and redundant
storage system. It may be necessary to configure storage policies that
have different requirements with regards to replicas, retention, and
other factors that could heavily affect the design of storage in a
specific zone.
Planning and scaling storage capacity
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
An important consideration in running a cloud over time is projecting growth
and utilization trends in order to plan capital expenditures for the short and
long term. Gather utilization meters for compute, network, and storage, along
with historical records of these meters. While securing major anchor tenants
can lead to rapid jumps in the utilization of resources, the average rate of
adoption of cloud services through normal usage also needs to be carefully
monitored.
Scaling Block Storage
---------------------
You can upgrade Block Storage pools to add storage capacity without
interrupting the overall Block Storage service. Add nodes to the pool by
installing and configuring the appropriate hardware and software and
then allowing that node to report in to the proper storage pool through the
message bus. Block Storage nodes generally report into the scheduler
service advertising their availability. As a result, after the node is
online and available, tenants can make use of those storage resources
instantly.
In some cases, the demand on Block Storage may exhaust the available
network bandwidth. As a result, design network infrastructure that
services Block Storage resources in such a way that you can add capacity
and bandwidth easily. This often involves the use of dynamic routing
protocols or advanced networking solutions to add capacity to downstream
devices easily. Both the front-end and back-end storage network designs
should encompass the ability to quickly and easily add capacity and
bandwidth.
.. note::
Sufficient monitoring and data collection should be in-place
from the start, such that timely decisions regarding capacity,
input/output metrics (IOPS) or storage-associated bandwidth can
be made.
Scaling Object Storage
----------------------
Adding back-end storage capacity to an Object Storage cluster requires
careful planning and forethought. In the design phase, it is important
to determine the maximum partition power required by the Object Storage
service, which determines the maximum number of partitions which can
exist. Object Storage distributes data among all available storage, but
a partition cannot span more than one disk, so the maximum number of
partitions can only be as high as the number of disks.
For example, a system that starts with a single disk and a partition
power of 3 can have 8 (2^3) partitions. Adding a second disk means that
each has 4 partitions. The one-disk-per-partition limit means that this
system can never have more than 8 disks, limiting its scalability.
However, a system that starts with a single disk and a partition power
of 10 can have up to 1024 (2^10) disks.
As you add back-end storage capacity to the system, the partition maps
redistribute data amongst the storage nodes. In some cases, this
involves replication of extremely large data sets. In these cases, we
recommend using back-end replication links that do not contend with
tenants' access to data.
As more tenants begin to access data within the cluster and their data
sets grow, it is necessary to add front-end bandwidth to service data
access requests. Adding front-end bandwidth to an Object Storage cluster
requires careful planning and design of the Object Storage proxies that
tenants use to gain access to the data, along with the high availability
solutions that enable easy scaling of the proxy layer. We recommend
designing a front-end load balancing layer that tenants and consumers
use to gain access to data stored within the cluster. This load
balancing layer may be distributed across zones, regions or even across
geographic boundaries, which may also require that the design encompass
geo-location solutions.
In some cases, you must add bandwidth and capacity to the network
resources servicing requests between proxy servers and storage nodes.
For this reason, the network architecture used for access to storage
nodes and proxy servers should make use of a design which is scalable.
Redundancy
----------
When making swift more redundant, one approach is to add additional proxy
servers and load balancing. HAProxy is one method of providing load
balancing and high availability and is often combined with keepalived
or pacemaker to ensure the HAProxy service maintains a stable VIP.
Sample HAProxy configurations can be found in the `OpenStack HA Guide.
<https://docs.openstack.org/ha-guide/>`_.
Replication
-----------
Replicas in Object Storage function independently, and clients only
require a majority of nodes to respond to a request in order for an
operation to be considered successful. Thus, transient failures like
network partitions can quickly cause replicas to diverge.
Fix These differences are eventually reconciled by
asynchronous, peer-to-peer replicator processes. The replicator processes
traverse their local filesystems, concurrently performing operations in a
manner that balances load across physical disks.
Replication uses a push model, with records and files generally only being
copied from local to remote replicas. This is important because data on the
node may not belong there (as in the case of handoffs and ring changes), and a
replicator can not know what data exists elsewhere in the cluster that it
should pull in. It is the duty of any node that contains data to ensure that
data gets to where it belongs. Replica placement is handled by the ring.
Every deleted record or file in the system is marked by a tombstone, so that
deletions can be replicated alongside creations. The replication process cleans
up tombstones after a time period known as the consistency window. The
consistency window encompasses replication duration and the length of time a
transient failure can remove a node from the cluster. Tombstone cleanup must be
tied to replication to reach replica convergence.
If a replicator detects that a remote drive has failed, the replicator uses the
``get_more_nodes`` interface for the ring to choose an alternative node with
which to synchronize. The replicator can maintain desired levels of replication
in the face of disk failures, though some replicas may not be in an immediately
usable location.
.. note::
The replicator does not maintain desired levels of replication when other
failures occur, such as entire node failures, because most failures are
transient.
Replication is an area of active development, andimplementation details
are likely to change over time.
There are two major classes of replicator: the db replicator, which replicates
accounts and containers, and the object replicator, which replicates object
data.
For more information, please see the `Swift replication page <https://docs.openstack.org/swift/latest/overview_replication.html>`_.

329
doc/arch-design/source/design-storage/design-storage-concepts.rst
View File

@ -1,329 +0,0 @@
================
Storage concepts
================
Storage is found in many parts of the OpenStack cloud environment. It is
important to understand the distinction between
:term:`ephemeral <ephemeral volume>` storage and
:term:`persistent <persistent volume>` storage:
- Ephemeral storage - If you only deploy OpenStack
:term:`Compute service (nova)`, by default your users do not have access to
any form of persistent storage. The disks associated with VMs are ephemeral,
meaning that from the user's point of view they disappear when a virtual
machine is terminated.
- Persistent storage - Persistent storage means that the storage resource
outlives any other resource and is always available, regardless of the state
of a running instance.
OpenStack clouds explicitly support three types of persistent
storage: *Object Storage*, *Block Storage*, and *File-based storage*.
Object storage
~~~~~~~~~~~~~~
Object storage is implemented in OpenStack by the
Object Storage service (swift). Users access binary objects through a REST API.
If your intended users need to archive or manage large datasets, you should
provide them with Object Storage service. Additional benefits include:
- OpenStack can store your virtual machine (VM) images inside of an Object
Storage system, as an alternative to storing the images on a file system.
- Integration with OpenStack Identity, and works with the OpenStack Dashboard.
- Better support for distributed deployments across multiple datacenters
through support for asynchronous eventual consistency replication.
You should consider using the OpenStack Object Storage service if you eventually
plan on distributing your storage cluster across multiple data centers, if you
need unified accounts for your users for both compute and object storage, or if
you want to control your object storage with the OpenStack Dashboard. For more
information, see the `Swift project page <https://www.openstack.org/software/releases/ocata/components/swift>`_.
Block storage
~~~~~~~~~~~~~
Block storage is implemented in OpenStack by the
Block Storage service (cinder). Because these volumes are
persistent, they can be detached from one instance and re-attached to another
instance and the data remains intact.
The Block Storage service supports multiple back ends in the form of drivers.
Your choice of a storage back end must be supported by a block storage
driver.
Most block storage drivers allow the instance to have direct access to
the underlying storage hardware's block device. This helps increase the
overall read/write IO. However, support for utilizing files as volumes
is also well established, with full support for NFS, GlusterFS and
others.
These drivers work a little differently than a traditional block
storage driver. On an NFS or GlusterFS file system, a single file is
created and then mapped as a virtual volume into the instance. This
mapping and translation is similar to how OpenStack utilizes QEMU's
file-based virtual machines stored in ``/var/lib/nova/instances``.
File-based storage
~~~~~~~~~~~~~~~~~~
In multi-tenant OpenStack cloud environment, the Shared File Systems service
(manila) provides a set of services for management of shared file systems. The
Shared File Systems service supports multiple back-ends in the form of drivers,
and can be configured to provision shares from one or more back-ends. Share
servers are virtual machines that export file shares using different file
system protocols such as NFS, CIFS, GlusterFS, or HDFS.
The Shared File Systems service is persistent storage and can be mounted to any
number of client machines. It can also be detached from one instance and
attached to another instance without data loss. During this process the data
are safe unless the Shared File Systems service itself is changed or removed.
Users interact with the Shared File Systems service by mounting remote file
systems on their instances with the following usage of those systems for
file storing and exchange. The Shared File Systems service provides shares
which is a remote, mountable file system. You can mount a share and access a
share from several hosts by several users at a time. With shares, you can also:
* Create a share specifying its size, shared file system protocol,
visibility level.
* Create a share on either a share server or standalone, depending on
the selected back-end mode, with or without using a share network.
* Specify access rules and security services for existing shares.
* Combine several shares in groups to keep data consistency inside the
groups for the following safe group operations.
* Create a snapshot of a selected share or a share group for storing
the existing shares consistently or creating new shares from that
snapshot in a consistent way.
* Create a share from a snapshot.
* Set rate limits and quotas for specific shares and snapshots.
* View usage of share resources.
* Remove shares.
Differences between storage types
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
:ref:`table_openstack_storage` explains the differences between Openstack
storage types.
.. _table_openstack_storage:
.. list-table:: Table. OpenStack storage
:widths: 20 20 20 20 20
:header-rows: 1
* -
- Ephemeral storage
- Block storage
- Object storage
- Shared File System storage
* - Application
- Run operating system and scratch space
- Add additional persistent storage to a virtual machine (VM)
- Store data, including VM images
- Add additional persistent storage to a virtual machine
* - Accessed through…
- A file system
- A block device that can be partitioned, formatted, and mounted
(such as, /dev/vdc)
- The REST API
- A Shared File Systems service share (either manila managed or an
external one registered in manila) that can be partitioned, formatted
and mounted (such as /dev/vdc)
* - Accessible from…
- Within a VM
- Within a VM
- Anywhere
- Within a VM
* - Managed by…
- OpenStack Compute (nova)
- OpenStack Block Storage (cinder)
- OpenStack Object Storage (swift)
- OpenStack Shared File System Storage (manila)
* - Persists until…
- VM is terminated
- Deleted by user
- Deleted by user
- Deleted by user
* - Sizing determined by…
- Administrator configuration of size settings, known as *flavors*
- User specification in initial request
- Amount of available physical storage
- * User specification in initial request
* Requests for extension
* Available user-level quotes
* Limitations applied by Administrator
* - Encryption configuration
- Parameter in ``nova.conf``
- Admin establishing `encrypted volume type
<https://docs.openstack.org/admin-guide/dashboard-manage-volumes.html>`_,
then user selecting encrypted volume
- Not yet available
- Shared File Systems service does not apply any additional encryption
above what the share’s back-end storage provides
* - Example of typical usage…
- 10 GB first disk, 30 GB second disk
- 1 TB disk
- 10s of TBs of dataset storage
- Depends completely on the size of back-end storage specified when
a share was being created. In case of thin provisioning it can be
partial space reservation (for more details see
`Capabilities and Extra-Specs
<https://docs.openstack.org/manila/latest/contributor/capabilities_and_extra_specs.html#common-capabilities>`_
specification)
.. note::
**File-level storage for live migration**
With file-level storage, users access stored data using the operating
system's file system interface. Most users who have used a network
storage solution before have encountered this form of networked
storage. The most common file system protocol for Unix is NFS, and for
Windows, CIFS (previously, SMB).
OpenStack clouds do not present file-level storage to end users.
However, it is important to consider file-level storage for storing
instances under ``/var/lib/nova/instances`` when designing your cloud,
since you must have a shared file system if you want to support live
migration.
Commodity storage technologies
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
There are various commodity storage back end technologies available. Depending
on your cloud user's needs, you can implement one or many of these technologies
in different combinations.
Ceph
----
Ceph is a scalable storage solution that replicates data across commodity
storage nodes.
Ceph utilises and object storage mechanism for data storage and exposes
the data via different types of storage interfaces to the end user it
supports interfaces for:
- Object storage
- Block storage
- File-system interfaces
Ceph provides support for the same Object Storage API as swift and can
be used as a back end for the Block Storage service (cinder) as well as
back-end storage for glance images.
Ceph supports thin provisioning implemented using copy-on-write. This can
be useful when booting from volume because a new volume can be provisioned
very quickly. Ceph also supports keystone-based authentication (as of
version 0.56), so it can be a seamless swap in for the default OpenStack
swift implementation.
Ceph's advantages include:
- The administrator has more fine-grained control over data distribution and
replication strategies.
- Consolidation of object storage and block storage.
- Fast provisioning of boot-from-volume instances using thin provisioning.
- Support for the distributed file-system interface
`CephFS <http://ceph.com/docs/master/cephfs/>`_.
You should consider Ceph if you want to manage your object and block storage
within a single system, or if you want to support fast boot-from-volume.
Gluster
-------
A distributed shared file system. As of Gluster version 3.3, you
can use Gluster to consolidate your object storage and file storage
into one unified file and object storage solution, which is called
Gluster For OpenStack (GFO). GFO uses a customized version of swift
that enables Gluster to be used as the back-end storage.
The main reason to use GFO rather than swift is if you also
want to support a distributed file system, either to support shared
storage live migration or to provide it as a separate service to
your end users. If you want to manage your object and file storage
within a single system, you should consider GFO.
LVM
---
The Logical Volume Manager (LVM) is a Linux-based system that provides an
abstraction layer on top of physical disks to expose logical volumes
to the operating system. The LVM back-end implements block storage
as LVM logical partitions.
On each host that will house block storage, an administrator must
initially create a volume group dedicated to Block Storage volumes.
Blocks are created from LVM logical volumes.
.. note::
LVM does *not* provide any replication. Typically,
administrators configure RAID on nodes that use LVM as block
storage to protect against failures of individual hard drives.
However, RAID does not protect against a failure of the entire
host.
iSCSI
-----
Internet Small Computer Systems Interface (iSCSI) is a network protocol that
operates on top of the Transport Control Protocol (TCP) for linking data
storage devices. It transports data between an iSCSI initiator on a server
and iSCSI target on a storage device.
iSCSI is suitable for cloud environments with Block Storage service to support
applications or for file sharing systems. Network connectivity can be
achieved at a lower cost compared to other storage back end technologies since
iSCSI does not require host bus adaptors (HBA) or storage-specific network
devices.
.. Add tips? iSCSI traffic on a separate network or virtual vLAN?
NFS
---
Network File System (NFS) is a file system protocol that allows a user or
administrator to mount a file system on a server. File clients can access
mounted file systems through Remote Procedure Calls (RPC).
The benefits of NFS is low implementation cost due to shared NICs and
traditional network components, and a simpler configuration and setup process.
For more information on configuring Block Storage to use NFS storage, see
`Configure an NFS storage back end
<https://docs.openstack.org/admin-guide/blockstorage-nfs-backend.html>`_ in the
OpenStack Administrator Guide.
Sheepdog
--------
Sheepdog is a userspace distributed storage system. Sheepdog scales
to several hundred nodes, and has powerful virtual disk management
features like snapshot, cloning, rollback and thin provisioning.
It is essentially an object storage system that manages disks and
aggregates the space and performance of disks linearly in hyper
scale on commodity hardware in a smart way. On top of its object store,
Sheepdog provides elastic volume service and http service.
Sheepdog does require a specific kernel version and can work
nicely with xattr-supported file systems.
ZFS
---
The Solaris iSCSI driver for OpenStack Block Storage implements
blocks as ZFS entities. ZFS is a file system that also has the
functionality of a volume manager. This is unlike on a Linux system,
where there is a separation of volume manager (LVM) and file system
(such as, ext3, ext4, xfs, and btrfs). ZFS has a number of
advantages over ext4, including improved data-integrity checking.
The ZFS back end for OpenStack Block Storage supports only
Solaris-based systems, such as Illumos. While there is a Linux port
of ZFS, it is not included in any of the standard Linux
distributions, and it has not been tested with OpenStack Block
Storage. As with LVM, ZFS does not provide replication across hosts
on its own, you need to add a replication solution on top of ZFS if
your cloud needs to be able to handle storage-node failures.

50
doc/arch-design/source/design.rst
View File

@ -1,50 +0,0 @@
.. _design:
======
Design
======
Designing an OpenStack cloud requires a understanding of the cloud user's
requirements and needs to determine the best possible configuration. This
chapter provides guidance on the decisions you need to make during the
design process.
To design, deploy, and configure OpenStack, administrators must
understand the logical architecture. OpenStack modules are one of the
following types:
Daemon
Runs as a background process. On Linux platforms, a daemon is usually
installed as a service.
Script
Installs a virtual environment and runs tests.
Command-line interface (CLI)
Enables users to submit API calls to OpenStack services through commands.
:ref:`logical_architecture` shows one example of the most common
integrated services within OpenStack and how they interact with each
other. End users can interact through the dashboard, CLIs, and APIs.
All services authenticate through a common Identity service, and
individual services interact with each other through public APIs, except
where privileged administrator commands are necessary.
.. _logical_architecture:
.. figure:: common/figures/osog_0001.png
:width: 100%
:alt: OpenStack Logical Architecture
OpenStack Logical Architecture
.. toctree::
:maxdepth: 2
design-compute.rst
design-storage.rst
design-networking.rst
design-identity.rst
design-images.rst
design-control-plane.rst
design-cmp-tools.rst

BIN
doc/arch-design/source/figures/Check_mark_23x20_02.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 3.0 KiB

BIN
doc/arch-design/source/figures/Compute_NSX.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 52 KiB

BIN
doc/arch-design/source/figures/Compute_Tech_Bin_Packing_CPU_optimized1.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 39 KiB

BIN
doc/arch-design/source/figures/Compute_Tech_Bin_Packing_General1.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 35 KiB

BIN
doc/arch-design/source/figures/ELKbasicArch.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 9.1 KiB

1
doc/arch-design/source/figures/ELKbasicArch.svg
View File

File diff suppressed because one or more lines are too long
Side by Side

Before

Width:  |  Height:  |  Size: 16 KiB

BIN
doc/arch-design/source/figures/General_Architecture3.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 79 KiB

BIN
doc/arch-design/source/figures/Generic_CERN_Architecture.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 70 KiB

BIN
doc/arch-design/source/figures/Generic_CERN_Example.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 24 KiB

BIN
doc/arch-design/source/figures/Massively_Scalable_Cells_regions_azs.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 42 KiB

BIN
doc/arch-design/source/figures/Multi-Cloud_Priv-AWS4.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 59 KiB

BIN
doc/arch-design/source/figures/Multi-Cloud_Priv-Pub3.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 54 KiB

BIN
doc/arch-design/source/figures/Multi-Cloud_failover2.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 54 KiB

BIN
doc/arch-design/source/figures/Multi-Site_Customer_Edge.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 68 KiB

BIN
doc/arch-design/source/figures/Multi-Site_shared_keystone1.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 50 KiB

BIN
doc/arch-design/source/figures/Multi-Site_shared_keystone_horizon_swift1.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 52 KiB

BIN
doc/arch-design/source/figures/Multi-site_Geo_Redundant_LB.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 75 KiB

BIN
doc/arch-design/source/figures/Network_Cloud_Storage2.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 37 KiB

BIN
doc/arch-design/source/figures/Network_Web_Services1.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 56 KiB

BIN
doc/arch-design/source/figures/Specialized_Hardware2.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 46 KiB

BIN
doc/arch-design/source/figures/Specialized_OOO.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 56 KiB

BIN
doc/arch-design/source/figures/Specialized_SDN_external.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 30 KiB

BIN
doc/arch-design/source/figures/Specialized_SDN_hosted.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 22 KiB

BIN
doc/arch-design/source/figures/Specialized_VDI1.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 25 KiB

BIN
doc/arch-design/source/figures/Storage_Database_+_Object5.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 50 KiB

BIN
doc/arch-design/source/figures/Storage_Hadoop3.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 50 KiB

BIN
doc/arch-design/source/figures/Storage_Object.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 35 KiB

BIN
doc/arch-design/source/figures/osog_0201.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 42 KiB

52
doc/arch-design/source/index.rst
View File

@ -1,52 +0,0 @@
.. meta::
:description: This guide targets OpenStack Architects
for architectural design
:keywords: Architecture, OpenStack
===================================
OpenStack Architecture Design Guide
===================================
.. important::
**This guide is no longer maintained by the OpenStack documentation
team. If you wish to update this guide, propose a patch at your
own leisure.**
This guide was last updated as of the Pike release, documenting
the OpenStack Ocata, Newton, and Mitaka releases. It may
not apply to EOL releases Kilo and Liberty.
We advise that you read this at your own discretion when planning
on your OpenStack cloud. This guide is intended as advice only.
Abstract
~~~~~~~~
The Architecture Design Guide provides information on planning and designing
an OpenStack cloud. It explains core concepts, cloud architecture design
requirements, and the design criteria of key components and services in an
OpenStack cloud. The guide also describes five common cloud use cases.
Before reading this book, we recommend:
* Prior knowledge of cloud architecture and principles.
* Linux and virtualization experience.
* A basic understanding of networking principles and protocols.
For information about deploying and operating OpenStack, see the
`Installation Guides <https://docs.openstack.org/ocata/install/>`_,
`Deployment Guides <https://docs.openstack.org/ocata/deploy/>`_,
and the `OpenStack Operations Guide <https://docs.openstack.org/operations-guide/>`_.
Contents
~~~~~~~~
.. toctree::
:maxdepth: 2
common/conventions.rst
arch-requirements.rst
design.rst
use-cases.rst
common/appendix.rst

7985
doc/arch-design/source/locale/tr_TR/LC_MESSAGES/arch-design.po
View File

File diff suppressed because it is too large Load Diff

14
doc/arch-design/source/use-cases.rst
View File

@ -1,14 +0,0 @@
.. _use-cases:
=========
Use cases
=========
.. toctree::
:maxdepth: 2
use-cases/use-case-development
use-cases/use-case-general-compute
use-cases/use-case-web-scale
use-cases/use-case-storage
use-cases/use-case-nfv

14
doc/arch-design/source/use-cases/use-case-development.rst
View File

@ -1,14 +0,0 @@
.. _development-cloud:
=================
Development cloud
=================
Design model
~~~~~~~~~~~~
Requirements
~~~~~~~~~~~~
Component block diagram
~~~~~~~~~~~~~~~~~~~~~~~

196
doc/arch-design/source/use-cases/use-case-general-compute.rst
View File

@ -1,196 +0,0 @@
.. _general-compute-cloud:
=====================
General compute cloud
=====================
Design model
~~~~~~~~~~~~
An online classified advertising company wants to run web applications
consisting of Tomcat, Nginx, and MariaDB in a private cloud. To meet the
policy requirements, the cloud infrastructure will run in their
own data center. The company has predictable load requirements but
requires scaling to cope with nightly increases in demand. Their current
environment does not have the flexibility to align with their goal of
running an open source API environment. The current environment consists
of the following:
* Between 120 and 140 installations of Nginx and Tomcat, each with 2
vCPUs and 4 GB of RAM
* A three node MariaDB and Galera cluster, each with 4 vCPUs and 8 GB
of RAM
The company runs hardware load balancers and multiple web applications
serving their websites and orchestrates environments using combinations
of scripts and Puppet. The website generates large amounts of log data
daily that requires archiving.
The solution would consist of the following OpenStack components:
* A firewall, switches and load balancers on the public facing network
connections.
* OpenStack Controller service running Image service, Identity service,
Networking service, combined with support services such as MariaDB and
RabbitMQ, configured for high availability on at least three controller
nodes.
* OpenStack compute nodes running the KVM hypervisor.
* OpenStack Block Storage for use by compute instances, requiring
persistent storage (such as databases for dynamic sites).
* OpenStack Object Storage for serving static objects (such as images).
.. figure:: ../figures/General_Architecture3.png
Running up to 140 web instances and the small number of MariaDB
instances requires 292 vCPUs available, as well as 584 GB of RAM. On a
typical 1U server using dual-socket hex-core Intel CPUs with
Hyperthreading, and assuming 2:1 CPU overcommit ratio, this would
require 8 OpenStack Compute nodes.
The web application instances run from local storage on each of the
OpenStack Compute nodes. The web application instances are stateless,
meaning that any of the instances can fail and the application will
continue to function.
MariaDB server instances store their data on shared enterprise storage,
such as NetApp or Solidfire devices. If a MariaDB instance fails,
storage would be expected to be re-attached to another instance and
rejoined to the Galera cluster.
Logs from the web application servers are shipped to OpenStack Object
Storage for processing and archiving.
Additional capabilities can be realized by moving static web content to
be served from OpenStack Object Storage containers, and backing the
OpenStack Image service with OpenStack Object Storage.
.. note::
Increasing OpenStack Object Storage means network bandwidth needs to
be taken into consideration. Running OpenStack Object Storage with
network connections offering 10 GbE or better connectivity is
advised.
Leveraging Orchestration and Telemetry services is also a potential
issue when providing auto-scaling, orchestrated web application
environments. Defining the web applications in a
:term:`Heat Orchestration Template (HOT)`
negates the reliance on the current scripted Puppet
solution.
OpenStack Networking can be used to control hardware load balancers
through the use of plug-ins and the Networking API. This allows users to
control hardware load balance pools and instances as members in these
pools, but their use in production environments must be carefully
weighed against current stability.
Requirements
~~~~~~~~~~~~
.. temporarily location of storage information until we establish a template
Storage requirements
--------------------
Using a scale-out storage solution with direct-attached storage (DAS) in
the servers is well suited for a general purpose OpenStack cloud. Cloud
services requirements determine your choice of scale-out solution. You
need to determine if a single, highly expandable and highly vertical,
scalable, centralized storage array is suitable for your design. After
determining an approach, select the storage hardware based on this
criteria.
This list expands upon the potential impacts for including a particular
storage architecture (and corresponding storage hardware) into the
design for a general purpose OpenStack cloud:
Connectivity
If storage protocols other than Ethernet are part of the storage solution,
ensure the appropriate hardware has been selected. If a centralized storage
array is selected, ensure that the hypervisor will be able to connect to
that storage array for image storage.
Usage
How the particular storage architecture will be used is critical for
determining the architecture. Some of the configurations that will
influence the architecture include whether it will be used by the
hypervisors for ephemeral instance storage, or if OpenStack Object
Storage will use it for object storage.
Instance and image locations
Where instances and images will be stored will influence the
architecture.
Server hardware
If the solution is a scale-out storage architecture that includes
DAS, it will affect the server hardware selection. This could ripple
into the decisions that affect host density, instance density, power
density, OS-hypervisor, management tools and others.
A general purpose OpenStack cloud has multiple options. The key factors
that will have an influence on selection of storage hardware for a
general purpose OpenStack cloud are as follows:
Capacity
Hardware resources selected for the resource nodes should be capable
of supporting enough storage for the cloud services. Defining the
initial requirements and ensuring the design can support adding
capacity is important. Hardware nodes selected for object storage
should be capable of support a large number of inexpensive disks
with no reliance on RAID controller cards. Hardware nodes selected
for block storage should be capable of supporting high speed storage
solutions and RAID controller cards to provide performance and
redundancy to storage at a hardware level. Selecting hardware RAID
controllers that automatically repair damaged arrays will assist
with the replacement and repair of degraded or deleted storage
devices.
Performance
Disks selected for object storage services do not need to be fast
performing disks. We recommend that object storage nodes take
advantage of the best cost per terabyte available for storage.
Contrastingly, disks chosen for block storage services should take
advantage of performance boosting features that may entail the use
of SSDs or flash storage to provide high performance block storage
pools. Storage performance of ephemeral disks used for instances
should also be taken into consideration.
Fault tolerance
Object storage resource nodes have no requirements for hardware
fault tolerance or RAID controllers. It is not necessary to plan for
fault tolerance within the object storage hardware because the
object storage service provides replication between zones as a
feature of the service. Block storage nodes, compute nodes, and
cloud controllers should all have fault tolerance built in at the
hardware level by making use of hardware RAID controllers and
varying levels of RAID configuration. The level of RAID chosen
should be consistent with the performance and availability
requirements of the cloud.
Network hardware requirements
-----------------------------
For a compute-focus architecture, we recommend designing the network
architecture using a scalable network model that makes it easy to add
capacity and bandwidth. A good example of such a model is the leaf-spine
model. In this type of network design, you can add additional
bandwidth as well as scale out to additional racks of gear. It is important to
select network hardware that supports port count, port speed, and
port density while allowing for future growth as workload demands
increase. In the network architecture, it is also important to evaluate
where to provide redundancy.
Network software requirements
-----------------------------
For a general purpose OpenStack cloud, the OpenStack infrastructure
components need to be highly available. If the design does not include
hardware load balancing, networking software packages like HAProxy will
need to be included.
Component block diagram
~~~~~~~~~~~~~~~~~~~~~~~

181
doc/arch-design/source/use-cases/use-case-nfv.rst
View File

@ -1,181 +0,0 @@
.. _nfv-cloud:
==============================
Network virtual function cloud
==============================
Design model
~~~~~~~~~~~~
Requirements
~~~~~~~~~~~~
Component block diagram
~~~~~~~~~~~~~~~~~~~~~~~
Network-focused cloud examples
------------------------------
An organization designs a large scale cloud-based web application. The
application scales horizontally in a bursting behavior and generates a
high instance count. The application requires an SSL connection to secure
data and must not lose connection state to individual servers.
The figure below depicts an example design for this workload. In this
example, a hardware load balancer provides SSL offload functionality and
connects to tenant networks in order to reduce address consumption. This
load balancer links to the routing architecture as it services the VIP
for the application. The router and load balancer use the GRE tunnel ID
of the application's tenant network and an IP address within the tenant
subnet but outside of the address pool. This is to ensure that the load
balancer can communicate with the application's HTTP servers without
requiring the consumption of a public IP address.
Because sessions persist until closed, the routing and switching
architecture provides high availability. Switches mesh to each
hypervisor and each other, and also provide an MLAG implementation to
ensure that layer-2 connectivity does not fail. Routers use VRRP and
fully mesh with switches to ensure layer-3 connectivity. Since GRE
provides an overlay network, Networking is present and uses the Open
vSwitch agent in GRE tunnel mode. This ensures all devices can reach all
other devices and that you can create tenant networks for private
addressing links to the load balancer.
.. figure:: ../figures/Network_Web_Services1.png
A web service architecture has many options and optional components. Due
to this, it can fit into a large number of other OpenStack designs. A
few key components, however, need to be in place to handle the nature of
most web-scale workloads. You require the following components:
* OpenStack Controller services (Image service, Identity service, Networking
service, and supporting services such as MariaDB and RabbitMQ)
* OpenStack Compute running KVM hypervisor
* OpenStack Object Storage
* Orchestration service
* Telemetry service
Beyond the normal Identity service, Compute service, Image service, and
Object Storage components, we recommend the Orchestration service component
to handle the proper scaling of workloads to adjust to demand. Due to the
requirement for auto-scaling, the design includes the Telemetry service.
Web services tend to be bursty in load, have very defined peak and
valley usage patterns and, as a result, benefit from automatic scaling
of instances based upon traffic. At a network level, a split network
configuration works well with databases residing on private tenant
networks since these do not emit a large quantity of broadcast traffic
and may need to interconnect to some databases for content.
Load balancing
~~~~~~~~~~~~~~
Load balancing spreads requests across multiple instances. This workload
scales well horizontally across large numbers of instances. This enables
instances to run without publicly routed IP addresses and instead to
rely on the load balancer to provide a globally reachable service. Many
of these services do not require direct server return. This aids in
address planning and utilization at scale since only the virtual IP
(VIP) must be public.
Overlay networks
~~~~~~~~~~~~~~~~
The overlay functionality design includes OpenStack Networking in Open
vSwitch GRE tunnel mode. In this case, the layer-3 external routers pair
with VRRP, and switches pair with an implementation of MLAG to ensure
that you do not lose connectivity with the upstream routing
infrastructure.
Performance tuning
~~~~~~~~~~~~~~~~~~
Network level tuning for this workload is minimal. :term:`Quality of
Service (QoS)` applies to these workloads for a middle ground Class
Selector depending on existing policies. It is higher than a best effort
queue but lower than an Expedited Forwarding or Assured Forwarding
queue. Since this type of application generates larger packets with
longer-lived connections, you can optimize bandwidth utilization for
long duration TCP. Normal bandwidth planning applies here with regards
to benchmarking a session's usage multiplied by the expected number of
concurrent sessions with overhead.
Network functions
~~~~~~~~~~~~~~~~~
Network functions is a broad category but encompasses workloads that
support the rest of a system's network. These workloads tend to consist
of large amounts of small packets that are very short lived, such as DNS
queries or SNMP traps. These messages need to arrive quickly and do not
deal with packet loss as there can be a very large volume of them. There
are a few extra considerations to take into account for this type of
workload and this can change a configuration all the way to the
hypervisor level. For an application that generates 10 TCP sessions per
user with an average bandwidth of 512 kilobytes per second per flow and
expected user count of ten thousand concurrent users, the expected
bandwidth plan is approximately 4.88 gigabits per second.
The supporting network for this type of configuration needs to have a
low latency and evenly distributed availability. This workload benefits
from having services local to the consumers of the service. Use a
multi-site approach as well as deploying many copies of the application
to handle load as close as possible to consumers. Since these
applications function independently, they do not warrant running
overlays to interconnect tenant networks. Overlays also have the
drawback of performing poorly with rapid flow setup and may incur too
much overhead with large quantities of small packets and therefore we do
not recommend them.
QoS is desirable for some workloads to ensure delivery. DNS has a major
impact on the load times of other services and needs to be reliable and
provide rapid responses. Configure rules in upstream devices to apply a
higher Class Selector to DNS to ensure faster delivery or a better spot
in queuing algorithms.
Cloud storage
~~~~~~~~~~~~~
Another common use case for OpenStack environments is providing a
cloud-based file storage and sharing service. You might consider this a
storage-focused use case, but its network-side requirements make it a
network-focused use case.
For example, consider a cloud backup application. This workload has two
specific behaviors that impact the network. Because this workload is an
externally-facing service and an internally-replicating application, it
has both :term:`north-south<north-south traffic>` and
:term:`east-west<east-west traffic>` traffic considerations:
north-south traffic
When a user uploads and stores content, that content moves into the
OpenStack installation. When users download this content, the
content moves out from the OpenStack installation. Because this
service operates primarily as a backup, most of the traffic moves
southbound into the environment. In this situation, it benefits you
to configure a network to be asymmetrically downstream because the
traffic that enters the OpenStack installation is greater than the
traffic that leaves the installation.
east-west traffic
Likely to be fully symmetric. Because replication originates from
any node and might target multiple other nodes algorithmically, it
is less likely for this traffic to have a larger volume in any
specific direction. However, this traffic might interfere with
north-south traffic.
.. figure:: ../figures/Network_Cloud_Storage2.png
This application prioritizes the north-south traffic over east-west
traffic: the north-south traffic involves customer-facing data.
The network design, in this case, is less dependent on availability and
more dependent on being able to handle high bandwidth. As a direct
result, it is beneficial to forgo redundant links in favor of bonding
those connections. This increases available bandwidth. It is also
beneficial to configure all devices in the path, including OpenStack, to
generate and pass jumbo frames.

210
doc/arch-design/source/use-cases/use-case-storage.rst
View File

@ -1,210 +0,0 @@
.. _storage-cloud:
=============
Storage cloud
=============
Design model
~~~~~~~~~~~~
Storage-focused architecture depends on specific use cases. This section
discusses three example use cases:
* An object store with a RESTful interface
* Compute analytics with parallel file systems
* High performance database
An object store with a RESTful interface
----------------------------------------
The example below shows a REST interface without a high performance
requirement. The following diagram depicts the example architecture:
.. figure:: ../figures/Storage_Object.png
The example REST interface, presented as a traditional Object Store
running on traditional spindles, does not require a high performance
caching tier.
This example uses the following components:
Network:
* 10 GbE horizontally scalable spine leaf back-end storage and front
end network.
Storage hardware:
* 10 storage servers each with 12x4 TB disks equaling 480 TB total
space with approximately 160 TB of usable space after replicas.
Proxy:
* 3x proxies
* 2x10 GbE bonded front end
* 2x10 GbE back-end bonds
* Approximately 60 Gb of total bandwidth to the back-end storage
cluster
.. note::
It may be necessary to implement a third party caching layer for some
applications to achieve suitable performance.
Compute analytics with data processing service
----------------------------------------------
Analytics of large data sets are dependent on the performance of the
storage system. Clouds using storage systems such as Hadoop Distributed
File System (HDFS) have inefficiencies which can cause performance
issues.
One potential solution to this problem is the implementation of storage
systems designed for performance. Parallel file systems have previously
filled this need in the HPC space and are suitable for large scale
performance-orientated systems.
OpenStack has integration with Hadoop to manage the Hadoop cluster
within the cloud. The following diagram shows an OpenStack store with a
high performance requirement:
.. figure:: ../figures/Storage_Hadoop3.png
The hardware requirements and configuration are similar to those of the
High Performance Database example below. In this case, the architecture
uses Ceph's Swift-compatible REST interface, features that allow for
connecting a caching pool to allow for acceleration of the presented
pool.
High performance database with Database service
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Databases are a common workload that benefit from high performance
storage back ends. Although enterprise storage is not a requirement,
many environments have existing storage that OpenStack cloud can use as
back ends. You can create a storage pool to provide block devices with
OpenStack Block Storage for instances as well as object interfaces. In
this example, the database I-O requirements are high and demand storage
presented from a fast SSD pool.
A storage system presents a LUN backed by a set of SSDs using a
traditional storage array with OpenStack Block Storage integration or a
storage platform such as Ceph or Gluster.
This system can provide additional performance. For example, in the
database example below, a portion of the SSD pool can act as a block
device to the Database server. In the high performance analytics
example, the inline SSD cache layer accelerates the REST interface.
.. figure:: ../figures/Storage_Database_+_Object5.png
In this example, Ceph presents a swift-compatible REST interface, as
well as a block level storage from a distributed storage cluster. It is
highly flexible and has features that enable reduced cost of operations
such as self healing and auto balancing. Using erasure coded pools are a
suitable way of maximizing the amount of usable space.
.. note::
There are special considerations around erasure coded pools. For
example, higher computational requirements and limitations on the
operations allowed on an object; erasure coded pools do not support
partial writes.
Using Ceph as an applicable example, a potential architecture would have
the following requirements:
Network:
* 10 GbE horizontally scalable spine leaf back-end storage and
front-end network
Storage hardware:
* 5 storage servers for caching layer 24x1 TB SSD
* 10 storage servers each with 12x4 TB disks which equals 480 TB total
space with about approximately 160 TB of usable space after 3
replicas
REST proxy:
* 3x proxies
* 2x10 GbE bonded front end
* 2x10 GbE back-end bonds
* Approximately 60 Gb of total bandwidth to the back-end storage
cluster
Using an SSD cache layer, you can present block devices directly to
hypervisors or instances. The REST interface can also use the SSD cache
systems as an inline cache.
Requirements
~~~~~~~~~~~~
Storage requirements
--------------------
Storage-focused OpenStack clouds must address I/O intensive workloads.
These workloads are not CPU intensive, nor are they consistently network
intensive. The network may be heavily utilized to transfer storage, but
they are not otherwise network intensive.
The selection of storage hardware determines the overall performance and
scalability of a storage-focused OpenStack design architecture. Several
factors impact the design process, including:
Latency
A key consideration in a storage-focused OpenStack cloud is latency.
Using solid-state disks (SSDs) to minimize latency and, to reduce CPU
delays caused by waiting for the storage, increases performance. Use
RAID controller cards in compute hosts to improve the performance of the
underlying disk subsystem.
Scale-out solutions
Depending on the storage architecture, you can adopt a scale-out
solution, or use a highly expandable and scalable centralized storage
array. If a centralized storage array meets your requirements, then the
array vendor determines the hardware selection. It is possible to build
a storage array using commodity hardware with Open Source software, but
requires people with expertise to build such a system.
On the other hand, a scale-out storage solution that uses
direct-attached storage (DAS) in the servers may be an appropriate
choice. This requires configuration of the server hardware to support
the storage solution.
Considerations affecting storage architecture (and corresponding storage
hardware) of a Storage-focused OpenStack cloud include:
Connectivity
Ensure the connectivity matches the storage solution requirements. We
recommend confirming that the network characteristics minimize latency
to boost the overall performance of the design.
Latency
Determine if the use case has consistent or highly variable latency.
Throughput
Ensure that the storage solution throughput is optimized for your
application requirements.
Server hardware
Use of DAS impacts the server hardware choice and affects host
density, instance density, power density, OS-hypervisor, and
management tools.
Component block diagram
~~~~~~~~~~~~~~~~~~~~~~~

14
doc/arch-design/source/use-cases/use-case-web-scale.rst
View File

@ -1,14 +0,0 @@
.. _web-scale-cloud:
===============
Web scale cloud
===============
Design model
~~~~~~~~~~~~
Requirements
~~~~~~~~~~~~
Component block diagram
~~~~~~~~~~~~~~~~~~~~~~~

5
tools/build-all-rst.sh
View File

@ -27,11 +27,10 @@ while [[ $# > 0 ]] ; do
done done
# PDF targets for Install guides are dealt in build-install-guides-rst.sh # PDF targets for Install guides are dealt in build-install-guides-rst.sh
PDF_TARGETS=( 'arch-design'\ PDF_TARGETS=( 'image-guide' \
'image-guide' \
'install-guide') 'install-guide')
for guide in arch-design doc-contrib-guide glossary \ for guide in doc-contrib-guide glossary \
image-guide install-guide; do image-guide install-guide; do
if [[ ${PDF_TARGETS[*]} =~ $guide ]]; then if [[ ${PDF_TARGETS[*]} =~ $guide ]]; then
tools/build-rst.sh doc/$guide --build build \ tools/build-rst.sh doc/$guide --build build \