openstack
/
octavia
Code Issues Proposed changes

Updated the Octavia 0.5 spec 

Change-Id: I4859e1b93c144eb0cd1492410768aeadf844b68b
This commit is contained in:
Susanne Balle 2014-10-02 11:55:09 -04:00
parent 9885233255
commit f24a6a861e
2 changed files with 112 additions and 109 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

199
doc/source/design/version0.5/component-design.rst
View File

@ -91,7 +91,7 @@ appliances or other "service providing entity."
components which need not be stateless.
* It is also possible for multiple instances of the driver will talk to the
same Octavia VM at the same time. Emphasis on the idempotency of the update
same amphora at the same time. Emphasis on the idempotency of the update
algorithms used should help minimize the issues this can potentially cause.
NETWORK DRIVER
@ -155,35 +155,35 @@ for Octavia to have its own operator API / interface.
CONTROLLER
----------
This is the component providing all the command and control for the Octavia
VMs. On the front end, it takes its commands and controls from the LBaaS
This is the component providing all the command and control for the
amphorae. On the front end, it takes its commands and controls from the LBaaS
driver.
It should be noted that in later releases of Octavia, the controller functions
will be split across several components. At this stage we are less concerned
with how this internal communcation will happen, and are most concerned with
ensuring communication with Octavia VMs, the LBaaS driver, and the Network
ensuring communication with amphorae, the amphora LB driver, and the Network
driver are all made as perfect as possible.
Among the controller's responsibilities are:
* Sending configuration and certificate information to an octavia backend
* Sending configuration and certificate information to an amphora LB
driver, which in the reference implementation will be generating
configuration files for haproxy and PEM-formatted user certificates and
sending these to individual octavia VMs. Configuration files will be
sending these to individual amphorae. Configuration files will be
generated from jinja templates kept in an template directory specific to
the haproxy driver.
* Processing the configuration updates that need to be applied to individual
octavia VMs, as sent by the LBaaS driver.
amphorae, as sent by the amphora LB driver.
* Interfacing with network driver to plumb additional interfaces on the octavia
VMs as necessary.
* Interfacing with network driver to plumb additional interfaces on the
amphorae as necessary.
* Monitoring the health of all octavia VMs (via a driver interface).
* Monitoring the health of all amphorae (via a driver interface).
* Receiving and routing certain kinds of notifications originating on the
octavia VMs (ex. "member down")
amphorae (ex. "member down")
* This is a stateful service, and should keep its state in a central, highly
available database of some sort.
@ -192,78 +192,79 @@ Among the controller's responsibilities are:
forth by users.
* Receiving notifications, statistics data and other short, regular messages
from octavia VMs and routing them to the appropriate entity.
from amphorae and routing them to the appropriate entity.
* Responding to requests from octavia VMs for configuration data.
* Responding to requests from amphorae for configuration data.
* Responding to requests from the user API or operator API handler driver for
data about specific loadbalancers or sub-objects, their status, and
statistics.
* octavia VM lifecycle management, including interfacing with Nova and Neutron
to spin up new octavia VMs as necessary and handle initial configuration and
network plumbing for their LB network interface, and cleaning this up when a
octavia VM is destroyed.
* Amphora lifecycle management, including interfacing with Nova and Neutron
to spin up new amphorae as necessary and handle initial configuration and
network plumbing for their LB network interface, and cleaning this up when an
amphora is destroyed.
* Maintaining a pool of spare octavia VMs (ie. spawning new ones as necessary
* Maintaining a pool of spare amphorae (ie. spawning new ones as necessary
and deleting ones from the pool when we have too much inventory here.)
* Gracefully spinning down "dirty old octavia VMs"
* Gracefully spinning down "dirty old amphorae"
* Loading and calling configured octavia backend drivers.
* Loading and calling configured amphora drivers.
**Notes:**
* Almost all the intelligence around putting together and validating
loadbalancer configurations will live here-- the octavia VM API is meant to
loadbalancer configurations will live here-- the Amphora API is meant to
be as simple as possible so that minor feature improvements do not
necessarily entail pushing out new octavia VMs across an entire installation.
necessarily entail pushing out new amphorae across an entire installation.
* The size of the spare octavia VM pool should be determined by the flavor
* The size of the spare amphora pool should be determined by the flavor
being offered.
* The controller also handles spinning up octavia VMs in the case of a true
* The controller also handles spinning up amphorae in the case of a true
active/standby topology (ie. where the spares pool is effectively zero.) It
should have enough intelligence to communicate to Nova that these octavia
VMs should not be on the same physical host in this topology.
should have enough intelligence to communicate to Nova that these amphorae
should not be on the same physical host in this topology.
* It also handles spinning up new octavia VMs when one fails in the above
* It also handles spinning up new amphorae when one fails in the above
topology.
* Since spinning up a new octavia VM is a task that can take a long time, the
* Since spinning up a new amphora is a task that can take a long time, the
controller should spawn a job or child process which handles this highly
asynchronous request.
VM DRIVER
---------
AMPHORA LOAD BALANCER (LB) DRIVER
---------------------------------
This is the abstraction layer that the controller talks to for communicating
with the Octavia VMs. Since we want to keep Octavia flexible enough so that
certain components (like the Octavia VM) can be replaced by third party
with the amphorae. Since we want to keep Octavia flexible enough so that
certain components (like the amphora) can be replaced by third party
products if the operator so desires, it's important to keep many of the
implementation-specific details contained within driver layers. A VM driver
also gives the operator the ability to have different open-source Octavia VMs
with potentially different capabilities (accessed via different flavors) which
can be handy for, for example, field-testing a new Octavia VM image.
implementation-specific details contained within driver layers. An amphora
LB driver also gives the operator the ability to have different open-source
amphorae with potentially different capabilities (accessed via different
flavors) which can be handy for, for example, field-testing a new amphora
image.
The reference implementation for the VM driver will be for the Octavia VM
The reference implementation for the amphora LB driver will be for the amphora
described below.
Responsibilities of the VM driver include:
Responsibilities of the amphora LB driver include:
* Generating configuration files for haproxy and PEM-formatted user
certificates and sending these to individual octavia VMs. Configuration
certificates and sending these to individual amphorae. Configuration
files will be generated from jinja templates kept in an template directory
specific to the haproxy driver.
* Handling all communication to and from Octavia VMs.
* Handling all communication to and from amphorae.
LB NETWORK
----------
This is the subnet that controllers will use to communicate with octavia VMs.
This is the subnet that controllers will use to communicate with amphorae.
This means that controllers must have connectivity (either layer 2 or routed)
to this subnet in order to function, and visa versa. Since Octavia VMs will be
to this subnet in order to function, and visa versa. Since amphorae will be
communicating on it, this means the network is not part of the "undercloud."
**Notes:**
@ -271,11 +272,11 @@ communicating on it, this means the network is not part of the "undercloud."
* As certain sensitive data (TLS private keys, for example) will be transmitted
over this communication infrastructure, all messages carrying a sensitive
payload should be done via encrypted and authenticated means. Further, we
recommend that messages to and from octavia VMs be signed regardless of the
recommend that messages to and from amphorae be signed regardless of the
sensitivity of their content.
OCTAVIA VM
AMPHORAE
----------
This is a Nova VM which actually provides the load balancing services as
configured by the user. Responsibilities of these entities include:
@ -297,36 +298,37 @@ configured by the user. Responsibilities of these entities include:
**Notes:**
* Each octavia VM will generally need its own dedicated LB network IP address,
* Each amphora will generally need its own dedicated LB network IP address,
both so that we don't accidentally bind to any IP:port the user wants to use
for loadbalancing services, and so that an octavia VM that is not yet in use
for loadbalancing services, and so that an amphora that is not yet in use
by any loadbalancer service can still communicate on the network and receive
commands from its controller. Whether this IP address exists on the same
subnet as the loadbalancer services it hosts is immaterial, so long as
front-end and back-end interfaces can be plumbed after a VM is launched.
front-end and back-end interfaces can be plumbed after an amphora is
launched.
* Since octavia VMs speak to controllers in a "trusted" way, it's important to
ensure that users do not have command-line access to the octavia VMs. In
other words, the octavia VMs should be a black box from the users'
* Since amphorae speak to controllers in a "trusted" way, it's important to
ensure that users do not have command-line access to the amphorae. In
other words, the amphorae should be a black box from the users'
perspective.
* Octavia VMs will be powered using haproxy 1.5 initially. We may decide to use
* Amphorae will be powered using haproxy 1.5 initially. We may decide to use
other software (especially for TLS termination) later on.
* The "glue scripts" which communicate with the controller should be as
lightweight as possible: Intelligence about how to put together an haproxy
config, for example, should not live on the octavia VM. Rather, the octavia
VM should perform simple syntax checks, start / restart haproxy if the checks
config, for example, should not live on the amphora. Rather, the amphora
should perform simple syntax checks, start / restart haproxy if the checks
pass, and report success/failure of the haproxy restart.
* With few exceptions, most of the API commands the octavia VM will ever do
* With few exceptions, most of the API commands the amphora will ever do
should be safely handled synchronously (ie. nothing should take longer than a
second or two to complete).
* Connection logs, and other things anticipated to generate a potential large
amount of data should be communicated by the octavia VM directly to which
amount of data should be communicated by the amphora directly to which
ever service is going to consume that data. (for example, if logs are being
shunted off to swift on a nightly basis, the octavia VM should handle this
shunted off to swift on a nightly basis, the amphora should handle this
directly and not go through the controller.)
@ -337,59 +339,59 @@ daemon(s) which periodically check that heartbeats from monitored entities are
both current and showing "good" status, if applicable. Specifically:
* Controllers need to be able to monitor the availability and overall health
of octavia VMs they control. For active node octavia VMs, this check should
happen pretty quickly: About once every 5 seconds. For spare node octavia
VMs, the check can happen much more infrequently (say, once per minute).
of amphorae they control. For active amphorae, this check should
happen pretty quickly: About once every 5 seconds. For spare amphorae,
the check can happen much more infrequently (say, once per minute).
The idea here is that internal health monitors will monitor a periodic
heartbeat coming from the octavia VMs, and take appropriate action (assuming
heartbeat coming from the amphorae, and take appropriate action (assuming
these are down) if they fail to check in with a heartbeat frequently enough.
This means that internal health monitors need to take the form of a daemon
which is constantly checking for and processing heartbeat requests (and
updating Controller or octavia VM statuses, and triggering other events as
updating controller or amphorae statuses, and triggering other events as
appropriate).
======================================================
Some notes on Controller <-> octavia VM communications
Some notes on Controller <-> Amphorae communications
======================================================
In order to keep things as scalable as possible, the thought was that short,
periodic and arguably less vital messages being emitted by the octavia VM and
periodic and arguably less vital messages being emitted by the amphora and
associated controller would be done via HMAC-signed UDP, and more vital, more
sensitive, and potentially longer transactional messages would be handled via a
RESTful API on the controller, accessed via bi-directionally authenticated
HTTPS.
Specifically, we should expect the following to happen over UDP:
* heartbeats from the octavia VM to the controller
* heartbeats from the amphora VM to the controller
* stats data from the octavia VM to the controller
* stats data from the amphora to the controller
* "edge" alert notifications (change in status) from the octavia VM to the
* "edge" alert notifications (change in status) from the amphora to the
controller
* Notification of pending tasks in queue from controller to octavia VM
* Notification of pending tasks in queue from controller to amphora
And the following would happen over TCP:
* haproxy / tls certificate configuration changes
=================================================
Supported Octavia VM Virtual Appliance Topologies
Supported Amphora Virtual Appliance Topologies
=================================================
Initially, I propose we support two topologies with version 0.5 of Octavia:
Option 1: "Single active node + spares pool"
--------------------------------------------
* This is similar to what HP is doing right now with Libra: Each octavia VM is
* This is similar to what HP is doing right now with Libra: Each amphora is
stand-alone with a frequent health-check monitor in place and upon failure,
an already-spun-up octavia VM is moved from the spares pool and configured to
an already-spun-up amphora is moved from the spares pool and configured to
take the old one's place. This allows for acceptable recovery times on
octavia VM failure while still remaining efficient, as far as VM resource
amphora failure while still remaining efficient, as far as VM resource
utilization is concerned.
Option 2: "True Active / Standby"
---------------------------------
* This is similar to what Blue Box is doing right now where octavia VMs are
* This is similar to what Blue Box is doing right now where amphorae are
deployed in pairs and use corosync / pacemaker to monitor each other's health
and automatically take over (usually in less than 5 seconds) if the "active"
node fails. This provides for the fastest possible recovery time on hardware
@ -397,10 +399,10 @@ Option 2: "True Active / Standby"
concerned.
* In this topology a floating IP address (different from a Neutron floating
IP!) is used to determine which octavia VM is the "active" one at any given
IP!) is used to determine which amphora is the "active" one at any given
time.
* In this topology, both octavia VMs need to be colocated on the same subnet.
* In this topology, both amphorae need to be colocated on the same subnet.
As such a "spares pool" doesn't make sense for this type of layout, unless
all spares are on the same mamangement network with the active nodes.
@ -410,7 +412,7 @@ the same thing as option 1 above with a spares pool size of zero.
============================
Supported Network Topologies
============================
This is actually where things get tricky, as far as octavia VM plumbing is
This is actually where things get tricky, as far as amphora plumbing is
concerned. And it only grows trickier when we consider that front-end
connectivity (ie. to the 'loadbalancer' vip_address) and back-end connectivity
(ie. to members of a loadbalancing pool) can be handled in different ways.
@ -419,39 +421,40 @@ back-end topology to discuss the various possible permutations here.
LB Network
----------
Each octavia VM needs to have a connection to a LB network. And each controller
Each amphora needs to have a connection to a LB network. And each controller
needs to have access to this management network (this could be layer-2 or
routed connectivity). Command and control will happen via the octavia VM's
routed connectivity). Command and control will happen via the amphorae's
LB network IP.
Front-end topologies
--------------------
There are generally two ways to handle the octavia VM's connection to the
There are generally two ways to handle the amphorae's connection to the
front-end IP address (this is the vip_address of the loadbalancer object):
**Option 1: Layer-2 connectivity**
The octavia VM can have layer-2 connectivity to the neutron network which is
The amphora can have layer-2 connectivity to the neutron network which is
host to the subnet on which the loadbalancer vip_address resides. In this
scenario, the octavia VM would need to send ARP responses to requests for the
vip_address, and therefore octavia VMs need to have interfaces plumbed on said
scenario, the amphora would need to send ARP responses to requests for the
vip_address, and therefore amphorae need to have interfaces plumbed on said
vip_address subnets which participate in ARP.
Note that this is somewhat problematic for active / standby virtual appliance
topologies because the vip_address for a given load balancer effectively
becomes a highly-available IP address (a true floating VIP), which means on
service failover from active to standby, the active octavia VM needs to
service failover from active to standby, the active amphora needs to
relenquish all the vip_addresses it has, and the standby needs to take them
over *and* start up haproxy services. This is OK if a given octavia
VM only has a few load balancers, but can lead to several minutes' down-time
over *and* start up haproxy services. This is OK if a given amphora
only has a few load balancers, but can lead to several minutes' down-time
during a graceful failover if there are a dozen or more load balancers on the
active/standby octavia VM pair. It's also more risky: The standby node might
active/standby amphora pair. It's also more risky: The standby node might
not be able to start up all the haproxy services during such a
failover. What's more, most types of VRRP-like services which handle floating
IPs require octavia VMs to have an additional IP address on the subnet housing
the floating vip_address in order for the standby VM to monitor the active VM.
IPs require amphorae to have an additional IP address on the subnet housing
the floating vip_address in order for the standby amphora to monitor the
active amphora.
Also note that in this topology, octavia VMs need an additional virtual network
Also note that in this topology, amphorae need an additional virtual network
interface plumbed when new front-end loadbalancer vip_addresses are assigned to
them which exist on subnets to which they don't already have access.
@ -459,7 +462,7 @@ them which exist on subnets to which they don't already have access.
In this layout, static routes are injected into the routing infrastructure
(Neutron) which essentially allow traffic destined for any given loadbalancer
vip_address to be routed to an IP address which lives on the octavia VM. (I
vip_address to be routed to an IP address which lives on the amphora. (I
would recommend this be something other than the LB network IP.) In this
topology, it's actually important that the loadbalancer vip_address does *not*
exist in any subnet with potential front-end clients because in order for
@ -468,38 +471,38 @@ infrastructure (and in this case, front-end clients would attempt layer-2
connectivity to the vip_address).
This topology also works much better for active/standby configurations, because
both the active and standby octavia VMs can bind to the vip_addresses of all
both the active and standby amphorae can bind to the vip_addresses of all
their assigned loadbalancer objects on a dummy, non-ARPing interface, both can
be running all haproxy services at the same time, and keep the
standby server processes from interfering with active loadbalancer traffic
through the use of fencing scripts on the octavia VMs. Static routing is
through the use of fencing scripts on the amphorae. Static routing is
accomplished to a highly available floating "routing IP" (using some VRRP-like
service for just this IP) which becomes the trigger for the fencing scripts on
the octavia VM. In this scenario, fail-overs are both much more reliable, and
the amphora. In this scenario, fail-overs are both much more reliable, and
can be accomplished in usually < 5 seconds.
Further, in this topology, octavia VMs do not need any additional virtual
Further, in this topology, amphorae do not need any additional virtual
interfaces plumbed when new front-end loadbalancer vip_addresses are assigned
to them.
Back-end topologies
-------------------
There are also two ways that octavia VMs can potentially talk to back-end
There are also two ways that amphorae can potentially talk to back-end
member IP addresses. Unlike the front-end topologies (where option 1 and option
2 are basically mutually exclusive, if not practically exclusive) both of these
types of connectivity can be used on a single octavia VM, and indeed, within a
types of connectivity can be used on a single amphora, and indeed, within a
single loadbalancer configuration.
**Option 1: Layer-2 connectivity**
This is layer-2 connectivity to back-end members, and is implied when a member
object has a subnet_id assigned to it. In this case, the existence of the
subnet_id implies octavia VMs need to have layer-2 connectivity to that subnet,
subnet_id implies amphorae need to have layer-2 connectivity to that subnet,
which means they need to have a virtual interface plumbed to it, as well as an
IP address on the subnet. This type of connectivity is useful for "secure"
back-end subnets that exist behind a NATing firewall where PAT is not in use on
the firewall. (In this way it effectively bypasses the firewall.) I anticipate
the firewall. (In this way it effectively bypasses the firewall.) We anticipate
this will be the most common form of back-end connectivity in use by most
OpenStack users.
@ -508,7 +511,7 @@ OpenStack users.
This is routed connectivity to back-end members. This is implied when a member
object does not have a subnet_id specified. In this topology, it is assumed
that member ip_addresses are reachable through standard neutron routing, and
therefore connections to them can be initiated from the octavia VM's default
therefore connections to them can be initiated from the amphora's default
gateway. No new virtual interfaces need to be plumbed for this type of
connectivity to members.

22
doc/source/design/version0.5/v0.5-component-design.dot
View File

@ -159,33 +159,33 @@ digraph G {
<td sides="tlrb"><font point-size="10"> Health Monitor </font></td>
</tr>
<tr>
<td sides="tlrb"><font point-size="10"> VM Driver </font></td>
<td sides="tlrb"><font point-size="10"> Amphora LB Driver </font></td>
</tr>
</table>>];
Database [shape=cylinder];
};
subgraph cluster_OctaviaVMs2 {
subgraph cluster_Amphoras2 {
label="";
color="white";
id="OctaviaMVs2";
OctaviaVM3 [shape=none margin=0 label=
id="AmphoraMVs2";
Amphora3 [shape=none margin=0 label=
<<table border="0" cellspacing="0" cellborder="1">
<tr>
<td>Octavia VM</td>
<td>Amphora</td>
</tr>
</table>>];
OctaviaVM4 [shape=none margin=0 label=
Amphora4 [shape=none margin=0 label=
<<table border="0" cellspacing="0" cellborder="1">
<tr>
<td>Octavia VM</td>
<td>Amphora</td>
</tr>
</table>>];
OctaviaVM5 [shape=none margin=0 label=
Amphora5 [shape=none margin=0 label=
<<table border="0" cellspacing="0" cellborder="1">
<tr>
<td>Octavia VM</td>
<td>Amphora</td>
</tr>
</table>>];
};
@ -212,8 +212,8 @@ digraph G {
OperatorAPIHandler -> Controller [dir=none];
Controller -> LBNetwork [style=invis];
LBNetwork -> {OctaviaVM3 OctaviaVM4 OctaviaVM5} [style=invis];
Controller -> {OctaviaVM3 OctaviaVM4 OctaviaVM5} [constraint=false dir=none];
LBNetwork -> {Amphora3 Amphora4 Amphora5} [style=invis];
Controller -> {Amphora3 Amphora4 Amphora5} [constraint=false dir=none];
Controller -> NetworkingDriver [constraint=false dir=none];
Controller -> Database [constraint=false dir=none];