openstack/octavia
Code Issues Proposed changes

Fix the amphora image support for RH Linux flavors

Browse Source 
Not all Linux flavors accept the same type of configuration to manage
NICs. The amphora-agent must be able to distinguish between different
Linux flavors and choose the appropriate type of jinja2 NIC
configuration template for each one, respectively.

Up until now, The amphora-agent had no notion of the operating system
it is running on, therefore it used NIC configuration templates that
only match Debian based Linux flavors (mostly Ubuntu). Making it
unusable for flavors such as RHEL, Fedora and CentOS.

This fix enhances how the amphora-agent is handling NIC hot plugs.
It will use the appropriate jinja2 template by checking the Amphora
distribution name when needed.

Co-Authored-By: Michael Johnson <johnsomor@gmail.com>

Closes-Bug #1548070

Change-Id: Id99948aec64656a0532afc68e146f0610bff1378
This commit is contained in:
Nir Magnezi 2017-01-12 11:32:50 +02:00
parent 67866ea193
commit c00488143d
38 changed files with 2620 additions and 828 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
diskimage-create/diskimage-create.sh
View File

@ -333,21 +333,21 @@ if [ "$AMP_BASEOS" = "ubuntu" ]; then
export UBUNTU_MIRROR="$BASE_OS_MIRROR"
fi
elif [ "$AMP_BASEOS" = "fedora" ]; then
AMP_element_sequence=${AMP_element_sequence:-"base vm fedora"}
AMP_element_sequence=${AMP_element_sequence:-"base vm fedora selinux-permissive"}
AMP_element_sequence="$AMP_element_sequence $AMP_BACKEND"
if [ "$BASE_OS_MIRROR" ]; then
AMP_element_sequence="$AMP_element_sequence fedora-mirror"
export FEDORA_MIRROR="$BASE_OS_MIRROR"
fi
elif [ "$AMP_BASEOS" = "centos" ]; then
AMP_element_sequence=${AMP_element_sequence:-"base vm centos7 epel"}
AMP_element_sequence=${AMP_element_sequence:-"base vm centos7 epel selinux-permissive"}
AMP_element_sequence="$AMP_element_sequence $AMP_BACKEND"
if [ "$BASE_OS_MIRROR" ]; then
AMP_element_sequence="$AMP_element_sequence centos-mirror"
export CENTOS_MIRROR="$BASE_OS_MIRROR"
fi
elif [ "$AMP_BASEOS" = "rhel" ]; then
AMP_element_sequence=${AMP_element_sequence:-"base vm rhel7"}
AMP_element_sequence=${AMP_element_sequence:-"base vm rhel7 selinux-permissive"}
AMP_element_sequence="$AMP_element_sequence $AMP_BACKEND"
fi

3
elements/amphora-agent/install.d/75-run_setup_install
View File

@ -8,6 +8,9 @@ set -o pipefail
pip install -U -c /opt/upper-constraints.txt /opt/amphora-agent
# Accommodate centos default install location
ln -s /bin/amphora-agent /usr/local/bin/amphora-agent || true
mkdir /etc/octavia
# we assume certs, etc will come in through the config drive
mkdir /etc/octavia/certs

9
elements/haproxy-octavia-ubuntu/os-refresh-config/configure.d/20-haproxy-selinux
View File

@ -1,9 +0,0 @@
#!/bin/bash
set -eux
set -o pipefail
# Allow haproxy to proxy any port if SELinux is in enforcing mode
# https://bugs.launchpad.net/tripleo/+bug/1339938
if [[ -x /usr/sbin/semanage ]]; then
setsebool -P haproxy_connect_any 1
fi

12
elements/haproxy-octavia-ubuntu/post-install.d/20-setup-haproxy-log Executable file
View File

@ -0,0 +1,12 @@
#!/bin/bash
set -eu
set -o pipefail
if [[ -d /etc/rsyslog.d ]] && [[ ! -e /etc/rsyslog.d/49-haproxy.conf ]]; then
cat >> /etc/rsyslog.d/49-haproxy.conf <<EOF
# Send HAProxy messages to a dedicated logfile
if \$programname startswith 'haproxy' then /var/log/haproxy.log
&~
EOF
fi

9
elements/haproxy-octavia/os-refresh-config/configure.d/20-haproxy-selinux
View File

@ -1,9 +0,0 @@
#!/bin/bash
set -eux
set -o pipefail
# Allow haproxy to proxy any port if SELinux is in enforcing mode
# https://bugs.launchpad.net/tripleo/+bug/1339938
if [[ -x /usr/sbin/semanage ]]; then
setsebool -P haproxy_connect_any 1
fi

0
elements/haproxy-octavia/post-install.d/20-disable-default-haproxy Normal file → Executable file
View File

12
elements/haproxy-octavia/post-install.d/20-setup-haproxy-log Executable file
View File

@ -0,0 +1,12 @@
#!/bin/bash
set -eu
set -o pipefail
if [[ -d /etc/rsyslog.d ]] && [[ ! -e /etc/rsyslog.d/49-haproxy.conf ]]; then
cat >> /etc/rsyslog.d/49-haproxy.conf <<EOF
# Send HAProxy messages to a dedicated logfile
if \$programname startswith 'haproxy' then /var/log/haproxy.log
&~
EOF
fi

6
elements/keepalived-octavia/post-install.d/20-enable-keepalived
View File

@ -1,6 +0,0 @@
#!/bin/bash
set -eu
set -o pipefail
chkconfig keepalived on

2
elements/no-resolvconf/README.rst
View File

@ -3,6 +3,8 @@ it with data from DHCP. This means that DNS resolution will not work from the
amphora. This is OK because all outbound connections from the amphora will
be based using raw IP addresses.
In addition we remove dns from the nsswitch.conf hosts setting.
This has the real benefit of speeding up host boot and configutation times.
This is especially helpful when running tempest tests in a devstack environment
where DNS resolution from the amphora usually doesn't work anyway: This means

4
elements/no-resolvconf/finalise.d/99-disable-resolv-conf
View File

@ -14,3 +14,7 @@ else
make_resolv_conf() { : ; }" > /etc/dhclient-enter-hooks
chmod +x /etc/dhclient-enter-hooks
fi
if [ -e /etc/nsswitch.conf ]; then
sed -i -e "/hosts:/ s/dns//g" /etc/nsswitch.conf
fi

8
etc/octavia.conf
View File

@ -224,7 +224,13 @@
[amphora_agent]
# agent_server_ca = /etc/octavia/certs/client_ca.pem
# agent_server_cert = /etc/octavia/certs/server.pem
# agent_server_network_dir = /etc/netns/amphora-haproxy/network/interfaces.d/
# Defaults for agent_server_network_dir when not specified here are:
# Ubuntu: /etc/netns/amphora-haproxy/network/interfaces.d/
# Centos/fedora/rhel: /etc/netns/amphora-haproxy/sysconfig/network-scripts/
#
# agent_server_network_dir =
# agent_server_network_file =
# agent_request_read_timeout = 120

314
octavia/amphorae/backends/agent/api_server/amphora_info.py
View File

@ -31,177 +31,177 @@ from octavia.common import constants as consts
LOG = logging.getLogger(__name__)
def compile_amphora_info():
return flask.jsonify(
{'hostname': socket.gethostname(),
'haproxy_version': _get_version_of_installed_package('haproxy'),
'api_version': api_server.VERSION})
class AmphoraInfo(object):
def __init__(self, osutils):
self._osutils = osutils
def compile_amphora_info(self):
return flask.jsonify(
{'hostname': socket.gethostname(),
'haproxy_version':
self._get_version_of_installed_package('haproxy'),
'api_version': api_server.VERSION})
def compile_amphora_details():
listener_list = util.get_listeners()
meminfo = _get_meminfo()
cpu = _cpu()
st = os.statvfs('/')
return flask.jsonify(
{'hostname': socket.gethostname(),
'haproxy_version': _get_version_of_installed_package('haproxy'),
'api_version': api_server.VERSION,
'networks': _get_networks(),
'active': True,
'haproxy_count': _count_haproxy_processes(listener_list),
'cpu': {
'total': cpu['total'],
'user': cpu['user'],
'system': cpu['system'],
'soft_irq': cpu['softirq'], },
'memory': {
'total': meminfo['MemTotal'],
'free': meminfo['MemFree'],
'buffers': meminfo['Buffers'],
'cached': meminfo['Cached'],
'swap_used': meminfo['SwapCached'],
'shared': meminfo['Shmem'],
'slab': meminfo['Slab'], },
'disk': {
'used': (st.f_blocks - st.f_bfree) * st.f_frsize,
'available': st.f_bavail * st.f_frsize},
'load': [
_load()],
'topology': consts.TOPOLOGY_SINGLE,
'topology_status': consts.TOPOLOGY_STATUS_OK,
'listeners': listener_list,
'packages': {}})
def compile_amphora_details(self):
listener_list = util.get_listeners()
meminfo = self._get_meminfo()
cpu = self._cpu()
st = os.statvfs('/')
return flask.jsonify(
{'hostname': socket.gethostname(),
'haproxy_version':
self._get_version_of_installed_package('haproxy'),
'api_version': api_server.VERSION,
'networks': self._get_networks(),
'active': True,
'haproxy_count': self._count_haproxy_processes(listener_list),
'cpu': {
'total': cpu['total'],
'user': cpu['user'],
'system': cpu['system'],
'soft_irq': cpu['softirq'], },
'memory': {
'total': meminfo['MemTotal'],
'free': meminfo['MemFree'],
'buffers': meminfo['Buffers'],
'cached': meminfo['Cached'],
'swap_used': meminfo['SwapCached'],
'shared': meminfo['Shmem'],
'slab': meminfo['Slab'], },
'disk': {
'used': (st.f_blocks - st.f_bfree) * st.f_frsize,
'available': st.f_bavail * st.f_frsize},
'load': self._load(),
'topology': consts.TOPOLOGY_SINGLE,
'topology_status': consts.TOPOLOGY_STATUS_OK,
'listeners': listener_list,
'packages': {}})
def _get_version_of_installed_package(self, name):
def _get_version_of_installed_package(name):
cmd = "dpkg --status {name}".format(name=name)
out = subprocess.check_output(cmd.split())
m = re.search(b'Version: .*', out)
return m.group(0)[len('Version: '):]
cmd = self._osutils.cmd_get_version_of_installed_package(name)
out = subprocess.check_output(cmd.split())
m = re.search(b'Version: .*', out)
return m.group(0)[len('Version: '):]
def _count_haproxy_processes(self, listener_list):
num = 0
for listener_id in listener_list:
if util.is_listener_running(listener_id):
# optional check if it's still running
num += 1
return num
def _count_haproxy_processes(listener_list):
num = 0
for listener_id in listener_list:
if util.is_listener_running(listener_id):
# optional check if it's still running
num += 1
return num
def _get_meminfo(self):
re_parser = re.compile(r'^(?P<key>\S*):\s*(?P<value>\d*)\s*kB')
result = dict()
with open('/proc/meminfo', 'r') as meminfo:
for line in meminfo:
match = re_parser.match(line)
if not match:
continue # skip lines that don't parse
key, value = match.groups(['key', 'value'])
result[key] = int(value)
return result
def _cpu(self):
with open('/proc/stat') as f:
cpu = f.readline()
vals = cpu.split(' ')
return {
'user': vals[2],
'nice': vals[3],
'system': vals[4],
'idle': vals[5],
'iowait': vals[6],
'irq': vals[7],
'softirq': vals[8],
'total': sum([int(i) for i in vals[2:]])
}
def _get_meminfo():
re_parser = re.compile(r'^(?P<key>\S*):\s*(?P<value>\d*)\s*kB')
result = dict()
for line in open('/proc/meminfo'):
match = re_parser.match(line)
if not match:
continue # skip lines that don't parse
key, value = match.groups(['key', 'value'])
result[key] = int(value)
return result
def _load(self):
with open('/proc/loadavg') as f:
load = f.readline()
vals = load.split(' ')
return vals[:3]
def _get_networks(self):
networks = dict()
with pyroute2.NetNS(consts.AMPHORA_NAMESPACE) as netns:
for interface in netns.get_links():
interface_name = None
for item in interface['attrs']:
if (item[0] == 'IFLA_IFNAME'
and not item[1].startswith('eth')):
break
elif item[0] == 'IFLA_IFNAME':
interface_name = item[1]
if item[0] == 'IFLA_STATS64':
networks[interface_name] = dict(
network_tx=item[1]['tx_bytes'],
network_rx=item[1]['rx_bytes'])
return networks
def _cpu():
with open('/proc/stat') as f:
cpu = f.readline()
vals = cpu.split(' ')
return {
'user': vals[2],
'nice': vals[3],
'system': vals[4],
'idle': vals[5],
'iowait': vals[6],
'irq': vals[7],
'softirq': vals[8],
'total': sum([int(i) for i in vals[2:]])
}
def get_interface(self, ip_addr):
try:
ip_version = ipaddress.ip_address(six.text_type(ip_addr)).version
except Exception:
return flask.make_response(
flask.jsonify(dict(message="Invalid IP address")), 400)
def _load():
with open('/proc/loadavg') as f:
load = f.readline()
vals = load.split(' ')
return vals[:3]
if ip_version == 4:
address_format = netifaces.AF_INET
elif ip_version == 6:
address_format = netifaces.AF_INET6
else:
return flask.make_response(
flask.jsonify(dict(message="Bad IP address version")), 400)
# We need to normalize the address as IPv6 has multiple representations
# fe80:0000:0000:0000:f816:3eff:fef2:2058 == fe80::f816:3eff:fef2:2058
normalized_addr = socket.inet_ntop(address_format,
socket.inet_pton(address_format,
ip_addr))
def _get_networks():
networks = dict()
with pyroute2.NetNS(consts.AMPHORA_NAMESPACE) as netns:
for interface in netns.get_links():
interface_name = None
for item in interface['attrs']:
if item[0] == 'IFLA_IFNAME' and not item[1].startswith('eth'):
break
elif item[0] == 'IFLA_IFNAME':
interface_name = item[1]
if item[0] == 'IFLA_STATS64':
networks[interface_name] = dict(
network_tx=item[1]['tx_bytes'],
network_rx=item[1]['rx_bytes'])
return networks
with pyroute2.NetNS(consts.AMPHORA_NAMESPACE) as netns:
for addr in netns.get_addr():
# Save the interface index as IPv6 records don't list a
# textual interface
interface_idx = addr['index']
# Save the address family (IPv4/IPv6) for use normalizing
# the IP address for comparison
interface_af = addr['family']
# Search through the attributes of each address record
for attr in addr['attrs']:
# Look for the attribute name/value pair for the address
if attr[0] == 'IFA_ADDRESS':
# Compare the normalized address with the address we
# we are looking for. Since we have matched the name
# above, attr[1] is the address value
if normalized_addr == socket.inet_ntop(
interface_af,
socket.inet_pton(interface_af, attr[1])):
# Lookup the matching interface name by
# getting the interface with the index we found
# in the above address search
lookup_int = netns.get_links(interface_idx)
# Search through the attributes of the matching
# interface record
for int_attr in lookup_int[0]['attrs']:
# Look for the attribute name/value pair
# that includes the interface name
if int_attr[0] == 'IFLA_IFNAME':
# Return the response with the matching
# interface name that is in int_attr[1]
# for the matching interface attribute
# name
return flask.make_response(
flask.jsonify(
dict(message='OK',
interface=int_attr[1])), 200)
def get_interface(ip_addr):
try:
ip_version = ipaddress.ip_address(six.text_type(ip_addr)).version
except Exception:
return flask.make_response(
flask.jsonify(dict(message="Invalid IP address")), 400)
if ip_version == 4:
address_format = netifaces.AF_INET
elif ip_version == 6:
address_format = netifaces.AF_INET6
else:
return flask.make_response(
flask.jsonify(dict(message="Bad IP address version")), 400)
# We need to normalize the address as IPv6 has multiple representations
# fe80:0000:0000:0000:f816:3eff:fef2:2058 == fe80::f816:3eff:fef2:2058
normalized_addr = socket.inet_ntop(address_format,
socket.inet_pton(address_format,
ip_addr))
with pyroute2.NetNS(consts.AMPHORA_NAMESPACE) as netns:
for addr in netns.get_addr():
# Save the interface index as IPv6 records don't list a
# textual interface
interface_idx = addr['index']
# Save the address family (IPv4/IPv6) for use normalizing
# the IP address for comparison
interface_af = addr['family']
# Search through the attributes of each address record
for attr in addr['attrs']:
# Look for the attribute name/value pair for the address
if attr[0] == 'IFA_ADDRESS':
# Compare the normalized address with the address we
# we are looking for. Since we have matched the name
# above, attr[1] is the address value
if normalized_addr == socket.inet_ntop(
interface_af,
socket.inet_pton(interface_af, attr[1])):
# Lookup the matching interface name by
# getting the interface with the index we found
# in the above address search
lookup_int = netns.get_links(interface_idx)
# Search through the attributes of the matching
# interface record
for int_attr in lookup_int[0]['attrs']:
# Look for the attribute name/value pair
# that includes the interface name
if int_attr[0] == 'IFLA_IFNAME':
# Return the response with the matching
# interface name that is in int_attr[1]
# for the matching interface attribute
# name
return flask.make_response(
flask.jsonify(
dict(message='OK',
interface=int_attr[1])), 200)
return flask.make_response(
flask.jsonify(dict(message="Error interface not found "
"for IP address")), 404)
flask.jsonify(dict(message="Error interface not found "
"for IP address")), 404)

7
octavia/amphorae/backends/agent/api_server/listener.py
View File

@ -26,6 +26,7 @@ import jinja2
import six
from werkzeug import exceptions
from octavia.amphorae.backends.agent.api_server import osutils
from octavia.amphorae.backends.agent.api_server import util
from octavia.amphorae.backends.utils import haproxy_query as query
from octavia.common import constants as consts
@ -74,6 +75,9 @@ class Wrapped(object):
class Listener(object):
def __init__(self):
self._osutils = osutils.BaseOS.get_os_util()
def get_haproxy_config(self, listener_id):
"""Gets the haproxy config
@ -171,7 +175,8 @@ class Listener(object):
respawn_count=util.CONF.haproxy_amphora.respawn_count,
respawn_interval=(util.CONF.haproxy_amphora.
respawn_interval),
amphora_nsname=consts.AMPHORA_NAMESPACE
amphora_nsname=consts.AMPHORA_NAMESPACE,
HasIFUPAll=self._osutils.has_ifup_all()
)
text_file.write(text)

444
octavia/amphorae/backends/agent/api_server/osutils.py Normal file
View File

@ -0,0 +1,444 @@
# Copyright 2017 Red Hat, Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import logging
import os
import platform
import shutil
import stat
import subprocess
import flask
import ipaddress
import jinja2
from oslo_config import cfg
import six
from werkzeug import exceptions
from octavia.common import constants as consts
from octavia.common import exceptions as octavia_exceptions
from octavia.common import utils
from octavia.i18n import _LE
CONF = cfg.CONF
LOG = logging.getLogger(__name__)
j2_env = jinja2.Environment(autoescape=True, loader=jinja2.FileSystemLoader(
os.path.dirname(os.path.realpath(__file__)) + consts.AGENT_API_TEMPLATES))
class BaseOS(object):
def __init__(self, os_name):
self.os_name = os_name
@classmethod
def get_os_util(cls):
os_name = platform.linux_distribution(full_distribution_name=False)[0]
for subclass in BaseOS.__subclasses__():
if subclass.is_os_name(os_name):
return subclass(os_name)
raise octavia_exceptions.InvalidAmphoraOperatingSystem(os_name=os_name)
def get_network_interface_file(self, interface):
if CONF.amphora_agent.agent_server_network_file:
return CONF.amphora_agent.agent_server_network_file
if CONF.amphora_agent.agent_server_network_dir:
return os.path.join(CONF.amphora_agent.agent_server_network_dir,
interface)
network_dir = consts.UBUNTU_AMP_NET_DIR_TEMPLATE.format(
netns=consts.AMPHORA_NAMESPACE)
return os.path.join(network_dir, interface)
def create_netns_dir(self, network_dir, netns_network_dir, ignore=None):
# We need to setup the netns network directory so that the ifup
# commands used here and in the startup scripts "sees" the right
# interfaces and scripts.
os.makedirs('/etc/netns/' + consts.AMPHORA_NAMESPACE)
shutil.copytree(
network_dir,
'/etc/netns/{netns}/{net_dir}'.format(
netns=consts.AMPHORA_NAMESPACE,
net_dir=netns_network_dir),
symlinks=True,
ignore=ignore)
def write_vip_interface_file(self, interface_file_path,
primary_interface, vip, ip, broadcast,
netmask, gateway, mtu, vrrp_ip, vrrp_version,
render_host_routes, template_vip):
# write interface file
mode = stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IROTH
# If we are using a consolidated interfaces file, just append
# otherwise clear the per interface file as we are rewriting it
# TODO(johnsom): We need a way to clean out old interfaces records
if CONF.amphora_agent.agent_server_network_file:
flags = os.O_WRONLY | os.O_CREAT | os.O_APPEND
else:
flags = os.O_WRONLY | os.O_CREAT | os.O_TRUNC
with os.fdopen(os.open(interface_file_path, flags, mode),
'w') as text_file:
text = template_vip.render(
interface=primary_interface,
vip=vip,
vip_ipv6=ip.version is 6,
prefix=utils.netmask_to_prefix(netmask),
broadcast=broadcast,
netmask=netmask,
gateway=gateway,
mtu=mtu,
vrrp_ip=vrrp_ip,
vrrp_ipv6=vrrp_version is 6,
host_routes=render_host_routes,
)
text_file.write(text)
def write_port_interface_file(self, netns_interface, fixed_ips, mtu,
interface_file_path, template_port):
# write interface file
# If we are using a consolidated interfaces file, just append
# otherwise clear the per interface file as we are rewriting it
# TODO(johnsom): We need a way to clean out old interfaces records
if CONF.amphora_agent.agent_server_network_file:
flags = os.O_WRONLY | os.O_CREAT | os.O_APPEND
else:
flags = os.O_WRONLY | os.O_CREAT | os.O_TRUNC
# mode 00644
mode = stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IROTH
with os.fdopen(os.open(interface_file_path, flags, mode),
'w') as text_file:
text = self._generate_network_file_text(netns_interface, fixed_ips,
mtu, template_port)
text_file.write(text)
def _generate_network_file_text(self, netns_interface, fixed_ips, mtu,
template_port):
text = ''
if fixed_ips is None:
text = template_port.render(interface=netns_interface)
else:
for index, fixed_ip in enumerate(fixed_ips, -1):
if index == -1:
netns_ip_interface = netns_interface
else:
netns_ip_interface = "{int}:{ip}".format(
int=netns_interface, ip=index)
try:
ip_addr = fixed_ip['ip_address']
cidr = fixed_ip['subnet_cidr']
ip = ipaddress.ip_address(
ip_addr if six.text_type == type(
ip_addr) else six.u(ip_addr))
network = ipaddress.ip_network(
cidr if six.text_type == type(
cidr) else six.u(cidr))
broadcast = network.broadcast_address.exploded
netmask = (network.prefixlen if ip.version is 6
else network.netmask.exploded)
host_routes = self.get_host_routes(fixed_ip)
except ValueError:
return flask.make_response(flask.jsonify(dict(
message="Invalid network IP")), 400)
new_text = template_port.render(interface=netns_ip_interface,
ipv6=ip.version is 6,
ip_address=ip.exploded,
broadcast=broadcast,
netmask=netmask,
mtu=mtu,
host_routes=host_routes)
text = '\n'.join([text, new_text])
return text
def get_host_routes(self, fixed_ip):
host_routes = []
for hr in fixed_ip.get('host_routes', []):
network = ipaddress.ip_network(
hr['destination'] if isinstance(
hr['destination'], six.text_type) else
six.u(hr['destination']))
host_routes.append({'network': network, 'gw': hr['nexthop']})
return host_routes
def _bring_if_up(self, interface, what):
# Note, we are not using pyroute2 for this as it is not /etc/netns
# aware.
cmd = ("ip netns exec {ns} ifup {params}".format(
ns=consts.AMPHORA_NAMESPACE, params=interface))
try:
subprocess.check_output(cmd.split(), stderr=subprocess.STDOUT)
except subprocess.CalledProcessError as e:
LOG.error(_LE('Failed to if up {0} due to '
'error: {1}').format(interface, str(e)))
raise exceptions.HTTPException(
response=flask.make_response(flask.jsonify(dict(
message='Error plugging {0}'.format(what),
details=e.output)), 500))
def _bring_if_down(self, interface):
# Note, we are not using pyroute2 for this as it is not /etc/netns
# aware.
cmd = ("ip netns exec {ns} ifdown {params}".format(
ns=consts.AMPHORA_NAMESPACE, params=interface))
try:
subprocess.check_output(cmd.split(), stderr=subprocess.STDOUT)
except subprocess.CalledProcessError:
pass
def bring_interfaces_up(self, ip, primary_interface, secondary_interface):
self._bring_if_down(primary_interface)
self._bring_if_down(secondary_interface)
self._bring_if_up(primary_interface, 'VIP')
self._bring_if_up(secondary_interface, 'VIP')
def has_ifup_all(self):
return True
class Ubuntu(BaseOS):
ETH_X_PORT_CONF = 'plug_port_ethX.conf.j2'
ETH_X_VIP_CONF = 'plug_vip_ethX.conf.j2'
@classmethod
def is_os_name(cls, os_name):
return os_name in ['Ubuntu']
def cmd_get_version_of_installed_package(self, package_name):
return "dpkg --status {name}".format(name=package_name)
def get_network_interface_file(self, interface):
if CONF.amphora_agent.agent_server_network_file:
return CONF.amphora_agent.agent_server_network_file
if CONF.amphora_agent.agent_server_network_dir:
return os.path.join(CONF.amphora_agent.agent_server_network_dir,
interface + '.cfg')
network_dir = consts.UBUNTU_AMP_NET_DIR_TEMPLATE.format(
netns=consts.AMPHORA_NAMESPACE)
return os.path.join(network_dir, interface + '.cfg')
def get_network_path(self):
return '/etc/network'
def get_netns_network_dir(self):
network_dir = self.get_network_path()
return os.path.basename(network_dir)
def create_netns_dir(
self, network_dir=None, netns_network_dir=None, ignore=None):
if not netns_network_dir:
netns_network_dir = self.get_netns_network_dir()
if not network_dir:
network_dir = self.get_network_path()
if not ignore:
ignore = shutil.ignore_patterns('eth0*', 'openssh*')
super(Ubuntu, self).create_netns_dir(
network_dir, netns_network_dir, ignore)
def write_interfaces_file(self):
name = '/etc/netns/{}/network/interfaces'.format(
consts.AMPHORA_NAMESPACE)
flags = os.O_WRONLY | os.O_CREAT | os.O_TRUNC
# mode 00644
mode = stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IROTH
with os.fdopen(os.open(name, flags, mode), 'w') as int_file:
int_file.write('auto lo\n')
int_file.write('iface lo inet loopback\n')
if not CONF.amphora_agent.agent_server_network_file:
int_file.write('source /etc/netns/{}/network/'
'interfaces.d/*.cfg\n'.format(
consts.AMPHORA_NAMESPACE))
def write_vip_interface_file(self, interface_file_path,
primary_interface, vip, ip, broadcast,
netmask, gateway, mtu, vrrp_ip, vrrp_version,
render_host_routes, template_vip=None):
if not template_vip:
template_vip = j2_env.get_template(self.ETH_X_VIP_CONF)
super(Ubuntu, self).write_vip_interface_file(
interface_file_path, primary_interface, vip, ip, broadcast,
netmask, gateway, mtu, vrrp_ip, vrrp_version, render_host_routes,
template_vip)
def write_port_interface_file(self, netns_interface, fixed_ips, mtu,
interface_file_path=None,
template_port=None):
if not interface_file_path:
interface_file_path = self.get_network_interface_file(
netns_interface)
if not template_port:
template_port = j2_env.get_template(self.ETH_X_PORT_CONF)
super(Ubuntu, self).write_port_interface_file(
netns_interface, fixed_ips, mtu, interface_file_path,
template_port)
def has_ifup_all(self):
return True
class RH(BaseOS):
ETH_X_PORT_CONF = 'rh_plug_port_ethX.conf.j2'
ETH_X_VIP_CONF = 'rh_plug_vip_ethX.conf.j2'
ETH_X_ALIAS_VIP_CONF = 'rh_plug_vip_ethX_alias.conf.j2'
ROUTE_ETH_X_CONF = 'rh_route_ethX.conf.j2'
@classmethod
def is_os_name(cls, os_name):
return os_name in ['fedora', 'redhat', 'centos']
def cmd_get_version_of_installed_package(self, package_name):
return "rpm -qi {name}".format(name=package_name)
def get_network_interface_file(self, interface):
if CONF.amphora_agent.agent_server_network_file:
return CONF.amphora_agent.agent_server_network_file
if CONF.amphora_agent.agent_server_network_dir:
return os.path.join(CONF.amphora_agent.agent_server_network_dir,
'ifcfg-' + interface)
network_dir = consts.RH_AMP_NET_DIR_TEMPLATE.format(
netns=consts.AMPHORA_NAMESPACE)
return os.path.join(network_dir, 'ifcfg-' + interface)
def get_alias_network_interface_file(self, interface):
return self.get_network_interface_file(interface + ':0')
def get_static_routes_interface_file(self, interface):
return self.get_network_interface_file('route-' + interface)
def get_network_path(self):
return '/etc/sysconfig/network-scripts'
def get_netns_network_dir(self):
network_full_path = self.get_network_path()
network_basename = os.path.basename(network_full_path)
network_dirname = os.path.dirname(network_full_path)
network_prefixdir = os.path.basename(network_dirname)
return os.path.join(network_prefixdir, network_basename)
def create_netns_dir(
self, network_dir=None, netns_network_dir=None, ignore=None):
if not netns_network_dir:
netns_network_dir = self.get_netns_network_dir()
if not network_dir:
network_dir = self.get_network_path()
if not ignore:
ignore = shutil.ignore_patterns('ifcfg-eth0*', 'ifcfg-lo*')
super(RH, self).create_netns_dir(
network_dir, netns_network_dir, ignore)
# Copy /etc/sysconfig/network file
src = '/etc/sysconfig/network'
dst = '/etc/netns/{netns}/sysconfig'.format(
netns=consts.AMPHORA_NAMESPACE)
shutil.copy2(src, dst)
def write_interfaces_file(self):
# No interfaces file in RH based flavors
return
def write_vip_interface_file(self, interface_file_path,
primary_interface, vip, ip, broadcast,
netmask, gateway, mtu, vrrp_ip, vrrp_version,
render_host_routes, template_vip=None):
if not template_vip:
template_vip = j2_env.get_template(self.ETH_X_VIP_CONF)
super(RH, self).write_vip_interface_file(
interface_file_path, primary_interface, vip, ip, broadcast,
netmask, gateway, mtu, vrrp_ip, vrrp_version, render_host_routes,
template_vip)
if ip.version == 4:
# Create an IPv4 alias interface, needed in RH based flavors
alias_interface_file_path = self.get_alias_network_interface_file(
primary_interface)
template_vip_alias = j2_env.get_template(self.ETH_X_ALIAS_VIP_CONF)
super(RH, self).write_vip_interface_file(
alias_interface_file_path, primary_interface, vip, ip,
broadcast, netmask, gateway, mtu, vrrp_ip, vrrp_version,
render_host_routes, template_vip_alias)
if render_host_routes:
routes_interface_file_path = (
self.get_static_routes_interface_file(primary_interface))
template_routes = j2_env.get_template(self.ROUTE_ETH_X_CONF)
self.write_static_routes_interface_file(
routes_interface_file_path, primary_interface,
render_host_routes, template_routes)
def write_static_routes_interface_file(self, interface_file_path,
interface, host_routes,
template_routes):
# write static routes interface file
mode = stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IROTH
# TODO(johnsom): We need a way to clean out old interfaces records
if CONF.amphora_agent.agent_server_network_file:
flags = os.O_WRONLY | os.O_CREAT | os.O_APPEND
else:
flags = os.O_WRONLY | os.O_CREAT | os.O_TRUNC
with os.fdopen(os.open(interface_file_path, flags, mode),
'w') as text_file:
text = template_routes.render(
interface=interface,
host_routes=host_routes,
)
text_file.write(text)
def write_port_interface_file(self, netns_interface, fixed_ips, mtu,
interface_file_path=None,
template_port=None):
if not interface_file_path:
interface_file_path = self.get_network_interface_file(
netns_interface)
if not template_port:
template_port = j2_env.get_template(self.ETH_X_PORT_CONF)
super(RH, self).write_port_interface_file(
netns_interface, fixed_ips, mtu, interface_file_path,
template_port)
if fixed_ips:
host_routes = []
for fixed_ip in fixed_ips:
host_routes.extend(self.get_host_routes(fixed_ip))
routes_interface_file_path = (
self.get_static_routes_interface_file(netns_interface))
template_routes = j2_env.get_template(self.ROUTE_ETH_X_CONF)
self.write_static_routes_interface_file(
routes_interface_file_path, netns_interface,
host_routes, template_routes)
def bring_interfaces_up(self, ip, primary_interface, secondary_interface):
if ip.version == 4:
super(RH, self).bring_interfaces_up(
ip, primary_interface, secondary_interface)
else:
# Secondary interface is not present in IPv6 configuration
self._bring_if_down(primary_interface)
self._bring_if_up(primary_interface, 'VIP')
def has_ifup_all(self):
return False

176
octavia/amphorae/backends/agent/api_server/plug.py
View File

@ -15,7 +15,6 @@
import logging
import os
import shutil
import socket
import stat
import subprocess
@ -29,9 +28,8 @@ import pyroute2
import six
from werkzeug import exceptions
from octavia.amphorae.backends.agent.api_server import util
from octavia.common import constants as consts
from octavia.i18n import _LE, _LI
from octavia.i18n import _LI
CONF = cfg.CONF
@ -48,6 +46,8 @@ template_vip = j2_env.get_template(ETH_X_VIP_CONF)
class Plug(object):
def __init__(self, osutils):
self._osutils = osutils
def plug_vip(self, vip, subnet_cidr, gateway,
mac_address, mtu=None, vrrp_ip=None, host_routes=None):
@ -97,57 +97,24 @@ class Plug(object):
secondary_interface = "{interface}:0".format(
interface=primary_interface)
# We need to setup the netns network directory so that the ifup
# commands used here and in the startup scripts "sees" the right
# interfaces and scripts.
interface_file_path = util.get_network_interface_file(
interface_file_path = self._osutils.get_network_interface_file(
primary_interface)
os.makedirs('/etc/netns/' + consts.AMPHORA_NAMESPACE)
shutil.copytree(
'/etc/network',
'/etc/netns/{}/network'.format(consts.AMPHORA_NAMESPACE),
symlinks=True,
ignore=shutil.ignore_patterns('eth0*', 'openssh*'))
name = '/etc/netns/{}/network/interfaces'.format(
consts.AMPHORA_NAMESPACE)
flags = os.O_WRONLY | os.O_CREAT | os.O_TRUNC
# mode 00644
mode = stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IROTH
with os.fdopen(os.open(name, flags, mode), 'w') as int_file:
int_file.write('auto lo\n')
int_file.write('iface lo inet loopback\n')
if not CONF.amphora_agent.agent_server_network_file:
int_file.write('source /etc/netns/{}/network/'
'interfaces.d/*.cfg\n'.format(
consts.AMPHORA_NAMESPACE))
# write interface file
self._osutils.create_netns_dir()
mode = stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IROTH
# If we are using a consolidated interfaces file, just append
# otherwise clear the per interface file as we are rewriting it
# TODO(johnsom): We need a way to clean out old interfaces records
if CONF.amphora_agent.agent_server_network_file:
flags = os.O_WRONLY | os.O_CREAT | os.O_APPEND
else:
flags = os.O_WRONLY | os.O_CREAT | os.O_TRUNC
with os.fdopen(os.open(interface_file_path, flags, mode),
'w') as text_file:
text = template_vip.render(
interface=primary_interface,
vip=vip,
vip_ipv6=ip.version is 6,
broadcast=broadcast,
netmask=netmask,
gateway=gateway,
mtu=mtu,
vrrp_ip=vrrp_ip,
vrrp_ipv6=vrrp_version is 6,
host_routes=render_host_routes,
)
text_file.write(text)
self._osutils.write_interfaces_file()
self._osutils.write_vip_interface_file(
interface_file_path=interface_file_path,
primary_interface=primary_interface,
vip=vip,
ip=ip,
broadcast=broadcast,
netmask=netmask,
gateway=gateway,
mtu=mtu,
vrrp_ip=vrrp_ip,
vrrp_version=vrrp_version,
render_host_routes=render_host_routes)
# Update the list of interfaces to add to the namespace
# This is used in the amphora reboot case to re-establish the namespace
@ -172,60 +139,14 @@ class Plug(object):
IFLA_IFNAME=primary_interface)
# bring interfaces up
self._bring_if_down(primary_interface)
self._bring_if_down(secondary_interface)
self._bring_if_up(primary_interface, 'VIP')
self._bring_if_up(secondary_interface, 'VIP')
self._osutils.bring_interfaces_up(
ip, primary_interface, secondary_interface)
return flask.make_response(flask.jsonify(dict(
message="OK",
details="VIP {vip} plugged on interface {interface}".format(
vip=vip, interface=primary_interface))), 202)
def _generate_network_file_text(self, netns_interface, fixed_ips, mtu):
text = ''
if fixed_ips is None:
text = template_port.render(interface=netns_interface)
else:
for index, fixed_ip in enumerate(fixed_ips, -1):
if index == -1:
netns_ip_interface = netns_interface
else:
netns_ip_interface = "{int}:{ip}".format(
int=netns_interface, ip=index)
try:
ip_addr = fixed_ip['ip_address']
cidr = fixed_ip['subnet_cidr']
ip = ipaddress.ip_address(
ip_addr if six.text_type == type(
ip_addr) else six.u(ip_addr))
network = ipaddress.ip_network(
cidr if six.text_type == type(
cidr) else six.u(cidr))
broadcast = network.broadcast_address.exploded
netmask = (network.prefixlen if ip.version is 6
else network.netmask.exploded)
host_routes = []
for hr in fixed_ip.get('host_routes', []):
network = ipaddress.ip_network(
hr['destination'] if isinstance(
hr['destination'], six.text_type) else
six.u(hr['destination']))
host_routes.append({'network': network,
'gw': hr['nexthop']})
except ValueError:
return flask.make_response(flask.jsonify(dict(
message="Invalid network IP")), 400)
new_text = template_port.render(interface=netns_ip_interface,
ipv6=ip.version is 6,
ip_address=ip.exploded,
broadcast=broadcast,
netmask=netmask,
mtu=mtu,
host_routes=host_routes)
text = '\n'.join([text, new_text])
return text
def _check_ip_addresses(self, fixed_ips):
if fixed_ips:
for ip in fixed_ips:
@ -267,27 +188,13 @@ class Plug(object):
'namespace {2}').format(default_netns_interface,
netns_interface,
consts.AMPHORA_NAMESPACE))
interface_file_path = util.get_network_interface_file(netns_interface)
# write interface file
# If we are using a consolidated interfaces file, just append
# otherwise clear the per interface file as we are rewriting it
# TODO(johnsom): We need a way to clean out old interfaces records
if CONF.amphora_agent.agent_server_network_file:
flags = os.O_WRONLY | os.O_CREAT | os.O_APPEND
else:
flags = os.O_WRONLY | os.O_CREAT | os.O_TRUNC
# mode 00644
mode = stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IROTH
with os.fdopen(os.open(interface_file_path, flags, mode),
'w') as text_file:
text = self._generate_network_file_text(netns_interface,
fixed_ips,
mtu)
text_file.write(text)
interface_file_path = self._osutils.get_network_interface_file(
netns_interface)
self._osutils.write_port_interface_file(
netns_interface=netns_interface,
fixed_ips=fixed_ips,
mtu=mtu,
interface_file_path=interface_file_path)
# Update the list of interfaces to add to the namespace
self._update_plugged_interfaces_file(netns_interface, mac_address)
@ -299,8 +206,8 @@ class Plug(object):
net_ns_fd=consts.AMPHORA_NAMESPACE,
IFLA_IFNAME=netns_interface)
self._bring_if_down(netns_interface)
self._bring_if_up(netns_interface, 'network')
self._osutils._bring_if_down(netns_interface)
self._osutils._bring_if_up(netns_interface, 'network')
return flask.make_response(flask.jsonify(dict(
message="OK",
@ -318,31 +225,6 @@ class Plug(object):
response=flask.make_response(flask.jsonify(dict(
details="No suitable network interface found")), 404))
def _bring_if_up(self, interface, what):
# Note, we are not using pyroute2 for this as it is not /etc/netns
# aware.
cmd = ("ip netns exec {ns} ifup {params}".format(
ns=consts.AMPHORA_NAMESPACE, params=interface))
try:
subprocess.check_output(cmd.split(), stderr=subprocess.STDOUT)
except subprocess.CalledProcessError as e:
LOG.error(_LE('Failed to if up {0} due to '
'error: {1}').format(interface, str(e)))
raise exceptions.HTTPException(
response=flask.make_response(flask.jsonify(dict(
message='Error plugging {0}'.format(what),
details=e.output)), 500))
def _bring_if_down(self, interface):
# Note, we are not using pyroute2 for this as it is not /etc/netns
# aware.
cmd = ("ip netns exec {ns} ifdown {params}".format(
ns=consts.AMPHORA_NAMESPACE, params=interface))
try:
subprocess.check_output(cmd.split(), stderr=subprocess.STDOUT)
except subprocess.CalledProcessError:
pass
def _update_plugged_interfaces_file(self, interface, mac_address):
# write interfaces to plugged_interfaces file and prevent duplicates
plug_inf_file = consts.PLUGGED_INTERFACES

11
octavia/amphorae/backends/agent/api_server/server.py
View File

@ -22,6 +22,7 @@ from octavia.amphorae.backends.agent.api_server import amphora_info
from octavia.amphorae.backends.agent.api_server import certificate_update
from octavia.amphorae.backends.agent.api_server import keepalived
from octavia.amphorae.backends.agent.api_server import listener
from octavia.amphorae.backends.agent.api_server import osutils
from octavia.amphorae.backends.agent.api_server import plug
PATH_PREFIX = '/' + api_server.VERSION
@ -43,9 +44,11 @@ def register_app_error_handler(app):
class Server(object):
def __init__(self):
self.app = flask.Flask(__name__)
self._osutils = osutils.BaseOS.get_os_util()
self._keepalived = keepalived.Keepalived()
self._listener = listener.Listener()
self._plug = plug.Plug()
self._plug = plug.Plug(self._osutils)
self._amphora_info = amphora_info.AmphoraInfo(self._osutils)
register_app_error_handler(self.app)
@ -119,10 +122,10 @@ class Server(object):
return self._listener.delete_listener(listener_id)
def get_details(self):
return amphora_info.compile_amphora_details()
return self._amphora_info.compile_amphora_details()
def get_info(self):
return amphora_info.compile_amphora_info()
return self._amphora_info.compile_amphora_info()
def get_all_listeners_status(self):
return self._listener.get_all_listeners_status()
@ -179,4 +182,4 @@ class Server(object):
return self._keepalived.manager_keepalived_service(action)
def get_interface(self, ip_addr):
return amphora_info.get_interface(ip_addr)
return self._amphora_info.get_interface(ip_addr)

2
octavia/amphorae/backends/agent/api_server/templates/plug_vip_ethX.conf.j2
View File

@ -16,7 +16,6 @@
#}
# Generated by Octavia agent
auto {{ interface }} {{ interface }}:0
{%- if vrrp_ip %}
iface {{ interface }} inet{{ '6' if vrrp_ipv6 }} static
address {{ vrrp_ip }}
@ -31,6 +30,7 @@ up route add -net {{ hr.network }} gw {{ hr.gw }} dev {{ interface }}
down route del -net {{ hr.network }} gw {{ hr.gw }} dev {{ interface }}
{%- endfor %}
{%- else %}
iface {{ interface }} inet{{ '6' if vip_ipv6 }} {{ 'auto' if vip_ipv6 else 'dhcp' }}
{%- endif %}

47
octavia/amphorae/backends/agent/api_server/templates/rh_plug_port_ethX.conf.j2 Normal file
View File

@ -0,0 +1,47 @@
{#
# Copyright 2017 Red Hat, Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#}
# Generated by Octavia agent
NM_CONTROLLED="no"
DEVICE="{{ interface }}"
ONBOOT="yes"
TYPE="Ethernet"
USERCTL="yes"
{%- if ipv6 %}
IPV6INIT="yes"
{%- if mtu %}
IPV6_MTU="{{ mtu }}"
{%- endif %}
{%- if ip_address %}
IPV6_AUTOCONF="no"
IPV6ADDR="{{ ip_address }}"
{%- else %}
IPV6_AUTOCONF="yes"
{%- endif %}
{%- else %}
IPV6INIT="no"
{%- if mtu %}
MTU="{{ mtu }}"
{%- endif %}
{%- if ip_address %}
BOOTPROTO="static"
IPADDR="{{ ip_address }}"
NETMASK="{{ netmask }}"
{%- else %}
BOOTPROTO="dhcp"
PERSISTENT_DHCLIENT="1"
{%- endif %}
{%- endif %}

54
octavia/amphorae/backends/agent/api_server/templates/rh_plug_vip_ethX.conf.j2 Normal file
View File

@ -0,0 +1,54 @@
{#
# Copyright 2017 Red Hat, Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#}
# Generated by Octavia agent
NM_CONTROLLED="no"
DEVICE="{{ interface }}"
ONBOOT="yes"
TYPE="Ethernet"
USERCTL="yes"
{%- if vrrp_ip %}
{%- if vrrp_ipv6 %}
IPV6INIT="yes"
IPV6_DEFROUTE="yes"
IPV6_AUTOCONF="no"
IPV6ADDR="{{ vrrp_ip }}/{{ prefix }}"
IPV6_DEFAULTGW="{{ gateway }}"
{%- if mtu %}
IPV6_MTU="{{ mtu }}"
{%- endif %}
{%- else %} {# not vrrp_ipv6 #}
BOOTPROTO="static"
IPADDR="{{ vrrp_ip }}"
NETMASK="{{ netmask }}"
GATEWAY="{{ gateway }}"
MTU="{{ mtu }}"
{%- endif %} {# end if vrrp_ipv6 #}
{%- else %} {# not vrrp_ip #}
{%- if vip_ipv6 %}
IPV6INIT="yes"
IPV6_DEFROUTE="yes"
IPV6_AUTOCONF="yes"
{%- else %}
BOOTPROTO="dhcp"
PERSISTENT_DHCLIENT="1"
{%- endif %} {# end if vip_ipv6 #}
{%- endif %} {# end if vrrp_ip #}
{%- if vip_ipv6 %}
IPV6ADDR_SECONDARIES="{{ vip }}/{{ prefix }}"
{%- endif %}

29
octavia/amphorae/backends/agent/api_server/templates/rh_plug_vip_ethX_alias.conf.j2 Normal file
View File

@ -0,0 +1,29 @@
{#
# Copyright 2017 Red Hat, Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software