openstack
/
kuryr-kubernetes
Code Issues Proposed changes

Add option to set listener timeouts for lb created by Kuryr 

The timeout-client-data and timeout-member-data configurations
for Octavia listeners default to 50 seconds for load balancers
created by Kuryr. This patch allows the creation and modification
of load balancers handled by Kuryr with different timeouts values.

Implements: blueprint configure-lb-listeners-timeout
Change-Id: I99016001c2263023d1fa2637d7b5aeb23b3b2d9d
This commit is contained in:
Tabitha Fasoyin 2021-02-24 14:31:07 +00:00
parent 10ea869858
commit a15617cdf3
14 changed files with 186 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
devstack/local.conf.ovn.sample
View File

@ -65,6 +65,9 @@ VAR_RUN_PATH=/var/run
# KURYR_K8S_OCTAVIA_MEMBER_MODE=L2
# KURYR_ENFORCE_SG_RULES=False
# KURYR_LB_ALGORITHM=SOURCE_IP_PORT
# Uncomment to modify listener client and member inactivity timeout.
# KURYR_TIMEOUT_CLIENT_DATA=50000
# KURYR_TIMEOUT_MEMBER_DATA=50000
# Octavia LBaaSv2

6
devstack/local.conf.sample
View File

@ -41,9 +41,15 @@ enable_service neutron-tag-ports-during-bulk-creation
# VAR_RUN_PATH=/var/run
# OCTAVIA
# =======
# Uncomment it to use L2 communication between loadbalancer and member pods
# KURYR_K8S_OCTAVIA_MEMBER_MODE=L2
# Uncomment to change Octavia loadbalancer listener client and member
# inactivity timeout from 50000ms.
# KURYR_TIMEOUT_CLIENT_DATA=50000
# KURYR_TIMEOUT_MEMBER_DATA=50000
# Octavia LBaaSv2
LIBS_FROM_GIT+=python-octaviaclient
enable_plugin octavia https://opendev.org/openstack/octavia

2
devstack/plugin.sh
View File

@ -485,6 +485,8 @@ function configure_neutron_defaults {
iniset "$KURYR_CONFIG" octavia_defaults member_mode "$KURYR_K8S_OCTAVIA_MEMBER_MODE"
iniset "$KURYR_CONFIG" octavia_defaults enforce_sg_rules "$KURYR_ENFORCE_SG_RULES"
iniset "$KURYR_CONFIG" octavia_defaults lb_algorithm "$KURYR_LB_ALGORITHM"
iniset "$KURYR_CONFIG" octavia_defaults timeout_client_data "$KURYR_TIMEOUT_CLIENT_DATA"
iniset "$KURYR_CONFIG" octavia_defaults timeout_member_data "$KURYR_TIMEOUT_MEMBER_DATA"
# Octavia takes a very long time to start the LB in the gate. We need
# to tweak the timeout for the LB creation. Let's be generous and give
# it up to 20 minutes.

2
devstack/settings
View File

@ -60,6 +60,8 @@ OPENSHIFT_CNI_BINARY_URL=${OPENSHIFT_CNI_BINARY_URL:-https://github.com/containe
KURYR_K8S_OCTAVIA_MEMBER_MODE=${KURYR_K8S_OCTAVIA_MEMBER_MODE:-L3}
KURYR_ENFORCE_SG_RULES=${KURYR_ENFORCE_SG_RULES:-True}
KURYR_LB_ALGORITHM=${KURYR_LB_ALGORITHM:-ROUND_ROBIN}
KURYR_TIMEOUT_CLIENT_DATA=${KURYR_TIMEOUT_CLIENT_DATA:-0}
KURYR_TIMEOUT_MEMBER_DATA=${KURYR_TIMEOUT_MEMBER_DATA:-0}
# Kuryr_ovs_baremetal
KURYR_CONFIGURE_BAREMETAL_KUBELET_IFACE=${KURYR_CONFIGURE_BAREMETAL_KUBELET_IFACE:-True}

1
doc/source/installation/index.rst
View File

@ -47,3 +47,4 @@ This section describes how you can install and configure kuryr-kubernetes
testing_udp_services
testing_sriov_functional
testing_sctp_services
listener_timeouts

63
doc/source/installation/listener_timeouts.rst Normal file
View File

@ -0,0 +1,63 @@
=====================================================
How to configure Listener timeouts for Load Balancers
=====================================================
By default, Kuryr uses the default Octavia timeout-client-data and
timeout-member-data values when creating or modifying loadbalancers.
To change the timeout values used in creating or modifying a particular
service:
#. Set the new timeout values for openstack.org/kuryr-timeout-client-data and
openstack.org/kuryr-timeout-member-data in the service annotation as seen
in the service manifest below. This specification sets the timeout-client-
data and the timeout-member-data to '70000' and '75000' respectively.
.. code-block:: yaml
apiVersion: v1
kind: Service
metadata:
name: kuryr-demo
annotations:
openstack.org/kuryr-timeout-client-data: '70000'
openstack.org/kuryr-timeout-member-data: '75000'
spec:
selector:
app: server
ports:
- protocol: TCP
port: 80
targetPort: 8080
Setting the timeouts via ConfigMap
----------------------------------
Alternatively, you can change the value of the timeout-client-data and/or the
timeout-member-data on the Kuryr ConfigMap. This option is ideal if the new
timeout values will be used for multiple loadbalancers. On DevStack deployment,
the Kuryr ConfigMap can be edited using:
.. code-block:: console
$ kubectl -n kube-system edit cm kuryr-config
The listener timeouts then needs to be changed at the ConfigMap:
.. code-block:: bash
[octavia_defaults]
timeout_member_data = 0
timeout_client_data = 0
Another option is to set the listener timeouts at the local.conf file.
.. code-block:: bash
#KURYR_TIMEOUT_CLIENT_DATA=0
#KURYR_TIMEOUT_MEMBER_DATA=0
.. note::
The listener timeouts values set via the ConfigMap or set at the local.conf
can be overridden by values set in the service annotations for a particular
service.

10
kubernetes_crds/kuryr_crds/kuryrloadbalancer.yaml
View File

@ -110,6 +110,10 @@ spec:
type: string
provider:
type: string
timeout_client_data:
type: integer
timeout_member_data:
type: integer
status:
type: object
properties:
@ -124,6 +128,8 @@ spec:
- port
- project_id
- protocol
- timeout_client_data
- timeout_member_data
properties:
id:
type: string
@ -137,6 +143,10 @@ spec:
type: string
protocol:
type: string
timeout_client_data:
type: integer
timeout_member_data:
type: integer
loadbalancer:
type: object
required:

10
kuryr_kubernetes/config.py
View File

@ -259,6 +259,16 @@ octavia_defaults = [
"to the pool members. The options are: ROUND_ROBIN, "
"LEAST_CONNECTIONS, SOURCE_IP and SOURCE_IP_PORT."),
default='ROUND_ROBIN'),
cfg.IntOpt('timeout_client_data',
help=_("Frontend client inactivity timeout in milliseconds. "
"When set to 0, the default timeout value set by "
"Octavia is used."),
default=0),
cfg.IntOpt('timeout_member_data',
help=_("Backend member inactivity timeout in milliseconds. "
"When set to 0, the default timeout value set by "
"Octavia is used."),
default=0),
]
cache_defaults = [

3
kuryr_kubernetes/constants.py
View File

@ -61,6 +61,9 @@ K8S_ANNOTATION_NET_CRD = K8S_ANNOTATION_PREFIX + '-net-crd'
K8S_ANNOTATION_NETPOLICY_CRD = K8S_ANNOTATION_PREFIX + '-netpolicy-crd'
K8S_ANNOTATION_POLICY = K8S_ANNOTATION_PREFIX + '-counter'
K8S_ANNOTATION_CLIENT_TIMEOUT = K8S_ANNOTATION_PREFIX + '-timeout-client-data'
K8S_ANNOTATION_MEMBER_TIMEOUT = K8S_ANNOTATION_PREFIX + '-timeout-member-data'
K8S_ANNOTATION_NPWG_PREFIX = 'k8s.v1.cni.cncf.io'
K8S_ANNOTATION_NPWG_NETWORK = K8S_ANNOTATION_NPWG_PREFIX + '/networks'
K8S_ANNOTATION_NPWG_CRD_SUBNET_ID = 'subnetId'

19
kuryr_kubernetes/controller/drivers/lbaasv2.py
View File

@ -357,7 +357,8 @@ class LBaaSv2Driver(base.LBaaSDriver):
'network-policy' not in rule.get('description'))
def ensure_listener(self, loadbalancer, protocol, port,
service_type='ClusterIP'):
service_type='ClusterIP', timeout_client_data=0,
timeout_member_data=0):
name = "%s:%s:%s" % (loadbalancer['name'], protocol, port)
listener = {
'name': name,
@ -366,6 +367,10 @@ class LBaaSv2Driver(base.LBaaSDriver):
'protocol': protocol,
'port': port
}
if timeout_client_data:
listener['timeout_client_data'] = timeout_client_data
if timeout_member_data:
listener['timeout_member_data'] = timeout_member_data
try:
result = self._ensure_provisioned(
loadbalancer, listener, self._create_listener,
@ -552,10 +557,22 @@ class LBaaSv2Driver(base.LBaaSDriver):
'protocol': listener['protocol'],
'protocol_port': listener['port'],
}
default_timeout_cli = CONF.octavia_defaults.timeout_client_data
default_timeout_mem = CONF.octavia_defaults.timeout_member_data
timeout_cli = listener['timeout_client_data'] if listener.get(
'timeout_client_data') else default_timeout_cli
timeout_mem = listener['timeout_member_data'] if listener.get(
'timeout_member_data') else default_timeout_mem
if timeout_cli:
request['timeout_client_data'] = timeout_cli
if timeout_mem:
request['timeout_member_data'] = timeout_mem
self.add_tags('listener', request)
lbaas = clients.get_loadbalancer_client()
response = lbaas.create_listener(**request)
listener['id'] = response.id
listener['timeout_client_data'] = response.timeout_client_data
listener['timeout_member_data'] = response.timeout_member_data
return listener
def _update_listener_acls(self, loadbalancer, listener_id, allowed_cidrs):

22
kuryr_kubernetes/controller/handlers/lbaas.py
View File

@ -158,6 +158,7 @@ class ServiceHandler(k8s_base.ResourceEventHandler):
kubernetes.post('{}/{}/kuryrloadbalancers'.format(
k_const.K8S_API_CRD_NAMESPACES, svc_namespace),
loadbalancer_crd)
LOG.debug("loadbalancer_crd: {}".format(loadbalancer_crd))
except k_exc.K8sConflict:
raise k_exc.ResourceNotReady(svc_name)
except k_exc.K8sNamespaceTerminating:
@ -182,6 +183,23 @@ class ServiceHandler(k8s_base.ResourceEventHandler):
LOG.exception('Error updating kuryrnet CRD %s', loadbalancer_crd)
raise
def _get_data_timeout_annotation(self, service):
try:
annotations = service['metadata']['annotations']
except KeyError:
return 0, 0
try:
timeout_cli = annotations[k_const.K8S_ANNOTATION_CLIENT_TIMEOUT]
data_timeout_cli = int(timeout_cli)
except KeyError:
data_timeout_cli = 0
try:
timeout_mem = annotations[k_const.K8S_ANNOTATION_MEMBER_TIMEOUT]
data_timeout_mem = int(timeout_mem)
except KeyError:
data_timeout_mem = 0
return data_timeout_cli, data_timeout_mem
def _build_kuryrloadbalancer_spec(self, service):
svc_ip = self._get_service_ip(service)
spec_lb_ip = service['spec'].get('loadBalancerIP')
@ -193,7 +211,6 @@ class ServiceHandler(k8s_base.ResourceEventHandler):
sg_ids = self._drv_sg.get_security_groups(service, project_id)
subnet_id = self._get_subnet_id(service, project_id, svc_ip)
spec_type = service['spec'].get('type')
spec = {
'ip': svc_ip,
'ports': ports,
@ -205,6 +222,9 @@ class ServiceHandler(k8s_base.ResourceEventHandler):
if spec_lb_ip is not None:
spec['lb_ip'] = spec_lb_ip
timeout_cli, timeout_mem = self._get_data_timeout_annotation(service)
spec['timeout_client_data'] = timeout_cli
spec['timeout_member_data'] = timeout_mem
return spec
def _has_lbaas_spec_changes(self, service, loadbalancer_crd):

6
kuryr_kubernetes/controller/handlers/loadbalancer.py
View File

@ -609,7 +609,11 @@ class KuryrLoadBalancerHandler(k8s_base.ResourceEventHandler):
loadbalancer=loadbalancer_crd['status'].get('loadbalancer'),
protocol=protocol,
port=port,
service_type=loadbalancer_crd['spec'].get('type'))
service_type=loadbalancer_crd['spec'].get('type'),
timeout_client_data=loadbalancer_crd['spec'].get(
'timeout_client_data'),
timeout_member_data=loadbalancer_crd['spec'].get(
'timeout_member_data'))
if listener is not None:
listeners = loadbalancer_crd['status'].get('listeners', [])
if listeners:

34
kuryr_kubernetes/tests/unit/controller/drivers/test_lbaasv2.py
View File

@ -195,8 +195,7 @@ class TestLBaaSv2Driver(test_base.TestCase):
# TODO(ivc): handle security groups
m_driver._ensure_provisioned.return_value = expected_resp
resp = cls.ensure_listener(m_driver, loadbalancer,
protocol, port)
resp = cls.ensure_listener(m_driver, loadbalancer, protocol, port)
m_driver._ensure_provisioned.assert_called_once_with(
loadbalancer, mock.ANY, m_driver._create_listener,
@ -561,6 +560,7 @@ class TestLBaaSv2Driver(test_base.TestCase):
'loadbalancer_id': '00EE9E11-91C2-41CF-8FD4-7970579E5C4C',
'protocol': 'TCP',
'port': 1234
}
listener_id = 'A57B7771-6050-4CA8-A63C-443493EC98AB'
@ -578,6 +578,36 @@ class TestLBaaSv2Driver(test_base.TestCase):
self.assertEqual(listener, ret)
self.assertEqual(listener_id, ret['id'])
def test_create_listener_with_different_timeouts(self):
cls = d_lbaasv2.LBaaSv2Driver
m_driver = mock.Mock(spec=d_lbaasv2.LBaaSv2Driver)
lbaas = self.useFixture(k_fix.MockLBaaSClient()).client
listener = {
'name': 'TEST_NAME',
'project_id': 'TEST_PROJECT',
'loadbalancer_id': '00EE9E11-91C2-41CF-8FD4-7970579E5C4C',
'protocol': 'TCP',
'port': 5678,
'timeout_client_data': 75000,
'timeout_member_data': 0
}
listener_id = 'A57B7771-6050-4CA8-A63C-443493EC98AB'
req = {
'name': listener['name'],
'project_id': listener['project_id'],
'loadbalancer_id': listener['loadbalancer_id'],
'protocol': listener['protocol'],
'protocol_port': listener['port'],
'timeout_client_data': listener['timeout_client_data']}
resp = o_lis.Listener(id=listener_id)
lbaas.create_listener.return_value = resp
ret = cls._create_listener(m_driver, listener)
lbaas.create_listener.assert_called_once_with(**req)
self.assertEqual(listener, ret)
self.assertEqual(listener_id, ret['id'])
def test_find_listener(self):
lbaas = self.useFixture(k_fix.MockLBaaSClient()).client
cls = d_lbaasv2.LBaaSv2Driver

10
releasenotes/notes/make-listener-timeouts-configurable-f563d85eg6c6fe6d.yaml Normal file
View File

@ -0,0 +1,10 @@
---
features:
- |
Kuryr will now support the configuration of Octavia listener timeouts.
The timeout-client-data and timeout-member-data settings of listeners
can be configured to use values other than the Octavia defaults when
creating or modifying loadbalancers handled by Kuryr. In order to use this
functionality, the new timeout values can be annotated to the Service or
set in the config-map. New options ``[octavia_defaults]timeout_client_data``
and ``[octavia_defaults]timeout_member_data`` were added.