openstack/kuryr-kubernetes
Code Issues Proposed changes

Implement NPWG multi-vif driver

Browse Source 
This patch creates a npwg multi-vif driver which can parse the
Pod annotations and CRD defined in Network Plumbing Working
Group CRD SPEC.

Implements: blueprint kuryr-npwg-spec-support
Change-Id: I9ee9643b468a5fe453541b9cf1acf31ca872a313
This commit is contained in:
Peng Liu 2018-06-25 21:24:19 +08:00
parent f4016637e0
commit 70ee5ad132
8 changed files with 424 additions and 28 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
doc/source/installation/index.rst
View File

@ -41,3 +41,4 @@ This section describes how you can install and configure kuryr-kubernetes
testing_nested_connectivity
containerized
ocp_route
multi_vif_with_npwg_spec

93
doc/source/installation/multi_vif_with_npwg_spec.rst Normal file
View File

@ -0,0 +1,93 @@
Configure Pod with Additional Interfaces
========================================
To create pods with additional Interfaces follow the Kubernetes Network Custom
Resource Definition De-facto Standard Version 1 [#]_, the next steps can be
followed:
1. Create Neutron net/subnets which you want the additional interfaces attach
to.
.. code-block:: bash
$ openstack network create net-a
$ openstack subnet create subnet-a --subnet-range 192.0.2.0/24 --network net-a
2. Create CRD of 'NetworkAttachmentDefinition' as defined in NPWG spec.
.. code-block:: bash
$ cat << EOF > nad.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: network-attachment-definitions.k8s.cni.cncf.io
spec:
group: k8s.cni.cncf.io
version: v1
scope: Namespaced
names:
plural: network-attachment-definitions
singular: network-attachment-definition
kind: NetworkAttachmentDefinition
shortNames:
- net-attach-def
validation:
openAPIV3Schema:
properties:
spec:
properties:
config:
type: string
EOF
$ kubectl apply -f nad.yal
3. Create NetworkAttachmentDefinition object with the UUID of Neutron subnet
defined in step 1.
.. code-block:: bash
$ cat << EOF > net-a.yaml
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: "net-a"
annotations:
openstack.org/kuryr-config: '{
"subnetId": "uuid-of-neutron-subnet-a"
}'
EOF
$ kubectl apply -f net-a.yaml
4. Enable the multi-vif driver by setting 'multi_vif_drivers' in kuryr.conf.
Then restart kuryr-controller.
.. code-block:: ini
[kubernetes]
multi_vif_drivers = npwg_multiple_interfaces
5. Add additional interfaces to pods definition. e.g.
.. code-block:: bash
$ cat << EOF > pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: nginx4
annotations:
k8s.v1.cni.cncf.io/networks: net-a
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80
EOF
$ kubectl apply -f pod.yaml
Reference
---------
.. [#] https://docs.google.com/document/d/1Ny03h6IDVy_e_vmElOqR7UdTPAG_RNydhVE1Kx54kFQ/edit?usp=sharing

56
doc/source/specs/rocky/npwg_spec_support.rst
View File

@ -56,7 +56,7 @@ Here's how a Pod Spec with additional networks requests might look like:
name: my-pod
namespace: my-namespace
annotations:
kubernetes.v1.cni.cncf.io/networks: net-a,net-b,other-ns/net-c
k8s.v1.cni.cncf.io/networks: net-a,net-b,other-ns/net-c
Or in JSON format like:
@ -67,7 +67,7 @@ Or in JSON format like:
name: my-pod
namespace: my-namespace
annotations:
kubernetes.v1.cni.cncf.io/networks: |
k8s.v1.cni.cncf.io/networks: |
[
{"name":"net-a"},
{"name":"net-b"},
@ -78,36 +78,36 @@ Or in JSON format like:
]
Then the VIF driver can parse the network information defined in 'Network'
objects. In NPWG spec, the 'Network' object definition is very flexible.
Implementations that are not CNI delegating plugins can add annotations to the
Network object and use those to store non-CNI configuration. And it is up to
the implementation to define the content it requires.
objects. In NPWG spec, the 'NetworkAttachmentDefinition' object definition is
very flexible. Implementations that are not CNI delegating plugins can add
annotations to the Network object and use those to store non-CNI configuration.
And it is up to the implementation to define the content it requires.
Here is how 'Network' CRD specified in the NPWG spec.
Here is how 'CustomResourceDefinition' CRD specified in the NPWG spec.
.. code-block:: yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: networks.kubernetes.cni.cncf.io
spec:
group: kubernetes.cni.cncf.io
version: v1
scope: Namespaced
names:
plural: networks
singular: network
kind: Network
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: network-attachment-definitions.k8s.cni.cncf.io
spec:
group: k8s.cni.cncf.io
version: v1
scope: Namespaced
names:
plural: network-attachment-definitions
singular: network-attachment-definition
kind: NetworkAttachmentDefinition
shortNames:
- net
validation:
openAPIV3Schema:
properties:
spec:
properties:
config:
type: string
- net-attach-def
validation:
openAPIV3Schema:
properties:
spec:
properties:
config:
type: string
For Kuryr-kubernetes, users should define the 'Network' object with a Neutron
subnet created previously like:
@ -123,7 +123,7 @@ subnet created previously like:
"subnetId": "id_of_neutron_subnet_created_previously"
}'
With information read from Pod annotation kubernetes.v1.cni.cncf.io/networks
With information read from Pod annotation k8s.v1.cni.cncf.io/networks
and 'Network' objects, the Neutron ports could either be created or retrieved.
Then the Pod annotation openstack.org/kuryr-vif will be updated accordingly.

7
kuryr_kubernetes/constants.py
View File

@ -18,6 +18,8 @@ K8S_API_NAMESPACES = K8S_API_BASE + '/namespaces'
K8S_API_CRD = '/apis/openstack.org/v1'
K8S_API_POLICIES = '/apis/networking.k8s.io/v1/networkpolicies'
K8S_API_NPWG_CRD = '/apis/k8s.cni.cncf.io/v1'
K8S_OBJ_NAMESPACE = 'Namespace'
K8S_OBJ_POD = 'Pod'
K8S_OBJ_SERVICE = 'Service'
@ -38,6 +40,11 @@ K8S_ANNOTATION_LBAAS_RT_NOTIF = K8S_ANNOTATION_PREFIX + '-lbaas-route-notif'
K8S_ANNOTATION_ROUTE_STATE = K8S_ANNOTATION_PREFIX + '-route-state'
K8S_ANNOTATION_ROUTE_SPEC = K8S_ANNOTATION_PREFIX + '-route-spec'
K8S_ANNOTATION_NPWG_PREFIX = 'k8s.v1.cni.cncf.io'
K8S_ANNOTATION_NPWG_NETWORK = K8S_ANNOTATION_NPWG_PREFIX + '/networks'
K8S_ANNOTATION_NPWG_CRD_SUBNET_ID = 'subnetId'
K8S_ANNOTATION_NPWG_CRD_DRIVER_TYPE = 'driverType'
K8S_OS_VIF_NOOP_PLUGIN = "noop"
CNI_EXCEPTION_CODE = 100

82
kuryr_kubernetes/controller/drivers/multi_vif.py
View File

@ -12,7 +12,16 @@
# License for the specific language governing permissions and limitations
# under the License.
from oslo_log import log as logging
from oslo_serialization import jsonutils
from kuryr_kubernetes import clients
from kuryr_kubernetes import constants
from kuryr_kubernetes.controller.drivers import base
from kuryr_kubernetes.controller.drivers import default_subnet
from kuryr_kubernetes import exceptions
LOG = logging.getLogger(__name__)
class NoopMultiVIFDriver(base.MultiVIFDriver):
@ -20,3 +29,76 @@ class NoopMultiVIFDriver(base.MultiVIFDriver):
def request_additional_vifs(
self, pod, project_id, security_groups):
return []
class NPWGMultiVIFDriver(base.MultiVIFDriver):
def __init__(self):
super(NPWGMultiVIFDriver, self).__init__()
self._drv_vif_pool = base.VIFPoolDriver.get_instance(
driver_alias='multi_pool')
self._drv_vif_pool.set_vif_driver()
def request_additional_vifs(self, pod, project_id, security_groups):
vifs = []
networks = self._get_networks(pod)
if not networks:
return vifs
kubernetes = clients.get_kubernetes_client()
namespace = pod['metadata']['namespace']
for network in networks:
if 'name' not in network:
raise exceptions.InvalidKuryrNetworkAnnotation()
if 'namespace' in network:
namespace = network['namespace']
try:
url = '%s/namespaces/%s/network-attachment-definitions/%s' % (
constants.K8S_API_NPWG_CRD, namespace, network['name'])
nad_obj = kubernetes.get(url)
except exceptions.K8sClientException:
LOG.exception("Kubernetes Client Exception")
raise
config = jsonutils.loads(nad_obj['metadata']['annotations']
['openstack.org/kuryr-config'])
subnet_id = config[constants.K8S_ANNOTATION_NPWG_CRD_SUBNET_ID]
subnet = {subnet_id: default_subnet._get_subnet(subnet_id)}
if constants.K8S_ANNOTATION_NPWG_CRD_DRIVER_TYPE not in config:
vif_drv = self._drv_vif_pool
else:
alias = config[constants.K8S_ANNOTATION_NPWG_CRD_DRIVER_TYPE]
vif_drv = base.PodVIFDriver.get_instance(
driver_alias=alias)
vif = vif_drv.request_vif(pod, project_id, subnet, security_groups)
if vif:
vifs.append(vif)
return vifs
def _get_networks(self, pod):
networks = []
try:
annotations = pod['metadata']['annotations']
key = constants.K8S_ANNOTATION_NPWG_NETWORK
networks_annotation = annotations[key]
except KeyError:
return []
try:
networks = jsonutils.loads(networks_annotation)
except ValueError:
# if annotation is not in json format, convert it to json.
net_list = networks_annotation.split(',')
for net in net_list:
net_details = net.split('/')
if len(net_details) == 1:
networks.append({'name': net_details[0]})
elif len(net_details) == 2:
networks.append(
{'namespace': net_details[0], 'name': net_details[1]}
)
else:
raise exceptions.InvalidKuryrNetworkAnnotation()
return networks

4
kuryr_kubernetes/exceptions.py
View File

@ -38,6 +38,10 @@ class InvalidKuryrNetCRD(Exception):
pass
class InvalidKuryrNetworkAnnotation(Exception):
pass
class CNIError(Exception):
pass

208
kuryr_kubernetes/tests/unit/controller/drivers/test_multi_vif.py Normal file
View File

@ -0,0 +1,208 @@
# Copyright 2018 Red Hat, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import mock
from kuryr_kubernetes import constants
from kuryr_kubernetes.controller.drivers import base as drivers
from kuryr_kubernetes.controller.drivers import multi_vif
from kuryr_kubernetes import exceptions
from kuryr_kubernetes.tests import base as test_base
from oslo_serialization import jsonutils
def get_pod_obj():
return {
'status': {
'qosClass': 'BestEffort',
'hostIP': '192.168.1.2',
},
'kind': 'Pod',
'spec': {
'schedulerName': 'default-scheduler',
'containers': [{
'name': 'busybox',
'image': 'busybox',
'resources': {}
}],
'nodeName': 'kuryr-devstack'
},
'metadata': {
'name': 'busybox-sleep1',
'namespace': 'default',
'resourceVersion': '53808',
'selfLink': '/api/v1/namespaces/default/pods/busybox-sleep1',
'uid': '452176db-4a85-11e7-80bd-fa163e29dbbb',
'annotations': {
'openstack.org/kuryr-vif': {},
'k8s.v1.cni.cncf.io/networks':
"net-a,net-b,other-ns/net-c"
}
}
}
def get_nets():
return [
{"name": "net-a"},
{"name": "net-b"},
{
"name": "net-c",
"namespace": "other-ns"
}
]
def get_crd_objs():
return [
{
'name': 'net-a',
'metadata': {
'annotations': {
'openstack.org/kuryr-config':
'''{"subnetId": "subnet-a"}'''
}
}
},
{
'name': 'net-b',
'metadata': {
'annotations': {
'openstack.org/kuryr-config':
'''{"subnetId": "subnet-b"}'''
}
}
},
{
'name': 'net-c',
'metadata': {
'annotations': {
'openstack.org/kuryr-config':
'''{"subnetId": "subnet-c"}'''
}
}
}
]
def get_subnet_objs():
return [
{'subnet-a': mock.sentinel.subneta},
{'subnet-b': mock.sentinel.subnetb},
{'subnet-c': mock.sentinel.subnetc},
]
class TestNPWGMultiVIFDriver(test_base.TestCase):
def setUp(self):
super(TestNPWGMultiVIFDriver, self).setUp()
self._project_id = mock.sentinel.project_id
self._subnet = mock.sentinel.subnet
self._vif = mock.sentinel.vif
self._subnets = [self._subnet]
self._security_groups = mock.sentinel.security_groups
self._pod = get_pod_obj()
self._vif_pool_drv = mock.Mock(spec=drivers.VIFPoolDriver)
self._request_vif = self._vif_pool_drv.request_vif
self._request_vif.return_value = self._vif
self._cls = multi_vif.NPWGMultiVIFDriver
self._drv = mock.Mock(spec=self._cls)
self._drv._get_networks = mock.Mock()
self._drv._drv_vif_pool = self._vif_pool_drv
@mock.patch.object(drivers.VIFPoolDriver, 'set_vif_driver')
@mock.patch.object(drivers.VIFPoolDriver, 'get_instance')
def test_init(self, m_get_vif_pool_driver, m_set_vifs_driver):
m_get_vif_pool_driver.return_value = self._vif_pool_drv
self._vif_pool_drv.set_vif_driver = m_set_vifs_driver
m_drv = multi_vif.NPWGMultiVIFDriver()
self.assertEqual(self._vif_pool_drv, m_drv._drv_vif_pool)
m_get_vif_pool_driver.assert_called_once_with(
driver_alias='multi_pool')
m_set_vifs_driver.assert_called_once()
@mock.patch('kuryr_kubernetes.controller.drivers'
'.default_subnet._get_subnet')
@mock.patch('kuryr_kubernetes.clients.get_kubernetes_client')
def test_request_additional_vifs(self, m_get_client, m_get_subnet):
vifs = [mock.sentinel.vif_a, mock.sentinel.vif_b, mock.sentinel.vif_c]
self._request_vif.side_effect = vifs
net_crds = get_crd_objs()
client = mock.Mock()
m_get_client.return_value = client
m_get_subnet.side_effect = [mock.sentinel.subneta,
mock.sentinel.subnetb,
mock.sentinel.subnetc]
client.get = mock.Mock()
client.get.side_effect = net_crds
self._drv._get_networks.return_value = get_nets()
self.assertEqual(vifs, self._cls.request_additional_vifs(
self._drv, self._pod, self._project_id, self._security_groups))
def test_get_networks_str(self):
networks = get_nets()
self.assertEqual(networks,
self._cls._get_networks(self._drv, self._pod))
def test_get_networks_json(self):
networks = get_nets()
self._pod['metadata']['annotations'][
'kubernetes.v1.cni.cncf.io/networks'] = jsonutils.dumps(networks)
self.assertEqual(networks,
self._cls._get_networks(self._drv, self._pod))
def test_get_networks_with_invalid_annotation(self):
self._pod['metadata']['annotations'][
constants.K8S_ANNOTATION_NPWG_NETWORK] = 'ns/net-a/invalid'
self.assertRaises(exceptions.InvalidKuryrNetworkAnnotation,
self._cls._get_networks, self._drv, self._pod)
def test_get_networks_without_annotation(self):
pod = {
'metadata': {
'annotations': {
}
}
}
self.assertEqual([], self._cls._get_networks(self._drv, pod))
@mock.patch('kuryr_kubernetes.clients.get_kubernetes_client')
def test_request_additional_vifs_without_networks(self, m_get_client):
self._drv._get_networks.return_value = []
self.assertEqual([],
self._cls.request_additional_vifs(
self._drv, self._pod, self._project_id,
self._security_groups))
m_get_client.assert_not_called()
@mock.patch('kuryr_kubernetes.clients.get_kubernetes_client')
def test_request_additional_vifs_with_invalid_network(self, m_get_client):
net_crds = get_crd_objs()
client = mock.Mock()
m_get_client.return_value = client
client.get = mock.Mock()
client.get.side_effects = net_crds
networks = [{'invalid_key': 'net-x'}]
self._drv._get_networks.return_value = networks
self.assertRaises(exceptions.InvalidKuryrNetworkAnnotation,
self._cls.request_additional_vifs,
self._drv, self._pod, self._project_id,
self._security_groups)

1
setup.cfg
View File

@ -99,6 +99,7 @@ kuryr_kubernetes.controller.handlers =
kuryr_kubernetes.controller.drivers.multi_vif =
noop = kuryr_kubernetes.controller.drivers.multi_vif:NoopMultiVIFDriver
npwg_multiple_interfaces = kuryr_kubernetes.controller.drivers.multi_vif:NPWGMultiVIFDriver
[files]
packages =