From 0eab8bab2dd29bfaa0b7d5e1906752b3d33e8169 Mon Sep 17 00:00:00 2001 From: Xinran Wang Date: Fri, 24 Jul 2020 03:03:32 +0000 Subject: [PATCH] Smart-nic Support Specification This spec proposes an overall design for smartnic management which will involve Nova, Neutron and Cyborg changes. After the discusstion during Wallaby PTG, we have a general agreement about the overall design. Please refer to: https://etherpad.opendev.org/p/nova-wallaby-ptg In this spec, we will introduce basically how to manage a smart-nic and how to use a smart-nic for a VM. Co-authored-by: Yongli He Change-Id: Ia425629ae5222d6fa7485845533231d0c04a989b --- .../approved/support-sriov-smartnic.rst | 515 ++++++++++++++++++ 1 file changed, 515 insertions(+) create mode 100644 specs/wallaby/approved/support-sriov-smartnic.rst diff --git a/specs/wallaby/approved/support-sriov-smartnic.rst b/specs/wallaby/approved/support-sriov-smartnic.rst new file mode 100644 index 000000000..5f0e63d85 --- /dev/null +++ b/specs/wallaby/approved/support-sriov-smartnic.rst @@ -0,0 +1,515 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +=================================== +Smartnic Management Overall Design +=================================== + +https://blueprints.launchpad.net/nova/+spec/sriov-smartnic-support + +This spec proposes an overall design for smartnic management which will involve +Nova, Neutron and Cyborg changes. In this spec, we will introduce how to manage +the smartnic's lifecycle, and how to attach a smartnic to the VM. +This spec aims at supporting VF management, the PF management is out-of-scope. + + +Problem description +=================== + +Nowadays, various devices are made to run specific workloads which will free up +CPU resources. Smart-nic is a network device that can be used to offload the +network-related workload. It goes beyond simple connectivity and implements +network traffic processing on the NIC that would necessarily be performed by +the CPU in the case of a conventional NIC. + +In the current design, OpenStack can not manage and schedule a smart-nic in +terms of supported features and readable traits automatically. + + +Use Cases +--------- +* Users want to boot up a VM with a specific port which is associated with a + smartnic resource managed by Cyborg. +* Users want to boot vms that leverage pre-programmed functionality to offload + their workload to a smart-nic. + +Proposed change +=============== + +There are multiple projects(Nova, Neutron, Cyborg, and Placement) involved in +this feature. + +Workflow +-------- +This workflow describes the basic operation to boot a VM with pre-programmed +nic. +The workflow can be divided into two parts: + +* Device discovery and report. +* Interaction when booting a VM. + + +Device discovery +^^^^^^^^^^^^^^^^ +Cyborg should implement a driver that the cyborg-agent can invoke periodically +to discover the smartnic resources. We will explain this part in detail in +Cyborg spec, please refer to https://review.opendev.org/#/c/759545/. + + +Device information report +^^^^^^^^^^^^^^^^^^^^^^^^^ + +How to report physnet trait +::::::::::::::::::::::::::: + +As we know, Neutron has a config option physical_device_mappings indicating the +mapping relation between physical_network and network_device. Admin need to +maintain another configuration file (Please refer to 1 in the flowchart) that +contains the device's name and the physical network name this device associate +to. +In this way, the Cyborg driver can directly read from this file and add a +physical network as a trait when reporting to Placement. The admin is +responsible to keep Cyborg's and Neutron's config file consistent, otherwise, +there will be a configuration conflict. + +Co-existence with Neutron's Guaranteed Minimum Bandwidth [1]_ Feature +::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: + +According to Wallaby PTG discussion [2]_ and the following discussions [3]_ +in the community, we propose to not support co-existence with Neutron's +Guaranteed Minimum Bandwidth Feature for the same physical device at the +first step, which means that one physical device can only be configured by +Cyborg or by Neutron's QoS feature if it is enabled. + + +The following flowchart illustrates the workflow during device discovery and +report:: + + +----------------+ + | cyborg-api | + +--------+-------+ + | + | +-------------+ +-------------+ + +----------|---------+ | | | | + | cyborg-conductor |----------->| Placement |<-------| Neutron | + +----------|---------+ | | | | | + | | +-------------+ +-------------+ + | | +------------+ + | +------>| Cyborg DB | + +--------|-------+ +------------+ + | cyborg-agent | + +--------|-------+ + | + | + | +----------------+ + +--------|-------+ 1 | | + | device-driver |<----------| device.conf | + +----------------+ | | + +----------------+ + + + +Interaction when booting the VM +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Currently, Nova can interact with Cyborg to boot up a VM with an accelerator +like FPGA, GPU. And other operations such as hard/soft reboot, pause/unpause +are also supported. But there is no mechanism to let Nova boot up a VM with a +nic associated with a specific network. To implement this, it requires +Nova, Cyborg, and Neutron change. And this spec also covers the scenario about +reboot/pause/stop/start and other operations. + +Here we take the "boot a VM" scenario as an example. Assuming that Cyborg has +reported the nic resources and related traits correctly. +The workflow is proposed as the following:: + + +-----------+ +-----------+ +-----------+ +-----------+ +---------+ + | admin | | Neutron | | Nova | | Placement | | Cyborg | + +-----|-----+ +-----|-----+ +-----|-----+ +-----|-----+ +-------+-+ + | | | | | + | 1.create device profile | | | + |----------------------------------------------------------------->| + | | | | + | | | | + | | | | + | | | | + +-----------+ | | | | + | end user | | | | | + +-----|-----+ | | | | + | 2. create port | | | | + |------------------>| | | | + | 3. create server with port | | | + |-------------------------------->| | | + 4.get port details and physnet trait | + + |<------------| | | + 5. get device profile + | |<------------------------------->| + | | 6.sheduling | | + | |<------------->| | + | | | | + | | | | + | |7.bind device and return PCI info| + | |<------------------------------->| + |8.update port| | | + |<------------| | | + | binding info| | | + | |-----+ | | + | | | | | + | | | 9. insert SRIOV VIF info | + | | | to libvirt XML section | + | | | | | + | |<----+ | | + | | | | + + +1. Firstly, admin needs to create a device profile that contains the smartnic's +description such as resource class, and traits. The CLI are already supported +as the following: + +`GRP="[{"resources:CUSTOM_NIC": "1","trait:CUSTOM_GTV1":"required"}]" +openstack accelerator device profile create sriov_dp1 $GRP` + +2. Secondly, user needs to create a port by passing device profile as a +parameter. Related API needs to be added to operate this. And also, +Neutron need to add a new vnic-type for the nic managed by Cyborg, we can name +it "cyborg" here. For example, we can create a port by: `openstack port +create --network providernet --vnic-type cyborg --device-profile sriov-dp1 +sriov_port1` in which `sriov-dp1` is the device profile created at the first +step. +The request body is the following: + +.. code-block:: + + { + "port": { + "name": "sriov-port1", + "network_id": "a87cc70a-3e15-4acf-8205-9b711a3531b7", + "vnic_type":"cyborg", + "device_profile": "sriov_dp1" # new extension contains device profile + } + } + + +3. Thirdly, user can boot up a VM by: +`openstack server create --image image-uuid -flavor flavor-name --nic +port-id=sriov_port1 test_vm1` + +4. Nova interacts with Neutron to get port’s details, including vnic type, +neutwork_id, physical network, etc. + +5. If the vnic type is "cyborg", then Nova need to extract the “device_profile” +extension of sriov_port1, and call Cyborg API to get details of this +device_profile [4]_, the ARQ creation is also in this step. + +6. In step 4, Nova has fetched the physical network from Neutron, now Nova need +to convert it into Placement's trait format and save it in port's resource +request field if vnic type is "cyborg". Then Nova need to merge the resource +class/trait obtained from Cyborg’s device profile and the port resource request +into a one single request group. And this request group will be merged into +request_spec which will be used in reboot/pause/start/stop and other supported +operations. After that, Nova schedules the VM to an available compute node +who matches all requested resources. + +7. After scheduling, Nova needs to call Cyborg to bind the ARQ with port id +and return attach_handle which contains the device's info such as PCI +address. + +8. Nova also needs to tell Neutron to update the port binding’s info +in this step. + +9. Libvirt driver need to insert SRIOV nic info to the XML +section. + + +API calls +^^^^^^^^^ + +1. Nova calls Neutron to get port details. + +* request URL: /v2.0/ports/{port_id} +* Method: GET +* response example(the new extension `device_profile` should be returned): + + .. code-block:: + + { + "port": { + ... + "binding_profile": {}, + "name": "sriov-port", + "network_id": "a87cc70a-3e15-4acf-8205-9b711a3531b7", + "qos_network_policy_id": "174dd0c1-a4eb-49d4-a807-ae80246d82f4", + "qos_policy_id": "29d5e02e-d5ab-4929-bee4-4a9fc12e22ae", + "device_profile":"sriov-dp1" # new extension + ... + } + } + +2. Nova calls Cyborg to get device profile's details. [5]_ + +3. Nova calls Cyborg to create and bind ARQs. + +* Create ARQ [6]_ +* Bind ARQ [7]_ + +4. Nova call Neutron to update port binding profile with interface info. [8]_ + +Neutron +------- + +In Neutron side, a new vnic type "cyborg" need to be added, as well as a new +port extension "device_profile". Please refer to Neutron's RFE [9]_ for +details. + +The proposed change includes: + +* Add a new vnic type "cyborg" indicating the port associated with device + managed by Cyborg. + +* Define device profile extension for port in neutorn lib. + +* Implement device profile extension for port in Neutron. + +* From DB side, we need to add a new table to store the mapping relation + between port and device_profile:: + + +---------------------------------------+------------------------+ + | port_uuid | device_profile | + +=======================================+========================+ + | 20f78856-0f73-4cf4-bcd0-1389086eb038 | sriov_dev_profile | + +---------------------------------------+------------------------+ + + +Please refer to: + +* Port resource request definition [10]_ +* neutron-lib port-resource-request Commits [11]_ +* neutron plugin port-resource-request Commits [12]_ + +Nova +---- +* Nova API: Nova calls Neutron API to get port details, including vnic type, + physical network etc. + "device_profile" should be returned as the return value. +* Nova API: Nova need to check if vnic type is "cyborg". If so, Nova will get + the device profile's name from neutron port and call Cyborg API to get the + details of this device profile. Meanwhile, Nova need to generate a trait + for physical network, for example, Nova get "physnet1" as the physical + network from Neutorn, the trait should looks like "CUSTOM_PHYSNET_PHYSNET1", + which is consistent with what Cyborg reports. +* Nova API: Once Nova gets all resource classes and traits, Nova should check + if a RequestGroup is created with port_resource_request, if so, we should add + resource and traits to this request group, if not, we should generate a new + resource group. Nova should have one single request_group to schedule to a + single nic resource provider [13]_. This request_group store the requested + resource information used by the scheduler, and other operations, such as + reboot/pause/unpause/start/stop, will use this request_group to do the + scheduling as well. +* Nova Compute: Nova should update port binding profile with sriov nic's info + (such as pci adderess etc), so that libvirt driver can generete related xml + section. + +Cyborg +------ + +* A new driver needs to be added in Cyborg in order to discover, program and + bind the device. More details is in the Cyborg spec. +* Cyborg should improve current bind ARQ API to support binding an ARQ to the + port ID. +* Cyborg needs to implement a device config file to configure the nic's + name, pci address, physnet name, etc, which are used for Cyborg driver to + generate resource provider, trait, etc. + + .. code-block:: RST + + [dev-name] + pci_address=0000:18:00.0 + function_name=GTPv1 + physnet=physnet1 + + +Alternatives +------------ + +Who reports physnet trait +^^^^^^^^^^^^^^^^^^^^^^^^^ +* Placement CLI to add trait. + + Admin need to add physnet trait to resource provider manually, this will be + done after Cyborg reports resource to Placement. It will cause redundant + Placement API call as well. + +* Neutron report trait for the resource provider created by Cyborg. + + In this way, Neutron should firstly get a resource provider created by Cyborg + by Placement API, and Neutron also need to find the right physnet tratis + according to nic's resource, which requires more changes in Neutron. + +* Let Neutron create RP and physnet trait. + + As we know, Neutron will create RP and related traits when minimum bandwidth + QoS is configured. We propose that Neutron could always report RP and physnet + traits all the time, no matter the bandwidth qos configured or not. In this + way, Cyborg will just find the RP created by Neutron by using some name + convention, and add accelerator-related traits to this RP. + + But we should consider how the RP tree structure look like when Neutron + report it, should it be directly under compute node RP, or still under Agent + RP like bandwidth qos feature does. + + +How Nova generate request_spec with request device +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +* Neutron calls Cyborg API to get device profile details and merge all RC and + trait into port_resource_reqeust, then return to Nova. + + It requires Neutron changes to interact with Cyborg, which seems that Cyborg + is a sub-component for Neutron. It's better to let Nova interact with Cyborg + and Neutron equally. + +* Implement a new DeviceProfileRequest to store the device profile's request + group, and pass it as a new parameter when Nova generates request_spec. + + It seems to be redundant to have a new request object. + +Others +^^^^^^ +* Cyborg provides its own SRIOV ML2 driver for the NICs it supports. + + Cyborg maintains a Neutron plugin driver that exceeds Cyborg project's scope. + +* ARQ's consumer could be an instance instead of port. + + May cause the confusion with current Nova-Cyborg integration(No Neutron + involvement). But it seems a choice, it's open for comments and suggestions. + +Data model impact +----------------- +New table `device_profile` needs to be added in Neutron DB. + + +REST API impact +--------------- +* A new extension 'device_profile' will be added in Neutron port, Neutron API + need changes. Neutron API should also forbid user to modify 'device_profile' + field once the port is bound with one instance(Neutorn API need to check the + binding:host-id before updating the 'device profile' field.). This + modification is only allowed when the port is unbound. +* Bind ARQ API should be improved to support binding an ARQ with port. +* Change Nova APIs for more operation supports. We plan to support + create/delete, start/stop, pause/unpause, suspend/resume, shelve/unshelve, + rebuild, reboot, evacuate operations for a VM having an "cyborg" vnic type + port. + The rest of the lifecycle operations will be rejected now with HTTP 403 Error + code. + +Security impact +--------------- +None + + +Notifications impact +-------------------- + +None + + +Other end user impact +--------------------- +None + + +Other deployer impact +--------------------- +None + + +Developer impact +---------------- +None + + +Performance Impact +------------------ +None + + +Upgrade impact +-------------- +None + + +Implementation +============== + +Assignee(s) +----------- +Yongli He(yongli.he@intel.com) + +Xinran Wang(xin-ran.wang@intel.com) + + +Feature Liaison +--------------- +None + + +Work Items +---------- +* Add new port extension in Neutron. +* Implement a new driver for specific nic in Cyborg. +* Add a configuration file in Cyborg to handle physnet. +* Parse and merge request into request spec in Nova. +* Bind ARQ to port id and update port binding profile. +* Re-use the current sriov nic xml generation code. + +Dependencies +============ + +None + + +Testing +======= + +* Need to add UT in the involved project. +* Functional test in Nova. +* Tempest test in Nova if necessary. +* Tempest test in cyborg-tempest-plugin. + + +Documentation Impact +==================== +Need to add documentation. + +References +========== + +.. [1] https://docs.openstack.org/neutron/latest/admin/config-qos-min-bw.html +.. [2] https://etherpad.opendev.org/p/nova-wallaby-ptg +.. [3] http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2020-11-09.log.html#t2020-11-09T05:48:48 +.. [4] https://docs.openstack.org/api-ref/accelerator/v2/index.html?expanded=#list-device-profiles +.. [5] https://docs.openstack.org/api-ref/accelerator/v2/index.html#list-device-profiles +.. [6] https://docs.openstack.org/api-ref/accelerator/v2/index.html#create-accelerator-requests +.. [7] https://docs.openstack.org/api-ref/accelerator/v2/index.html#update-accelerator-requests +.. [8] https://docs.openstack.org/api-ref/network/v2/#update-port +.. [9] https://bugs.launchpad.net/neutron/+bug/1906603 +.. [10] https://review.opendev.org/#/c/508149/14/specs/rocky/minimum-bandwidth-allocation-placement-api.rst +.. [11] https://github.com/openstack/neutron-lib/search?q=port-resource-request&type=Commits +.. [12] https://github.com/openstack/neutron/search?q=port-resource-request&type=Commits +.. [13] https://docs.openstack.org/placement/latest/user/provider-tree.html#granular-resource-requests + +History +======= + +.. list-table:: Revisions + :header-rows: 1 + + * - Release Name + - Description + * - Wallaby + - Introduced