Networking guide: Add Guaranteed Minimum Packet Rate
Add Neutron doc for the Guaranteed Minimum Packet Rate feature. Co-Authored-By: Balazs Gibizer <balazs.gibizer@est.tech> Co-Authored-By: Bence Romsics <bence.romsics@est.tech> Partial-Bug: #1922237 See-Also: https://review.opendev.org/785236 Change-Id: I8bbd503488ae0997c0e5f33e0ef7709841c310b5
This commit is contained in:
Przemyslaw Szczerbik
parent
12ca5e9a40
commit
77b3a1a774
|
|
@ -0,0 +1,518 @@
|
|||
Quality of Service (QoS): Guaranteed Minimum Packet Rate
|
||||
========================================================
|
||||
|
||||
Similarly to how bandwidth can be a limiting factor of a network interface,
|
||||
packet processing capacity tend to be a limiting factor of the soft switching
|
||||
solutions like OVS. At the same time certain applications are dependent on not
|
||||
just guaranteed bandwidth, but also on guaranteed packet rate to function
|
||||
properly. OpenStack already supports bandwidth guarantees via the
|
||||
minimum bandwidth QoS policy rules, which is described in detail in
|
||||
:doc:`Quality of Service (QoS): Guaranteed Minimum Bandwidth
|
||||
<config-qos-min-bw>`. It's recommended to read Guaranteed Minimum Bandwidth
|
||||
guide first, but it's not strictly required.
|
||||
|
||||
Just like :doc:`Quality of Service (QoS): Guaranteed Minimum Bandwidth
|
||||
<config-qos-min-bw>` guide, this chapter does not aim to replace Nova or
|
||||
Placement documentation in any way, but gives a brief overview of the feature
|
||||
and explains how it can be configured.
|
||||
|
||||
In a similar way to guaranteed bandwidth, we can distinguish two levels of
|
||||
enforcement for guaranteeing packet processing capacity constraint:
|
||||
|
||||
* placement: Avoiding over-subscription when placing (scheduling) VMs and their
|
||||
ports.
|
||||
|
||||
* data plane: Enforcing the guarantee on the soft switch
|
||||
|
||||
.. note::
|
||||
|
||||
At the time of writing this guide, only placement enforcement is supported.
|
||||
For detailed list of supported enforcement types and backends, please refer
|
||||
to :doc:`QoS configuration chapter of the Networking Guide <config-qos>`.
|
||||
|
||||
The solution needs to differentiate between two different deployment scenarios:
|
||||
|
||||
1) The packet processing functionality is implemented on the compute host CPUs
|
||||
and therefore packets processed from both ingress and egress directions are
|
||||
handled by the same set of CPU cores. This is the case in the
|
||||
non-hardware-offloaded OVS deployments. In this scenario OVS represents a
|
||||
single packet processing resource pool, which is represented with a
|
||||
single resource class called ``NET_PACKET_RATE_KILOPACKET_PER_SEC``.
|
||||
|
||||
2) The packet processing functionality is implemented in a specialized hardware
|
||||
where the incoming and outgoing packets are handled by independent
|
||||
hardware resources. This is the case for hardware-offloaded OVS. In this
|
||||
scenario a single OVS has two independent resource pools one for the
|
||||
incoming packets and one for the outgoing packets. Therefore these needs to
|
||||
be represented with two different resource classes
|
||||
``NET_PACKET_RATE_EGR_KILOPACKET_PER_SEC`` and
|
||||
``NET_PACKET_RATE_IGR_KILOPACKET_PER_SEC``.
|
||||
|
||||
Limitations
|
||||
-----------
|
||||
|
||||
Since Guaranteed Minimum Packet Rate and Guaranteed Minimum Bandwidth features
|
||||
have a lot in common, they also share certain limitations.
|
||||
|
||||
* A pre-created port with a ``minimum-packet-rate`` rule must be passed
|
||||
when booting a server (``openstack server create``). Passing a network
|
||||
with a minimum-packet-rate rule at boot is not supported because of
|
||||
technical reasons (in this case the port is created too late for
|
||||
Neutron to affect scheduling).
|
||||
|
||||
* Changing the guarantee of a QoS policy (adding/deleting a
|
||||
``minimum_packet_rate`` rule, or changing the ``min_kpps`` field of a
|
||||
``minimum_packet_rate`` rule) is only possible while the policy is not in
|
||||
effect. That is ports of the QoS policy are not yet bound by Nova. Requests
|
||||
to change guarantees of in-use policies are rejected.
|
||||
|
||||
* Changing the QoS policy of the port with new ``minimum_packet_rate`` rules
|
||||
changes placement ``allocations`` from Yoga release.
|
||||
If the VM was booted with port without QoS policy and ``minimum_packet_rate``
|
||||
rules the port update succeeds but placement allocations will not change.
|
||||
The same is true if the port had no allocation record in Placement before
|
||||
QoS policy update. But if the VM was booted with a port with QoS policy and
|
||||
``minimum_packet_rate`` rules the update is possible and the allocations are
|
||||
changed in placement as well.
|
||||
|
||||
.. note::
|
||||
|
||||
As it is possible to update a port to remove the QoS policy, updating it
|
||||
back to have QoS policy with ``minimum_packet_rate`` rule will not result in
|
||||
``placement allocation`` record. In this case only dataplane enforcement will
|
||||
happen.
|
||||
|
||||
.. note::
|
||||
|
||||
Updating the ``minimum_packet_rate`` rule of a QoS policy that is attached
|
||||
to a port which is bound to a VM is still not possible.
|
||||
|
||||
* When QoS is used with a trunk, Placement enforcement is applied only to the
|
||||
trunk's parent port. Subports are not going to have Placement allocation.
|
||||
As a workaround, parent port QoS policy should take into account subports
|
||||
needs and request enough minimum packet rate resources to accommodate every
|
||||
port in the trunk.
|
||||
|
||||
Placement pre-requisites
|
||||
------------------------
|
||||
|
||||
Placement must support `microversion 1.36
|
||||
<https://docs.openstack.org/placement/latest/placement-api-microversion-history.html#support-same-subtree-queryparam-on-get-allocation-candidates>`_.
|
||||
This was first released in Train.
|
||||
|
||||
Nova pre-requisites
|
||||
-------------------
|
||||
|
||||
Nova must support top of `microversion 2.72
|
||||
<https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#maximum-in-stein>`_,
|
||||
additionally the Nova Xena release is needed to support the new
|
||||
``port-resource-request-groups`` Neutron API extension.
|
||||
|
||||
Not all Nova virt drivers are supported, please refer to the
|
||||
`Virt Driver Support section of the Nova Admin Guide
|
||||
<https://docs.openstack.org/nova/latest/admin/port_with_resource_request.html#virt-driver-support>`_.
|
||||
|
||||
Neutron pre-requisites
|
||||
----------------------
|
||||
|
||||
Neutron must support the following API extensions:
|
||||
|
||||
* ``qos-pps-minimum``
|
||||
* ``port-resource-request-groups``
|
||||
|
||||
These were all first released in Yoga.
|
||||
|
||||
Neutron DB sanitization
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The ``resource_request`` field of the Neutron port is used to express the
|
||||
resource needs of the port. The information in this field is calculated from
|
||||
the QoS policy rules attached to the port. Initially, only the minimum
|
||||
bandwidth rule was used as a source of requested resources. The format of
|
||||
``resource_request`` looked like this:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
{
|
||||
"required": [<CUSTOM_PHYSNET_ traits>, <CUSTOM_VNIC_TYPE traits>],
|
||||
"resources":
|
||||
{
|
||||
<NET_BW_[E|I]GR_KILOBIT_PER_SEC resource class name>:
|
||||
<requested bandwidth amount from the QoS policy>
|
||||
}
|
||||
},
|
||||
|
||||
This structure allowed to describe only one group of resources and traits,
|
||||
which was sufficient at the time. However, with the introduction of QoS minimum
|
||||
packet rate rule, ports can now have multiple sources of requested resources
|
||||
and traits. Because of that, the format of ``resource_request`` field was
|
||||
incapable of expressing such request and it had to be changed.
|
||||
|
||||
To solve this issue, ``port-resource-request-groups`` extension was
|
||||
added in Neutron Yoga release. It provides support for the new format of
|
||||
``resource_request`` field, that allows to request multiple groups of
|
||||
resources and traits from the same RP subtree. The new format looks like this:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
{
|
||||
"request_groups":
|
||||
[
|
||||
{
|
||||
"id": <min-pps-group-uuid>
|
||||
"required": [<CUSTOM_VNIC_TYPE traits>],
|
||||
"resources":
|
||||
{
|
||||
NET_PACKET_RATE_[E|I]GR_KILOPACKET_PER_SEC:
|
||||
<amount requested via the QoS policy>
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": <min-bw-group-uuid>
|
||||
"required": [<CUSTOM_PHYSNET_ traits>,
|
||||
<CUSTOM_VNIC_TYPE traits>],
|
||||
"resources":
|
||||
{
|
||||
<NET_BW_[E|I]GR_KILOBIT_PER_SEC resource class name>:
|
||||
<requested bandwidth amount from the QoS policy>
|
||||
}
|
||||
}
|
||||
],
|
||||
"same_subtree":
|
||||
[
|
||||
<min-pps-group-uuid>,
|
||||
<min-bw-group-uuid>
|
||||
]
|
||||
}
|
||||
|
||||
The main drawback about the new structure of ``resource_request`` field is lack
|
||||
of backwards compatibility. This can cause issues if ``ml2_port_bindings``
|
||||
table in Neutron DB contains port bindings that were created before the
|
||||
introduction of ``port-resource-request-groups`` extension. Because
|
||||
``port-resource-request-groups`` extension is enabled by default in Yoga
|
||||
release, it's necessary to perform DB sanitization before upgrading Neutron to
|
||||
Yoga.
|
||||
|
||||
DB sanitization will ensure that every row of ``ml2_port_bindings`` table
|
||||
uses the new format. Upgrade check can be run before DB sanitization, to see
|
||||
if there are any rows in the DB that require sanitization.
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ neutron-status upgrade check
|
||||
# If 'Port Binding profile sanity check' fails, DB sanitization is needed
|
||||
$ neutron-sanitize-port-binding-profile-allocation --config-file /etc/neutron/neutron.conf
|
||||
|
||||
Supported drivers and agents
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
In release Yoga the following agent-based ML2 mechanism drivers are
|
||||
supported:
|
||||
|
||||
* Open vSwitch (``openvswitch``) vnic_types: ``normal``, ``direct``
|
||||
|
||||
neutron-server config
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
QoS minimum packet rate rule requires exactly the same configuration in the
|
||||
``neutron-server`` as QoS minimum bandwidth rule. Please refer to
|
||||
``neutron-server config`` section of
|
||||
:doc:`Quality of Service (QoS): Guaranteed Minimum Bandwidth guide
|
||||
<config-qos-min-bw>` for more details.
|
||||
|
||||
neutron-openvswitch-agent config
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Set the agent configuration as the authentic source of the resources available.
|
||||
Depending on OVS deployment type, packet processing capacity can be configured
|
||||
with:
|
||||
|
||||
* :oslo.config:option:`ovs.resource_provider_packet_processing_without_direction`
|
||||
Format for this option is ``<hypervisor>:<packet_rate>``. This option should
|
||||
be used for non-hardware-offloaded OVS deployments.
|
||||
* :oslo.config:option:`ovs.resource_provider_packet_processing_with_direction`
|
||||
Format for this option is
|
||||
``<hypervisor>:<egress_packet_rate>:<ingress_packet_rate>``. You may set only
|
||||
one direction and omit the other. This option should be used for
|
||||
hardware-offloaded OVS deployments.
|
||||
|
||||
Regardless if direction-less or direction-oriented packet processing mode is
|
||||
used, configuration is always applied to the whole OVS instance.
|
||||
|
||||
.. note::
|
||||
|
||||
``egress`` / ``ingress`` is meant from the VM point of view.
|
||||
That is ``egress`` = cloud server upload, ``ingress`` = download.
|
||||
|
||||
Egress and ingress available packet rate values are in ``kilo packet/sec
|
||||
(kpps)``.
|
||||
|
||||
Direction-less and direction-oriented modes are mutually exclusive options.
|
||||
Only one can be used at a time.
|
||||
|
||||
The hypervisor name is optional, and needs to be set only in the rare case
|
||||
cases. For more information, please refer to Neutron agent documentation.
|
||||
|
||||
If desired, resource provider inventory fields can be tweaked on a
|
||||
per-agent basis by setting
|
||||
:oslo.config:option:`ovs.resource_provider_packet_processing_inventory_defaults`.
|
||||
Valid values are all the
|
||||
`optional parameters of the update resource provider inventory call
|
||||
<https://docs.openstack.org/api-ref/placement/?expanded=update-resource-provider-inventory-detail#update-resource-provider-inventory>`_.
|
||||
|
||||
``/etc/neutron/plugins/ml2/ovs_agent.ini`` (on compute and network nodes):
|
||||
|
||||
.. code-block:: ini
|
||||
|
||||
[ovs]
|
||||
resource_provider_packet_processing_with_direction = :10000000:10000000,...
|
||||
#resource_provider_packet_processing_inventory_defaults = step_size:1000,...
|
||||
|
||||
|
||||
Propagation of resource information
|
||||
-----------------------------------
|
||||
|
||||
Propagation of resource information is explained in detail in
|
||||
:doc:`Quality of Service (QoS): Guaranteed Minimum Bandwidth guide
|
||||
<config-qos-min-bw>`.
|
||||
|
||||
Sample usage
|
||||
------------
|
||||
|
||||
Network and QoS policies (together with their rules) are usually pre-created
|
||||
by a cloud admin:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
# as admin
|
||||
|
||||
$ openstack network create net0
|
||||
|
||||
$ openstack subnet create subnet0 \
|
||||
--network net0 \
|
||||
--subnet-range 10.0.4.0/24
|
||||
|
||||
$ openstack network qos policy create policy0
|
||||
|
||||
$ openstack network qos rule create policy0 \
|
||||
--type minimum-packet-rate \
|
||||
--min-kpps 1000000 \
|
||||
--egress
|
||||
|
||||
$ openstack network qos rule create policy0 \
|
||||
--type minimum-packet-rate \
|
||||
--min-kpps 1000000 \
|
||||
--ingress
|
||||
|
||||
Then a normal user can use the pre-created policy to create ports and boot
|
||||
servers with those ports:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
# as an unprivileged user
|
||||
|
||||
# an ordinary soft-switched port: ``--vnic-type normal`` is the default
|
||||
$ openstack port create port-normal-qos \
|
||||
--network net0 \
|
||||
--qos-policy policy0
|
||||
|
||||
$ openstack server create server0 \
|
||||
--os-compute-api-version 2.72 \
|
||||
--flavor cirros256 \
|
||||
--image cirros-0.5.2-x86_64-disk \
|
||||
--port port-normal-qos
|
||||
|
||||
On Healing of Allocations
|
||||
-------------------------
|
||||
|
||||
Since Placement carries a global view of a cloud deployment's resources
|
||||
(what is available, what is used) it may in some conditions get out of sync
|
||||
with reality.
|
||||
|
||||
One important case stems from OpenStack not having distributed transactions to
|
||||
allocate resources provided by multiple OpenStack components (here Nova and
|
||||
Neutron). There are known race conditions in which Placement's view may get
|
||||
out of sync with reality. The design knowingly minimizes the race condition
|
||||
windows, but there are known problems:
|
||||
|
||||
* If a QoS policy is modified after Nova read a port's ``resource_request``
|
||||
but before the port is bound its state before the modification will be
|
||||
applied.
|
||||
* If a bound port with a resource allocation is deleted. The port's allocation
|
||||
is leaked. `<https://bugs.launchpad.net/nova/+bug/1820588>`_
|
||||
|
||||
.. note::
|
||||
|
||||
Deleting a bound port has no known use case. Please consider detaching
|
||||
the interface first by ``openstack server remove port`` instead.
|
||||
|
||||
Incorrect allocations may be fixed by:
|
||||
|
||||
* Moving the server, which will delete the wrong allocation and create the
|
||||
correct allocation. Moving servers fixes local overallocations.
|
||||
* With `placement heal_allocations
|
||||
<https://docs.openstack.org/nova/latest/cli/nova-manage.html#placement-heal-allocations>`_
|
||||
tool.
|
||||
* Manually, by using `openstack resource provider allocation set
|
||||
<https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-allocation-set>`_
|
||||
/`delete <https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-allocation-delete>`_.
|
||||
|
||||
Debugging
|
||||
---------
|
||||
|
||||
* Is Nova running at least Xena release and Neutron at least the Yoga release?
|
||||
|
||||
* Are ``qos-pps-minimum`` and ``port-resource-request-groups`` extensions
|
||||
available?
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ openstack extension show qos-pps-minimum
|
||||
$ openstack extension show port-resource-request-groups
|
||||
|
||||
* Is the ``placement`` service plugin enabled in neutron-server?
|
||||
|
||||
* Is ``resource_provider_packet_processing_with_direction`` or
|
||||
``resource_provider_packet_processing_without_direction`` configured for the
|
||||
relevant neutron agent?
|
||||
|
||||
* Was the agent restarted since changing the configuration file?
|
||||
|
||||
* Is ``resource_provider_packet_processing_with_direction`` or
|
||||
``resource_provider_packet_processing_without_direction`` reaching
|
||||
neutron-server?
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
# as admin
|
||||
$ openstack network agent show ... -c configuration -f json
|
||||
|
||||
Please find an example in section `Propagation of resource information`_.
|
||||
|
||||
* Did neutron-server successfully sync to Placement?
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
# as admin
|
||||
$ openstack network agent show ... | grep resources_synced
|
||||
|
||||
Please find an example in section `Propagation of resource information`_.
|
||||
|
||||
* Is the resource provider tree correct? Is the root a compute host? One level
|
||||
below the agents?
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ openstack --os-placement-api-version 1.17 resource provider list
|
||||
+--------------------------------------+------------------------------------------+------------+--------------------------------------+--------------------------------------+
|
||||
| uuid | name | generation | root_provider_uuid | parent_provider_uuid |
|
||||
+--------------------------------------+------------------------------------------+------------+--------------------------------------+--------------------------------------+
|
||||
| 3b36d91e-bf60-460f-b1f8-3322dee5cdfd | devstack0 | 2 | 3b36d91e-bf60-460f-b1f8-3322dee5cdfd | None |
|
||||
| 89ca1421-5117-5348-acab-6d0e2054239c | devstack0:Open vSwitch agent | 0 | 3b36d91e-bf60-460f-b1f8-3322dee5cdfd | 3b36d91e-bf60-460f-b1f8-3322dee5cdfd |
|
||||
+--------------------------------------+------------------------------------------+------------+--------------------------------------+--------------------------------------+
|
||||
|
||||
* Does Placement have the expected traits?
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
# as admin
|
||||
$ openstack --os-placement-api-version 1.17 trait list | awk '/CUSTOM_/ { print $2 }' | sort
|
||||
CUSTOM_VNIC_TYPE_NORMAL
|
||||
CUSTOM_VNIC_TYPE_SMART_NIC
|
||||
CUSTOM_VNIC_TYPE_VDPA
|
||||
|
||||
* Do the OVS agent resource provider have the proper trait associations and
|
||||
inventories?
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
# as admin
|
||||
$ openstack --os-placement-api-version 1.17 resource provider trait list <RP-UUID>
|
||||
$ openstack --os-placement-api-version 1.17 resource provider inventory list <RP-UUID>
|
||||
|
||||
* Does the QoS policy have a ``minimum-packet-rate`` rule?
|
||||
|
||||
* Does the port have the proper policy?
|
||||
|
||||
* Does the port have a ``resource_request``?
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
# as admin
|
||||
$ openstack port show port-normal-qos | grep resource_request
|
||||
|
||||
* Was the server booted with a port (as opposed to a network)?
|
||||
|
||||
* Did nova allocate resources for the server in Placement?
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
# as admin
|
||||
$ openstack --os-placement-api-version 1.17 resource provider allocation show <SERVER-UUID>
|
||||
|
||||
* Does the allocation have a part on the expected OVS agent resource provider?
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
# as admin
|
||||
$ openstack --os-placement-api-version 1.17 resource provider show --allocations <RP-UUID>
|
||||
|
||||
* Did placement manage to produce an allocation candidate list to nova during
|
||||
scheduling?
|
||||
|
||||
* Did nova manage to schedule the server?
|
||||
|
||||
* Did nova tell neutron which OVS agent resource provider was allocated to
|
||||
satisfy the packet rate request?
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
# as admin
|
||||
$ openstack port show port-normal-qos | grep binding.profile.*allocation
|
||||
|
||||
* Did neutron manage to bind the port?
|
||||
|
||||
Links
|
||||
-----
|
||||
|
||||
* Nova documentation on using a port with ``resource_request``
|
||||
|
||||
* `API Guide <https://docs.openstack.org/api-guide/compute/port_with_resource_request.html>`_
|
||||
* `Admin Guide <https://docs.openstack.org/nova/latest/admin/port_with_resource_request.html>`_
|
||||
|
||||
* Neutron spec: QoS minimum guaranteed packet rate
|
||||
|
||||
* `on specs.openstack.org <https://specs.openstack.org/openstack/neutron-specs/specs/xena/qos-minimum-guaranteed-packet-rate.html>`__
|
||||
* `on review.opendev.org <https://review.opendev.org/785236>`__
|
||||
|
||||
* Nova spec: QoS minimum guaranteed packet rate
|
||||
|
||||
* `on specs.openstack.org
|
||||
<https://specs.openstack.org/openstack/nova-specs/specs/xena/approved/qos-minimum-guaranteed-packet-rate.html>`__
|
||||
* `on review.opendev.org
|
||||
<https://review.opendev.org/785014>`__
|
||||
|
||||
* Relevant OpenStack Networking API references
|
||||
|
||||
* https://docs.openstack.org/api-ref/network/v2/#agent-resources-synced-extension
|
||||
* https://docs.openstack.org/api-ref/network/v2/#port-resource-request
|
||||
* https://docs.openstack.org/api-ref/network/v2/#port-resource-request-groups
|
||||
* https://docs.openstack.org/api-ref/network/v2/#qos-minimum-packet-rate-rules
|
||||
|
||||
* Microversion histories
|
||||
|
||||
* `Compute 2.72
|
||||
<https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#maximum-in-stein>`_
|
||||
* `Placement 1.36
|
||||
<https://docs.openstack.org/placement/latest/placement-api-microversion-history.html#support-same-subtree-queryparam-on-get-allocation-candidates>`_
|
||||
|
||||
* Implementation
|
||||
|
||||
* `on review.opendev.org
|
||||
<https://review.opendev.org/#/q/topic:bp/qos-minimum-guaranteed-packet-rate">`_
|
||||
|
||||
* Known Bugs
|
||||
|
||||
* `Bandwidth resource is leaked
|
||||
<https://bugs.launchpad.net/nova/+bug/1820588>`_ this issue also affects
|
||||
packet rate resources.
|
||||
|
|
@ -30,6 +30,7 @@ Configuration
|
|||
config-ovsfwdriver
|
||||
config-qos
|
||||
config-qos-min-bw
|
||||
config-qos-min-pps
|
||||
config-rbac
|
||||
config-routed-networks
|
||||
config-sfc
|
||||
|
|
|
|||
Loading…
Reference in New Issue