Lots of corrections / clarifications to install guides.
(for Greg W) - replaced abbrev substitutions inside **bold** with plain text as these this combination does not expand. Also in one additional file that came up in search. Some related margin adjustments ... Initial content submit Signed-off-by: Greg Waines <greg.waines@windriver.com> Change-Id: If8805359bf80b0f1359ef58c24493307310e7e28 Signed-off-by: Ron Stone <ronald.stone@windriver.com>
This commit is contained in:
Greg Waines
Ron Stone
parent
53d86a6d49
commit
50bc21226b
@ -64,10 +64,12 @@ Install software on worker nodes
|
||||
Configure worker nodes
|
||||
----------------------
|
||||
|
||||
#. Assign the cluster-host network to the MGMT interface for the worker nodes:
|
||||
#. The MGMT interfaces are partially set up by the network install procedure;
|
||||
configuring the port used for network install as the MGMT port and
|
||||
specifying the attached network of "mgmt".
|
||||
|
||||
(Note that the MGMT interfaces are partially set up automatically by the
|
||||
network install procedure.)
|
||||
Complete the MGMT interface configuration of the worker nodes by specifying
|
||||
the attached network of "cluster-host".
|
||||
|
||||
::
|
||||
|
||||
@ -78,100 +80,175 @@ Configure worker nodes
|
||||
#. Configure data interfaces for worker nodes. Use the DATA port names, for
|
||||
example eth0, that are applicable to your deployment environment.
|
||||
|
||||
This step is optional for Kubernetes: Do this step if using |SRIOV| network
|
||||
attachments in hosted application containers.
|
||||
|
||||
.. only:: starlingx
|
||||
|
||||
.. important::
|
||||
|
||||
This step is **required** for OpenStack.
|
||||
|
||||
|
||||
* Configure the data interfaces
|
||||
|
||||
::
|
||||
|
||||
DATA0IF=<DATA-0-PORT>
|
||||
DATA1IF=<DATA-1-PORT>
|
||||
PHYSNET0='physnet0'
|
||||
PHYSNET1='physnet1'
|
||||
SPL=/tmp/tmp-system-port-list
|
||||
SPIL=/tmp/tmp-system-host-if-list
|
||||
|
||||
# configure the datanetworks in sysinv, prior to referencing it
|
||||
# in the ``system host-if-modify`` command'.
|
||||
system datanetwork-add ${PHYSNET0} vlan
|
||||
system datanetwork-add ${PHYSNET1} vlan
|
||||
|
||||
for NODE in worker-0 worker-1; do
|
||||
echo "Configuring interface for: $NODE"
|
||||
set -ex
|
||||
system host-port-list ${NODE} --nowrap > ${SPL}
|
||||
system host-if-list -a ${NODE} --nowrap > ${SPIL}
|
||||
DATA0PCIADDR=$(cat $SPL | grep $DATA0IF |awk '{print $8}')
|
||||
DATA1PCIADDR=$(cat $SPL | grep $DATA1IF |awk '{print $8}')
|
||||
DATA0PORTUUID=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $2}')
|
||||
DATA1PORTUUID=$(cat $SPL | grep ${DATA1PCIADDR} | awk '{print $2}')
|
||||
DATA0PORTNAME=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $4}')
|
||||
DATA1PORTNAME=$(cat $SPL | grep ${DATA1PCIADDR} | awk '{print $4}')
|
||||
DATA0IFUUID=$(cat $SPIL | awk -v DATA0PORTNAME=$DATA0PORTNAME '($12 ~ DATA0PORTNAME) {print $2}')
|
||||
DATA1IFUUID=$(cat $SPIL | awk -v DATA1PORTNAME=$DATA1PORTNAME '($12 ~ DATA1PORTNAME) {print $2}')
|
||||
system host-if-modify -m 1500 -n data0 -c data ${NODE} ${DATA0IFUUID}
|
||||
system host-if-modify -m 1500 -n data1 -c data ${NODE} ${DATA1IFUUID}
|
||||
system interface-datanetwork-assign ${NODE} ${DATA0IFUUID} ${PHYSNET0}
|
||||
system interface-datanetwork-assign ${NODE} ${DATA1IFUUID} ${PHYSNET1}
|
||||
set +ex
|
||||
done
|
||||
|
||||
* To enable using |SRIOV| network attachments for the above interfaces in
|
||||
Kubernetes hosted application containers:
|
||||
|
||||
* Configure |SRIOV| device plug in:
|
||||
|
||||
::
|
||||
|
||||
for NODE in worker-0 worker-1; do
|
||||
system host-label-assign $NODE sriovdp=enabled
|
||||
done
|
||||
|
||||
* If planning on running |DPDK| in containers on this host, configure the
|
||||
number of 1G Huge pages required on both |NUMA| nodes:
|
||||
|
||||
::
|
||||
|
||||
for NODE in worker-0 worker-1; do
|
||||
|
||||
# assign 10x 1G huge page on processor/numa-node 0 on worker-node to applications
|
||||
system host-memory-modify -f application $NODE 0 -1G 10
|
||||
|
||||
# assign 10x 1G huge page on processor/numa-node 1 on worker-node to applications
|
||||
system host-memory-modify -f application $NODE 1 -1G 10
|
||||
|
||||
done
|
||||
|
||||
|
||||
.. only:: starlingx
|
||||
|
||||
*************************************
|
||||
OpenStack-specific host configuration
|
||||
*************************************
|
||||
|
||||
.. important::
|
||||
|
||||
This step is **required** for OpenStack.
|
||||
**This step is required only if the StarlingX OpenStack application
|
||||
(stx-openstack) will be installed.**
|
||||
|
||||
This step is optional for Kubernetes: Do this step if using SRIOV network
|
||||
attachments in hosted application containers.
|
||||
#. **For OpenStack only:** Assign OpenStack host labels to the worker nodes in
|
||||
support of installing the stx-openstack manifest and helm-charts later.
|
||||
|
||||
For Kubernetes SRIOV network attachments:
|
||||
::
|
||||
|
||||
* Configure SRIOV device plug in:
|
||||
for NODE in worker-0 worker-1; do
|
||||
system host-label-assign $NODE openstack-compute-node=enabled
|
||||
system host-label-assign $NODE openvswitch=enabled
|
||||
system host-label-assign $NODE sriov=enabled
|
||||
done
|
||||
|
||||
::
|
||||
#. **For OpenStack only:** Configure the host settings for the vSwitch.
|
||||
|
||||
system host-label-assign controller-1 sriovdp=enabled
|
||||
**If using OVS-DPDK vswitch, run the following commands:**
|
||||
|
||||
* If planning on running DPDK in containers on this host, configure the number
|
||||
of 1G Huge pages required on both NUMA nodes:
|
||||
Default recommendation for worker node is to use a single core on each
|
||||
numa-node for |OVS|-|DPDK| vswitch. This should have been automatically
|
||||
configured, if not run the following command.
|
||||
|
||||
::
|
||||
::
|
||||
|
||||
system host-memory-modify controller-1 0 -1G 100
|
||||
system host-memory-modify controller-1 1 -1G 100
|
||||
for NODE in worker-0 worker-1; do
|
||||
|
||||
For both Kubernetes and OpenStack:
|
||||
# assign 1 core on processor/numa-node 0 on worker-node to vswitch
|
||||
system host-cpu-modify -f vswitch -p0 1 $NODE
|
||||
|
||||
::
|
||||
# assign 1 core on processor/numa-node 1 on worker-node to vswitch
|
||||
system host-cpu-modify -f vswitch -p1 1 $NODE
|
||||
|
||||
DATA0IF=<DATA-0-PORT>
|
||||
DATA1IF=<DATA-1-PORT>
|
||||
PHYSNET0='physnet0'
|
||||
PHYSNET1='physnet1'
|
||||
SPL=/tmp/tmp-system-port-list
|
||||
SPIL=/tmp/tmp-system-host-if-list
|
||||
done
|
||||
|
||||
# configure the datanetworks in sysinv, prior to referencing it
|
||||
# in the ``system host-if-modify`` command'.
|
||||
system datanetwork-add ${PHYSNET0} vlan
|
||||
system datanetwork-add ${PHYSNET1} vlan
|
||||
|
||||
for NODE in worker-0 worker-1; do
|
||||
echo "Configuring interface for: $NODE"
|
||||
set -ex
|
||||
system host-port-list ${NODE} --nowrap > ${SPL}
|
||||
system host-if-list -a ${NODE} --nowrap > ${SPIL}
|
||||
DATA0PCIADDR=$(cat $SPL | grep $DATA0IF |awk '{print $8}')
|
||||
DATA1PCIADDR=$(cat $SPL | grep $DATA1IF |awk '{print $8}')
|
||||
DATA0PORTUUID=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $2}')
|
||||
DATA1PORTUUID=$(cat $SPL | grep ${DATA1PCIADDR} | awk '{print $2}')
|
||||
DATA0PORTNAME=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $4}')
|
||||
DATA1PORTNAME=$(cat $SPL | grep ${DATA1PCIADDR} | awk '{print $4}')
|
||||
DATA0IFUUID=$(cat $SPIL | awk -v DATA0PORTNAME=$DATA0PORTNAME '($12 ~ DATA0PORTNAME) {print $2}')
|
||||
DATA1IFUUID=$(cat $SPIL | awk -v DATA1PORTNAME=$DATA1PORTNAME '($12 ~ DATA1PORTNAME) {print $2}')
|
||||
system host-if-modify -m 1500 -n data0 -c data ${NODE} ${DATA0IFUUID}
|
||||
system host-if-modify -m 1500 -n data1 -c data ${NODE} ${DATA1IFUUID}
|
||||
system interface-datanetwork-assign ${NODE} ${DATA0IFUUID} ${PHYSNET0}
|
||||
system interface-datanetwork-assign ${NODE} ${DATA1IFUUID} ${PHYSNET1}
|
||||
set +ex
|
||||
done
|
||||
When using |OVS|-|DPDK|, configure 1x 1G huge page for vSwitch memory on
|
||||
each |NUMA| node where vswitch is running on this host, with the
|
||||
following command:
|
||||
|
||||
*************************************
|
||||
OpenStack-specific host configuration
|
||||
*************************************
|
||||
::
|
||||
|
||||
.. important::
|
||||
for NODE in worker-0 worker-1; do
|
||||
|
||||
**This step is required only if the StarlingX OpenStack application
|
||||
(stx-openstack) will be installed.**
|
||||
# assign 1x 1G huge page on processor/numa-node 0 on worker-node to vswitch
|
||||
system host-memory-modify -f vswitch -1G 1 $NODE 0
|
||||
|
||||
#. **For OpenStack only:** Assign OpenStack host labels to the worker nodes in
|
||||
support of installing the stx-openstack manifest and helm-charts later.
|
||||
# assign 1x 1G huge page on processor/numa-node 0 on worker-node to vswitch
|
||||
system host-memory-modify -f vswitch -1G 1 $NODE 1
|
||||
|
||||
::
|
||||
done
|
||||
|
||||
for NODE in worker-0 worker-1; do
|
||||
system host-label-assign $NODE openstack-compute-node=enabled
|
||||
system host-label-assign $NODE openvswitch=enabled
|
||||
system host-label-assign $NODE sriov=enabled
|
||||
done
|
||||
|
||||
#. **For OpenStack only:** Setup disk partition for nova-local volume group,
|
||||
needed for stx-openstack nova ephemeral disks.
|
||||
.. important::
|
||||
|
||||
::
|
||||
|VMs| created in an |OVS|-|DPDK| environment must be configured to use
|
||||
huge pages to enable networking and must use a flavor with property:
|
||||
hw:mem_page_size=large
|
||||
|
||||
for NODE in worker-0 worker-1; do
|
||||
echo "Configuring Nova local for: $NODE"
|
||||
ROOT_DISK=$(system host-show ${NODE} | grep rootfs | awk '{print $4}')
|
||||
ROOT_DISK_UUID=$(system host-disk-list ${NODE} --nowrap | grep ${ROOT_DISK} | awk '{print $2}')
|
||||
PARTITION_SIZE=10
|
||||
NOVA_PARTITION=$(system host-disk-partition-add -t lvm_phys_vol ${NODE} ${ROOT_DISK_UUID} ${PARTITION_SIZE})
|
||||
NOVA_PARTITION_UUID=$(echo ${NOVA_PARTITION} | grep -ow "| uuid | [a-z0-9\-]* |" | awk '{print $4}')
|
||||
system host-lvg-add ${NODE} nova-local
|
||||
system host-pv-add ${NODE} nova-local ${NOVA_PARTITION_UUID}
|
||||
done
|
||||
Configure the huge pages for |VMs| in an |OVS|-|DPDK| environment for
|
||||
this host with the command:
|
||||
|
||||
::
|
||||
|
||||
for NODE in worker-0 worker-1; do
|
||||
|
||||
# assign 10x 1G huge page on processor/numa-node 0 on worker-node to applications
|
||||
system host-memory-modify -f application -1G 10 $NODE 0
|
||||
|
||||
# assign 10x 1G huge page on processor/numa-node 1 on worker-node to applications
|
||||
system host-memory-modify -f application -1G 10 $NODE 1
|
||||
|
||||
done
|
||||
|
||||
#. **For OpenStack only:** Setup disk partition for nova-local volume group,
|
||||
needed for stx-openstack nova ephemeral disks.
|
||||
|
||||
::
|
||||
|
||||
for NODE in worker-0 worker-1; do
|
||||
echo "Configuring Nova local for: $NODE"
|
||||
ROOT_DISK=$(system host-show ${NODE} | grep rootfs | awk '{print $4}')
|
||||
ROOT_DISK_UUID=$(system host-disk-list ${NODE} --nowrap | grep ${ROOT_DISK} | awk '{print $2}')
|
||||
PARTITION_SIZE=10
|
||||
NOVA_PARTITION=$(system host-disk-partition-add -t lvm_phys_vol ${NODE} ${ROOT_DISK_UUID} ${PARTITION_SIZE})
|
||||
NOVA_PARTITION_UUID=$(echo ${NOVA_PARTITION} | grep -ow "| uuid | [a-z0-9\-]* |" | awk '{print $4}')
|
||||
system host-lvg-add ${NODE} nova-local
|
||||
system host-pv-add ${NODE} nova-local ${NOVA_PARTITION_UUID}
|
||||
done
|
||||
|
||||
|
||||
-------------------
|
||||
|
||||
@ -171,21 +171,32 @@ Configure controller-0
|
||||
|
||||
source /etc/platform/openrc
|
||||
|
||||
#. Configure the |OAM| and MGMT interfaces of controller-0 and specify the
|
||||
attached networks. Use the |OAM| and MGMT port names, for example eth0, that
|
||||
are applicable to your deployment environment.
|
||||
#. Configure the |OAM| interface of controller-0 and specify the
|
||||
attached network as "oam".
|
||||
|
||||
Use the |OAM| port name that is applicable to your deployment environment,
|
||||
for example eth0:
|
||||
|
||||
::
|
||||
|
||||
OAM_IF=<OAM-PORT>
|
||||
system host-if-modify controller-0 $OAM_IF -c platform
|
||||
system interface-network-assign controller-0 $OAM_IF oam
|
||||
|
||||
#. Configure the MGMT interface of controller-0 and specify the attached
|
||||
networks of both "mgmt" and "cluster-host".
|
||||
|
||||
Use the MGMT port name that is applicable to your deployment environment,
|
||||
for example eth1:
|
||||
|
||||
::
|
||||
|
||||
MGMT_IF=<MGMT-PORT>
|
||||
system host-if-modify controller-0 lo -c none
|
||||
IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if ($6=="lo") print $4;}')
|
||||
for UUID in $IFNET_UUIDS; do
|
||||
system interface-network-remove ${UUID}
|
||||
done
|
||||
system host-if-modify controller-0 $OAM_IF -c platform
|
||||
system interface-network-assign controller-0 $OAM_IF oam
|
||||
system host-if-modify controller-0 $MGMT_IF -c platform
|
||||
system interface-network-assign controller-0 $MGMT_IF mgmt
|
||||
system interface-network-assign controller-0 $MGMT_IF cluster-host
|
||||
@ -196,26 +207,83 @@ Configure controller-0
|
||||
|
||||
system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org
|
||||
|
||||
**************************************************************
|
||||
Optionally, initialize a Ceph-based Persistent Storage Backend
|
||||
**************************************************************
|
||||
#. Configure data interfaces for controller-0. Use the DATA port names, for example
|
||||
eth0, applicable to your deployment environment.
|
||||
|
||||
.. important::
|
||||
This step is optional for Kubernetes: Do this step if using |SRIOV| network
|
||||
attachments in hosted application containers.
|
||||
|
||||
A persistent storage backend is required if your application requires
|
||||
Persistent Volume Claims (PVCs). The StarlingX OpenStack application
|
||||
(stx-openstack) requires |PVCs|, therefore if you plan on using the
|
||||
stx-openstack application, then you must configure a persistent storage
|
||||
backend.
|
||||
.. important::
|
||||
|
||||
.. only:: starlingx
|
||||
This step is **required** for OpenStack.
|
||||
|
||||
There are two options for persistent storage backend:
|
||||
1) the host-based Ceph solution and
|
||||
2) the Rook container-based Ceph solution.
|
||||
|
||||
The Rook container-based Ceph backend is installed after both
|
||||
AIO-Controllers are configured and unlocked.
|
||||
* Configure the data interfaces
|
||||
|
||||
::
|
||||
|
||||
DATA0IF=<DATA-0-PORT>
|
||||
DATA1IF=<DATA-1-PORT>
|
||||
export NODE=controller-0
|
||||
PHYSNET0='physnet0'
|
||||
PHYSNET1='physnet1'
|
||||
SPL=/tmp/tmp-system-port-list
|
||||
SPIL=/tmp/tmp-system-host-if-list
|
||||
system host-port-list ${NODE} --nowrap > ${SPL}
|
||||
system host-if-list -a ${NODE} --nowrap > ${SPIL}
|
||||
DATA0PCIADDR=$(cat $SPL | grep $DATA0IF |awk '{print $8}')
|
||||
DATA1PCIADDR=$(cat $SPL | grep $DATA1IF |awk '{print $8}')
|
||||
DATA0PORTUUID=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $2}')
|
||||
DATA1PORTUUID=$(cat $SPL | grep ${DATA1PCIADDR} | awk '{print $2}')
|
||||
DATA0PORTNAME=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $4}')
|
||||
DATA1PORTNAME=$(cat $SPL | grep ${DATA1PCIADDR} | awk '{print $4}')
|
||||
DATA0IFUUID=$(cat $SPIL | awk -v DATA0PORTNAME=$DATA0PORTNAME '($12 ~ DATA0PORTNAME) {print $2}')
|
||||
DATA1IFUUID=$(cat $SPIL | awk -v DATA1PORTNAME=$DATA1PORTNAME '($12 ~ DATA1PORTNAME) {print $2}')
|
||||
|
||||
system datanetwork-add ${PHYSNET0} vlan
|
||||
system datanetwork-add ${PHYSNET1} vlan
|
||||
|
||||
system host-if-modify -m 1500 -n data0 -c data ${NODE} ${DATA0IFUUID}
|
||||
system host-if-modify -m 1500 -n data1 -c data ${NODE} ${DATA1IFUUID}
|
||||
system interface-datanetwork-assign ${NODE} ${DATA0IFUUID} ${PHYSNET0}
|
||||
system interface-datanetwork-assign ${NODE} ${DATA1IFUUID} ${PHYSNET1}
|
||||
|
||||
* To enable using |SRIOV| network attachments for the above interfaces in
|
||||
Kubernetes hosted application containers:
|
||||
|
||||
* Configure the Kubernetes |SRIOV| device plugin.
|
||||
|
||||
::
|
||||
|
||||
system host-label-assign controller-0 sriovdp=enabled
|
||||
|
||||
* If planning on running |DPDK| in kuberentes hosted appliction containers
|
||||
on this host, configure the number of 1G Huge pages required on both
|
||||
|NUMA| nodes.
|
||||
|
||||
::
|
||||
|
||||
# assign 10x 1G huge page on processor/numa-node 0 on controller-0 to applications
|
||||
system host-memory-modify -f application controller-0 0 -1G 10
|
||||
|
||||
# assign 10x 1G huge page on processor/numa-node 1 on controller-0 to applications
|
||||
system host-memory-modify -f application controller-0 1 -1G 10
|
||||
|
||||
|
||||
***************************************************************
|
||||
If required, initialize a Ceph-based Persistent Storage Backend
|
||||
***************************************************************
|
||||
|
||||
A persistent storage backend is required if your application requires |PVCs|.
|
||||
|
||||
.. only:: starlingx
|
||||
|
||||
.. important::
|
||||
|
||||
The StarlingX OpenStack application **requires** |PVCs|.
|
||||
|
||||
There are two options for persistent storage backend: the host-based Ceph
|
||||
solution and the Rook container-based Ceph solution.
|
||||
|
||||
For host-based Ceph:
|
||||
|
||||
@ -251,74 +319,17 @@ For host-based Ceph:
|
||||
system host-label-assign controller-0 ceph-mon-placement=enabled
|
||||
system host-label-assign controller-0 ceph-mgr-placement=enabled
|
||||
|
||||
#. Configure data interfaces for controller-0. Use the DATA port names, for example
|
||||
eth0, applicable to your deployment environment.
|
||||
|
||||
.. important::
|
||||
***********************************
|
||||
If required, configure Docker Proxy
|
||||
***********************************
|
||||
|
||||
This step is **required** for OpenStack.
|
||||
StarlingX uses publicly available container runtime registries. If you are
|
||||
behind a corporate firewall or proxy, you need to set docker proxy settings.
|
||||
|
||||
This step is optional for Kubernetes: Do this step if using |SRIOV| network
|
||||
attachments in hosted application containers.
|
||||
Refer to :ref:`Docker Proxy Configuration <docker_proxy_config>` for
|
||||
details about configuring Docker proxy settings.
|
||||
|
||||
For Kubernetes |SRIOV| network attachments:
|
||||
|
||||
* Configure the |SRIOV| device plugin.
|
||||
|
||||
::
|
||||
|
||||
system host-label-assign controller-0 sriovdp=enabled
|
||||
|
||||
* If planning on running |DPDK| in containers on this host, configure the number
|
||||
of 1G Huge pages required on both |NUMA| nodes.
|
||||
|
||||
::
|
||||
|
||||
system host-memory-modify controller-0 0 -1G 100
|
||||
system host-memory-modify controller-0 1 -1G 100
|
||||
|
||||
|
||||
For both Kubernetes and OpenStack:
|
||||
|
||||
::
|
||||
|
||||
DATA0IF=<DATA-0-PORT>
|
||||
DATA1IF=<DATA-1-PORT>
|
||||
export NODE=controller-0
|
||||
PHYSNET0='physnet0'
|
||||
PHYSNET1='physnet1'
|
||||
SPL=/tmp/tmp-system-port-list
|
||||
SPIL=/tmp/tmp-system-host-if-list
|
||||
system host-port-list ${NODE} --nowrap > ${SPL}
|
||||
system host-if-list -a ${NODE} --nowrap > ${SPIL}
|
||||
DATA0PCIADDR=$(cat $SPL | grep $DATA0IF |awk '{print $8}')
|
||||
DATA1PCIADDR=$(cat $SPL | grep $DATA1IF |awk '{print $8}')
|
||||
DATA0PORTUUID=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $2}')
|
||||
DATA1PORTUUID=$(cat $SPL | grep ${DATA1PCIADDR} | awk '{print $2}')
|
||||
DATA0PORTNAME=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $4}')
|
||||
DATA1PORTNAME=$(cat $SPL | grep ${DATA1PCIADDR} | awk '{print $4}')
|
||||
DATA0IFUUID=$(cat $SPIL | awk -v DATA0PORTNAME=$DATA0PORTNAME '($12 ~ DATA0PORTNAME) {print $2}')
|
||||
DATA1IFUUID=$(cat $SPIL | awk -v DATA1PORTNAME=$DATA1PORTNAME '($12 ~ DATA1PORTNAME) {print $2}')
|
||||
|
||||
system datanetwork-add ${PHYSNET0} vlan
|
||||
system datanetwork-add ${PHYSNET1} vlan
|
||||
|
||||
system host-if-modify -m 1500 -n data0 -c data ${NODE} ${DATA0IFUUID}
|
||||
system host-if-modify -m 1500 -n data1 -c data ${NODE} ${DATA1IFUUID}
|
||||
system interface-datanetwork-assign ${NODE} ${DATA0IFUUID} ${PHYSNET0}
|
||||
system interface-datanetwork-assign ${NODE} ${DATA1IFUUID} ${PHYSNET1}
|
||||
|
||||
#. If required, and not already done as part of bootstrap, configure Docker to
|
||||
use a proxy server.
|
||||
|
||||
#. List Docker proxy parameters:
|
||||
|
||||
::
|
||||
|
||||
system service-parameter-list platform docker
|
||||
|
||||
#. Refer to :ref:`Docker Proxy Configuration <docker_proxy_config>` for
|
||||
details about Docker proxy settings.
|
||||
|
||||
.. only:: starlingx
|
||||
|
||||
@ -326,7 +337,116 @@ For host-based Ceph:
|
||||
OpenStack-specific host configuration
|
||||
*************************************
|
||||
|
||||
.. include:: inc-openstack-specific-host-config.rest
|
||||
.. important::
|
||||
|
||||
**This step is required only if the StarlingX OpenStack application
|
||||
(stx-openstack) will be installed.**
|
||||
|
||||
#. **For OpenStack only:** Assign OpenStack host labels to controller-0 in
|
||||
support of installing the stx-openstack manifest and helm-charts later.
|
||||
|
||||
::
|
||||
|
||||
system host-label-assign controller-0 openstack-control-plane=enabled
|
||||
system host-label-assign controller-0 openstack-compute-node=enabled
|
||||
system host-label-assign controller-0 openvswitch=enabled
|
||||
system host-label-assign controller-0 sriov=enabled
|
||||
|
||||
#. **For OpenStack only:** Configure the system setting for the vSwitch.
|
||||
|
||||
StarlingX has |OVS| (kernel-based) vSwitch configured as default:
|
||||
|
||||
* Runs in a container; defined within the helm charts of stx-openstack
|
||||
manifest.
|
||||
* Shares the core(s) assigned to the platform.
|
||||
|
||||
If you require better performance, |OVS|-|DPDK| (|OVS| with the Data
|
||||
Plane Development Kit, which is supported only on bare metal hardware)
|
||||
should be used:
|
||||
|
||||
* Runs directly on the host (it is not containerized).
|
||||
* Requires that at least 1 core be assigned/dedicated to the vSwitch function.
|
||||
|
||||
**To deploy the default containerized OVS:**
|
||||
|
||||
::
|
||||
|
||||
system modify --vswitch_type none
|
||||
|
||||
This does not run any vSwitch directly on the host, instead, it uses the
|
||||
containerized |OVS| defined in the helm charts of stx-openstack
|
||||
manifest.
|
||||
|
||||
**To deploy OVS-DPDK, run the following command:**
|
||||
|
||||
::
|
||||
|
||||
system modify --vswitch_type ovs-dpdk
|
||||
|
||||
Default recommendation for an |AIO|-controller is to use a single core
|
||||
for |OVS|-|DPDK| vswitch.
|
||||
|
||||
::
|
||||
|
||||
# assign 1 core on processor/numa-node 0 on controller-0 to vswitch
|
||||
system host-cpu-modify -f vswitch -p0 1 controller-0
|
||||
|
||||
Once vswitch_type is set to |OVS|-|DPDK|, any subsequent nodes created
|
||||
will default to automatically assigning 1 vSwitch core for |AIO|
|
||||
controllers and 2 vSwitch cores for compute-labeled worker nodes.
|
||||
|
||||
|
||||
When using |OVS|-|DPDK|, configure 1x 1G huge page for vSwitch memory on
|
||||
each |NUMA| node where vswitch is running on this host, with the
|
||||
following command:
|
||||
|
||||
::
|
||||
|
||||
# assign 1x 1G huge page on processor/numa-node 0 on controller-0 to vswitch
|
||||
system host-memory-modify -f vswitch -1G 1 controller-0 0
|
||||
|
||||
|
||||
.. important::
|
||||
|
||||
|VMs| created in an |OVS|-|DPDK| environment must be configured to use
|
||||
huge pages to enable networking and must use a flavor with property:
|
||||
hw:mem_page_size=large
|
||||
|
||||
Configure the huge pages for |VMs| in an |OVS|-|DPDK| environment on this host with
|
||||
the commands:
|
||||
|
||||
::
|
||||
|
||||
# assign 10x 1G huge page on processor/numa-node 0 on controller-0 to applications
|
||||
system host-memory-modify -f application -1G 10 controller-0 0
|
||||
|
||||
# assign 10x 1G huge page on processor/numa-node 1 on controller-0 to applications
|
||||
system host-memory-modify -f application -1G 10 controller-0 1
|
||||
|
||||
.. note::
|
||||
|
||||
After controller-0 is unlocked, changing vswitch_type requires
|
||||
locking and unlocking controller-0 to apply the change.
|
||||
|
||||
#. **For OpenStack only:** Set up disk partition for nova-local volume
|
||||
group, which is needed for stx-openstack nova ephemeral disks.
|
||||
|
||||
::
|
||||
|
||||
export NODE=controller-0
|
||||
|
||||
echo ">>> Getting root disk info"
|
||||
ROOT_DISK=$(system host-show ${NODE} | grep rootfs | awk '{print $4}')
|
||||
ROOT_DISK_UUID=$(system host-disk-list ${NODE} --nowrap | grep ${ROOT_DISK} | awk '{print $2}')
|
||||
echo "Root disk: $ROOT_DISK, UUID: $ROOT_DISK_UUID"
|
||||
|
||||
echo ">>>> Configuring nova-local"
|
||||
NOVA_SIZE=34
|
||||
NOVA_PARTITION=$(system host-disk-partition-add -t lvm_phys_vol ${NODE} ${ROOT_DISK_UUID} ${NOVA_SIZE})
|
||||
NOVA_PARTITION_UUID=$(echo ${NOVA_PARTITION} | grep -ow "| uuid | [a-z0-9\-]* |" | awk '{print $4}')
|
||||
system host-lvg-add ${NODE} nova-local
|
||||
system host-pv-add ${NODE} nova-local ${NOVA_PARTITION_UUID}
|
||||
sleep 2
|
||||
|
||||
|
||||
-------------------
|
||||
@ -366,8 +486,9 @@ Install software on controller-1 node
|
||||
|
||||
system host-update 2 personality=controller
|
||||
|
||||
#. Wait for the software installation on controller-1 to complete, for controller-1 to
|
||||
reboot, and for controller-1 to show as locked/disabled/online in 'system host-list'.
|
||||
#. Wait for the software installation on controller-1 to complete, for
|
||||
controller-1 to reboot, and for controller-1 to show as
|
||||
locked/disabled/online in 'system host-list'.
|
||||
|
||||
This can take 5-10 minutes, depending on the performance of the host machine.
|
||||
|
||||
@ -385,82 +506,96 @@ Install software on controller-1 node
|
||||
Configure controller-1
|
||||
----------------------
|
||||
|
||||
#. Configure the |OAM| and MGMT interfaces of controller-1 and specify the
|
||||
attached networks. Use the |OAM| and MGMT port names, for example eth0, that are
|
||||
applicable to your deployment environment:
|
||||
#. Configure the |OAM| interface of controller-1 and specify the
|
||||
attached network of "oam".
|
||||
|
||||
(Note that the MGMT interface is partially set up automatically by the network
|
||||
install procedure.)
|
||||
Use the |OAM| port name that is applicable to your deployment environment,
|
||||
for example eth0:
|
||||
|
||||
::
|
||||
|
||||
OAM_IF=<OAM-PORT>
|
||||
MGMT_IF=<MGMT-PORT>
|
||||
system host-if-modify controller-1 $OAM_IF -c platform
|
||||
system interface-network-assign controller-1 $OAM_IF oam
|
||||
|
||||
#. The MGMT interface is partially set up by the network install procedure;
|
||||
configuring the port used for network install as the MGMT port and
|
||||
specifying the attached network of "mgmt".
|
||||
|
||||
Complete the MGMT interface configuration of controller-1 by specifying the
|
||||
attached network of "cluster-host".
|
||||
|
||||
::
|
||||
|
||||
system interface-network-assign controller-1 mgmt0 cluster-host
|
||||
|
||||
#. Configure data interfaces for controller-1. Use the DATA port names, for example
|
||||
eth0, applicable to your deployment environment.
|
||||
#. Configure data interfaces for controller-1. Use the DATA port names, for
|
||||
example eth0, applicable to your deployment environment.
|
||||
|
||||
This step is optional for Kubernetes. Do this step if using |SRIOV|
|
||||
network attachments in hosted application containers.
|
||||
|
||||
.. important::
|
||||
|
||||
This step is **required** for OpenStack.
|
||||
|
||||
This step is optional for Kubernetes: Do this step if using |SRIOV|
|
||||
network attachments in hosted application containers.
|
||||
|
||||
For Kubernetes |SRIOV| network attachments:
|
||||
|
||||
* Configure the |SRIOV| device plugin:
|
||||
* Configure the data interfaces
|
||||
|
||||
::
|
||||
|
||||
system host-label-assign controller-1 sriovdp=enabled
|
||||
DATA0IF=<DATA-0-PORT>
|
||||
DATA1IF=<DATA-1-PORT>
|
||||
export NODE=controller-1
|
||||
PHYSNET0='physnet0'
|
||||
PHYSNET1='physnet1'
|
||||
SPL=/tmp/tmp-system-port-list
|
||||
SPIL=/tmp/tmp-system-host-if-list
|
||||
system host-port-list ${NODE} --nowrap > ${SPL}
|
||||
system host-if-list -a ${NODE} --nowrap > ${SPIL}
|
||||
DATA0PCIADDR=$(cat $SPL | grep $DATA0IF |awk '{print $8}')
|
||||
DATA1PCIADDR=$(cat $SPL | grep $DATA1IF |awk '{print $8}')
|
||||
DATA0PORTUUID=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $2}')
|
||||
DATA1PORTUUID=$(cat $SPL | grep ${DATA1PCIADDR} | awk '{print $2}')
|
||||
DATA0PORTNAME=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $4}')
|
||||
DATA1PORTNAME=$(cat $SPL | grep ${DATA1PCIADDR} | awk '{print $4}')
|
||||
DATA0IFUUID=$(cat $SPIL | awk -v DATA0PORTNAME=$DATA0PORTNAME '($12 ~ DATA0PORTNAME) {print $2}')
|
||||
DATA1IFUUID=$(cat $SPIL | awk -v DATA1PORTNAME=$DATA1PORTNAME '($12 ~ DATA1PORTNAME) {print $2}')
|
||||
|
||||
* If planning on running DPDK in containers on this host, configure the number
|
||||
of 1G Huge pages required on both NUMA nodes:
|
||||
system datanetwork-add ${PHYSNET0} vlan
|
||||
system datanetwork-add ${PHYSNET1} vlan
|
||||
|
||||
::
|
||||
system host-if-modify -m 1500 -n data0 -c data ${NODE} ${DATA0IFUUID}
|
||||
system host-if-modify -m 1500 -n data1 -c data ${NODE} ${DATA1IFUUID}
|
||||
system interface-datanetwork-assign ${NODE} ${DATA0IFUUID} ${PHYSNET0}
|
||||
system interface-datanetwork-assign ${NODE} ${DATA1IFUUID} ${PHYSNET1}
|
||||
|
||||
system host-memory-modify controller-1 0 -1G 100
|
||||
system host-memory-modify controller-1 1 -1G 100
|
||||
* To enable using |SRIOV| network attachments for the above interfaes in
|
||||
Kubernetes hosted application containers:
|
||||
|
||||
.. only:: starlingx
|
||||
* Configure the Kubernetes |SRIOV| device plugin:
|
||||
|
||||
For both Kubernetes and OpenStack:
|
||||
::
|
||||
|
||||
::
|
||||
system host-label-assign controller-1 sriovdp=enabled
|
||||
|
||||
DATA0IF=<DATA-0-PORT>
|
||||
DATA1IF=<DATA-1-PORT>
|
||||
export NODE=controller-1
|
||||
PHYSNET0='physnet0'
|
||||
PHYSNET1='physnet1'
|
||||
SPL=/tmp/tmp-system-port-list
|
||||
SPIL=/tmp/tmp-system-host-if-list
|
||||
system host-port-list ${NODE} --nowrap > ${SPL}
|
||||
system host-if-list -a ${NODE} --nowrap > ${SPIL}
|
||||
DATA0PCIADDR=$(cat $SPL | grep $DATA0IF |awk '{print $8}')
|
||||
DATA1PCIADDR=$(cat $SPL | grep $DATA1IF |awk '{print $8}')
|
||||
DATA0PORTUUID=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $2}')
|
||||
DATA1PORTUUID=$(cat $SPL | grep ${DATA1PCIADDR} | awk '{print $2}')
|
||||
DATA0PORTNAME=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $4}')
|
||||
DATA1PORTNAME=$(cat $SPL | grep ${DATA1PCIADDR} | awk '{print $4}')
|
||||
DATA0IFUUID=$(cat $SPIL | awk -v DATA0PORTNAME=$DATA0PORTNAME '($12 ~ DATA0PORTNAME) {print $2}')
|
||||
DATA1IFUUID=$(cat $SPIL | awk -v DATA1PORTNAME=$DATA1PORTNAME '($12 ~ DATA1PORTNAME) {print $2}')
|
||||
* If planning on running |DPDK| in Kubernetes hosted application containers
|
||||
on this host, configure the number of 1G Huge pages required on both
|
||||
|NUMA| nodes:
|
||||
|
||||
system datanetwork-add ${PHYSNET0} vlan
|
||||
system datanetwork-add ${PHYSNET1} vlan
|
||||
::
|
||||
|
||||
system host-if-modify -m 1500 -n data0 -c data ${NODE} ${DATA0IFUUID}
|
||||
system host-if-modify -m 1500 -n data1 -c data ${NODE} ${DATA1IFUUID}
|
||||
system interface-datanetwork-assign ${NODE} ${DATA0IFUUID} ${PHYSNET0}
|
||||
system interface-datanetwork-assign ${NODE} ${DATA1IFUUID} ${PHYSNET1}
|
||||
# assign 10x 1G huge page on processor/numa-node 0 on controller-0 to applications
|
||||
system host-memory-modify -f application controller-1 0 -1G 10
|
||||
|
||||
*************************************************************************************
|
||||
Optionally, configure host-specific details for Ceph-based Persistent Storage Backend
|
||||
*************************************************************************************
|
||||
# assign 10x 1G huge page on processor/numa-node 1 on controller-0 to applications
|
||||
system host-memory-modify -f application controller-1 1 -1G 10
|
||||
|
||||
|
||||
|
||||
***************************************************************************************
|
||||
If configuring a Ceph-based Persistent Storage Backend, configure host-specific details
|
||||
***************************************************************************************
|
||||
|
||||
For host-based Ceph:
|
||||
|
||||
@ -503,6 +638,48 @@ For host-based Ceph:
|
||||
system host-label-assign controller-1 openvswitch=enabled
|
||||
system host-label-assign controller-1 sriov=enabled
|
||||
|
||||
#. **For OpenStack only:** Configure the host settings for the vSwitch.
|
||||
|
||||
**If using OVS-DPDK vswitch, run the following commands:**
|
||||
|
||||
Default recommendation for an AIO-controller is to use a single core
|
||||
for |OVS|-|DPDK| vswitch. This should have been automatically configured,
|
||||
if not run the following command.
|
||||
|
||||
::
|
||||
|
||||
# assign 1 core on processor/numa-node 0 on controller-1 to vswitch
|
||||
system host-cpu-modify -f vswitch -p0 1 controller-1
|
||||
|
||||
|
||||
When using |OVS|-|DPDK|, configure 1x 1G huge page for vSwitch memory on
|
||||
each |NUMA| node where vswitch is running on this host, with the
|
||||
following command:
|
||||
|
||||
::
|
||||
|
||||
# assign 1x 1G huge page on processor/numa-node 0 on controller-1 to vswitch
|
||||
system host-memory-modify -f vswitch -1G 1 controller-1 0
|
||||
|
||||
|
||||
.. important::
|
||||
|
||||
|VMs| created in an |OVS|-|DPDK| environment must be configured to use
|
||||
huge pages to enable networking and must use a flavor with property:
|
||||
hw:mem_page_size=large
|
||||
|
||||
Configure the huge pages for |VMs| in an |OVS|-|DPDK| environment for
|
||||
this host with the command:
|
||||
|
||||
::
|
||||
|
||||
# assign 10x 1G huge page on processor/numa-node 0 on controller-1 to applications
|
||||
system host-memory-modify -f application -1G 10 controller-1 0
|
||||
|
||||
# assign 10x 1G huge page on processor/numa-node 1 on controller-1 to applications
|
||||
system host-memory-modify -f application -1G 10 controller-1 1
|
||||
|
||||
|
||||
#. **For OpenStack only:** Set up disk partition for nova-local volume group,
|
||||
which is needed for stx-openstack nova ephemeral disks.
|
||||
|
||||
@ -539,15 +716,13 @@ machine.
|
||||
|
||||
.. only:: starlingx
|
||||
|
||||
--------------------------------------------------------------------------
|
||||
Optionally, finish configuration of Ceph-based Persistent Storage Backend
|
||||
--------------------------------------------------------------------------
|
||||
|
||||
For host-based Ceph: Nothing else is required.
|
||||
-----------------------------------------------------------------------------------------------
|
||||
If using Rook container-based Ceph, finish configuring the ceph-rook Persistent Storage Backend
|
||||
-----------------------------------------------------------------------------------------------
|
||||
|
||||
For Rook container-based Ceph:
|
||||
|
||||
On **virtual** controller-0 and controller-1:
|
||||
On active controller:
|
||||
|
||||
#. Wait for the ``rook-ceph-apps`` application to be uploaded
|
||||
|
||||
|
||||
@ -188,16 +188,90 @@ The newly installed controller needs to be configured.
|
||||
|
||||
system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org
|
||||
|
||||
**************************************************************
|
||||
Optionally, initialize a Ceph-based Persistent Storage Backend
|
||||
**************************************************************
|
||||
#. Configure data interfaces for controller-0. Use the DATA port names, for example
|
||||
eth0, applicable to your deployment environment.
|
||||
|
||||
This step is **optional** for Kubernetes. Do this step if using |SRIOV|
|
||||
network attachments in hosted application containers.
|
||||
|
||||
.. only:: starlingx
|
||||
|
||||
.. important::
|
||||
|
||||
This step is **required** for OpenStack.
|
||||
|
||||
* Configure the data interfaces
|
||||
|
||||
::
|
||||
|
||||
DATA0IF=<DATA-0-PORT>
|
||||
DATA1IF=<DATA-1-PORT>
|
||||
export NODE=controller-0
|
||||
PHYSNET0='physnet0'
|
||||
PHYSNET1='physnet1'
|
||||
SPL=/tmp/tmp-system-port-list
|
||||
SPIL=/tmp/tmp-system-host-if-list
|
||||
system host-port-list ${NODE} --nowrap > ${SPL}
|
||||
system host-if-list -a ${NODE} --nowrap > ${SPIL}
|
||||
DATA0PCIADDR=$(cat $SPL | grep $DATA0IF |awk '{print $8}')
|
||||
DATA1PCIADDR=$(cat $SPL | grep $DATA1IF |awk '{print $8}')
|
||||
DATA0PORTUUID=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $2}')
|
||||
DATA1PORTUUID=$(cat $SPL | grep ${DATA1PCIADDR} | awk '{print $2}')
|
||||
DATA0PORTNAME=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $4}')
|
||||
DATA1PORTNAME=$(cat $SPL | grep ${DATA1PCIADDR} | awk '{print $4}')
|
||||
DATA0IFUUID=$(cat $SPIL | awk -v DATA0PORTNAME=$DATA0PORTNAME '($12 ~ DATA0PORTNAME) {print $2}')
|
||||
DATA1IFUUID=$(cat $SPIL | awk -v DATA1PORTNAME=$DATA1PORTNAME '($12 ~ DATA1PORTNAME) {print $2}')
|
||||
|
||||
system datanetwork-add ${PHYSNET0} vlan
|
||||
system datanetwork-add ${PHYSNET1} vlan
|
||||
|
||||
system host-if-modify -m 1500 -n data0 -c data ${NODE} ${DATA0IFUUID}
|
||||
system host-if-modify -m 1500 -n data1 -c data ${NODE} ${DATA1IFUUID}
|
||||
system interface-datanetwork-assign ${NODE} ${DATA0IFUUID} ${PHYSNET0}
|
||||
system interface-datanetwork-assign ${NODE} ${DATA1IFUUID} ${PHYSNET1}
|
||||
|
||||
|
||||
* To enable using |SRIOV| network attachments for the above interfaces in
|
||||
Kubernetes hosted application containers:
|
||||
|
||||
* Configure the Kubernetes |SRIOV| device plugin.
|
||||
|
||||
::
|
||||
|
||||
system host-label-assign controller-0 sriovdp=enabled
|
||||
|
||||
* If planning on running |DPDK| in kubernetes hosted application
|
||||
containers on this host, configure the number of 1G Huge pages required
|
||||
on both |NUMA| nodes.
|
||||
|
||||
::
|
||||
|
||||
# assign 10x 1G huge page on processor/numa-node 0 on controller-0 to applications
|
||||
system host-memory-modify -f application controller-0 0 -1G 10
|
||||
|
||||
# assign 10x 1G huge page on processor/numa-node 1 on controller-0 to applications
|
||||
system host-memory-modify -f application controller-0 1 -1G 10
|
||||
|
||||
|
||||
|
||||
***************************************************************
|
||||
If required, initialize a Ceph-based Persistent Storage Backend
|
||||
***************************************************************
|
||||
|
||||
A persistent storage backend is required if your application requires
|
||||
|PVCs|.
|
||||
|
||||
.. only:: starlingx
|
||||
|
||||
.. important::
|
||||
|
||||
The StarlingX OpenStack application **requires** |PVCs|.
|
||||
|
||||
There are two options for persistent storage backend: the host-based Ceph solution and the Rook container-based Ceph solution.
|
||||
|
||||
For host-based Ceph:
|
||||
|
||||
#. Initialize with add ceph backend:
|
||||
#. Add host-based ceph backend:
|
||||
|
||||
::
|
||||
|
||||
@ -215,7 +289,7 @@ For host-based Ceph:
|
||||
|
||||
For Rook container-based Ceph:
|
||||
|
||||
#. Initialize with add ceph-rook backend:
|
||||
#. Add Rook container-based backend:
|
||||
|
||||
::
|
||||
|
||||
@ -229,87 +303,15 @@ For host-based Ceph:
|
||||
system host-label-assign controller-0 ceph-mon-placement=enabled
|
||||
system host-label-assign controller-0 ceph-mgr-placement=enabled
|
||||
|
||||
#. Configure data interfaces for controller-0. Use the DATA port names, for example
|
||||
eth0, applicable to your deployment environment.
|
||||
***********************************
|
||||
If required, configure Docker Proxy
|
||||
***********************************
|
||||
|
||||
.. important::
|
||||
StarlingX uses publicly available container runtime registries. If you are
|
||||
behind a corporate firewall or proxy, you need to set docker proxy settings.
|
||||
|
||||
This step is **required** for OpenStack.
|
||||
|
||||
This step is optional for Kubernetes: Do this step if using |SRIOV|
|
||||
network attachments in hosted application containers.
|
||||
|
||||
For Kubernetes |SRIOV| network attachments:
|
||||
|
||||
* Configure the |SRIOV| device plugin.
|
||||
|
||||
::
|
||||
|
||||
system host-label-assign controller-0 sriovdp=enabled
|
||||
|
||||
* If planning on running |DPDK| in containers on this host, configure the
|
||||
number of 1G Huge pages required on both |NUMA| nodes.
|
||||
|
||||
::
|
||||
|
||||
system host-memory-modify controller-0 0 -1G 100
|
||||
system host-memory-modify controller-0 1 -1G 100
|
||||
|
||||
For both Kubernetes and OpenStack:
|
||||
|
||||
::
|
||||
|
||||
DATA0IF=<DATA-0-PORT>
|
||||
DATA1IF=<DATA-1-PORT>
|
||||
export NODE=controller-0
|
||||
PHYSNET0='physnet0'
|
||||
PHYSNET1='physnet1'
|
||||
SPL=/tmp/tmp-system-port-list
|
||||
SPIL=/tmp/tmp-system-host-if-list
|
||||
system host-port-list ${NODE} --nowrap > ${SPL}
|
||||
system host-if-list -a ${NODE} --nowrap > ${SPIL}
|
||||
DATA0PCIADDR=$(cat $SPL | grep $DATA0IF |awk '{print $8}')
|
||||
DATA1PCIADDR=$(cat $SPL | grep $DATA1IF |awk '{print $8}')
|
||||
DATA0PORTUUID=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $2}')
|
||||
DATA1PORTUUID=$(cat $SPL | grep ${DATA1PCIADDR} | awk '{print $2}')
|
||||
DATA0PORTNAME=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $4}')
|
||||
DATA1PORTNAME=$(cat $SPL | grep ${DATA1PCIADDR} | awk '{print $4}')
|
||||
DATA0IFUUID=$(cat $SPIL | awk -v DATA0PORTNAME=$DATA0PORTNAME '($12 ~ DATA0PORTNAME) {print $2}')
|
||||
DATA1IFUUID=$(cat $SPIL | awk -v DATA1PORTNAME=$DATA1PORTNAME '($12 ~ DATA1PORTNAME) {print $2}')
|
||||
|
||||
system datanetwork-add ${PHYSNET0} vlan
|
||||
system datanetwork-add ${PHYSNET1} vlan
|
||||
|
||||
system host-if-modify -m 1500 -n data0 -c data ${NODE} ${DATA0IFUUID}
|
||||
system host-if-modify -m 1500 -n data1 -c data ${NODE} ${DATA1IFUUID}
|
||||
system interface-datanetwork-assign ${NODE} ${DATA0IFUUID} ${PHYSNET0}
|
||||
system interface-datanetwork-assign ${NODE} ${DATA1IFUUID} ${PHYSNET1}
|
||||
|
||||
#. Add an |OSD| on controller-0 for Ceph. The following example adds an |OSD|
|
||||
to the `sdb` disk:
|
||||
|
||||
.. important::
|
||||
|
||||
This step requires a configured Ceph storage backend
|
||||
|
||||
::
|
||||
|
||||
echo ">>> Add OSDs to primary tier"
|
||||
system host-disk-list controller-0
|
||||
system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {}
|
||||
system host-stor-list controller-0
|
||||
|
||||
#. If required, and not already done as part of bootstrap, configure Docker to
|
||||
use a proxy server.
|
||||
|
||||
#. List Docker proxy parameters:
|
||||
|
||||
::
|
||||
|
||||
system service-parameter-list platform docker
|
||||
|
||||
Refer to :ref:`Docker Proxy Configuration <docker_proxy_config>` for
|
||||
details about Docker proxy settings.
|
||||
Refer to :ref:`Docker Proxy Configuration <docker_proxy_config>` for
|
||||
details about configuring Docker proxy settings.
|
||||
|
||||
.. only:: starlingx
|
||||
|
||||
@ -349,84 +351,81 @@ For host-based Ceph:
|
||||
* Runs directly on the host (it is not containerized).
|
||||
* Requires that at least 1 core be assigned/dedicated to the vSwitch function.
|
||||
|
||||
To deploy the default containerized |OVS|:
|
||||
**To deploy the default containerized OVS:**
|
||||
|
||||
::
|
||||
|
||||
system modify --vswitch_type none
|
||||
|
||||
Do not run any vSwitch directly on the host, instead, use the
|
||||
This does not run any vSwitch directly on the host, instead, it uses the
|
||||
containerized |OVS| defined in the helm charts of stx-openstack
|
||||
manifest.
|
||||
|
||||
To deploy |OVS|-|DPDK|, run the following command:
|
||||
**To deploy OVS-DPDK, run the following command:**
|
||||
|
||||
::
|
||||
|
||||
system modify --vswitch_type ovs-dpdk
|
||||
|
||||
Default recommendation for an AIO-controller is to use a single core
|
||||
for |OVS|-|DPDK| vswitch.
|
||||
|
||||
::
|
||||
|
||||
# assign 1 core on processor/numa-node 0 on controller-0 to vswitch
|
||||
system host-cpu-modify -f vswitch -p0 1 controller-0
|
||||
|
||||
Once vswitch_type is set to |OVS|-|DPDK|, any subsequent nodes created
|
||||
will default to automatically assigning 1 vSwitch core for |AIO|
|
||||
controllers and 2 vSwitch cores for compute-labeled worker nodes.
|
||||
|
||||
When using |OVS|-|DPDK|, configure vSwitch memory per |NUMA| node with
|
||||
the following command:
|
||||
When using |OVS|-|DPDK|, configure 1x 1G huge page for vSwitch memory on each |NUMA| node
|
||||
where vswitch is running on this host, with the following command:
|
||||
|
||||
::
|
||||
|
||||
system host-memory-modify -f <function> -1G <1G hugepages number> <hostname or id> <processor>
|
||||
# assign 1x 1G huge page on processor/numa-node 0 on controller-0 to vswitch
|
||||
system host-memory-modify -f vswitch -1G 1 controller-0 0
|
||||
|
||||
For example:
|
||||
.. important::
|
||||
|
||||
::
|
||||
|VMs| created in an |OVS|-|DPDK| environment must be configured to use
|
||||
huge pages to enable networking and must use a flavor with property:
|
||||
hw:mem_page_size=large
|
||||
|
||||
system host-memory-modify -f vswitch -1G 1 worker-0 0
|
||||
Configure the huge pages for |VMs| in an |OVS|-|DPDK| environment on this host with
|
||||
the commands:
|
||||
|
||||
|VMs| created in an |OVS|-|DPDK| environment must be configured to use
|
||||
huge pages to enable networking and must use a flavor with property:
|
||||
hw:mem_page_size=large
|
||||
::
|
||||
|
||||
Configure the huge pages for |VMs| in an |OVS|-|DPDK| environment with
|
||||
the command:
|
||||
# assign 1x 1G huge page on processor/numa-node 0 on controller-0 to applications
|
||||
system host-memory-modify -f application -1G 10 controller-0 0
|
||||
|
||||
::
|
||||
|
||||
system host-memory-modify -1G <1G hugepages number> <hostname or id> <processor>
|
||||
|
||||
For example:
|
||||
|
||||
::
|
||||
|
||||
system host-memory-modify worker-0 0 -1G 10
|
||||
# assign 1x 1G huge page on processor/numa-node 1 on controller-0 to applications
|
||||
system host-memory-modify -f application -1G 10 controller-0 1
|
||||
|
||||
.. note::
|
||||
|
||||
After controller-0 is unlocked, changing vswitch_type requires
|
||||
locking and unlocking all compute-labeled worker nodes (and/or AIO
|
||||
controllers) to apply the change.
|
||||
locking and unlocking controller-0 to apply the change.
|
||||
|
||||
#. **For OpenStack only:** Set up disk partition for nova-local volume
|
||||
group, which is needed for stx-openstack nova ephemeral disks.
|
||||
#. **For OpenStack only:** Set up disk partition for nova-local volume
|
||||
group, which is needed for stx-openstack nova ephemeral disks.
|
||||
|
||||
::
|
||||
::
|
||||
|
||||
export NODE=controller-0
|
||||
export NODE=controller-0
|
||||
|
||||
echo ">>> Getting root disk info"
|
||||
ROOT_DISK=$(system host-show ${NODE} | grep rootfs | awk '{print $4}')
|
||||
ROOT_DISK_UUID=$(system host-disk-list ${NODE} --nowrap | grep ${ROOT_DISK} | awk '{print $2}')
|
||||
echo "Root disk: $ROOT_DISK, UUID: $ROOT_DISK_UUID"
|
||||
echo ">>> Getting root disk info"
|
||||
ROOT_DISK=$(system host-show ${NODE} | grep rootfs | awk '{print $4}')
|
||||
ROOT_DISK_UUID=$(system host-disk-list ${NODE} --nowrap | grep ${ROOT_DISK} | awk '{print $2}')
|
||||
echo "Root disk: $ROOT_DISK, UUID: $ROOT_DISK_UUID"
|
||||
|
||||
echo ">>>> Configuring nova-local"
|
||||
NOVA_SIZE=34
|
||||
NOVA_PARTITION=$(system host-disk-partition-add -t lvm_phys_vol ${NODE} ${ROOT_DISK_UUID} ${NOVA_SIZE})
|
||||
NOVA_PARTITION_UUID=$(echo ${NOVA_PARTITION} | grep -ow "| uuid | [a-z0-9\-]* |" | awk '{print $4}')
|
||||
system host-lvg-add ${NODE} nova-local
|
||||
system host-pv-add ${NODE} nova-local ${NOVA_PARTITION_UUID}
|
||||
sleep 2
|
||||
echo ">>>> Configuring nova-local"
|
||||
NOVA_SIZE=34
|
||||
NOVA_PARTITION=$(system host-disk-partition-add -t lvm_phys_vol ${NODE} ${ROOT_DISK_UUID} ${NOVA_SIZE})
|
||||
NOVA_PARTITION_UUID=$(echo ${NOVA_PARTITION} | grep -ow "| uuid | [a-z0-9\-]* |" | awk '{print $4}')
|
||||
system host-lvg-add ${NODE} nova-local
|
||||
system host-pv-add ${NODE} nova-local ${NOVA_PARTITION_UUID}
|
||||
sleep 2
|
||||
|
||||
.. incl-config-controller-0-openstack-specific-aio-simplex-end:
|
||||
.. incl-config-controller-0-openstack-specific-aio-simplex-end:
|
||||
|
||||
-------------------
|
||||
Unlock controller-0
|
||||
@ -448,17 +447,13 @@ machine.
|
||||
|
||||
.. only:: starlingx
|
||||
|
||||
--------------------------------------------------------------------------
|
||||
Optionally, finish configuration of Ceph-based Persistent Storage Backend
|
||||
--------------------------------------------------------------------------
|
||||
-----------------------------------------------------------------------------------------------
|
||||
If using Rook container-based Ceph, finish configuring the ceph-rook Persistent Storage Backend
|
||||
-----------------------------------------------------------------------------------------------
|
||||
|
||||
For host-based Ceph: Nothing else is required.
|
||||
On controller-0:
|
||||
|
||||
For Rook container-based Ceph:
|
||||
|
||||
On **virtual** controller-0:
|
||||
|
||||
#. Wait for application rook-ceph-apps uploaded
|
||||
#. Wait for application rook-ceph-apps to be uploaded
|
||||
|
||||
::
|
||||
|
||||
@ -498,7 +493,7 @@ machine.
|
||||
|
||||
system application-apply rook-ceph-apps
|
||||
|
||||
#. Wait for |OSDs| pod ready.
|
||||
#. Wait for |OSDs| pod to be ready.
|
||||
|
||||
::
|
||||
|
||||
|
||||
@ -176,26 +176,34 @@ Configure controller-0
|
||||
|
||||
source /etc/platform/openrc
|
||||
|
||||
#. Configure the OAM and MGMT interfaces of controller-0 and specify the
|
||||
attached networks. Use the OAM and MGMT port names, for example eth0, that are
|
||||
applicable to your deployment environment.
|
||||
#. Configure the |OAM| interface of controller-0 and specify the
|
||||
attached network as "oam".
|
||||
|
||||
Use the |OAM| port name that is applicable to your deployment environment, for example eth0:
|
||||
|
||||
::
|
||||
|
||||
OAM_IF=<OAM-PORT>
|
||||
system host-if-modify controller-0 $OAM_IF -c platform
|
||||
system interface-network-assign controller-0 $OAM_IF oam
|
||||
|
||||
#. Configure the MGMT interface of controller-0 and specify the attached networks of both "mgmt" and "cluster-host".
|
||||
|
||||
Use the MGMT port name that is applicable to your deployment environment, for example eth1:
|
||||
|
||||
::
|
||||
|
||||
MGMT_IF=<MGMT-PORT>
|
||||
system host-if-modify controller-0 lo -c none
|
||||
IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if ($6=="lo") print $4;}')
|
||||
for UUID in $IFNET_UUIDS; do
|
||||
system interface-network-remove ${UUID}
|
||||
done
|
||||
system host-if-modify controller-0 $OAM_IF -c platform
|
||||
system interface-network-assign controller-0 $OAM_IF oam
|
||||
system host-if-modify controller-0 $MGMT_IF -c platform
|
||||
system interface-network-assign controller-0 $MGMT_IF mgmt
|
||||
system interface-network-assign controller-0 $MGMT_IF cluster-host
|
||||
|
||||
#. Configure NTP servers for network time synchronization:
|
||||
#. Configure |NTP| servers for network time synchronization:
|
||||
|
||||
::
|
||||
|
||||
@ -203,33 +211,31 @@ Configure controller-0
|
||||
|
||||
#. Configure Ceph storage backend:
|
||||
|
||||
.. important::
|
||||
This step is required only if your application requires persistent storage.
|
||||
|
||||
This step required only if your application requires
|
||||
persistent storage.
|
||||
.. only:: starlingx
|
||||
|
||||
**If you want to install the StarlingX Openstack application
|
||||
(stx-openstack), this step is mandatory.**
|
||||
.. important::
|
||||
|
||||
**If you want to install the StarlingX Openstack application
|
||||
(stx-openstack), this step is mandatory.**
|
||||
|
||||
::
|
||||
|
||||
system storage-backend-add ceph --confirmed
|
||||
system storage-backend-add ceph --confirmed
|
||||
|
||||
#. If required, and not already done as part of bootstrap, configure Docker to
|
||||
use a proxy server.
|
||||
|
||||
#. List Docker proxy parameters:
|
||||
StarlingX uses publicly available container runtime registries. If you are behind a
|
||||
corporate firewall or proxy, you need to set docker proxy settings.
|
||||
|
||||
::
|
||||
Refer to :ref:`Docker Proxy Configuration <docker_proxy_config>` for
|
||||
details about configuring Docker proxy settings.
|
||||
|
||||
system service-parameter-list platform docker
|
||||
|
||||
#. Refer to :ref:`Docker Proxy Configuration <docker_proxy_config>` for
|
||||
details about Docker proxy settings.
|
||||
|
||||
.. only:: starlingx
|
||||
|
||||
|
||||
*************************************
|
||||
OpenStack-specific host configuration
|
||||
*************************************
|
||||
@ -248,71 +254,43 @@ Configure controller-0
|
||||
|
||||
#. **For OpenStack only:** Configure the system setting for the vSwitch.
|
||||
|
||||
StarlingX has OVS (kernel-based) vSwitch configured as default:
|
||||
StarlingX has |OVS| (kernel-based) vSwitch configured as default:
|
||||
|
||||
* Runs in a container; defined within the helm charts of stx-openstack
|
||||
manifest.
|
||||
* Shares the core(s) assigned to the platform.
|
||||
|
||||
If you require better performance, OVS-DPDK (OVS with the Data Plane
|
||||
Development Kit, which is supported only on bare metal hardware) should be
|
||||
used:
|
||||
If you require better performance, |OVS|-|DPDK| (OVS with the Data Plane
|
||||
Development Kit, which is supported only on bare metal hardware) should
|
||||
be used:
|
||||
|
||||
* Runs directly on the host (it is not containerized).
|
||||
* Requires that at least 1 core be assigned/dedicated to the vSwitch function.
|
||||
|
||||
To deploy the default containerized OVS:
|
||||
**To deploy the default containerized OVS:**
|
||||
|
||||
::
|
||||
|
||||
system modify --vswitch_type none
|
||||
|
||||
Do not run any vSwitch directly on the host, instead, use the containerized
|
||||
OVS defined in the helm charts of stx-openstack manifest.
|
||||
This does not run any vSwitch directly on the host, instead, it uses the
|
||||
containerized |OVS| defined in the helm charts of stx-openstack manifest.
|
||||
|
||||
To deploy OVS-|DPDK|, run the following command:
|
||||
**To deploy OVS-DPDK, run the following command:**
|
||||
|
||||
::
|
||||
|
||||
system modify --vswitch_type ovs-dpdk
|
||||
system host-cpu-modify -f vswitch -p0 1 controller-0
|
||||
|
||||
Once vswitch_type is set to OVS-|DPDK|, any subsequent nodes created
|
||||
will default to automatically assigning 1 vSwitch core for AIO
|
||||
controllers and 2 vSwitch cores for compute-labeled worker nodes.
|
||||
|
||||
When using OVS-|DPDK|, configure vSwitch memory per node with the
|
||||
following command:
|
||||
|
||||
::
|
||||
|
||||
system host-memory-modify -f <function> -1G <1G hugepages number> <hostname or id> <processor>
|
||||
|
||||
For example:
|
||||
|
||||
::
|
||||
|
||||
system host-memory-modify -f vswitch -1G 1 worker-0 0
|
||||
|
||||
|VMs| created in an OVS-|DPDK| environment must be configured to use huge pages
|
||||
to enable networking and must use a flavor with property: hw:mem_page_size=large
|
||||
|
||||
Configure the huge pages for VMs in an OVS-|DPDK| environment with the command:
|
||||
|
||||
::
|
||||
|
||||
system host-memory-modify -1G <1G hugepages number> <hostname or id> <processor>
|
||||
|
||||
For example:
|
||||
|
||||
::
|
||||
|
||||
system host-memory-modify worker-0 0 -1G 10
|
||||
Once vswitch_type is set to OVS-|DPDK|, any subsequent AIO-controller or
|
||||
worker nodes created will default to automatically assigning 1 vSwitch
|
||||
core for |AIO| controllers and 2 vSwitch cores for compute-labeled worker
|
||||
nodes.
|
||||
|
||||
.. note::
|
||||
|
||||
After controller-0 is unlocked, changing vswitch_type requires
|
||||
locking and unlocking all compute-labeled worker nodes (and/or AIO
|
||||
locking and unlocking all compute-labeled worker nodes (and/or |AIO|
|
||||
controllers) to apply the change.
|
||||
|
||||
.. incl-config-controller-0-storage-end:
|
||||
@ -341,8 +319,8 @@ Install software on controller-1 and worker nodes
|
||||
#. As controller-1 boots, a message appears on its console instructing you to
|
||||
configure the personality of the node.
|
||||
|
||||
#. On the console of controller-0, list hosts to see newly discovered controller-1
|
||||
host (hostname=None):
|
||||
#. On the console of controller-0, list hosts to see newly discovered
|
||||
controller-1 host (hostname=None):
|
||||
|
||||
::
|
||||
|
||||
@ -373,16 +351,16 @@ Install software on controller-1 and worker nodes
|
||||
|
||||
system host-update 3 personality=worker hostname=worker-0
|
||||
|
||||
Repeat for worker-1. Power on worker-1 and wait for the new host (hostname=None) to
|
||||
be discovered by checking 'system host-list':
|
||||
Repeat for worker-1. Power on worker-1 and wait for the new host
|
||||
(hostname=None) to be discovered by checking 'system host-list':
|
||||
|
||||
::
|
||||
|
||||
system host-update 4 personality=worker hostname=worker-1
|
||||
|
||||
#. Wait for the software installation on controller-1, worker-0, and worker-1 to
|
||||
complete, for all servers to reboot, and for all to show as locked/disabled/online in
|
||||
'system host-list'.
|
||||
#. Wait for the software installation on controller-1, worker-0, and worker-1
|
||||
to complete, for all servers to reboot, and for all to show as
|
||||
locked/disabled/online in 'system host-list'.
|
||||
|
||||
::
|
||||
|
||||
@ -403,20 +381,27 @@ Configure controller-1
|
||||
|
||||
.. incl-config-controller-1-start:
|
||||
|
||||
Configure the OAM and MGMT interfaces of controller-1 and specify the attached
|
||||
networks. Use the OAM and MGMT port names, for example eth0, that are applicable
|
||||
to your deployment environment.
|
||||
#. Configure the |OAM| interface of controller-1 and specify the
|
||||
attached network of "oam".
|
||||
|
||||
(Note that the MGMT interface is partially set up automatically by the network
|
||||
install procedure.)
|
||||
Use the |OAM| port name that is applicable to your deployment environment, for example eth0:
|
||||
|
||||
::
|
||||
::
|
||||
|
||||
OAM_IF=<OAM-PORT>
|
||||
system host-if-modify controller-1 $OAM_IF -c platform
|
||||
system interface-network-assign controller-1 $OAM_IF oam
|
||||
|
||||
#. The MGMT interface is partially set up by the network install procedure; configuring
|
||||
the port used for network install as the MGMT port and specifying the attached network of "mgmt".
|
||||
|
||||
Complete the MGMT interface configuration of controller-1 by specifying the attached
|
||||
network of "cluster-host".
|
||||
|
||||
::
|
||||
|
||||
system interface-network-assign controller-1 mgmt0 cluster-host
|
||||
|
||||
OAM_IF=<OAM-PORT>
|
||||
MGMT_IF=<MGMT-PORT>
|
||||
system host-if-modify controller-1 $OAM_IF -c platform
|
||||
system interface-network-assign controller-1 $OAM_IF oam
|
||||
system interface-network-assign controller-1 $MGMT_IF cluster-host
|
||||
|
||||
.. only:: starlingx
|
||||
|
||||
@ -498,34 +483,16 @@ Configure worker nodes
|
||||
#. Configure data interfaces for worker nodes. Use the DATA port names, for
|
||||
example eth0, that are applicable to your deployment environment.
|
||||
|
||||
.. important::
|
||||
This step is optional for Kubernetes: Do this step if using |SRIOV| network
|
||||
attachments in hosted application containers.
|
||||
|
||||
This step is **required** for OpenStack.
|
||||
.. only:: starlingx
|
||||
|
||||
This step is optional for Kubernetes: Do this step if using |SRIOV| network
|
||||
attachments in hosted application containers.
|
||||
.. important::
|
||||
|
||||
For Kubernetes |SRIOV| network attachments:
|
||||
This step is **required** for OpenStack.
|
||||
|
||||
* Configure |SRIOV| device plug in:
|
||||
|
||||
::
|
||||
|
||||
for NODE in worker-0 worker-1; do
|
||||
system host-label-assign ${NODE} sriovdp=enabled
|
||||
done
|
||||
|
||||
* If planning on running DPDK in containers on this host, configure the number
|
||||
of 1G Huge pages required on both |NUMA| nodes:
|
||||
|
||||
::
|
||||
|
||||
for NODE in worker-0 worker-1; do
|
||||
system host-memory-modify ${NODE} 0 -1G 100
|
||||
system host-memory-modify ${NODE} 1 -1G 100
|
||||
done
|
||||
|
||||
For both Kubernetes and OpenStack:
|
||||
* Configure the data interfaces
|
||||
|
||||
::
|
||||
|
||||
@ -561,6 +528,32 @@ Configure worker nodes
|
||||
set +ex
|
||||
done
|
||||
|
||||
* To enable using |SRIOV| network attachments for the above interfaces in Kubernetes hosted application containers:
|
||||
|
||||
* Configure |SRIOV| device plug in:
|
||||
|
||||
::
|
||||
|
||||
for NODE in worker-0 worker-1; do
|
||||
system host-label-assign ${NODE} sriovdp=enabled
|
||||
done
|
||||
|
||||
* If planning on running DPDK in containers on this host, configure the number
|
||||
of 1G Huge pages required on both |NUMA| nodes:
|
||||
|
||||
::
|
||||
|
||||
for NODE in worker-0 worker-1; do
|
||||
|
||||
# assign 10x 1G huge page on processor/numa-node 0 on worker-node to applications
|
||||
system host-memory-modify -f application $NODE 0 -1G 10
|
||||
|
||||
# assign 10x 1G huge page on processor/numa-node 1 on worker-node to applications
|
||||
system host-memory-modify -f application $NODE 1 -1G 10
|
||||
|
||||
done
|
||||
|
||||
|
||||
.. only:: starlingx
|
||||
|
||||
*************************************
|
||||
@ -583,6 +576,64 @@ Configure worker nodes
|
||||
system host-label-assign $NODE sriov=enabled
|
||||
done
|
||||
|
||||
#. **For OpenStack only:** Configure the host settings for the vSwitch.
|
||||
|
||||
**If using OVS-DPDK vswitch, run the following commands:**
|
||||
|
||||
Default recommendation for worker node is to use a single core on each numa-node
|
||||
for |OVS|-|DPDK| vswitch. This should have been automatically configured,
|
||||
if not run the following command.
|
||||
|
||||
::
|
||||
|
||||
for NODE in worker-0 worker-1; do
|
||||
|
||||
# assign 1 core on processor/numa-node 0 on worker-node to vswitch
|
||||
system host-cpu-modify -f vswitch -p0 1 $NODE
|
||||
|
||||
# assign 1 core on processor/numa-node 1 on worker-node to vswitch
|
||||
system host-cpu-modify -f vswitch -p1 1 $NODE
|
||||
|
||||
done
|
||||
|
||||
|
||||
When using |OVS|-|DPDK|, configure 1x 1G huge page for vSwitch memory on each |NUMA| node
|
||||
where vswitch is running on this host, with the following command:
|
||||
|
||||
::
|
||||
|
||||
for NODE in worker-0 worker-1; do
|
||||
|
||||
# assign 1x 1G huge page on processor/numa-node 0 on worker-node to vswitch
|
||||
system host-memory-modify -f vswitch -1G 1 $NODE 0
|
||||
|
||||
# assign 1x 1G huge page on processor/numa-node 0 on worker-node to vswitch
|
||||
system host-memory-modify -f vswitch -1G 1 $NODE 1
|
||||
|
||||
done
|
||||
|
||||
|
||||
.. important::
|
||||
|
||||
|VMs| created in an |OVS|-|DPDK| environment must be configured to use
|
||||
huge pages to enable networking and must use a flavor with property:
|
||||
hw:mem_page_size=large
|
||||
|
||||
Configure the huge pages for |VMs| in an |OVS|-|DPDK| environment for this host with
|
||||
the command:
|
||||
|
||||
::
|
||||
|
||||
for NODE in worker-0 worker-1; do
|
||||
|
||||
# assign 10x 1G huge page on processor/numa-node 0 on worker-node to applications
|
||||
system host-memory-modify -f application -1G 10 $NODE 0
|
||||
|
||||
# assign 10x 1G huge page on processor/numa-node 1 on worker-node to applications
|
||||
system host-memory-modify -f application -1G 10 $NODE 1
|
||||
|
||||
done
|
||||
|
||||
#. **For OpenStack only:** Set up disk partition for nova-local volume group,
|
||||
which is needed for stx-openstack nova ephemeral disks.
|
||||
|
||||
|
||||
@ -191,7 +191,7 @@ Configure storage nodes
|
||||
system interface-network-assign $NODE mgmt0 cluster-host
|
||||
done
|
||||
|
||||
#. Add OSDs to storage-0. The following example adds OSDs to the `sdb` disk:
|
||||
#. Add |OSDs| to storage-0. The following example adds |OSDs| to the `sdb` disk:
|
||||
|
||||
::
|
||||
|
||||
@ -206,7 +206,7 @@ Configure storage nodes
|
||||
|
||||
system host-stor-list $HOST
|
||||
|
||||
#. Add OSDs to storage-1. The following example adds OSDs to the `sdb` disk:
|
||||
#. Add |OSDs| to storage-1. The following example adds |OSDs| to the `sdb` disk:
|
||||
|
||||
::
|
||||
|
||||
@ -255,34 +255,16 @@ Configure worker nodes
|
||||
#. Configure data interfaces for worker nodes. Use the DATA port names, for
|
||||
example eth0, that are applicable to your deployment environment.
|
||||
|
||||
.. important::
|
||||
This step is optional for Kubernetes: Do this step if using |SRIOV| network
|
||||
attachments in hosted application containers.
|
||||
|
||||
This step is **required** for OpenStack.
|
||||
.. only:: starlingx
|
||||
|
||||
This step is optional for Kubernetes: Do this step if using SRIOV network
|
||||
attachments in hosted application containers.
|
||||
.. important::
|
||||
|
||||
For Kubernetes SRIOV network attachments:
|
||||
This step is **required** for OpenStack.
|
||||
|
||||
* Configure SRIOV device plug in:
|
||||
|
||||
::
|
||||
|
||||
for NODE in worker-0 worker-1; do
|
||||
system host-label-assign ${NODE} sriovdp=enabled
|
||||
done
|
||||
|
||||
* If planning on running DPDK in containers on this host, configure the number
|
||||
of 1G Huge pages required on both NUMA nodes:
|
||||
|
||||
::
|
||||
|
||||
for NODE in worker-0 worker-1; do
|
||||
system host-memory-modify ${NODE} 0 -1G 100
|
||||
system host-memory-modify ${NODE} 1 -1G 100
|
||||
done
|
||||
|
||||
For both Kubernetes and OpenStack:
|
||||
* Configure the data interfaces
|
||||
|
||||
::
|
||||
|
||||
@ -318,41 +300,123 @@ Configure worker nodes
|
||||
set +ex
|
||||
done
|
||||
|
||||
*************************************
|
||||
OpenStack-specific host configuration
|
||||
*************************************
|
||||
* To enable using SRIOV network attachments for the above interfaces in Kubernetes hosted application containers:
|
||||
|
||||
.. important::
|
||||
* Configure SRIOV device plug in:
|
||||
|
||||
**This step is required only if the StarlingX OpenStack application
|
||||
(stx-openstack) will be installed.**
|
||||
::
|
||||
|
||||
#. **For OpenStack only:** Assign OpenStack host labels to the worker nodes in
|
||||
support of installing the stx-openstack manifest and helm-charts later.
|
||||
for NODE in worker-0 worker-1; do
|
||||
system host-label-assign ${NODE} sriovdp=enabled
|
||||
done
|
||||
|
||||
::
|
||||
* If planning on running |DPDK| in containers on this host, configure the
|
||||
number of 1G Huge pages required on both NUMA nodes:
|
||||
|
||||
for NODE in worker-0 worker-1; do
|
||||
system host-label-assign $NODE openstack-compute-node=enabled
|
||||
system host-label-assign $NODE openvswitch=enabled
|
||||
system host-label-assign $NODE sriov=enabled
|
||||
done
|
||||
::
|
||||
|
||||
#. **For OpenStack only:** Set up disk partition for nova-local volume group,
|
||||
which is needed for stx-openstack nova ephemeral disks.
|
||||
for NODE in worker-0 worker-1; do
|
||||
system host-memory-modify ${NODE} 0 -1G 100
|
||||
system host-memory-modify ${NODE} 1 -1G 100
|
||||
done
|
||||
|
||||
::
|
||||
|
||||
for NODE in worker-0 worker-1; do
|
||||
echo "Configuring Nova local for: $NODE"
|
||||
ROOT_DISK=$(system host-show ${NODE} | grep rootfs | awk '{print $4}')
|
||||
ROOT_DISK_UUID=$(system host-disk-list ${NODE} --nowrap | grep ${ROOT_DISK} | awk '{print $2}')
|
||||
PARTITION_SIZE=10
|
||||
NOVA_PARTITION=$(system host-disk-partition-add -t lvm_phys_vol ${NODE} ${ROOT_DISK_UUID} ${PARTITION_SIZE})
|
||||
NOVA_PARTITION_UUID=$(echo ${NOVA_PARTITION} | grep -ow "| uuid | [a-z0-9\-]* |" | awk '{print $4}')
|
||||
system host-lvg-add ${NODE} nova-local
|
||||
system host-pv-add ${NODE} nova-local ${NOVA_PARTITION_UUID}
|
||||
done
|
||||
.. only:: starlingx
|
||||
|
||||
*************************************
|
||||
OpenStack-specific host configuration
|
||||
*************************************
|
||||
|
||||
.. important::
|
||||
|
||||
**This step is required only if the StarlingX OpenStack application
|
||||
(stx-openstack) will be installed.**
|
||||
|
||||
#. **For OpenStack only:** Assign OpenStack host labels to the worker nodes in
|
||||
support of installing the stx-openstack manifest and helm-charts later.
|
||||
|
||||
::
|
||||
|
||||
for NODE in worker-0 worker-1; do
|
||||
system host-label-assign $NODE openstack-compute-node=enabled
|
||||
system host-label-assign $NODE openvswitch=enabled
|
||||
system host-label-assign $NODE sriov=enabled
|
||||
done
|
||||
|
||||
#. **For OpenStack only:** Configure the host settings for the vSwitch.
|
||||
|
||||
**If using OVS-DPDK vswitch, run the following commands:**
|
||||
|
||||
Default recommendation for worker node is to use a single core on each
|
||||
numa-node for |OVS|-|DPDK| vswitch. This should have been automatically
|
||||
configured, if not run the following command.
|
||||
|
||||
::
|
||||
|
||||
for NODE in worker-0 worker-1; do
|
||||
|
||||
# assign 1 core on processor/numa-node 0 on worker-node to vswitch
|
||||
system host-cpu-modify -f vswitch -p0 1 $NODE
|
||||
|
||||
# assign 1 core on processor/numa-node 1 on worker-node to vswitch
|
||||
system host-cpu-modify -f vswitch -p1 1 $NODE
|
||||
|
||||
done
|
||||
|
||||
|
||||
When using |OVS|-|DPDK|, configure 1x 1G huge page for vSwitch memory on
|
||||
each |NUMA| node where vswitch is running on this host, with the
|
||||
following command:
|
||||
|
||||
::
|
||||
|
||||
for NODE in worker-0 worker-1; do
|
||||
|
||||
# assign 1x 1G huge page on processor/numa-node 0 on worker-node to vswitch
|
||||
system host-memory-modify -f vswitch -1G 1 $NODE 0
|
||||
|
||||
# assign 1x 1G huge page on processor/numa-node 0 on worker-node to vswitch
|
||||
system host-memory-modify -f vswitch -1G 1 $NODE 1
|
||||
|
||||
done
|
||||
|
||||
|
||||
.. important::
|
||||
|
||||
|VMs| created in an |OVS|-|DPDK| environment must be configured to use
|
||||
huge pages to enable networking and must use a flavor with property:
|
||||
hw:mem_page_size=large
|
||||
|
||||
Configure the huge pages for |VMs| in an |OVS|-|DPDK| environment for
|
||||
this host with the command:
|
||||
|
||||
::
|
||||
|
||||
for NODE in worker-0 worker-1; do
|
||||
|
||||
# assign 10x 1G huge page on processor/numa-node 0 on worker-node to applications
|
||||
system host-memory-modify -f application -1G 10 $NODE 0
|
||||
|
||||
# assign 10x 1G huge page on processor/numa-node 1 on worker-node to applications
|
||||
system host-memory-modify -f application -1G 10 $NODE 1
|
||||
|
||||
done
|
||||
|
||||
#. **For OpenStack only:** Set up disk partition for nova-local volume group,
|
||||
which is needed for stx-openstack nova ephemeral disks.
|
||||
|
||||
::
|
||||
|
||||
for NODE in worker-0 worker-1; do
|
||||
echo "Configuring Nova local for: $NODE"
|
||||
ROOT_DISK=$(system host-show ${NODE} | grep rootfs | awk '{print $4}')
|
||||
ROOT_DISK_UUID=$(system host-disk-list ${NODE} --nowrap | grep ${ROOT_DISK} | awk '{print $2}')
|
||||
PARTITION_SIZE=10
|
||||
NOVA_PARTITION=$(system host-disk-partition-add -t lvm_phys_vol ${NODE} ${ROOT_DISK_UUID} ${PARTITION_SIZE})
|
||||
NOVA_PARTITION_UUID=$(echo ${NOVA_PARTITION} | grep -ow "| uuid | [a-z0-9\-]* |" | awk '{print $4}')
|
||||
system host-lvg-add ${NODE} nova-local
|
||||
system host-pv-add ${NODE} nova-local ${NOVA_PARTITION_UUID}
|
||||
done
|
||||
|
||||
-------------------
|
||||
Unlock worker nodes
|
||||
|
||||
@ -30,16 +30,16 @@
|
||||
* Runs directly on the host (it is not containerized).
|
||||
* Requires that at least 1 core be assigned/dedicated to the vSwitch function.
|
||||
|
||||
To deploy the default containerized |OVS|:
|
||||
**To deploy the default containerized OVS:**
|
||||
|
||||
::
|
||||
|
||||
system modify --vswitch_type none
|
||||
|
||||
Do not run any vSwitch directly on the host, instead, use the containerized
|
||||
This does not run any vSwitch directly on the host, instead, it uses the containerized
|
||||
|OVS| defined in the helm charts of stx-openstack manifest.
|
||||
|
||||
To deploy |OVS|-|DPDK|, run the following command:
|
||||
**To deploy OVS-DPDK, run the following command:**
|
||||
|
||||
::
|
||||
|
||||
@ -53,32 +53,23 @@
|
||||
When using |OVS|-|DPDK|, configure vSwitch memory per |NUMA| node with
|
||||
the following command:
|
||||
|
||||
::
|
||||
|
||||
system host-memory-modify -f <function> -1G <1G hugepages number> <hostname or id> <processor>
|
||||
|
||||
For example:
|
||||
|
||||
::
|
||||
|
||||
system host-memory-modify -f vswitch -1G 1 worker-0 0
|
||||
|
||||
|VMs| created in an |OVS|-|DPDK| environment must be configured to use
|
||||
huge pages to enable networking and must use a flavor with property:
|
||||
hw:mem_page_size=large
|
||||
.. important::
|
||||
|
||||
Configure the huge pages for |VMs| in an |OVS|-|DPDK| environment with
|
||||
the command:
|
||||
|VMs| created in an |OVS|-|DPDK| environment must be configured to use
|
||||
huge pages to enable networking and must use a flavor with property:
|
||||
hw:mem_page_size=large
|
||||
|
||||
::
|
||||
Configure the huge pages for |VMs| in an |OVS|-|DPDK| environment with
|
||||
the command:
|
||||
|
||||
system host-memory-modify -1G <1G hugepages number> <hostname or id> <processor>
|
||||
::
|
||||
|
||||
For example:
|
||||
|
||||
::
|
||||
|
||||
system host-memory-modify worker-0 0 -1G 10
|
||||
system host-memory-modify -f application -1G 10 worker-0 0
|
||||
system host-memory-modify -f application -1G 10 worker-1 1
|
||||
|
||||
.. note::
|
||||
|
||||
@ -86,24 +77,24 @@
|
||||
locking and unlocking all compute-labeled worker nodes (and/or AIO
|
||||
controllers) to apply the change.
|
||||
|
||||
#. **For OpenStack only:** Set up disk partition for nova-local volume
|
||||
group, which is needed for stx-openstack nova ephemeral disks.
|
||||
#. **For OpenStack only:** Set up disk partition for nova-local volume
|
||||
group, which is needed for stx-openstack nova ephemeral disks.
|
||||
|
||||
::
|
||||
::
|
||||
|
||||
export NODE=controller-0
|
||||
export NODE=controller-0
|
||||
|
||||
echo ">>> Getting root disk info"
|
||||
ROOT_DISK=$(system host-show ${NODE} | grep rootfs | awk '{print $4}')
|
||||
ROOT_DISK_UUID=$(system host-disk-list ${NODE} --nowrap | grep ${ROOT_DISK} | awk '{print $2}')
|
||||
echo "Root disk: $ROOT_DISK, UUID: $ROOT_DISK_UUID"
|
||||
echo ">>> Getting root disk info"
|
||||
ROOT_DISK=$(system host-show ${NODE} | grep rootfs | awk '{print $4}')
|
||||
ROOT_DISK_UUID=$(system host-disk-list ${NODE} --nowrap | grep ${ROOT_DISK} | awk '{print $2}')
|
||||
echo "Root disk: $ROOT_DISK, UUID: $ROOT_DISK_UUID"
|
||||
|
||||
echo ">>>> Configuring nova-local"
|
||||
NOVA_SIZE=34
|
||||
NOVA_PARTITION=$(system host-disk-partition-add -t lvm_phys_vol ${NODE} ${ROOT_DISK_UUID} ${NOVA_SIZE})
|
||||
NOVA_PARTITION_UUID=$(echo ${NOVA_PARTITION} | grep -ow "| uuid | [a-z0-9\-]* |" | awk '{print $4}')
|
||||
system host-lvg-add ${NODE} nova-local
|
||||
system host-pv-add ${NODE} nova-local ${NOVA_PARTITION_UUID}
|
||||
sleep 2
|
||||
echo ">>>> Configuring nova-local"
|
||||
NOVA_SIZE=34
|
||||
NOVA_PARTITION=$(system host-disk-partition-add -t lvm_phys_vol ${NODE} ${ROOT_DISK_UUID} ${NOVA_SIZE})
|
||||
NOVA_PARTITION_UUID=$(echo ${NOVA_PARTITION} | grep -ow "| uuid | [a-z0-9\-]* |" | awk '{print $4}')
|
||||
system host-lvg-add ${NODE} nova-local
|
||||
system host-pv-add ${NODE} nova-local ${NOVA_PARTITION_UUID}
|
||||
sleep 2
|
||||
|
||||
.. incl-config-controller-0-openstack-specific-aio-simplex-end:
|
||||
.. incl-config-controller-0-openstack-specific-aio-simplex-end:
|
||||
|
||||
@ -244,7 +244,7 @@ OpenStack-specific host configuration
|
||||
* Runs directly on the host (it is not containerized).
|
||||
* Requires that at least 1 core be assigned/dedicated to the vSwitch function.
|
||||
|
||||
To deploy the default containerized |OVS|:
|
||||
To deploy the default containerized OVS|:
|
||||
|
||||
::
|
||||
|
||||
@ -333,7 +333,8 @@ Unlock controller-0 in order to bring it into service:
|
||||
system host-unlock controller-0
|
||||
|
||||
Controller-0 will reboot in order to apply configuration changes and come into
|
||||
service. This can take 5-10 minutes, depending on the performance of the host machine.
|
||||
service. This can take 5-10 minutes, depending on the performance of the host
|
||||
machine.
|
||||
|
||||
-------------------------------------------------
|
||||
Install software on controller-1 and worker nodes
|
||||
@ -377,26 +378,27 @@ Install software on controller-1 and worker nodes
|
||||
|
||||
system host-update 3 personality=worker hostname=worker-0
|
||||
|
||||
Repeat for worker-1. Power on worker-1 and wait for the new host (hostname=None) to
|
||||
be discovered by checking 'system host-list':
|
||||
Repeat for worker-1. Power on worker-1 and wait for the new host
|
||||
(hostname=None) to be discovered by checking 'system host-list':
|
||||
|
||||
::
|
||||
|
||||
system host-update 4 personality=worker hostname=worker-1
|
||||
|
||||
For rook storage, there is no storage personality. Some hosts with worker personality
|
||||
providers storage service. Here we still named these worker host storage-x.
|
||||
Repeat for storage-0 and storage-1. Power on storage-0, storage-1 and wait for the
|
||||
new host (hostname=None) to be discovered by checking 'system host-list':
|
||||
For rook storage, there is no storage personality. Some hosts with worker
|
||||
personality providers storage service. Here we still named these worker host
|
||||
storage-x. Repeat for storage-0 and storage-1. Power on storage-0, storage-1
|
||||
and wait for the new host (hostname=None) to be discovered by checking
|
||||
'system host-list':
|
||||
|
||||
::
|
||||
|
||||
system host-update 5 personality=worker hostname=storage-0
|
||||
system host-update 6 personality=worker hostname=storage-1
|
||||
|
||||
#. Wait for the software installation on controller-1, worker-0, and worker-1 to
|
||||
complete, for all servers to reboot, and for all to show as locked/disabled/online in
|
||||
'system host-list'.
|
||||
#. Wait for the software installation on controller-1, worker-0, and worker-1
|
||||
to complete, for all servers to reboot, and for all to show as
|
||||
locked/disabled/online in 'system host-list'.
|
||||
|
||||
::
|
||||
|
||||
@ -419,8 +421,8 @@ Configure controller-1
|
||||
|
||||
.. incl-config-controller-1-start:
|
||||
|
||||
Configure the OAM and MGMT interfaces of controller-0 and specify the attached
|
||||
networks. Use the OAM and MGMT port names, for example eth0, that are applicable
|
||||
Configure the |OAM| and MGMT interfaces of controller-0 and specify the attached
|
||||
networks. Use the |OAM| and MGMT port names, for example eth0, that are applicable
|
||||
to your deployment environment.
|
||||
|
||||
(Note that the MGMT interface is partially set up automatically by the network
|
||||
@ -522,8 +524,8 @@ Configure worker nodes
|
||||
system host-label-assign ${NODE} sriovdp=enabled
|
||||
done
|
||||
|
||||
* If planning on running DPDK in containers on this host, configure the number
|
||||
of 1G Huge pages required on both NUMA nodes:
|
||||
* If planning on running |DPDK| in containers on this host, configure the number
|
||||
of 1G Huge pages required on both |NUMA| nodes:
|
||||
|
||||
::
|
||||
|
||||
@ -616,8 +618,9 @@ Unlock worker nodes in order to bring them into service:
|
||||
system host-unlock $NODE
|
||||
done
|
||||
|
||||
The worker nodes will reboot in order to apply configuration changes and come into
|
||||
service. This can take 5-10 minutes, depending on the performance of the host machine.
|
||||
The worker nodes will reboot in order to apply configuration changes and come
|
||||
into service. This can take 5-10 minutes, depending on the performance of the
|
||||
host machine.
|
||||
|
||||
-----------------------
|
||||
Configure storage nodes
|
||||
@ -625,7 +628,8 @@ Configure storage nodes
|
||||
|
||||
#. Assign the cluster-host network to the MGMT interface for the storage nodes.
|
||||
|
||||
Note that the MGMT interfaces are partially set up by the network install procedure.
|
||||
Note that the MGMT interfaces are partially set up by the network install
|
||||
procedure.
|
||||
|
||||
::
|
||||
|
||||
@ -653,7 +657,8 @@ Unlock storage nodes in order to bring them into service:
|
||||
done
|
||||
|
||||
The storage nodes will reboot in order to apply configuration changes and come
|
||||
into service. This can take 5-10 minutes, depending on the performance of the host machine.
|
||||
into service. This can take 5-10 minutes, depending on the performance of the
|
||||
host machine.
|
||||
|
||||
-------------------------------------------------
|
||||
Install Rook application manifest and helm-charts
|
||||
|
||||
Loading…
Reference in New Issue
Block a user