Ron Stone cf755b146c KubeVirt/CDI introduction

Inital draft guide for KubeVirt introduction
Conditionalize version
Implement patchset 2 review changes
Implement patchset 3 review changes
Resolve merge conflict
Implement patchset 4 review changes

Story: 2010466

Signed-off-by: Ron Stone <ronald.stone@windriver.com>
Change-Id: I57a16fca9b78992b249a1aa04e6b12893c94fe9f

2022-12-16 19:32:48 -05:00

13 KiB

Raw Blame History

Create an Ubuntu VM

This section provides a more complex, but likely more real-life, example of deploying a linux-based with KubeVirt.

The example uses:

An ubuntu 22.04 jammy cloud image https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img
The Upload Proxy service to upload the ubuntu jammy image into a DataVolume/, for the root disk. A production linux distribution and a DataVolume/ for persistent storage.
A cloudInitNoCloud volume type.

Used to pass in cloud init formatted 'userData' in order to create a user and password for initial login.
Explicit resource request for 8x CPUs and 16G of Memory.
Multus and CNIs in order to add an additional -based interface. This allows the to be assigned a unique IP Address from the IP Subnet attached to the -based interface.
Connection using the serial console interface via virtctl in order to configure the IP Interface on the SRIOV-based interface.
SSH from a REMOTE WORKSTATION to the 's unique IP Address on the IP Subnet attached to the -based interface.

Make the infrastructure changes to support -based interfaces on containers (including KubeVirt containers):
1. Create a new Data Network of type 'vlan' for SRIOV interfaces to be used by your KubeVirt .
```
~(keystone_admin)$ system datanetwork-add kube-sriov vlan
```
2. Create pci-sriov type interfaces.
  
  For every AIO-Controller and Worker/Compute, create a PCI-SRIOV type interface and attach the new data network to this interface. e.g. for compute-1's enp24s0f0 ethernet interface
```
~(keystone_admin)$ system host-if-modify -n sriov0 -c pci-sriov -N 64 --vf-driver vfio compute-1 enp24s0f0
~(keystone_admin)$ system interface-datanetwork-assign compute-1 sriov0 kube-sriov
```
3. Create a NetworkAttachmentDefinition in Kubernetes to point to the new Data Network and specify a particular vlan-id to use within that Data Network.
  
  In the example below, a NetworkAttachmentDefinition is created in the stx-lab namespace for vlan-id=20 on interfaces attached to the kube-sriov data network. In this example, vlan-id=20 is attached to a router which has this interface configured as the 10.10.186.0/24 IP Subnet.
  
  Note that the k8s.v1.cni.cncf.io/resourceName annotation is used to reference the specific Data Network, and has a structure of intel.com/pci_sriov_net_<data_network_name> (with all dashes in the data network name, if any, converted to underscores).
```
$ cat <<EOF > 186-subnet.yaml 
apiVersion: "k8s.cni.cncf.io/v1" 
kind: NetworkAttachmentDefinition 
metadata:
  name: 186-subnet
  namespace: stx-lab
  annotations:
    k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_kube_sriov
spec:
  config: '{
    "cniVersion": "0.3.0",
    "type": "sriov",
    "vlan": 20
  }'
EOF
```
4. Apply the configuration.
```
$ kubectl apply -f 186-subnet.yaml
```

Create the , login to the console and configure the 's interface on the 10.10.186.0/24 network. Then SSH to the from a remote workstation.

Use virtctl and the Upload Proxy service to load the ubuntu jammy cloud image into a new DataVolume of size 500G, in the stx-lab namespace.

$ virtctl image-upload dv stx-lab-ubuntu-test-disk -n stx-lab --insecure \
  --access-mode ReadWriteOnce --size 500Gi \
  --image-path /home/sysadmin/admin/kubevirt/images/jammy-server-cloudimg-amd64.img \
  --uploadproxy-url https://admin.starlingx.abc.com:32111

Create the yaml file defining the VirtualMachine instance.

$ cat <<EOF > stx-lab-ubuntu-test-vm.yaml
apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
  labels:
  kubevirt.io/vm: stx-lab-ubuntu-test
  name: stx-lab-ubuntu-test
  namespace: stx-lab
spec:
  running: true
  template:
    metadata:
      labels:
        kubevirt.io/vm: stx-lab-ubuntu-test
    spec:
      domain:
        devices:
          disks:
          - disk:
              bus: virtio
            name: myrootdisk
          - disk:
              bus: virtio
              name: cloudinitdisk
          interfaces:
          - masquerade: {}
            name: default
          - name: 186-subnet
            sriov: {}
        machine:
          type: ""
        resources:
          requests:
            cpu: 8 
            memory: 16Gi
      networks:
      - name: default
        pod: {}
      - multus:
          networkName: stx-lab/186-subnet
        name: 186-subnet
      terminationGracePeriodSeconds: 0
      volumes:
      - name: myrootdisk
        dataVolume:
          name: stx-lab-ubuntu-test-disk
      - cloudInitNoCloud:
        userData: |-
          #cloud-config
          user: jenkins
          password: myP@ssw0rd
          chpasswd: { expire: False }
          ssh_pwauth: True
        name: cloudinitdisk
EOF

Apply the configuration.

$ kubectl apply -f stx-lab-ubuntu-test-vm.yaml

Connect to console and configure and the 's interface on the 10.10.186.0/24 network.

$ virtctl -n stx-lab console stx-lab-ubuntu-test-vm
Successfully connected to stx-lab-ubuntu-test console. The escape sequence is ^]4

stx-lab-ubuntu-test login: jenkins 
Password:
Welcome to Ubuntu 22.04 LTS (GNU/Linux 5.15.0-39-generic x86_64)

* Documentation: https://help.ubuntu.com
* Management:    https://landscape.canonical.com
* Support:       https://ubuntu.com/advantage

 System information as of Thu Dec 8 16:55:12 UTC 2022

 System information as of Thu   Dec     8 16:55:12 UTC 2022

 System load:    0.2587890625        Processes:               178
 Usage of /:     0.3% of 476.62GB    Users logged in:         0
 Memory usage:   1%                  IPv4 address for enp1s0: 10.0.2.2
 Swap usage:     0%

 0 updates can be applied immediately.

 ...

Still in the console, list the interfaces.

Note that this has 2x interfaces.

enp1s0 is the default container interface
enp6s0 is the interface

jenkins@stx-lab-ubuntu-test:~$ ip link

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
  link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group
default qlen 1000
  link/ether 52:54:00:41:84:a0 brd ff:ff:ff:ff:ff:ff
3: enp6s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
  link/ether 46:c5:53:3b:b3:b3 brd ff:ff:ff:ff:ff:ff

jenkins@stx-lab-ubuntu-test:~$ ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
   inet 127.0.0.1/8 scope host lo
     valid_lft forever preferred_lft forever
   inet6 ::1/128 scope host
     valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
   link/ether 52:54:00:41:84:a0 brd ff:ff:ff:ff:ff:ff
   inet 10.0.2.2/24 metric 100 brd 10.0.2.255 scope global dynamic enp1s0
     valid_lft 86313505sec preferred_lft 86313505sec
   inet6 fe80::5054:ff:fe41:84a0/64 scope link
     valid_lft forever preferred_lft forever
3: enp6s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
   link/ether 46:c5:53:3b:b3:b3 brd ff:ff:ff:ff:ff:ff cumulus@stx-lab-ubuntu-test:~$

Still in the console, using the default interface (which has connectivity out the interface), update and upgrade the ubuntu deployment in the .
```
jenkins@stx-lab-ubuntu-test:~$ sudo apt-get update
jenkins@stx-lab-ubuntu-test:~$ sudo apt-get -y upgrade
```

Still in the console, configure Networking persistently with netplan and reboot

Specifically disable default CNI interface (enp1s0), and configure the interface (enp6s0).

$ sudo su -

$ cat <<EOF > /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg
network: {config: disabled}
EOF

# Update /etc/netplan/50-cloud-init.yaml as shown below.

$ vi /etc/netplan/50-cloud-init.yaml

network:
   ethernets:
     # enp1s0:
       # dhcp4: true
     enp6s0:
       dhcp4: no
       addresses:
         - 10.10.186.97/24
       nameservers:
         addresses: [10.10.186.130]
       routes:
         - to: default
           via: 10.10.186.1
           version: 2

# Apply the updates
$ netplan apply

# Restart the system
$ /sbin/reboot

< LOGS FROM BOOTING ON CONSOLE >

[ OK   ] Finished Execute cloud user/final scripts.
[ OK   ] Reached target Cloud-init target.
Ubuntu 22.04.1 LTS stx-lab-ubuntu-test ttyS0
stx-lab-ubuntu-test login: jenkins
Password:

Last login: Thu   Dec     8 16:55:13 UTC 2022 on   ttyS0

jenkins@stx-lab-ubuntu-test:~$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
   link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp1s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
   link/ether 52:54:00:41:84:a0 brd ff:ff:ff:ff:ff:ff
3: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
   link/ether 46:c5:53:3b:b3:b3 brd ff:ff:ff:ff:ff:ff 

jenkins@stx-lab-ubuntu-test:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
   link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
   inet 127.0.0.1/8 scope host lo
     valid_lft forever preferred_lft forever
   inet6 ::1/128 scope host
     valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
   link/ether 52:54:00:41:84:a0 brd ff:ff:ff:ff:ff:ff
3: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
   link/ether 46:c5:53:3b:b3:b3 brd ff:ff:ff:ff:ff:ff
   inet 10.10.186.97/24 brd 10.10.186.255 scope global enp6s0
     valid_lft forever preferred_lft forever
   inet6 2620:10a:a001:a106:44c5:53ff:fe3b:b3b3/64 scope global dynamic mngtmpaddr noprefixroute
     valid_lft 2591972sec preferred_lft 604772sec
   inet6 fe80::44c5:53ff:fe3b:b3b3/64 scope link
     valid_lft forever preferred_lft forever

jenkins@stx-lab-ubuntu-test:~$ ip route
default via 10.10.186.1 dev enp6s0 proto static
10.10.186.0/24 dev enp6s0 proto kernel scope link src 10.10.186.97

Connect from a remote workstation.

$ ssh jenkins@10.10.186.97
password:

Last login: Thu Dec 8 18:14:18 2022

jenkins@stx-lab-ubuntu-test:~$ ip link

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
   link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp1s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
   link/ether 52:54:00:41:84:a0 brd ff:ff:ff:ff:ff:ff
3: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
   link/ether 46:c5:53:3b:b3:b3 brd ff:ff:ff:ff:ff:ff

13 KiB Raw Blame History

Create an Ubuntu VM

13 KiB

Raw Blame History