Go to file

Rei Oliveira 04bb563752 Recover k8s certs after long period offline

These two scripts can be used to recover k8s certificates after they
expire due to a long period offline.

Recovery can be performed by first running kube-cert-rotation.sh and
then kube-expired-kubelet-cert-recovery.sh.

Note:

- As it is now, it only works for AIO-SX standalone and subcloud
environments. Additional work will be done for other types of
environments.

- The k8s Root CA will not be addressed in this commit. It's under
consideration whether the K8s Root CA recovery is required as Root CAs
have very long durations. Incremental work for it will be posted in
a followup review if necessary.

Test case:
PASS: In a system where k8s certificates are not expired, run sudo
      show-certs.sh and take note of the kubelet certificates dates
      (kubelet-client-current.pem, kubelet-server, kubelet CA).
      Now run kube-expired-kubelet-cert-recovery.sh and verify
      that dates have not changed.
PASS: In a system where k8s certificates are not expired, run sudo
      show-certs.sh and take note of the k8s leaf certificates dates
      (admin.conf,apiserver, apiserver-kubelet-client,
      controller-manager.conf, front-proxy-client, scheduler.conf).
      Now run kube-cert-rotation.sh and verify that dates have
      not changed.
PASS: After the 2 steps above, run 'kubectl get po -A' to verify the
      health of the cluster.

PASS:
- Change 'hardware clock' of vbox vm to more than 1 year in the future.
- Verified, after turning on vm, that system date was now 1 year ahead
- Verified that kubernetes was not responding. No kubectl commands were
  accepted. 'sudo show-certs.sh' showed etcd and kubelet certs expired
  and 'sudo kubeadm certs check-expiration' replies with an error, which
  indicates certificates have already expired.
- Run kube-cert-rotation.sh followed by kube-expired-kubelet-cert-recovery.sh
- Verify that after 1 minute kubectl starts to respond again.
- Run 'sudo show-certs.sh' and verify that k8s certificates (admin.conf,
  apiserver, apiserver-kubelet-client, controller-manager.conf,
  front-proxy-client, scheduler.conf), the etcd certificates
  (etcd-client.crt, etcd-server.crt, apiserver-etcd-client.crt)
  and kubelet certificates (kubelet-client-current.pem, kubelet-server,
  kubelet CA) now show valid dates.
- Run kubectl commands such as 'kubectl delete pod' and verify k8s is
  able to respond and restart pods successfully.

Story: 2010815
Task: 48343

Change-Id: I7f71201e7f9ce83f79dc68c68a8a4d55ee3c69aa
Signed-off-by: Rei Oliveira <Reinildes.JoseMateusOliveira@windriver.com>

2023-07-24 17:43:05 -03:00