The Divingbell Ubuntu job is currently a voting job, even though it does
not run as a gate job on merged changes. Since the OpenDev migration,
the job has failed consistently. New issues have surfaced while
attempting to correct the issue [0].
This change moves the job to non-voting in order to allow the merge of
critical work while the cause of the failures is investigated.
[0] https://review.opendev.org/663392
Change-Id: Ie44bb7f0160acc362af212028cc553ec99090acd
Signed-off-by: Drew Walters <andrew.walters@att.com>
This makes the main container within the apt daemonset run as
privileged, which is required to perform kernel upgrades through it.
It was confirmed that even with all capabilities enabled, an
unprivileged apt is unable to perform the necessary updates to
the boot partition during a kernel upgrade.
Change-Id: I4e996794f24fcfc9d8ced7a58cecd2ceec36f6c5
Use the common logger for consistent log output for some echo statements
that were not making use of it.
Change-Id: I7fae2a950318f5cd3245a4571dc464009726d4ae
This PS allows to avoid of using assignments which are not supported
in older versions of Helm (GO<1.11).
Change-Id: Ic0dad4d1b60071c4366c63834f1ad7e3a76fdcd8
This commit introduces a non-voting job to lint Helm charts against the
latest version of Helm toolkit from OpenStack-Helm Infra. This job
should serve as an indicator of when it's safe to advance the version of
Helm toolkit used by Airship.
Additionally, this commit modifies all Helm chart lint jobs to run on
each commit, regardless of the files modified by a change. This should
not introduce a noticeable difference in CI runtime, as these jobs
execute quicker than the tox jobs.
Change-Id: Iffbe718f2f8cabaac74910e0c40a13e17e3f0578
A recent change made most Divingbell Daemonsets run as unprivileged containers:
https://review.openstack.org/#/c/639435/
Change-Id: If4e04368a3de3c7de7a3cf64692e5dd1294234b6
Enhanced rendering of docs, expanded introductory section.
Documented `perm` module, alphabetically sorted documenation for
modules, replaced dead recorded demo links with new links,
documented apt package blacklisting capability.
Change-Id: Ifd889efe73287c13d839ab40b1a78ffa357fd00e
Divingbell runs all its containers as privileged. Some Divingbell
containers can perform their jobs with the default set of Linux
capabilities that Docker gives to unprivileged containers while others
need additional capabilities. The default list of capabilties include
the following:
- SETPCAP
- MKNOD
- AUDIT_WRITE
- CHOWN
- NET_RAW
- DAC_OVERRIDE
- FOWNER
- FSETID
- KILL
- SETGID
- SETUID
- NET_BIND_SERVICE
- SYS_CHROOT
- SETFCAP
The capabilities listed in the daemonset templates function as a
whitelist in that the corresponding containers have access to the Linux
capabilities listed in their SecurityContext, but also the
aforementioned capabilties included by default by Docker.
Summary of testing for each daemonset:
The bcc-capable tool [0] was used to discover which Linux capabilities
the Divingbell containers invoke. The tool was ran against all the
processes running in the container. The Divingbell logs for each
container were also carefully analyzed for failed permission checks.
daemonset-exec:
A recent change to use nsenter to enter all host namespaces when running
exec prevents divingbell-exec from being able to run unprivileged as
there are no Linux capabilties that allows write access to '/proc'.
When trying to run as unprivileged, the following prevents the pod from
coming up:
"nsenter: cannot open /proc/1/ns/ipc: Permission denied"
daemonset-sysctl:
Ran the divingbell-sys containers as unprivileged and the kernel config
on the host updated as defined in the manifest. Kernel configs were
checked before and after running divingbell-sys container as
unprivileged. Beyond the default Linux capabilties given by
Docker, the 'SYS_PTRACE', 'SYS_ADMIN', and 'SYS_RAWIO' Linux
capabilities are needed. The following is a snippet of the logs showing
under which circumstance these privileges are needed:
"INFO * Applying /etc/sysctl.d/10-kernel-hardening.conf ...
INFO sysctl: setting key "kernel.kptr_restrict": Operation not permitted
INFO * Applying /etc/sysctl.d/10-ptrace.conf ...
INFO sysctl: setting key "kernel.yama.ptrace_scope": Operation not
permitted
INFO * Applying /etc/sysctl.d/10-zeropage.conf ...
INFO sysctl: setting key "vm.mmap_min_addr": Operation not permitted"
daemonset-perm:
Ran the divingbell-perm containers as unprivileged and the file
ownership and permissions on the host updated as defined in the
manifest. As a test, the daemon was configured to run every minute
and the targeted files ownership and permissions were manually
changed. It was then verified that divingbell restored the ownership
and permissions of the file to what it should be. This applies to
the divingbell-perm-default and the divingbell-perm-calico containers.
daemonset-limits:
Ran the divingbell-limits containers as unprivileged and checked the
ulimits on the host before and after running divingbell and the ulimit
updated to the value defined in the manifest. The capable tool also
showed that no additional Linux capabilties are needed.
daemonset-apparmor:
Ran the divingbell-apparmor containers as unprivileged and logs show no
evidence of failed permission checks. Additionally, the apparmor config
was updated in the manifest and the apparmor profile successfully
loaded. Beyond the default Linux capabilties given by Docker, the
'MAC_ADMIN' Linux capability is needed to load an apparmor profile.
daemonset-apt:
Ran the divingbell-apt containers as unprivileged and was able to
successfully install package without issues. As a test, the
manifest was updated to install 'htop' and after running Divingbell,
it was confirmed that 'htop' installed successfully. Here is
a snippet from the logs:
DEBUG + INSTALLED_THIS_TIME=' htop'
DEBUG + REQUESTED_PACKAGES=' htop'
daemonset-ethtool:
Ran the divingbell-ethtool containers as unprivileged and was able to
manage NIC tunables. As a check, the NIC tunables for ens3 was checked
before and after running Divingbell - 'ethtool -k ens3'. Divingbell
configured the NIC as defined in the manifest. Beyond the default Linux
capabilties given by Docker, the 'NET_ADMIN' Linux capability is needed.
The following is a log snippet showing what happens when the 'NET_ADMIN'
capability is not added:
"DEBUG + /sbin/ethtool -K cali86cb821b7db tx-nocache-copy off
INFO Cannot set device feature settings: Operation not permitted"
daemonset-uamlite:
Ran the divingbell-uamlite containers as unprivileged and was able to
successfully add user accounts as defined in the manifest. No additional
Linux capabilities are needed.
daemonset_mounts:
Ran the divingbell-mounts containers as unprivileged and was able to
successfully add host level mounts as defined in the manifest. No
additional Linux capabilities are needed.
[0]https://github.com/iovisor/bcc/blob/master/tools/capable.py
Change-Id: I26a1b5e06ad27c854d95e6675de05b884ce3bdc1
This pins the version of helm-toolkit used during the build process to a
compatible version. This should be updated to master once helm 2.13.1
releases with this fix [0].
[0]: https://github.com/helm/helm/pull/5411
Change-Id: I8f273b96e35e66479e62dd2c7f362fd10c1e3bff
This PS moves to pivot to the hosts namespaces rather than chroot
so as to allow scripts to run fully in the context of the host.
Change-Id: I6b4dab92b6f8a7f9fa5b895d546117fdae43d731
Signed-off-by: Pete Birley <pete@port.direct>
- When reverting permissions on a file, there is no check for existence
causing a deleted file to CL the perm module
Change-Id: Ifae0ac196acf8ac2ccef84102967b6b4305a7691
- Adds the ability to rerun divingbell-perm at specified interval.
- Adds the ability to specify a rerun policy of
'always', 'never', 'once_successfully'. Default value is 'always'.
Demo: https://asciinema.org/a/220289
Change-Id: I3909b4d92f8e2bdb0d826ca1cfbd62f937c2532d
The previous README file for Divingbell did not render correctly
as a md. This change converts it to a small RST file.
Change-Id: Iabfc3eb6ed763ddf306d85f85399f2f3f99069ae
Add support for retries and reruns at specified intervals for
divingbell-exec scripts. Also adds support for timeouts.
Also update osh-infra-upgrade-host to allow gate to run.
Change-Id: I5f4cd43b13a467d94f67b358f3190f515256ae66
via new module 'perm'
1) DaemonSet
2) Secret (instead of old ConfigMap)
3) Include module /bin/_perm.sh.tpl
4) Commented example in values.yaml
5) Demo: https://asciinema.org/a/209509
6) Increased # of expected DaemonSets
7) Rebased after a few merges
8) Addressing comments
9) Migrated from ConfigMap to Secret
10) Got rid of 'eval'
11) Test
12) Demo for host targeting: https://asciinema.org/a/213125
Change-Id: Ia3181dcb7fc1ccc7422c635b010000f6d3fbcf4d
In Makefile there is no target which actually installs
Helm binary.
Change-Id: Idca3ed4d0c6d8734b7b6bcfc9d8a1ba9e50693ee
Signed-off-by: Dimitrios Markou <dm844v@att.com>
This change also adds an apt-get update call and a possibility to
provide debconf options that might be needed for some packages.
In case of dpkg interruptions dpkg --configure -a is added to
try to handle the failures.
Change-Id: Ib1f9a412bc544b4f7754634740fb04569bae6d34
For some versions of Tiller, the newline in Secrets were causing
Tiller to error with illegal base64 data error messages.
Change-Id: Ibd005c9b81e620590b0025bd32da4d589eba91ae
- Extends apt daemonset to remove packages.
- Uses a list of packages in remove and automove, so if a
package gets installed that should not be it will be
uninstalled when Divingbell runs again.
Change-Id: Id5c7ccead399a8c78621a0e593033e55412ff315
This change adds a possibility to install or upgrade to packages
with a specific version. The daemonset also tracks the packages
installed, and will be removing the packages that were deleted
from the chart but were previously installed by divingbell.
Change-Id: Ia6066679e549190054eb2cf71589065177447447
Change configmaps to secrets to maintain compatibility with [0].
[0] https://review.openstack.org/#/c/617039
Change-Id: Ie95aee1a4104008ca93c23ac9d19245a87fade20
Avoid filename collision between limits and sysctl module.
Bugfix for sysctl to print reverted setting before it's deleted.
Change-Id: I31269c413f884a25ecf3588b52677ca427b4c082
1) 'Values' configures limit settings to be persisted.
2) Previous DivingBell controlled limits those were set
but now are gone are cleared.
3) Previous values of newly set limits are backed up
to /var/divingbell/limits
4) New limit is applied via adding a separate conf file
to /etc/security/limits.d
5) The Doc is updated with appropriate details.
6) Dev env with Vagrant
7) Increase number of expected DaemonSets in 020-test
8) Demo: https://asciinema.org/a/209619
Change-Id: I5efb39c498c2b666b4ba97271b59757f4a0c1ca7
