AppArmor support for StarlingX
This specification describes the introduction of AppArmor into the StarlingX solution. Story: 2010310 Task: 46355 Signed-off-by: jmishra <jagatguruprasad.mishra@windriver.com> Change-Id: Id76a3da52c148c77bfd78cafbe583b94345e491a
This commit is contained in:
jmishra
parent
f618ca6764
commit
a09b153e2f
|
|
@ -0,0 +1,357 @@
|
|||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License. http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
========================================
|
||||
StarlingX AppArmor Support
|
||||
========================================
|
||||
|
||||
Storyboard:
|
||||
https://storyboard.openstack.org/#!/story/2010310
|
||||
|
||||
This specification describes the introduction of AppArmor into the
|
||||
StarlingX solution, initially for the purposes of running end user's
|
||||
hosted containers under AppArmor profiles.
|
||||
|
||||
AppArmor is a Mandatory Access Control (MAC) system built on Linux's
|
||||
LSM (Linux Security Modules) interface. In practice, the kernel queries
|
||||
AppArmor before each system call to know whether the process is authorized
|
||||
to do the given operation. Through this mechanism, AppArmor confines programs
|
||||
to a limited set of resources.
|
||||
|
||||
AppArmor applies a set of rules (known as a “profile”) on each program. The
|
||||
profile applied by the kernel depends on the installation path of the program
|
||||
being executed, the rules applied do not depend on the user. All users face
|
||||
the same set of rules when they are executing the same program but traditional
|
||||
user permissions still apply and might result in different behavior.
|
||||
|
||||
AppArmor profiles contain a list of access control rules on resources that each
|
||||
program can make use of. Each profile can be loaded either in enforcing or
|
||||
complaining mode. The former enforces the policy and reports violation attempts,
|
||||
while the latter does not enforce the policy but still logs the system calls
|
||||
that would have been denied.
|
||||
|
||||
AppArmor helps administrators in running a more secure kubernetes deployment by
|
||||
restricting what containers/pods are allowed to do, and/or provide better
|
||||
auditing through system logs.The access needed by a container/pod is configured
|
||||
through profiles tuned to allow the access such as Linux capabilities, network
|
||||
access, file permissions, etc.
|
||||
|
||||
|
||||
|
||||
Problem description
|
||||
===================
|
||||
Need a mechanism for restricting application pods run by the end users. AppArmor does
|
||||
this by allowing the system administrator to restrict programs' capabilities
|
||||
with per-program profiles.Then Kubernetes' container.apparmor.security.beta.kubernetes.io
|
||||
annotation can be used in a pod spec to assign an AppArmor profile to a pod.
|
||||
|
||||
Management of AppArmor profiles across all the kubernetes hosts of a StarlingX kubernetes
|
||||
cluster should be simple for end user.
|
||||
|
||||
There are concerns over the system performance degradation when enabling the
|
||||
AppArmor. However, some end users want it enabled for security reasons, and are willing
|
||||
to accept the system performance degradation.
|
||||
|
||||
|
||||
Use Cases
|
||||
---------
|
||||
|
||||
AppArmor can be optionally enabled on a per host basis:
|
||||
|
||||
* Default recommendation will be that AppArmor is enabled on all hosts
|
||||
* However, if required (e.g. to avoid performance impact on some cards), user could choose which host to enable AppArmor profiles (e.g., Enable AppArmor only on controllers in a Standard System) NOTE In this case, a user would need to use labels and node-selector in pod spec to ensure the pod is only scheduled on nodes with AppArmor enabled.
|
||||
|
||||
If AppArmor is enabled on host(s) of a StarlingX system, a user will be able to run selected user's hosted application pods under a selected AppArmor profile by specifying the AppArmor profile in the pod's spec.
|
||||
|
||||
**NOTE** Support for AppArmor profiles on the StarlingX Platform entities (both host processes and containers) themselves, is left to a future release.
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
This change provides a way to enable/disable on a particular host.
|
||||
|
||||
* Default recommendation will be that AppArmor is enabled on all hosts
|
||||
* However, if required (e.g. to avoid performance impact on some hosts), user could choose
|
||||
which host to enable AppArmor profiles (e.g., Enable AppArmor only on controllers in a Standard System)
|
||||
**NOTE** In this case, a user would need to use labels and node-selector in pod spec to ensure the pod
|
||||
is only scheduled on nodes with AppArmor enabled.
|
||||
|
||||
In order apply the profiles to a particular pod, the profiles need to be available to the host machine
|
||||
where pod is launched. Security Profile Operator (SPO, https://github.com/kubernetes-sigs/security-profiles-operator)
|
||||
provides AppArmor profile management (i.e. loading/unloading) across k8s nodes. SPO defines an AppArmor Profile CRD, such
|
||||
that end users' can define AppArmor profiles for SPO to manage.
|
||||
|
||||
SPO will be integrated as a system application using FluxCD manifest which utilizes helm chart that
|
||||
deploys a controller/operator for centrally managing the AppArmor Profile resources, a webhook for validation of AppArmor
|
||||
Profile resources and a daemonset for loading/unloading the AppArmor Profiles. The DaemonSet configuration creates an SPO
|
||||
pod/container on each host in the StarlingX cluster.
|
||||
Creating a DaemonSet configuration for AppArmor container ensures that SPO will run on all kubernetes nodes,
|
||||
such as controllers, all-in-one controllers and worker nodes.
|
||||
|
||||
Once an AppArmor profile is loaded to the K8S nodes, it can be applied to a particular pod using annotations.
|
||||
| example: container.apparmor.security.beta.kubernetes.io/<pod_name>:localhost/<profile_ref>. Please refer
|
||||
https://kubernetes.io/docs/tutorials/security/apparmor/#example
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
In order apply the profiles to a particular pod, the profiles need to be available to the host
|
||||
machine where pod is launched. There are several solutions available and below alternatives were evaluated:
|
||||
|
||||
* security-profiles-operator
|
||||
* apparmor-loader
|
||||
* kube_apparmor_Manager
|
||||
* Manually copying and loading profiles on each host (DEV Options)
|
||||
|
||||
Below is a comparision table
|
||||
|
||||
.. list-table:: Alternatives to load AppArmor profiles in K8s cluster
|
||||
:widths: 10 20 30 20 20
|
||||
:header-rows: 1
|
||||
|
||||
* -
|
||||
- AppArmor Loader
|
||||
- Security Profile Operator(SPO)
|
||||
- AppArmor_Kube_Manager
|
||||
- Manually Loading Profiles (DEV options)
|
||||
* - Community Type
|
||||
- K8S
|
||||
- K8S
|
||||
- Private, sysdiglab
|
||||
- NA
|
||||
* - Approach
|
||||
- Daemon Set
|
||||
- Daemon Set
|
||||
- SSH based copy
|
||||
- Manual copy (SSH based)
|
||||
* - Developement Type
|
||||
- POC
|
||||
- Controlled as Alpha, beta release
|
||||
- Not sure
|
||||
- NA
|
||||
* - Active
|
||||
- Last updated 4 years back
|
||||
- Very Much Active
|
||||
- Last updated 2 years back
|
||||
- NA
|
||||
* - Release Frequency
|
||||
- NA
|
||||
- Quarterly basis
|
||||
- NA
|
||||
- NA
|
||||
* - Latest Version
|
||||
- No version control
|
||||
- V0.4.3
|
||||
- V0.0.5
|
||||
- NA
|
||||
* - Last Release Data
|
||||
-
|
||||
- Jun 07, 2022
|
||||
- Apr 04, 2020
|
||||
- NA
|
||||
* - Roadmap
|
||||
- No
|
||||
- Yes, KEP and Roadmap in place
|
||||
- No
|
||||
- NA
|
||||
* - Github Link
|
||||
- https://github.com/kubernetes/kubernetes/tree/master/test/images/apparmor-loader
|
||||
- https://github.com/kubernetes-sigs/security-profiles-operator
|
||||
- https://github.com/sysdiglabs/kube-apparmor-manager
|
||||
- NA
|
||||
* - Functionality
|
||||
- Only Load AppArmor profiles on all worker nodes in k8s cluster
|
||||
- Out-of-tree Kubernetes enhancement | Supports SELinux, seccomp, AppArmor | AppArmor profiles load and unload | Feature enabling and disabling using feature gate | Supplementary features – Metrics , service monitoring, log enricher, | Debugging features- CPU and Memory profile endpoints
|
||||
- Load AppArmor profiles on all worker nodes in k8s cluster | Sync feature to cater profile change synchronize across worker nodes
|
||||
- Copy the profiles on each host using SSH | Load the profile on each host using aa-enforce, aa-complain, or apparmor_parser | Unload the profiles on each host using aa-disable or apparmer_parser
|
||||
|
||||
|
||||
Security Profile Operator (SPO) is the best option available and will continue our effort in incorporating it in StarlingX based on below advantages.
|
||||
|
||||
* Better community support
|
||||
* Feature rich
|
||||
* Roadmap in place
|
||||
|
||||
There are concerns about the stability of AppArmor support in SPO and the basic functionality of deploying AppArmor
|
||||
profiles to various nodes is not working at this point. We will continue our effort to
|
||||
track SPO fixes to apparmor functionality.
|
||||
|
||||
* Retest updates to SPO and provide feedback to contributor quickly
|
||||
* Offer any assistance we can to SPO contributor
|
||||
* Identify any missing/desired functionality (e.g. complain vs enforce)
|
||||
|
||||
For now, we will proceed in parallel with
|
||||
|
||||
* using DEV option of linux AppArmor tools (aa commands, https://gitlab.com/apparmor/apparmor/-/wikis/ManPages) to load apparmor profiles on hosts AND
|
||||
* work closely in testing and providing feedback (and possibly changes) to SPO project
|
||||
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
A host level attribute "apparmor" would be added to the host table.
|
||||
This can contain two possible values "enabled" or "disabled"
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
AppArmor can be enabled or disabled at runtime on a per-host basis.
|
||||
|
||||
* Below cli command(s) would be used to enable or disable AppArmor in kernel for a particular host
|
||||
* system host-lock <hostname>
|
||||
* system host-update <hostname> apparmor=enabled/disabled **Example:** system host-update controller-0 apparmor=enabled
|
||||
* system host-unlock <hostname>
|
||||
|
||||
The above command(s) would enable the AppArmor in kernel with NO services (on host or containerized)
|
||||
in the WRCP/StarlingX Solution running under an apparmor profile
|
||||
**NOTE** will explore whether default profiles enabled in Debian Bullseye can be used as default
|
||||
when AppArmor is enabled
|
||||
|
||||
* Changes in the below API to reflect AppArmor. |
|
||||
GET /v1/ihosts/{host_id} |
|
||||
Description: Shows detailed information about a specific host |
|
||||
INPUT: |
|
||||
host_id : csapi:UUID: part of user-info |
|
||||
Output: |
|
||||
"apparmor": "enabled" or "apparmor": "disabled" would be added to the output json |
|
||||
|
||||
* Addition of a new attribute 'apparmor' to the below API:
|
||||
|
||||
| PATCH /v1/ihosts/{host_id}
|
||||
| Apparmor attribute can be modified. Support would be added for apparmor (xsd:string)
|
||||
| INPUT:
|
||||
| host_id : csapi:UUID: part of user-info
|
||||
| Example: [
|
||||
| {
|
||||
| "path":"/apparmor",
|
||||
| "value":"enabled",
|
||||
| "op":"replace"
|
||||
| }
|
||||
| ]
|
||||
| Output:
|
||||
| "apparmor": "enabled" or "apparmor": "disabled" would be added to the output json
|
||||
| **NOTE**: Changing the 'apparmor' attribute, will ONLY be allowed when host is locked.
|
||||
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
The AppArmor helps in running a more secure kubernetes deployment by restricting what
|
||||
containers/pods are allowed to do. User should be able to run selected user's hosted
|
||||
application pods under a selected AppArmor profile. AppArmor enforces the policy defined
|
||||
in the profile as well as reports policy violation attempts.
|
||||
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
* If apparmor is NOT enabled (disabled by default), then there is NO impact on end user
|
||||
* If apparmor is used,
|
||||
* there is admin user impact to enable apparmor on hosts and load apparmor profiles on host and
|
||||
* there is general user impact that they 'can' run hosted containers under apparmor profiles for enhanced security.
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
AppArmor is known to have a performance impact when running. There will be a performance impact characterization
|
||||
(StarlingX Infrastructure Operations and Datapaths) of enabling AppArmor in the StarlingX which involved getting data with:
|
||||
|
||||
* apparmor disabled
|
||||
* apparmor enabled but NO apparmor profile applied to datapath app pod
|
||||
* apparmor enabled and apparmor profile applied to datapath app pod
|
||||
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
None.
|
||||
|
||||
Upgrade impact
|
||||
--------------
|
||||
|
||||
A new host level attribute "apparmor" would be added to the host table. There should not be any impact on backup and restore, upgrade, and rollback.
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
* Jagatguru Prasad Mishra (jmishra)
|
||||
|
||||
Other contributors:
|
||||
* Sirin Shaikh (sshaikh)
|
||||
* Kirti Singh (ksingh)
|
||||
* Rahul Roshan Kachchap (rkachchap)
|
||||
|
||||
|
||||
Repos Impacted
|
||||
--------------
|
||||
|
||||
* starlingx/config and starlingx/stx-puppet repo would be modified to add API/CLI support for AppArmor.
|
||||
* A new repo would be created for security-profiles-operator.
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
* Integration of SPO as a system application
|
||||
* CLI, RESTAPI, DB update and config update backend
|
||||
* Performance impact testing
|
||||
* End-to-end integration testing
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
* Security Profile Operator (SPO, https://github.com/kubernetes-sigs/security-profiles-operator)
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
Testing will involve:
|
||||
|
||||
* Unit test cases for the new code introduced
|
||||
* Installing/enabling the SPO system app
|
||||
* verifying the enabling/disabling of Apparmor at host level using API and CLI
|
||||
* Configuring Apparmor profiles across hosts
|
||||
* Configuring Kubernetes Pods with Apparmor profiles
|
||||
* Verifying that a policy violation by a container is logged in file "/var/log/syslog".
|
||||
* Backup and Restore testing
|
||||
* Upgrades testing
|
||||
* DC testing
|
||||
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
Some information that is useful to be documented will be:
|
||||
|
||||
* enabling Apparmor per host
|
||||
* Configuring Apparmor profiles across hosts
|
||||
* Configuring Kubernetes Pods with Apparmor profiles with example to do simple test
|
||||
* Guidelines on how to write/develop apparmor profiles for your containers using aa-genprof
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
* https://www.appvia.io/blog/how-to-use-the-security-profiles-operator
|
||||
* https://medium.com/information-and-technology/so-what-is-apparmor-64d7ae211ed
|
||||
* https://github.com/kubernetes-sigs/security-profiles-operator
|
||||
* https://github.com/kubernetes/kubernetes/tree/master/test/images/apparmor-loader
|
||||
* https://github.com/sysdiglabs/kube-apparmor-manager
|
||||
* https://kubernetes.io/docs/tutorials/security/apparmor/#example
|
||||
|
||||
|
||||
History
|
||||
=======
|
||||
|
||||
.. list-table:: Revisions
|
||||
:header-rows: 1
|
||||
|
||||
* - Release Name
|
||||
- Description
|
||||
* - STX-8.0
|
||||
- Introduced
|
||||
Loading…
Reference in New Issue