From 9c47eec8c38df14c9016ca08cd29719ac54eea43 Mon Sep 17 00:00:00 2001 From: Jiangyuan Date: Thu, 29 Mar 2018 12:48:56 +0000 Subject: [PATCH] spec doc for adding customized mem/cpu for function Change-Id: Ie14a2d24f31e4a1a9c7c278bcd560a7b20030507 --- doc/source/specs/customize_resource.rst | 165 ++++++++++++++++++++++++ doc/source/specs/index.rst | 1 + 2 files changed, 166 insertions(+) create mode 100644 doc/source/specs/customize_resource.rst diff --git a/doc/source/specs/customize_resource.rst b/doc/source/specs/customize_resource.rst new file mode 100644 index 00000000..98220758 --- /dev/null +++ b/doc/source/specs/customize_resource.rst @@ -0,0 +1,165 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +========================================== +Add customized memory and cpu for function +========================================== + +https://storyboard.openstack.org/#!/story/2001586 + +Only users know the amount of resources that their functions may need, so +qinling should provide cpu and memory_size options for users to customize +resources. When creating function, allow user to specify cpu and memory_size +resources for the function so that qinling can allocate and manage resources +of cpu and memory more reasonably in this way. + + +Problem description +=================== + +- For deployers, they may want to manage system resources reasonably and + safely by restricting functions' resource occupancy so that more executions + could share the fixed amount of resources and non-interfering with each + other. + +- For users, applying for resources according to actual needs can help them + prevent the waste of resources, which will make sense when users need to pay + for the resources they use. + + +Proposed change +=============== + +For different types of functions, the ways of limiting resources are +distinguishing. + +- For image type functions, we could config the resource limitation in pod + definition, and then kubernetes will help us limit the whole pod, which + means the total amount of resources used by the function inside the pod + could never exceed the pod's limit. + +- For package/swift type function, it turns to be the runtime server's + responsibility to set resource limits for subprocess that function runs in. + +Some details are as following: + +First of all, make sure the values of memory_size and cpu saved in the function +database are valid. In ``api.controllers.v1.function.post``, we do type and +size check for cpu and memory_size params so that they must be integers and +within the range set in config.py, and qinling will supply default values for +them if users do not input anything. The default values are set to be the +minimum in this range. + +Besides, in ``api.controllers.v1.function.put``, we add cpu and memory_size to +``UPDATE_ALLOWED``. If user wants to update them, we will do type and size +check for cpu and memory_size params before updating function database. + +When creating execution, pass cpu and memory_size values saved in function +database to ``qinling.engine.default_engine.create_execution``. Here we will +do the check for cpu and memory_size values again in case the limited scope of +resources set in config.py have been reset. + +Then for image type function, pass both of them to +``orchestrator.kubernetes.manager._create_pod``, and we can get a pod with +limited cpu and memory resources. For package/swift type function, pass both of +them to the worker pod in the k8s deployment by using +``qinling.engine.utils.url_request``. In the selected worker pod, cpu and +memory_size values will be used to limit the total amount of resources that +function process and its subprocesses can use. + +We are considering using 'cgroup' in linux to limit cpu and memory because by +'cgroup' we can add pids of function process and its subprocesses to the same +'control group' and limit the total amount of resources. But we need to use +different users to set resource files in 'cgroup' and run functions because +function should not be granted permission to modify the resource setting files +in 'cgroup'. We use root to create a 'control group', and qinling can only +write to the 'tasks' file. + +- For cpu resource limitation, 'cpu.cfs_quota_us' and 'cpu.cfs_period_us' + files in 'cgroup' will be used to convert millicpu value. + +- For memory resource limitation, only 'memory.limit_in_bytes' file will be + used to limit RAM because now in k8s source code, the kubelet does not + support running with swap enabled. Although it also provides a workaround + '--fail-swap-on=false' to allow swap on, which may cause some performance + impacts, we would better to disable swap now. For more details about swap, + please see the references. + +Data model impact +----------------- + +Add a cpu column for function database to save cpu value specified by user. + +REST API impact +--------------- + +* Add cpu and memory_size options for function creation. + +* Allow to update cpu and memory_size values saved in function database. + +End user impact +--------------- + +When using python-qinlingclient to create/update function, the CLI may look +like 'openstack function create/update --cpu xxx --memory_size xxx ...'. + +Performance Impact +------------------ + +None. + +Deployer impact +--------------- + +The config options for min/max size of cpu and memory_size will be provided in +``qinling.config``. The unit of cpu is 'millicpu' and the unit of memory_size +is 'bytes'. + +Alternatives +------------ + +We have considered using resource.RLIMIT_AS to limit memory resource that +function can use. However if function forks other child processes, the child +processes will inherits its parent's resource limits, instead of sharing the +limits. + + +Implementation +============== + +Assignee(s) +----------- + +Jiangyuan + + +Dependencies +============ + +None. + + +Testing +======= + +None. + + +References +========== + +* Resource model + https://docs.python.org/2.7/library/resource.html#resource-limits +* Patch for image type function's resource limit + https://review.openstack.org/#/c/553947 +* IRC discussions + http://eavesdrop.openstack.org/irclogs/%23openstack-qinling/%23openstack-qinling.2018-03-23.log.html +* K8s source code about swap + https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cm/container_manager_linux.go#L203 +* An open issue in k8s about swap + https://github.com/kubernetes/kubernetes/issues/53533 +* Some discussions about why disable swap + https://serverfault.com/questions/881517/why-disable-swap-on-kubernetes diff --git a/doc/source/specs/index.rst b/doc/source/specs/index.rst index dc4301ca..56a5cff0 100644 --- a/doc/source/specs/index.rst +++ b/doc/source/specs/index.rst @@ -29,6 +29,7 @@ Specs :maxdepth: 1 function_versioning + customize_resource Template --------