Commit Graph

68 Commits (c19c4dca002ffaf352d63de5be2d294d6e49893f)

Author SHA1 Message Date
Lingxian Kong c19c4dca00 Improve function execution performance
- Improve/Simplify python runtime
- Use non-locking for communication with db when invoking function.
- Add config option to run job handler.
- Pre-fetch runtime when query function in db.

Change-Id: I9e791aecf03d5bf64a39548e6c9bad1b678882af
6 years ago
Guoqiang Ding 01be0d2a1f Fix the deprecated usage of "get_transport"
As log says, 'oslo_messaging.transport.get_transport()' is deprecated.
The reference link of oslo_messaging is at [1].

[1] https://review.openstack.org/#/c/454194/

Change-Id: Ib4e8cc7b30a7541221d9ae2d244577b2dae5518d
6 years ago
Lingxian Kong 6c013ddb7b Limit cpu resource consumption of function
Partially implements: blueprint qinling-container-resource-limitation

Change-Id: I2bb850e2d9bcf33e69d562abdbf88a226e1f4b92
6 years ago
Lingxian Kong cfd63da7b5 Limit function memory consumption
- Limit memory utilization of function execution.
- Timeout handling(3 min by default)
- Python runtime improvement

Partially implements: blueprint qinling-container-resource-limitation

Change-Id: I4ea991035a446acf78f8d12991a616f8637f95ec
6 years ago
Lingxian Kong c22dac0df1 Fix function package update
Change-Id: I534978de7ccc65f00ca09e32eddc1b829af47925
6 years ago
Zuul 87fee6a936 Merge "Support to update function code" 6 years ago
Lingxian Kong 9f5b474b6f Support to update function code
Change-Id: If7698350925119140b46cf319ad74f3e063ef0a6
Closes-Bug: #1733477
6 years ago
chenghuiyu 56616ac535 Remove unused variables ‘LOG'
Perhaps it will be used for debbuging, but now it's unused.

Change-Id: Ie7f5b8461f45a30455c2278b5d67cd392a4f49f8
6 years ago
Zuul bc238162fa Merge "Fix the job task failure" 6 years ago
Lingxian Kong 7154c9bd06 Fix the job task failure
Change-Id: Ief9eec6833c0379f9c04fdd4c5440d9984247f79
Closes-Bug: #1726594
6 years ago
junboli a7502caefd Minor fix for log message in rest_utils.py
TrivialFix


Change-Id: I0adbb40d2b779dea8fe4af3da9959f06aa311882
6 years ago
junboli a248d388d5 Fix the depreciate warning of get_transport
Use get_rpc_transport instead of get_transport, because
get_transport has been deprecate.

Change-Id: Ied4c0291fb06907aae5130940549ec7892eaaaf7
6 years ago
Lingxian Kong 601410e119 Support filtering for execution query
Change-Id: Iab965386f57948d4c333dfadde589c76ea639764
Implements: blueprint qinling-execution-filter
6 years ago
Lingxian Kong a7496f4e16 Create trust for functions
When user creates function, qinling will create trust for the function
that can be used when function is invoked. This feature is especially
useful when the function is invoked by a trustee user.

Remove the trust for job accordingly because the job will always use
trust for the function.

Change-Id: I68c608a1f25f1008e13bff33325e7cd9914653ae
6 years ago
Lingxian Kong d2dd717a2f Add session support for function
Also add the documentation.

Partially implements: blueprint qinling-support-user-session
Change-Id: Ic11b3b9f6ffe65bb08cfd880b81d41a40590919b
6 years ago
Lingxian Kong 4272c94a08 Add functional tests for execution
Partially implements: blueprint qinling-functional-tests
Change-Id: I448fd148c6d046412a459ec933ca1e88ddfa9cad
6 years ago
Jenkins ade2e5f8a7 Merge "Support policy check for api access" 6 years ago
Lingxian Kong cff88e1b64 Support policy check for api access
- Support oslo.policy in api layer
- Add tempest test for basic qinling operation

Change-Id: Icf12ee4df54652e81013e951b91163c085d961b7
6 years ago
liuyamin fccc7c2d35 [DOC BLD FIX] Correct reST field lists in docstrings
Most common format for documenting arguments is reST field, the
incorrect docstring will cause a Sphinx warning to be generated
and caused the documentation to be improperly rendered.

[1] http://sphinx-doc.org/domains.html#info-field-lists

Change-Id: I8860624c923ebdd3dc64ca52613eb79fd959da2b
6 years ago
Lingxian Kong 96a777f09f Improve runtime functional tests
Change-Id: I5f7fd65592097cffd55ebab07d9d00d261aa8a44
6 years ago
Lingxian Kong ac9d477112 Scale down function
This is admin only operation. The load monitoring of function execution
depends on the monitoring solution of underlying orchestrator.

Implements: blueprint qingling-autoscaling
Change-Id: I2eff62a45a718d230b51f18c5a49e0abea5f3164
6 years ago
Lingxian Kong 503bee80d6 Fix gate script for devstack-gate
Change-Id: I8913144c88aa4de76af5224dc0337d02e63fc39d
6 years ago
Jenkins 5dba23dc84 Merge "Add devstack plugin and scripts" 6 years ago
Lingxian Kong 3760a84a5c Add devstack plugin and scripts
Change-Id: I99c43b9033ac92284c03686a22cf9952f8bf20a1
6 years ago
Lingxian Kong f2189a0468 Fix logs query for executions
After function autoscaling support, it's impossible for qinling-engine
to get execution log because it doesn't know which pod it is talking
to.

So, it's neccessary for the runtime to return execution logs to
qinling engine.

The qinling client is not affected.

Change-Id: I96dfd00cc83d8b8a5e8c601ee3800b1ef1a45b1b
6 years ago
Lingxian Kong c74ac2faac Add scale up ability for function execution
Provide an API to do scale up operation, leave the container
monitoring to orchestrator underlay, although k8s already supports
HPA, but we should not rely on specific container orchestrator
capability for general purpose.

Partially implements: blueprint qingling-autoscaling
Change-Id: Iff1ff646a6df263b2770b8cebc74e80ab18c7613
6 years ago
Jenkins 55a06cfb77 Fix delete function with running job associated
User should not delete a function with running jobs associated.

Change-Id: I627ac7ef057768e7c2d99f56e17da9eacbf4ad76
6 years ago
Lingxian Kong b7562a4bd3 Add unit test for execution operations
Partially implements: blueprint add-unit-tests

Change-Id: Ib418364ec7164d0bad0402501f14d8340be99c89
6 years ago
Jenkins 22b4443858 Merge "Add unit test for function operations" 6 years ago
Lingxian Kong eb91b6d6f3 Add unit test for function operations
Partially implements: blueprint add-unit-tests
Change-Id: I00c3c7beca8cda7057f63850026c0f3f0526f5dd
6 years ago
shangxiaobj a3e95af382 [Trivialfix]Fix typos in qinling
Fix all the typos found in qinling.

Change-Id: Id3dbfd9b5280ad6ed1ca6d3400a9a00a925e186a
6 years ago
Lingxian Kong 089f41de2b Support function execution log query
After execution is finished, Qinling provides API to query execution
log for audit, debug or other purposes.

Implements: blueprint qinling-logs
Change-Id: Ie92619bf238c8e72e15c350c65bfbbb88a065f97
6 years ago
Lingxian Kong 35e652f43a Fix job creation with pattern
Change-Id: I0222b853464f6bf3589f11477a0624d72c64a22b
6 years ago
Lingxian Kong 55c7e5f496 Do not get openstack session in container
If the function is triggered by job, it's very hard to get openstack
serssion based on trust id, unless the qinling service credentials are
passed which will lead to security issue.

So, this patch will remove 'context' param for user function.

Change-Id: Ib0e185ad9729cf59e308b7ded683911a92624ad0
6 years ago
Lingxian Kong 1763f29ce6 Fix log message
Change-Id: If30b192539981003149311c25d776f9a73819beb
6 years ago
Lingxian Kong 13e2d13195 Fix getting update params
Change-Id: I0724417cfe974c23813117cf3733de89f8f5f1c5
6 years ago
Jenkins d768711939 Merge "Add job update api" 6 years ago
Lingxian Kong 16876649e5 Add job update api
By using update api, users could pause/cancel/resume jobs, or change
job execution pattern.

Change-Id: I05103316a91ad79c4d0593c4a02a400aac45a762
6 years ago
Lingxian Kong ff873c3f95 Fix authentication failure when token expires
OpenStack service session is generated when downloading code from
inside the container, but when the function is invoked at a deferred
time, the token may already be expiried.

Need to create the session when executing the function instead of
when downloading.

Change-Id: I8170bfb1c5f8b6cf88a744547db12cc12248aaca
6 years ago
Lingxian Kong a2213f3f6d Fix relationship of function service mapping
When deleting function, the service mapping record should be
deleted automatically.

Provide a helper script for resource clean up.

Change-Id: Ie3db454b5bcfb74a0826f01c4626fb45fca58478
6 years ago
Lingxian Kong 30c8e7dd60 Log improvement
Change-Id: Ia7e0dee3820179c22fb8281c62d0a8241b72b718
6 years ago
Lingxian Kong d23551c465 Fully support sync/async execution
Change-Id: I008025f710f8d61eb6f8a4998fe6990abdbc7203
6 years ago
Lingxian Kong 49c4c10adb Fix a bug in job periodic task
Change-Id: I4c26e08e3738023b1a7e7e8f0f2c1f59d7e80d79
6 years ago
Lingxian Kong 997c78131e Implement periodic task for jobs
Change-Id: I9093333cc9b1a64b5e386af87db80df5aa4dfb4b
6 years ago
Lingxian Kong 5baa3f1e45 Add jobs api
This is part-1 of jobs support in Qinling.

With job API, users can create a job with an existing function
that can be ran in cron job's fashion, but the job definition
has more capabilities than cron.

Change-Id: I0cb885dd6005ba024e3816bae9b79b0d3f35d335
6 years ago
Lingxian Kong 0f7520b88c Python runtime improvement
Change-Id: I8a77324df07c2d98773c20aa04eb81cc54152be3
6 years ago
wanghao 94b9d32ae6 Improve config options
Change-Id: I8f73a544333b02a4bc37736d850bf06cc09f8263
6 years ago
Lingxian Kong 00885b42ff Add UT for runtime API operation
Partially implements: blueprint add-unit-tests
Change-Id: I2bf92b36e4bee273171bd626638a61e313377a44
6 years ago
liusheng b460a55df7 Fix minor typos
Change-Id: I7a42166e24e27083ab34c7dcea87c28a56bbf7f7
6 years ago
Lingxian Kong eae0caf17d Add basic test for engine
Partially implements: blueprint add-unit-tests

Change-Id: I7fff946e8c05cf50722afdc54be746bde9240d43
6 years ago