From cb3bf9262f17812b9beb9e77a94131211367852a Mon Sep 17 00:00:00 2001 From: Michal Gershenzon Date: Mon, 5 Jun 2017 14:43:35 +0000 Subject: [PATCH] Create and run workflows within a namespace Creating and running workflows within a namespace will allow users to create many workflows with the same name. This is useful when a user already has many workflows that are connected to each other implemented and one of the workflow names is already in use and the user does not want to edit the that workflow and all the ones referencing it or merge them to a workbook. blueprint create-and-run-workflows-within-a-namespace Change-Id: I5a8e6c8761bfe31bf04026083c4755468f580429 --- ...e-and-run-workflows-within-a-namespace.rst | 411 ++++++++++++++++++ 1 file changed, 411 insertions(+) create mode 100644 specs/pike/approved/create-and-run-workflows-within-a-namespace.rst diff --git a/specs/pike/approved/create-and-run-workflows-within-a-namespace.rst b/specs/pike/approved/create-and-run-workflows-within-a-namespace.rst new file mode 100644 index 0000000..f2b3bd6 --- /dev/null +++ b/specs/pike/approved/create-and-run-workflows-within-a-namespace.rst @@ -0,0 +1,411 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +=========================================== +Create and run workflows within a namespace +=========================================== + +Launchpad blueprint: + +https://blueprints.launchpad.net/mistral/+spec/create-and-run-workflows-within-a-namespace + +Creating and running workflows within a namespace will allow users to create +many workflows with the same name. This is useful when a user already has many +workflows that are connected to each other implemented and one of the workflow +names is already in use and the user does not want to edit that workflow and +all the ones referencing it or merge them to a workbook. This is possible +because the namespace is not a part of the Mistral language. + + +Problem description +=================== + +When a workflow name is already in use, it takes editing the workflow the user +wants to add to mistral or delete the existing one. + +Use Cases +--------- + +* When there are many users writing workflows in the same tenant simultaneously + each can use a different namespace, so there will be no clashes preventing + any of the users to add the workflow he created to mistral. + +* If a workflow definition is not allowed to be modified, but a workflow with + the same name already exists in Mistral and the user wants to upload the + workflow and create an execution from it. + + +Proposed change +=============== + +Add a new namespace parameter to both workflow definition creation, workflow +definition deletion, workflow execution creation and other relevant APIs. + +If a workflow definition creation request has a namespace specified in it, the +workflow name should be unique only within the namespace. If no namespace is +passed, the name should be unique within the group of workflow definitions +without a namespace (will be referred as the default namespace from here on). + +If a workflow execution request from the user, has a namespace specified in it, +mistral will search for the workflow definition within the namespace only. +However, if the workflow execution came from the mistral engine (e.g +sub-workflow execution), mistral will first try finding the workflow definition +within the namespace, and only if one does not exist, mistral will try finding +it the default namespace. By default the namespace passes to the sub-workflows +execution. If a workflow definition only exists in another namespace that is +not the default namespace, it will not be found, and the create workflow +execution request will fail. + +In the future we might want to add a namespace for actions and workbooks (but +for now workbooks and workflows created from them, are always in the default +namespace). + +An example to explain how the namespace moves recursively to sub-workflow +executions. + +Given there are 3 workflows in the workflow table. Two workflows definitions +called 'wf' and 'sub_sub_workflow'are in the same namespace - a namespace we +will call 'abc', and one workflow definition called 'sub_workflow' in the +default namespace. + +Visualization of a partial workflow definitions table: + + +----+---------------------+-----------+ + | ID | name | namespace | + +----+---------------------+-----------+ + | 1 | wf | abc | + +----+---------------------+-----------+ + | 2 | sub_wf | | + +----+---------------------+-----------+ + | 3 | sub_sub_wf | abc | + +----+---------------------+-----------+ + | 4 | sub_sub_wf | | + +----+---------------------+-----------+ + +Workflow definition for 'wf' with ID 1: + .. code-block:: yaml + + --- + version: '2.0' + wf: + tasks: + t1: + workflow: sub_wf + + +Workflow definition for 'sub_wf' with ID 2: + .. code-block:: yaml + + --- + version: '2.0' + sub_wf: + tasks: + t2: + workflow: sub_sub_wf + +Workflow definition for 'sub_sub_wf' with ID 3: + .. code-block:: yaml + + --- + version: '2.0' + sub_sub_wf: + tasks: + t3: + action: std.noop + +Workflow definition for 'sub_sub_wf' with ID 4: + .. code-block:: yaml + + --- + version: '2.0' + sub_sub_wf: + tasks: + should_not_run: + action: std.fail + +As you notice, namespace is not and should never be a part of the language. + +By calling the execution of workflow with name 'wf' within namespace 'abc', it +is required for workflow with name 'wf' in namespace 'abc' to run, and when +task t1 is executed to call workflow 'sub_wf' within the default namespace +(since no workflow with name 'sub_wf' exist within namespace 'abc'), but still +remember the namespace is 'abc' so that when task t2 will be executed, the +workflow that will be executed is workflow 'sub_sub_wf' in namespace 'abc' with +ID '3', rather than workflow 'sub_sub_wf' in the default namespace with ID '4'. +The execution described above should result in success. + +More strictly speaking, when it comes to calling nested workflows the namespace +of the top most workflow is propagated down to its children. So that when +Mistral needs to resolve a workflow name, it first searches the configured name +in that propagated namespace, and if it doesn't exist there, Mistral will try +to find it in the default namespace. + +A workflow execution can only trigger an execution of a workflow within both +the same tenant and namespace or within both the same tenant and the default +namespace. + +For workbooks that means that all workflows within the workbook could only call +workflows in the default namespace. + +Leading suggestion for the creation API of the 'wf' execution is this: + .. code-block:: + + POST /v2/executions + { + "workflow_name": "wf", + "workflow_namespace": "abc" + } + +Leading suggestion for passing the namespace from execution to sub-execution +recursively is putting it in the params of the execution possible under env. +A user is not allowed to add any key that starts with two underscores to the +env. + +Example of how such row might look like in the database: + .. code-block:: + + mysql> select * from workflow_executions_v2 where id='3'\G; + *************************** 1. row *************************** + created_at: 2017-06-19 10:59:29 + updated_at: 2017-06-19 10:59:30 + scope: private + project_id: 1 + id: 3 + name: sub_sub_wf + description: + workflow_name: sub_sub_wf + workflow_namespace: abc + workflow_id: 3 + spec: {"tasks": {"t3": {"action": "std.noop", "version": "2.0", "type": "direct", "name": "t3"}}, "name": "sub_sub_wf", "version": "2.0"} + state: SUCCESS + state_info: NULL + tags: NULL + runtime_context: {"index": 0} + accepted: 1 + input: {} + output: {} + params: {"env": {"__namespace": "abc"}} + +Notice the last line where under params->env we have a key called '__namespace' + +In the example described above, if a user decides to add a workflow with the +name 'sub_wf' to the 'abc' namespace, the next time the workflow will be +executed, the new workflow called 'sub_wf' from the 'abc' namespace will be +triggered by the workflow with the name 'wf' from the 'abc' namespace, instead +of the workflow 'sub_wf' from the default namespace. + +Regarding the results of the current APIs see the next examples that all assume +that the workflows described in the next table are the only one that exist. + +Table: + +----+---------------------+-----------+ + | ID | name | namespace | + +----+---------------------+-----------+ + | 1 | wf | abc | + +----+---------------------+-----------+ + | 2 | sub_wf | | + +----+---------------------+-----------+ + | 3 | sub_sub_wf | abc | + +----+---------------------+-----------+ + | 4 | sub_sub_wf | | + +----+---------------------+-----------+ + | 5 | example_wf | example_1 | + +----+---------------------+-----------+ + | 6 | example_wf | example_a | + +----+---------------------+-----------+ + +Examples: + + * **GET /v2/workflows** + Will return all 6 workflows + + * **GET /v2/workflows/wf** + Will return an error "workflow not found [workflow_identifier=wf] + + * **GET /v2/workflows/sub_wf** + Will return workflow 'sub_wf' from the default namespace (ID=2). + + * **GET /v2/workflows/sub_sub_wf** + Will return workflow 'sub_sub_wf' from the default namespace (ID=4). + + * **GET /v2/workflows/example_wf** + Will return an error "workflow not found [workflow_identifier=example_wf] + + * **DELETE /v2/workflows/wf** + Will throw an exception, because no namespace supplied and no such workflow + exist in the default namespace + + * **DELETE /v2/workflows/sub_wf** + Will delete the workflow with the name 'sub_wf' from the default namespace + (ID=2). + This should be allowed in order to let users that don't use namespaces to + work as they are used to. + + * **DELETE /v2/workflows/sub_sub_wf** + Will delete the workflow with the name 'sub_sub_wf' from the default + namespace (ID=4). + + * **DELETE /v2/workflows/example_wf** + Will return an error "workflow not found [workflow_identifier=example_wf] + + * PUT will have similar results to DELETE + + +Alternatives +------------ + +We can try and use workbooks, but the down side is it forces the user to merge +his workflows, and might result in a hugh file, that a user might find to be +hard to edit and read. + +For the described namespace design, we can use different names. For example in +the create execution API we can call the new key 'workflow_namespace' instead +of 'namespace'. Also the default namespace currently described is the empty +string (''), but it can be something like "". We should also +consider saving some namespaces to future system use (for example namespaces +that starts with 2 underscores '__') + + +Data model impact +----------------- + +The proposed change must come with a change to the data model. + +For workflow definition, a namespace should be added to the model and the DB +workflow_definitions_v2 table. And the same for workflow execution, plus it +should also be under env in params, so it will seep easily to the sub-workflow +executions. In the case of workflow execution there is also the option of just +adding it to the env under params. + +In the future we might create a namespace table. Migration from current +suggested model to one that includes a separate table for namespace, should be +easy using SQLAlchemy. + +REST API impact +--------------- + +Optional namespace parameter will be added to relevant requests: + + * create workflow definition within a namespace:: + + POST /v2/workflows?namespace=NAMESPACE + RAW_WF_DEFINITION + + * delete workflow definition within a namespace:: + + DELETE /v2/workflows/WORKFLOW_IDENTIFIER?namespace=NAMESPACE + + * get a workflow definition within a namespace:: + + GET /v2/workflows/WORKFLOW_IDENTIFIER?namespace=NAMESPACE + + * get all the workflow definitions within a given namespace:: + + GET /v2/workflows?namespace=NAMESPACE + + * update a workflow definition within a given namespace:: + + PUT /v2/workflows?namespace=NAMESPACE + RAW_WF_DEFINITION + + * create an execution of a workflow where the workflow belongs to given:: + + POST /v2/executions + { + "workflow_name": "WORKFLOW_NAME", + "workflow_namespace": "NAMESPACE" + } + + * get a list of all the namespaces:: + + GET /v2/namespaces + + +End user impact +--------------- + +The new namespace request parameter should be added to the python-mistralclient +as well. + +Performance Impact +------------------ + +None. + +Deployer impact +--------------- + +Database migration should be done when upgrading mistral to a version that +includes this change. + + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + michal-gershenzon + +Other contributors: + melisha + +Work Items +---------- + +* Adding namespace parameter to create workflow definition, delete workflow + definition and create workflow execution requests. + +* Change the way workflow definition are queried during execution. + +* Tests + +* Database migration script + +* Documentation + +* Add new parameter to python-mistralclient + + +Nice to have work items: + +* Adding namespace as a filter parameter of get workflow definition + +* Adding namespace as a filter parameter of update workflow definition + +* Adding namespace as a filter parameter of get workflow executions + +* Supporting the namespace feature with workbooks + +* Adding namespaces API endpoint + + +Dependencies +============ + +None. + + +Testing +======= + +* Create a workflow under some namespace that already exist in the default + namespace. + +* Create a workflow under the default namespace that calls the workflow above. + Run it once under the default namespace and once under the namespace from + previous section and see each time the expected sub-workflow execution is + created. + +* Create a workflow under some namespace that does not exist in the default + namespace and see trying to execute it without specifying a namespace fails. + + +References +========== + +None.