fix: Testing with multiple workers

This PS allows Deckhand to be able to run functional tests with
multiple workers. To achieve that, a document validation bug
was fixed: undeleted data schemas from all previous revisions are
considered. (The test schema-validation-success_add_invalid_document
was failing sporadically because of data race conditions with
only considering data schemas from the last revision with multiple
workers.)

The number of workers for running functional tests via uwsgi
has been increased to the number of CPU cores available on
the server to consistently validate concurrency.

Change-Id: I12589c2ed10495a1eb30757b6bacc5370503d0f4

deckhand/engine/document_validation.py

@@ -113,7 +113,7 @@ class DocumentValidation(object):
             been registered by external services via ``DataSchema`` documents.
             """
             data_schemas = db_api.document_get_all(
-                schema=types.DATA_SCHEMA_SCHEMA, revision_id='latest')
+                schema=types.DATA_SCHEMA_SCHEMA, deleted=False)
 
             for data_schema in data_schemas:
                 if cls.schema_re.match(data_schema['metadata']['name']):
@@ -202,7 +202,7 @@ class DocumentValidation(object):
             jsonschema.validate(raw_dict, base_schema.schema)
         except jsonschema.exceptions.ValidationError as e:
             LOG.debug('Document failed top-level schema validation. Details: '
-                      '%s.', e.message)
+                      '%s', e.message)
             # NOTE(fmontei): Raise here because if we fail basic schema
             # validation, then there is no point in continuing.
             raise errors.InvalidDocumentFormat(
@@ -228,7 +228,8 @@ class DocumentValidation(object):
         # ignored.
         if document.is_abstract():
             LOG.info('Skipping schema validation for abstract '
-                     'document: %s.', raw_dict)
+                     'document: [%s] %s.', document.get_schema(),
+                                           document.get_name())
         else:
 
             for schema_to_use in schemas_to_use:

tools/functional-tests.sh

@@ -26,9 +26,16 @@ function cleanup {
         sudo docker stop $DECKHAND_ID
     fi
     rm -rf $CONF_DIR
-    kill %1
+
+    if [ -z "$DECKHAND_IMAGE" ]; then
+        # Kill all processes and child processes (for example, if workers > 1)
+        # if using uwsgi only.
+        PGID=$(ps -o comm -o pgid | grep uwsgi | grep -o [0-9]* | head -n 1)
+        setsid kill -- -$PGID
+    fi
 }
 
+
 trap cleanup EXIT
 
 
@@ -179,11 +186,15 @@ log_section Starting Deckhand image
 
 if [ -z "$DECKHAND_IMAGE" ]; then
     echo "Running Deckhand via uwsgi"
+
+    # Set --workers 2, so that concurrency is always tested.
     uwsgi \
     --http :9000 \
     -w deckhand.cmd \
     --callable deckhand_callable \
     --enable-threads \
+    --workers 2 \
+    --threads 1 \
     -L \
     --pyargv "--config-file $CONF_DIR/deckhand.conf" &> $STDOUT &
 else