Stage logs and artifacts

Solum needs to collect the output artifacts of each stage of assembly creation,
and associate those artifacts with the assemblies for reference and debugging
by users of Solum.

Change-Id: I64de1407dfa2f6d137adf4c502a166fa6255af59

specs/juno/stage-logs.rst

@@ -0,0 +1,203 @@
+..
+ This work is licensed under a Creative Commons Attribution 3.0 Unported
+ License.
+
+ http://creativecommons.org/licenses/by/3.0/legalcode
+
+===================================
+Storage and retrieval of stage logs
+===================================
+
+https://blueprints.launchpad.net/solum/+spec/stage-logs
+
+Solum needs to collect the logs of each stage of assembly creation, and
+associate those logs with the assemblies for reference and debugging by users
+of Solum.
+
+
+Problem description
+===================
+
+At present, users of Solum cannot see the log output from tasks such as
+build-app and unit-test. I identify three important steps to provide these
+logs to users:
+
+- Solum's worker agent must store the output of the tasks it performs.
+- Solum must associate these logs with the appropriate Assembly.
+- Solum must communicate this association to users in a simple way. A user must
+  be able to find the logs of each of the stages the user's assembly has been
+  through with no more information than the assembly id.
+
+I also identify several constraints on this problem:
+
+- Users must only have their own logs presented to them.
+- Storage and organization of logs must be as simple as possible to facilitate
+  consumption by a service such as Kibana to be run by deployers of Solum, and
+  also simple manual consumption by deployers not wishing to deploy and
+  maintain additional dependent services to run Solum.
+- Multiple worker agents may be running one or more hosts concurrently, so
+  individual agents must not inhibit the work of other agents.
+- An assembly will likely be rebuilt over the course of an application's
+  development; therefore a single assembly may bear more than one set of logs
+  from a particular stage. These multiple runs must be distinguished from each
+  other, and of special note is finding which of a set of logs is the most
+  recent. Once Mistral is used for workflow execution, we can associate these
+  logs by Execution, but until then, timestamps is the simplest solution.
+
+
+Proposed change
+===============
+
+These features are best achieved in incremental steps:
+
+- Store the logs of a stage on the host of the worker agent.
+
+  - Develop a naming scheme for ouput files for storage. I propose ::
+
+    /var/log/solum/worker/<assembly>/<stage>/<iso8601>.log
+
+    - For logging the output of a stage, maintaining metadata is important,
+      and a file path is not a strong way to associate it, especially with
+      multiple hosts simultaneously recording logs. It is then important that
+      the metadata remain in the body of the log. To that end, I propose
+      recording logs using the following format: ::
+
+        { "@timestamp": "<iso8601 timestamp>",
+          "assembly_id": "<assembly id>",
+          "solum_stage": "<'unittest', 'build-app', etc>",
+           ...
+          "message": "<captured output>"
+        }
+
+      This format is suggested as a simple way for this logging mechanism to
+      interface with log-consuming tools like Logstash or ElasticSearch, or
+      syslog.
+
+  - In addition, attaching a field to Assembly to convey the location of its
+    relevant logs would be helpful at this stage, using a text field to store
+    a dictionary shaped nominally like the following: ::
+
+      {
+        unittest: {
+          <date1>: <host>:<path/to/file>,
+          <date2>: <host>:<path/to/file>
+        },
+        build-app: {
+          <date1>: <host>:<path/to/file>
+       }
+     }
+
+- Collect and organize the logs from the hosts to a central location.
+
+  - This may be achieved most easily by storing the logs initially on network
+    storage, or emitting them direclty to a service like syslog.
+
+- Present the organized logs to users, ideally via additional API methods.
+
+  - Develop a URI scheme for presenting the logs to users. I propose ::
+
+      SOLUM/v2/assemblies/<assembly>/logs/<stage>/<iso8601>.log
+
+
+Alternatives
+------------
+
+- Swift can likely be used for storage, but Solum does not already make use of
+  the service. Initially, we will focus on local storage, and work in other
+  services such as Swift in later stages.
+
+Data model impact
+-----------------
+
+Minimal. Initially, an extra field on Assembly is proposed.
+
+REST API impact
+---------------
+
+None
+
+Security impact
+---------------
+
+Exposing the output of a docker container running user unit tests against
+user code is no more dangerous than running those tests against the code is
+originally.
+
+One concern of this format is making sure only the owner of an assembly can
+retrieve its logs.
+
+The concern of a user filling up storage is not a large one. Ideally, system
+monitor triggers ought to warn long before a user exhausts all available
+storage with a single docker container.
+
+Notifications impact
+--------------------
+
+None
+
+Other end user impact
+---------------------
+
+None
+
+Performance Impact
+------------------
+
+None
+
+Other deployer impact
+---------------------
+
+None
+
+Developer impact
+----------------
+
+This setup can be reused by future stages--
+functional testing, for example.
+
+Implementation
+==============
+
+Assignee(s)
+-----------
+
+Ed Cranford (ed--cranford) will implement this logging proposal.
+
+Work Items
+----------
+
+- Add configurable path and ensure the directory is created as worker starts.
+- Modify worker shell handler to capture and store the output of build and
+  unittest tasks.
+
+
+Dependencies
+============
+
+None
+
+
+Testing
+=======
+
+This is a practical change, and difficult to test automatically. This is
+further complicated by the differences between the development environment VM
+and an actual deployed Solum environment, which may be distributed across
+several distinct hosts and present accessibility problems we cannot foresee in
+development.
+
+
+Documentation Impact
+====================
+
+The documentation changes must cover minimally:
+- For operators, how to configure the logging location.
+- For users, how to retrieve the output of assembly stages.
+
+
+References
+==========
+
+None
+