openstack/kolla
Code Issues Proposed changes

Delete the unnecessary space

Browse Source 
TrivialFix

Change-Id: Ifca64d8c4d6e7654d25001022e806f082acfa0b8
This commit is contained in:
caoyuan 2016-10-15 04:07:29 +00:00
parent 57203424c2
commit a31a70d265
5 changed files with 118 additions and 118 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

30
specs/ansible-multi.rst
View File

@ -9,13 +9,13 @@ Multi-node Ansible
==================
This blueprint specifies an approach to automate the deployment of OpenStack
using Ansible and Docker best practices. The overriding principles used in
using Ansible and Docker best practices. The overriding principles used in
this specification are simplicity, flexibility and optimized deployment speed.
Problem description
===================
Kolla can be deployed multi-node currently. To do so, the environment
Kolla can be deployed multi-node currently. To do so, the environment
variables must be hand edited to define the hosts to connect to for various
services.
@ -42,10 +42,10 @@ Proposed change
===============
The docker-compose tool is single node and does nearly the same job as Ansible
would in this specification. As a result, we recommend deprecating
would in this specification. As a result, we recommend deprecating
docker-compose as the default deployment system for Kolla.
To replace it, we recommend Ansible as a technology choice. Ansible is easy
To replace it, we recommend Ansible as a technology choice. Ansible is easy
to learn, easy to use, and offers a base set of functionality to solve
deployment as outlined in our four use cases.
@ -53,36 +53,36 @@ We recommend three models of configuration.
The first model is based upon internally configuring the container and having
the container take responsibility for all container configuration including
database setup, database synchronization, and keystone registration. This
model uses docker-compose and docker as dependencies. Existing containers will
database setup, database synchronization, and keystone registration. This
model uses docker-compose and docker as dependencies. Existing containers will
be maintained but new container content will use either of the two remaining
models. James Slagle (TripleO PTL on behalf of our downstream TripleO
models. James Slagle (TripleO PTL on behalf of our downstream TripleO
community) was very clear that he would prefer to see this model stay available
and maintained. As TripleO enters the world of Big Tent, they don't intend to
and maintained. As TripleO enters the world of Big Tent, they don't intend to
deploy all of the services, and as such it doesn't make sense to maintain this
legacy operational mode for new container content except on demand of our
downstreams, hopefully with their assistance. This model is called
downstreams, hopefully with their assistance. This model is called
CONFIG_INSIDE.
The second model and third model configure the containers outside of the
container. These models depend on Ansible and Docker. In the future, the
container. These models depend on Ansible and Docker. In the future, the
OpenStack Puppet, OpenStack Chef and TripleO communities may decide to switch
to one of these two models in which case these communities may maintain tooling
to integrate with Kolla. The major difference between these two models is that
to integrate with Kolla. The major difference between these two models is that
one offers immutability and single source of truth (CONFIG_OUTSIDE_COPY_ONCE),
while the third model trades these two properties to allow an Operator to
directly modify configuration files on a system and have the configuration be
live in the container (CONFIG_OUTSIDE_COPY_ALWAYS). Because
live in the container (CONFIG_OUTSIDE_COPY_ALWAYS). Because
CONFIG_OUTSIDE_COPY_ALWAYS requires direct Operator intervention on a node, and
we prefer as a community Operators interact with the tools provided by Kolla,
CONFIG_OUTSIDE_COPY_ONCE will be the default.
We do not have to further enhance two sets of container configuration, but
instead can focus our development effort on the default Ansible configuration
methods. If a defect is found in one of the containers based upon the
methods. If a defect is found in one of the containers based upon the
CONFIG_INSIDE model, the community will repair it.
Finally we will implement a complete Ansible deployment system. The details
Finally we will implement a complete Ansible deployment system. The details
of the implementation are covered in a later section in this specification.
We estimate this will be approximately ~1000 LOC defining ~100 Ansible tasks.
We further estimate the total code base when complete will be under 6 KLOC.
@ -97,7 +97,7 @@ best practices while introducing completely customizable configuration.
The CONFIG_OUTSIDE_COPY_ALWAYS model of configuration offers the Operator
greater flexibility in managing their deployment, at greater risk of damaging
their deployment. It trades one set of best practices for another,
their deployment. It trades one set of best practices for another,
specifically the Kolla container best practices for flexibility.
Security impact

46
specs/containerize-openstack.rst
View File

@ -9,8 +9,8 @@ Containerize OpenStack
======================
When upgrading or downgrading OpenStack, it is possible to use package based
management or image-based management. Containerizing OpenStack is meant to
optimize image-based management of OpenStack. Containerizing OpenStack
management or image-based management. Containerizing OpenStack is meant to
optimize image-based management of OpenStack. Containerizing OpenStack
solves a manageability and availability problem with the current state of the
art deployment systems in OpenStack.
@ -20,34 +20,34 @@ Problem description
Current state of the art deployment systems use either image based or package
based upgrade.
Image based upgrades are utilized by TripleO. When TripleO updates a system,
Image based upgrades are utilized by TripleO. When TripleO updates a system,
it creates an image of the entire disk and deploys that rather than just the
parts that compose the OpenStack deployment. This results in significant
loss of availability. Further running VMs are shut down in the imaging
process. However, image based systems offer atomicity, because all related
parts that compose the OpenStack deployment. This results in significant
loss of availability. Further running VMs are shut down in the imaging
process. However, image based systems offer atomicity, because all related
software for a service is updated in one atomic action by reimaging the system.
Other systems use package based upgrade. Package based upgrades suffer from
a non-atomic nature. An update may update 1 or more RPM packages. The update
Other systems use package based upgrade. Package based upgrades suffer from
a non-atomic nature. An update may update 1 or more RPM packages. The update
process could fail for any number of reasons, and there is no way to back
out the existing changes. Typically in an OpenStack deployment it is
out the existing changes. Typically in an OpenStack deployment it is
desirable to update a service that does one thing including it's dependencies
as an atomic unit. Package based upgrades do not offer atomicity.
as an atomic unit. Package based upgrades do not offer atomicity.
To solve this problem, containers can be used to provide an image-based update
approach which offers atomic upgrade of a running system with minimal
interruption in service. A rough prototype of compute upgrade [1] shows
interruption in service. A rough prototype of compute upgrade [1] shows
approximately a 10 second window of unavailability during a software update.
The prototype keeps virtual machines running without interruption.
Use cases
---------
1. Upgrade or rollback OpenStack deployments atomically. End-user wants to
1. Upgrade or rollback OpenStack deployments atomically. End-user wants to
change the running software versions in her system to deploy a new upstream
release without interrupting service for significant periods.
2. Upgrade OpenStack based by component. End-user wants to upgrade her system
2. Upgrade OpenStack based by component. End-user wants to upgrade her system
in fine-grained chunks to limit damage from a failed upgrade.
3. Rollback OpenStack based by component. End-user experienced a failed
3. Rollback OpenStack based by component. End-user experienced a failed
upgrade and wishes to rollback to the last known good working version.
@ -180,16 +180,16 @@ The various container sets are composed in more detail as follows:
* swift-proxy-server
In order to achieve the desired results, we plan to permit super-privileged
containers. A super-privileged container is defined as any container launched
containers. A super-privileged container is defined as any container launched
with the --privileged=true flag to docker that:
* bind-mounts specific security-crucial host operating system directories
with -v. This includes nearly all directories in the filesystem except for
with -v. This includes nearly all directories in the filesystem except for
leaf directories with no other host operating system use.
* shares any namespace with the --ipc=host, --pid=host, or --net=host flags
We will not use the Docker EXPOSE operation since all containers will use
--net=host. One motive for using --net=host is it is inherently simpler.
--net=host. One motive for using --net=host is it is inherently simpler.
A different motive for not using EXPOSE is the 20 microsecond penalty
applied to every packet forwarded and returned by docker-proxy.
If EXPOSE functionality is desired, it can be added back by
@ -207,12 +207,12 @@ If the container does not pass its healthcheck operation, it should be
restarted.
Integration of metadata with fig or a similar single node Docker orchestration
tool will be implemented. Even though fig executes on a single node, the
tool will be implemented. Even though fig executes on a single node, the
containers will be designed to run multi-node and the deploy tool should take
some form of information to allow it to operate multi-node. The deploy tool
some form of information to allow it to operate multi-node. The deploy tool
should take a set of key/value pairs as inputs and convert them into inputs
into the environment passed to Docker. These key/value pairs could be a file
or environment variables. We will not offer integration with multi-node
into the environment passed to Docker. These key/value pairs could be a file
or environment variables. We will not offer integration with multi-node
scheduling or orchestration tools, but instead expect our consumers to manage
each bare metal machine using our fig or similar in nature tool integration.
@ -220,7 +220,7 @@ Any contributions from the community of the required metadata to run these
containers using a multi-node orchestration tool will be warmly received but
generally won't be maintained by the core team.
The technique for launching the deploy script is not handled by Kolla. This
The technique for launching the deploy script is not handled by Kolla. This
is a problem for a higher level deployment tool such as TripleO or Fuel to
tackle.
@ -229,7 +229,7 @@ Logs from the individual containers will be retrievable in some consistent way.
Security impact
---------------
Container usage with super-privileged mode may possibly impact security. For
Container usage with super-privileged mode may possibly impact security. For
example, when using --net=host mode and bind-mounting /run which is necessary
for a compute node, it is possible that a compute breakout could corrupt the
host operating system.

46
specs/kubernetes-deployment.rst
View File

@ -6,7 +6,7 @@ https://blueprints.launchpad.net/kolla/+spec/kolla-kubernetes
Kubernetes was evaluated by the Kolla team in the first two months of the
project and it was found to be problematic because it did not support net=host,
pid=host, and --privileged features in docker. Since then, it has developed
pid=host, and --privileged features in docker. Since then, it has developed
these features [1].
The objective is to manage the lifecycle of containerized OpenStack services by
@ -51,7 +51,7 @@ Orchestration
-------------
OpenStack on Kubernetes will be orchestrated by outside tools in order to create
a production ready OpenStack environment. The kolla-kubernetes repo is where
a production ready OpenStack environment. The kolla-kubernetes repo is where
any deployment tool can join the community and be a part of orchestrating a
kolla-kubernetes deployment.
@ -60,10 +60,10 @@ Service Config Management
Config generation will be completely decoupled from the deployment. The
containers only expect a config file to land in a specific directory in
the container in order to run. With this decoupled model, any tool could be
used to generate config files. The kolla-kubernetes community will evaluate
the container in order to run. With this decoupled model, any tool could be
used to generate config files. The kolla-kubernetes community will evaluate
any config generation tool, but will likely use Ansible for config generation
in order to reuse existing work from the community. This solution uses
in order to reuse existing work from the community. This solution uses
customized Ansible and jinja2 templates to generate the config. Also, there will
be a maintained set of defaults and a global yaml file that can override the
defaults.
@ -82,7 +82,7 @@ will be a Kubernetes Job, which will run the task until completion then
terminate the pods [7].
Each service will have a bootstrap task so that when the operator upgrades,
the bootstrap tasks are reused to upgrade the database. This will allow
the bootstrap tasks are reused to upgrade the database. This will allow
deployment and upgrades to follow the same pipeline.
The Kolla containers will communicate with the Kubernetes API server to in order
@ -96,14 +96,14 @@ require some orchestration and the bootstrap pod will need to be setup to
never restart or be replicated.
2) Use a sidecar container in the pod to handle the database sync with proper
health checking to make sure the services are coming up healthy. The big
health checking to make sure the services are coming up healthy. The big
difference between kolla's old docker-compose solution and Kubernetes, is that
docker-compose would only restart the containers. Kubernetes will completely
reschedule them. Which means, removing the pod and restarting it. The reason
docker-compose would only restart the containers. Kubernetes will completely
reschedule them. Which means, removing the pod and restarting it. The reason
this would fix that race condition failure kolla saw from docker-compose is
because glance would be rescheduled on failure allowing keystone to get a
chance to sync with the database and become active instead of constantly being
piled with glance requests. There can also be health checks around this to help
piled with glance requests. There can also be health checks around this to help
determine order.
If kolla-kubernetes used this sidecar approach, it would regain the use of
@ -116,12 +116,12 @@ Dependencies
- Docker >= 1.10.0
- Jinja2 >= 2.8.0
Kubernetes does not support dependencies between pods. The operator will launch
Kubernetes does not support dependencies between pods. The operator will launch
all the services and use kubernetes health checks to bring the deployment to an
operational state.
With orchestration around Kubernetes, the operator can determine what tasks are
run and when the tasks are run. This way, dependencies are handled at the
run and when the tasks are run. This way, dependencies are handled at the
orchestration level, but they are not required because proper health checking
will bring up the cluster in a healthy state.
@ -133,7 +133,7 @@ desired state for the pods and the deployment will move the cluster to the
desired state when a change is detected.
Kolla-kubernetes will provide Jobs that will provide the operator with the
flexibility needed to under go a step wise upgrade. In future releases,
flexibility needed to under go a step wise upgrade. In future releases,
kolla-kubernetes will look to Kubernetes to provide a means for operators to
plugin these jobs into a Deployment.
@ -141,22 +141,22 @@ Reconfigure
-----------
The operator generates a new config and loads it into the Kubernetes configmap
by changing the configmap version in the service yaml file. Then, the operator
by changing the configmap version in the service yaml file. Then, the operator
will trigger a rolling upgrade, which will scale down old pods and bring up new
ones that will run with the updated configuration files.
There's an open issue upstream in Kubernetes where the plan is to add support
around detecting if a pod has a changed in the configmap [6]. Depending on what
the solution is, kolla-kubernetes may or may not use it. The rolling
around detecting if a pod has a changed in the configmap [6]. Depending on what
the solution is, kolla-kubernetes may or may not use it. The rolling
upgrade feature will provide kolla-kubernetes with an elegant way to handle
restarting the services.
HA Architecture
---------------
Kubernetes uses health checks to bring up the services. Therefore,
Kubernetes uses health checks to bring up the services. Therefore,
kolla-kubernetes will use the same checks when monitoring if a service is
healthy. When a service fails, the replication controller will be responsible
healthy. When a service fails, the replication controller will be responsible
for bringing up a new container in its place [8][9].
However, Kubernetes does not cover all the HA corner cases, for instance,
@ -178,14 +178,14 @@ guarantee a pod will always be scheduled to a host, it makes node based
persistent storage unlikely, unless the community uses labels for every pod.
Persistent storage in kolla-kubernetes will come from volumes backed by
different storage offerings to provide persistent storage. Kolla-kubernetes
different storage offerings to provide persistent storage. Kolla-kubernetes
will provide a default solution using Ceph RBD, that the community will use to
deploy multinode with. From there, kolla-kubernetes can add any additional
persistent storage options as well as support options for the operator to
reference an existing storage solution.
To deploy Ceph, the community will use the Ansible playbooks from Kolla to
deploy a containerized Ceph at least for the 1.0 release. After Kubernetes
deploy a containerized Ceph at least for the 1.0 release. After Kubernetes
deployment matures, the community can evaluate building its own Ceph deployment
solution.
@ -198,9 +198,9 @@ Service Roles
At the broadest level, OpenStack can split up into two main roles, Controller
and Compute. With Kubernetes, the role definition layer changes.
Kolla-kubernetes will still need to define Compute nodes, but not Controller
nodes. Compute nodes hold the libvirt container and the running vms. That
nodes. Compute nodes hold the libvirt container and the running vms. That
service cannont migrate because the vms associated with it exist on the node.
However, the Controller role is more flexible. The Kubernetes layer provides IP
However, the Controller role is more flexible. The Kubernetes layer provides IP
persistence so that APIs will remain active and abstracted from the operator's
view [15]. kolla-kubernetes can direct Controller services away from the Compute
node using labels, while managing Compute services more strictly.
@ -244,7 +244,7 @@ To reuse Kolla's containers, kolla-kubernetes will use elastic search, heka, and
kibana as the default logging mechanism.
The community will implement centralized logging by using a 'side car' container
in the Kubernetes pod [17]. The logging service will trace the logs from the
in the Kubernetes pod [17]. The logging service will trace the logs from the
shared volume of the running serivce and send the data to elastic search. This
solution is ideal because volumes are shared amoung the containers in a pod.

110
specs/logging-with-heka.rst
View File

@ -4,15 +4,15 @@ Logging with Heka
https://blueprints.launchpad.net/kolla/+spec/heka
Kolla currently uses Rsyslog for logging. And Change Request ``252968`` [1]
Kolla currently uses Rsyslog for logging. And Change Request ``252968`` [1]
suggests to use ELK (Elasticsearch, Logstash, Kibana) as a way to index all the
logs, and visualize them.
This spec suggests using Heka [2] instead of Logstash, while still using
Elasticsearch for indexing and Kibana for visualization. It also discusses
Elasticsearch for indexing and Kibana for visualization. It also discusses
the removal of Rsyslog along the way.
What is Heka? Heka is a open-source stream processing software created and
What is Heka? Heka is a open-source stream processing software created and
maintained by Mozilla.
Using Heka will provide a lightweight and scalable log processing solution
@ -22,7 +22,7 @@ Problem description
===================
Change Request ``252968`` [1] adds an Ansible role named "elk" that enables
deploying ELK (Elasticsearch, Logstash, Kibana) on nodes with that role. This
deploying ELK (Elasticsearch, Logstash, Kibana) on nodes with that role. This
spec builds on that work, proposing a scalable log processing architecture
based on the Heka [2] stream processing software.
@ -34,7 +34,7 @@ OpenStack nodes rather than using a centralized log processing engine that
represents a bottleneck and a single-point-of-failure.
We also know from experience that Heka provides all the necessary flexibility
for processing other types of data streams than log messages. For example, we
for processing other types of data streams than log messages. For example, we
already use Heka together with Elasticsearch for logs, but also with collectd
and InfluxDB for statistics and metrics.
@ -53,16 +53,16 @@ in a dedicated container, referred to as the Heka container in the rest of this
document.
Each Heka instance reads and processes the logs local to the node it runs on,
and sends these logs to Elasticsearch for indexing. Elasticsearch may be
and sends these logs to Elasticsearch for indexing. Elasticsearch may be
distributed on multiple nodes for resiliency and scalability, but that part is
outside the scope of that specification.
Heka, written in Go, is fast and has a small footprint, making it possible to
run it on every node of the cluster. In contrast, Logstash runs in a JVM and
run it on every node of the cluster. In contrast, Logstash runs in a JVM and
is known [3] to be too heavy to run on every node.
Another important aspect is flow control and avoiding the loss of log messages
in case of overload. Heka’s filter and output plugins, and the Elasticsearch
in case of overload. Heka’s filter and output plugins, and the Elasticsearch
output plugin in particular, support the use of a disk based message queue.
This message queue allows plugins to reprocess messages from the queue when
downstream servers (Elasticsearch) are down or cannot keep up with the data
@ -74,20 +74,20 @@ which introduces some complexity and other points-of-failures.
Remove Rsyslog
--------------
Kolla currently uses Rsyslog. The Kolla services are configured to write their
logs to Syslog. Rsyslog gets the logs from the ``/var/lib/kolla/dev/log`` Unix
socket and dispatches them to log files on the local file system. Rsyslog
Kolla currently uses Rsyslog. The Kolla services are configured to write their
logs to Syslog. Rsyslog gets the logs from the ``/var/lib/kolla/dev/log`` Unix
socket and dispatches them to log files on the local file system. Rsyslog
running in a Docker container, the log files are stored in a Docker volume
(named ``rsyslog``).
With Rsyslog already running on each cluster node, the question of using two
log processing daemons, namely ``rsyslogd`` and ``hekad``, has been raised on
the mailing list. The spec evaluates the possibility of using ``hekad`` only,
log processing daemons, namely ``rsyslogd`` and ``hekad``, has been raised on
the mailing list. The spec evaluates the possibility of using ``hekad`` only,
based on some prototyping work we have conducted [4].
Note: Kolla doesn't currently collect logs from RabbitMQ, HAProxy and
Keepalived. For RabbitMQ the problem is related to RabbitMQ not having the
capability to write its logs to Syslog. HAProxy and Keepalived do have that
Keepalived. For RabbitMQ the problem is related to RabbitMQ not having the
capability to write its logs to Syslog. HAProxy and Keepalived do have that
capability, but the ``/var/lib/kolla/dev/log`` Unix socket file is currently
not mounted into the HAProxy and Keepalived containers.
@ -96,21 +96,21 @@ Use Heka's ``DockerLogInput`` plugin
To remove Rsyslog and only use Heka one option would be to make the Kolla
services write their logs to ``stdout`` (or ``stderr``) and rely on Heka's
``DockerLogInput`` plugin [5] for reading the logs. Our experiments have
``DockerLogInput`` plugin [5] for reading the logs. Our experiments have
revealed a number of problems with this option:
* The ``DockerLogInput`` plugin doesn't currently work for containers that have
a ``tty`` allocated. And Kolla currently allocates a tty for all containers
a ``tty`` allocated. And Kolla currently allocates a tty for all containers
(for good reasons).
* When ``DockerLogInput`` is used there is no way to differentiate log messages
for containers producing multiple log streams. ``neutron-agents`` is an
example of such a container. (Sam Yaple has raised that issue multiple
for containers producing multiple log streams. ``neutron-agents`` is an
example of such a container. (Sam Yaple has raised that issue multiple
times.)
* If Heka is stopped and restarted later then log messages will be lost, as the
``DockerLogInput`` plugin doesn't currently have a mechanism for tracking its
positions in the log streams. This is in contrast to the ``LogstreamerInput``
positions in the log streams. This is in contrast to the ``LogstreamerInput``
plugin [6] which does include that mechanism.
For these reasons we think that relying on the ``DockerLogInput`` plugin may
@ -119,7 +119,7 @@ not be a practical option.
For the note, our experiments have also shown that the OpenStack containers
logs written to ``stdout`` are visible to neither Heka nor ``docker logs``.
This problem is not reproducible when ``stderr`` is used rather than
``stdout``. The cause of this problem is currently unknown. And it looks like
``stdout``. The cause of this problem is currently unknown. And it looks like
other people have come across that issue [7].
Use local log files
@ -129,7 +129,7 @@ Another option consists of configuring all the Kolla services to log into local
files, and using Heka's ``LogstreamerInput`` plugin [5].
This option involves using a Docker named volume, mounted both into the service
containers (in ``rw`` mode) and into the Heka container (in ``ro`` mode). The
containers (in ``rw`` mode) and into the Heka container (in ``ro`` mode). The
services write logs into files placed in that volume, and Heka reads logs from
the files found in that volume.
@ -138,28 +138,28 @@ And it relies on Heka's ``LogstreamerInput`` plugin, which, based on our
experience, is efficient and robust.
Keeping file logs locally on the nodes has been established as a requirement by
the Kolla developers. With this option, and the Docker volume used, meeting
the Kolla developers. With this option, and the Docker volume used, meeting
that requirement necessitates no additional mechanism.
For this option to be applicable the services must have the capability of
logging into files. Most of the Kolla services have this capability. The
logging into files. Most of the Kolla services have this capability. The
exceptions are HAProxy and Keepalived, for which a different mechanism should
be used (described further down in the document). Note that this will make it
be used (described further down in the document). Note that this will make it
possible to collect logs from RabbitMQ, which does not support logging to
Syslog but does support logging to a file.
Also, this option requires that the services have the permission to create
files into the Docker volume, and that Heka has the permission to read these
files. This means that the Docker named volume will have to have appropriate
owner, group and permission bits. With the Heka container running under
files. This means that the Docker named volume will have to have appropriate
owner, group and permission bits. With the Heka container running under
a specific user (see below) this will mean using an ``extend_start.sh`` script
including ``sudo chown`` and possibly ``sudo chmod`` commands. Our prototype
including ``sudo chown`` and possibly ``sudo chmod`` commands. Our prototype
[4] already includes this.
As mentioned already the ``LogstreamerInput`` plugin includes a mechanism for
tracking positions in log streams. This works with journal files stored on the
file system (in ``/var/cache/hekad``). A specific volume, private to Heka,
will be used for these journal files. In this way no logs will be lost if the
tracking positions in log streams. This works with journal files stored on the
file system (in ``/var/cache/hekad``). A specific volume, private to Heka,
will be used for these journal files. In this way no logs will be lost if the
Heka container is removed and a new one is created.
Handling HAProxy and Keepalived
@ -174,7 +174,7 @@ This works by using Heka's ``UdpInput`` plugin with its ``net`` option set
to ``unixgram``.
This also requires that a Unix socket is created by Heka, and that socket is
mounted into the HAProxy and Keepalived containers. For that we will use the
mounted into the HAProxy and Keepalived containers. For that we will use the
same technique as the one currently used in Kolla with Rsyslog, that is
mounting ``/var/lib/kolla/dev`` into the Heka container and mounting
``/var/lib/kolla/dev/log`` into the service containers.
@ -182,7 +182,7 @@ mounting ``/var/lib/kolla/dev`` into the Heka container and mounting
Our prototype already includes some code demonstrating this. See [4].
Also, to be able to store a copy of the HAProxy and Keepalived logs locally on
the node, we will use Heka's ``FileOutput`` plugin. We will possibly create
the node, we will use Heka's ``FileOutput`` plugin. We will possibly create
two instances of that plugin, one for HAProxy and one for Keepalived, with
specific filters (``message_matcher``).
@ -190,29 +190,29 @@ Read Python Tracebacks
----------------------
In case of exceptions the OpenStack services log Python Tracebacks as multiple
log messages. If no special care is taken then the Python Tracebacks will be
log messages. If no special care is taken then the Python Tracebacks will be
indexed as separate documents in Elasticsearch, and displayed as distinct log
entries in Kibana, making them hard to read. To address that issue we will use
entries in Kibana, making them hard to read. To address that issue we will use
a custom Heka decoder, which will be responsible for coalescing the log lines
making up a Python Traceback into one message. Our prototype includes that
making up a Python Traceback into one message. Our prototype includes that
decoder [4].
Collect system logs
-------------------
In addition to container logs we think it is important to collect system logs
as well. For that we propose to mount the host's ``/var/log`` directory into
as well. For that we propose to mount the host's ``/var/log`` directory into
the Heka container, and configure Heka to get logs from standard log files
located in that directory (e.g. ``kern.log``, ``auth.log``, ``messages``). The
located in that directory (e.g. ``kern.log``, ``auth.log``, ``messages``). The
list of system log files will be determined at development time.
Log rotation
------------
Log rotation is an important aspect of the logging system. Currently Kolla
doesn't rotate logs. Logs just accumulate in the ``rsyslog`` Docker volume.
Log rotation is an important aspect of the logging system. Currently Kolla
doesn't rotate logs. Logs just accumulate in the ``rsyslog`` Docker volume.
The work on Heka proposed in this spec isn't directly related to log rotation,
but we are suggesting to address this issue for Mitaka. This will mean
but we are suggesting to address this issue for Mitaka. This will mean
creating a new container that uses ``logrotate`` to manage the log files
created by the Kolla containers.
@ -220,33 +220,33 @@ Create an ``heka`` user
-----------------------
For security reasons an ``heka`` user will be created in the Heka container and
the ``hekad`` daemon will run under that user. The ``heka`` user will be added
the ``hekad`` daemon will run under that user. The ``heka`` user will be added
to the ``kolla`` group, to make sure that Heka can read the log files created
by the services.
Security impact
---------------
Heka is a mature product maintained and used in production by Mozilla. So we
trust Heka as being secure. We also trust the Heka developers as being serious
Heka is a mature product maintained and used in production by Mozilla. So we
trust Heka as being secure. We also trust the Heka developers as being serious
should security vulnerabilities be found in the Heka code.
As described above we are proposing to use a Docker volume between the service
containers and the Heka container. The group of the volume directory and the
log files will be ``kolla``. And the owner of the log files will be the user
that executes the service producing logs. But the ``gid`` of the ``kolla``
containers and the Heka container. The group of the volume directory and the
log files will be ``kolla``. And the owner of the log files will be the user
that executes the service producing logs. But the ``gid`` of the ``kolla``
group and the ``uid``'s of the users executing the services may correspond
to a different group and different users on the host system. This means
that the permissions may not be right on the host system. This problem is
to a different group and different users on the host system. This means
that the permissions may not be right on the host system. This problem is
not specific to this specification, and it already exists in Kolla (for
the mariadb data volume for example).
Performance Impact
------------------
The ``hekad`` daemon will run in a container on each cluster node. But the
``rsyslogd`` will be removed. And we have assessed that Heka is lightweight
enough to run on every node. Also, a possible option would be to constrain the
The ``hekad`` daemon will run in a container on each cluster node. But the
``rsyslogd`` will be removed. And we have assessed that Heka is lightweight
enough to run on every node. Also, a possible option would be to constrain the
Heka container to only use a defined amount of resources.
Alternatives
@ -256,12 +256,12 @@ An alternative to this proposal involves using Logstash in a centralized
way as done in [1].
Another alternative would be to execute Logstash on each cluster node, as this
spec proposes with Heka. But this would mean running a JVM on each cluster
spec proposes with Heka. But this would mean running a JVM on each cluster
node, and using Redis as a centralized queue.
Also, as described above, we initially considered relying on services writing
their logs to ``stdout`` and use Heka's ``DockerLogInput`` plugin. But our
prototyping work has demonstrated the limits of that approach. See the
their logs to ``stdout`` and use Heka's ``DockerLogInput`` plugin. But our
prototyping work has demonstrated the limits of that approach. See the
``DockerLogInput`` section above for more information.
Implementation

4
specs/template.rst
View File

@ -8,8 +8,8 @@
This template should be in ReSTructured text. The filename in the git
repository should match the launchpad URL, for example a URL of
https://blueprints.launchpad.net/kolla/+spec/awesome-thing should be named
awesome-thing.rst . Please do not delete any of the sections in this
template. If you have nothing to say for a whole section, just write: None
awesome-thing.rst . Please do not delete any of the sections in this
template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, see http://www.tele3.cz/jbar/rest/rest.html