Delete the unnecessary space
TrivialFix Change-Id: Ifca64d8c4d6e7654d25001022e806f082acfa0b8
This commit is contained in:
parent
57203424c2
commit
a31a70d265
@ -9,13 +9,13 @@ Multi-node Ansible
|
|||||||
==================
|
==================
|
||||||
|
|
||||||
This blueprint specifies an approach to automate the deployment of OpenStack
|
This blueprint specifies an approach to automate the deployment of OpenStack
|
||||||
using Ansible and Docker best practices. The overriding principles used in
|
using Ansible and Docker best practices. The overriding principles used in
|
||||||
this specification are simplicity, flexibility and optimized deployment speed.
|
this specification are simplicity, flexibility and optimized deployment speed.
|
||||||
|
|
||||||
Problem description
|
Problem description
|
||||||
===================
|
===================
|
||||||
|
|
||||||
Kolla can be deployed multi-node currently. To do so, the environment
|
Kolla can be deployed multi-node currently. To do so, the environment
|
||||||
variables must be hand edited to define the hosts to connect to for various
|
variables must be hand edited to define the hosts to connect to for various
|
||||||
services.
|
services.
|
||||||
|
|
||||||
@ -42,10 +42,10 @@ Proposed change
|
|||||||
===============
|
===============
|
||||||
|
|
||||||
The docker-compose tool is single node and does nearly the same job as Ansible
|
The docker-compose tool is single node and does nearly the same job as Ansible
|
||||||
would in this specification. As a result, we recommend deprecating
|
would in this specification. As a result, we recommend deprecating
|
||||||
docker-compose as the default deployment system for Kolla.
|
docker-compose as the default deployment system for Kolla.
|
||||||
|
|
||||||
To replace it, we recommend Ansible as a technology choice. Ansible is easy
|
To replace it, we recommend Ansible as a technology choice. Ansible is easy
|
||||||
to learn, easy to use, and offers a base set of functionality to solve
|
to learn, easy to use, and offers a base set of functionality to solve
|
||||||
deployment as outlined in our four use cases.
|
deployment as outlined in our four use cases.
|
||||||
|
|
||||||
@ -53,36 +53,36 @@ We recommend three models of configuration.
|
|||||||
|
|
||||||
The first model is based upon internally configuring the container and having
|
The first model is based upon internally configuring the container and having
|
||||||
the container take responsibility for all container configuration including
|
the container take responsibility for all container configuration including
|
||||||
database setup, database synchronization, and keystone registration. This
|
database setup, database synchronization, and keystone registration. This
|
||||||
model uses docker-compose and docker as dependencies. Existing containers will
|
model uses docker-compose and docker as dependencies. Existing containers will
|
||||||
be maintained but new container content will use either of the two remaining
|
be maintained but new container content will use either of the two remaining
|
||||||
models. James Slagle (TripleO PTL on behalf of our downstream TripleO
|
models. James Slagle (TripleO PTL on behalf of our downstream TripleO
|
||||||
community) was very clear that he would prefer to see this model stay available
|
community) was very clear that he would prefer to see this model stay available
|
||||||
and maintained. As TripleO enters the world of Big Tent, they don't intend to
|
and maintained. As TripleO enters the world of Big Tent, they don't intend to
|
||||||
deploy all of the services, and as such it doesn't make sense to maintain this
|
deploy all of the services, and as such it doesn't make sense to maintain this
|
||||||
legacy operational mode for new container content except on demand of our
|
legacy operational mode for new container content except on demand of our
|
||||||
downstreams, hopefully with their assistance. This model is called
|
downstreams, hopefully with their assistance. This model is called
|
||||||
CONFIG_INSIDE.
|
CONFIG_INSIDE.
|
||||||
|
|
||||||
The second model and third model configure the containers outside of the
|
The second model and third model configure the containers outside of the
|
||||||
container. These models depend on Ansible and Docker. In the future, the
|
container. These models depend on Ansible and Docker. In the future, the
|
||||||
OpenStack Puppet, OpenStack Chef and TripleO communities may decide to switch
|
OpenStack Puppet, OpenStack Chef and TripleO communities may decide to switch
|
||||||
to one of these two models in which case these communities may maintain tooling
|
to one of these two models in which case these communities may maintain tooling
|
||||||
to integrate with Kolla. The major difference between these two models is that
|
to integrate with Kolla. The major difference between these two models is that
|
||||||
one offers immutability and single source of truth (CONFIG_OUTSIDE_COPY_ONCE),
|
one offers immutability and single source of truth (CONFIG_OUTSIDE_COPY_ONCE),
|
||||||
while the third model trades these two properties to allow an Operator to
|
while the third model trades these two properties to allow an Operator to
|
||||||
directly modify configuration files on a system and have the configuration be
|
directly modify configuration files on a system and have the configuration be
|
||||||
live in the container (CONFIG_OUTSIDE_COPY_ALWAYS). Because
|
live in the container (CONFIG_OUTSIDE_COPY_ALWAYS). Because
|
||||||
CONFIG_OUTSIDE_COPY_ALWAYS requires direct Operator intervention on a node, and
|
CONFIG_OUTSIDE_COPY_ALWAYS requires direct Operator intervention on a node, and
|
||||||
we prefer as a community Operators interact with the tools provided by Kolla,
|
we prefer as a community Operators interact with the tools provided by Kolla,
|
||||||
CONFIG_OUTSIDE_COPY_ONCE will be the default.
|
CONFIG_OUTSIDE_COPY_ONCE will be the default.
|
||||||
|
|
||||||
We do not have to further enhance two sets of container configuration, but
|
We do not have to further enhance two sets of container configuration, but
|
||||||
instead can focus our development effort on the default Ansible configuration
|
instead can focus our development effort on the default Ansible configuration
|
||||||
methods. If a defect is found in one of the containers based upon the
|
methods. If a defect is found in one of the containers based upon the
|
||||||
CONFIG_INSIDE model, the community will repair it.
|
CONFIG_INSIDE model, the community will repair it.
|
||||||
|
|
||||||
Finally we will implement a complete Ansible deployment system. The details
|
Finally we will implement a complete Ansible deployment system. The details
|
||||||
of the implementation are covered in a later section in this specification.
|
of the implementation are covered in a later section in this specification.
|
||||||
We estimate this will be approximately ~1000 LOC defining ~100 Ansible tasks.
|
We estimate this will be approximately ~1000 LOC defining ~100 Ansible tasks.
|
||||||
We further estimate the total code base when complete will be under 6 KLOC.
|
We further estimate the total code base when complete will be under 6 KLOC.
|
||||||
@ -97,7 +97,7 @@ best practices while introducing completely customizable configuration.
|
|||||||
|
|
||||||
The CONFIG_OUTSIDE_COPY_ALWAYS model of configuration offers the Operator
|
The CONFIG_OUTSIDE_COPY_ALWAYS model of configuration offers the Operator
|
||||||
greater flexibility in managing their deployment, at greater risk of damaging
|
greater flexibility in managing their deployment, at greater risk of damaging
|
||||||
their deployment. It trades one set of best practices for another,
|
their deployment. It trades one set of best practices for another,
|
||||||
specifically the Kolla container best practices for flexibility.
|
specifically the Kolla container best practices for flexibility.
|
||||||
|
|
||||||
Security impact
|
Security impact
|
||||||
|
@ -9,8 +9,8 @@ Containerize OpenStack
|
|||||||
======================
|
======================
|
||||||
|
|
||||||
When upgrading or downgrading OpenStack, it is possible to use package based
|
When upgrading or downgrading OpenStack, it is possible to use package based
|
||||||
management or image-based management. Containerizing OpenStack is meant to
|
management or image-based management. Containerizing OpenStack is meant to
|
||||||
optimize image-based management of OpenStack. Containerizing OpenStack
|
optimize image-based management of OpenStack. Containerizing OpenStack
|
||||||
solves a manageability and availability problem with the current state of the
|
solves a manageability and availability problem with the current state of the
|
||||||
art deployment systems in OpenStack.
|
art deployment systems in OpenStack.
|
||||||
|
|
||||||
@ -20,34 +20,34 @@ Problem description
|
|||||||
Current state of the art deployment systems use either image based or package
|
Current state of the art deployment systems use either image based or package
|
||||||
based upgrade.
|
based upgrade.
|
||||||
|
|
||||||
Image based upgrades are utilized by TripleO. When TripleO updates a system,
|
Image based upgrades are utilized by TripleO. When TripleO updates a system,
|
||||||
it creates an image of the entire disk and deploys that rather than just the
|
it creates an image of the entire disk and deploys that rather than just the
|
||||||
parts that compose the OpenStack deployment. This results in significant
|
parts that compose the OpenStack deployment. This results in significant
|
||||||
loss of availability. Further running VMs are shut down in the imaging
|
loss of availability. Further running VMs are shut down in the imaging
|
||||||
process. However, image based systems offer atomicity, because all related
|
process. However, image based systems offer atomicity, because all related
|
||||||
software for a service is updated in one atomic action by reimaging the system.
|
software for a service is updated in one atomic action by reimaging the system.
|
||||||
|
|
||||||
Other systems use package based upgrade. Package based upgrades suffer from
|
Other systems use package based upgrade. Package based upgrades suffer from
|
||||||
a non-atomic nature. An update may update 1 or more RPM packages. The update
|
a non-atomic nature. An update may update 1 or more RPM packages. The update
|
||||||
process could fail for any number of reasons, and there is no way to back
|
process could fail for any number of reasons, and there is no way to back
|
||||||
out the existing changes. Typically in an OpenStack deployment it is
|
out the existing changes. Typically in an OpenStack deployment it is
|
||||||
desirable to update a service that does one thing including it's dependencies
|
desirable to update a service that does one thing including it's dependencies
|
||||||
as an atomic unit. Package based upgrades do not offer atomicity.
|
as an atomic unit. Package based upgrades do not offer atomicity.
|
||||||
|
|
||||||
To solve this problem, containers can be used to provide an image-based update
|
To solve this problem, containers can be used to provide an image-based update
|
||||||
approach which offers atomic upgrade of a running system with minimal
|
approach which offers atomic upgrade of a running system with minimal
|
||||||
interruption in service. A rough prototype of compute upgrade [1] shows
|
interruption in service. A rough prototype of compute upgrade [1] shows
|
||||||
approximately a 10 second window of unavailability during a software update.
|
approximately a 10 second window of unavailability during a software update.
|
||||||
The prototype keeps virtual machines running without interruption.
|
The prototype keeps virtual machines running without interruption.
|
||||||
|
|
||||||
Use cases
|
Use cases
|
||||||
---------
|
---------
|
||||||
1. Upgrade or rollback OpenStack deployments atomically. End-user wants to
|
1. Upgrade or rollback OpenStack deployments atomically. End-user wants to
|
||||||
change the running software versions in her system to deploy a new upstream
|
change the running software versions in her system to deploy a new upstream
|
||||||
release without interrupting service for significant periods.
|
release without interrupting service for significant periods.
|
||||||
2. Upgrade OpenStack based by component. End-user wants to upgrade her system
|
2. Upgrade OpenStack based by component. End-user wants to upgrade her system
|
||||||
in fine-grained chunks to limit damage from a failed upgrade.
|
in fine-grained chunks to limit damage from a failed upgrade.
|
||||||
3. Rollback OpenStack based by component. End-user experienced a failed
|
3. Rollback OpenStack based by component. End-user experienced a failed
|
||||||
upgrade and wishes to rollback to the last known good working version.
|
upgrade and wishes to rollback to the last known good working version.
|
||||||
|
|
||||||
|
|
||||||
@ -180,16 +180,16 @@ The various container sets are composed in more detail as follows:
|
|||||||
* swift-proxy-server
|
* swift-proxy-server
|
||||||
|
|
||||||
In order to achieve the desired results, we plan to permit super-privileged
|
In order to achieve the desired results, we plan to permit super-privileged
|
||||||
containers. A super-privileged container is defined as any container launched
|
containers. A super-privileged container is defined as any container launched
|
||||||
with the --privileged=true flag to docker that:
|
with the --privileged=true flag to docker that:
|
||||||
|
|
||||||
* bind-mounts specific security-crucial host operating system directories
|
* bind-mounts specific security-crucial host operating system directories
|
||||||
with -v. This includes nearly all directories in the filesystem except for
|
with -v. This includes nearly all directories in the filesystem except for
|
||||||
leaf directories with no other host operating system use.
|
leaf directories with no other host operating system use.
|
||||||
* shares any namespace with the --ipc=host, --pid=host, or --net=host flags
|
* shares any namespace with the --ipc=host, --pid=host, or --net=host flags
|
||||||
|
|
||||||
We will not use the Docker EXPOSE operation since all containers will use
|
We will not use the Docker EXPOSE operation since all containers will use
|
||||||
--net=host. One motive for using --net=host is it is inherently simpler.
|
--net=host. One motive for using --net=host is it is inherently simpler.
|
||||||
A different motive for not using EXPOSE is the 20 microsecond penalty
|
A different motive for not using EXPOSE is the 20 microsecond penalty
|
||||||
applied to every packet forwarded and returned by docker-proxy.
|
applied to every packet forwarded and returned by docker-proxy.
|
||||||
If EXPOSE functionality is desired, it can be added back by
|
If EXPOSE functionality is desired, it can be added back by
|
||||||
@ -207,12 +207,12 @@ If the container does not pass its healthcheck operation, it should be
|
|||||||
restarted.
|
restarted.
|
||||||
|
|
||||||
Integration of metadata with fig or a similar single node Docker orchestration
|
Integration of metadata with fig or a similar single node Docker orchestration
|
||||||
tool will be implemented. Even though fig executes on a single node, the
|
tool will be implemented. Even though fig executes on a single node, the
|
||||||
containers will be designed to run multi-node and the deploy tool should take
|
containers will be designed to run multi-node and the deploy tool should take
|
||||||
some form of information to allow it to operate multi-node. The deploy tool
|
some form of information to allow it to operate multi-node. The deploy tool
|
||||||
should take a set of key/value pairs as inputs and convert them into inputs
|
should take a set of key/value pairs as inputs and convert them into inputs
|
||||||
into the environment passed to Docker. These key/value pairs could be a file
|
into the environment passed to Docker. These key/value pairs could be a file
|
||||||
or environment variables. We will not offer integration with multi-node
|
or environment variables. We will not offer integration with multi-node
|
||||||
scheduling or orchestration tools, but instead expect our consumers to manage
|
scheduling or orchestration tools, but instead expect our consumers to manage
|
||||||
each bare metal machine using our fig or similar in nature tool integration.
|
each bare metal machine using our fig or similar in nature tool integration.
|
||||||
|
|
||||||
@ -220,7 +220,7 @@ Any contributions from the community of the required metadata to run these
|
|||||||
containers using a multi-node orchestration tool will be warmly received but
|
containers using a multi-node orchestration tool will be warmly received but
|
||||||
generally won't be maintained by the core team.
|
generally won't be maintained by the core team.
|
||||||
|
|
||||||
The technique for launching the deploy script is not handled by Kolla. This
|
The technique for launching the deploy script is not handled by Kolla. This
|
||||||
is a problem for a higher level deployment tool such as TripleO or Fuel to
|
is a problem for a higher level deployment tool such as TripleO or Fuel to
|
||||||
tackle.
|
tackle.
|
||||||
|
|
||||||
@ -229,7 +229,7 @@ Logs from the individual containers will be retrievable in some consistent way.
|
|||||||
Security impact
|
Security impact
|
||||||
---------------
|
---------------
|
||||||
|
|
||||||
Container usage with super-privileged mode may possibly impact security. For
|
Container usage with super-privileged mode may possibly impact security. For
|
||||||
example, when using --net=host mode and bind-mounting /run which is necessary
|
example, when using --net=host mode and bind-mounting /run which is necessary
|
||||||
for a compute node, it is possible that a compute breakout could corrupt the
|
for a compute node, it is possible that a compute breakout could corrupt the
|
||||||
host operating system.
|
host operating system.
|
||||||
|
@ -6,7 +6,7 @@ https://blueprints.launchpad.net/kolla/+spec/kolla-kubernetes
|
|||||||
|
|
||||||
Kubernetes was evaluated by the Kolla team in the first two months of the
|
Kubernetes was evaluated by the Kolla team in the first two months of the
|
||||||
project and it was found to be problematic because it did not support net=host,
|
project and it was found to be problematic because it did not support net=host,
|
||||||
pid=host, and --privileged features in docker. Since then, it has developed
|
pid=host, and --privileged features in docker. Since then, it has developed
|
||||||
these features [1].
|
these features [1].
|
||||||
|
|
||||||
The objective is to manage the lifecycle of containerized OpenStack services by
|
The objective is to manage the lifecycle of containerized OpenStack services by
|
||||||
@ -51,7 +51,7 @@ Orchestration
|
|||||||
-------------
|
-------------
|
||||||
|
|
||||||
OpenStack on Kubernetes will be orchestrated by outside tools in order to create
|
OpenStack on Kubernetes will be orchestrated by outside tools in order to create
|
||||||
a production ready OpenStack environment. The kolla-kubernetes repo is where
|
a production ready OpenStack environment. The kolla-kubernetes repo is where
|
||||||
any deployment tool can join the community and be a part of orchestrating a
|
any deployment tool can join the community and be a part of orchestrating a
|
||||||
kolla-kubernetes deployment.
|
kolla-kubernetes deployment.
|
||||||
|
|
||||||
@ -60,10 +60,10 @@ Service Config Management
|
|||||||
|
|
||||||
Config generation will be completely decoupled from the deployment. The
|
Config generation will be completely decoupled from the deployment. The
|
||||||
containers only expect a config file to land in a specific directory in
|
containers only expect a config file to land in a specific directory in
|
||||||
the container in order to run. With this decoupled model, any tool could be
|
the container in order to run. With this decoupled model, any tool could be
|
||||||
used to generate config files. The kolla-kubernetes community will evaluate
|
used to generate config files. The kolla-kubernetes community will evaluate
|
||||||
any config generation tool, but will likely use Ansible for config generation
|
any config generation tool, but will likely use Ansible for config generation
|
||||||
in order to reuse existing work from the community. This solution uses
|
in order to reuse existing work from the community. This solution uses
|
||||||
customized Ansible and jinja2 templates to generate the config. Also, there will
|
customized Ansible and jinja2 templates to generate the config. Also, there will
|
||||||
be a maintained set of defaults and a global yaml file that can override the
|
be a maintained set of defaults and a global yaml file that can override the
|
||||||
defaults.
|
defaults.
|
||||||
@ -82,7 +82,7 @@ will be a Kubernetes Job, which will run the task until completion then
|
|||||||
terminate the pods [7].
|
terminate the pods [7].
|
||||||
|
|
||||||
Each service will have a bootstrap task so that when the operator upgrades,
|
Each service will have a bootstrap task so that when the operator upgrades,
|
||||||
the bootstrap tasks are reused to upgrade the database. This will allow
|
the bootstrap tasks are reused to upgrade the database. This will allow
|
||||||
deployment and upgrades to follow the same pipeline.
|
deployment and upgrades to follow the same pipeline.
|
||||||
|
|
||||||
The Kolla containers will communicate with the Kubernetes API server to in order
|
The Kolla containers will communicate with the Kubernetes API server to in order
|
||||||
@ -96,14 +96,14 @@ require some orchestration and the bootstrap pod will need to be setup to
|
|||||||
never restart or be replicated.
|
never restart or be replicated.
|
||||||
|
|
||||||
2) Use a sidecar container in the pod to handle the database sync with proper
|
2) Use a sidecar container in the pod to handle the database sync with proper
|
||||||
health checking to make sure the services are coming up healthy. The big
|
health checking to make sure the services are coming up healthy. The big
|
||||||
difference between kolla's old docker-compose solution and Kubernetes, is that
|
difference between kolla's old docker-compose solution and Kubernetes, is that
|
||||||
docker-compose would only restart the containers. Kubernetes will completely
|
docker-compose would only restart the containers. Kubernetes will completely
|
||||||
reschedule them. Which means, removing the pod and restarting it. The reason
|
reschedule them. Which means, removing the pod and restarting it. The reason
|
||||||
this would fix that race condition failure kolla saw from docker-compose is
|
this would fix that race condition failure kolla saw from docker-compose is
|
||||||
because glance would be rescheduled on failure allowing keystone to get a
|
because glance would be rescheduled on failure allowing keystone to get a
|
||||||
chance to sync with the database and become active instead of constantly being
|
chance to sync with the database and become active instead of constantly being
|
||||||
piled with glance requests. There can also be health checks around this to help
|
piled with glance requests. There can also be health checks around this to help
|
||||||
determine order.
|
determine order.
|
||||||
|
|
||||||
If kolla-kubernetes used this sidecar approach, it would regain the use of
|
If kolla-kubernetes used this sidecar approach, it would regain the use of
|
||||||
@ -116,12 +116,12 @@ Dependencies
|
|||||||
- Docker >= 1.10.0
|
- Docker >= 1.10.0
|
||||||
- Jinja2 >= 2.8.0
|
- Jinja2 >= 2.8.0
|
||||||
|
|
||||||
Kubernetes does not support dependencies between pods. The operator will launch
|
Kubernetes does not support dependencies between pods. The operator will launch
|
||||||
all the services and use kubernetes health checks to bring the deployment to an
|
all the services and use kubernetes health checks to bring the deployment to an
|
||||||
operational state.
|
operational state.
|
||||||
|
|
||||||
With orchestration around Kubernetes, the operator can determine what tasks are
|
With orchestration around Kubernetes, the operator can determine what tasks are
|
||||||
run and when the tasks are run. This way, dependencies are handled at the
|
run and when the tasks are run. This way, dependencies are handled at the
|
||||||
orchestration level, but they are not required because proper health checking
|
orchestration level, but they are not required because proper health checking
|
||||||
will bring up the cluster in a healthy state.
|
will bring up the cluster in a healthy state.
|
||||||
|
|
||||||
@ -133,7 +133,7 @@ desired state for the pods and the deployment will move the cluster to the
|
|||||||
desired state when a change is detected.
|
desired state when a change is detected.
|
||||||
|
|
||||||
Kolla-kubernetes will provide Jobs that will provide the operator with the
|
Kolla-kubernetes will provide Jobs that will provide the operator with the
|
||||||
flexibility needed to under go a step wise upgrade. In future releases,
|
flexibility needed to under go a step wise upgrade. In future releases,
|
||||||
kolla-kubernetes will look to Kubernetes to provide a means for operators to
|
kolla-kubernetes will look to Kubernetes to provide a means for operators to
|
||||||
plugin these jobs into a Deployment.
|
plugin these jobs into a Deployment.
|
||||||
|
|
||||||
@ -141,22 +141,22 @@ Reconfigure
|
|||||||
-----------
|
-----------
|
||||||
|
|
||||||
The operator generates a new config and loads it into the Kubernetes configmap
|
The operator generates a new config and loads it into the Kubernetes configmap
|
||||||
by changing the configmap version in the service yaml file. Then, the operator
|
by changing the configmap version in the service yaml file. Then, the operator
|
||||||
will trigger a rolling upgrade, which will scale down old pods and bring up new
|
will trigger a rolling upgrade, which will scale down old pods and bring up new
|
||||||
ones that will run with the updated configuration files.
|
ones that will run with the updated configuration files.
|
||||||
|
|
||||||
There's an open issue upstream in Kubernetes where the plan is to add support
|
There's an open issue upstream in Kubernetes where the plan is to add support
|
||||||
around detecting if a pod has a changed in the configmap [6]. Depending on what
|
around detecting if a pod has a changed in the configmap [6]. Depending on what
|
||||||
the solution is, kolla-kubernetes may or may not use it. The rolling
|
the solution is, kolla-kubernetes may or may not use it. The rolling
|
||||||
upgrade feature will provide kolla-kubernetes with an elegant way to handle
|
upgrade feature will provide kolla-kubernetes with an elegant way to handle
|
||||||
restarting the services.
|
restarting the services.
|
||||||
|
|
||||||
HA Architecture
|
HA Architecture
|
||||||
---------------
|
---------------
|
||||||
|
|
||||||
Kubernetes uses health checks to bring up the services. Therefore,
|
Kubernetes uses health checks to bring up the services. Therefore,
|
||||||
kolla-kubernetes will use the same checks when monitoring if a service is
|
kolla-kubernetes will use the same checks when monitoring if a service is
|
||||||
healthy. When a service fails, the replication controller will be responsible
|
healthy. When a service fails, the replication controller will be responsible
|
||||||
for bringing up a new container in its place [8][9].
|
for bringing up a new container in its place [8][9].
|
||||||
|
|
||||||
However, Kubernetes does not cover all the HA corner cases, for instance,
|
However, Kubernetes does not cover all the HA corner cases, for instance,
|
||||||
@ -178,14 +178,14 @@ guarantee a pod will always be scheduled to a host, it makes node based
|
|||||||
persistent storage unlikely, unless the community uses labels for every pod.
|
persistent storage unlikely, unless the community uses labels for every pod.
|
||||||
|
|
||||||
Persistent storage in kolla-kubernetes will come from volumes backed by
|
Persistent storage in kolla-kubernetes will come from volumes backed by
|
||||||
different storage offerings to provide persistent storage. Kolla-kubernetes
|
different storage offerings to provide persistent storage. Kolla-kubernetes
|
||||||
will provide a default solution using Ceph RBD, that the community will use to
|
will provide a default solution using Ceph RBD, that the community will use to
|
||||||
deploy multinode with. From there, kolla-kubernetes can add any additional
|
deploy multinode with. From there, kolla-kubernetes can add any additional
|
||||||
persistent storage options as well as support options for the operator to
|
persistent storage options as well as support options for the operator to
|
||||||
reference an existing storage solution.
|
reference an existing storage solution.
|
||||||
|
|
||||||
To deploy Ceph, the community will use the Ansible playbooks from Kolla to
|
To deploy Ceph, the community will use the Ansible playbooks from Kolla to
|
||||||
deploy a containerized Ceph at least for the 1.0 release. After Kubernetes
|
deploy a containerized Ceph at least for the 1.0 release. After Kubernetes
|
||||||
deployment matures, the community can evaluate building its own Ceph deployment
|
deployment matures, the community can evaluate building its own Ceph deployment
|
||||||
solution.
|
solution.
|
||||||
|
|
||||||
@ -198,9 +198,9 @@ Service Roles
|
|||||||
At the broadest level, OpenStack can split up into two main roles, Controller
|
At the broadest level, OpenStack can split up into two main roles, Controller
|
||||||
and Compute. With Kubernetes, the role definition layer changes.
|
and Compute. With Kubernetes, the role definition layer changes.
|
||||||
Kolla-kubernetes will still need to define Compute nodes, but not Controller
|
Kolla-kubernetes will still need to define Compute nodes, but not Controller
|
||||||
nodes. Compute nodes hold the libvirt container and the running vms. That
|
nodes. Compute nodes hold the libvirt container and the running vms. That
|
||||||
service cannont migrate because the vms associated with it exist on the node.
|
service cannont migrate because the vms associated with it exist on the node.
|
||||||
However, the Controller role is more flexible. The Kubernetes layer provides IP
|
However, the Controller role is more flexible. The Kubernetes layer provides IP
|
||||||
persistence so that APIs will remain active and abstracted from the operator's
|
persistence so that APIs will remain active and abstracted from the operator's
|
||||||
view [15]. kolla-kubernetes can direct Controller services away from the Compute
|
view [15]. kolla-kubernetes can direct Controller services away from the Compute
|
||||||
node using labels, while managing Compute services more strictly.
|
node using labels, while managing Compute services more strictly.
|
||||||
@ -244,7 +244,7 @@ To reuse Kolla's containers, kolla-kubernetes will use elastic search, heka, and
|
|||||||
kibana as the default logging mechanism.
|
kibana as the default logging mechanism.
|
||||||
|
|
||||||
The community will implement centralized logging by using a 'side car' container
|
The community will implement centralized logging by using a 'side car' container
|
||||||
in the Kubernetes pod [17]. The logging service will trace the logs from the
|
in the Kubernetes pod [17]. The logging service will trace the logs from the
|
||||||
shared volume of the running serivce and send the data to elastic search. This
|
shared volume of the running serivce and send the data to elastic search. This
|
||||||
solution is ideal because volumes are shared amoung the containers in a pod.
|
solution is ideal because volumes are shared amoung the containers in a pod.
|
||||||
|
|
||||||
|
@ -4,15 +4,15 @@ Logging with Heka
|
|||||||
|
|
||||||
https://blueprints.launchpad.net/kolla/+spec/heka
|
https://blueprints.launchpad.net/kolla/+spec/heka
|
||||||
|
|
||||||
Kolla currently uses Rsyslog for logging. And Change Request ``252968`` [1]
|
Kolla currently uses Rsyslog for logging. And Change Request ``252968`` [1]
|
||||||
suggests to use ELK (Elasticsearch, Logstash, Kibana) as a way to index all the
|
suggests to use ELK (Elasticsearch, Logstash, Kibana) as a way to index all the
|
||||||
logs, and visualize them.
|
logs, and visualize them.
|
||||||
|
|
||||||
This spec suggests using Heka [2] instead of Logstash, while still using
|
This spec suggests using Heka [2] instead of Logstash, while still using
|
||||||
Elasticsearch for indexing and Kibana for visualization. It also discusses
|
Elasticsearch for indexing and Kibana for visualization. It also discusses
|
||||||
the removal of Rsyslog along the way.
|
the removal of Rsyslog along the way.
|
||||||
|
|
||||||
What is Heka? Heka is a open-source stream processing software created and
|
What is Heka? Heka is a open-source stream processing software created and
|
||||||
maintained by Mozilla.
|
maintained by Mozilla.
|
||||||
|
|
||||||
Using Heka will provide a lightweight and scalable log processing solution
|
Using Heka will provide a lightweight and scalable log processing solution
|
||||||
@ -22,7 +22,7 @@ Problem description
|
|||||||
===================
|
===================
|
||||||
|
|
||||||
Change Request ``252968`` [1] adds an Ansible role named "elk" that enables
|
Change Request ``252968`` [1] adds an Ansible role named "elk" that enables
|
||||||
deploying ELK (Elasticsearch, Logstash, Kibana) on nodes with that role. This
|
deploying ELK (Elasticsearch, Logstash, Kibana) on nodes with that role. This
|
||||||
spec builds on that work, proposing a scalable log processing architecture
|
spec builds on that work, proposing a scalable log processing architecture
|
||||||
based on the Heka [2] stream processing software.
|
based on the Heka [2] stream processing software.
|
||||||
|
|
||||||
@ -34,7 +34,7 @@ OpenStack nodes rather than using a centralized log processing engine that
|
|||||||
represents a bottleneck and a single-point-of-failure.
|
represents a bottleneck and a single-point-of-failure.
|
||||||
|
|
||||||
We also know from experience that Heka provides all the necessary flexibility
|
We also know from experience that Heka provides all the necessary flexibility
|
||||||
for processing other types of data streams than log messages. For example, we
|
for processing other types of data streams than log messages. For example, we
|
||||||
already use Heka together with Elasticsearch for logs, but also with collectd
|
already use Heka together with Elasticsearch for logs, but also with collectd
|
||||||
and InfluxDB for statistics and metrics.
|
and InfluxDB for statistics and metrics.
|
||||||
|
|
||||||
@ -53,16 +53,16 @@ in a dedicated container, referred to as the Heka container in the rest of this
|
|||||||
document.
|
document.
|
||||||
|
|
||||||
Each Heka instance reads and processes the logs local to the node it runs on,
|
Each Heka instance reads and processes the logs local to the node it runs on,
|
||||||
and sends these logs to Elasticsearch for indexing. Elasticsearch may be
|
and sends these logs to Elasticsearch for indexing. Elasticsearch may be
|
||||||
distributed on multiple nodes for resiliency and scalability, but that part is
|
distributed on multiple nodes for resiliency and scalability, but that part is
|
||||||
outside the scope of that specification.
|
outside the scope of that specification.
|
||||||
|
|
||||||
Heka, written in Go, is fast and has a small footprint, making it possible to
|
Heka, written in Go, is fast and has a small footprint, making it possible to
|
||||||
run it on every node of the cluster. In contrast, Logstash runs in a JVM and
|
run it on every node of the cluster. In contrast, Logstash runs in a JVM and
|
||||||
is known [3] to be too heavy to run on every node.
|
is known [3] to be too heavy to run on every node.
|
||||||
|
|
||||||
Another important aspect is flow control and avoiding the loss of log messages
|
Another important aspect is flow control and avoiding the loss of log messages
|
||||||
in case of overload. Heka’s filter and output plugins, and the Elasticsearch
|
in case of overload. Heka’s filter and output plugins, and the Elasticsearch
|
||||||
output plugin in particular, support the use of a disk based message queue.
|
output plugin in particular, support the use of a disk based message queue.
|
||||||
This message queue allows plugins to reprocess messages from the queue when
|
This message queue allows plugins to reprocess messages from the queue when
|
||||||
downstream servers (Elasticsearch) are down or cannot keep up with the data
|
downstream servers (Elasticsearch) are down or cannot keep up with the data
|
||||||
@ -74,20 +74,20 @@ which introduces some complexity and other points-of-failures.
|
|||||||
Remove Rsyslog
|
Remove Rsyslog
|
||||||
--------------
|
--------------
|
||||||
|
|
||||||
Kolla currently uses Rsyslog. The Kolla services are configured to write their
|
Kolla currently uses Rsyslog. The Kolla services are configured to write their
|
||||||
logs to Syslog. Rsyslog gets the logs from the ``/var/lib/kolla/dev/log`` Unix
|
logs to Syslog. Rsyslog gets the logs from the ``/var/lib/kolla/dev/log`` Unix
|
||||||
socket and dispatches them to log files on the local file system. Rsyslog
|
socket and dispatches them to log files on the local file system. Rsyslog
|
||||||
running in a Docker container, the log files are stored in a Docker volume
|
running in a Docker container, the log files are stored in a Docker volume
|
||||||
(named ``rsyslog``).
|
(named ``rsyslog``).
|
||||||
|
|
||||||
With Rsyslog already running on each cluster node, the question of using two
|
With Rsyslog already running on each cluster node, the question of using two
|
||||||
log processing daemons, namely ``rsyslogd`` and ``hekad``, has been raised on
|
log processing daemons, namely ``rsyslogd`` and ``hekad``, has been raised on
|
||||||
the mailing list. The spec evaluates the possibility of using ``hekad`` only,
|
the mailing list. The spec evaluates the possibility of using ``hekad`` only,
|
||||||
based on some prototyping work we have conducted [4].
|
based on some prototyping work we have conducted [4].
|
||||||
|
|
||||||
Note: Kolla doesn't currently collect logs from RabbitMQ, HAProxy and
|
Note: Kolla doesn't currently collect logs from RabbitMQ, HAProxy and
|
||||||
Keepalived. For RabbitMQ the problem is related to RabbitMQ not having the
|
Keepalived. For RabbitMQ the problem is related to RabbitMQ not having the
|
||||||
capability to write its logs to Syslog. HAProxy and Keepalived do have that
|
capability to write its logs to Syslog. HAProxy and Keepalived do have that
|
||||||
capability, but the ``/var/lib/kolla/dev/log`` Unix socket file is currently
|
capability, but the ``/var/lib/kolla/dev/log`` Unix socket file is currently
|
||||||
not mounted into the HAProxy and Keepalived containers.
|
not mounted into the HAProxy and Keepalived containers.
|
||||||
|
|
||||||
@ -96,21 +96,21 @@ Use Heka's ``DockerLogInput`` plugin
|
|||||||
|
|
||||||
To remove Rsyslog and only use Heka one option would be to make the Kolla
|
To remove Rsyslog and only use Heka one option would be to make the Kolla
|
||||||
services write their logs to ``stdout`` (or ``stderr``) and rely on Heka's
|
services write their logs to ``stdout`` (or ``stderr``) and rely on Heka's
|
||||||
``DockerLogInput`` plugin [5] for reading the logs. Our experiments have
|
``DockerLogInput`` plugin [5] for reading the logs. Our experiments have
|
||||||
revealed a number of problems with this option:
|
revealed a number of problems with this option:
|
||||||
|
|
||||||
* The ``DockerLogInput`` plugin doesn't currently work for containers that have
|
* The ``DockerLogInput`` plugin doesn't currently work for containers that have
|
||||||
a ``tty`` allocated. And Kolla currently allocates a tty for all containers
|
a ``tty`` allocated. And Kolla currently allocates a tty for all containers
|
||||||
(for good reasons).
|
(for good reasons).
|
||||||
|
|
||||||
* When ``DockerLogInput`` is used there is no way to differentiate log messages
|
* When ``DockerLogInput`` is used there is no way to differentiate log messages
|
||||||
for containers producing multiple log streams. ``neutron-agents`` is an
|
for containers producing multiple log streams. ``neutron-agents`` is an
|
||||||
example of such a container. (Sam Yaple has raised that issue multiple
|
example of such a container. (Sam Yaple has raised that issue multiple
|
||||||
times.)
|
times.)
|
||||||
|
|
||||||
* If Heka is stopped and restarted later then log messages will be lost, as the
|
* If Heka is stopped and restarted later then log messages will be lost, as the
|
||||||
``DockerLogInput`` plugin doesn't currently have a mechanism for tracking its
|
``DockerLogInput`` plugin doesn't currently have a mechanism for tracking its
|
||||||
positions in the log streams. This is in contrast to the ``LogstreamerInput``
|
positions in the log streams. This is in contrast to the ``LogstreamerInput``
|
||||||
plugin [6] which does include that mechanism.
|
plugin [6] which does include that mechanism.
|
||||||
|
|
||||||
For these reasons we think that relying on the ``DockerLogInput`` plugin may
|
For these reasons we think that relying on the ``DockerLogInput`` plugin may
|
||||||
@ -119,7 +119,7 @@ not be a practical option.
|
|||||||
For the note, our experiments have also shown that the OpenStack containers
|
For the note, our experiments have also shown that the OpenStack containers
|
||||||
logs written to ``stdout`` are visible to neither Heka nor ``docker logs``.
|
logs written to ``stdout`` are visible to neither Heka nor ``docker logs``.
|
||||||
This problem is not reproducible when ``stderr`` is used rather than
|
This problem is not reproducible when ``stderr`` is used rather than
|
||||||
``stdout``. The cause of this problem is currently unknown. And it looks like
|
``stdout``. The cause of this problem is currently unknown. And it looks like
|
||||||
other people have come across that issue [7].
|
other people have come across that issue [7].
|
||||||
|
|
||||||
Use local log files
|
Use local log files
|
||||||
@ -129,7 +129,7 @@ Another option consists of configuring all the Kolla services to log into local
|
|||||||
files, and using Heka's ``LogstreamerInput`` plugin [5].
|
files, and using Heka's ``LogstreamerInput`` plugin [5].
|
||||||
|
|
||||||
This option involves using a Docker named volume, mounted both into the service
|
This option involves using a Docker named volume, mounted both into the service
|
||||||
containers (in ``rw`` mode) and into the Heka container (in ``ro`` mode). The
|
containers (in ``rw`` mode) and into the Heka container (in ``ro`` mode). The
|
||||||
services write logs into files placed in that volume, and Heka reads logs from
|
services write logs into files placed in that volume, and Heka reads logs from
|
||||||
the files found in that volume.
|
the files found in that volume.
|
||||||
|
|
||||||
@ -138,28 +138,28 @@ And it relies on Heka's ``LogstreamerInput`` plugin, which, based on our
|
|||||||
experience, is efficient and robust.
|
experience, is efficient and robust.
|
||||||
|
|
||||||
Keeping file logs locally on the nodes has been established as a requirement by
|
Keeping file logs locally on the nodes has been established as a requirement by
|
||||||
the Kolla developers. With this option, and the Docker volume used, meeting
|
the Kolla developers. With this option, and the Docker volume used, meeting
|
||||||
that requirement necessitates no additional mechanism.
|
that requirement necessitates no additional mechanism.
|
||||||
|
|
||||||
For this option to be applicable the services must have the capability of
|
For this option to be applicable the services must have the capability of
|
||||||
logging into files. Most of the Kolla services have this capability. The
|
logging into files. Most of the Kolla services have this capability. The
|
||||||
exceptions are HAProxy and Keepalived, for which a different mechanism should
|
exceptions are HAProxy and Keepalived, for which a different mechanism should
|
||||||
be used (described further down in the document). Note that this will make it
|
be used (described further down in the document). Note that this will make it
|
||||||
possible to collect logs from RabbitMQ, which does not support logging to
|
possible to collect logs from RabbitMQ, which does not support logging to
|
||||||
Syslog but does support logging to a file.
|
Syslog but does support logging to a file.
|
||||||
|
|
||||||
Also, this option requires that the services have the permission to create
|
Also, this option requires that the services have the permission to create
|
||||||
files into the Docker volume, and that Heka has the permission to read these
|
files into the Docker volume, and that Heka has the permission to read these
|
||||||
files. This means that the Docker named volume will have to have appropriate
|
files. This means that the Docker named volume will have to have appropriate
|
||||||
owner, group and permission bits. With the Heka container running under
|
owner, group and permission bits. With the Heka container running under
|
||||||
a specific user (see below) this will mean using an ``extend_start.sh`` script
|
a specific user (see below) this will mean using an ``extend_start.sh`` script
|
||||||
including ``sudo chown`` and possibly ``sudo chmod`` commands. Our prototype
|
including ``sudo chown`` and possibly ``sudo chmod`` commands. Our prototype
|
||||||
[4] already includes this.
|
[4] already includes this.
|
||||||
|
|
||||||
As mentioned already the ``LogstreamerInput`` plugin includes a mechanism for
|
As mentioned already the ``LogstreamerInput`` plugin includes a mechanism for
|
||||||
tracking positions in log streams. This works with journal files stored on the
|
tracking positions in log streams. This works with journal files stored on the
|
||||||
file system (in ``/var/cache/hekad``). A specific volume, private to Heka,
|
file system (in ``/var/cache/hekad``). A specific volume, private to Heka,
|
||||||
will be used for these journal files. In this way no logs will be lost if the
|
will be used for these journal files. In this way no logs will be lost if the
|
||||||
Heka container is removed and a new one is created.
|
Heka container is removed and a new one is created.
|
||||||
|
|
||||||
Handling HAProxy and Keepalived
|
Handling HAProxy and Keepalived
|
||||||
@ -174,7 +174,7 @@ This works by using Heka's ``UdpInput`` plugin with its ``net`` option set
|
|||||||
to ``unixgram``.
|
to ``unixgram``.
|
||||||
|
|
||||||
This also requires that a Unix socket is created by Heka, and that socket is
|
This also requires that a Unix socket is created by Heka, and that socket is
|
||||||
mounted into the HAProxy and Keepalived containers. For that we will use the
|
mounted into the HAProxy and Keepalived containers. For that we will use the
|
||||||
same technique as the one currently used in Kolla with Rsyslog, that is
|
same technique as the one currently used in Kolla with Rsyslog, that is
|
||||||
mounting ``/var/lib/kolla/dev`` into the Heka container and mounting
|
mounting ``/var/lib/kolla/dev`` into the Heka container and mounting
|
||||||
``/var/lib/kolla/dev/log`` into the service containers.
|
``/var/lib/kolla/dev/log`` into the service containers.
|
||||||
@ -182,7 +182,7 @@ mounting ``/var/lib/kolla/dev`` into the Heka container and mounting
|
|||||||
Our prototype already includes some code demonstrating this. See [4].
|
Our prototype already includes some code demonstrating this. See [4].
|
||||||
|
|
||||||
Also, to be able to store a copy of the HAProxy and Keepalived logs locally on
|
Also, to be able to store a copy of the HAProxy and Keepalived logs locally on
|
||||||
the node, we will use Heka's ``FileOutput`` plugin. We will possibly create
|
the node, we will use Heka's ``FileOutput`` plugin. We will possibly create
|
||||||
two instances of that plugin, one for HAProxy and one for Keepalived, with
|
two instances of that plugin, one for HAProxy and one for Keepalived, with
|
||||||
specific filters (``message_matcher``).
|
specific filters (``message_matcher``).
|
||||||
|
|
||||||
@ -190,29 +190,29 @@ Read Python Tracebacks
|
|||||||
----------------------
|
----------------------
|
||||||
|
|
||||||
In case of exceptions the OpenStack services log Python Tracebacks as multiple
|
In case of exceptions the OpenStack services log Python Tracebacks as multiple
|
||||||
log messages. If no special care is taken then the Python Tracebacks will be
|
log messages. If no special care is taken then the Python Tracebacks will be
|
||||||
indexed as separate documents in Elasticsearch, and displayed as distinct log
|
indexed as separate documents in Elasticsearch, and displayed as distinct log
|
||||||
entries in Kibana, making them hard to read. To address that issue we will use
|
entries in Kibana, making them hard to read. To address that issue we will use
|
||||||
a custom Heka decoder, which will be responsible for coalescing the log lines
|
a custom Heka decoder, which will be responsible for coalescing the log lines
|
||||||
making up a Python Traceback into one message. Our prototype includes that
|
making up a Python Traceback into one message. Our prototype includes that
|
||||||
decoder [4].
|
decoder [4].
|
||||||
|
|
||||||
Collect system logs
|
Collect system logs
|
||||||
-------------------
|
-------------------
|
||||||
|
|
||||||
In addition to container logs we think it is important to collect system logs
|
In addition to container logs we think it is important to collect system logs
|
||||||
as well. For that we propose to mount the host's ``/var/log`` directory into
|
as well. For that we propose to mount the host's ``/var/log`` directory into
|
||||||
the Heka container, and configure Heka to get logs from standard log files
|
the Heka container, and configure Heka to get logs from standard log files
|
||||||
located in that directory (e.g. ``kern.log``, ``auth.log``, ``messages``). The
|
located in that directory (e.g. ``kern.log``, ``auth.log``, ``messages``). The
|
||||||
list of system log files will be determined at development time.
|
list of system log files will be determined at development time.
|
||||||
|
|
||||||
Log rotation
|
Log rotation
|
||||||
------------
|
------------
|
||||||
|
|
||||||
Log rotation is an important aspect of the logging system. Currently Kolla
|
Log rotation is an important aspect of the logging system. Currently Kolla
|
||||||
doesn't rotate logs. Logs just accumulate in the ``rsyslog`` Docker volume.
|
doesn't rotate logs. Logs just accumulate in the ``rsyslog`` Docker volume.
|
||||||
The work on Heka proposed in this spec isn't directly related to log rotation,
|
The work on Heka proposed in this spec isn't directly related to log rotation,
|
||||||
but we are suggesting to address this issue for Mitaka. This will mean
|
but we are suggesting to address this issue for Mitaka. This will mean
|
||||||
creating a new container that uses ``logrotate`` to manage the log files
|
creating a new container that uses ``logrotate`` to manage the log files
|
||||||
created by the Kolla containers.
|
created by the Kolla containers.
|
||||||
|
|
||||||
@ -220,33 +220,33 @@ Create an ``heka`` user
|
|||||||
-----------------------
|
-----------------------
|
||||||
|
|
||||||
For security reasons an ``heka`` user will be created in the Heka container and
|
For security reasons an ``heka`` user will be created in the Heka container and
|
||||||
the ``hekad`` daemon will run under that user. The ``heka`` user will be added
|
the ``hekad`` daemon will run under that user. The ``heka`` user will be added
|
||||||
to the ``kolla`` group, to make sure that Heka can read the log files created
|
to the ``kolla`` group, to make sure that Heka can read the log files created
|
||||||
by the services.
|
by the services.
|
||||||
|
|
||||||
Security impact
|
Security impact
|
||||||
---------------
|
---------------
|
||||||
|
|
||||||
Heka is a mature product maintained and used in production by Mozilla. So we
|
Heka is a mature product maintained and used in production by Mozilla. So we
|
||||||
trust Heka as being secure. We also trust the Heka developers as being serious
|
trust Heka as being secure. We also trust the Heka developers as being serious
|
||||||
should security vulnerabilities be found in the Heka code.
|
should security vulnerabilities be found in the Heka code.
|
||||||
|
|
||||||
As described above we are proposing to use a Docker volume between the service
|
As described above we are proposing to use a Docker volume between the service
|
||||||
containers and the Heka container. The group of the volume directory and the
|
containers and the Heka container. The group of the volume directory and the
|
||||||
log files will be ``kolla``. And the owner of the log files will be the user
|
log files will be ``kolla``. And the owner of the log files will be the user
|
||||||
that executes the service producing logs. But the ``gid`` of the ``kolla``
|
that executes the service producing logs. But the ``gid`` of the ``kolla``
|
||||||
group and the ``uid``'s of the users executing the services may correspond
|
group and the ``uid``'s of the users executing the services may correspond
|
||||||
to a different group and different users on the host system. This means
|
to a different group and different users on the host system. This means
|
||||||
that the permissions may not be right on the host system. This problem is
|
that the permissions may not be right on the host system. This problem is
|
||||||
not specific to this specification, and it already exists in Kolla (for
|
not specific to this specification, and it already exists in Kolla (for
|
||||||
the mariadb data volume for example).
|
the mariadb data volume for example).
|
||||||
|
|
||||||
Performance Impact
|
Performance Impact
|
||||||
------------------
|
------------------
|
||||||
|
|
||||||
The ``hekad`` daemon will run in a container on each cluster node. But the
|
The ``hekad`` daemon will run in a container on each cluster node. But the
|
||||||
``rsyslogd`` will be removed. And we have assessed that Heka is lightweight
|
``rsyslogd`` will be removed. And we have assessed that Heka is lightweight
|
||||||
enough to run on every node. Also, a possible option would be to constrain the
|
enough to run on every node. Also, a possible option would be to constrain the
|
||||||
Heka container to only use a defined amount of resources.
|
Heka container to only use a defined amount of resources.
|
||||||
|
|
||||||
Alternatives
|
Alternatives
|
||||||
@ -256,12 +256,12 @@ An alternative to this proposal involves using Logstash in a centralized
|
|||||||
way as done in [1].
|
way as done in [1].
|
||||||
|
|
||||||
Another alternative would be to execute Logstash on each cluster node, as this
|
Another alternative would be to execute Logstash on each cluster node, as this
|
||||||
spec proposes with Heka. But this would mean running a JVM on each cluster
|
spec proposes with Heka. But this would mean running a JVM on each cluster
|
||||||
node, and using Redis as a centralized queue.
|
node, and using Redis as a centralized queue.
|
||||||
|
|
||||||
Also, as described above, we initially considered relying on services writing
|
Also, as described above, we initially considered relying on services writing
|
||||||
their logs to ``stdout`` and use Heka's ``DockerLogInput`` plugin. But our
|
their logs to ``stdout`` and use Heka's ``DockerLogInput`` plugin. But our
|
||||||
prototyping work has demonstrated the limits of that approach. See the
|
prototyping work has demonstrated the limits of that approach. See the
|
||||||
``DockerLogInput`` section above for more information.
|
``DockerLogInput`` section above for more information.
|
||||||
|
|
||||||
Implementation
|
Implementation
|
||||||
|
@ -8,8 +8,8 @@
|
|||||||
This template should be in ReSTructured text. The filename in the git
|
This template should be in ReSTructured text. The filename in the git
|
||||||
repository should match the launchpad URL, for example a URL of
|
repository should match the launchpad URL, for example a URL of
|
||||||
https://blueprints.launchpad.net/kolla/+spec/awesome-thing should be named
|
https://blueprints.launchpad.net/kolla/+spec/awesome-thing should be named
|
||||||
awesome-thing.rst . Please do not delete any of the sections in this
|
awesome-thing.rst . Please do not delete any of the sections in this
|
||||||
template. If you have nothing to say for a whole section, just write: None
|
template. If you have nothing to say for a whole section, just write: None
|
||||||
For help with syntax, see http://sphinx-doc.org/rest.html
|
For help with syntax, see http://sphinx-doc.org/rest.html
|
||||||
To test out your formatting, see http://www.tele3.cz/jbar/rest/rest.html
|
To test out your formatting, see http://www.tele3.cz/jbar/rest/rest.html
|
||||||
|
|
||||||
|
Loading…
Reference in New Issue
Block a user