diff --git a/elk_metrics_6x/assets/Elastic-Stack-Diagram.svg b/elk_metrics_6x/assets/Elastic-Stack-Diagram.svg new file mode 100644 index 00000000..cc360841 --- /dev/null +++ b/elk_metrics_6x/assets/Elastic-Stack-Diagram.svg @@ -0,0 +1,2 @@ + +
elastic APM
[Not supported by viewer]
Sensor Data
Sensor Data
Sensor Data
Sensor Data
Compute
<font style="font-size: 50px">Compute</font>
Storage
[Not supported by viewer]
Switching
[Not supported by viewer]
Sensor Data
Sensor Data
Sensor Data
Sensor Data
Control Plane
<font style="font-size: 50px">Control Plane</font>
APIs and Services
APIs and Services
APM Data
APM Data
Instances
<font style="font-size: 50px">Instances</font>
APM Data
APM Data
Logging Server
<font style="font-size: 50px">Logging Server</font>
Logstash
[Not supported by viewer]
APM Data
APM Data
Legand
[Not supported by viewer]
Filebeat
[Not supported by viewer]
Metricbeat
[Not supported by viewer]
Packetbeat
[Not supported by viewer]
Auditbeat
[Not supported by viewer]
Heartbeat
[Not supported by viewer]
Journalbeat
[Not supported by viewer]
Connbeat
[Not supported by viewer]
Beats
Beats

Architecture Diagram

  • The Elastic Stack is used to collect sensor data and store it for analysis and to facilitate observability.
  • Intra cluster communication and the transfer of data within the Elasticsearch cluster is done using a mesh topology.
  • Any component of the elastic stack can be scaled out independently of any other component and has no dependency on the version of software deployed within an environment.
  • Collectors will self-adjust based on application discovery.


[Not supported by viewer]
In control theory, observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs.
<i><font color="#545454">In control theory, observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs.</font></i> \ No newline at end of file diff --git a/elk_metrics_6x/assets/Elastic-dataflow.svg b/elk_metrics_6x/assets/Elastic-dataflow.svg new file mode 100644 index 00000000..5ccdf433 --- /dev/null +++ b/elk_metrics_6x/assets/Elastic-dataflow.svg @@ -0,0 +1,2 @@ + +
Load-balancer Path
Load-balancer Path
Elastic APM
Elastic APM
Community Beats
Community Beats
Master(s) 
Master(s) 
Logstash
[Not supported by viewer]
Node(n)
Node(n)
Ingest(s)
Ingest(s)
Logstash
[Not supported by viewer]
Node(n)
Node(n)
elastic APM
[Not supported by viewer]
Node(n)
Node(n)
Data(s) 
Data(s) 
Logstash
[Not supported by viewer]
Node(n)
Node(n)
Elasticsearch Master Path
Elasticsearch Master Path<br style="font-size: 20px">
Elasticsearch Coordinator Path
Elasticsearch Coordinator Path
Elasticsearch Data Path
Elasticsearch Data Path<br style="font-size: 20px">
Core Beats
Core Beats
Other Components
[Not supported by viewer]

Overview

Diagram shows all components as individual blocks. This is done for better understanding of the data flow and what each block is responsible for.

<h1><font style="font-size: 40px">Overview</font></h1><p style="font-size: 25px">Diagram shows all components as individual blocks. This is done for better understanding of the data flow and what each block is responsible for.</p>

Components

  • Elasticsearch:
    • Master node: Maintains cluster quorum 
    • Data node: Persistent data storage
    • Ingest node: Accepts ingress traffic to be processed and written to a data node.
    • Coordination node: Load balances traffic throughout the cluster.
  • Logstash:
    • Logstash will grok and process data before forwarding it to an Elasticsearch ingest node.
  • Kibana:
    • User front-end used to explore data held within Elasticsearch.
  • APM (optional):
    • Provides an interface to monitor application specific transactions. While not required, a load-balancer is recommended.
  • Beats:
    • Monitors responsible for collecting sensor data and system information. Where possible, all monitors will send data to Logstash. If the beat is incapable of using Logstash as an output plugin (some community beats) Elasticsearch will be used instead.
  • Other Components:
    • Other ingress traffic will generally send data directly to Elasticsearch. This can be done by directly connecting to an Elasticsearch ingest node or through the use of a load-balancer (recommended) which will balance traffic to all known ingest/coordination nodes.
    • Other components may also send data to logstash.

[Not supported by viewer]
elastic APM Agent
[Not supported by viewer]
Application(s)
Application(s)
Coordinator(s) 
Coordinator(s) 
Logstash
Logstash
API Access
API Access
USERS
USERS
User Data Path
User Data Path
Node(n)
Node(n)
Server(s)
[Not supported by viewer]
Other Components(s)
Other Components(s)
Server(s)
[Not supported by viewer]
beat(s)
beat(s) \ No newline at end of file diff --git a/elk_metrics_6x/readme.rst b/elk_metrics_6x/readme.rst index ed60d933..33995fda 100644 --- a/elk_metrics_6x/readme.rst +++ b/elk_metrics_6x/readme.rst @@ -12,6 +12,14 @@ the elastic stack. **These playbooks require Ansible 2.5+.** +Highlevel overview of the Elastic-Stack infrastructure these playbooks will +build and operate against. + +.. image:: assets/Elastic-Stack-Diagram.svg + :scale: 50 % + :alt: Elasticsearch Architecture Diagram + :align: center + OpenStack-Ansible Integration ----------------------------- @@ -315,6 +323,15 @@ deploy logstash, deploy Kibana, and then deploy all of the service beats. The individual playbooks found within this repository can be independently run at anytime. +Architecture | Data flow +^^^^^^^^^^^^^^^^^^^^^^^^ + +This diagram outlines the data flow from within an Elastic-Stack deployment. + +.. image:: assets/Elastic-dataflow.svg + :scale: 50 % + :alt: Elastic-Stack Data Flow Diagram + :align: center Optional | Enable uwsgi stats ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^