.. Copyright 2019 Red Hat, Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ============================== Octavia Amphora Log Offloading ============================== The default logging configuration will store the logs locally, on the amphora filesystem with file rotation. Octavia Amphorae can offload their log files via the syslog protocol to syslog receivers via the load balancer management network (lb-mgmt-net). This allows log aggregation of both administrative logs and also tenant traffic flow logs. The syslog receivers can either be local to the load balancer management network or routable via the load balancer management network. By default any syslog receiver that supports UDP or TCP syslog protocol can be used, however the operator also has the option to create an override rsyslog configuration template to enable other features or protocols their Amphora image may support. This guide will discuss the features of :term:`Amphora` log offloading and how to configure them. Administrative Logs =================== The administrative log offloading feature of the :term:`Amphora` covers all of the system logging inside the :term:`Amphora` except for the tenant flow logs. Tenant flow logs can be sent to and processed by the same syslog receiver used by the administrative logs, but they are configured separately. All administrative log messages will be sent using the native log format for the application sending the message. Enabling Administrative Log Offloading -------------------------------------- One or more syslog receiver endpoints must be configured in the Octavia configuration file to enable administrative log offloading. The first endpoint will be the primary endpoint to receive the syslog packets. Read the :ref:`rsyslog failover considerations` section for information about how to use multiple target servers. To configure administrative log offloading, set the following setting in your Octavia configuration file for all of the controllers and restart them: .. code-block:: ini [amphora_agent] admin_log_targets = 192.0.2.1:10514 In this example, the syslog receiver will be 192.0.2.1 on port 10514. If *log_protocol* is not specified UDP will be used. .. note:: Make sure your syslog receiver endpoints are accessible from the load balancer management network and you have configured the required security group or firewall rules to allow the traffic. These endpoints can be routable addresses from the load balancer management network. The load balancer related administrative logs will be sent using a LOG_LOCAL[0-7] facility. The facility number defaults to 1, but is configurable using the administrative_log_facility setting in the Octavia configuration file. To configure administrative log facility, set the following setting in your Octavia configuration file for all of the controllers and restart them: .. code-block:: ini [amphora_agent] administrative_log_facility = 1 Forwarding All Administrative Logs ---------------------------------- By default, the Amphorae will only forward load balancer related administrative logs, such as the haproxy admin logs, keepalived, and :term:`Amphora` agent logs. You can optionally configure the Amphorae to send all of the administrative logs from the :term:`Amphora`, such as the kernel, system, and security logs. Even with this setting the tenant flow logs will not be included. You can configure tenant flow log forwarding in the `Tenant Flow Logs`_ section. The load balancer related administrative logs will be sent using the LOG_LOCAL[0-7] configured using the administrative_log_facility setting. All other administrative log messages will use their native syslog facilities. To configure the Amphorae to forward all administrative logs, set the following setting in your Octavia configuration file for all of the controllers and restart them: .. code-block:: ini [amphora_agent] forward_all_logs = True Tenant Flow Logs ================ Enabling Tenant Flow Log Offloading ----------------------------------- One or more syslog receiver endpoints must be configured in the Octavia configuration file to enable tenant flow log offloading. The first endpoint will be the primary endpoint to receive the syslog packets. The endpoints configured for tenant flow log offloading may be the same endpoints as the administrative log offloading configuration. Read the :ref:`rsyslog failover considerations` section for information about how to use multiple target servers. .. warning:: Tenant flow logging can produce a large number of syslog messages depending on how many connections the load balancers are receiving. Tenant flow logging produces one log entry per connection to the load balancer. We recommend you monitor, size, and configure your syslog receivers appropriately based on the expected number of connections your load balancers will be handling. To configure tenant flow log offloading, set the following setting in your Octavia configuration file for all of the controllers and restart them: .. code-block:: ini [amphora_agent] tenant_log_targets = 192.0.2.1:10514 In this example, the syslog receiver will be 192.0.2.1 on port 10514. If *log_protocol* is not specified UDP will be used. .. note:: Make sure your syslog receiver endpoints are accessible from the load balancer management network and you have configured the required security group or firewall rules to allow the traffic. These endpoints can be routable addresses from the load balancer management network. The load balancer related tenant flow logs will be sent using a LOG_LOCAL[0-7] facility. The facility number defaults to 0, but is configurable using the user_log_facility setting in the Octavia configuration file. To configure the tenant flow log facility, set the following setting in your Octavia configuration file for all of the controllers and restart them: .. code-block:: ini [amphora_agent] user_log_facility = 0 Tenant Flow Log Format ---------------------- The default tenant flow log format is: .. code-block:: project_id loadbalancer_id listener_id client_ip client_port data_time request_string http_status bytes_read bytes_uploaded client_certificate_verify(0 or 1) client_certificate_distinguised_name pool_id member_id processing_time(ms) termination_state Any field that is unknown or not applicable to the connection will have a '-' character in its place. An example log entry when using rsyslog as the syslog receiver is: .. note:: The prefix[1] in this example comes from the rsyslog receiver and is not part of the syslog message from the amphora. [1] "Jun 12 00:44:13 amphora-3e0239c3-5496-4215-b76c-6abbe18de573 haproxy[1644]:" .. code-block:: Jun 12 00:44:13 amphora-3e0239c3-5496-4215-b76c-6abbe18de573 haproxy[1644]: 5408b89aa45b48c69a53dca1aaec58db fd8f23df-960b-4b12-ba62-2b1dff661ee7 261ecfc2-9e8e-4bba-9ec2-3c903459a895 172.24.4.1 41152 12/Jun/2019:00:44:13.030 "GET / HTTP/1.1" 200 76 73 - "" e37e0e04-68a3-435b-876c-cffe4f2138a4 6f2720b3-27dc-4496-9039-1aafe2fee105 4 -- Custom Tenant Flow Log Format ----------------------------- You can optionally specify a custom log format for the tenant flow logs. This string follows the HAProxy log format variables with the exception of the "{{ project_id }}" and "{{ lb_id }}" variables that will be replaced by the Octavia :term:`Amphora` driver. These custom variables are optional. See the HAProxy documentation for `Custom log format `_ variable definitions. To configure a custom log format, set the following setting in your Octavia configuration file for all of the controllers and restart them: .. code-block:: ini [haproxy_amphora] user_log_format = '{{ project_id }} {{ lb_id }} %f %ci %cp %t %{+Q}r %ST %B %U %[ssl_c_verify] %{+Q}[ssl_c_s_dn] %b %s %Tt %tsc' .. _rsyslog failover considerations: Failover Considerations ======================= In order to provide protection against potential data loss because of downtime of a single syslog server, it may be a advisable to use multiple log targets. In such configuration *log_protocol* needs to be set to *TCP*. With the UDP syslog protocol, RSyslog is unable to detect if the primary endpoint has failed. Also pay attention to the *log_retry_count* and *log_retry_interval* settings when using multiple log targets. You might want to set *log_retry_count* to 0 and use a higher value for *log_retry_interval*. Values up to 1800 (30 minutes) are possible. That way the failover will happen immediately after the client detects that the server became unavailable. In such case, that server won't be used again for at least *log_retry_interval* seconds after that event. In the following example the primary syslog receiver will be 192.0.2.1 on port 10514. The backup syslog receiver will be 2001:db8:1::10 on port 10514. .. code-block:: ini [amphora_agent] admin_log_targets = 192.0.2.1:10514, 2001:db8:1::10:10514 tenant_log_targets = 192.0.2.1:10514, 2001:db8:1::10:10514 log_protocol = TCP log_retry_count = 0 log_retry_interval = 1800 Disabling Logging ================= There may be cases where you need to disable logging inside the :term:`Amphora`, such as complying with regulatory standards. Octavia provides multiple options for disabling :term:`Amphora` logging. Disable Local Log Storage ------------------------- This setting stops log entries from being written to the disk inside the :term:`Amphora`. Logs can still be sent via :term:`Amphora` log offloading if log offloading is configured for the Amphorae. Enabling this setting may provide a performance benefit to the load balancer. .. warning:: This feature disables ALL log storage in the :term:`Amphora`, including kernel, system, and security logging. .. note:: If you enable this setting and are not using :term:`Amphora` log offloading, we recommend you also `Disable Tenant Flow Logging`_ to improve load balancing performance. To disable local log storage in the :term:`Amphora`, set the following setting in your Octavia configuration file for all of the controllers and restart them: .. code-block:: ini [amphora_agent] disable_local_log_storage = True Disable Tenant Flow Logging --------------------------- This setting allows you to disable tenant flow logging irrespective of the other logging configuration settings. It will take precedent over the other settings. When this setting is enabled, no tenant flow (connection) logs will be written to the disk inside the :term:`Amphora` or be sent via the :term:`Amphora` log offloading. .. note:: Disabling tenant flow logging can also improve the load balancing performance of the amphora. Due to the potential performance improvement, we recommend you enable this setting when using the `Disable Local Log Storage`_ setting. To disable tenant flow logging, set the following setting in your Octavia configuration file for all of the controllers and restart them: .. code-block:: ini [haproxy_amphora] connection_logging = False