shipyard/images/airflow/config/__init__.py
Bryan Strassner d6c72d19e6 Update to Airflow 1.10.1 and restore sysout
Updates to Airflow 1.10.1; See (1), (2) for some notes

Related, and additionally: configures Airflow to restore logging of
workflow steps to a console/sdtout logger, supporting the desired
ability to attach logging and monitoring to standard container
mechanisms. This does not change the behavior of also logging to the
airflow-arranged log files for steps and DAG runs.

A side effect of updating to 1.10.1 includes a major decrease in
resource usage by the Airflow scheudler process (reducing from ~ 1 core
fully consumed to less than 5% of a core consumed YMMV, but significant)

Additional adjustment downward of resources allocated, threads produced,
and frequency of polling leads to an overall significant reduction in
resource usage.

Airship note:
Because Airflow 1.10.0 and 1.10.1 use compatible versions of celery and
dag_run information, updating from 1.10.0 - 1.10.1 in place is possible
if airflow-worker pods are allowed to continue to run.

(1) https://github.com/apache/incubator-airflow/blob/master/UPDATING.md
(2) https://github.com/apache/incubator-airflow/releases/tag/1.10.1

Change-Id: I9b024e3996c528c7b74e2888191d48c7a45a1f04
2018-12-05 12:37:14 -06:00

0 lines
Python