Updates to Airflow 1.10.1; See (1), (2) for some notes
Related, and additionally: configures Airflow to restore logging of
workflow steps to a console/sdtout logger, supporting the desired
ability to attach logging and monitoring to standard container
mechanisms. This does not change the behavior of also logging to the
airflow-arranged log files for steps and DAG runs.
A side effect of updating to 1.10.1 includes a major decrease in
resource usage by the Airflow scheudler process (reducing from ~ 1 core
fully consumed to less than 5% of a core consumed YMMV, but significant)
Additional adjustment downward of resources allocated, threads produced,
and frequency of polling leads to an overall significant reduction in
Because Airflow 1.10.0 and 1.10.1 use compatible versions of celery and
dag_run information, updating from 1.10.0 - 1.10.1 in place is possible
if airflow-worker pods are allowed to continue to run.