Commit Graph

93 Commits

Author SHA1 Message Date
Steve Wilkerson
290df62223 Elasticsearch: Update image for s3 bucket creation
This updates the Elasticsearch image used for s3 bucket creation
to use the same ceph daemon image used in the ceph-rgw chart now
that the Mimic release is supported

Change-Id: I416a283b8ac41f6b360d20aac1be8374c07badcd
2019-01-07 13:51:55 -06:00
Zuul
632742b5f7 Merge "Remove unused pod-etc-apache volumes" 2019-01-07 19:40:20 +00:00
Zuul
5347636108 Merge "Elasticsearch: Add security context for exporter pod/container" 2019-01-07 16:26:08 +00:00
Steve Wilkerson
8180635733 Helm-toolkit: Update job for creating s3 buckets
This updates the helm-toolkit manifest template and scipts for
creating an S3 bucket and linking it to a user. This moves away
from the previous python implementation that used rgwadmin, and
instead uses s3cmd for a cleaner approach that can support more
recent versions of ceph

Change-Id: I305062a5daa063bfe21a12448d7a3957bca00bf4
2019-01-05 14:37:47 +00:00
Steve Wilkerson
30d2cf00d4 Remove unused pod-etc-apache volumes
This removes unused pod-etc-apache volumes from the charts that
use an apache sidecar container as a reverse proxy.

Change-Id: Ibafff3b53f9d3c20f5aed30d40ee6470cb515a8a
2019-01-04 10:31:35 -06:00
Chris Wedgwood
0c4e37391f 'NOP' cleanup for more consistent white-space use in charts
Where we have the style '{{ ...' we should use the style '... }}'.

Change-Id: Ic3e779e4681370d396f95d3804ca27db5b9d3642
2019-01-03 22:45:49 +00:00
Steve Wilkerson
0679ed49bd Elasticsearch: Add security context for exporter pod/container
This adds the security context snippet for the elasticsearch
prometheus exporter pod. This changes the pod's user from root to
the nobody user instead

This also adds the container security context to explicitly set
allowPrivilegeEscalation to false

Change-Id: If692fccaf4dd362b28fecb4656036289a3a97122
2019-01-03 16:12:17 -06:00
Zuul
6d354f0f7b Merge "Revert "Add Egress Helm-toolkit function & enforce the nework policy at OSH-INFRA"" 2018-12-16 08:57:09 +00:00
Pete Birley
0bf3674539 Revert "Add Egress Helm-toolkit function & enforce the nework policy at OSH-INFRA"
This reverts commit 8d33a2911c.

Change-Id: Ic861b9bf9b337449b47a3558da8355e7a5bcacee
2018-12-16 04:21:46 +00:00
Zuul
b90bf10b89 Merge "Add Egress Helm-toolkit function & enforce the nework policy at OSH-INFRA" 2018-12-15 09:32:21 +00:00
Mike Pham
8d33a2911c Add Egress Helm-toolkit function & enforce the nework policy at OSH-INFRA
This PS implements the helm toolkit function to generate the
Egress in kubernetes network policy manifest based on overrideable values.
It also enbale the K8s network policy at Osh-infra gate.

Change-Id: Icbe2a18c98dba795d15398dcdcac64228f6a7b4c
2018-12-14 16:32:40 -05:00
Zuul
d984f3c782 Merge "Elasticsearch: Define success criteria for adding snapshot repo" 2018-12-12 23:23:48 +00:00
Steve Wilkerson
7be42d3cd5 Elasticsearch: Define success criteria for adding snapshot repo
This adds a simple check to the Elasticsearch snapshot repo job
that will cause the job to fail if the repository isn't added
successfully

Change-Id: I9dca6ef545b43c52a37542319fa2f706b174c44b
2018-12-12 14:44:49 -06:00
Steve Wilkerson
d3e046d803 Elasticsearch: Update helm test
This updates the Elasticsearch helm test to execute a clean on the
test index before attempting to create it, in cases where a
stranded test index may exist

Change-Id: I87533f94f6ea55b0b2f929543f8d3e75baa81bed
2018-12-12 12:43:13 -06:00
Steve Wilkerson
00f6a4a9c1 Elasticsearch: Remove default Curator action configuration
This removes the default Curator action configuration. As these
values will potentially be merged with any supplied overrides, it
could result in undesirable behavior. As a result, we should leave
the existing defaults commented out as a reference instead.

Change-Id: Idaf1dc8f3e476f1189058b69b841588a15deb7cd
2018-12-10 14:06:35 +00:00
Steve Wilkerson
4c29bafcbc Gates: Update fluent-logging/elasticsearch configurations
This updates the fluentd buffer output configurations to account
for the restraints of the jobs deploying fluentd. This also
renames the fluentd configuration key from td_agent to fluentd to
reflect the fact we're no longer deploying td-agent

This also updates the Elasticsearch default replicas and overrides
the replica counts in each Elasticsearch deployment to account for
resource constraints

Change-Id: I55dee410eced99c3e1645f7452e4306ad646e601
2018-10-19 17:30:08 +00:00
Steve Wilkerson
92717bdc72 Ceph: Remove fluentbit sidecars, mount hostpath for logs
This removes the fluentbit sidecars from the ceph-mon and ceph-osd
charts. Instead, we mount /var/log/ceph as a hostpath, and use the
fluentbit daemonset to target the mounted log files instead

This also updates the fluentd configuration to better handle the
correct configuration type for flush_interval (time vs int), as
well as updates the fluentd elasticsearch output values to help
address the gate failures resulting from the Elasticsearch bulk
endpoints failing

Change-Id: If3f2ff6371f267ed72379de25ff463079ba4cddc
2018-10-17 11:05:03 -05:00
Tin Lam
92e68d33ea Add network policy toolkit function
This patch set implements the helm toolkit function to generate a
kubernetes network policy manifest based on overrideable values.
This also adds a chart that shuts down all the ingress and egress
traffics in the namespace. This can be used to ensure the
whitelisted network policy works as intended.

Additionally, implementation is done for some infrastructure charts.

Change-Id: I78e87ef3276e948ae4dd2eb462b4b8012251c8c8
Co-Authored-By: Mike Pham <tp6510@att.com>
Signed-off-by: Tin Lam <tin@irrational.io>
2018-10-15 13:50:50 +00:00
Steve Wilkerson
c7cbb9f4dd Charts: Update heat image used for jobs and helm tests
This changes the image used for various jobs and helm tests in the
osh-infra charts. This replaces the kolla heat image with the loci
based heat image used for jobs and helm tests in openstack-helm in
order to drive consistency

Change-Id: Ie9deedadb7507282fe62723ec4641dd508040364
2018-10-11 14:47:58 -05:00
Zuul
e231a7c5fd Merge "Elasticsearch: Update log4j2 configuration settings" 2018-10-10 19:23:47 +00:00
Zuul
922d7d3d26 Merge "Charts: Update helm test pod templates" 2018-10-10 06:00:44 +00:00
Zuul
9f63330f0b Merge "Add missing labels to cronJobs" 2018-10-09 21:49:41 +00:00
Steve Wilkerson
bfa237d347 Charts: Update helm test pod templates
This updates the helm test pod templates in the charts with helm
tests defined. This change includes the addition of:

- Generate test pod cluster roles and role bindings
- Generate service accounts for test pods
- Add node selectors to the test pods
- Add service accounts to the test pods
- Addition of entrypoint container to the test pods
- Indentation fix for rabbitmq test pod template

Change-Id: I9a0dd8a1a87bfe5eaf1362e92b37bc004f9c2cdb
2018-10-09 21:00:00 +00:00
Mike Pham
0ad481d9c6 Add missing labels to cronJobs
This PS adds the missing labels so it is able to target by the network policy.

Change-Id: I5da7e0a9b17d00a5e98488780d72c77a93af999f
2018-10-09 13:58:17 -04:00
Steve Wilkerson
30f339a080 Elasticsearch: Add node selector to Curator cron job
This adds the node selector key and value configuration to the
Curator cron job for Elasticsearch, as it was previously omitted

Change-Id: Id702007fa827a1e1f90dee9b2a855e4197f4567c
2018-10-09 11:01:41 -05:00
Steve Wilkerson
1f4a890343 Elasticsearch: Update log4j2 configuration settings
This updates the configuration settings used for the log4j2
template for Elasticsearch. The previous settings weren't
compatible with the version of Elasticsearch currently being used
(5.6.4)

Change-Id: Id4b02ad022c46d599ae02ef77bb0f81f7e62c9e4
2018-10-09 08:01:39 -05:00
Steve Wilkerson
a084769410 Elasticsearch S3 repo
This ps adds the ability to use the ceph radosgw s3 api for
snapshot repositories. It removes the ability to use a RWM pvc, as
the radosgw solution provides a more robust approach for storing
index snapshots

Change-Id: Ie56ac41ccdc61bfadcac52b400cceb35403e9fae
2018-09-19 15:53:21 -05:00
Zuul
a98b14d541 Merge "Elasticsearch: Add ingress, remove node ports" 2018-09-19 02:57:09 +00:00
Pete Birley
bb3ff98d53 Add release uuid to pods and rc objects
This PS adds the ability to attach a release uuid to pods and rc
objects as desired. A follow up ps will add the ability to add arbitary
annotations to the same objects.

Change-Id: Iceedba457a03387f6fc44eb763a00fd57f9d84a5
Signed-off-by: Pete Birley <pete@port.direct>
2018-09-13 05:35:35 +00:00
Steve Wilkerson
cd88fc44fc Elasticsearch: Add ingress, remove node ports
This adds an ingress to the Elasticsearch chart, allowing for the
exposure of the Elasticsearch cluster externally if required.

This also removes the node ports from the data and discovery
services, as these ports should not be used beyond service
discovery by the elasticsearch nodes. It moves the node port for
the client service under the network.elasticsearch key to match
the network tree for the other services

Change-Id: Ia989eff87b8c9f112c697ae309bbb971dc699aa5
2018-09-04 14:19:13 +00:00
Steve Wilkerson
9a311475ba Charts: Use secrets for configs in chart
This updates the osh-infra charts to use a secret for their
configuration files instead of a configmap, allowing for the
storage of sensitive information

Change-Id: Ia32587162288df0b297c45fd43b55cef381cb064
2018-08-24 15:56:53 -05:00
Seungkyu Ahn
a430533e6a Quoting node_select_value in Ingress Controller
In most cases, the ingress controller's nodeSelector key and value
are "node-role.kubernetes.io/ingress" and "true".
Using quote to treat the nodeSelector value as a string.

Change-Id: Ie1745629b90795e4d888d85f35565e6d6350e09b
2018-08-01 02:39:05 +00:00
Zuul
9c7169e40e Merge "Resources: Fix erroneous resource definitions" 2018-07-30 18:29:10 +00:00
Steve Wilkerson
4f78e1f6fc Drive apache proxy configuration via values templates
This proposes defining the apache proxy hosts entirely via values
templates. While complicated on its face, this gives flexibility
by allowing the ability to define the desired authentication
mechanism via values templates. These options can range from
using http basic auth for development purposes to defining more
complex ldap configurations without a need to modify the chart
directly

Change-Id: Ief1b6890444ff90cc9c0ca872087af74836c0771
Signed-off-by: Pete Birley <pete@port.direct>
2018-07-30 07:52:26 -05:00
Steve Wilkerson
397eebf995 Resources: Fix erroneous resource definitions
This fixes the resource trees for the fluent-logging and
openstack-exporter charts to match the other charts. This
also fixes the elasticsearch master template to use the
correct indentation level for the resource template

Change-Id: Ic6ec270a880216daff10d1f22128c6377ebf9933
2018-07-27 16:35:37 -05:00
Steve Wilkerson
dc16a897d7 Add missing labels to helm test pods
This adds missing labels to the helm test pods in osh-infra

Change-Id: I618d9089bfde2d847411f5f876f0ff6afd9cce7f
2018-07-10 08:55:40 -05:00
Steve Wilkerson
cb7bf2c0b3 Add missing readiness probes to openstack-helm-infra charts
This adds missing readiness probes to the following charts in
openstack-helm-infra: elasticsearch, fluent-logging, kibana,
nagios, prometheus-kube-state-metrics, prometheus-node-exporter,
and prometheus-openstack-exporter

Change-Id: I6a2635b08667c31eadb1b05ba848c658935a17e5
2018-06-26 12:25:36 +00:00
Steve Wilkerson
2dd5bf0594 Update ordering of auth providers in apache reverse proxy
This updates the ordering of the basic auth providers in the
elasticsearch and nagios chart to check the file provider first
before going out to check the configured ldap server.

Change-Id: I47ff8a1c7b2cefa8425914c5d4d7a76aa8d43216
Signed-off-by: Steve Wilkerson <wilkers.steve@gmail.com>
2018-06-25 12:43:06 -05:00
Pete Birley
fa629cdbbd Daemonsets: Use current kubernetes daemonset api version
This PS moves to use the current ga version for kubernetes daemonsets,
additionally any remaining deployments that were using the
`extensions/v1beta1` have been updated to `apps/v1`.

Story: 2002205
Task: 21735

Change-Id: If9703162dc472af1e6096bf2b9062802fd5ce8ab
Signed-off-by: Pete Birley <pete@port.direct>
2018-06-13 21:53:18 +00:00
Pete Birley
b6a51fb57f Use current kubernetes API version
This PS moves to use the current API version for kubernetes rcs'
that were previously using `apps/v1beta1`.

Story: 2002205
Task: 21735

Change-Id: Icb4e7aa2392da6867427a58926be2da6f424bd56
Signed-off-by: Pete Birley <pete@port.direct>
2018-06-12 17:35:13 -05:00
Steve Wilkerson
f6fe167278 Elasticsearch: Update tests to clean up test data and index
This ps adds a function for cleaning up the test data used to
verify Elasticsearch is functioning properly. It removes the test
index created and populated with test data to ensure the resulting
elasticsearch cluster is clean and does not contain random data

Change-Id: Ibdeb90e3f3b6307bf16c68469556bef256ed2d78
2018-06-10 12:56:57 +00:00
Steve Wilkerson
de9c46bcfa Charts: Tidy up openstack-helm-infra charts
This moves the charts in openstack-helm-infra closer towards a
standard structure. It addresses multiple deviations, including:
missing resources for init containers, incorrect indents for
disabled resources in some charts, incorrect indents for volumes
and volumemounts added via values, missing resources for some
helm test templates, missing helm-toolkit image functions, and
moving the resource template declarations to be under the image
template declarations

Change-Id: I4834a5d476ef7fc69c5583caacc0229050f20a76
2018-05-21 12:58:22 -07:00
Steve Wilkerson
9e11fc11af Update resource tree for elasticsearch/kibana
This adds the entry for resources for the apache proxy running in
the elasticsearch client and kibana pods. This also fixes an
incorrect enabled flag for resources in the kibana chart

Change-Id: Ifcd33a680167d7debfae2c4d71bdcb693632fce9
2018-05-15 20:55:24 +00:00
Steve Wilkerson
3c692abd6e Add ldap support in elasticsearch/kibana apache proxies
This adds required configuration for enabling LDAP through
the apache proxy in the elasticsearch and kibana charts by
default

Change-Id: Iaff8f328ff50944ddad94ec86b1134ca73750176
2018-05-14 13:53:30 +00:00
Sean Eagan
f402171e42 Move to v0.3.1 of kubernetes-entrypoint
Move to v0.3.1 of kubernetes-entrypoint which has 2 breaking changes to
pod dependencies, and also adds support for depending on jobs via
labels.

Change-Id: I2bafc2153ddd46b3833b253a2e7950bccbccf8ed
2018-04-25 12:38:44 -05:00
Zuul
f059e9a6df Merge "Update curator auth config and actions" 2018-04-19 21:35:08 +00:00
Zuul
e36dfcd21d Merge "Add manifest for image_repo_sync job" 2018-04-19 20:55:15 +00:00
Steve Wilkerson
5a4d56d068 Update curator auth config and actions
Curator is unable to use environment variables for configuration
values if the configured option contains more than the env
variable.  In the case of the http_auth value (which expects
user:password), using ${USER}:${PASS} prevents curator from
successfully authenticating to elasticsearch.  This moves to
dynamically define these values in the configmap if the value is
empty

This also updates values for curators actions to target logstash-
indices for its actions

Change-Id: Id5b49171e00847432e4ab0cf4be60005b70c21e3
2018-04-19 10:01:10 -05:00
Steve Wilkerson
e166432a98 Add manifest for image_repo_sync job
This ps proposes adding a common template for the image_repo_sync
jobs for consumption by the charts

Change-Id: I48476d1e4fd94bd1b08b13b46983e3d999f8d8ca
2018-04-19 14:10:08 +00:00
Steve Wilkerson
ee7516f565 add elasticsearch, fluent-logging, grafana registry endpoints
This adds the local image registry endpoint to elasticsearch,
fluent-logging and grafana.  This endpoint was missing from the
values.yaml in those charts

Change-Id: I30dc1f0cab40ccf8a493e13f407e2f0d37af1eee
2018-04-19 01:12:47 +00:00