Elasticsearch by Elastic

Elasticsearch by Elastic

Installation for elastic-provided elasticsearch.


Elasticsearch Installation

Initial Configuration

Copy default fluentd-config (and edit as required)

cp ./seldon-deploy-install/prerequisites-setup/efk/fluentd-values.yaml fluentd-values.yaml

Elasticsearch

Install Elasticsearch using the following script

kubectl create namespace seldon-logs || echo "namespace seldon-logs exists"

helm upgrade --install elasticsearch elasticsearch \
    --version 7.6.0 \
    --namespace seldon-logs \
    --set service.type=ClusterIP \
    --set antiAffinity="soft" \
    --repo https://helm.elastic.co \
    --set image=docker.elastic.co/elasticsearch/elasticsearch-oss

kubectl rollout status statefulset/elasticsearch-master -n seldon-logs

Fluentd and Kibana

If not using auth (not by default) then set elasticsearch.auth.enabled to false in the fluentd values file.

Install fluentd and kibana using the following script

helm upgrade --install fluentd fluentd-elasticsearch \
    --version 8.0.0 \
    --namespace seldon-logs -f fluentd-values.yaml \
    --repo https://kiwigrid.github.io

helm upgrade --install kibana kibana \
    --version 7.6.0 \
    --namespace seldon-logs \
    --set service.type=ClusterIP \
    --repo https://helm.elastic.co \
    --set image=docker.elastic.co/kibana/kibana-oss

kubectl rollout status deployment/kibana-kibana -n seldon-logs

Configure Seldon Deploy

Following helm values needs to be set in deploy-values.yaml when using Elastisearch by Elastic:

requestLogger:
  create: true
  elasticsearch:
    host: elasticsearch-master.seldon-logs.svc.cluster.local
    port: "9200"
    protocol: http

elasticsearch:
  basicAuth: false
  url: http://elasticsearch-master.seldon-logs.svc.cluster.local:9200

Auth is not required for the elastic-provided version of elasticsearch. The elasticsearch.basicAuth option in the seldon deploy helm chart can be set to false, as illustrated above.

Authentication

If authentication is required to access your ElasticSearch cluster, you will need to configure your credentials so that Seldon can access it. To do this, you can provide your ElasticSearch user and password through a secret. By default, Seldon will look for a secret named elastic-credentials.

As an example, if we assume that ElasticSearch can be accessed using the admin / admin credentials, we could create the relevant secrets as:

ELASTIC_USER=admin
ELASTIC_PASSWORD=admin

kubectl create secret generic elastic-credentials -n seldon-logs \
  --from-literal=username="${ELASTIC_USER}" \
  --from-literal=password="${ELASTIC_PASSWORD}" \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret generic elastic-credentials -n seldon-system \
  --from-literal=username="${ELASTIC_USER}" \
  --from-literal=password="${ELASTIC_PASSWORD}" \
  --dry-run=client -o yaml | kubectl apply -f -

Configure EFK Ingress (Optional)

Kibana

It can be useful to access kibana’s UI without having to port-forward.

To expose kibana externally it needs to have its own path. This means a custom values file:

extraEnvs:
- name: SERVER_BASEPATH
  value: "/kibana"

That should be referenced with -f as an additional parameter on the previous helm install command.

Then create a following VirtualService for Kibana to enable its ingress

cat << EOF > kibana-vs.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: kibana
  namespace: seldon-logs
spec:
  gateways:
  - istio-system/seldon-gateway
  hosts:
  - '*'
  http:
  - match:
    - uri:
        prefix: /kibana/
    rewrite:
      uri: /
    route:
    - destination:
        host: kibana-kibana
        port:
          number: 5601
EOF

Kubectl apply -f kibana-vs.yaml
Last modified May 21, 2021