Request Logging

Request Logging with Elasticsearch

Install KNative Eventing

Knative Eventing is used for request logging. See Knative section for steps.

Knative eventing is recommended but see Running without KNative below if this represents a major barrier for you.

Install EFK Stack

We suggest using the OpenDistro version of elastic for a trial and provide instructions.

Other flavours are also available.

Fluentd is used for collecting application logs. We use the request logger component for storing the bodies of http requests to models and other data about those, as elastic documents.

Seldon Core and Deploy Configuration

For Seldon Core add to your core-values.yaml following options

executor:
  requestLogger:
    defaultEndpoint: "http://broker-ingress.knative-eventing.svc.cluster.local/seldon-logs/default"

For Seldon Deploy add to your deploy-values.yaml following options

requestLogger:
  create: true

If using opendistro, it should also have auth enabled, in line with the instructions in that section (referenced above).

Custom Request Logger

It’s possible for you to add your own custom request logger, with any custom logic you’d like to add. In order to do this, you need to make sure each Seldon Deployment points to the endpoint of your custom request logger.

For this, you will need to make sure you enable the Seldon Operator Environment variable for the logger. Prior to v1.2.x this was "EXECUTOR_REQUEST_LOGGER_DEFAULT_ENDPOINT_PREFIX" and then became "EXECUTOR_REQUEST_LOGGER_DEFAULT_ENDPOINT". It can be enabled through the core helm chart’s executor.requestLogger.defaultEndpoint.

Below we show an example of how you would do this for our non-knative default request logger

Running without KNative

It’s also possible to set up request logging without KNative dependency. For this, you will have to run a non-knative request logger, which you can trigger by running the configuration below.

Make sure that you edit the elasticsearch host variable below to point to the correct elasticsearch service address.

Important: for a normal install use the helm chart to install the logger. But that uses knative eventing. If you don’t want that then you can refer to the helm chart to create your own logger spec like the below.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: seldon-request-logger
  namespace: default
  labels:
    app: seldon-request-logger
spec:
  replicas: 2
  selector:
    matchLabels:
      app: seldon-request-logger
  template:
    metadata:
      labels:
        app: seldon-request-logger
    spec:
      containers:
        - name: user-container
          image: docker.io/seldonio/seldon-request-logger:1.6.0
          imagePullPolicy: Always
          env:
            - name: ELASTICSEARCH_HOST
              value: "elasticsearch-master.logs.svc.cluster.local"
            - name: ELASTICSEARCH_PORT
              value: "9200"
---
apiVersion: v1
kind: Service
metadata:
  name: seldon-request-logger
  namespace: default
spec:
  selector:
    app: seldon-request-logger
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080

For a secured elastic you would also set ELASTICSEARCH_PROTOCOL, ELASTICSEARCH_USER and ELASTICSEARCH_PASS.

Configure Seldon Core to use that endpoint

In order to make sure the Seldon Deployments are sending the requests to that endpoint, you will need to make sure you provide the request logger prefix. In this case you will need the following extra attributes in the Seldon Core values.yaml:

        - name: EXECUTOR_REQUEST_LOGGER_DEFAULT_ENDPOINT
          value: "http://seldon-request-logger."

It’s important that you make sure it’s on that format, which is http://<LOGGER_SERVICE>.. The Seldon Service Orchestrator will then add the namespace where the Seldon DEployment is running as a suffix.

Overriding Request Logger Endpoint for specific Seldon Deployment

Once you have created the request logger, now you have to make sure your deployments are pointing to the correct custom request logger. You can set up the custom request logger address by adding the following configuration to every Seldon Core SeldonDeployment file:

      logger:
        mode: all
        url: http://seldon-request-logger.default

The mode configuration can be set to request, response or all.

The url is where this should be logged to. There’s a similar example for KFServing.

Authentication on Elasticsearch

The Seldon Deploy helm values file has two options for connecting to a secured elastic.

One is token-based authentication. Use this if you have an auth token. This is used for openshift cluster logging flavour of elastic.

The other option is basic authentication.

Elastic can be configured with basic auth. Note this requires an xpack feature.

Similar then needs to be applied for kibana - note the env vars are not quite the same.

And fluentd.

For Deploy this would need secrets in the namespaces seldon-logs (containing elastic and req logger) and seldon-system (containing Deploy) as Deploy would need to speak to elastic using the secret.

This could look like:

kubectl create secret generic elastic-credentials -n seldon-logs \
  --from-literal=username="${ELASTIC_USER}" \
  --from-literal=password="${ELASTIC_PASSWORD}" \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret generic elastic-credentials -n seldon-system \
  --from-literal=username="${ELASTIC_USER}" \
  --from-literal=password="${ELASTIC_PASSWORD}" \
  --dry-run=client -o yaml | kubectl apply -f -
Last modified April 16, 2021