Request Logging

Request Logging with Elasticsearch

Note

Before starting the installation procedure, please download installation resources as explained here and make sure that all pre-requisites are satisfied.

This page also assumes that main Seldon components are installed.

Install KNative Eventing

KNATIVE_EVENTING_URL=https://github.com/knative/eventing/releases/download/
EVENTING_VERSION=v0.11.0

 kubectl apply --selector knative.dev/crd-install=true \
    -f ${KNATIVE_EVENTING_URL}/${EVENTING_VERSION}/eventing.yaml
 sleep 5
 kubectl apply -f ${KNATIVE_EVENTING_URL}/${EVENTING_VERSION}/eventing.yaml
 kubectl apply -f ${KNATIVE_EVENTING_URL}/${EVENTING_VERSION}/in-memory-channel.yaml

Install EFK Stack

Initial Configuration

Copy default fluentd-config (and edit as required)

cp ./seldon-deploy-install/prerequisites-setup/efk/fluentd-values.yaml fluentd-values.yaml

Prepare kibana-values.yaml file

image: docker.elastic.co/kibana/kibana-oss

service:
  type: ClusterIP

extraEnvs:
- name: SERVER_BASEPATH
  value: "/kibana"

Elasticsearch

Install Elasticsearch using the following script

kubectl create namespace seldon-logs || echo "namespace seldon-logs exists"

helm upgrade --install elasticsearch elasticsearch \
    --version 7.6.0 \
    --namespace seldon-logs \
    --set service.type=ClusterIP \
    --set antiAffinity="soft" \
    --repo https://helm.elastic.co \
    --set image=docker.elastic.co/elasticsearch/elasticsearch-oss

kubectl rollout status statefulset/elasticsearch-master -n seldon-logs

Fluentd and Kibana

Install fluentd and kibana using the following script

helm upgrade --install fluentd fluentd-elasticsearch \
    --version 8.0.0 \
    --namespace seldon-logs -f fluentd-values.yaml \
    --repo https://kiwigrid.github.io

helm upgrade --install kibana kibana \
    --version 7.6.0 \
    --namespace seldon-logs -f kibana-values.yaml \
    --repo https://helm.elastic.co

kubectl rollout status deployment/kibana-kibana -n seldon-logs

Configure EFK Ingress

Kibana

Create a following VirtualService for Kibana to enable its ingress

cat << EOF > kibana-vs.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: kibana
  namespace: seldon-logs
spec:
  gateways:
  - seldon-gateway.istio-system.svc.cluster.local
  hosts:
  - '*'
  http:
  - match:
    - uri:
        prefix: /kibana/
    rewrite:
      uri: /
    route:
    - destination:
        host: kibana-kibana
        port:
          number: 5601
EOF

Kubectl apply -f kibana-vs.yaml

Elasticsearch

Create a following VirtualService for Elasticsearch to enable its ingress

cat << EOF > elasticsearch-vs.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: elasticsearch
  namespace: seldon-logs
spec:
  gateways:
  - seldon-gateway.istio-system.svc.cluster.local
  hosts:
  - '*'
  http:
  - match:
    - uri:
        prefix: /elasticsearch/
    rewrite:
      uri: /
    route:
    - destination:
        host: elasticsearch-master
        port:
          number: 9200
EOF

Kubectl apply -f elasticsearch-vs.yaml

Configure KNative Event broker

Create knative event broker that will handle the logging.

kubectl delete broker -n seldon-logs default || true
kubectl label namespace seldon-logs knative-eventing-injection- --overwrite=true
kubectl label namespace seldon-logs knative-eventing-injection=enabled --overwrite=true

Seldon Core and Deploy Configuration

For Seldon Core add to your core-values.yaml following options

executor:
  requestLogger:
    defaultEndpoint: "http://default-broker.seldon-logs"

For Seldon Deploy add to your deploy-values.yaml following options

requestLogger:
  create: true

Custom Request Logger

It’s possible for you to add your own custom request logger, with any custom loggic you’d like to add. In order to do this, you need to make sure each Seldon Deployment points to the endpoint of your custom request logger.

For this, you will need to make sure you enable the Seldon Operator Environment variable for the logger. Prior to v1.2.x this was "EXECUTOR_REQUEST_LOGGER_DEFAULT_ENDPOINT_PREFIX" and then became "EXECUTOR_REQUEST_LOGGER_DEFAULT_ENDPOINT". It can be enabled through the core helm chart’s executor.requestLogger.defaultEndpoint.

Below we show an example of how you would do this for our non-knative default request logger

Running without KNative

It’s also possible to set up request logging without KNative dependency. For this, you will have to run a non-knative request logger, which you can trigger by running the configuration below.

Make sure that you edit the elasticsearch host variable below to point to the correct elasticsearch service address.

Important: you need to make sure that you are running a request logger in every namespace where you plan to deploy seldon core models

apiVersion: apps/v1
kind: Deployment
metadata:
  name: seldon-request-logger
  namespace: default
  labels:
    app: seldon-request-logger
spec:
  replicas: 2
  selector:
    matchLabels:
      app: seldon-request-logger
  template:
    metadata:
      labels:
        app: seldon-request-logger
    spec:
      containers:
        - name: user-container
          image: docker.io/seldonio/seldon-request-logger:0.2.0
          imagePullPolicy: Always
          env:
            - name: ELASTICSEARCH_HOST
              value: "elasticsearch-master.logs.svc.cluster.local"
            - name: ELASTICSEARCH_PORT
              value: "9200"
---
apiVersion: v1
kind: Service
metadata:
  name: seldon-request-logger
  namespace: default
spec:
  selector:
    app: seldon-request-logger
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080

Configure Seldon Core to use that endpoint

In order to make sure the Seldon Deployments are sending the requests to that endpoint, you will need to make sure you provide the request logger prefix. In this case you will need the following extra attributes in the Seldon Core values.yaml:

        - name: EXECUTOR_REQUEST_LOGGER_DEFAULT_ENDPOINT
          value: "http://seldon-request-logger."

It’s important that you make sure it’s on that format, which is http://<LOGGER_SERVICE>.. The Seldon Service Orchestrator will then add the namespace where the Seldon DEployment is running as a suffix.

Overriding Request Logger Endpoint for specific Seldon Deployment

Once you have created the request logger, now you have to make sure your deployments are pointing to the correct custom request logger. You can set up the custom request logger address by adding the following configuration to every Seldon Core SeldonDeployment file:

      logger:
        mode: all
        url: http://seldon-request-logger.default

The mode configuration can be set to request, response or all.

The url is where this should be logged to. There’s a similar example for KFServing.

Last modified September 14, 2020: add kfserving installation instructions (a73fd87)