Upgrading

Upgrading Seldon Deploy to 1.3.0

• 1.3.0 introduces improvement to the Batch Processing of requests - generated Argo workflows now use storage initializer mechanism for pulling in input file and uploading output file. The default storage initializer is seldonio/rclone-storage-initializer which requires adjustment of credential secret. It is also possible to fallback to kfserving/storage-initializer that will work with previous secret format. Please consult our documentation for more details.
• The request logger now looks up metadata from seldon deploy’s metadata service. For configuring this:
  • Metadata is not required. If you have metadata.pg.enabled false in your values file then you don’t need to configure anything for request logger metadata lookup.
  • If you are using metadata then adding auth from the request logger to deploy enables the prediction schema for a model to be found and used to enrich prediction data and enrich monitoring.
  • Enabling auth from request logger to deploy currently requires a user and the password grant flow. If your identity provider doesn’t support this then you can wait for a future release to get more options (please get in touch about preferred option).
  • It’s ok to disable the metadata lookup from the request logger. It will still work, you just won’t be able to enrich logged predictions with the schema (e.g. category names).
    • To disable any metadata lookup, either set metadata.pg.enabled to false in the values file (though this also disables metadata) or set requestLogger.deployHost and requestLogger.authSecret to empty
  • See the request logger page for details of the auth setup

Upgrading Seldon Deploy to 1.2.0

• 1.2.0 introduces the Model Metadata storage to Seldon Deploy. To enable/disable the model metadata storage functionality with all required prerequisites follow the instructions. The quick overview of the changes is:
  • If you do not want the model metadata functionality disable it by setting metadata.pg.enabled in the Seldon Deploy helm chart to false. Not recommended since all model metadata related features will be unavailable.
  • Install a self-hosted or managed PostgreSQL solution (instructions).
  • Configure the connection details to the PostgreSQL instance by adding a metadata-postgres secret in the Seldon Deploy namespace (instructions).
  • Enable the model metadata storage by setting metadata.pg.enabled in the Seldon Deploy helm chart to true (instructions).
• Detector components for outlier detection, drift detection and metrics are now moving out of the seldon-logs namespace and into the user namespaces. To handle this
  • The easiest way to migrate is to delete any existing detectors from the UI and recreate them after upgrade.
  • To see what detectors you have, do kubectl get ksvc -n seldon-logs. Also check kubectl get trigger -n seldon-logs.
  • Detectors will have -outlier, -drift or -metrics in their names. Each will have a ksvc and a corresponding trigger. The name and namespace of the model they are for will also be in the name.
  • If you find any issues while carrying out this process, please contact Seldon.
• The ElasticSearch integration will now enable basic authentication by default.
  • This is controlled by the elasticsearch.basicAuth flag in the Seldon Deploy Helm chart.
  • To disable it (and revert to the default behaviour of <= 1.1.0), you can set its value as elasticsearch.basicAuth=false.
  • To keep it enabled and configure ElasticSearch’s user / password credentials, please check the ElasticSearch integration setup guide.
• The Seldon Core version is now 1.7.0. See the Getting Started > Production Installation section for installation. A helm upgrade --install can be used to upgrade.
• The demo drift detector has changed from gs://seldon-models/alibi-detect/cd/ks/cifar10-0_4_3 to gs://seldon-models/alibi-detect/cd/ks/cifar10-0_4_4. This is because the alibi version in Seldon Core is upgraded.
  

Upgrading Seldon Deploy to 1.1.0

• The helm values file now has an option to set user and password for basic auth to Elastic. For more information, see the request logger documentation.
• The recommended prometheus settings have now changed to improve performance. We suggest uninstalling the prometheus helm chart and installing with the below changes.
  • See the metrics page for the full helm values.
  • The parts that have changed are:
    • prometheus.server.livenessProbePeriodSeconds is now 30
    • prometheus.server.extraArgs has been added with entries query.max-concurrency: 400 and storage.remote.read-concurrent-limit: 30
    • prometheus.server.resources.limits now has cpu: 2 and memory: 4Gi
    • prometheus.server.resources.requests now has cpu: 800m and memory: 1Gi
      
  • Remove analytics with helm delete -n seldon-system seldon-core-analytics.
  • Install again using the instructions

• The recommended seldon deploy helm values have changed slightly to improve performance. Set the following:

  • resources.limits should now have cpu: 800m and memory: 800Mi
    

• A further change to the seldon deploy helm values is the addition of a configuration for the runAsUser for knative pods started by deploy:

  defaultUserID: "8888"

Upgrading Seldon Deploy to 1.0.0

• If you are upgrading by more than one version, please see previous notes.
• The self-install trial customers, the file dockercreds.txt is now called sdconfig.txt (anyone using Product Installation can ignore this).
  • The variable KUBEFLOW_USER_EMAIL has become SD_USER_EMAIL and KUBEFLOW_PASSWORD has become SD_PASSWORD
• New components have been introduced for Batch support. Argo and minio. See Getting Started > Production Installation
  • Note the new Batch component requires per-namespace config for any Batch-enabled namespaces. See Getting Started > Production Installation section on Argo and refer to references to .
• For upgrading Seldon Deploy specifically
  • The requestLogger.image in the helm values file is now docker.io/seldonio/seldon-request-logger:1.5.0
  • The alibidetect.image in the helm values is now seldonio/alibi-detect-server:1.5.0.
  • There’s a new entry in the batchjobs section of the helm values - serviceAccount: "workflow"
  • The Seldon Core version is now 1.5.0. See the Getting Started > Production Installation section for installation.
  • The KFServing version is 0.4.1.
    

Upgrading Seldon Deploy to 0.9.0

• Seldon Deploy 0.9.0 uses knative 0.18. To upgrade this, see the Upgrading Knative section.
• For upgrading Seldon Deploy specifically
  
  • A license is required at app install/startup time. Contact Seldon for a license.
  • Seldon Core Analytics or Prometheus needs to be upgraded. Prometheus recording rules have been introduced. See Getting Started > Production Installation > Metrics Monitoring
  • If Seldon Core Analytics is used, upgrade with using seldon-deploy-install/prerequisites-setup/prometheus/seldon-core-analytics.sh from the scripts extracted in Getting Started > Download Installation Resources.
  • Upgrading Seldon Core Analytics will reset prometheus data.
  • The requestLogger.image in the helm values file is now docker.io/seldonio/seldon-request-logger:1.3.0-json
  • The requestLogger.trigger in the helm values now has apiVersion “eventing.knative.dev/v1” and broker “default”.
  • The alibidetect.image in the helm values is now seldonio/alibi-detect-server:1.4.0.
  • There is a new batchjobs section in the helm values.

Upgrading Seldon Deploy to 0.8.2

• In installations that have restricted external access (eg. air gapped), the following image will need to be downloaded:
  • docker.io/seldonio/seldon-request-logger:1.3.0-json

Upgrading Seldon Deploy to 0.8.0

• The sd-install script is now called sd-install-default. Or for a kubeflow install there’s sd-install-kubeflow.

• The helm values file now has a single gitops section. This involves the following changes.

  • The recommended way to configure git credentials is in a git secret, though a token parameter to the chart can still be used.

  • The sd-install-default script contains an example of this. It is:

    kubectl create secret generic git-creds -n seldon-system --from-file ~/.ssh/id_rsa --from-file ~/.ssh/known_hosts --from-literal=passphrase="$GIT_SSHKEY_PASSPHRASE" --from-literal=username="$GIT_USER" --from-literal=token="$GIT_TOKEN" --from-literal=email="$GIT_EMAIL"  --dry-run -o yaml | kubectl apply -f -

  • skipVerifyGit, gitwebhook, GITOPS_FORMAT and argocd config have all moved from github to the new gitops section in helm values file, see documentation.

  • if your ArgoCD Application does not follow naming pattern seldon-gitops-${namespace}, please, specify the ArgoCD application using namespace labels, see documentation.

• The contents of the helm chart have also changed. In particular:

  • The istio virtual service no longer does path rewriting.
  • Permission is now required to watch argocd applications.
  • The liveness and readiness probes are now on /seldon-deploy/api/status

Upgrading Seldon Deploy to 0.7.0

• The deploy helm chart now contains an option to install a request logger in the seldon-logs namespace. This is on by default. Previously loggers were per-namespace. To use a single request logger installed through the chart, ensure the seldon-logs namespace is labelled for knative-evenint and that seldon core is v1.2.1 and has executor.requestLogger.defaultEndpoint http://default-broker.seldon-logs. For KfServing, if using it, the default broker will for now continue to be in the current namespace unless set at the point of creating the model deployment.
• The helm values file has seen some changes.
  • There’s now a request logger section for installing the request logger to a dedicated namespace, if desired.
  • The istioIngress section has been renamed ingressGateway
  • There are extra options in the prometheus section for for using a secured prometheus (assumes not secure by default, as before).
  • There are extra options in the elasticsearch section for for using a secured prometheus (assumes not secure by default, as before).