We use analytics and cookies to understand site traffic. Information about your use of our site is shared with Google for that purpose.You can read our privacy policies and terms of use etc by clicking here.
VCS GitOps
This page provides a detailed overview of how to set up the GitOps functionality for Seldon Deploy. Setting up GitOps is highly recommended.
Note
Before starting the installation procedure, please download installation resources as explained here and make sure that all pre-requisites are satisfied.
This page also assumes that main Seldon components are installed.
Seldon Deploy leverages ArgoCD to implement GitOps in your cluster. You can read more about Seldon Deploy’s GitOps integration in the architecture and operation sections.
This documentation page will walk you through the different steps required to set up ArgoCD and GitOps in your cluster. Note that this guide assumes a fresh cluster without ArgoCD, thus it must only be treated as a general reference. Feel free to adapt these steps to your particular infrastructure.
Installation of ArgoCD
Since this guide assumes that you currently don’t have ArgoCD installed in your cluster, the first step will be on installing and configuring it. You can treat these steps as a quick way to get started with ArgoCD. Therefore, it is highly encouraged to check ArgoCD’s official documentation for further configuration.
To install and configure ArgoCD we will first install their official K8s manifest into our cluster.
For version v1.6.2
of ArgoCD, this can be done as follows:
ARGOCD_VERSION=v1.6.2
kubectl create namespace argocd || echo "namespace exist"
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/${ARGOCD_VERSION}/manifests/install.yaml
kubectl rollout status deployment/argocd-application-controller -n argocd
kubectl rollout status deployment/argocd-repo-server -n argocd
kubectl rollout status deployment/argocd-server -n argocd
kubectl rollout status deployment/argocd-redis -n argocd
kubectl rollout status deployment/argocd-dex-server -n argocd
Configuration of ArgoCD
Now that we have ArgoCD installed in our cluster, the next step will involve configuring it so that:
- The admin user credentials get changed.
- SSL termination is disabled in ArgoCD.
- ArgoCD gets exposed under the
/argocd/
path of our load balancer.
Note that these steps must be treated just as a guideline. Therefore, you may need to adapt them to your infrastructure. For more information on ArgoCD’s configuration, we suggest checking ArgoCD’s official documentation.
Admin user and password
Upon the initial installation of ArgoCD, it is recommended to change its user and password. This guideline will show how to change ArgoCD’s default admin user as an example. However, it’s also possible to hook ArgoCD to a centralised OIDC provider. This could be the same as Seldon Deploy’s OIDC provider, providing a Single Sign-On across your entire stack.
The following command will assume that we want to set the admin credentials as
admin
// $ARGOCDINITIALPASS
.
With this in mind, we can set those credentials as:
ARGOCDINITIALPASS=12341234
kubectl patch secret \
-n argocd argocd-secret \
-p '{"stringData": { "admin.password": "'$(htpasswd -bnBC 10 "" ${ARGOCDINITIALPASS} | tr -d ':\n')'"}}'
This command requires htpasswd
that can be installed on Ubuntu, Debian and Mint systemms with:
sudo apt install apache2-utils
SSL Termination
As a general choice, we recommend to handle SSL termination at the ingress / load balancer level. This simplifies the components setup within the cluster.
Following this approach, we will need to disable the SSL termination in ArgoCD.
Otherwise, ArgoCD will expect to receive SSL traffic by default.
To do this, we can ask ArgoCD to run in insecure
mode by running the command
below:
kubectl patch deploy argocd-server \
-n argocd \
-p '[{"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "--insecure"}]' \
--type json
Exposing ArgoCD through our ingress
There are different ways to access ArgoCD.
However, the most convenient is usually to expose it through our ingress layer
so that it can be accessed as a different subpath (e.g. /argocd/
).
This will make the ArgoCD API and UI reachable as
http(s)://<load-balancer-domain>/argocd/
.
This will need to be configured in a couple of places.
On one hand, we need to tell ArgoCD to expect its static assets to be served under the
/argocd/
path. This can be done by running the following command:kubectl patch deploy argocd-server \ -n argocd \ -p '[{"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "--rootpath"}, {"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "/argocd"}]' \ --type json
On the other hand, we will need to create a
VirtualService
that maps the/argocd/
path to ArgoCD’s service. Note that this assumes that we are using Istio as an ingress layer. To expose ArgoCD, it should be enough to apply the followingVirtualService
resource to our cluster:cat << EOF > ./argocd-vs.yaml apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: argocd-server namespace: argocd spec: gateways: - istio-system/seldon-gateway hosts: - '*' http: - match: - uri: prefix: /argocd/ route: - destination: host: argocd-server port: number: 80 EOF
If we assume that the above has been saved as a file called
argocd-vs.yaml
, we can then apply it as:
kubectl apply -f argocd-vs.yaml
Accessing ArgoCD UI and API
After configuring ArgoCD, we should now be able to reach ArgoCD under:
http(s)://<load-balancer-domain>/argocd/
In a standard setup, you should be able to obtain your load balancer domain by checking the Istio resources created in your cluster. You can do this as:
ISTIO_INGRESS=$(\
kubectl get svc -n istio-system istio-ingressgateway \
-o jsonpath='{.status.loadBalancer.ingress[0].ip}'\
)
ISTIO_INGRESS+=$(\
kubectl get svc -n istio-system istio-ingressgateway \
-o jsonpath='{.status.loadBalancer.ingress[0].hostname}'\
)
echo $ISTIO_INGRESS
Alternatively, if your ingress layer doesn’t expose an external IP address or
domain, it’s also possible to port-forward a local port into ArgoCD.
For example, to map your local localhost:8080
(or localhost:8080/argocd/
if you defined the rootpath
above) domain to ArgoCD, you can run:
kubectl port-forward -n argocd svc/argocd-server 8080:80
Once we know our load balancer domain, we can then access ArgoCD through its UI
or API.
For the latter, we will leverage ArgoCD’s CLI, called argocd
.
Below, you can find instructions on how to set up and access both of them.
ArgoCD ships with a CLI tool called argocd
.
This tool allows you to interact with ArgoCD from the command line.
The rest of this guide will provide command examples that can be used to
configure ArgoCD using their argocd
tool.
However, it’s also possible to perform the same actions from the ArgoCD UI.
To use the argocd
CLI, we will need to:
Install the
argocd
CLI locally. This can be achieved by running the following commands:ARGOCD_VERSION=v1.6.2 wget -q -O argocd "https://github.com/argoproj/argo-cd/releases/download/${ARGOCD_VERSION}/argocd-linux-amd64" chmod +x ./argocd
Authenticate the CLI against our ArgoCD instance. Assuming that our load balancer domain can be found under
$ISTIO_INGRESS
, it should be enough to run the command below../argocd login \ "$ISTIO_INGRESS:80" \ --username admin \ --password ${ARGOCDINITIALPASS} \ --insecure \ --grpc-web-root-path /argocd
ArgoCD exposes a UI that can be used to configure your GitOps repository.
Assuming that you have followed the ArgoCD setup guidelines
above, this UI should be accessible under the
/argocd/
path of your load balancer domain.
After accessing the UI, you should be prompted with a login form asking for the
admin credentials.
You should be able to access this UI by using ArgoCD’s user / password
combination (admin
and $ARGOCDINITIALPASS
by default, respectively).
You can learn more about how to navigate ArgoCD’s UI in their documentation.
GitOps for logging components
The request logger is an infrastructure component that runs in the
centralised logging namespace (named as seldon-logs
by default).
Alongside this, Triggers can also be created for detectors and these tied to particular models. Thus is good to
back them through a GitOps environment.
With this goal in mind, it’s encouraged to create a system-level GitOps
environment to track these infrastructure resources.
To do this, you can follow the instructions to create a new GitOps
environment, using the seldon-logs
namespace as
target.
Note that if you’ve configured Seldon Deploy to use a different centralised
namespace for logging, you’ll need to point to that namespace instead.
Once the logging GitOps environment is created, Seldon Deploy should be able to pick it up and use it without any further configuration.
Creating a new GitOps environment
Now that ArgoCD has been installed and configured, we can then proceed to create a new GitOps environment. This environment will provide a Kubernetes namespace to deploy our machine learning models, which is backed by a GitOps repository.
This section will walk you through how to create a new GitOps-tracked
environment for a namespace with name $namespace
.
The GitOps repository which will act as the source of truth for this
environment will exist under $GIT_REPO_URL
.
Note that $GIT_REPO_URL
should not contain the http
or https
protocol
schema.
Git Repository
The only requirements for the Git repository are:
- It contains a non-empty folder named as the namespace (i.e.
./$namespace
). It is accessible from within the cluster. This can be verified using this one-liner command:
kubectl run --quiet=true -it --rm git-clone --image=radial/busyboxplus:git --restart=Never -- git clone https://$GIT_USER:$GIT_TOKEN@$GIT_REPO_URL
ArgoCD Resources
In order for ArgoCD to keep track of our new environment, it’s necessary to configure a Project and an Application. These are concepts native to ArgoCD. We suggest checking the ArgoCD official documentation for more information about them.
ArgoCD projects and applications can be created via the ArgoCD UI, argocd
CLI
tool and through CRDs.
As
declarative
approach guarantees best reproducibility this is the approach that we
recommend.
We create ArgoCD project by creating the following AppProject
resources named argocd-project.yaml
cat << EOF > ./argocd-project.yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: seldon
namespace: argocd
spec:
description: Seldon Deploy Project
sourceRepos:
- https://${GIT_REPO_URL}
destinations:
- namespace: ${namespace}
server: https://kubernetes.default.svc
clusterResourceWhitelist:
- group: '*'
kind: '*'
roles:
- name: seldon-admin
policies:
- p, proj:seldon:seldon-admin, applications, get, seldon/*, allow
- p, proj:seldon:seldon-admin, applications, create, seldon/*, allow
- p, proj:seldon:seldon-admin, applications, update, seldon/*, allow
- p, proj:seldon:seldon-admin, applications, delete, seldon/*, allow
- p, proj:seldon:seldon-admin, applications, sync, seldon/*, allow
EOF
kubectl apply -f argocd-project.yaml
Similarly we create the ArgoCD Application
resource:
ARGO_APP_NAME=seldon-gitops-"${namespace}"
cat << EOF > ./argocd-application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: ${ARGO_APP_NAME}
namespace: argocd
spec:
project: seldon
destination:
namespace: ${namespace}
server: https://kubernetes.default.svc
source:
directory:
recurse: true
path: ${namespace}
repoURL: https://${GIT_REPO_URL}
syncPolicy:
automated: {}
EOF
kubectl apply -f argocd-application.yaml
Kubernetes Model Namespace
To use a namespace with GitOps, you will need to annotate it with the Git
repository URL (using key git-repo
) and enable it for GitOps access by adding
the git-repo
label.
As an example, to enable a namespace named $namespace
, you could run the
following commands:
kubectl create ns $namespace || echo "namespace $namespace already exists"
kubectl label ns $namespace seldon.restricted=false --overwrite=true
kubectl label namespace $namespace seldon.gitops=enabled --overwrite=true
kubectl annotate namespace $namespace git-repo="https://${GIT_REPO_URL}" --overwrite=true
Note
If your ArgoCD application does not follow seldon-gitops-${namespace}
naming convention you can label the namespace accordingly
kubectl label namespace $namespace argocdapp=${ARGO_APP_NAME} --overwrite=true
If you intend to use Batch jobs on the namespace then you’ll need additional config. You can visit the Argo section of the docs for more details.
Configuring Git Credentials
The last required step is to provide the relevant Git credentials to access our GitOps repository. Note that these credentials need to be set at both the ArgoCD and Seldon Deploy level.
ArgoCD
You can configure your GitOps repository access in ArgoCD using either SSH or
user / password credentials.
These credentials can be provided using the argocd
CLI tool, or directly
through the ArgoCD UI.
Assuming that the argocd
CLI tool is already
authenticated, and that $GIT_SSH_PATH
and $GIT_TOKEN
represent our GitOps repository user and password, we should be able to run:
./argocd repo add "ssh://${GIT_REPO_URL}" \
--ssh-private-key-path ${GIT_SSH_PATH} \
--upsert
Assuming that the argocd
CLI tool has already been
authenticated, and that ${GIT_USER}
and ${GIT_TOKEN}
represent our GitOps repository user and password, we should be able to run:
./argocd repo add "https://${GIT_REPO_URL}" \
--username ${GIT_USER} \
--password ${GIT_TOKEN} \
--upsert
ArgoCD exposes a UI that can be used to configure your GitOps repository.
Assuming that you have followed the ArgoCD setup guidelines
above, this UI should be accessible under the /argocd
path.
You should be able to access this UI by using ArgoCD’s user / password
combination (admin
and 12341234
by default, respectively).
You can learn more about how to configure repositories using ArgoCD’s UI in their documentation.
Seldon Deploy
Seldon Deploy also requires access to the repository. This access will be used to monitor the status of the current resources in the cluster, as well as to create new ones.
To configure our Git credentials in Seldon Deploy, we will follow these steps:
- Create a Kubernetes secret containing our credentials, either as a SSH key or a User / Password combination. This secret can have any arbitrary name, but must live in the same namespace as Seldon Deploy.
If we assume that our private key is present under $GIT_SSH_PATH
, we can
create the credentials secret as:
kubectl create secret generic git-creds -n seldon-system \
--from-file ${GIT_SSH_PATH} \
--from-file ${GIT_KNOWN_HOSTS_PATH} \
--from-literal=passphrase="${GIT_SSHKEY_PASSPHRASE}" \
--from-literal=username="${GIT_USER}" \
--from-literal=email="${GIT_EMAIL}" \
--dry-run=client -o yaml | kubectl apply -f -
Note that the passphrase
field can be left empty if they SSH key doesn’t
have a passphrase.
We can create the credentials secret using a User / Password combination (or User / Personal Access Token) as:
kubectl create secret generic git-creds -n seldon-system \
--from-literal=username="${GIT_USER}" \
--from-literal=token="${GIT_TOKEN}" \
--from-literal=email="${GIT_EMAIL}" \
--dry-run=client -o yaml | kubectl apply -f -
Update Seldon Deploy’s configuration to point to our newly created secret. In particular, we will need to modify the
gitops
section of the values of the Seldon Deploy Helm chart. Here, we will need to set thegitops.argocd.enabled
flag totrue
, and we will point thegitops.git.secret
field to the right secret name (e.g.git-creds
if we’ve followed the commands above).gitops: git: secret: git-creds argocd: enabled: true
(Optional) Webhooks Configuration
By default, the resource synchronisation against the GitOps repository will happen on a poll basis. That is, Seldon Deploy will check the repo periodically for updates.
The main caveat of this approach is that there may be a small delay between taking an action and seeing it reflected in the cluster / UI. To work around this, you can configure a set of webhooks in the cluster in the that will get pinged everytime there is a change in the cluster. Note that this is an optional step, and its feasibility may depend on your networking infrastructure and your Git provider.
Note
If webhooks are not arriving at its destination it may be helpful to enable webhook’s history.
It may be necessary to also select Skip certificate verification
option.
You can find instructions below on how to set up these webhooks for common Git providers
First we need to configure the webhook that will be sending updates to the Seldon Deploy endpoint:
whIp=$(kubectl get service seldon-deploy-webhook -o jsonpath='{.status.loadBalancer.ingress[*].ip}')
curl -u ${GIT_USER}:${GIT_TOKEN} \
-v -H "Content-Type: application/json" \
-d '
{
"name": "web",
"active": true,
"events": ["*"],
"config": {
"url": "http://${whIp}/api/git-webhook",
"content_type": "json"
}
}' \
https://${GIT_API}/repos/${GIT_USER}/${REPONAME}/hooks
Then we set up the webhook that will be sending the updates to ArgoCD.
curl -u ${GIT_USER}:${GIT_TOKEN} \
-v -H "Content-Type: application/json" \
-d '\
{
"name": "web",
"active": true,
"events": ["*"],
"config": {
"url": "https://${ARGOCDURL}/api/webhook",
"content_type": "json",
"secret": "${ARGOCDINITIALPASS}",
"insecure_ssl": "1"
}
}' \
https://${GIT_API}/repos/${GIT_USER}/${REPONAME}/hooks
First we need to configure the webhook that will be sending updates to the Seldon Deploy endpoint:
whIp=$(kubectl get service seldon-deploy-webhook -o jsonpath='{.status.loadBalancer.ingress[*].ip}')
curl -u ${GIT_USER}:${GIT_TOKEN} \
-v -H "Content-Type: application/json" \
-d '
{
"description": "web",
"events": ["repo:push"],
"url": "'"http://${whIp}/api/git-webhook"'",
"active": true
}' \
https://${GIT_API}/2.0/repositories/${GIT_USER}/${REPONAME}/hooks
Then we set up the webhook that will be sending the updates to ArgoCD.
curl -u ${GIT_USER}:${GIT_TOKEN} \
-v -H "Content-Type: application/json" \
-d '
{
"description": "web",
"events": ["repo:push"],
"url": "'"https://${ARGOCDURL}/api/webhook"'",
"active": true
}' \
https://${GIT_API}/2.0/repositories/${GIT_USER}/${REPONAME}/hooks
Troubleshooting
You should see one application per gitops namespace when running the below:
kubectl get application -n argocd
You can check the status of an argocd app with kubectl get application -n argocd seldon-gitops-<namespace_name> -o yaml
.
The argocd UI (described above) can also be used to inspect applications. You can trigger a manual sync from there. If a particular application is missing or in error try running its setup again in order to debug.
Example Script for Namespace Setup
Download installation resources as explained here
Copy the sd-setup directory containing gitops-setup.sh
cp -r ./seldon-deploy-install/sd-setup/ sd-setup/
The gitops-setup.sh is an example script for setting up gitops in the cluster by installing argocd and configuring namespaces. It can be used for reference.
See the trial installation docs for more on how to run that script. More context is provided in Namespace Setup