How to upgrade Kubeflow from 1.7 to 1.7 Patch 1
Charmed Kubeflow (CKF) 1.7 Patch 1 is a patch release that includes bug fixes and feature enhancements, documented in the release notes. This guide shows how to upgrade from CKF 1.7 to 1.7 Patch 1.
- An active and idle Charmed Kubeflow 1.7 deployment. This requires all charms in the bundle to be in that state.
- Access to the dashboard of the existing Charmed Kubeflow 1.7 deployment.
- Admin access to a Kubernetes cluster where the existing Charmed Kubeflow 1.7 is deployed.
- Do you need an upgrade?
- Before upgrade
- Upgrade Istio
- Before charms upgrade
- Upgrade charms
- Verify upgrade
The Patch 1 release touched only a subset of charms in the Charmed Kubeflow bundle. Compare the revisions of your deployed charms to the revisions listed below. If your deployed charms have a smaller revision, then proceed with the upgrade.
Before upgrading Charmed Kubeflow it is recommended to do the following:
- Stop all Notebooks.
- Review any important data that needs to be backed up and perform backup procedures according to the policies of your organisation.
- Record all charm versions in existing Charmed Kubeflow deployment.
All upgrade steps should be performed in the
kubeflow model. If you haven’t already, switch to the
# switch to kubeflow model juju switch kubeflow
It is assumed that the deployed
istio-ingressgateway versions alongside Charmed Kubeflow 1.7 are 1.16.
- Remove the
istio-ingressgatewayapplication and the corresponding relation with
# remove relation and istio-ingressgateway application juju remove-relation istio-pilot istio-ingressgateway juju remove-application istio-ingressgateway
- Run the following command to ensure that the
istio-ingressgatewayapplication and all related resources are properly removed. If removal was successful, the command should succeed (return
juju show-application istio-ingressgateway 2> >(grep -q "not found" && echo $?) kubectl -n kubeflow get deploy istio-ingressgateway-workload 2> >(grep -q "NotFound" && echo $?)
Troubleshooting of removal of `istio-ingressgateway` application
WARNING: Removing application using
--force option should be the last resort. There could be potential stability issues if application is not shutdown cleanly.
If required, remove
istio-ingressgateway application with
--force option and remove
juju remove-application --force istio-ingressgateway kubectl -n kubeflow delete deploy istio-ingressgateway-workload
juju refresh istio-pilot --channel 1.16/stable
Troubleshooting of Istio upgrade
Refer to this document for troubleshooting tips.
istio-ingressgatewayand add a relation between
# deploy istio-ingressgateway juju deploy istio-gateway --channel 1.16/stable --trust --config kind=ingress istio-ingressgateway juju relate istio-pilot istio-ingressgateway
Before charms can be upgraded, you need to enable trust on deployed charms (required).
Because of changes in the charm code, some charms in Charmed Kubeflow 1.7 have to be trusted by juju before the upgrade.
WARNING: Please note that if you do not execute
juju trust for these charms, you may encounter authorization errors. If that is the case, please refer to the Troubleshooting guide.
# enable trust on charms juju trust katib-db-manager --scope=cluster juju trust kfp-api --scope=cluster
To upgrade Charmed Kubeflow each charm needs to be refreshed. It is recommended to wait for each charm to finish its upgrade before proceeding with the next.
Depending on the original deployment of Charmed Kuberflow version 1.7, the refresh command will report that charm is up-to-date which indicates that there is no need to upgrade that particular charm.
During the upgrade some charms can temporarily go into
blocked state, but they should go
active after a while.
# upgrade charms juju refresh admission-webhook --channel 1.7/stable juju refresh katib-controller --channel 0.15/stable juju refresh katib-db-manager --channel 0.15/stable juju refresh katib-ui --channel 0.15/stable juju refresh kfp-api --channel 2.0/stable juju refresh kfp-persistence --channel 2.0/stable juju refresh kfp-profile-controller --channel 2.0/stable juju refresh kfp-schedwf --channel 2.0/stable juju refresh kfp-ui --channel 2.0/stable juju refresh kfp-viewer --channel 2.0/stable juju refresh kfp-viz --channel 2.0/stable juju refresh knative-eventing --channel 1.8/stable juju refresh knative-operator --channel 1.8/stable juju refresh knative-serving --channel 1.8/stable juju refresh kserve-controller --channel 0.10/stable juju refresh seldon-controller-manager --channel 1.15/stable juju relate istio-pilot:gateway-info kserve-controller:ingress-gateway
Troubleshooting charm upgrade
If a charm fails to upgrade or is stuck in a
maintenance state for a long time it is possible to recover by running the refresh command with the version that was there prior to deployment, i.e. downgrade the charm. After that, repeat the upgrade.
You can verify the progress of the upgrade by running
watch -c juju status --color
when all services are in
idle state then the upgrade should be finished.
Last updated 10 days ago.