Back up control plane

This guide describes how to back up and restore the Charmed Kubeflow (CKF) control plane data to a compatible S3 storage.

It is expected that these steps are followed all at once, backing up all databases, pipelines MinIO bucket, and ML Metadata database at the same time. Failing to do so may result in data loss.

Running Kubeflow pipelines and Katib experiments can affect the outcome of the backup, please make sure all pipelines and experiments are stopped and no other processes are calling them, such as Jupyter Notebooks.

User workloads in user namespaces are not backed up.

Requirements

Access to a S3 storage used for the backup data - only AWS S3 and S3 RadosGW are supported.
Admin access to the Kubernetes cluster where CKF is deployed.
Juju admin access to the kubeflow model.
yq binary.
Ensure the local storage is big enough to back up the data.

Configure `rclone`

rclone is a tool that allows file management in cloud storage. This tool will be used for backing up several files throughout this guide and it can be installed as a snap:

sudo snap install rclone

Connect to a shared S3 storage

Configure rclone to connect to the shared S3 storage. The following can be used as reference:

[remote-s3]
type = s3
provider = AWS
env_auth = true
access_key_id = ...
secret_access_key = ...
region = eu-central-1
acl = private
server_side_encryption = AES256

You can check where this configuration file is located with rclone config file.

Save the name of the S3 remote in an ENV variable:

RCLONE_S3_REMOTE=remote-s3

Connect to CKF MinIO

The following steps require an accessible MinIO endpoint, which can be done port forwarding the minio service:

kubectl port-forward -n kubeflow svc/minio 9000:9000

Get minio’s secret-key value:

juju show-unit kfp-ui/0 \
        | yq '.kfp-ui/0.relation-info.[] | select (.endpoint == "object-storage") | .application-data.data' \
        | yq '.secret-key'

Get minio’s access-key:

juju config minio access-key

Configure rclone to connect to CKF MinIO. The following can be used as reference:

[minio-ckf]
type = s3
provider = Minio
access_key_id = minio
secret_access_key = ...
endpoint = http://localhost:9000
acl = private

Save the name of the MinIO remote in an ENV variable:

RCLONE_MINIO_CKF_REMOTE=minio-ckf

Back up CKF databases to S3 storage

CKF uses katib-db and kfp-db as databases for Katib and Kubeflow pipelines respectively.

Deploy and configure the s3-integrator to connect to the shared S3 storage.

See S3 AWS and S3 Radowsg configuration guides for this step.

Scale up kfp-db and katib-db.

This step avoids the Primary database from becoming unavailable during backup.

juju scale-application kfp-db 2
juju scale-application katib-db 2

Create a backup for each database.

Replace mysql-k8s with the name of the database you intend to create a backup for in the commands from that guide.

Back up ML Metadata using `sqlite3`

The mlmd charm uses a SQLite database to store ML metadata generated from Kubeflow pipelines.

Install the required tools inside the application container:

This step expects the mlmd application container to have Internet access.

# MLMD > 1.14, CKF 1.9
MLMD_POD="mlmd-0"
MLMD_CONTAINER="mlmd-grpc-server"

# MLMD 1.14, CKF 1.8
MLMD_POD="mlmd-0"
MLMD_CONTAINER="mlmd"

kubectl exec -n kubeflow $MLMD_POD -c $MLMD_CONTAINER -- \
    /bin/bash -c "apt update && apt install sqlite3 -y"

Scale down kfp-metadata-writer. This is done to prevent any additional writes to mlmd.

juju scale-application kfp-metadata-writer 0

Perform a database backup.

This moves all the database contents into a compressed text file inside the mlmd-0 container:

MLMD_BACKUP=mlmd-$(date -d "today" +"%Y-%m-%d-%H-%M").dump.gz

kubectl exec -n kubeflow $MLMD_POD -c $MLMD_CONTAINER -- \
	/bin/bash -c \
	"sqlite3 /data/mlmd.db .dump | gzip -c >/tmp/$MLMD_BACKUP"

Copy the backup file to local storage:

kubectl cp -n kubeflow -c $MLMD_CONTAINER \
	$MLMD_POD:/tmp/$MLMD_BACKUP \
	./$MLMD_BACKUP

Copy the mlmd backup data to the S3 storage:

S3_BUCKET=backup-bucket-2024
RCLONE_S3_REMOTE=remote-s3
RCLONE_BWIDTH_LIMIT=20M

rclone --size-only copy \
	--bwlimit $RCLONE_BWIDTH_LIMIT \
	./$MLMD_BACKUP \
	$RCLONE_S3_REMOTE:$S3_BUCKET

Optionally, you can remove the mlmd data from your local machine:

rm -rf $MLMD_BACKUP

Scale up kfp-metadata-writer:

juju scale-application kfp-metadata-writer 1

Back up `mlpipeline` MinIO bucket

Sync all files from minio to the shared S3 storage:

S3_BUCKET=backup-bucket-2024
RCLONE_S3_REMOTE=remote-s3
RCLONE_BWIDTH_LIMIT=20M

rclone --size-only sync \
	--bwlimit $RCLONE_BWIDTH_LIMIT \
	$RCLONE_MINIO_REMOTE:mlpipeline \
	$RCLONE_S3_REMOTE:$S3_BUCKET/mlpipeline

Back up ML Metadata with `kubectl`

You can also perform the backup using kubectl.

Scale down kfp-metadata-writer. This is done to prevent any additional writes to mlmd:

juju scale-application kfp-metadata-writer 0

Copy the backup file to local storage:

# MLMD > 1.14, CKF 1.9
MLMD_POD="mlmd-0"
MLMD_CONTAINER="mlmd-grpc-server"

# MLMD 1.14, CKF 1.8
MLMD_POD="mlmd-0"
MLMD_CONTAINER="mlmd"

kubectl cp -n kubeflow -c $MLMD_CONTAINER \
	$MLMD_POD:/data/mlmd.db \
	./$MLMD_BACKUP

Copy the mlmd backup data to the S3 storage:

S3_BUCKET=backup-bucket-2024
RCLONE_S3_REMOTE=remote-s3
RCLONE_BWIDTH_LIMIT=20M

rclone --size-only copy \
	--bwlimit $RCLONE_BWIDTH_LIMIT \
	./$MLMD_BACKUP \
	$RCLONE_S3_REMOTE:$S3_BUCKET

Optionally, you can remove the `mlmd’ backup data from your local machine:

rm -rf $MLMD_BACKUP

Scale up kfp-metadata-writer:

juju scale-application kfp-metadata-writer 1

Last updated 6 days ago.

Help improve this document in the forum.

Requirements

Configure rclone