Set up remote access

Charmed Kubeflow delivers a powerful, sophisticated end-to-end MLOps platform which you can deploy in half an hour or less, using MicroK8s or another conformant Kubernetes distribution.

Exposing Charmed Kubeflow over the network so that users can access it is straightforward when you know how, and this tutorial will guide you through the steps needed to get things deployed and configured right for your users.

What you’ll learn

• How to configure MicroK8s to expose Charmed Kubeflow to the outside world
• How to configure your Charmed Kubeflow for round-robin DNS to simplify accessing the platform, with Dnsmasq and Systemd-Resolved

What you’ll need

• A MicroK8s cluster running on Ubuntu 20.04 with one or more nodes running Charmed Kubeflow and MetalLB - see the install guide to get up and running
• Another computer running a modern web browser from which to test remote access to Charmed Kubeflow
• Some command line knowledge

The first step we need to complete will be to enable an ingress on our MicroK8s cluster. On one of the MicroK8s nodes, run the following command:

microk8s enable ingress

Next, we’ll configure the ingress add-on so that it exposes our Charmed Kubeflow MLOps platform. Just make sure that you’ve followed the steps in the install guide and have MetalLB enabled on MicroK8s for your cluster first. Run the following commands to expose your Charmed Kubeflow deployment for remote access:

cat << EOF > ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kubeflow-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: public
  rules:
  - host: kubeflow.ubuntu.local
    http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: istio-ingressgateway
              port:
                number: 80
EOF

microk8s.kubectl apply -f ingress.yaml -nkubeflow

That wasn’t so bad! Wait a few moments until MicroK8s makes the configuration change:

microk8s.kubectl get ingress -nkubeflow
NAME               CLASS    HOSTS                   ADDRESS     PORTS   AGE
kubeflow-ingress   public   kubeflow.ubuntu.local   127.0.0.1   80      29s

Now we’ll make remote access for your clients highly available with round-robin DNS so that if any node of the MicroK8s cluster goes offline, your users will still be able to reach Charmed Kubeflow. Read on…

The next step is to set up Dnsmasq on two of the MicroK8s nodes. We are going to use Dnsmasq as a lightweight caching DNS service that also resolves all of our local MicroK8s hosts to the domain name kubeflow.ubuntu.local, in order to set up round-robin DNS.

First let’s install the Dnsmasq package. Again, on each node in the MicroK8s cluster, run the following command:

sudo apt install dnsmasq -y

Next we need to configure the Dnsmasq daemon. We want Dnsmasq to listen for DNS lookup requests on our MicroK8s nodes’ loopback interface as well as on the public interface. Note that your server may have more than one network interface - make sure you choose the right one!

You can find a lot of information about your MicroK8s node’s network configuration by running the following command:

ip addr

Ok, let’s go ahead and configure Dnsmasq. Note that you should replace the IP address assigned to the variable PUBLIC_IP with your own MicroK8s cluster node’s public IP address, which will likely be different to the one shown below.

Run these commands on each MicroK8s cluster node to set up Dnsmasq as a caching DNS server that recursively forwards unknown lookups to Google’s public DNS:

PUBLIC_IP=192.168.100.10

cat << EOF > dnsmasq.local.conf
domain-needed
bogus-priv
no-poll
server=8.8.8.8
server=8.8.4.4
listen-address=$PUBLIC_IP
except-interface=l0
bind-interfaces
EOF

cat dnsmasq.local.conf | sudo tee -a /etc/dnsmasq.d/local.conf

The next step is to set up static records for Dnsmasq on each of the nodes where Dnsmasq is running. We want to create a static record for each node in the MicroK8s cluster. On each node in the MicroK8s cluster, run the following commands:

declare -a microk8s_cluster=("192.168.100.10" "192.168.100.11" "192.168.100.12")

for node in ${microk8s_cluster[@]}; do
echo "$node kubeflow.ubuntu.local" | sudo tee -a /etc/hosts
done

Alright, we want to bounce Dnsmasq so that it picks up our configuration changes.

sudo systemctl enable dnsmasq
sudo systemctl restart dnsmasq

But there are some extra steps here, to ensure that our servers and clients using systemd-resolved can make use of Dnsmasq. On each node of the cluster, and the systems that will access it, run the following commands:

echo "DNS=192.168.100.10" | sudo tee -a /etc/systemd/resolved.conf
echo "FallbackDNS=192.168.100.11" | sudo tee -a /etc/systemd/resolved.conf
sudo systemctl restart systemd-resolved
sudo systemd-resolve --flush-caches

In order for internal services like the authentication system OIDC-gatekeeper to be able to resolve our Charmed Kubeflow service’s hostname correctly, we need to configure CoreDNS to recursively lookup external addresses via Dnsmasq. Let’s do that now:

microk8s disable dns
microk8s enable dns:192.168.100.10,192.168.100.11

The final step is to reconfigure the Charmed Kubeflow public URL so that the OIDC Gatekeeper and Dex operators know how users will access Charmed Kubeflow. On the computer running Juju, execute the following two commands to complete the setup:

juju config oidc-gatekeeper public-url=http://kubeflow.ubuntu.local
juju config dex-auth public-url=http://kubeflow.ubuntu.local

Congratulations on setting up highly available remote access to your Charmed Kubeflow MLOps platform running on MicroK8s! But if you’re having difficulties, don’t worry - head over to the forum to ask a question and get in touch with the community.

• Read the Charmed Kubeflow authentication guide
• How to use Charmed Kubeflow with GPUs