K3s is a lightweight Kubernetes deployment by Rancher that is fully compliant, yet also compact enough to run on development boxes and edge devices.
In this article, I will show you how to deploy a three-node K3s cluster on Ubuntu nodes that are created using Terraform and a local KVM libvirt provider. Ansible is used for installation of the cluster.
Creating node VMs
We will deploy this K3s cluster on three independent guests running Ubuntu.
These Ubuntu VMs could actually be created using any hypervisor or hyperscaler, but for this article we will use Terraform and the local KVM libvirt provider to create guests named: k3s-1, k3s-2, k3s-3.
Install Terraform, its libvirt provider, and KVM as described in a previous article. Also create a ‘br0’ host bridge and KVM ‘host-bridge’ so that the two additional NIC on k3s-1 can be assigned as explained in a previous article.
Then use my github project to create the three Ubuntu guest OS.
# required packages sudo apt install make git curl -y # github project with terraform to create guest OS git clone https://github.com/fabianlee/k3s-cluster-kvm.git cd k3s-cluster-kvm # set first 3 octets of your host br0 network # this is where MetalLB endpoints .143, .144 will be created sed -i 's/metal_lb_prefix: .*/metal_lb_prefix: 192.168.1/' group_vars/all # download dependencies and modules ansible-playbook install_dependencies.yml # invoke terraform apply from tf-libvirt directory ansible-playbook playbook_terraform_kvm.yml
K3s is a lightweight Kubernetes deployment by Rancher that is fully compliant, yet also compact enough to run on development boxes and edge devices.
In this article, I will show you how to deploy a three-node K3s cluster on Ubuntu nodes that are created using Terraform and a local KVM libvirt provider. Ansible is used for installation of the cluster.
Creating node VMs
We will deploy this K3s cluster on three independent guests running Ubuntu.
These Ubuntu VMs could actually be created using any hypervisor or hyperscaler, but for this article we will use Terraform and the local KVM libvirt provider to create guests named: k3s-1, k3s-2, k3s-3.
Install Terraform, its libvirt provider, and KVM as described in a previous article. Also create a ‘br0’ host bridge and KVM ‘host-bridge’ so that the two additional NIC on k3s-1 can be assigned as explained in a previous article.
Then use my github project to create the three Ubuntu guest OS.
# required packages sudo apt install make git curl -y # github project with terraform to create guest OS git clone https://github.com/fabianlee/k3s-cluster-kvm.git cd k3s-cluster-kvm # set first 3 octets of your host br0 network # this is where MetalLB endpoints .143, .144 will be created sed -i 's/metal_lb_prefix: .*/metal_lb_prefix: 192.168.1/' group_vars/all # download dependencies and modules ansible-playbook install_dependencies.yml # invoke terraform apply from tf-libvirt directory ansible-playbook playbook_terraform_kvm.yml
The KVM guests can be listed using virsh. I have embedded the IP address in the libvirt domain name to make the address obvious.
# should show three running K3s VMs $ export LIBVIRT_DEFAULT_URI=qemu:///system $ virsh list Id Name State -------------------------------------------- ... 10 k3s-1-192.168.122.213 running 11 k3s-2-192.168.122.214 running 12 k3s-1-192.168.122.215 running
cloud-init has been used to give the ‘ubuntu’ user an ssh keypair for login, which allows us to validate the login for each host using the command below.
# accept key as known_hosts for octet in $(seq 213 215); do ssh-keyscan -H 192.168.122.$octet >> ~/.ssh/known_hosts; done # test ssh into remote host for octet in $(seq 213 215); do ssh -i tf-libvirt/id_rsa ubuntu@192.168.122.$octet "hostname -f; uptime"; done
K3s cluster installation overview
As discussed in detail in my other article where the K3s cluster is deployed manually, K3s is first installed on the primary guest VM which generates a node token.
Then K3s is run on the additional worker nodes using the master IP address and node token as parameters so they join the Kubernetes cluster.
The first guest ‘k3s-1’ will serve as the master, with k3s-2 and k3s-3 joining the Kubernetes cluster.
Ansible configuration
In order for Ansible to do its work, we need to inform it of the inventory of guest VMs available and their group variables. Then we use an Ansible playbook to run the specific tasks and roles.
Ansible inventory
The available hosts as well as their group memberships are defined in ‘ansible_inventory’.
$ cat ansible_inventory ... # K3s 'master' [master] k3s-1 # K3s hosts participating as worker nodes [node] k3s-2 k3s-3 # all nodes in K3s cluster [k3s_cluster:children] master node ...
Ansible group variables
The group variables are found in the ‘group_vars’ directory.
# list of group variables $ ls group_vars all master # show variables applying to every host # notice that 'servicelb' and 'traefik' are disabled # which allows us to pick MetalLB and custom ingress later cat group_vars/all # variables applying to just master cat group_vars/master
Ansible playbook
The playbook we use for K3s cluster execution executes a set of roles against the groups defined in the ansible_inventory. The primary role we leverage is k3s-ansible, which comes directly from the k3s-io team.
$ cat playbook_k3s.yml ... - hosts: k3s_cluster gather_facts: yes become: yes roles: - role: prereq - role: download - hosts: master become: yes roles: - role: k3s/master - hosts: node become: yes roles: - role: k3s/node ...
We surround this with prerequisites as well as post-configuration steps to apply additional NIC to the master, generate certificates, and finally installing ‘k9s’ as a graphical utility for Kubernetes management.
Ansible external dependencies
We still have dependencies to fulfill before installing the K3s cluster using Ansible.
- Downloading the ‘k3s-ansible’ role found on github
- Dependent collections found on Ansible galaxy
I have written a playbook that can fetch all dependencies automatically, simply run the command below.
# pulls external dependencies ansible-playbook install_dependencies.yml
Install K3s cluster using Ansible
We can now have Ansible install the K3s cluster on our three guest VMs by invoking the playbook.
ansible-playbook playbook_k3s.yml
Validate Kubernetes deployment to cluster
As a quick test of the Kubernetes cluster, create a test deployment of nginx. Then check the pod status from each of the cluster nodes.
# deploy from k3s-1, deploys to entire cluster $ ssh -i tf-libvirt/id_rsa ubuntu@192.168.122.213 "sudo kubectl create deployment nginx --image=nginx" deployment.apps/nginx created # show pod deployment $ ssh -i tf-libvirt/id_rsa ubuntu@192.168.122.213 "sudo kubectl get pods -l app=nginx" NAME READY STATUS RESTARTS AGE nginx-6799fc88d8-txcnh 1/1 Running 0 2m58s
Validate remotely using kubectl
The playbook_k3s.yml contains a role named ‘k3s-get-kubeconfig-local’ that copies the remote kubeconfig to a local file named ‘/tmp/k3s-kubeconfig’.
So if you have kubectl installed on your host VM, you can also query the Kubernetes cluster remotely.
$ kubectl --kubeconfig /tmp/k3s-kubeconfig get pods -l app=nginx NAME READY STATUS RESTARTS AGE nginx-6799fc88d8-txcnh 1/1 Running 0 4m24s
