April 21, 2023
Introduction
I've been running digrin.com on a Scaleway.com VM instance since 2017 without too many issues.It was a server with 4GB of RAM and 4 shared cores, it was running Rancher 1.6,that was orchestrating docker containers. It costs like $10 monthly.However, I was running close to memory limit,I had to merge workers to a single container and limit redis memory.Rancher 1.6 was deprecated for quite few years now, I originally installed it,because we used it at work and I wanted to get familiar with it.While I didn't experience outages or unknown issues, I was fine with running it even when it was deprecated.But on 13.3.2023 I got an email from server provider titled "Deprecation of Bootscript feature",so I thought I would try to migrate to something newer.As we use kubernetes at work, I decided I will give that a try.I will write my notes and links here, in case I need to come back to them in the future.
Requirements
As I mentioned, I'm running digrin.com on a Scaleway.com instance for $10.I like to keep my costs low, so I don't need to squeeze money out of digrin.com.I compared hetzner to scaleway and hetzner was a bit cheaper. I got 8GB of RAM and 4 shared cores for €15.72.Two weeks later hetzner released new ARM64 servers, 8GB of RAM and 4 shared cores for €7.73(I tried to run my stack on ARM64 server, but Gitlab does not support builds there)I was considering managed PostgreSQL, Redis and k8s cluster,but prices there are nowhere near what I am willing to pay for hobby projects.IMO I should be fine with 8GB of RAM and 4 cores and I will not need multiple servers for quite some time.I chose to use k3s, because it's lightweight, and I've read Rancher made quite nice job there.
Setup
I got a server from hetzner with latest Debian.
I installed k3s with curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server" sh,but it didn't work out of the box, I also had to install
$ apt update && apt install apparmor apparmor-utilsthen install k3s
[emailprotected]:~# curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server" sh[INFO] Finding release for channel stable[INFO] Using v1.26.3+k3s1 as release[INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.26.3+k3s1/sha256sum-amd64.txt[INFO] Skipping binary downloaded, installed k3s matches hash[INFO] Skipping installation of SELinux RPM[INFO] Skipping /usr/local/bin/kubectl symlink to k3s, already exists[INFO] Skipping /usr/local/bin/crictl symlink to k3s, already exists[INFO] Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr[INFO] Creating killall script /usr/local/bin/k3s-killall.sh[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env[INFO] systemd: Creating service file /etc/systemd/system/k3s.service[INFO] systemd: Enabling k3s unitCreated symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.[INFO] systemd: Starting k3sAs I already have some work servers in my kubeconfig, I could not just copy kubeconfig from k3s server.I used info from here to copy just required parts in kubeconfig file, so now I have 3 contexts in my kubeconfig file.
Redis container
Next step was to install redis container I will share with all my apps.I used this tutorial with memory limits and this configmap:
apiVersion: v1kind: ConfigMapmetadata: name: redis-configdata: redis-config: | # limit memory usage to something sane maxmemory 700mb # drop only least recently used keys on overuse maxmemory-policy volatile-lruPostgres container
I used this tutorial to setup postgres container, but when I tested restarting deployment,it always broke the database within few restarts.I googled if it's common issue and found this command,but that is dangerous and data loss might happen (it happened to me, only 4 tables out of 40 were there :D ),so this was not acceptable:
docker run --rm -it -u=postgres -e POSTGRES_PASSWORD={password} -v /mnt/data:/some_dir postgres:15.2 /bin/bash -c "pg_resetwal -f /some_dir"As this was my first day playing with k3s,I was quite sad random tutorial wasn't foolproof.I asked about it in stackoverflow andit turns out I should have used StatefulSet instead of Deployment.In Rancher 1.6 I was using PostgreSQL 13, so I thought I might migrate to postgres:15.2 here.Works fine, no breaking changes so far.
Secrets
In Rancher 1.6 I used environment variables to pass secrets to containers, in k3s I use kubernetes secrets.I didn't want to manage vault service, I created a separate private repo where I will store secrets.
Container registry
Gitlab.com provides registry, which I was using in Rancher 1.6. But in Rancher 1.6 there was a UI,where I would just input login and password and it would just work.I even had a backup registry there, because one day gitlab registry was down and my server could not pull the image -> downtime.
In k3s it seems to be a bit harder, I used this tutorial to connect to gitlab registry.
Staging web service
As I had Redis, PostgreSQL and registry setup, I could finally start working on the web service.I didn't have sandbox/staging in Rancher 1.6 because of memory limits,but I could build one here to make sure everything works before migrating production.
First thing was to setup ALLOWED_HOSTS in Django settings, so I could access the service from k3s cluster.In k3s I needed django-allow-cidr middleware for django,following this tutorial.I got stuck here for some time, as I was forcing CIDR ranges into ALLOWED_HOSTS instead of ALLOWED_CIDR_NETS(names totally look the same :D ).
Ingress controller
In Rancher 1.6 I used nginx container.So in k3s, I also wanted nginx ingress controller, but as I didn't disable Traefik from first k3s install,I never made it work and thought if Traefik is default, I will just use that.There is a simple command to forward portsand I could view Traefik dashboard on localhost(not that this was useful, but I wanna keep command here for later use just in case):
kubectl port-forward -n kube-system traefik-56b8c5fb5c-sp67z 9000:9000Dashboard is available on http://localhost:9000/dashboard/#/
In Rancher 1.6, I had to run certbot to update https certificates quarterly (thank you Letsencrypt!).There was a template for that, so it was easy.With k3s, I found out Traefik can do that by default. It even supports multiple providers,and as Cloudflare was one of them, I used that.I found a github gist that worked out of the box.
Static files
In Rancher 1.6, I was using nginx container, which mounted static files from host and served them.In k3s I wasn't using nginx, so I moved static files to web container with whitenoise.I liked that nginx handles static files and I thought it is fast. With whitenoise I have a bit more work in web container,but I guess at least it's closer to the app.Apart from installing whitenoise there, I also had to exclude static files from logging (I don't like polluted logs).
Observability
logz.io was my chosen logging in Rancher 1.6, as we also used to use it at work in 2017.But there is just 1 day of logs in free tier. At work, we started playing with grafana.I thought I will give that a try. Shipping logs there was easy, just running one command grafana provided.I would prefer manifests I can edit instead of one command, so I just dumped manifests from k3s.For k3s logs, I had to set cri: {}, so I can json parse logs in grafana(my app produces json logs).Free tier in grafana seems to be superior to logz.io (14 days retention),and I believe I can hook up metrics there was well (maybe in the future).
I used to use datadog in rancher 1.6, but I had to remove it when I was facing memory issues.Only time will tell, if grafana will be sufficient, or I will need to install datadog agent as well.
Release
With Rancher 1.6, I was releasing from gitlab CI job. Gitlab had a support for kubernetes,but they deprecated it in favour of some agent. As I thought releasing is just running one kubectl command,I wanted to avoid having unnecessary agents running on my server. So I use gitlab CI to run a release job,as mentioned in this blogpost.
deploy-web: stage: deploy-thor-production image: name: bitnami/kubectl:latest entrypoint: [""] script: - kubectl config set-cluster k3s --server="${K3S_SERVER}" - kubectl config set clusters.k3s.certificate-authority-data ${CERTIFICATE_AUTHORITY_DATA} - kubectl config set-credentials gitlab --token="${GITLAB_SA_TOKEN}" - kubectl config set-context default --cluster=k3s --user=gitlab - kubectl config use-context default - sed -i "s/<VERSION>/${CI_COMMIT_SHA}/g" k8s/digrin-production/deployment-web.yaml - kubectl apply -f k8s/digrin-production/deployment-web.yaml when: manualWhen my deployment-web.yaml might look like this:
apiVersion: apps/v1kind: Deploymentmetadata: name: digrin-web labels: app: digrin-webspec: replicas: 2 strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 0% progressDeadlineSeconds: 120 selector: matchLabels: app: digrin-web template: metadata: labels: app: digrin-web spec: containers: - image: registry.gitlab.com/digrin/digrin:<VERSION> name: digrin-web envFrom: - secretRef: name: digrin-production-secret command: ['/bin/sh', 'extras/docker-entrypoint.sh'] ports: - containerPort: 8000 name: gunicorn livenessProbe: httpGet: path: /robots.txt port: 8000 scheme: HTTP initialDelaySeconds: 20 periodSeconds: 10 timeoutSeconds: 3 failureThreshold: 9 readinessProbe: httpGet: path: /robots.txt port: 8000 scheme: HTTP initialDelaySeconds: 20 periodSeconds: 2 failureThreshold: 5 timeoutSeconds: 2 imagePullSecrets: - name: registry-credentialsProduction Migration Plan
- Disable caching on Cloudflare. Add firewall rule to block all traffic except my IP (cloudflare doesn't have maintenance mode)
- Create new backup from up-to-date production database.Download backup and push it to k3s PostgreSQL server:
PGPASSWORD={password} psql -h {host/ip} -p {exposed_pg_port} -d {db_name} -U {pg_user} -f digrin_dump.sql -W - Release production version with async and periodic tasks from production (they were disabled while testing)
- Run script to backup PostgreSQL database
- Test web works fine:
- Login
- Check portfolio and all tabs
- Create portfolio
- Edit portfolio
- Import test portfolio
- Update DNS records to point to k3s server.
- Disable firewall rule on cloudflare.com.
- Profit!
Production Migration
In reality, export and import DB took longer than I expected:
9:57 Cloudflare firewall rule applied.
10:06 psql DB load finished.
10:07 Workers being released.
10:11 DNS updated.
10:17 DNS applied, website runs.
Conclusion
I was quite happy with Rancher 1.6, but now I am even happier with k3s.I like having all manifests versioned now. It was not the case with rancher 1.6,where I was storing docker compose files, but I believe they were mostly outdated. Same for secrets/env variables.
I started with rancher 1.6, because as Python developer, I was quite lost when something devops related had to be done at work.With installing my own rancher 1.6 instance, I lost a few hair in the learning curve, but was much more confident after setting it up.I believe it is the same case with k3s, I was struggling for a few evenings (few hours every other day),but now I am more confident with kubernetes (still noob, but It runs my hobby projects).If you also have a technology you are not confident in, I recommend to play with it!
FAQs
Is K3s production ready? ›
K3s is a production-ready Kubernetes distribution that can handle many types of workloads. Even though it is a streamlined version of Kubernetes, it is still a fully compliant Kubernetes distribution that includes the core features of K8s, including container orchestration, networking, security, and storage.
Does K3s use Traefik? ›K3s versions 1.20 and earlier include Traefik v1. K3s versions 1.21 and later install Traefik v2, unless an existing installation of Traefik v1 is found, in which case Traefik is not upgraded to v2. For more information on the specific version of Traefik included with K3s, consult the Release Notes for your version.
What is limitation of K3s? ›Among its limitations is that K3s does not come with a distributed database by default. This limits the control plane's high availability capabilities. You need to point K3s servers to an external database endpoint (etcd, PostgreSQL, and MySQL) to achieve high availability of its control plane.
What are the minimum requirements for K3s server? ›...
Hardware
| Spec | Minimum | Recommended |
|---|---|---|
| CPU | 1 core | 2 cores |
| RAM | 512 MB | 1 GB |
K3s includes and defaults to containerd, an industry-standard container runtime.
Do I need to install Docker before K3s? ›Docker is not required for RKE2 or K3s clusters. There are a couple of options for installing Docker. One option is to refer to the official Docker documentation about how to install Docker on Linux. The steps will vary based on the Linux distribution.
Is Traefik good for production? ›For containerized workloads, Traefik Proxy does a great job as easy-to-configure reverse proxy that detects backends based on other running Dockers containers and their labels, which works well for simpler production environments.
Why is traefik better than nginx? ›Overall, Traefik presents a more user-friendly experience with its intuitive web interface and auto-discovery features. NGINX is lacking in this regard but provides more fine-grained control.
Is traefik faster than nginx? ›In terms of transfer speed, Nginx was able to handle a higher amount of data per second, with a transfer rate of 13.53MB/sec compared to Traefik v3's 7.19MB/sec.
Is k3s high availability? ›An HA K3s cluster is comprised of: Two or more server nodes that will serve the Kubernetes API and run other control plane services. Zero or more agent nodes that are designated to run your apps and services. An external datastore (as opposed to the embedded SQLite datastore used in single-server setups)
Should I use k3s or K8s? ›
If you're running on bare metal or a virtual machine, K3s is a better option. This also applies to your existing architecture. If you're running a hybrid architecture or a cloud-based hybrid architecture, K3s is likely the best choice for you. If you're running an all-cloud architecture, K8s is your best option.
What is the difference between k3s agent and server? ›A server node is defined as a host running the k3s server command, with control-plane and datastore components managed by K3s. An agent node is defined as a host running the k3s agent command, without any datastore or control-plane components. Both servers and agents run the kubelet, container runtime, and CNI.
How many master nodes for k3s? ›HA K3S requires at least three nodes, including two masters and one worker. The only requirement for the nodes is that they have static IP-Addresses.
What is the difference between deployment and service in k3s? ›What's the difference between a Service and a Deployment in Kubernetes? A deployment is responsible for keeping a set of pods running. A service is responsible for enabling network access to a set of pods.
What is the difference between k3d and k3s? ›k3d is a wrapper of k3s but one of the apparent differences is that k3s deploys a virtual machine-based Kubernetes cluster while k3d deploys Docker-based k3s Kubernetes clusters.
Why is K3s lightweight? ›Lightweight — The binary containing the non-containerized components is smaller than K8s. Fast deployment — You can use a single command to install and deploy K3s, and it will take you less than 30 seconds to do so. Simplified — Thanks to the self-contained single binary package.
Does rancher use K3s? ›Rancher helps K3s users manage the high volume of clusters with Rancher Continuous Delivery which gives users a controller that allows them to efficiently manage Kubernetes at the edge.
Does K3s use etcd? ›Important: K3s v1. 22.2 and newer support migration from SQLite to etcd. Older versions will create a new empty datastore if you add --cluster-init to an existing server.
Should I still use Docker with Kubernetes? ›Can Kubernetes Run Without Docker? The answer is both yes and no. Kubernetes, in itself, is not a complete solution. It depends on a container runtime to orchestrate; you can't manage containers without having containers in the first place.
Can Kubernetes work without Docker? ›You can decide to use Kubernetes without Docker, or even Docker without Kubernetes for that matter (but we advise you to use it for different purposes than running containers). Still, even though Kubernetes is a rather extensive tool, you will have to find a good container runtime for it – one that has implemented CRI.
Can I use Kubernetes instead of Docker? ›
Kubernetes (sometimes referred to as K8s) is a popular open source platform that orchestrates container runtime systems across a cluster of networked resources. Kubernetes can be used with or without Docker.
Why should I use Traefik? ›One of the key advantages of Traefik is its ability to automatically discover and route traffic to backend services without requiring manual configuration. This is achieved through Traefik's support for service discovery providers such as Docker, Kubernetes, and Consul.
What is the point of Traefik? ›Traefik is an open-source Edge Router that makes publishing your services a fun and easy experience.
What is Traefik good for? ›Traefik is a leading modern reverse proxy and load balancer that makes deploying microservices easy. Traefik integrates with your existing infrastructure components and configures itself automatically and dynamically.
Is Traefik an ingress controller? ›The Traefik Kubernetes Ingress provider is a Kubernetes Ingress controller; that is to say, it manages access to cluster services by supporting the Ingress specification.
What is the difference between Istio and Traefik? ›Istio's control plane provides an abstraction layer over the underlying cluster management platform, such as Kubernetes, Mesos, etc; Traefik: Load Balancer for Microservices. Træfɪk is a modern HTTP reverse proxy and load balancer made to deploy microservices with ease.
What is the difference between Traefik Daemonset and deployment? ›DaemonSets automatically scale to new nodes, when the nodes join the cluster, whereas Deployment pods are only scheduled on new nodes if required. DaemonSets ensure that only one replica of pods run on any single node.
What is the maximum size of Traefik? ›Traefik uses the Go net/http module to handle request. This module has a default value for the maximum permitted size of the headers in an HTTP request. By default the DefaultMaxHeaderBytes value is set at 1MB (const DefaultMaxHeaderBytes = 1 << 20 // 1 MB).
How many requests can Traefik handle? ›Traefik is obviously slower than Nginx, but not so much: Traefik can serve 28392 requests/sec and Nginx 33591 requests/sec which gives a ratio of 85%.
What is the memory limit for Traefik? ›By default, Traefik Enterprise proxies have a limit of 4 gigabytes of memory on many platforms, but this can be customized by modifying the generated installation manifests (see the section Installing/Customizing for more information).
Can K3s run on a single node? ›
To set up a single-node K3s cluster, run the Rancher server installation command on just one node instead of two nodes. In both single-node setups, Rancher can be installed with Helm on the Kubernetes cluster in the same way that it would be installed on any other cluster.
Does K3s support multiple nodes? ›k3d makes it very easy to create single- and multi-node k3s clusters in docker, e.g. for local development on Kubernetes.
What is the port range for K3s? ›The K3s server needs port 6443 to be accessible by the nodes. The nodes need to be able to reach other nodes over UDP port 8472 when Flannel VXLAN is used.
Who owns K3s? ›Eric Johnson - Business Owner - K3S Motorsports | LinkedIn.
What is the best OS to install Kubernetes? ›Kubernetes is designed for its control plane to run on Linux. Within your cluster you can run applications on Linux or other operating systems, including Windows.
Why is it called K3s? ›What's with the name? We wanted an installation of Kubernetes that was half the size in terms of memory footprint. Kubernetes is a 10-letter word stylized as K8s. So something half as big as Kubernetes would be a 5-letter word stylized as K3s.
How many pods should I run on a node? ›About default maximum Pods per node. By default, GKE allows up to 110 Pods per node on Standard clusters, however Standard clusters can be configured to allow up to 256 Pods per node. Autopilot clusters have a maximum of 32 Pods per node.
What is the recommended master node size in Kubernetes? ›More specifically, Kubernetes is designed to accommodate configurations that meet all of the following criteria: No more than 110 pods per node. No more than 5,000 nodes. No more than 150,000 total pods.
How many VMS needed for Kubernetes? ›A Kubernetes cluster consists of master and worker nodes. Best practice is to have three or more master nodes for High availability and a set of worker nodes based on the cluster needs.
How do you check if deployment is successful in Kubernetes? ›To see the Deployment rollout status, run kubectl rollout status deployment/nginx-deployment . Notice that the Deployment has created all three replicas, and all replicas are up-to-date (they contain the latest Pod template) and available.
How do you expose a service in k3s? ›
- Create a cluster, mapping the ingress port 80 to localhost:8081. ...
- Get the kubeconfig file (redundant, as k3d cluster create already merges it into your default kubeconfig file) ...
- Create a nginx deployment. ...
- Create a ClusterIP service for it.
In short, Deployment and ReplicaSet are used to manage the lifecycle of pods in Kubernetes. Deployment provides higher-level abstractions and additional features such as rolling updates, rollbacks, and versioning of the application. ReplicaSet is a lower-level abstraction that provides basic scaling mechanisms.
What are the disadvantages of K3s? ›Among its limitations is that K3s does not come with a distributed database by default. This limits the control plane's high availability capabilities. You need to point K3s servers to an external database endpoint (etcd, PostgreSQL, and MySQL) to achieve high availability of its control plane.
Does K3s include Traefik? ›K3s versions 1.20 and earlier include Traefik v1. K3s versions 1.21 and later install Traefik v2, unless an existing installation of Traefik v1 is found, in which case Traefik is not upgraded to v2. For more information on the specific version of Traefik included with K3s, consult the Release Notes for your version.
Can k3d be used for production? ›K3s and k3d are lightweight Kubernetes distributions that allow you to quickly deploy production-level Kubernetes in your local environments without much storage or network requirements.
Is Kubernetes ready for production? ›We refer to a Kubernetes environment as “production-ready” when it has everything needed to serve traffic to real end users. There are many requirements to make Kubernetes production-ready.
When was k3s released? ›It was officially released on Feb. 26, 2019. Rancher started K3s as an embedded component of Rio, which was one of their experimental projects.
Why use K8s instead of k3s? ›The primary differences between K3s Vs. K8s is that K3s is a lightweight, easy-to-use version of Kubernetes designed for resource-constrained environments, while K8s is a more feature-rich and robust container orchestration tool.
What is the difference between k3s and K3d performance? ›One of the key differences is that k3d deploys Docker-based k3s Kubernetes clusters while k3s deploys a virtual machine-based Kubernetes cluster. K3d offers a more scalable version of k3s which might make it preferable to the standard k3s.
What is the alternative to K3d? ›Kubernetes, Docker Compose, Rancher, Docker Swarm, and Argo are the most popular alternatives and competitors to K3d.
What is the difference between K3d server and agent? ›
A server node is defined as a host running the k3s server command, with control-plane and datastore components managed by K3s. An agent node is defined as a host running the k3s agent command, without any datastore or control-plane components. Both servers and agents run the kubelet, container runtime, and CNI.
What are the minimum requirements for Kubernetes production? ›A minimum Kubernetes worker node configuration is a bare-metal server or an Oracle Linux Kernel-based Virtual Machine (KVM) instance with: 1 CPU cores (Intel VT-capable CPU) 8GB RAM. 1GB Ethernet NIC.
What will replace Kubernetes? ›- AWS Fargate. ...
- Azure Container Instances. ...
- Google Cloud Run. ...
- Google Kubernetes Engine (GKE) ...
- Amazon Elastic Kubernetes Service (EKS) ...
- Openshift Container Platform. ...
- Rancher. ...
- Docker Swarm.
There are multiple issues that can cause the Kubernetes node not ready state, such as the following; Lack of Resources in the Node to run the Pod. It means that the node does not have the adequate processing power, memory, or disk space to accommodate the resources requested by the Pod.
What is the minimum number of nodes in K3s? ›Because K3s server nodes are schedulable by default, the minimum number of nodes for an HA K3s server cluster is two server nodes and zero agent nodes.
Is K3s a single node cluster? ›An HA K3s cluster consists of 2 or more server nodes that serve the Kubernetes API and run other control plane services and an external datastore.