Hello K8s with kops

an.tran

— 1 min read

I have spent a big chunk of my Friday evening trying to spin up a k8s cluster using kops. After 5 hours of tinkering, here is the result.

Lesson learned:

  • Don't just jump straight into using kops. Maybe provision yourself a minikube first to learn k8s.
  • For a simpler k8s instance, maybe try using microk8s.

Obstacles

The latest stable version of kops at the time of writing is 1.27.1. This version has a weird bug https://github.com/kubernetes/kops/issues/15852, which give me a fair amount of hair pulling trying to start it up. Solution: install the beta version of 1.28.1, which fixes the issue.

After this, your k8s cluster is generally available. I tried installing a helm chart to test the cluster: https://github.com/paulbouwer/hello-kubernetes. The installation works, but the cluster fail to provision a LoadBalancer to our service. Here is a rough output of kubectl get svc -n hello-kubernetes

NAME                           TYPE           CLUSTER-IP      EXTERNAL-IP
hello-kubernetes-hello-world   LoadBalancer   100.56.13.192   <pending>

Running kubectl describe svc/hello-kubernetes-hello-world -n hello-kubernetes will give us the detail of the error

Warning  SyncLoadBalancerFailed  76s                  service-controller  Error syncing load balancer: failed to ensure load balancer: Unable to update load balancer attributes during attribute sync: "AccessDenied: User: arn:aws:sts::1123242:assumed-role/aws-cloud-controller-manager.kube-system.sa.cluster1.kube-xyzdfe/24242 is not authorized to perform: iam:CreateServiceLinkedRole

Here is the solution for this: https://github.com/kubernetes/kops/issues/15158. Possibly kops failed to set the correct permission for the IAM role.

Afterthought

I tried this approach as a way to learn about k8s ecosystem. However, we see this as a very expensive approach, especially for us when we don't have such a need for a HA deployment yet.

Read more