Minikube high CPU usage even with no workload on Linux machine

Minikube is a tool that allows you to run Kubernetes for local development. It’s adviced to run all over the k8s documentation, in the tech blogs. Still Minikube looks like a piece of software that is not quite ready for a smooth development experience. At the time of writing minikube’s got a bunch of issues. Some of them are open for a long time, hardly reproducible and quite frustrating. CPU usage with no workload to services is one of them.

Recently I’ve bumped into a problem when an idle minikube process consumes up to 100% of CPU. After a while apiserver stops responding making your cluster not really functioning. It turned out I’m not alone. I run minikube on my Lenovo Thinkpad X1 Carbon machine with 16G RAM and 8 cores (i7 8th gen) on board. I use Ubuntu 18.04 with kvm2 VM driver. So I found this high CPU consumption as quite unexpected and annoying. I’ve solved this problem. But as it sometimes happens in the software development, I’m still not sure what exactly helped. So taking into account some black magic, let’s see what witch brew’s ingredients may help.

Before you begin: monitoring

If you can’t measure it, you can’t improve it”. Before making any tweaks start measuring CPU usage. Altough top tool is cool, it just cannot beat its successor htop. It allows you to tag processes, filter and sort quickly, see processes and threads in a flat and a tree modes.

htop tool

On GNU/Linux you most likely want to track QEMU processes resource usage.

Minikube’s dashboard plugin is also a great tool for monitoing resources — per pod (Overview section) or for the whole cluster (Cluster section). To use dashboard, enable the addon and open it:

minikube addons enable dashboard
minikube dashboard

Use latest OS virtualisation libraries, QEMU tools, VM drivers

This is what I believe really fixed my problem. Make sure you:

  1. Update your OS package information:

    # in my case of Ubuntu 18.04 I use
    sudo apt update
    
  2. Install the latest libs and tools (like libvirt, qemu), enable systemD services needed as per KVM drivers guide

  3. Install the latest driver

Run Minikube with correct VM drivers and resources

If you had already run minikube with the default virtualbox drivers, remove your cluster with:

# don't forget ``--p <your-profile-name>``
# if you had started minikube with custom profile name
minikube delete

Make sure you start your new cluster with drivers needed and resources allocation that meets your services requirements. In my case I used more memory and cpus than it’s set by default:

minikube start --vm-driver kvm2 --memory 4096 --cpus 4

Use only the addons you really need

Minikube’s shipped with a dozen of addons like dashboard for web UI or heapster for Grafana-based monitoring. They may be cool and useful, but don’t try to enable them all at once.

In my case enabling efk addons has led to a high CPU usage, so that I have to disable it.

Make sure you enabled only the addons you really need:

# get list of addons
minikube addons list

# check for addons in the global config too
# for the case of recreating your cluster
minikube config view

Conclusion

Minikube’s idea is cool. You can really save some money by running k8s cluster on you laptop instead of setting it up in a cloud service for testing. But all the abstractions that Kubernetes introduces, virtualisation with KVM and QEMU on GNU/Linux make Minikube complex. Minikube’s niche (a developer’s local machine) prevents it from being quickly fixed as, say, cloud production-ready solutions that earn their owners piles of money. So let’s hope that magic I’ve posted here will work out for you too. I also hope the issue with high CPU usage will be fixed by someone with proper skills.

Comments