From Kubernetes to Low Carbon-netes: Optimizing K8s Infrastructure and Workloads for Sustainability
Wondering what is the environmental impact of your Kubernetes clusters? How much energy is consumed and Carbon emitted by them? You are in the right place! Discover the steps that can help you measure and reduce the environmental footprint of your Kubernetes environments.
Listen to me read this post here (not an AI-generated voice!) or subscribe to the feed in your podcast app.
Now back to the main article.
Some time ago I've listened an episode of Environmental Variables podcast about New Research Horizons, and the host Chris Adams used the term - from Kubernetes to Low Carbon-netes (around 42nd minute). So I jotted this sentence down somewhere for it to be used in some future article. The time has finally come for the article to be published. So, thanks Chris for the great idea!
Green Software Foundation
We will start this article by describing how can we measure energy usage of Kubernetes cluster, and with this, the Carbon footprint of the cluster(s).
How can we measure emissions?
If you can't measure it, you can't manage it.
Peter Drucker
The first step in any effort that is related to reduction, especially Carbon emissions, is to measure what our current usage is. In this case - what is our Carbon footprint.
Being the first step, it is for sure one of the hardest, because we can't say with the exact certainty if the numbers are true or not. The overall Carbon emissions of our infrastructure can be mainly calculated from:
- the amount of CO2/Carbon emitted in the production and delivery of the hardware to our premises (e.g. data centre) - this is known as embodied carbon - and
- the amount of electricity this hardware uses while in running - known as operational emissions.
Calculating the embodied carbon can be really difficult, so we need to rely on the data from our manufacturers and distributors. The embodied carbon is already emitted to the atmosphere so we cannot do much about that other than using our hardware for longer period and opt for re-use rather than buying new hardware. To find out more about embodied or embedded carbon, check out one of my previous articles.
Marjan Bugarinović
In this article however, we'll focus on the second point. On the things we can do while operating our hardware.
If the most of the electricity we're using is coming from renewable sources - that is great! However, a lot of times, the source of electricity is rather dynamic - we cannot be 100% certain from which source we're getting the power. What we can do is to measure the power our infrastructure uses, and based on the numbers of Watt hours (Wh), we make calculations and predictions.
For the Kubernetes infrastructure, and other infrastructure for that matter, we can use one of the following tools:
- Scaphandre - a monitoring agent that keeps track of energy consumption of your system.
- Kepler - a Prometheus exporter that uses eBPF to probe energy-related system stats and exports them as metrics.
And guess what - I've written about both these tools here! Checkout the links below to find out more!
Marjan Bugarinović
Marjan Bugarinović
These tools mainly export power usage metrics that we can later convert into CO2 emissions based on the data from our power source provider. You can check my article from a while back, where I'm discussing the emission data sources, or Green APIs and how you can use them.
Marjan Bugarinović
What can we do to reduce the emissions?
So, we have completed the first step - measuring. We now know where we stand. Or at least have some idea where we stand. Next, let's go through some possible emission reduction steps starting from the easier ones first towards the more complicated ones.
Adding resource requests and limits
By default, workloads in Kubernetes cluster run without any resource limitations. Because of this, as a first step, we should always have limits and requests defined on them. It is a good practice, and as well a recommended step, for two reasons:
- Adding them limits uncontrolled usage of resources by cluster workloads.
- Adding them allows Kubernetes to better apply its Quality of Service classes.
The yaml definition, should look something like below.
resources:
requests:
memory: "64Mi"
cpu: "250m"
ephemeral-storage: "100M"
limits:
memory: "128Mi"
cpu: "500m"
ephemeral-storage: "500M"
More information can be found on the following link from Kubernetes documentation.
Add limit ranges and resource quotas
We all know users cannot be trusted (being that end users, developers, system administrators, etc). Therefore, we have some mechanisms that we can apply to make sure that resource requests and limits are always defined.
First, if we want to force no workload runs unbound in the cluster, we can add LimitRange to the namespace. This resource allows us the following:
- enforce min/max resources usage per Pod or Container,
- enforce min/max storage request per Persistent Volume Claim,
- enforce ratio between request and limit for a resource,
- set default request/limit for resources and automatically inject them to Containers at runtime.
The main thing to remember here is that LimitRange works inside a namespace we define. It won't work on cluster level, so we might apply this resource by default to every namespace that is created.
To find out more about limit ranges, go to the link below.
If our resources on the cluster(s) are limited, we might go one step further and enable ResourceQuota on each namespace. This will make sure that no workload takes too much of the resources that are available. If a workload is being deployed and requests resources that crosses the quota, the deployment will be blocked. This way you can safeguard the limited resources of your cluster.
To find out more about resource quotas, checkout the link below.
Application changes
Not all applications handle containerization the same way. For example, when calculating default Heap memory, Java applications (before version 17) didn't use the resource limitations of the Container level, but on a Node level. It didn't properly handle container awareness. This is a problem because application crashes with java.lang.OutOfMemoryError and you don't know why. To fix this, you would need to add -Xms and -Xmx arguments to your JAVA_OPTS.
Turn on/off workloads when not used
One of the most used fix in an endless amount of IT problems - turning it off and on again, can also be quite effective when it comes to reducing the CO2 emissions.
Some data shows that just turning off machines when not used can save us a lot of energy, and therefore reduce Carbon emissions. What is known as the LightSwitchOps.
To do this in a Kubernetes cluster environments, there are two approaches:
- manually scale up/down resources when not used
- automatically (on a schedule) turn off/on resources when not used with kube-green.
The first option is quite simple, but rather painful. If you have multiple resources in the cluster, as you probably do, it can be quite taxing and boring to go from one resource to another and just scale it down when not used.
Another option is to have sort of a schedule when your resources will automatically scale down and up. This can be done with the help of kube-green tool. It is quite easy and simple to set up. And guess what - I already mention this in one of my previous articles.
Marjan Bugarinović
Run batch jobs when energy is greener
Based on the metrics in the first part, you can easily determine when your energy is greener - coming from renewable sources, and when it's not. You can take this information and reschedule your CronJob resources or any batch jobs to run on electricity from renewable sources.
The simple solution would be to adjust the schedule of your cron jobs to run when the energy is greener. The problem with this is approach is the dynamic nature of power sources. We can't easily predict from which sources the energy comes from all the time. Therefore, we need some automation that can help us here. We discuss this in the following section.
Dynamically scale resources based on carbon intensity
Due to its dynamic nature, energy sources cannot be predicted 100% of the time. This is where you can do what is called event based scheduling. There is a tool that allows you to do just that, called KEDA - Kubernetes-based Event Driven Autoscaler. This tool works as a dynamic autoscaler of workloads based on certain events.
Following is the idea of the process you can apply to dynamically autoscale workloads:
- Call one of the Green APIs (e.g. Electricity Maps) to check for Carbon intensity of your location.
- If the Carbon Intensity is currently low, trigger the scale up of certain workloads through KEDA.
- If the Carbon Intensity is getting high, trigger the scale down of the workloads in the same way.
This solution is, from my point of view, the most complicated one to configure. I am going to spend some time in the next weeks to try it out and write up a demo in one of my next articles.
Summary
When looking into reducing your Kubernetes electricity and Carbon footprint, we need to start with the baseline - what amount of electricity we use, and what amount of CO2 do we emit. Then we can go to the next steps of actually reducing the footprint. The ones I mentioned here are the following:
- adding resource requests and limits, either through individual definitions or limit ranges,
- rejecting the workloads that use more than available with resource quotas,
- make applications changes (e.g.
XmxandXmsoptions for Java), - turning the workloads off when not used,
- running batch jobs when energy is greener,
- dynamically scale resources based on locations' carbon intensity.
All the above options should help you reduce the carbon and energy footprint of your cluster. Let me know if you have tried some of them and what have you noticed. If you have something else I didn't consider here, even better! Write down your thoughts, feedback in the comment section below, I am eager to find out more about this topic!