Prerequisites #
Kubernetes Metrics Server #
Overview #
-
The Kubernetes Metrics Server provides resource usage statistics like CPU & RAM, that can be queried by tools like
kubectl topand used by the Horizontal Pod Autoscaler (HPA) to make decisions about scaling. -
Resource requests and limits work regardless of whether the Kubernetes Metrics Server is deployed.
Download Manifest #
# Download official metrics server YAML manifest from SIGs (Kubernetes Special Interest Groups)
wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
Edit Manifest #
Note: It’snecessary to add the --kubelet-insecure-tls option in the “Deployment / spec.containers.args” section.
# Edit the manifest
vi components.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
k8s-app: metrics-server
rbac.authorization.k8s.io/aggregate-to-admin: "true"
rbac.authorization.k8s.io/aggregate-to-edit: "true"
rbac.authorization.k8s.io/aggregate-to-view: "true"
name: system:aggregated-metrics-reader
rules:
- apiGroups:
- metrics.k8s.io
resources:
- pods
- nodes
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
k8s-app: metrics-server
name: system:metrics-server
rules:
- apiGroups:
- ""
resources:
- nodes/metrics
verbs:
- get
- apiGroups:
- ""
resources:
- pods
- nodes
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
k8s-app: metrics-server
name: metrics-server-auth-reader
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
k8s-app: metrics-server
name: metrics-server:system:auth-delegator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
k8s-app: metrics-server
name: system:metrics-server
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:metrics-server
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
spec:
ports:
- name: https
port: 443
protocol: TCP
targetPort: https
selector:
k8s-app: metrics-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
spec:
selector:
matchLabels:
k8s-app: metrics-server
strategy:
rollingUpdate:
maxUnavailable: 0
template:
metadata:
labels:
k8s-app: metrics-server
spec:
containers:
- args:
- --cert-dir=/tmp
- --secure-port=10250
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --kubelet-use-node-status-port
- --metric-resolution=15s
- --kubelet-insecure-tls # Add this line
image: registry.k8s.io/metrics-server/metrics-server:v0.7.2
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /livez
port: https
scheme: HTTPS
periodSeconds: 10
name: metrics-server
ports:
- containerPort: 10250
name: https
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /readyz
port: https
scheme: HTTPS
initialDelaySeconds: 20
periodSeconds: 10
resources:
requests:
cpu: 100m
memory: 200Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
volumeMounts:
- mountPath: /tmp
name: tmp-dir
nodeSelector:
kubernetes.io/os: linux
priorityClassName: system-cluster-critical
serviceAccountName: metrics-server
volumes:
- emptyDir: {}
name: tmp-dir
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
labels:
k8s-app: metrics-server
name: v1beta1.metrics.k8s.io
spec:
group: metrics.k8s.io
groupPriorityMinimum: 100
insecureSkipTLSVerify: true
service:
name: metrics-server
namespace: kube-system
version: v1beta1
versionPriority: 100
Deploy the Metrics Server #
# Deploy the metrics server
kubectl apply -f components.yaml
# Shell output:
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
Verify the Metric Server Resources #
# List deployments: Wait till the "metrics-server" deployment is ready
kubectl get deployments --namespace kube-system
# Shell output: (Wait till ready)
NAME READY UP-TO-DATE AVAILABLE AGE
cilium-operator 1/1 1 1 74d
coredns 2/2 2 2 74d
metrics-server 1/1 1 1 21s
Deployment Resource Management #
Overview #
Resource Request:
-
In the pod specification it’s possible to optionally define how much of each resources like CPU & RAM a container needs
-
The kube-scheduler uses this information to decide which node to place the Pod on. If a node does not have at least this much CPU or RAM available, the pod will not be scheduled on that node.
-
The kubelet also reserves at least the request amount of that system resource specifically for that container to use
-
If the node where the Pod is running has enough of a resource available, it’s possible and allowed for a container to use more resource than its defined request
Resource Limit:
- The kubelet enforces those limits so that the running container is not allowed to use more of that resource than the defined limit
Deployment Example: Resource Requests and Limits #
# Create manifest for the deployment
vi resource-management-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: resource-management-deployment
spec:
replicas: 2
selector:
matchLabels:
app: example-app
template:
metadata:
labels:
app: example-app
spec:
containers:
- name: nginx-container
image: nginx:latest
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
# Deploy the manifest
kubectl apply -f resource-management-deployment.yaml
Deployment Details #
-
resources.limits.memory: "128Mi"The container will be killed if it allocates more then 128Mi MB of RAM. Kubernetes will attempt to restart the container regarding on the pods restartPolicy. -
resources.limits.cpu: "500m"The container will be throttled if it uses more then 500 millicores / half a CPU core.
CPU limit definition:
-
cpu: "1"1 CPU core -
cpu: ""500m"500 millicores / half a CPU core
Verify Resource Requests and Limits #
List pods:
# List pods in "default" namespace
kubectl get pods
# Shell output:
NAME READY STATUS RESTARTS AGE
resource-management-deployment-74bd69b985-4kb7h 1/1 Running 0 7s
resource-management-deployment-74bd69b985-qhmzg 1/1 Running 0 7s
List pod details:
# List pod details
kubectl describe pod resource-management-deployment-74bd69b985-4kb7h
# Shell output:
...
Containers:
nginx-container:
Container ID: containerd://4766781731382231a9388806f97d906a9c848ce1f57159dc1d727ee9c15c2c1c
Image: nginx:latest
Image ID: docker.io/library/nginx@sha256:04ba374043ccd2fc5c593885c0eacddebabd5ca375f9323666f28dfd5a9710e3
Port: <none>
Host Port: <none>
State: Running
Started: Wed, 18 Sep 2024 09:42:53 +0000
Ready: True
Restart Count: 0
Limits: # Resource limits
cpu: 500m
memory: 128Mi
Requests: # Resource request
cpu: 250m
memory: 64Mi
...
List Current Resource Usage #
# Check the current usage of the pods CPU and memory
kubectl top pod resource-management-deployment-74bd69b985-4kb7h
# Shell output:
NAME CPU(cores) MEMORY(bytes)
resource-management-deployment-74bd69b985-4kb7h 0m 4Mi
Delete Deployment #
# Delete the deployment
kubectl delete -f resource-management-deployment.yaml
Links #
# Official Documentation
https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/