Prerequisites #
Kubernetes Metrics Server #
Overview #
-
The Kubernetes Metrics Server provides resource usage statistics like CPU & RAM, that can be queried by tools like
kubectl topand used by the Horizontal Pod Autoscaler (HPA) to make decisions about scaling. -
Resource requests and limits work regardless of whether the Kubernetes Metrics Server is deployed.
Download Manifest #
# Download official metrics server YAML manifest from SIGs (Kubernetes Special Interest Groups)
wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
Edit Manifest #
Note: It’snecessary to add the --kubelet-insecure-tls option in the “Deployment / spec.containers.args” section.
# Edit the manifest
vi components.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
k8s-app: metrics-server
rbac.authorization.k8s.io/aggregate-to-admin: "true"
rbac.authorization.k8s.io/aggregate-to-edit: "true"
rbac.authorization.k8s.io/aggregate-to-view: "true"
name: system:aggregated-metrics-reader
rules:
- apiGroups:
- metrics.k8s.io
resources:
- pods
- nodes
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
k8s-app: metrics-server
name: system:metrics-server
rules:
- apiGroups:
- ""
resources:
- nodes/metrics
verbs:
- get
- apiGroups:
- ""
resources:
- pods
- nodes
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
k8s-app: metrics-server
name: metrics-server-auth-reader
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
k8s-app: metrics-server
name: metrics-server:system:auth-delegator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
k8s-app: metrics-server
name: system:metrics-server
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:metrics-server
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
spec:
ports:
- name: https
port: 443
protocol: TCP
targetPort: https
selector:
k8s-app: metrics-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
spec:
selector:
matchLabels:
k8s-app: metrics-server
strategy:
rollingUpdate:
maxUnavailable: 0
template:
metadata:
labels:
k8s-app: metrics-server
spec:
containers:
- args:
- --cert-dir=/tmp
- --secure-port=10250
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --kubelet-use-node-status-port
- --metric-resolution=15s
- --kubelet-insecure-tls # Add this line
image: registry.k8s.io/metrics-server/metrics-server:v0.7.2
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /livez
port: https
scheme: HTTPS
periodSeconds: 10
name: metrics-server
ports:
- containerPort: 10250
name: https
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /readyz
port: https
scheme: HTTPS
initialDelaySeconds: 20
periodSeconds: 10
resources:
requests:
cpu: 100m
memory: 200Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
volumeMounts:
- mountPath: /tmp
name: tmp-dir
nodeSelector:
kubernetes.io/os: linux
priorityClassName: system-cluster-critical
serviceAccountName: metrics-server
volumes:
- emptyDir: {}
name: tmp-dir
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
labels:
k8s-app: metrics-server
name: v1beta1.metrics.k8s.io
spec:
group: metrics.k8s.io
groupPriorityMinimum: 100
insecureSkipTLSVerify: true
service:
name: metrics-server
namespace: kube-system
version: v1beta1
versionPriority: 100
Deploy the Metrics Server #
# Deploy the metrics server
kubectl apply -f components.yaml
# Shell output:
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
Verify the Metric Server Resources #
# List deployments: Wait till the "metrics-server" deployment is ready
kubectl get deployments --namespace kube-system
# Shell output: (Wait till ready)
NAME READY UP-TO-DATE AVAILABLE AGE
cilium-operator 1/1 1 1 74d
coredns 2/2 2 2 74d
metrics-server 1/1 1 1 21s
Deployment Resource Management #
Overview #
Resource Request:
-
In the pod specification it’s possible to optionally define how much of each resources like CPU & RAM a container needs
-
The kube-scheduler uses this information to decide which node to place the Pod on. If a node does not have at least this much CPU or RAM available, the pod will not be scheduled on that node.
-
The kubelet also reserves at least the request amount of that system resource specifically for that container to use
-
If the node where the Pod is running has enough of a resource available, it’s possible and allowed for a container to use more resource than its defined request
Resource Limit:
- The kubelet enforces those limits so that the running container is not allowed to use more of that resource than the defined limit
Deployment Example: Resource Requests and Limits #
# Create manifest for the deployment
vi resource-management-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: resource-management-deployment
spec:
replicas: 2
selector:
matchLabels:
app: example-app
template:
metadata:
labels:
app: example-app
spec:
containers:
- name: nginx-container
image: nginx:latest
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
# Deploy the manifest
kubectl apply -f resource-management-deployment.yaml
Deployment Details #
-
resources.limits.memory: "128Mi"The container will be killed if it allocates more then 128Mi MB of RAM. Kubernetes will attempt to restart the container regarding on the pods restartPolicy. -
resources.limits.cpu: "500m"The container will be throttled if it uses more then 500 millicores / half a CPU core.
CPU limit definition:
-
cpu: "1"1 CPU core -
cpu: ""500m"500 millicores / half a CPU core
Verify Resource Requests and Limits #
List pods:
# List pods in "default" namespace
kubectl get pods
# Shell output:
NAME READY STATUS RESTARTS AGE
resource-management-deployment-74bd69b985-4kb7h 1/1 Running 0 7s
resource-management-deployment-74bd69b985-qhmzg 1/1 Running 0 7s
List pod details:
# List pod details
kubectl describe pod resource-management-deployment-74bd69b985-4kb7h
# Shell output:
...
Containers:
nginx-container:
Container ID: containerd://4766781731382231a9388806f97d906a9c848ce1f57159dc1d727ee9c15c2c1c
Image: nginx:latest
Image ID: docker.io/library/nginx@sha256:04ba374043ccd2fc5c593885c0eacddebabd5ca375f9323666f28dfd5a9710e3
Port: <none>
Host Port: <none>
State: Running
Started: Wed, 18 Sep 2024 09:42:53 +0000
Ready: True
Restart Count: 0
Limits: # Resource limits
cpu: 500m
memory: 128Mi
Requests: # Resource request
cpu: 250m
memory: 64Mi
...
List Current Resource Usage #
# Check the current usage of the pods CPU and memory
kubectl top pod resource-management-deployment-74bd69b985-4kb7h
# Shell output:
NAME CPU(cores) MEMORY(bytes)
resource-management-deployment-74bd69b985-4kb7h 0m 4Mi
Delete Deployment #
# Delete the deployment
kubectl delete -f resource-management-deployment.yaml
Deployment Resource Management: Vish/Stress Example #
Create a Configuration #
# Create a configuration for the deployment of a "vish/stress" container
kubectl create deployment vish-stress --image=vish/stress --dry-run=client -o yaml > vish-stress.yaml
Adopt the Configuration #
# Edit the configuration
vi vish-stress.yaml
Original configuration:
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: vish-stress
name: vish-stress
spec:
replicas: 1
selector:
matchLabels:
app: vish-stress
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: vish-stress
spec:
containers:
- image: vish/stress
name: stress
resources: {}
status: {}
Add the following RAM limits and container args:
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: vish-stress
name: vish-stress
spec:
replicas: 1
selector:
matchLabels:
app: vish-stress
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: vish-stress
spec:
containers:
- image: vish/stress
name: stress
resources:
limits:
memory: "4Gi"
requests:
memory: "2500Mi"
args:
- -cpus
- "2"
- -mem-total
- "950Mi"
- -mem-alloc-size
- "100Mi"
- -mem-alloc-sleep
- "1s"
status: {}
Create Deployment #
# Create the deployment
kubectl apply -f vish-stress.yaml
# List pods
kubectl get pods -o wide
# Shell output:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
vish-stress-8474cfbd68-br6bl 1/1 Running 0 8s 10.0.1.165 ubuntu2 <none> <none>
Verify the Allocation with top #
# Run the top command on the "ubuntu2" worker node
top
#PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5454 root 20 0 959052 955008 3072 S 201.0 11.8 1:31.77 stress
Delete the Deployment #
# Delete the deployment
kubectl delete deployment vish-stress
Resource Limits for a Namespace #
Create Example Namespace #
# Create an example namespace with the name "usage-limit"
kubectl create ns usage-limit
Create LimitRange Configuration #
# Create a configuration for the resource limitation
vi resource-range.yaml
apiVersion: v1
kind: LimitRange
metadata:
name: resource-range
spec:
limits:
- default:
cpu: 1
memory: 500Mi
defaultRequest:
cpu: 0.5
memory: 100Mi
type: Container
-
default:Specifies the default resource limits for CPU and memory that will be applied to containers in the namespace if no explicit limits are set. -
defaultRequest:Specifies the default resource requests for containers in the namespace if no explicit requests are set. -
type: ContainerLimits apply to individual containers.
# Create the LimitRange
kubectl create -n usage-limit -f resource-range.yaml
# Shell output:
limitrange/resource-range created
Verify LimitRange #
# List the LimitRange for all namespaces
kubectl get LimitRange -A
# Shell output:
usage-limit resource-range 2024-11-05T13:52:22Z
Create Example Deployment #
Create Deployment #
# Deploy a "vish/stress" container
kubectl create deployment vish-stress -n usage-limit --image vish/stress
Save Deployment Configuration #
# List pods in "usage-limit" namespace
kubectl get pod -n usage-limit
# Shell output:
NAME READY STATUS RESTARTS AGE
vish-stress-54d56f6855-6c4d4 1/1 Running 0 94s
# Save the pod configuration
kubectl get pod vish-stress-54d56f6855-6c4d4 -n usage-limit -o yaml > vish-stress-limited.yaml
Verify the LimitRange #
# cat the pod configuration
cat vish-stress-limited.yaml
# Shell output:
spec:
containers:
- image: vish/stress
imagePullPolicy: Always
name: stress
resources:
limits:
cpu: "1"
memory: 500Mi
requests:
cpu: 500m
memory: 100Mi
-
When a pod (or a deployment, which creates pods) is created without explicitly specifying resource requests and limits, Kubernetes applies the defaults specified in the LimitRange.
-
The LimitRange does not directly appear in the Deployment manifest.
Delete the Deployment #
# Delete the deployment
kubectl delete deployment vish-stress -n usage-limit
Links #
# Official Documentation
https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/