Kubernetes Networking - Liveness, Readiness & Startup Probe Examples: Liveness Probe (TCP), Readiness Probes (TCP / Single Pod and Multi Pod Dependency), Startup Probe (Command, TCP & HTTP)

Checks whether the application / container has started properly and is ready to handle requests.
Unlike liveness & readiness probes ( which run continuously), startup probes only run once during the container startup phase.
Useful for containers with long startup time on their first initialization. Prevents the container from beeing killed before they are ready.

Liveness Probe
#

The kubelet uses liveness probes to know when to restart a container.
If the liveness probe is successful, no action is taken and no logs are recorded.
If the liveness probe fails, the event is logged, and the kubelet kills the container according to the configured restartPolicy.

TCP Liveness Probe
#

Pod Manifest
#

vi livenessprobe-tcp.yaml

apiVersion: v1
kind: Pod
metadata:
  name: nginx-livenessprobe-tcp
spec:
  containers:
  - name: nginx-livenessprobe
    image: nginx:latest
    ports:
    - containerPort: 80
    livenessProbe:
      tcpSocket:
        port: 80
      initialDelaySeconds: 15 # Time after the container has started before liveness probes are initiated
      periodSeconds: 10 # Frequency in seconds with which to perform the probe
      timeoutSeconds: 1 # How long to wait for a response
      failureThreshold: 3 # Number of failures before marking the container as Unready

kubectl apply -f livenessprobe-tcp.yaml

Verify Pod
#

# List pods
kubectl get pods

# Shell output: (Wait till READY)
NAME                      READY   STATUS    RESTARTS   AGE
nginx-livenessprobe-tcp   1/1     Running   0          26s

Verify Liveness Status
#

# List pod details
kubectl describe pod nginx-livenessprobe-tcp

# Shell output:
Containers:
  nginx-livenessprobe:
    Container ID:   containerd://7badd7c84248aa78c43d7859324d30d97759059ff85fd886c3b7f1262bf65218
    Image:          nginx:latest
    Image ID:       docker.io/library/nginx@sha256:447a8665cc1dab95b1ca778e162215839ccbb9189104c79d7ec3a81e14577add
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Fri, 16 Aug 2024 12:54:31 +0000
    Ready:          True
    Restart Count:  0
    Liveness:       tcp-socket :80 delay=15s timeout=1s period=10s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-c28x6 (ro)

# Kubernetes is now checking the pod's readiness using the specified tcpSocket probe on port "80"
Liveness:       tcp-socket :80 delay=15s timeout=1s period=10s #success=1 #failure=3

Details:

delay=15s The probe waits 15 seconds before starting the first check
timeout=1s Each check must complete within 1 second
period=10s The check is repeated every 10 seconds
#success=1 The pod needs to pass 1 ongoing check to be marked as ready
#failure=3 The pod can fail up to 3 consecutive checks before being marked as unready

Verify Liveness Probe
#

Continuously watch the pod status:

# Continuously watch the pod status
kubectl get pod nginx-livenessprobe-tcp -w

Trigger pod failure: (In a seperate terminal)

# Access container terminal
kubectl exec -it nginx-livenessprobe-tcp -- sh

# Stop Nginx / make port 80 unavailable
nginx -s stop

Verify the pod status

# (Continuously watch the pod status)
kubectl get pod nginx-livenessprobe-tcp -w

# Shell output:
NAME                      READY   STATUS    RESTARTS   AGE
nginx-livenessprobe-tcp   1/1     Running   0          2m23s
nginx-livenessprobe-tcp   0/1     Completed   0          3m17s
nginx-livenessprobe-tcp   1/1     Running     1 (3s ago)   3m19s

Note: Even without a liveness probe configured, Kubernetes still expects the main process inside the container to be running. If the main process inside the container stops, the container will exit, and Kubernetes will automatically restart the pod.

Delete Pod
#

kubectl delete -f livenessprobe-tcp.yaml

Readiness Probe
#

Readiness probes are used to determine whether a container is ready to accept incoming traffic.

TCP Readiness Probe
#

Pod Manifest
#

Kubernetes will periodically check if the container is able to accept TCP connections on port 80:

vi readinessprobe-tcp.yaml

apiVersion: v1
kind: Pod
metadata:
  name: nginx-readinessprobe-tcp
spec:
  containers:
  - name: nginx-readinessprobe
    image: nginx:latest
    ports:
    - containerPort: 80
    readinessProbe:
      tcpSocket:
        port: 80
      initialDelaySeconds: 15  # Time after the container has started before readiness probes are initiated
      periodSeconds: 10 # Frequency in seconds with which to perform the probe
      timeoutSeconds: 1  # How long to wait for a response
      failureThreshold: 3  # Number of failures before marking the container as Unready

kubectl apply -f readinessprobe-tcp.yaml

Verify Pod
#

# List pods
kubectl get pods

# Shell output: (Wait till READY)
NAME                       READY   STATUS    RESTARTS   AGE
nginx-readinessprobe-tcp   1/1     Running   0          31s

Verify Readiness Status
#

# List pod details
kubectl describe pod nginx-readinessprobe-tcp

# Shell output:
Containers:
  nginx-readinessprobe:
    Container ID:   containerd://f1ca30235e5685294e356f01ef694e0952f0e002dc56740fe4a1b4c2b7d876e8
    Image:          nginx:latest
    Image ID:       docker.io/library/nginx@sha256:447a8665cc1dab95b1ca778e162215839ccbb9189104c79d7ec3a81e14577add
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Fri, 16 Aug 2024 12:03:51 +0000
    Ready:          True
    Restart Count:  0
    Readiness:      tcp-socket :80 delay=15s timeout=1s period=10s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cv67l (ro)

# Kubernetes is now checking the pod's readiness using the specified tcpSocket probe on port "80"
Readiness:      tcp-socket :80 delay=15s timeout=1s period=10s #success=1 #failure=3

Details:

delay=15s The probe waits 15 seconds before starting the first check
timeout=1s Each check must complete within 1 second
period=10s The check is repeated every 10 seconds
#success=1 The pod needs to pass 1 ongoing check to be marked as ready
#failure=3 The pod can fail up to 3 consecutive checks before being marked as unready

List Logs
#

# List logs
kubectl logs nginx-readinessprobe-tcp

Delete Pod
#

kubectl delete -f readinessprobe-tcp.yaml

TCP Readiness Probe: 2 Pod Version
#

Pod 1 Manifest
#

vi nginx-1.yaml

apiVersion: v1
kind: Pod
metadata:
  name: nginx-1
  labels:
    run: nginx-1
spec:
  containers:
  - name: nginx-1
    image: nginx:latest
    ports:
    - containerPort: 80

---
apiVersion: v1
kind: Service
metadata:
  name: nginx-1
spec:
  selector:
    run: nginx-1
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

kubectl apply -f nginx-1.yaml

Pod 1 ClusterIP
#

# List services
kubectl get svc

# Shell ouput:
NAME         TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.43.0.1     <none>        443/TCP   23m
nginx-1      ClusterIP   10.43.32.31   <none>        80/TCP    22m

Pod 2 Manifest
#

vi nginx-2.yaml

apiVersion: v1
kind: Pod
metadata:
  name: nginx-2
spec:
  containers:
  - name: nginx-2
    image: nginx:latest
    ports:
    - containerPort: 80
    readinessProbe:
      tcpSocket:
        host: 10.43.32.31 # Use ClusterIP of Nginx-1
        port: 80
      initialDelaySeconds: 15
      periodSeconds: 10
      timeoutSeconds: 1
      failureThreshold: 3

kubectl apply -f nginx-2.yaml

Verify Pods
#

# List pods
kubectl get pods

# Shell output:
NAME      READY   STATUS    RESTARTS   AGE
nginx-1   1/1     Running   0          19m
nginx-2   1/1     Running   0          30s

Verify Readiness Probe
#

# Delete pod "nginx-1"
kubectl delete pod nginx-1

# Verify "nginx-2" pod readiness
kubectl get pod nginx-2

# Shell output:
NAME      READY   STATUS    RESTARTS   AGE
nginx-2   0/1     Running   0          6m23s

Explanation:

The nginx-2 pod’s readiness probe will fail if nginx-1 is no longer available, because the probe is configured to check connectivity to nginx-1 on port 80.
When the readiness probe fails, Kubernetes will mark the nginx-2 pod as not ready. The pod is still running, but it won’t receive any traffic from services that rely on pod readiness.

Delete Pod
#

kubectl delete -f nginx-2.yaml

HTTP Readiness Probe
#

Pod Manifest
#

vi readinessprobe-http.yaml

apiVersion: v1
kind: Pod
metadata:
  name: nginx-readinessprobe-http
spec:
  containers:
  - name: nginx-readinessprobe
    image: nginx:latest
    ports:
    - containerPort: 80
    readinessProbe:
      httpGet:
        path: /
        port: 80
      initialDelaySeconds: 15  # Time after the container has started before readiness probes are initiated
      periodSeconds: 10 # Frequency in seconds with which to perform the probe
      timeoutSeconds: 1  # How long to wait for a response
      failureThreshold: 3  # Number of failures before marking the container as Unready

kubectl apply -f readinessprobe-http.yaml

Verify Pod
#

# List pods
kubectl get pods

# Shell output: (Wait till READY)
NAME                        READY   STATUS    RESTARTS   AGE
nginx-readinessprobe-http   1/1     Running   0          40s

Verify Readiness Status
#

# List pod details
kubectl describe pod nginx-readinessprobe-http

# Shell output:
Containers:
  nginx-readinessprobe:
    Container ID:   containerd://6aef76936106193c27a877f281a4532d2d24ee46f1954e0fceeba15afe645f8f
    Image:          nginx:latest
    Image ID:       docker.io/library/nginx@sha256:447a8665cc1dab95b1ca778e162215839ccbb9189104c79d7ec3a81e14577add
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Fri, 16 Aug 2024 12:25:35 +0000
    Ready:          True
    Restart Count:  0
    Readiness:      http-get http://:80/ delay=15s timeout=1s period=10s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dsbcg (ro)

# Kubernetes does perform an HTTP GET request on port 80, targeting the root path /
Readiness:      http-get http://:80/ delay=15s timeout=1s period=10s #success=1 #failure=3

Details:

delay=15s The probe waits 15 seconds before starting the first check
timeout=1s Each check must complete within 1 second
period=10s The check is repeated every 10 seconds
#success=1 The pod needs to pass 1 ongoing check to be marked as ready
#failure=3 The pod can fail up to 3 consecutive checks before being marked as unready

Delete Pod
#

kubectl delete -f readinessprobe-http.yaml

Startup Probes
#

Overview
#

Verifies whether the application in a container has started
Runs before any other probe. Unless it finishes successfully, it disables the other probes
Is only executed at startup, unlike readiness probes, which are run periodically
If a container fails its startup probe, it is killed and follows the pods restartPolicy

Example: Command Probe
#

Deploy Pod
#

Allows to run a custom command inside your container during startup
Used to check if a specific process or action within the container has been completed successfully

vi startupprobe-command-probe.yaml

apiVersion: v1
kind: Pod
metadata:
  name: nginx-startupprobe-command-probe
spec:
  containers:
  - name: nginx-livenessprobe
    image: nginx:latest
    ports:
    - containerPort: 80
    startupProbe:
      exec:
        command: ["curl", "-f", "http://localhost"]
      initialDelaySeconds: 15 # Seconds before probe should be active
      periodSeconds: 5 # Frequency of checks once started
      failureThreshold: 3 # Number of times the probe can fail before the container failings
      successThreshold: 1 # Number of times the probe must succeed before the container is healthy
      timeoutSeconds: 1 # Number of seconds to which the probe finally times out

command: ["curl", "-f", "http://localhost"] Checks if the container is ready to start serving traffic. -f Makes curl “fail silently” on server errors.

kubectl apply -f startupprobe-command-probe.yaml

Verify the Startup Probe
#

List the Pod:

# List pods
kubectl get pods

# Shell output:
NAME                               READY   STATUS    RESTARTS   AGE
nginx-startupprobe-command-probe   1/1     Running   0          38s

# Check the pod logs:
kubectl logs nginx-startupprobe-command-probe

# Shell output:
127.0.0.1 - - [06/Nov/2024:09:49:27 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/7.88.1" "-"

200 HTTP status code for success.
615 Size of the response body in bytes.

Delete the Pod
#

# Delete the pod
kubectl delete pod nginx-startupprobe-command-probe

Example: TCP Probe
#

Checks for the readiness of the container by establishing a TCP connection to a specific port

Deploy Pod
#

vi startupprobe-tcp-probe.yaml

apiVersion: v1
kind: Pod
metadata:
  name: postgresss-startupprobe-tcp-probe
spec:
  containers:
  - name: postgresss-database
    image: postgres:latest
    ports:
      - containerPort: 5432
    env:
      - name: POSTGRES_PASSWORD
        value: "my-secure-pw"
    startupProbe:
      tcpSocket:
        port: 5432
      initialDelaySeconds: 15 # Seconds before probe should be active
      periodSeconds: 5 # Frequency of checks once started
      failureThreshold: 3 # Number of times the probe can fail before the container failings
      successThreshold: 1 # Number of times the probe must succeed before the container is healthy
      timeoutSeconds: 1 # Number of seconds to which the probe finally times out

kubectl apply -f startupprobe-tcp-probe.yaml

Verify the TCP Probe
#

List the Pod:

# List pods
kubectl get pods

# Shell output:
NAME                                READY   STATUS    RESTARTS   AGE
postgresss-startupprobe-tcp-probe   1/1     Running   0          50s

Note: The TCP startup probe will not show up in the logs, but the readiness of the pod verifies that the probe worked.

Delete the Pod
#

# Delete the pod
kubectl delete pod postgresss-startupprobe-tcp-probe

Example: HTTP Probe
#

Makes a HTTP requests to a specific endpoint within the application.

Deploy Pod
#

vi startupprobe-http-probe.yaml

apiVersion: v1
kind: Pod
metadata:
  name: echo-startupprobe-tcp-probe
spec:
  containers:
  - name: echo-server
    image: ealen/echo-server
    ports:
      - containerPort: 80
    startupProbe:
      httpGet:
        path: /health
        port: 80
      initialDelaySeconds: 15 # Seconds before probe should be active
      periodSeconds: 5 # Frequency of checks once started
      failureThreshold: 3 # Number of times the probe can fail before the container failings
      successThreshold: 1 # Number of times the probe must succeed before the container is healthy
      timeoutSeconds: 1 # Number of seconds to which the probe finally times out

kubectl apply -f startupprobe-http-probe.yaml

Verify the HTTP Probe
#

List the Pod:

# List pods
kubectl get pods

# Shell output:
NAME                          READY   STATUS    RESTARTS   AGE     IP           NODE      NOMINATED NODE   READINESS GATES
echo-startupprobe-tcp-probe   1/1     Running   0          2m36s   10.0.1.232   ubuntu2   <none>           <none>

curl http://10.0.1.232:80/health

# Check the pod logs:
kubectl logs echo-startupprobe-tcp-probe

# Shell output:
Listening on port 80.
{"name":"echo-server","hostname":"echo-startupprobe-tcp-probe","pid":1,"level":30,"host":{"hostname":"10.0.1.232","ip":"::ffff:10.0.1.197","ips":[]},"http":{"method":"GET","baseUrl":"","originalUrl":"/health","protocol":"http"},"request":{"params":{},"query":{},"cookies":{},"body":{},"headers":{"host":"10.0.1.232:80","user-agent":"kube-probe/1.28","accept":"*/*","connection":"close"}},"environment":{"PATH":"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","HOSTNAME":"echo-startupprobe-tcp-probe","NODE_VERSION":"20.11.0","YARN_VERSION":"1.22.19","KUBERNETES_PORT_443_TCP_PROTO":"tcp","KUBERNETES_PORT_443_TCP_PORT":"443","KUBERNETES_PORT_443_TCP_ADDR":"10.96.0.1","KUBERNETES_SERVICE_HOST":"10.96.0.1","KUBERNETES_SERVICE_PORT":"443","KUBERNETES_SERVICE_PORT_HTTPS":"443","KUBERNETES_PORT":"tcp://10.96.0.1:443","KUBERNETES_PORT_443_TCP":"tcp://10.96.0.1:443","HOME":"/root"},"msg":"Wed, 06 Nov 2024 10:18:01 GMT | [GET] - http://10.0.1.232:80/health","time":"2024-11-06T10:18:01.040Z","v":0}

Delete the Pod
#

# Delete the pod
kubectl delete pod echo-startupprobe-tcp-probe

Kubernetes-Components - This article is part of a series.

Part 1: Kubernetes Configuration Manifests: Create Manifests from RAW Output and Dry-Run Command

Part 2: Kubernetes Non-Disruptive & Disruptive Configuration Updates: Kubectl Apply, Edit, Patch & Replace; Update Rollouts and Rollbacks with Set Image Command

Part 3: Kubernetes Pods: Create Pods with Run-Command and YAML Configuration; Single & Multi Container Pods, Port-Forwarding, Find Container on Worker Node

Part 4: Kubernetes Pods: Init & Sidecar Container Overview, Init Container Examples

Part 6: Kubernetes Services: Example ClusterIP, NodePort & LoadBalancer Services with Expose-Command and YAML Configuration; Service for External Endpoint

Part 8: Kubernetes Monitoring & Logs: Monitor Applications with top, Monitor Events Pod specific and Cluster wide, Container STDOUT and STDERR Logs

Part 9: Kubernetes Security: Immutable Deployment - Deploy Container with ReadOnly-Filesystem and Writable-Volume

Part 10: Kubernetes Security: Pod Security Admission (PSA) - Overview, Enforce Pod Security Standard at a Namespace; Example Nginx Pod SecurityContext for Restricted PSS

Part 12: Kubernetes Secrets: Opaque Secret Configuration, Pod Examples with Environment Variable Secrets and Volume Secrets; SSH Authentication Secret with Pod Example

Part 13: Kubernetes ConfigMaps: Mount ConfigMap to Pod as Volume, Mount ConfigMap as Environment Variable

Part 14: Kubernetes Sets - ReplicaSets & DaemonSets: Overview, Example ReplicaSet, Example DaemonSet with and without NodeSelector / Node Labeling

Part 15: Kubernetes Sets - StatefulSets: Difference between StatefulSet & ReplcaSet; StatefulSet Example with VolumeClaimTemplate and Stateless Service

Part 17: Kubernetes Configuration Management - Helm Charts: Create a Custom Helm Chart

Part 18: Kubernetes Configuration Management - Kustomize: Kustomize Example

Part 19: Kubernetes Networking - Network Policies: Ingress and Egress Policy Examples

Part 20: This Article

Part 21: Kubernetes Networking - DNS: CoreDNS custom Hosts entry (K3s & K8s Version), Deployment with custom Hosts entry; Backup and Restore the CoreDNS ConfigMap; DNS Troubleshooting

Part 22: Kubernetes Horizontal Pod Autoscaling: Install Kubernetes Metrics Server, Example Deployment with Horizontal Pod Autoscaler (HPA)

Part 23: Kubernetes Jobs: Jobs Overview, Basic Non-parallel & Parallel Job Examples; CronJob & RBAC Example that Restarts a Deployment

Part 24: Kubernetes Role Based Access Control (RBAC): RBAC Overview, Create Service Account, Example Role and RoleBinding

Part 25: Kubernetes Kubeconfig: Create example Kubeconfig with new (RBAC) Service Account and ClusterRole / ClusterRole Binding

Part 26: Kubernetes Etcd Snapshot: Etcd Snapshot and Restore with Etcdctl, Verify Etcd Member Health; Etcdctl Commands

Part 27: Kubernetes Troubleshooting: Cluster, Nodes, Pods & Kubelet

Kubernetes Probes #

Overview #

Liveness Probe #

TCP Liveness Probe #

Pod Manifest #

Verify Pod #

Verify Liveness Status #

Verify Liveness Probe #

Delete Pod #

Readiness Probe #

TCP Readiness Probe #

Pod Manifest #

Verify Pod #

Verify Readiness Status #

List Logs #

Delete Pod #

TCP Readiness Probe: 2 Pod Version #

Pod 1 Manifest #

Pod 1 ClusterIP #

Pod 2 Manifest #

Verify Pods #

Verify Readiness Probe #

Delete Pod #

HTTP Readiness Probe #

Pod Manifest #

Verify Pod #

Verify Readiness Status #

Delete Pod #

Startup Probes #

Overview #

Example: Command Probe #

Deploy Pod #

Verify the Startup Probe #

Delete the Pod #

Example: TCP Probe #

Deploy Pod #

Verify the TCP Probe #

Delete the Pod #

Example: HTTP Probe #

Deploy Pod #

Verify the HTTP Probe #

Delete the Pod #

Kubernetes Probes
#

Overview
#

Liveness Probe
#

TCP Liveness Probe
#

Pod Manifest
#

Verify Pod
#

Verify Liveness Status
#

Verify Liveness Probe
#

Delete Pod
#

Readiness Probe
#

TCP Readiness Probe
#

Pod Manifest
#

Verify Pod
#

Verify Readiness Status
#

List Logs
#

Delete Pod
#

TCP Readiness Probe: 2 Pod Version
#

Pod 1 Manifest
#

Pod 1 ClusterIP
#

Pod 2 Manifest
#

Verify Pods
#

Verify Readiness Probe
#

Delete Pod
#

HTTP Readiness Probe
#

Pod Manifest
#

Verify Pod
#

Verify Readiness Status
#

Delete Pod
#

Startup Probes
#

Overview
#

Example: Command Probe
#

Deploy Pod
#

Verify the Startup Probe
#

Delete the Pod
#

Example: TCP Probe
#

Deploy Pod
#

Verify the TCP Probe
#

Delete the Pod
#

Example: HTTP Probe
#

Deploy Pod
#

Verify the HTTP Probe
#

Delete the Pod
#