Ansible AWX: Kubernetes Deployment with AWX Operator, Storage Setup (Storage Class, PersistentVolume with NFS, PersistentVolumeClaim), TLS Encryption with Nginx Ingress and Kubernetes Secret

Table of Contents

Overview
#

AWX offers a web-based user interface for Ansible. I’m providing two PersistentVolume variants, the first uses a directory on Kubernetes node 3 for storage, the second uses a NFS share.

In this tutorial I’m using a Kubernetes K8s cluster with MetalLB deployed with Kubespray on Debian 12 servers:

192.168.30.71 node1 # Controller / Master Node
192.168.30.72 node2 # Controller / Master Node
192.168.30.73 node3 # Worker Node
192.168.30.74 node4 # Worker Node

192.168.30.60 NFS Server

NFS Prerequisites
#

NFS Server Setup
#

AWX Folder Structure
#

Create the folder structure for the PersistentVolume:

# Create folder structure
sudo mkdir -p /srv/nfs/k8s_share/awx

# Change the owner
sudo chmod 777 /srv/nfs/k8s_share/awx

NFS Exports
#

I’m using the following NFS server configuration:

# Install NFS package
sudo apt install nfs-kernel-server

# Open NFS configuration
sudo vi /etc/exports

# NFS configuration: Define the kubernetes nodes
/srv/nfs/k8s_share 192.168.30.71(rw,sync,no_root_squash)
/srv/nfs/k8s_share 192.168.30.72(rw,sync,no_root_squash)
/srv/nfs/k8s_share 192.168.30.73(rw,sync,no_root_squash)
/srv/nfs/k8s_share 192.168.30.74(rw,sync,no_root_squash)

# Restart NFS server
sudo systemctl restart nfs-server

Install NFS on Kubernetes Nodes
#

Install the NFS utilities package on the Kubernetes nodes:

# Install NFS utilities package
sudo apt install nfs-common -y

Verify the NFS connectivity:

# Verify that the NFS server is correctly configured
/usr/sbin/showmount -e 192.168.30.60

Helm
#

Install Helm
#

# Install Helm with script
cd /tmp && curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 &&
chmod +x get_helm.sh &&
./get_helm.sh

# Verify the installation / check version
helm version

Add the AWX Operator Repository
#

# Add the AWX Operator repository
helm repo add awx-operator https://ansible.github.io/awx-operator/

# Update the repository index
helm repo update

Install AWX Operator
#

# Install the AWX Operator
helm install ansible-awx-operator awx-operator/awx-operator -n awx --create-namespace

# Delete the AWX Operator
helm delete ansible-awx-operator awx-operator/awx-operator -n awx

Storage Class, PV & PVC
#

Create Storage Class
#

# Create manifest for storage class
vi local-storage-class.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-storage
  namespace: awx
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: Immediate

# Deploy storage class
kubectl create -f local-storage-class.yaml

# Verify the storage class
kubectl get sc -n awx

# Shell output:
NAME            PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
local-storage   kubernetes.io/no-provisioner   Delete          Immediate           false                  9s

Create PersistentVolume
#

Create Manifest
#

Persistent Volume: Kubernetes Node Directory

# Create manifest for PersistentVolume
vi pv-awx.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: awx-postgres-pv
  namespace: awx
spec:
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /mnt/pv-awx
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node3

Create the following directory on Kubernetes node 3:

# Create directory & set permission
sudo mkdir -p /mnt/pv-awx &&
sudo chmod 777 /mnt/pv-awx

Persistent Volume: NFS Share

# Create manifest for PersistentVolume
vi pv-awx.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: awx-postgres-pv
  namespace: awx
spec:
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  nfs:
    path: /srv/nfs/k8s_share/awx
    server: 192.168.30.60

Deploy Manifest
#

# Deploy the PersistentVolume
kubectl create -f pv-awx.yaml

Create PersistentVolumeClaim
#

# Create manifest for PersistentVolumeClaim
vi pvc-awx.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-15-ansible-awx-postgres-15-0
  namespace: awx
spec:
  storageClassName: local-storage
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 3Gi

# Deploy the PersistentVolumeClaim
kubectl create -f pvc-awx.yaml

Verify Resources
#

# List PV & PVC
kubectl get pv,pvc -n awx

# Shell output:
NAME                               CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                       STORAGECLASS    VOLUMEATTRIBUTESCLASS   REASON   AGE
persistentvolume/awx-postgres-pv   10Gi       RWO            Retain           Bound    awx/postgres-15-ansible-awx-postgres-15-0   local-storage   <unset>                          8s

NAME                                                          STATUS   VOLUME            CAPACITY   ACCESS MODES   STORAGECLASS    VOLUMEATTRIBUTESCLASS   AGE
persistentvolumeclaim/postgres-15-ansible-awx-postgres-15-0   Bound    awx-postgres-pv   10Gi       RWO            local-storage   <unset>                 3s

Delete Resources
#

# Delete PVC
kubectl delete pvc postgres-15-ansible-awx-postgres-15-0 -n awx

# Delete PV
kubectl delete pv awx-postgres-pv

Deploy a AWX Instance
#

YAML Manifest
#

# Create AWX manifest 
vi ansible-awx.yaml

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: ansible-awx # Define AWX instance name
  namespace: awx
spec:
  service_type: nodeport
  postgres_storage_class: local-storage

# Deploy the AWX instance: This takes a view minutes!
kubectl create -f ansible-awx.yaml

# Delete the AWX instance
kubectl delete -f ansible-awx.yaml

Verify Deployment
#

# List pods in "awx" namespace 
kubectl get pods -n awx

# Shell output:
NAME                                               READY   STATUS      RESTARTS   AGE
ansible-awx-migration-24.5.0-5qkzj                 0/1     Completed   0          10m
ansible-awx-postgres-15-0                          1/1     Running     0          12m
ansible-awx-task-896c85586-87p7z                   4/4     Running     0          11m
ansible-awx-web-5fdc956d96-pc8n7                   3/3     Running     0          11m
awx-operator-controller-manager-5fd4474775-2cxfv   2/2     Running     0          40m

AWX Instance Service
#

Find NodePort IP
#

# List services in "awx" namespace
kubectl get svc -n awx

# Shell output:
NAME                                              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
ansible-awx-postgres-15                           ClusterIP   None            <none>        5432/TCP       17m
ansible-awx-service                               NodePort    10.233.46.247   <none>        80:32175/TCP   17m
awx-operator-controller-manager-metrics-service   ClusterIP   10.233.60.200   <none>        8443/TCP       46m

Kubernetes TLS Certificate Secret
#

In this setup I’m using a Let’s Encrypt wildcard certificate.

# Create a Kubernetes secret for the TLS certificate
kubectl create secret tls awx-tls --cert=./fullchain.pem --key=./privkey.pem -n awx

Nginx Ingress for NodePort Service
#

# Create ingress manifest
vi awx-ingress.yml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: awx
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
spec:
  tls:
  - hosts:
    - awx.jklug.work
    secretName: awx-tls
  rules:
  - host: awx.jklug.work
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: ansible-awx-service  # Define AWX ingress service name
            port:
              number: 80

# Deploy ingress resource
kubectl create -f awx-ingress.yml -n awx

# Delete ingress resource
kubectl delete -f awx-ingress.yml -n awx

DNS Entry
#

The DNS entry for the Nginx ingress must point to one of the worker nodes:

192.168.30.73 awx.jklug.work
# Or 
192.168.30.74 awx.jklug.work

AWX Webinterface
#

Access Webinterface
#

# Access the webinterface: Via NodePort
192.168.30.71:32175
192.168.30.72:32175
192.168.30.73:32175
192.168.30.74:32175

# Access the webinterface: Via Ingress
https://awx.jklug.work

Retrieve Password
#

# Default user
admin

# Retrieve the admin password
kubectl get secret ansible-awx-admin-password -o jsonpath="{.data.password}" -n awx | base64 --decode ; echo

# Shell output:
uki2yUchay5yLAXf6IV39RIwxJbfhE1U

Links
#

# GitHub Repository
https://github.com/ansible/awx-operator

# Installation
https://ansible.readthedocs.io/projects/awx-operator/en/latest/installation/basic-install.html
https://ansible.readthedocs.io/projects/awx-operator/en/latest/installation/helm-install-on-existing-cluster.html
https://github.com/ansible/awx/blob/devel/tools/docker-compose/README.md

Overview #

NFS Prerequisites #

NFS Server Setup #

AWX Folder Structure #

NFS Exports #

Install NFS on Kubernetes Nodes #

Helm #

Install Helm #

Add the AWX Operator Repository #

Install AWX Operator #

Storage Class, PV & PVC #

Create Storage Class #

Create PersistentVolume #

Create Manifest #

Deploy Manifest #

Create PersistentVolumeClaim #

Verify Resources #

Delete Resources #

Deploy a AWX Instance #

YAML Manifest #

Verify Deployment #

AWX Instance Service #

Find NodePort IP #

Kubernetes TLS Certificate Secret #

Nginx Ingress for NodePort Service #

DNS Entry #

AWX Webinterface #

Access Webinterface #

Retrieve Password #

Links #