↓Skip to main content

Enterprise K8s - OpenShift: Single Node Cluster Deployment based on vSphere with CLI-based Local Agent, Static IPv4 Network Configuration; Deploy Image Registry, Example Deployment from Cluster Web Console

1155 words·
OpenShift Kubernetes vSphere VMware RHEL
Table of Contents
OpenShift - This article is part of a series.
Part 1: Enterprise K8s - OpenShift: Single Node Cluster Deployment based on vSphere with Interactive Installer, DHCP Network Configuration
Part 2: Enterprise K8s - OpenShift: Multi Node Cluster Deployment based on vSphere with Interactive Installer, Static IPv4 Network Configuration, Example Deployment with Ingress
Part 3: This Article
Part 4: Enterprise K8s - OpenShift: Multi Node Cluster Deployment based on vSphere with CLI-based Local Agent, Static IPv4 Network Configuration, HAproxy Load Balancer

Overview
#

I’m using a Red Hat Enterprise Linux 9.4 (Plow) server to run the OpenShift Installer.


Deploy OpenShift Cluster
#

vSphere
#

Create VM
#

Create a VM in vSphere with the following specs:

  • 16 GB RAM

  • 8 CPU Cores

  • 2 Disks with each 120 GB storage

  • Add the following configuration parameters

# Name
disk.EnableUUID

# Value
TRUE

For more details about the VM configuration parameters, check out my previous OpenShift posts.

  • Copy the VM MAC address and add it to agent-config.yaml

Prerequisites
#

Install the following packages on the RHEL server, that is used to create the OpenShift ISO:

Register the VM
# 

sudo subscription-manager register --username user.name --password 'my-secure-pw'

Install OpenShift CLI
# 

# Unpack the archive
tar xvf oc-4.16.2-linux.tar.gz

# Move binary
sudo mv oc /usr/local/bin/

# Verify the installation / check version
oc version

Install nmstatectl
# 

# Install nmstatectl
sudo yum -y install /usr/bin/nmstatectl

# Verify installation / check version
nmstatectl --version

OpenShift Installer
#

Installer
#

Download the OpenShift Installer:
https://console.redhat.com/openshift/install/vsphere/agent-based

# Unpack the archive
tar xvf openshift-install-linux.tar.gz

# Create directory and move installer binary
mkdir openshift &&
mv openshift-install openshift/ && cd openshift

Pull Secret & SSH Key
#

Copy your Pull-secret:
https://console.redhat.com/openshift/install/vsphere/agent-based

# Create SSH key pair
ssh-keygen -t rsa -b 4096

# Copy the public SSH key
cat ~/.ssh/id_rsa.pub

Install Config
#

Add the Pull-secret and the public SSH key to the install-config.yaml manifest:

vi install-config.yaml

apiVersion: v1
baseDomain: jklug.local # Define domain
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  replicas: 0
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: openshift-sn
  replicas: 1
metadata:
  name: openshift-sn # Define cluster name
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineNetwork:
  - cidr: 192.168.70.0/24
  networkType: OVNKubernetes 
  serviceNetwork:
  - 172.30.0.0/16
platform:
  none: {}
pullSecret: 'your-pull-secret' # Add your Pull-secret
sshKey: 'your-public-ssh-key' # Add your public SSH key

It should look like this:

apiVersion: v1
baseDomain: jklug.local # Define domain
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  replicas: 0
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: openshift-sn
  replicas: 1
metadata:
  name: openshift-sn # Define cluster name
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineNetwork:
  - cidr: 192.168.70.0/24
  networkType: OVNKubernetes 
  serviceNetwork:
  - 172.30.0.0/16
platform:
  none: {}
pullSecret: '{"auths":{"cloud.openshift.com":{"auth":"some-pullsecret...","email":"..."}}}'
sshKey: 'ssh-rsa some-ssh-key...'

Agent Config
# 

vi agent-config.yaml

apiVersion: v1beta1
kind: AgentConfig
metadata:
  name: openshift-sn # Define cluster name (must be the same as install-config.yaml)
rendezvousIP: 192.168.70.100 # Define VM IP
hosts: 
  - hostname: openshift-sn
    interfaces:
      - name: eth1
        macAddress: 00:50:56:85:0d:d0 # Define VM MAC address
    rootDeviceHints: 
      deviceName: /dev/sda
    networkConfig:
      interfaces:
        - name: eth1
          type: ethernet
          state: up
          mac-address: 00:50:56:85:0d:d0 # Define VM MAC address
          ipv4:
            enabled: true
            address:
              - ip: 192.168.70.100 # Define VM IP
                prefix-length: 24
            dhcp: false
      dns-resolver:
        config:
          server:
            - 192.168.70.1
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: 192.168.70.1
            next-hop-interface: eth1
            table-id: 254

File & Folder Structure
#

Verify the file structure:

├── agent-config.yaml
├── install-config.yaml
└── openshift-install

Generate Installer ISO
# 

# Generate the installer ISO
./openshift-install agent create image

# Shell output:
INFO Configuration has 1 master replicas and 0 worker replicas
INFO The rendezvous host IP (node0 IP) is 192.168.70.100
INFO Extracting base ISO from release payload
INFO Verifying cached file
INFO Using cached Base ISO /home/rhel/.cache/agent/image_cache/coreos-x86_64.iso
INFO Consuming Install Config from target directory
INFO Consuming Agent Config from target directory
INFO Generated ISO at agent.x86_64.iso

File & Folder Structure
# 

├── agent.x86_64.iso
├── auth
│   ├── kubeadmin-password
│   └── kubeconfig
├── openshift-install
└── rendezvousIP

vSphere
#

Upload the ISO & Start VM
#

  • Upload the agent.x86_64.iso to a vSphere Datastore

  • Mount the agent.x86_64.iso to the OpenShift VM

  • Start the OpenShift VM

  • Wait till the OpenShift single node cluster is deployed


Access OpenShift Cluster
#

DNS Entry
# 

192.168.70.100	api.openshift-sn.jklug.local
192.168.70.100	oauth-openshift.apps.openshift-sn.jklug.local
192.168.70.100	console-openshift-console.apps.openshift-sn.jklug.local
192.168.70.100	grafana-openshift-monitoring.apps.openshift-sn.jklug.local
192.168.70.100	thanos-querier-openshift-monitoring.apps.openshift-sn.jklug.local
192.168.70.100	prometheus-k8s-openshift-monitoring.apps.openshift-sn.jklug.local
192.168.70.100	alertmanager-main-openshift-monitoring.apps.openshift-sn.jklug.local

Set Kubeconfig Environment Variable
#

Export the kubeconfig environment variable so that it points to the kubeconfig file.

Temporary:

# Export the kubeconfig environment variable
export KUBECONFIG=/home/rhel/openshift/auth/kubeconfig

Permanent:

# Export the kubeconfig environment variable
echo 'export KUBECONFIG=/home/rhel/openshift/auth/kubeconfig' >> ~/.bashrc

# Apply changes
source /home/rhel/.bashrc

List Cluster Nodes
# 

# List nodes
oc get nodes

# Shell output:
NAME           STATUS   ROLES                         AGE   VERSION
openshift-sn   Ready    control-plane,master,worker   13m   v1.29.6+aba1e8d

SSH Into VM
# 

# SSH into VM
ssh core@192.168.70.100

# Verify the disk scheme
lsblk

# Shell output:
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda      8:0    0   120G  0 disk
sdb      8:16   0   120G  0 disk
├─sdb1   8:17   0     1M  0 part
├─sdb2   8:18   0   127M  0 part
├─sdb3   8:19   0   384M  0 part /boot
└─sdb4   8:20   0 119.5G  0 part /var
                                 /sysroot/ostree/deploy/rhcos/var
                                 /usr
                                 /etc
                                 /
                                 /sysroot
sr0     11:0    1   1.1G  0 rom

Cluster Web Console
#

Open Cluster Web COnsole
# 

# Open the web console
https://console-openshift-console.apps.openshift-sn.jklug.local/

Kubeadmin PW
# 

# User
kubeadmin

# Retrieve pasword
cat auth/kubeadmin-password

# Shell output:
JhNE5-wXaYk-5Rdp8-cUmcv

Image Registry
#

The second disk is available as /dev/sda.

Persistent Volume
# 

# Create manifest
vi image-registry.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-image-registry
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  volumeName: pv-image-registry
  storageClassName: local-storage
  local:
    path: /dev/sda
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - openshift-sn

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-image-registry
  namespace: openshift-image-registry
spec:
  accessModes:
  - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 100Gi
  storageClassName: local-storage
  volumeName: pv-image-registry

# Apply the image registry
oc apply -f image-registry.yaml

Verify Persistent Volume
# 

# Verify the persistent volume
oc get pv

# Shell output:
NAME                CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                         STORAGECLASS    VOLUMEATTRIBUTESCLASS   REASON   AGE
pv-image-registry   100Gi      RWO            Delete           Bound    openshift-image-registry/pvc-image-registry   local-storage   <unset>

# Verify the persistent volume clame
oc get pvc -n openshift-image-registry

# Shell output:
NAME                 STATUS   VOLUME              CAPACITY   ACCESS MODES   STORAGECLASS    VOLUMEATTRIBUTESCLASS   AGE
pvc-image-registry   Bound    pv-image-registry   100Gi      RWO            local-storage   <unset>                 22s

Configure Image Registry Management State
# 

# Configures OpenShift image registry storage
oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"storage":{"pvc":{"claim":"pvc-image-registry"}},"rolloutStrategy":"Recreate","replicas":1,"managementState":"Managed"}}'

# Shell output:
config.imageregistry.operator.openshift.io/cluster patched

Verify Image Registry Pods
# 

# List pods in namespace "openshift-image-registry" 
oc get pods -n openshift-image-registry

# Shell output:
NAME                                              READY   STATUS    RESTARTS   AGE
cluster-image-registry-operator-8dbf56445-whkqt   1/1     Running   2          15h
image-registry-6779b4d954-tpqsh                   1/1     Running   0          94s
node-ca-6dvp6                                     1/1     Running   2          15h

Example Deployment from Cluster Web Console
#

Create Apache Deployment
#

  • Switch to “Developer” view

  • Go to: “Project: All Projects” > “Create Project”

  • Define a project name, for example apache-example
  • Click “+Add”
  • Click “All services”
  • Select “Apache HTTP Server”
  • Click “Create”

  • Click “Try sample”

  • Click “Create”

  • Open the location in a browser
# Create a hosts entry
192.168.70.100 apache-example-apache-example.apps.openshift-sn.jklug.local

# Open the URL
https://apache-example-apache-example.apps.openshift-sn.jklug.local/

Verify Pods with OpenShift CLI
# 

# List pods in "apache-example" namespace
oc get pods -n apache-example

# Shell output:
NAME                              READY   STATUS      RESTARTS   AGE
apache-example-1-build            0/1     Completed   0          112s
apache-example-64f8c65dbb-jwncd   1/1     Running     0          112s

# List services in "apache-example" namespace
oc get svc -n apache-example

# Shell output:
NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
apache-example   ClusterIP   172.30.134.33   <none>        8080/TCP,8443/TCP   16m

Links # 

# Install nmstate
https://nmstate.io/user/install.html

# Image Registry
https://docs.openshift.com/container-platform/4.14/registry/configuring_registry_storage/configuring-registry-storage-baremetal.html#installation-registry-storage-non-production_configuring-registry-storage-baremetal
OpenShift - This article is part of a series.
Part 1: Enterprise K8s - OpenShift: Single Node Cluster Deployment based on vSphere with Interactive Installer, DHCP Network Configuration
Part 2: Enterprise K8s - OpenShift: Multi Node Cluster Deployment based on vSphere with Interactive Installer, Static IPv4 Network Configuration, Example Deployment with Ingress
Part 3: This Article
Part 4: Enterprise K8s - OpenShift: Multi Node Cluster Deployment based on vSphere with CLI-based Local Agent, Static IPv4 Network Configuration, HAproxy Load Balancer