↓Skip to main content

Enterprise K8s - OpenShift: Multi Node Cluster Deployment based on vSphere with CLI-based Local Agent, Static IPv4 Network Configuration, HAproxy Load Balancer

1694 words·
OpenShift Kubernetes vSphere VMware HAproxy DNSmasq RHEL
Table of Contents
OpenShift - This article is part of a series.
Part 1: Enterprise K8s - OpenShift: Single Node Cluster Deployment based on vSphere with Interactive Installer, DHCP Network Configuration
Part 2: Enterprise K8s - OpenShift: Multi Node Cluster Deployment based on vSphere with Interactive Installer, Static IPv4 Network Configuration, Example Deployment with Ingress
Part 3: Enterprise K8s - OpenShift: Single Node Cluster Deployment based on vSphere with CLI-based Local Agent, Static IPv4 Network Configuration; Deploy Image Registry, Example Deployment from Cluster Web Console
Part 4: This Article

Overview
#

I’m using a Red Hat Enterprise Linux 9.4 (Plow) server to run the OpenShift Installer and also to deploy the HAproxy load balancer.

192.168.70.20 # OpenShift Installer, HAproxy Load Balancer
192.168.70.2 # DNSmasq DNS Server

192.168.70.90 master-01.openshift-mn.jklug.local # master-01
192.168.70.91 master-02.openshift-mn.jklug.local # master-02
192.168.70.92 master-03.openshift-mn.jklug.local # master-03
192.168.70.93 worker-01.openshift-mn.jklug.local # worker-01
192.168.70.94 worker-02.openshift-mn.jklug.local # worker-01

Deploy OpenShift Cluster
#

vSphere
#

Create VMs
#

Create 5 VMs in vSphere with the following specs:

Master Nodes:

  • 16 GB RAM

  • 8 CPU Cores

  • 120 GB storage

Worker Nodes:

  • 8 GB RAM

  • 4 CPU Cores

  • 120 GB storage

Add the following configuration parameters to all VMs:

# Name
disk.EnableUUID

# Value
TRUE

For more details about the VM configuration parameters, check out my previous OpenShift posts.

  • Copy the VM MAC address and add it to agent-config.yaml
# VM      MAC Address
master-01 00:50:56:85:3e:5d
master-02 00:50:56:85:ee:a5
master-03 00:50:56:85:fb:78
worker-01 00:50:56:85:ca:4d
worker-02 00:50:56:85:92:42

Prerequisites
#

Install the following packages on the RHEL server, that is used to create the OpenShift ISO:

Register the VM
# 

sudo subscription-manager register --username user.name --password 'my-secure-pw'

Install OpenShift CLI
# 

# Unpack the archive
tar xvf oc-4.16.2-linux.tar.gz

# Move binary
sudo mv oc /usr/local/bin/

# Verify the installation / check version
oc version

Install nmstatectl
# 

# Install nmstatectl
sudo yum -y install /usr/bin/nmstatectl

# Verify installation / check version
nmstatectl --version

DNS Entries
#

Make sure the DNS server can resolve the following DNS & reverse DNS entries:

192.168.70.90 master-01.openshift-mn.jklug.local
192.168.70.91 master-02.openshift-mn.jklug.local
192.168.70.92 master-03.openshift-mn.jklug.local
192.168.70.93 worker-01.openshift-mn.jklug.local
192.168.70.94 worker-02.openshift-mn.jklug.local

192.168.70.20 api.openshift-mn.jklug.local.
192.168.70.20 api-int.openshift-mn.jklug.local.
192.168.70.20 *.apps.openshift-mn.jklug.local.

I’m using the following DNSmasq configuration:

domain-needed
bogus-priv
expand-hosts
 
listen-address=127.0.0.1
listen-address=192.168.70.2 # Define static IP of interface
bind-interfaces

server=8.8.8.8 # Define public DNS server
server=8.8.4.4 # Define public DNS server

# Define DNS entries: Forward DNS
address=/master-01.openshift-mn.jklug.local/192.168.70.90
address=/master-02.openshift-mn.jklug.local/192.168.70.91
address=/master-03.openshift-mn.jklug.local/192.168.70.92
address=/worker-01.openshift-mn.jklug.local/192.168.70.93
address=/worker-02.openshift-mn.jklug.local/192.168.70.94
address=/api.openshift-mn.jklug.local/192.168.70.20
address=/api-int.openshift-mn.jklug.local/192.168.70.20
address=/.apps.openshift-mn.jklug.local/192.168.70.20


# Reverse DNS
ptr-record=90.70.168.192.in-addr.arpa,master-01.openshift-mn.jklug.local
ptr-record=91.70.168.192.in-addr.arpa,master-02.openshift-mn.jklug.local
ptr-record=92.70.168.192.in-addr.arpa,master-03.openshift-mn.jklug.local
ptr-record=93.70.168.192.in-addr.arpa,worker-01.openshift-mn.jklug.local
ptr-record=94.70.168.192.in-addr.arpa,worker-02.openshift-mn.jklug.local
ptr-record=20.70.168.192.in-addr.arpa,api.openshift-mn.jklug.local
ptr-record=20.70.168.192.in-addr.arpa,api-int.openshift-mn.jklug.local
ptr-record=20.70.168.192.in-addr.arpa,apps.openshift-mn.jklug.local

Define DNS Server
# 

# List network connection names
nmcli con show

# Shell output:
NAME    UUID                                  TYPE      DEVICE
ens160  4d4a23ec-d12b-3109-a57b-f815dbcc4c6f  ethernet  ens160
lo      805a327f-4c6e-4fa3-b35f-607627d13e7d  loopback  lo

# Edit network config
sudo vi /etc/NetworkManager/system-connections/ens160.nmconnection

# Define DNS server
[ipv4]
address1=192.168.70.20/24,192.168.70.1
dns=192.168.70.2;8.8.8.8;
method=manual

# Restart NetworkManager
sudo systemctl restart NetworkManager

Install HAproxy
# 

# Install HAproxy
sudo yum install haproxy -y

# Create a backup of the config
sudo cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy_backup.cfg

# Edit the HAproxy configuration
sudo vi /etc/haproxy/haproxy.cfg

global
  log         127.0.0.1 local2
  pidfile     /var/run/haproxy.pid
  maxconn     4000
  daemon
defaults
  mode                    http
  log                     global
  option                  dontlognull
  option http-server-close
  option                  redispatch
  retries                 3
  timeout http-request    10s
  timeout queue           1m
  timeout connect         10s
  timeout client          1m
  timeout server          1m
  timeout http-keep-alive 10s
  timeout check           10s
  maxconn                 3000
listen api-server-6443 
  bind *:6443
  mode tcp
  server master-01 master-01.openshift-mn.jklug.local:6443 check inter 1s
  server master-02 master-02.openshift-mn.jklug.local:6443 check inter 1s
  server master-03 master-03.openshift-mn.jklug.local:6443 check inter 1s
listen machine-config-server-22623 
  bind *:22623
  mode tcp
  server master-01 master-01.openshift-mn.jklug.local:22623 check inter 1s
  server master-02 master-02.openshift-mn.jklug.local:22623 check inter 1s
  server master-03 master-03.openshift-mn.jklug.local:22623 check inter 1s
listen ingress-router-443 
  bind *:443
  mode tcp
  balance source
  server worker-01 worker-01.openshift-mn.jklug.local:443 check inter 1s
  server worker-02 worker-02.openshift-mn.jklug.local:443 check inter 1s
listen ingress-router-80 
  bind *:80
  mode tcp
  balance source
  server worker-01 worker-01.openshift-mn.jklug.local:80 check inter 1s
  server worker-02 worker-02.openshift-mn.jklug.local:80 check inter 1s

# Update firewall settings
sudo firewall-cmd --add-port=6443/tcp --permanent # kube-api-server on control plane
sudo firewall-cmd --add-port=22623/tcp --permanent # machine-config server
sudo firewall-cmd --add-service=http --permanent
sudo firewall-cmd --add-service=https --permanent
sudo firewall-cmd --add-port=9000/tcp --permanent # haproxy stats
sudo firewall-cmd --reload

# Enable & Start HAproxy
sudo setsebool -P haproxy_connect_any 1
sudo systemctl enable haproxy
sudo systemctl start haproxy

# Verify HAproxy status
sudo systemctl status haproxy --no-pager

OpenShift Installer
#

Installer
#

Download the OpenShift Installer:
https://console.redhat.com/openshift/install/vsphere/agent-based

# Unpack the archive
tar xvf openshift-install-linux.tar.gz

# Create directory and move installer binary
mkdir openshift-mn &&
mv openshift-install openshift-mn/ && cd openshift-mn

Pull Secret & SSH Key
#

Copy your Pull-secret:
https://console.redhat.com/openshift/install/vsphere/agent-based

# Create SSH key pair
ssh-keygen -t rsa -b 4096

# Copy the public SSH key
cat ~/.ssh/id_rsa.pub

Install Config
#

Add the Pull-secret and the public SSH key to the install-config.yaml manifest:

vi install-config.yaml

apiVersion: v1
baseDomain: jklug.local # Define domain
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  replicas: 2
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  replicas: 3
metadata:
  name: openshift-mn # Define cluster name
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineNetwork:
  - cidr: 192.168.70.0/24
  networkType: OVNKubernetes 
  serviceNetwork:
  - 172.30.0.0/16
platform:
  none: {}
pullSecret: 'your-pull-secret' # Add your Pull-secret
sshKey: 'your-public-ssh-key' # Add your public SSH key

It should look like this:

apiVersion: v1
baseDomain: jklug.local # Define domain
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  replicas: 2
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  replicas: 3
metadata:
  name: openshift-mn # Define cluster name
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineNetwork:
  - cidr: 192.168.70.0/24
  networkType: OVNKubernetes 
  serviceNetwork:
  - 172.30.0.0/16
platform:
  none: {}
pullSecret: '{"auths":{"cloud.openshift.com":{"some-pull-secret..."}}}'
sshKey: 'ssh-rsa some-ssh-key...'

Agent Config
# 

vi agent-config.yaml

apiVersion: v1alpha1
kind: AgentConfig
metadata:
  name: openshift-mn # Define cluster name (must be the same as install-config.yaml)
rendezvousIP: 192.168.70.90 # Define IP of master-01
hosts:
  - hostname: master-01
    role: master
    interfaces:
      - name: eth1
        macAddress: 00:50:56:85:3e:5d # Define master-01 MAC
    networkConfig:
      interfaces:
        - name: eth1
          type: ethernet
          state: up
          mac-address: 00:50:56:85:3e:5d # Define master-01 MAC
          ipv4:
            enabled: true
            address:
              - ip: 192.168.70.90 # Define master-01 IP
                prefix-length: 24 
            dhcp: false
      dns-resolver:
        config:
          server:
            - 192.168.70.2
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: 192.168.70.1
            next-hop-interface: eth1
            table-id: 254
  - hostname: master-02
    role: master
    interfaces:
      - name: eth1
        macAddress: 00:50:56:85:ee:a5 # Define master-02 MAC
    networkConfig:
      interfaces:
        - name: eth1
          type: ethernet
          state: up
          mac-address: 00:50:56:85:ee:a5 # Define master-02 MAC
          ipv4:
            enabled: true
            address:
              - ip: 192.168.70.91 # Define master-02 IP
                prefix-length: 24 
            dhcp: false
      dns-resolver:
        config:
          server:
            - 192.168.70.2
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: 192.168.70.1
            next-hop-interface: eth1
            table-id: 254
  - hostname: master-03
    role: master
    interfaces:
      - name: eth1
        macAddress: 00:50:56:85:fb:78 # Define master-03 MAC
    networkConfig:
      interfaces:
        - name: eth1
          type: ethernet
          state: up
          mac-address: 00:50:56:85:fb:78 # Define master-03 MAC
          ipv4:
            enabled: true
            address:
              - ip: 192.168.70.92 # Define master-03 IP
                prefix-length: 24 
            dhcp: false
      dns-resolver:
        config:
          server:
            - 192.168.70.2
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: 192.168.70.1
            next-hop-interface: eth1
            table-id: 254
  - hostname: worker-01
    role: worker
    interfaces:
      - name: eth1
        macAddress: 00:50:56:85:ca:4d # Define worker-01 MAC
    networkConfig:
      interfaces:
        - name: eth1
          type: ethernet
          state: up
          mac-address: 00:50:56:85:ca:4d # Define worker-01 MAC
          ipv4:
            enabled: true
            address:
              - ip: 192.168.70.93 # Define worker-01 IP
                prefix-length: 24 
            dhcp: false
      dns-resolver:
        config:
          server:
            - 192.168.70.2
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: 192.168.70.1
            next-hop-interface: eth1
            table-id: 254
  - hostname: worker-02
    role: worker
    interfaces:
      - name: eth1
        macAddress: 00:50:56:85:92:42 # Define worker-02 MAC
    networkConfig:
      interfaces:
        - name: eth1
          type: ethernet
          state: up
          mac-address: 00:50:56:85:92:42 # Define worker-02 MAC
          ipv4:
            enabled: true
            address:
              - ip: 192.168.70.94 # Define worker-02 IP
                prefix-length: 24 
            dhcp: false
      dns-resolver:
        config:
          server:
            - 192.168.70.2
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: 192.168.70.1
            next-hop-interface: eth1
            table-id: 254

File & Folder Structure
#

Verify the file structure:

├── agent-config.yaml
├── install-config.yaml
└── openshift-install

Generate Installer ISO
# 

# Generate the installer ISO
./openshift-install agent create image

# Shell output:
INFO Configuration has 3 master replicas and 2 worker replicas
INFO The rendezvous host IP (node0 IP) is 192.168.70.90
INFO Extracting base ISO from release payload
INFO Base ISO obtained from release and cached at [/home/rhel/.cache/agent/image_cache/coreos-x86_64.iso]
INFO Consuming Install Config from target directory
INFO Consuming Agent Config from target directory
INFO Generated ISO at agent.x86_64.iso

File & Folder Structure
# 

├── agent.x86_64.iso
├── auth
│   ├── kubeadmin-password
│   └── kubeconfig
├── openshift-install
└── rendezvousIP

vSphere
#

Upload the ISO & Start VM
#

  • Upload the agent.x86_64.iso to a vSphere Datastore

  • Mount the agent.x86_64.iso to the OpenShift VM

  • Start the OpenShift VM

  • Wait till the OpenShift single node cluster is deployed


Track Cluster Installation
# 

# Track the cluster installation process
./openshift-install agent wait-for install-complete

# Shell output:
INFO Bootstrap Kube API Initialized
INFO Cluster installation in progress
INFO Host: master-03, reached installation stage Rebooting
INFO Host: master-01, reached installation stage Waiting for bootkube
INFO Host: master-03, reached installation stage Done
INFO Node master-02 has been rebooted 1 times before completing installation
INFO Host: worker-02, reached installation stage Rebooting
INFO Host: worker-01, reached installation stage Rebooting
INFO Host: master-01, reached installation stage Waiting for bootkube: waiting for ETCD bootstrap to be complete
INFO Host: worker-02, reached installation stage Joined
INFO Bootstrap configMap status is complete
INFO Bootstrap is complete
INFO cluster bootstrap is complete
INFO Cluster is installed
INFO Install complete!
INFO To access the cluster as the system:admin user when using 'oc', run
INFO     export KUBECONFIG=/home/rhel/openshift-mn/auth/kubeconfig
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.openshift-mn.jklug.local
INFO Login to the console with user: "kubeadmin", and password: "qMm67-5nqcj-4EDji-aAK5w"

Access OpenShift Cluster
#

Set Kubeconfig Environment Variable
#

Export the kubeconfig environment variable so that it points to the kubeconfig file.

Temporary:

# Export the kubeconfig environment variable
export KUBECONFIG=/home/rhel/openshift-mn/auth/kubeconfig

Permanent:

# Export the kubeconfig environment variable
echo 'export KUBECONFIG=/home/rhel/openshift-mn/auth/kubeconfig' >> ~/.bashrc

# Apply changes
source /home/rhel/.bashrc

List Cluster Nodes
# 

# List nodes
oc get nodes

# Shell output:
NAME        STATUS   ROLES                  AGE   VERSION
master-01   Ready    control-plane,master   11m   v1.29.6+aba1e8d
master-02   Ready    control-plane,master   25m   v1.29.6+aba1e8d
master-03   Ready    control-plane,master   25m   v1.29.6+aba1e8d
worker-01   Ready    worker                 16m   v1.29.6+aba1e8d
worker-02   Ready    worker                 16m   v1.29.6+aba1e8d

# List nodes: More details
oc get nodes -o wide

# Shell output:
NAME        STATUS   ROLES                  AGE   VERSION           INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                                                KERNEL-VERSION                 CONTAINER-RUNTIME
master-01   Ready    control-plane,master   12m   v1.29.6+aba1e8d   192.168.70.90   <none>        Red Hat Enterprise Linux CoreOS 416.94.202407081958-0   5.14.0-427.26.1.el9_4.x86_64   cri-o://1.29.6-3.rhaos4.16.gitfd433b7.el9
master-02   Ready    control-plane,master   25m   v1.29.6+aba1e8d   192.168.70.91   <none>        Red Hat Enterprise Linux CoreOS 416.94.202407081958-0   5.14.0-427.26.1.el9_4.x86_64   cri-o://1.29.6-3.rhaos4.16.gitfd433b7.el9
master-03   Ready    control-plane,master   25m   v1.29.6+aba1e8d   192.168.70.92   <none>        Red Hat Enterprise Linux CoreOS 416.94.202407081958-0   5.14.0-427.26.1.el9_4.x86_64   cri-o://1.29.6-3.rhaos4.16.gitfd433b7.el9
worker-01   Ready    worker                 16m   v1.29.6+aba1e8d   192.168.70.93   <none>        Red Hat Enterprise Linux CoreOS 416.94.202407081958-0   5.14.0-427.26.1.el9_4.x86_64   cri-o://1.29.6-3.rhaos4.16.gitfd433b7.el9
worker-02   Ready    worker                 17m   v1.29.6+aba1e8d   192.168.70.94   <none>        Red Hat Enterprise Linux CoreOS 416.94.202407081958-0   5.14.0-427.26.1.el9_4.x86_64   cri-o://1.29.6-3.rhaos4.16.gitfd433b7.el9

SSH Into VMs
# 

# SSH into VM
ssh core@192.168.70.90
ssh core@192.168.70.91
ssh core@192.168.70.92
ssh core@192.168.70.93
ssh core@192.168.70.94

Verify Cluster in OpenShift Console
#

  • Select the new cluster

Cluster Web Console
#

Open Cluster Web COnsole
# 

# Open the web console
https://console-openshift-console.apps.openshift-mn.jklug.local/

Kubeadmin PW
# 

# User
kubeadmin

# Retrieve pasword
cat auth/kubeadmin-password

# Shell output:
qMm67-5nqcj-4EDji-aAK5w
OpenShift - This article is part of a series.
Part 1: Enterprise K8s - OpenShift: Single Node Cluster Deployment based on vSphere with Interactive Installer, DHCP Network Configuration
Part 2: Enterprise K8s - OpenShift: Multi Node Cluster Deployment based on vSphere with Interactive Installer, Static IPv4 Network Configuration, Example Deployment with Ingress
Part 3: Enterprise K8s - OpenShift: Single Node Cluster Deployment based on vSphere with CLI-based Local Agent, Static IPv4 Network Configuration; Deploy Image Registry, Example Deployment from Cluster Web Console
Part 4: This Article