Posted in: Technical Blogs
Experience of installing OpenShift on a virtualised environment
Our Container Platform Consultant Jonathan shares his experience of installing Red Hat OpenShift on a virtualised environment, VMware vSphere, using Installer-Provisioned Infrastructure.
This guide is based on the official Red Hat documentation, including added notes and a description of some of the snags requiring action that aren’t covered in the main docs. Tier 2 offer accredited skills and services around Red Hat OpenShift Container Platform. Discover more about how Tier 2 is able to help:
Written by Jonathan Gazeley, Container Platform Consultant at Tier 2 Consulting.
Today I’m going to share my experience of installing Red Hat OpenShift on a virtualised environment. Red Hat do have their own documentation for this, and this guide is based upon the official docs, and should be read alongside them. I’ll provide links at key points, and I’ll also add a few notes and describe some of the ‘gotchas’ I ran into.
OpenShift comes with an automated installer for supported platforms which means it handles creation and management of the underlying infrastructure (such as VMs, disks, and networks) as well as its own components. This is called Installer-Provisioned Infrastructure, or IPI for short. The automated installer supports many options, including private virtualisation platforms as well as public cloud platforms.
If you want to run on bare metal, or on some other unsupported platform – you can still do this, but you will need to set up the infrastructure yourself. This is called User-Provisioned Infrastructure, or UPI for short.
The platform we have available today is VMware vSphere, which is a supported platform, and so we can benefit from IPI.
Prep
There are a few things we need before we can start:
- A set of privileged credentials for vCenter. It doesn’t have to be admin, but you need to make sure you have all of the required privileges (it’s quite a long list)
- Some network space to build into, e.g. a
/24private network like10.250.130.0/24with internet access - Two pre-allocated IP addresses in this range, with DNS records. Let’s say our base domain is
mybiz.local, so we can choosesandbox.mybiz.localfor our cluster domain. We need to create a fixed record for the API server and a wildcard record for the apps ingress.
10.250.130.1 api.sandbox.mybiz.local
10.250.130.2 *.apps.sandbox.mybiz.localThere are also a couple of things you need to do with Red Hat. You will need a Red Hat account to download the installer, and you will need a pull secret so the installer can download the OpenShift images.
Certificates
If your vCenter server or vSphere nodes have self-signed certificates on them (on a production system, they shouldn’t), you will have to jump through an extra hoop. You can test this out by visiting the vCenter server in your browser. If you don’t get any warnings and the padlock icon is there, you should be fine. If you get any warnings, you’ll need to do these manual trust steps on whatever system you’ll be running the install from.
First you’ll need to retrieve the certificate from the vCenter server and every vSphere node. You can either do this in the browser by browsing to each endpoint and saving the certificate, or you can use a curl command to fetch them:
echo | openssl s_client -showcerts -servername vcenter.mybiz.local -connect vcenter.mybiz.local:443 2>/dev/null > vcenter.mybiz.local.pemNow import these certificates into your system trust store and mark them as trusted. The method for this depends on your local OS, but as I’m on a Mac I followed these instructions for MacOS
Installation
With the installer downloaded and unpacked, and all the prerequisites satisfied, we can now run the installer to deploy the cluster.
When we run the installer, we get interactively prompted for various information. Here we have the output of an example deployment. Most of these fields are related to vCenter and the choices you have will depend on your specific vSphere deployment.
$ ./openshift-install create cluster
? Platform vsphere
? vCenter vcenter.mybiz.local
? Username openshift@vcenter.mybiz.local
? Password [? for help] ************
INFO Connecting to vCenter vcenter.mybiz.local
? Datacenter MyDatacenter
? Cluster MyCluster
? Default Datastore vsanDatastore
? Network MyNetwork
? Virtual IP Address for API 10.250.130.1
? Virtual IP Address for Ingress 10.250.130.2
? Base Domain mybiz.local
? Cluster Name sandbox
? Pull Secret [? for help] ******Deployment will take around half an hour, depending on your configuration. When it is complete, you should see some output like this.
INFO Install complete!
INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/home/myuser/install_dir/auth/kubeconfig'
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.sandbox.mybiz.local
INFO Login to the console with user: "kubeadmin", and password: "4vYBz-Ee6gm-ymBZj-Wt5AL"
INFO Time elapsed: 36m22sMake a note of the username and password given, as these are required for the web interface. One of the first tasks you should do is set up user accounts and some form of user management, to avoid logging in as the cluster admin. Look after the kubeconfig file as it contains root credentials for the cluster. It can be used to connect to the cluster with the command-line tool oc.
Scaling
One of the key benefits of installing OpenShift with IPI is that it manages the underlying infrastructure as well as the Kubernetes components.
All Kubernetes distributions have the concept of Nodes, and you can view the nodes in your cluster like this:
$ oc get nodes
NAME STATUS ROLES AGE VERSION
sandbox-xfvg5-master-0 Ready master 20d v1.24.6+5658434
sandbox-xfvg5-master-1 Ready master 20d v1.24.6+5658434
sandbox-xfvg5-master-2 Ready master 20d v1.24.6+5658434
sandbox-xfvg5-worker-9c9jf Ready worker 20d v1.24.6+5658434
sandbox-xfvg5-worker-fd6cn Ready worker 20d v1.24.6+5658434
sandbox-xfvg5-worker-vx6wb Ready worker 20d v1.24.6+5658434Where OpenShift differs is that it also has the concept of Machines, which represent the hardware or VM that the Nodes are actually running on:
$ oc get machines
NAME PHASE TYPE REGION ZONE AGE
sandbox-xfvg5-master-0 Running 20d
sandbox-xfvg5-master-1 Running 20d
sandbox-xfvg5-master-2 Running 20d
sandbox-xfvg5-worker-9c9jf Running 20d
sandbox-xfvg5-worker-fd6cn Running 20d
sandbox-xfvg5-worker-vx6wb Running 20dMachines (excluding Masters) are grouped into MachineSets:
$ oc get machinesets
NAME DESIRED CURRENT READY AVAILABLE AGE
sandbox-xfvg5-worker 3 3 3 3 20dIf this was a bare-metal or UPI cluster, this would be about as far as you could go. But in an IPI cluster, OpenShift can manage your MachineSet and Machine objects by interacting with the virtualisation solution.
If you deploy some workloads, and down the line you find that your cluster is too full and you need more nodes, no need to worry! You can scale a MachineSet and easily add more node(s) to your cluster:
$ oc scale machineset sandbox-xfvg5-worker --replicas=4
machineset.machine.openshift.io/sandbox-xfvg5-worker scaledYou can see that the MachineSet now has a desired number of 4 nodes…
$ oc get machinesets
NAME DESIRED CURRENT READY AVAILABLE AGE
sandbox-xfvg5-worker 4 4 3 3 20dAnd that there is a fourth worker node in Provisioning state…
$ oc get machines
NAME PHASE TYPE REGION ZONE AGE
sandbox-xfvg5-master-0 Running 20d
sandbox-xfvg5-master-1 Running 20d
sandbox-xfvg5-master-2 Running 20d
sandbox-xfvg5-worker-4rgjw Provisioning 68s
sandbox-xfvg5-worker-9c9jf Running 20d
sandbox-xfvg5-worker-fd6cn Running 20d
sandbox-xfvg5-worker-vx6wb Running 20dAfter a few minutes, the Machine will switch to Running, and shortly after that the Node will be available too, and workloads will start to run on it. It really is that easy to scale up!
