OpenShift Operations Training

Footer

Mon, 01 Jan 0001 00:00:00 +0000

Header

Mon, 01 Jan 0001 00:00:00 +0000

- acend gmbh

1.1 Installation

Mon, 01 Jan 0001 00:00:00 +0000

In this lab, you are going to install an OpenShift 4 cluster on AWS.

Task 1.1.1: Preparing the environment

Using the information provided by your trainer, ssh into your bastion host and change into the ocp4-ops directory.

It is best practice to create a timestamped directory for each cluster installation, for example:

cd ~/ocp4-ops
mkdir $(date +"%Y-%m-%d")

Task 1.1.2: Customizing the installation

First, we need an SSH keypair. The public key will be used to grant us access to the OpenShift machines we are going to create. Either use an existing keypair or create a new one by executing:

1.2 Cluster inspection

Mon, 01 Jan 0001 00:00:00 +0000

Before diving headfirst into configuring the freshly set-up cluster, let’s first take a look at certain features. As we installed the OpenShift cluster on AWS, some components have already been configured for us to use right away, amongst them:

An AWS cloud provider, making it possible for controllers to, e.g., create and update AWS load balancers and do node lifecycle management
A storage class to provision AWS Elastic Block Store (EBS) volumes

Cloud provider

In order to understand what a cloud provider is or does, it’s easiest to look at the description of the AWS cloud provider :

1.3 Configuration

Mon, 01 Jan 0001 00:00:00 +0000

After the initial installation, we can start configuring the cluster. Because of the extended use of operators, nearly everything in OpenShift 4 is configured via custom resources. In some cases configmaps or templates are used, but this is clearly the minority.

Most of the configuration is done post-installation and can be used to better adapt the cluster to its environment or change its default behaviour. We are going to configure our own kubelet arguments and change the default project template in order to automatically create some NetworkPolicy resources.

1.4 Additional components

Mon, 01 Jan 0001 00:00:00 +0000

In this lab, you will install additional components that are commonly used in production.

Task 1.4.1: Install cert-manager

cert-manager massively simplifies certificate management for Kubernetes. It provides easy to use tools to issue, manage and automatically renew certificates and supports the ACME protocol. This allows us to easily and automatically get certificates signed by Let’s Encrypt for our cluster.

Install cert-manager as an Operator using the OperatorHub:

In the Administrator view, navigate to Operators -> OperatorHub
Enter cert-manager into the filter box
Select the cert-manager Operator for Red Hat OpenShift and click Install
Leave everything as it is, except that you set the Update approval to Manual
Click Install
As soon as it appears, approve the install plan presented to you by clicking Approve
Check for the existence of the cert-manager-operator pod

Hints

oc --namespace cert-manager-operator get pods

Also, the Operator should already have created at least the webhook and cainjector pods in the cert-manager namespace

Hints

oc --namespace cert-manager get pods

The Operator will also automatically create a CertManager custom resource named cluster. In order to use DNS01 challenges on OpenShift clusters installed on AWS, we need to add some configuration to cert-manager’s configuration :

1.5 Uptime app

Mon, 01 Jan 0001 00:00:00 +0000

In this lab you’re going to deploy an application on your freshly-installed OpenShift cluster.

The application’s availability will be monitored and recorded by an external monitoring system. Your main goal for the rest of this training is to keep this application up and running, no matter what you do.

Note

At any time you can view the availability of your cluster console and the uptime app on this dashboard. Credentials to login will be provided by your trainers.

Uptime app

The application you’re going to install is a simple Python app. It’s going to be scaled to 2 replicas in order to achieve high availability.

1.6 Backup

Mon, 01 Jan 0001 00:00:00 +0000

In this lab, you will create scheduled backups for the most important cluster components.

Task 1.6.1: Create user workload backups

First, check the backupStorageLocation’s health and name:

oc -n openshift-adp get backupstoragelocations.velero.io

The output should show you that you’ve got backupStorageLocation resource named default with PHASE Available. If the PHASE shows something else, there’s most probably a permissions or typo problem in your dataProtectionApplication resource.

NAME PHASE LAST VALIDATED AGE DEFAULT
default Available 4s 3m true

Now, create a daily backup of the resources in namespace uptime-app-prod with a lifetime of 5 days by creating the following Schedule manifest:

2.1 Infrastructure nodes

Mon, 01 Jan 0001 00:00:00 +0000

Not exactly a day 2-operations task, but one that immensily simplifies operations. The concept of infrastructure nodes was more prominently advertised by Red Hat when OpenShift 3 was the latest and greatest. This evidently changed somehow for OpenShift 4, but with a bit of legwork we’re able to create those, too.

Infra nodes are dedicated worker nodes intended for infrastructure components such as the image registry, routers or the whole monitoring stack. Providing dedicated nodes helps ensuring that there’s always enough resources for infrastructure components.

2.2 Infrastructure components

Mon, 01 Jan 0001 00:00:00 +0000

Now that we have fresh infra nodes up and running, let’s see to it that they have something to do. The usual suspects to run on infrastructure nodes are:

routers
registry
most of the monitoring components
most of the logging components

Red Hat provides a detailed list of what components don’t incur OpenShift Container Platform worker subscriptions . So if you have any other components listed there that are not going to be looked at in this training, it could make perfect sense to move these components as well.

2.3 Restore

Mon, 01 Jan 0001 00:00:00 +0000

Task 2.3.1: Restoring user workload backups

In the backup lab you created a scheduled backup of all resources in namespace uptime-app-prod. Now we are going to test the restore procedure.

Warning

To keep your restore time as short as possible, make sure you read through the whole procedure, understand it and maybe even prepare all resources for the restore before deleting the project!

Make sure you read above warning, then go ahead and delete the project uptime-app-prod.

2.4 Update

Mon, 01 Jan 0001 00:00:00 +0000

In this lab we will update our OpenShift 4 cluster to the latest stable errata release.

Updating an OpenShift 4 cluster is a fully automated process that is managed by several cluster operators. If the cluster has Internet access, updates are provided over-the-air, otherwise you need to synchronize the new images to an internal registry prior to starting the update or use a pull-through registry.

Check “Updating a restricted network cluster” for details on updating a cluster in a disconnected environment.

2.5 Worker nodes

Mon, 01 Jan 0001 00:00:00 +0000

We initially installed this cluster using only worker nodes. We then successfully created infra nodes and moved all infrastructure components onto them. Now, essentially the only real workload that’s running on the 3 worker nodes is our uptime application.

Without going into detail yet on how and where you can find dashboards and metrics for capacity monitoring, it’s safe to say that one small Python app doesn’t need 3 worker nodes with 8 CPU cores and 32 GB of memory. So we are going to decrease the number of worker nodes to 2 and also reduce their size.

2.6 Control plane nodes

Mon, 01 Jan 0001 00:00:00 +0000

Note

While many occurrences of “master” have been replaced throughout OpenShift and Kubernetes, this is unfortunately not yet the case for Machine resources. This is why you are going to encounter a mixture of “control plane” and “master” terms in this lab.

During this lab, we will learn how to replace a failed control plane node. As long as we do not lose the majority of our control plane nodes, we can simply replace those that failed. If we lose the majority (e.g., 2 out of 3), we need to restore the state of etcd from a previously created snapshot, which will not be covered in this lab.

3.1 Rules and dashboards

Mon, 01 Jan 0001 00:00:00 +0000

Task 3.1.1: Provided dashboards

OpenShift provides a set of default dashboards for cluster adminstrators and OpenShift users.

Cluster dashboards

To get a sense of where these dashboards can be found, let’s take a look at the performance of the etcd cluster.

Navigate to Observe -> Dashboards on the web console and select etcd in the dashboard drop-down list.

User dashboards

Switch to the Developer console and select the uptime-app-prod project. Then navigate to the Observing tab. Select the uptime-app in the Workload drop-down list. This will display a wide selection of different metrics relevant to your app like CPU usage, memory usage and networking performance.

3.2 Monitoring stack configuration

Mon, 01 Jan 0001 00:00:00 +0000

Task 3.2.1: Enable alerting

Let’s take a look at an example of how you can send alerts to a receiver of your choice. Configuring a new receiver can either be done using the web console or by directly configuring the appropriate secret. Before going into detail on where this configuration lies, let’s have a look at an example:

---
apiVersion: v1
kind: Secret
metadata:
 name: alertmanager-main
 namespace: openshift-monitoring
stringData:
 alertmanager.yaml: |
 ---
 global:
 resolve_timeout: 5m
 smtp_from: monitoring@ops-training.openshift.ch
 smtp_smarthost: smtp.ops-training.openshift.ch:587
 smtp_hello: ops-training.openshift.ch
 smtp_auth_username: test
 smtp_auth_password: test
 smtp_require_tls: true
 inhibit_rules:
 - equal:
 - namespace
 - alertname
 source_match:
 severity: critical
 target_match_re:
 severity: warning|info
 - equal:
 - namespace
 - alertname
 source_match:
 severity: warning
 target_match_re:
 severity: info
 receivers:
 - name: Default
 - name: Watchdog
 - name: Critical
 - name: mail
 email_configs:
 - to: ops@ops-training.openshift.ch
 route:
 group_by:
 - namespace
 group_interval: 5m
 group_wait: 30s
 receiver: Default
 repeat_interval: 12h
 routes:
 - match:
 alertname: Watchdog
 receiver: Watchdog
 - match:
 severity: critical
 receiver: mail 
type: Opaque

The main components that can be configured when applying a custom Alertmanager configuration comprise:

3.3 User-defined monitoring rules

Mon, 01 Jan 0001 00:00:00 +0000

Task 3.3.1: Enable the user-workload monitoring stack

As mentioned in a lab before, it is not supported to alter the OpenShift-provided monitoring stack. If you want to add additional monitoring targets or add custom Prometheus rules, you need to enable the user-workload monitoring stack.

And this is exactly what we are going to do. First, edit the cluster-monitoring-config configmap in the openshift-monitoring namespace.

Hints

oc -n openshift-monitoring edit cm cluster-monitoring-config

Add the line enableUserWorkload: true under data/config.yaml (leave the rest as it is).

3.4 Logging

Mon, 01 Jan 0001 00:00:00 +0000

In this lab we will install and configure the logging stack.

Task 3.4.1 Install cluster logging

You can install OpenShift Logging to aggregate all the logs from your OpenShift cluster, such as node logs, application logs and infrastructure logs.

To deploy the cluster logging stack from the CLI, we need to create the following objects. Have a look at them first:

OpenShift Elasticsearch Operator Namespace

apiVersion: v1
kind: Namespace
metadata:
 name: openshift-operators-redhat 
 annotations:
 openshift.io/node-selector: ""
 labels:
 openshift.io/cluster-monitoring: "true"

Red Hat OpenShift Logging Namespace

apiVersion: v1
kind: Namespace
metadata:
 name: openshift-logging
 annotations:
 openshift.io/node-selector: ""
 labels:
 openshift.io/cluster-monitoring: "true"

OpenShift Elasticsearch Operator OperatorGroup

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
 name: openshift-operators-redhat
 namespace: openshift-operators-redhat 
spec: {}

Red Hat OpenShift Logging OperatorGroup

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
 name: cluster-logging
 namespace: openshift-logging 
spec:
 targetNamespaces:
 - openshift-logging

OpenShift Elasticsearch Operator Subscription

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
 name: "elasticsearch-operator"
 namespace: "openshift-operators-redhat" 
spec:
 channel: "stable" 
 installPlanApproval: "Automatic"
 source: "redhat-operators" 
 sourceNamespace: "openshift-marketplace"
 name: "elasticsearch-operator"

Red Hat OpenShift Logging Subscription

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
 name: cluster-logging
 namespace: openshift-logging 
spec:
 channel: "stable"
 name: cluster-logging
 source: redhat-operators 
 sourceNamespace: openshift-marketplace

Now either copy and paste above resource definitions and then apply them to your cluster or directly use our provided files:

4.1 Troubleshooting basics

Mon, 01 Jan 0001 00:00:00 +0000

In chapter 3 you got to know the monitoring components, its rules and more. Now, imagine you get an alert. “Where to begin?”, you ask. How do you find out what’s amiss and led to the alert?

That’s what we are going to look at in this first lab.

Task 4.1.1: Cluster operators

As you already know, OpenShift 4 usually consists of about 30 different operators. Each operator is responsible that the actual state of the cluster matches the desired, configured state. If this is not the case, the operator tells us.

4.2 Evictions

Mon, 01 Jan 0001 00:00:00 +0000

This second troubleshooting lab will go into detail on what evictions are, how to analyze and how to avoid them.

Task 4.2.1: Setup

First, let’s get set up. Create a new namespace using the following command:

Warning

It is essential that you use this command or else this lab won’t work!

oc create namespace <namespace>

Now, deploy a test pod into the freshly created namespace using the provided definition:

5.1 Setup

Mon, 01 Jan 0001 00:00:00 +0000

In order to explore GitOps processes, we are going to need appropriate tools: Gitea and ArgoCD.

Task 5.1.1: Gitea

Gitea is a lightweight Git server written in Go. It allows us to easily host a Git repository which we can use as our single source of truth.

Unfortunately, the default Operator catalog sources don’t contain a Gitea operator, but let’s change that.

Create the Gitea Operator by applying the YAML files from the Github repository:

5.2 Argo CD

Mon, 01 Jan 0001 00:00:00 +0000

Task 5.2.1: Argo CD

Go out into the new world of GitOps and explore Argo CD.

Find out how to access Argo CD’s web interface (which is very handy).

Hints

As usual, there are multiple ways to find out what Argo CD’s dashboard URL is:

OpenShift exposes the link via its web console:

Or get it via route from the openshift-gitops namespace:

oc -n openshift-gitops get routes

Task 5.2.2: Gitea repository

Create a new Git repository on your Gitea instance.

6.1 Destroy the cluster

Mon, 01 Jan 0001 00:00:00 +0000

Usually our work centers around keeping everything up and running, building and engineering stuff and so on. This is not always easy, so there’s some extra joy when it comes to finally be allowed to do the opposite and destroy something :)

Task 6.1.1: Destroy your cluster

Change into the installation directory you created on day 1, it should be under /home/ec2-user/ocp4-ops and have a timestamp as its name.

Destroy the cluster: