STF 1.5 Fully deconnected

· 545 words · 3 minute read

Based on : https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.2/html/service_telemetry_framework_1.5/assembly-installing-the-core-components-of-stf_assembly list all needed operators and images and generate imageset-config.yaml

apiVersion: mirror.openshift.io/v1alpha2
kind: ImageSetConfiguration
storageConfig:
  local:
    path: ./
mirror:
  operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.10
    packages:
      - name: service-telemetry-operator
        channels:
          - name: stable-1.5
      - name: openshift-cert-manager-operator
        channels:
          - name: tech-preview
      - name: amq7-interconnect-operator
        channels:
          - name: 1.10.x
      - name: smart-gateway-operator
        channels:
          - name: stable-1.5

  - catalog: registry.redhat.io/redhat/certified-operator-index:v4.10
    packages:
      - name: elasticsearch-eck-operator-certified
        minVersion: '2.6.2'
        channels:
          - name: stable

  - catalog: quay.io/operatorhubio/catalog:latest
    packages:
      - name: prometheus
        channels:
        - name: beta
          minVersion: '0.47.0'
      - name: grafana-operator
        channels:
        - name: alpha
          minVersion: '3.10.3'
        - name: v4
    	minVersion: '4.8.0'
    	maxVersion: '4.8.0'

  additionalImages:
  - name: registry.redhat.io/openshift4/ose-oauth-proxy:v4.4
  - name: gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0
  - name: quay.io/prometheus-operator/prometheus-config-reloader:v0.47.0
  - name: quay.io/prometheus/alertmanager:v0.21.0
  - name: quay.io/prometheus/prometheus:latest
  - name: docker.elastic.co/elasticsearch/elasticsearch-ubi8:7.16.1
  - name: registry.redhat.io/rhel8/grafana:7
  - name: quay.io/grafana-operator/grafana_plugins_init:0.1.0
  - name: docker.io/grafana/grafana:8.4.11 # needed for polystat 2.4+

Based on https://github.com/openshift/oc-mirror/issues/538, we’re not able to sync non-default channel by itself; if it errors out, you need to also include at least one version from the default channel, had issue with last version 4.9 so used lower one.

From a server with access to internet, Download images locally

oc mirror --config imageset-config.yaml file://archives

Copy archives disconnected server, and populate images to mirror registry

oc mirror --from archives/ docker://quay.gnali.lab:8443

⚠️ Don’t forget to add quay rootCA to /etc/pki/ca-trust/source/anchors/, and quay authentication to .docker/config.json

Tag following quay image:

Image Tag
prometheus-operator/prometheus-operator v0.47.0
grafana-operator/grafana-operator v4.8.0

Edit the generated catalog source file from mirroring to match the official source. Doing this we can just copy/paste commands from STF documentation.

$ grep  metadata -A1  ./oc-mirror-workspace/results-1639608409/catalogSource-*
catalogSource-catalog.yaml:metadata:
catalogSource-catalog.yaml-  name: operatorhubio-operators
--
catalogSource-certified-operator-index.yaml:metadata:
catalogSource-certified-operator-index.yaml-  name: certified-operators
--
catalogSource-redhat-operator-index.yaml:metadata:
catalogSource-redhat-operator-index.yaml-  name: redhat-operators

Disable default operator sources

oc patch OperatorHub cluster --type json -p '[{"op": "add", "path": "/spec/disableAllDefaultSources", "value": true}]'

Apply image content policy

oc apply -f ./oc-mirror-workspace/results-1639608409/

Connect to Openshift worker, and push oauth-proxy image

oc login -u kubeadmin -p FnJKS-hJTcm-A69VN-aNhDC https://api.ocp.gnali.lab:6443
podman login -u kubeadmin -p $(oc whoami -t) image-registry.openshift-image-registry.svc:5000
podman pull quay.gnali.lab:8443/openshift4/ose-oauth-proxy:v4.4 
podman tag quay.gnali.lab:8443/openshift4/ose-oauth-proxy:v4.4 image-registry.openshift-image-registry.svc:5000/openshift/oauth-proxy:v4.4
podman push image-registry.openshift-image-registry.svc:5000/openshift/oauth-proxy:v4.4

Allow image to be pull via tag instead of digest (https://access.redhat.com/solutions/4817401)

$ cat << EOF > prometheus.conf
[[registry]]
  prefix = ""
  location = "quay.io/prometheus-operator"
  mirror-by-digest-only = false

  [[registry.mirror]]
    location = "quay.gnali.lab:8443/prometheus-operator"

[[registry]]
  prefix = ""
  location = "quay.io/prometheus"
  mirror-by-digest-only = false

  [[registry.mirror]]
    location = "quay.gnali.lab:8443/prometheus"

[[registry]]
  prefix = ""
  location = "docker.elastic.co/elasticsearch"
  mirror-by-digest-only = false

  [[registry.mirror]]
    location = "quay.gnali.lab:8443/elasticsearch"

[[registry]]
  prefix = ""
  location = "gcr.io/kubebuilder"
  mirror-by-digest-only = false

  [[registry.mirror]]
	location = "quay.gnali.lab:8443/kubebuilder"

[[registry]]
  prefix = ""
  location = "quay.io/grafana-operator"
  mirror-by-digest-only = false

  [[registry.mirror]]
	location = "quay.gnali.lab:8443/grafana-operator"

[[registry]]
  prefix = ""
  location = "registry.redhat.io/rhel8"
  mirror-by-digest-only = false

  [[registry.mirror]]
	location = "quay.gnali.lab:8443/rhel8"

[[registry]]
  prefix = ""
  location = "docker.io/grafana"
  mirror-by-digest-only = false

  [[registry.mirror]]
	location = "sopra-quay.gnali.lab:8443/grafana"
EOF

$ base64_reg=$(base64 -w0 prometheus.conf)

$ cat << EOF > worker_registry.yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 99-mirror-by-digest-registries
spec:
  config:
    ignition:
      version: 3.1.0
    storage:
      files:
      - contents:
          source: data:text/plain;charset=utf-8;base64,${base64_reg}
        filesystem: root
        mode: 420
        path: /etc/containers/registries.conf.d/99-mirror-by-digest-registries.conf
EOF

$ oc  apply -f worker_registry.yaml

Important : it’s needed to create a separate machineconfig for each node role

Follow official documentation to install STF: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.2/html/service_telemetry_framework_1.5/assembly-installing-the-core-components-of-stf_assembly#deploying-stf-to-the-openshift-environment_assembly-installing-the-core-components-of-stf

Download grafana polystat plugin from https://grafana.com/grafana/plugins/grafana-polystat-panel/?tab=installation

Unzip and copy plugins to grafana containers:

$ oc rsync grafana-polystat-panel grafana-deployment-68684776cc-ss28s:/var/lib/grafana/plugins/

Connect to grafana containers and reload grafana

$ oc exec grafana-deployment-68684776cc-ss28s -it -- /bin/bash
$ killall grafana-server

References: