August 13, 2018

Deploy Granafa with Prometheus and OAuth2 on OpenShift

Grafana has an official docker image. Though permission problems occurred in previous versions, it can running flawlessly. There is also an example in OKD. We will start with the examples, but also do further configuration with data sources and dashboards.

Deploy Granafa with Prometheus and OAuth2 on OpenShift

Overview

Grafana has an official docker image. Though permission problems occurred in previous versions, it can running flawlessly. There is also an example in OKD[1]. We will start with the examples, but also do further configuration with data sources and dashboards.

Deploying Prometheus

Prometheus is supported officially by OpenShift and hence could be quicky deployed using openshift-ansible. Simply run

$ ansible-playbook -vvv -i ${INVENTORY_FILE} playbooks/openshift-prometheus/config.yml

would automatically deploy Prometheus.

Alternatively, Prometheus installation could be customized by adding more options into inventory file. Details could be found at enabling cluster metrics chapter in documents.

Deploying Grafana

We could create a template named grafana.yaml

---
kind: Template
apiVersion: v1
metadata:
  name: grafana
  annotations:
    "openshift.io/display-name": Grafana
    description: |
      Grafana server with patched Prometheus datasource.
    iconClass: fa fa-cogs
    tags: "metrics,monitoring,grafana,prometheus"
parameters:
- description: The location of the grafana image
  name: IMAGE_GRAFANA
  value: docker.io/grafana/grafana:master 
- description: The location of the proxy image
  name: IMAGE_PROXY
  value: openshift/oauth-proxy:v1.0.0
- description: External URL for the grafana route
  name: ROUTE_URL
  value: ""
- description: The namespace to instantiate heapster under. Defaults to 'grafana'.
  name: NAMESPACE
  value: grafana
- description: The session secret for the proxy
  name: SESSION_SECRET
  generate: expression
  from: "[a-zA-Z0-9]{43}"
objects:
- apiVersion: v1
  kind: ServiceAccount
  metadata:
    name: grafana
    namespace: "${NAMESPACE}"
    annotations:
      serviceaccounts.openshift.io/oauth-redirectreference.primary: '{"kind":"OAuthRedirectReference","apiVersion":"v1","reference":{"kind":"Route","name":"grafana"}}'
- apiVersion: authorization.openshift.io/v1
  kind: ClusterRoleBinding
  metadata:
    name: grafana-cluster-reader
  roleRef:
    name: cluster-reader
  subjects:
  - kind: ServiceAccount
    name: grafana
    namespace: "${NAMESPACE}"
- apiVersion: route.openshift.io/v1
  kind: Route
  metadata:
    name: grafana
    namespace: "${NAMESPACE}"
  spec:
    host: "${ROUTE_URL}"
    to:
      name: grafana
    tls:
      termination: Reencrypt
- apiVersion: v1
  kind: Service
  metadata:
    name: grafana
    annotations:
      prometheus.io/scrape: "true"
      prometheus.io/scheme: https
      service.alpha.openshift.io/serving-cert-secret-name: grafana-tls
    namespace: "${NAMESPACE}"
    labels:
      metrics-infra: grafana
      name: grafana
  spec:
    ports:
    - name: grafana
      port: 443
      protocol: TCP
      targetPort: 8443
    selector:
      app: grafana
- apiVersion: v1
  kind: Secret
  metadata:
    name: grafana-proxy
    namespace: "${NAMESPACE}"
  stringData:
    session_secret: "${SESSION_SECRET}="
# Deploy Grafana behind an oauth proxy
- apiVersion: extensions/v1beta1
  kind: Deployment
  metadata:
    labels:
      app: grafana
    name: grafana
    namespace: "${NAMESPACE}"
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: grafana
    template:
      metadata:
        labels:
          app: grafana
        name: grafana
      spec:
        serviceAccountName: grafana
        containers:
        - name: oauth-proxy
          image: ${IMAGE_PROXY}
          imagePullPolicy: IfNotPresent
          ports:
          - containerPort: 8443
            name: web
          args:
          - -https-address=:8443
          - -http-address=
          - -email-domain=*
          - -client-id=system:serviceaccount:${NAMESPACE}:grafana
          - -upstream=http://localhost:3000
          - -provider=openshift
#          - '-openshift-delegate-urls={"/api/datasources": {"resource": "namespace", "verb": "get", "resourceName": "grafana", "namespace": "${NAMESPACE}"}}'
          - '-openshift-sar={"namespace": "${NAMESPACE}", "verb": "list", "resource": "services"}'
          - -tls-cert=/etc/tls/private/tls.crt
          - -tls-key=/etc/tls/private/tls.key
          - -client-secret-file=/var/run/secrets/kubernetes.io/serviceaccount/token
          - -cookie-secret-file=/etc/proxy/secrets/session_secret
          - -skip-auth-regex=^/metrics,/api/datasources,/api/dashboards
          volumeMounts:
          - mountPath: /etc/tls/private
            name: grafana-tls
          - mountPath: /etc/proxy/secrets
            name: secrets

        - name: grafana
          image: ${IMAGE_GRAFANA}
          ports:
          - name: grafana-http
            containerPort: 3000
          volumeMounts:
          - mountPath: "/root/go/src/github.com/grafana/grafana/data"
            name: grafana-data
          - mountPath: "/usr/share/grafana/conf"
            name: grafanaconfig
          - mountPath: /etc/tls/private
            name: grafana-tls
          - mountPath: /etc/proxy/secrets
            name: secrets

        volumes:
        - name: grafanaconfig
          configMap:
            name: grafana-config
        - name: secrets
          secret:
            secretName: grafana-proxy
        - name: grafana-tls
          secret:
            secretName: grafana-tls
        - emptyDir: {}
          name: grafana-data

In this template, we defined 2 containers in a pod, which are grafana itself and a oauth-proxy for access authorization. We also defined several volumes for grafana container, including a config maps named grafana-config for configuration persistence.
By now, the real configuration is still missing in the template. We could follow the default configuration in Grafana to minimize installation difficulty. Add the following content to the template:

- apiVersion: v1
  kind: ConfigMap
  metadata:
    name: grafana-config
    namespace: "${NAMESPACE}"
  data:
    defaults.ini: |-
      [...content of the configuration...]

Then, create a new project and upload the template to it.

$ oc create project grafana
$ NAMESPACE=grafana oc process -f grafana.yml | oc create -f -
$ oc rollout status deployment/grafana

Now, the grafana pod should be up and running. We still needs to setup OAuth by adding proper permission to users.

$ oc adm policy add-cluster-role-to-user cluster-reader grafana

Now point your browser to your apps route, and OAuth should be working in front of grafana.

Datasource Provisioning

From Grafana 5.0, datasource could be provisioned by placing YAML files in given directory. This directory could be specified in configuration. With default configration, we could add a config maps named grafana-datasources, and mount it to grafana-datasources → /usr/share/grafana/datasources. In the config map, create a pair whose key is datasources.yaml, and insert datasource parameters. For example, to bind a Prometheues behind an OAuth proxy:

apiVersion: 1
datasources:
  - name: "OCP Prometheus"
    type: prometheus
    access: proxy
    url: https://route.to.prometheues.app
    basicAuth: false
    withCredentials: false
    isDefault: true
    jsonData:
        tlsSkipVerify: true
        "httpHeaderName1": "Authorization"
    secureJsonData:
        "httpHeaderValue1": "Bearer [Prometheues Secret]"
    editable: true

where [Prometheues Secret] could be gained by oc sa get-token prometheus -n openshift-metrics.

Dashboard Provisioning

Dashboard could be provisioned in Grafana 5 as well. Slightly different from datasource provisioning, we need to create a config map to tell grafana where we place our dashboards, and then another config map which contains real dashboard configrations.
First we create and mount a config map grafana-provisioning-dashboards → /usr/share/grafana/dashboards, and create a pair named dashboards.yaml with value:

apiVersion: 1
providers:
- name: 'default'
  orgId: 1
  folder: ''
  type: file
  disableDeletion: false
  updateIntervalSeconds: 3
  options:
    path: /var/lib/grafana/dashboards

Then create and mount grafana-dashboards → /var/lib/grafana/dashboards, where the path of the mountpoint should be changed upon the configuration in grafana-provisioning-dashboards config map. Finally we could create pairs whose key is any-filename.json and value is the JSON of the dashboard. Grafana should load it as it is mounted and changed.


  1. Previously OpenShift Origin ↩︎