Introduction

It's recommended to familiarize yourself with inspecting workloads on Kubernetes. This particular cheat sheet is very useful to have readily available.

Sanity checks

Once the CSI driver has been deployed either through object configuration files, Helm or an Operator. This view should be representative of what a healthy system should look like after install. If any of the workload deployments lists anything but Running, proceed to inspect the logs of the problematic workload.

kubectl get pods --all-namespaces -l 'app in (nimble-csp, hpe-csi-node, hpe-csi-controller)'
NAMESPACE     NAME                                  READY   STATUS    RESTARTS   AGE
kube-system   hpe-csi-controller-7d9cd6b855-zzmd9   5/5     Running   0          15s
kube-system   hpe-csi-node-dk5t4                    2/2     Running   0          15s
kube-system   hpe-csi-node-pwq2d                    2/2     Running   0          15s
kube-system   nimble-csp-546c9c4dd4-5lsdt           1/1     Running   0          15s
kubectl get pods --all-namespaces -l 'app in (primera3par-csp, hpe-csi-node, hpe-csi-controller)'
NAMESPACE     NAME                                  READY   STATUS    RESTARTS   AGE
kube-system   hpe-csi-controller-7d9cd6b855-fqppd   5/5     Running   0          14s
kube-system   hpe-csi-node-86kh6                    2/2     Running   0          14s
kube-system   hpe-csi-node-k8p4p                    2/2     Running   0          14s
kube-system   hpe-csi-node-r2mg8                    2/2     Running   0          14s
kube-system   hpe-csi-node-vwb5r                    2/2     Running   0          14s
kube-system   primera3par-csp-546c9c4dd4-bcwc6      1/1     Running   0          14s

A Custom Resource Definition (CRD) named hpenodeinfos.storage.hpe.com holds important network and host initiator information.

Retrieve list of nodes.

kubectl get hpenodeinfos
$ kubectl get hpenodeinfos
NAME               AGE
tme-lnx-worker1   57m
tme-lnx-worker3   57m
tme-lnx-worker2   57m
tme-lnx-worker4   57m

Inspect a node.

kubectl get hpenodeinfos/tme-lnx-worker1 -o yaml
apiVersion: storage.hpe.com/v1
kind: HPENodeInfo
metadata:
  creationTimestamp: "2020-08-24T23:50:09Z"
  generation: 1
  managedFields:
  - apiVersion: storage.hpe.com/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        .: {}
        f:chap_password: {}
        f:chap_user: {}
        f:iqns: {}
        f:networks: {}
        f:uuid: {}
    manager: csi-driver
    operation: Update
    time: "2020-08-24T23:50:09Z"
  name: tme-lnx-worker1
  resourceVersion: "30337986"
  selfLink: /apis/storage.hpe.com/v1/hpenodeinfos/tme-lnx-worker1
  uid: 3984752b-29ac-48de-8ca0-8381532cbf06
spec:
  chap_password: RGlkIHlvdSByZWFsbHkgZGVjb2RlIHRoaXM/
  chap_user: chap-user
  iqns:
  - iqn.1994-05.com.redhat:828e7a4eef40
  networks:
  - 10.2.2.2/16
  - 172.16.6.115/24
  - 172.16.8.115/24
  - 172.17.0.1/16
  - 10.1.1.0/12
  uuid: 0242f811-3995-746d-652d-6c6e78352d77

NFS Server Provisioner resources

The NFS Server Provisioner consists of a number of Kubernetes resources per PVC. The default Namespace where the resources are deployed is "hpe-nfs" but is configurable in the StorageClass. See base StorageClass parameters for more details.

Object Name                           Purpose
ConfigMap hpe-nfs-config This ConfigMap holds the configuration file for the NFS server. Local tweaks may be wanted. Please see the config file reference for more details.
Deployment hpe-nfs-<UUID> The Deployment that is running the NFS Pod.
Service hpe-nfs-<UUID> Pod Service the NFS clients perform mounts against.
PVC hpe-nfs-<UUID> The RWO claim serving the NFS workload.

Tip

The <UUID> stems from the user request RWX claim UUID for easy tracking.

Logging

Log files associated with the HPE CSI Driver logs data to the standard output stream. If the logs need to be retained for long term, use a standard logging solution for Kubernetes such as Fluentd. Some of the logs on the host are persisted which follow standard logrotate policies.

CSI driver logs

Node driver

kubectl logs -f  daemonset.apps/hpe-csi-node  hpe-csi-driver -n kube-system

Controller driver

kubectl logs -f deployment.apps/hpe-csi-controller hpe-csi-driver -n kube-system

Tip

The logs for both node and controller drivers are persisted at /var/log/hpe-csi.log

Log level

Log levels for both CSI Controller and Node driver can be controlled using LOG_LEVEL environment variable. Possible values are info, warn, error, debug, and trace. Apply the changes using kubectl apply -f <yaml> command after adding this to CSI controller and node container spec as below. For Helm charts this is controlled through logLevel variable in values.yaml.

          env:
            - name: LOG_LEVEL
              value: trace

CSP logs

CSP logs can be accessed from their respective services.

kubectl logs -f svc/nimble-csp-svc -n kube-system
kubectl logs -f svc/primera3par-csp-svc -n kube-system

Log collector

Log collector script hpe-logcollector.sh can be used to collect the logs from any node which has kubectl access to the cluster.

curl -O https://raw.githubusercontent.com/hpe-storage/csi-driver/master/hpe-logcollector.sh
chmod 555 hpe-logcollector.sh

Usage:

./hpe-logcollector.sh -h
Collect HPE storage diagnostic logs using kubectl.

Usage:
     hpe-logcollector.sh [-h|--help] [--node-name NODE_NAME] \
                         [-n|--namespace NAMESPACE] [-a|--all]
Options:
-h|--help                  Print this usage text
--node-name NODE_NAME      Collect logs only for Kubernetes node
                           NODE_NAME
-n|--namespace NAMESPACE   Collect logs from HPE CSI deployment in namespace
                           NAMESPACE (default: kube-system)
-a|--all                   Collect logs from all nodes (the default)

Tuning

HPE provides a set of well tested defaults for the CSI driver and all the supported CSPs. In certain case it may be necessary to fine tune the CSI driver to accommodate a certain workload or behavior.

Data path configuration

The HPE CSI Driver for Kubernetes automatically configures Linux iSCSI/multipath settings based on config.json. In order to tune these values, edit the config map with kubectl edit configmap hpe-linux-config -n kube-system and restart node plugin using kubectl delete pod -l app=hpe-csi-node to apply.

Important

HPE provide a set of general purpose default values for the IO paths, tuning is only required if prescribed by HPE.