Introduction¶

The HPE Alletra Storage MP B10000, Alletra 9000 and Primera and 3PAR Storage Container Storage Provider (CSP) for Kubernetes is part of the HPE CSI Driver for Kubernetes. The CSP abstract the data management capabilities of the array for use by Kubernetes.

Platform Requirements¶

Check the corresponding CSI driver version in the compatibility and support table for the latest updates on supported Kubernetes version, orchestrators and host OS.

Network Port Requirements¶

The HPE Alletra Storage MP B10000, Alletra 9000, Primera and 3PAR Container Storage Provider requires the following TCP ports to be open inbound to the array from the Kubernetes cluster worker nodes running the HPE CSI Driver for Kubernetes.

User Role Requirements¶

The CSP requires access to a user with either edit or the super role. It's recommended to use the edit role for security best practices.

Virtual Domains¶

Virtual Domains are not yet fully supported by the CSP. From HPE CSI Driver v2.5.0, it's possible to manually create the Kubernetes hosts connecting to storage within the Virtual Domain. Once the hosts have been created, deploy the CSI driver with the Helm chart using the "disableHostDeletion" parameter set to "true". The Virtual Domain user may create the hosts through the Virtual Domain if the "AllowDomainUsersAffectNoDomain" parameter is set to either "hostonly" or "yes" on the array.

Detailed steps to use Virtual Domains¶

These steps assumes access to the storage platform with privileges to create domains and change settings.

Login to the storage platform with SSH. Create an new domain:

cli% createdomain -comment "This is a test domain." my-kubernetes-domain-0

Then, create a new user and assign to the domain. These credentials will be used by the CSI driver.

cli% createuser -c my-password-0 domain-user-0 my-kubernetes-domain-0 edit

Next, make sure domain users are allowed to create hosts outside the domain.

cli% setsys AllowDomainUsersAffectNoDomain hostonly

The next steps involve installing the HPE CSI Driver for Kubernetes with disableHostDeletion set to true. The steps to supply the parameter depends on if the Helm chart or Operator is being used.

• Helm chart install from ArtifactHub.io.
• Operator install for OpenShift.

Once the CSI driver is installed and running, add an HPE storage backend with the credentials provided in the steps above.

Limitations¶

These are the generally known limitation of the CSP.

• The CSP has been tested using iSCSI with up to 250 VolumeAttachments per compute node. HPE recommends not exceeding 200 VolumeAttachments per node and leave headroom for emergencies. It's always recommended to test the upper bounds before deploying to production. Increasing the "maxVolumesPerNode" parameter from the default of 100 is explained in the Helm chart.
• Compute node hostnames may not exceed 27 characters. The storage platform limitation is 31 characters. Since HPE CSI Driver 3.0.0, the node name has a 4 character protocol prefix such as "iqn-" or "wwn-". Further, the CSP truncates the domain name from the Kubernetes node name.

VLUN Templates¶

A VLUN template enables the export of a virtual volume as a VLUN to hosts. For more information, see the HPE Primera OS Commmand Line Interface - Installation and Reference Guide.

The CSP supports the following types of VLUN templates:

The boolean string "hostSeesVLUN" StorageClass parameter controls which VLUN template to use.

Change VLUN Template for existing PVCs¶

To modify an existing PVC, "hostSeesVLUN" needs to be specified with the "allowMutations" parameter along with adding the PVC annotation "csi.hpe.com/hostSeesVLUN" with the string values of either "true" or "false". The HPE CSI Driver creates the VLUN template based upon the hostSeesVLUN parameter during the volume publish operation. For the change to take effect, the Pod will need to be scheduled on another node by either deleting the Pod or draining the node.

StorageClass Parameters¶

All parameters enumerated reflects the current version and may contain unannounced features and capabilities.

Example default StorageClass (download):

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    # There can only be one default StorageClass per cluster
    storageclass.kubernetes.io/is-default-class: "true"
  name: hpe-standard
parameters:
  csi.storage.k8s.io/controller-expand-secret-name: hpe-backend
  csi.storage.k8s.io/controller-expand-secret-namespace: hpe-storage
  csi.storage.k8s.io/controller-publish-secret-name: hpe-backend
  csi.storage.k8s.io/controller-publish-secret-namespace: hpe-storage
  csi.storage.k8s.io/node-publish-secret-name: hpe-backend
  csi.storage.k8s.io/node-publish-secret-namespace: hpe-storage
  csi.storage.k8s.io/node-stage-secret-name: hpe-backend
  csi.storage.k8s.io/node-stage-secret-namespace: hpe-storage
  csi.storage.k8s.io/provisioner-secret-name: hpe-backend
  csi.storage.k8s.io/provisioner-secret-namespace: hpe-storage
  csi.storage.k8s.io/fstype: xfs
  accessProtocol: iscsi
  description: Volume created by the HPE CSI Driver for Kubernetes
  cpg: SSD_r6
  snapCpg: SSD_r6
  hostSeesVLUN: "true"
  provisioningType: tpvv
  # compression: "true" # For 3PAR only
provisioner: csi.hpe.com
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true

Common Provisioning Parameters¶

Restrictions applicable when using the CSI volume mutator:
¹ = Parameters that are editable after provisioning.
² = Volumes with snapshots/clones can't be modified.
³ = HPE 3PAR only parameter
⁴ = HPE Primera/Alletra 9000 only parameter

Please see using the HPE CSI Driver for additional StorageClass examples like CSI snapshots and clones.

Cloning Parameters¶

Cloning supports two modes of cloning. Either use cloneOf and reference a PersistentVolumeClaim in the current namespace to clone or use importVolAsClone and reference an array volume name to clone and import into the Kubernetes cluster. Volumes with clones are immutable once created.

Array Snapshot Parameters¶

During the snapshotting process, any existing PersistentVolumeClaim defined in the virtualCopyOf parameter within a StorageClass, will be snapped as PersistentVolumeClaim and exposed through the HPE CSI Driver and made available to the Kubernetes cluster. Volumes with snapshots are immutable once created.

Import Parameters¶

During the import volume process, any legacy (non-container volumes) defined in the ImportVol parameter, within a StorageClass, will be renamed to match the PersistentVolumeClaim that leverages the StorageClass. The new volumes will be exposed through the HPE CSI Driver and made available to the Kubernetes cluster. Note: All previous Access Control Records and Initiator Groups will be removed from the volume when it is imported.

Peer Persistence Configuration¶

The HPE Alletra Storage MP B10000 CSP supports two modes of performing synchrounous replication between two arrays, Active Peer Persistence (APP) and Classic Peer Persistence (CPP) using Remote Copy Groups (RCGs).

To enable replication within the HPE CSI Driver, the following steps must be completed:

• Create Secrets for both primary and target array. Refer to Configuring Additional Storage Backends.
• Create a replication HPEReplicationDeviceInfos CRD.
• Create a replication enabled StorageClass.

A CustomResourceDefinition (CRD) of type hpereplicationdeviceinfos.storage.hpe.com must be created to define the target array information. The CRD resource name will be used to define the StorageClass parameter "replicationDevices".

apiVersion: storage.hpe.com/v2
kind: HPEReplicationDeviceInfo
metadata:
  name: my-peer-persistence-target
  namespace: hpe-storage
spec:
  target_array_details:
  - targetCpg: <CPG name>
    targetSnapCpg: <Snap CPG name> # optional
    targetName: <Target array name>
    targetSecret: <Target Secret name>
    targetSecretNamespace: <Target Secret Namespace>

Next, review and perform the prerequisites for:

• Active Peer Persistence
• Classic Peer Persistence

Active Peer Persistence Prerequisites¶

Explaining all the requirements for using Active Peer Persistence is beyond the scope of this document. Be understood with the Active Peer Persistence limitations with the HPE CSI Driver before proceeding.

When using Active Peer Persistence, the CSP can't be allowed to manage the hosts on the backend arrays. While it's capable of creating hosts, the host needs to be admitted to RCGs manually to ensure the correct parameters are applied (see Manual Host and RCG Creation).

HPE CSI Driver installation parameters¶

To prevent the CSP from deleting unused hosts, the HPE CSI Driver needs to be installed with the Helm chart or Operator using the "disableHostDeletion" parameter set to "true".

Helm chart install example:

helm install --create-namespace -n hpe-storage my-hpe-csi-driver --set disableHostDeletion=true hpe-storage/hpe-csi-driver

Manual Host and RCG Creation¶

The CSP is at this time unable to add hosts with the correct proximity to RCGs. This is due to an API limitation, the limitation will be removed in a future version of the HPE CSI Driver.

Create the hosts (iSCSI example):

createhost -iscsi iqn-my-compute-node-1 iqn.1994-05.com.redhat:my-compute-node-1
createhost -iscsi iqn-my-compute-node-2 iqn.1994-05.com.redhat:my-compute-node-2
createhost -iscsi iqn-my-compute-node-3 iqn.1994-05.com.redhat:my-compute-node-3

Create the RCG and set the correct policy:

creatercopygroup -usr_cpg SSD_r6 my-target-FC:SSD_r6 my-csi-rcg my-target-FC:sync
setrcopygroup pol active_active my-csi-rcg

Admit the hosts to the RCG:

admitrcopyhost -proximity all my-csi-rcg iqn-my-compute-node-1
admitrcopyhost -proximity all my-csi-rcg iqn-my-compute-node-2
admitrcopyhost -proximity all my-csi-rcg iqn-my-compute-node-3

Verify that the hostset exist on both primary and target array:

showhostset RH2_my-csi-rcg*

Example output:

 Id Name           Members
102 RH2_my-csi-rcg iqn-my-compute-node-1
                   iqn-my-compute-node-2
                   iqn-my-compute-node-3
----------------------------------------
  1 total          3

 Id Name                   Members
374 RH2_my-csi-rcg.r123352 iqn-my-compute-node-1
                           iqn-my-compute-node-2
                           iqn-my-compute-node-3
----------------------------------------
  1 total                  3

It's highly desirable for an Active Peer Persistence protected workload to be rescheduled during a site outage. To allow Kubernetes to reschedule the workloads that ran on a partitioned host, the HPE CSI Driver Pod Monitor need to remove the VolumeAttachment from the partitioned host.

Learn how to label Pods to be monitored by the HPE CSI Driver:

• HPE CSI Driver for Kubernetes Pod Monitor

StorageClass Parameters for Active Peer Persistence¶

Due to an API limitation and the necessary manual steps to pre-create all the storage resources prior to creating PVCs, it's not currently recommended to add replication parameters directly in the StorageClass. Instead, PVCs are annotated with the necessary parameters, either during creation or afterwards.

This section will be updated in a future revision, meanwhile, Add Non-Replicated Volume to Remote Copy Group.

Classic Peer Persistence Prerequisites¶

If using existing RCGs for replication, the RCGs need to have the correct sync mode policy applied, "path_management". Periodic sync mode is not supported at this time.

setrcopygroup pol path_management my-csi-rcg

Be understood with the limitations of Classic Peer Persistence integration with the HPE CSI Driver before proceeding.

For a tutorial on how to enable Classic Peer Persistence, check out the blog Enabling Remote Copy using the HPE CSI Driver for Kubernetes on HPE Primera

StorageClass Parameters for Classic Peer Persistence¶

These StorageClass parameters are applicable only for replication, "remoteCopyGroup" and "replicationDevices" are mandatory. If the RCG defined in "remoteCopyGroup" doesn't exist on the array, then a new RCG will be created.

¹ = Existing RCGs must have local CPG and target CPG configured along with the correct policy, "path_management".

Add Non-Replicated Volume to Remote Copy Group¶

In order to add an existing PVC to a RCG, the StorageClass will utilize the Volume Mutator. PVC overrides may be used for creating PVCs with replication.

In the "parameter" section of the StorageClass, add the following (example):

...
paramters:
  allowOverrides: replicationPolicy,remoteCopyGroup,replicationDevices
  allowMutations: replicationPolicy,remoteCopyGroup,replicationDevices
...

Description of the parameters.

Learn in the using section how to apply overrides and mutations in more detail, as an example for replication, this PVC, when either created or edited, add the underlying PersistentVolume to "my-csi-rcg" RCG using Active Peer Persistence using the "my-peer-persistence-target" HPEReplicationDeviceInfo.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-replicated-pvc
  annotations:
    csi.hpe.com/replicationPolicy: active
    csi.hpe.com/remoteCopyGroup: my-csi-rcg
    csi.hpe.com/replicationDevices: my-peer-persistence-target
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 374Gi
  storageClassName: hpe-standard

The same "edit" operation may be performed non-interactive with kubectl:

kubectl annotate pvc/my-replicated-pvc \
    csi.hpe.com/replicationPolicy=active \
    csi.hpe.com/remoteCopyGroup=my-csi-rcg \
    csi.hpe.com/replicationDevices=my-peer-persistence-target

The underlying PersistentVolume is now being replicated.

VolumeSnapshotClass Parameters¶

These parameters are for VolumeSnapshotClass objects when using CSI snapshots. The external snapshotter needs to be deployed on the Kubernetes cluster and is usually performed by the Kubernetes vendor. Check enabling CSI snapshots for more information. Volumes with snapshots are immutable.

How to use VolumeSnapshotClass and VolumeSnapshot objects is elaborated on in using CSI snapshots.

VolumeGroupClass Parameters¶

In the HPE CSI Driver version 1.4.0+, a volume set with QoS settings can be created dynamically using the QoS parameters for the VolumeGroupClass. The following parameters are available for a VolumeGroup on the array. Learn more about VolumeGroups in the provisioning concepts documentation.

¹ = Parameter is deprecated and have no effect on HPE Alletra Storage MP B10000 10.5 and later.

Example for HPE Primera:

apiVersion: storage.hpe.com/v1
kind: VolumeGroupClass
metadata:
  name: my-volume-group-class
provisioner: csi.hpe.com
deletionPolicy: Delete
parameters:
  description: "HPE CSI Driver for Kubernetes Volume Group"
  csi.hpe.com/volume-group-provisioner-secret-name: hpe-backend
  csi.hpe.com/volume-group-provisioner-secret-namespace: hpe-storage
  priority: normal
  ioMinGoal: "300"
  ioMaxLimit: "10000"
  bwMinGoalKb: "3000"
  bwMaxLimitKb: "30000"
  latencyGoal: "300ms"

SnapshotGroupClass Parameters¶

These parameters are for SnapshotGroupClass objects when using CSI snapshots. The external snapshotter needs to be deployed on the Kubernetes cluster and is usually performed by the Kubernetes vendor. Check enabling CSI snapshots for more information. Volumes with snapshots are immutable.

How to use VolumeSnapshotClass and VolumeSnapshot objects is elaborated on in using CSI snapshots.

Static Provisioning¶

Static provisioning of PVs and PVCs may be used when absolute control over physical volumes are required by the storage administrator. This CSP also supports importing volumes and clones of volumes using the import parameters in a StorageClass.

Prerequisites¶

The CSP expects a certain naming convention for PersistentVolumes and Virtual Volumes on the array.

• Persistent Volume: pvc-00000000-0000-0000-0000-000000000000
• Virtual Volume: pvc-00000000-0000-0000-0000-000

The following example uses the above scheme as a naming convention. Have a storage administrator rename the existing Virtual Volume on the array:

setvv -name pvc-00000000-0000-0000-0000-000 my-existing-virtual-volume

HPEVolumeInfo¶

Create a new HPEVolumeInfo resource.

apiVersion: storage.hpe.com/v2
kind: HPEVolumeInfo
metadata:
  name: pvc-00000000-0000-0000-0000-000000000000
spec:
  record:
    Id: pvc-00000000-0000-0000-0000-000000000000
    Name: pvc-00000000-0000-0000-0000-000
  uuid: pvc-00000000-0000-0000-0000-000000000000

Persistent Volume¶

Create a PV referencing the HPEVolumeInfo resource.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pvc-00000000-0000-0000-0000-000000000000
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 16Gi
  csi:
    volumeHandle: pvc-00000000-0000-0000-0000-000000000000
    driver: csi.hpe.com
    fsType: xfs
    volumeAttributes:
      volumeAccessMode: mount
      fsType: xfs
    controllerPublishSecretRef:
      name: hpe-backend
      namespace: hpe-storage
    nodePublishSecretRef:
      name: hpe-backend
      namespace: hpe-storage
    controllerExpandSecretRef:
      name: hpe-backend
      namespace: hpe-storage
  persistentVolumeReclaimPolicy: Retain
  volumeMode: Filesystem

Persistent Volume Claim¶

Now, a user may claim the static PV by creating a PVC referencing the PV name in .spec.volumeName.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 16Gi
  volumeName: my-static-pv-1
  storageClassName: ""

Active Peer Persistence Limitations¶

These are the current limitations of Active Peer Persistence when used with the HPE CSI Driver.

• Active Peer Persistence was introduced in version 3.0.0 of the CSI driver. Prior versions won't work.
• Three sites are required to provide a redundant Kubernetes control-plane and Quorum Witness for the RCGs.
• RCGs, volumes and VLUNs may not be managed outside of the CSI driver.
• Pods (containers or VMs) needs to be labeled accordingly to be monitored by the HPE CSI Driver Pod Monitor to prevent multi-attach errors.
• Only intra-cluster disaster recovery is supported.
• Only symmetric host proximity is supported and running Active Peer Persistence beyond a campus distance (around 1km) is not recommended.
• Only manual host and RCG creation is supported (this limitation will be removed in the future).
• Once an automatic failover has occurred, the recovered workloads needs to be manually restarted when the previous primary is restored. This will resume full redundancy with VLUNs created on both arrays for the workloads (a future platform update will address this workaround).

Classic Peer Persistence Limitations¶

These are the current limitations of the Remote Copy Classic Peer Persistence integration with the HPE CSI Driver.

• Classic Peer Persistence does not provide disaster recovery for workloads running on Kubernetes. Classic Peer Persistence provide disaster recovery for the storage system.
• Classic Peer Persistence only provide data path resilience. If the primary array is unreachable for the CSP or the role of the remote copy group has changed due to disaster recovery operations (manual or automatic switchover/failover), all CSI operations will cease to function until the primary array comes back up and the role of the remote copy groups returned to original state.
• When the primary array is unavailable for the Kubernetes cluster and remote copy group has failed over to the target array successfully, running workloads will continue to run if the host the workload was running on has redundant data paths to the target array (current primary array).
• It's possible to access volumes from the target array by statically provisioning PersistentVolumes without renaming the volume on the array. This is only safe if it has been determined that the primary array does not have active hosts accessing the volume against the primary array.

Support¶

Please refer to the HPE Alletra Storage MP B10000, Alletra 9000 and Primera and 3PAR Storage CSP support statement.