Alauda Container Platform

Block Storage Disaster Recovery

RBD Mirror is a feature of Ceph Block Storage (RBD) that enables asynchronous data replication between different Ceph clusters, providing cross-cluster Disaster Recovery (DR). Its core function is to synchronize data in a primary-backup mode, ensuring rapid service takeover by the backup cluster when the primary cluster fails.

WARNING
  • RBD Mirror performs incremental synchronization based on snapshots, with a default snapshot interval of once per hour (configurable). The differential data between primary and backup clusters typically corresponds to writes within one snapshot cycle.
  • RBD Mirror only provides underlying storage data backup and does not handle Kubernetes resource backups. Please use the platform's Backup and Restore feature to back up PVC and PV resources.

TOC

TerminologyBackup ConfigurationPrerequisitesProceduresFailoverPrerequisitesProcedures

Terminology

TermExplanation
Primary ClusterThe cluster currently providing storage services.
Secondary ClusterThe standby cluster used for backup purposes.

Backup Configuration

Prerequisites

  • Prepare two clusters capable of deploying Alauda Container Platform (ACP) Storage with Ceph: a Primary cluster and a Secondary cluster, with network connectivity between them.
  • Both clusters must run the same platform version (v3.12 or later).
  • Create distributed storage services in both Primary and Secondary clusters.
  • Create block storage pools with identical names in both Primary and Secondary clusters.
  • Please ensure that the following three images have been uploaded to the platform's private image repository:
    • quay.io/csiaddons/k8s-controller:v0.5.0
    • quay.io/csiaddons/k8s-sidecar:v0.8.0
    • quay.io/brancz/kube-rbac-proxy:v0.8.0

Procedures

Enable Mirroring for Primary Cluster's Block Storage Pool

Execute the following command on the Primary cluster's Control node:

Command
Output
kubectl -n rook-ceph patch cephblockpool <block-pool-name> \
--type merge -p '{"spec":{"mirroring":{"enabled":true,"mode":"image"}}}'

Parameters:

  • <block-pool-name>: Block storage pool name.

Retrieve Peer Token

This token serves as the critical credential for establishing mirror connections between clusters.

Execute the following command on the Primary cluster's Control node:

Command
Output
kubectl get secret -n rook-ceph \
$(kubectl get cephblockpool.ceph.rook.io <block-pool-name> -n rook-ceph -o jsonpath='{.status.info.rbdMirrorBootstrapPeerSecretName}') \
-o jsonpath='{.data.token}' | base64 -d

Parameters:

  • <block-pool-name>: Block storage pool name.

Create Peer Token Secret in Secondary Cluster

Execute the following command on the Secondary cluster's Control node:

Command
Output
kubectl -n rook-ceph create secret generic rbd-primary-site-secret \
--from-literal=token=<token> \
--from-literal=pool=<block-pool-name>

Parameters:

  • <token>: Token obtained from Step 2.
  • <block-pool-name>: Block storage pool name.

Enable Mirroring for Secondary Cluster's Block Storage Pool

在 Execute the following command on the Secondary cluster's Control node:

Command
Output
kubectl -n rook-ceph patch cephblockpool <block-pool-name> --type merge -p \
'{
  "spec": {
    "mirroring": {
      "enabled": true,
      "mode": "image",
      "peers": {
        "secretNames": [
          "rbd-primary-site-secret"
        ]
      }
    }
  }
}'

Parameters:

  • <block-pool-name>: Block storage pool name.

Deploy Mirror Daemon in Secondary Cluster

This daemon is responsible for monitoring and managing RBD mirror synchronization processes, including data synchronization and error handling.

Execute the following command on the Secondary cluster's Control node:

Command
Output
cat << EOF | kubectl apply -f -
apiVersion: ceph.rook.io/v1
kind: CephRBDMirror
metadata:
  name: rbd-mirror
  namespace: rook-ceph
spec:
  count: 1
EOF

Verify Mirror Status

Execute the following command on the Secondary cluster's Control node:

Command
Output
kubectl get cephblockpools.ceph.rook.io <block-pool-name> -n rook-ceph -o jsonpath='{.status.mirroringStatus.summary}'

Parameters:

  • <block-pool-name>: Block storage pool name.

Enable Replication Sidecar

This feature enables efficient data replication and synchronization without interrupting primary application operations, enhancing system reliability and availability.

  1. Deploy csiaddons-controller

Execute the following commands on both Primary and Secondary clusters' Control nodes:

Click to view
kubectl create -f https://raw.githubusercontent.com/csi-addons/kubernetes-csi-addons/v0.5.0/deploy/controller/crds.yaml
kubectl create -f https://raw.githubusercontent.com/csi-addons/kubernetes-csi-addons/v0.5.0/deploy/controller/rbac.yaml

cat << EOF | kubectl apply -f -
apiVersion: v1
data:
  controller_manager_config.yaml: |
    apiVersion: controller-runtime.sigs.k8s.io/v1alpha1
    kind: ControllerManagerConfig
    health:
      healthProbeBindAddress: :8081
    metrics:
      bindAddress: 127.0.0.1:8080
    webhook:
      port: 9443
    leaderElection:
      leaderElect: true
      resourceName: e8cd140a.openshift.io
kind: ConfigMap
metadata:
  name: csi-addons-manager-config
  namespace: csi-addons-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: csi-addons
  name: csi-addons-controller-manager
  namespace: csi-addons-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: csi-addons
  template:
    metadata:
      annotations:
        kubectl.kubernetes.io/default-container: manager
      labels:
        app.kubernetes.io/name: csi-addons
    spec:
      containers:
      - args:
        - --secure-listen-address=0.0.0.0:8443
        - --upstream=http://127.0.0.1:8080/
        - --logtostderr=true
        - --v=10
        image: <registry>/brancz/kube-rbac-proxy:v0.8.0
        name: kube-rbac-proxy
        ports:
        - containerPort: 8443
          name: https
          protocol: TCP
        resources:
          limits:
            cpu: 500m
            memory: 128Mi
          requests:
            cpu: 10m
            memory: 64Mi
      - args:
        - --health-probe-bind-address=:8081
        - --metrics-bind-address=127.0.0.1:8080
        - --leader-elect
        command:
        - /manager
        image: <registry>/csiaddons/k8s-controller:v0.5.0
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8081
          initialDelaySeconds: 15
          periodSeconds: 20
        name: manager
        readinessProbe:
          httpGet:
            path: /readyz
            port: 8081
          initialDelaySeconds: 5
          periodSeconds: 10
        resources:
          limits:
            cpu: 500m
            memory: 128Mi
          requests:
            cpu: 10m
            memory: 64Mi
        securityContext:
          allowPrivilegeEscalation: false
      securityContext:
        runAsNonRoot: true
      serviceAccountName: csi-addons-controller-manager
      terminationGracePeriodSeconds: 10
EOF

Parameters:

  • <registry>: Registry address of platform.
  1. Enable csi sidecar

Execute the following commands on both Primary and Secondary clusters' Control nodes:

kubectl patch cm rook-ceph-operator-config -n rook-ceph --type json --patch \
'[
  {
    "op": "add",
    "path": "/data/CSI_ENABLE_OMAP_GENERATOR",
    "value": "true"
  },
  {
    "op": "add",
    "path": "/data/CSI_ENABLE_CSIADDONS",
    "value": "true"
  }
]'

Create VolumeReplicationClass

Execute the following commands on both Primary and Secondary clusters' Control nodes:

Command
Output
cat << EOF | kubectl apply -f -
apiVersion: replication.storage.openshift.io/v1alpha1
kind: VolumeReplicationClass
metadata:
  name: rbd-volumereplicationclass
spec:
  provisioner: rook-ceph.rbd.csi.ceph.com
  parameters:
    mirroringMode: snapshot
    schedulingInterval: "<scheduling-interval>"
    replication.storage.openshift.io/replication-secret-name: rook-csi-rbd-provisioner
    replication.storage.openshift.io/replication-secret-namespace: rook-ceph
EOF
  1. <scheduling-interval>: Scheduling interval, (e.g., schedulingInterval: "1h" indicates execution every 1 hour.)

Enable Mirror for PVC

Execute the following command on the Primary cluster's Control node:

Command
Output
cat << EOF | kubectl apply -f -
apiVersion: replication.storage.openshift.io/v1alpha1
kind: VolumeReplication
metadata:
  name: <vr-name>
  namespace: <namespace>
spec:
  autoResync: false
  volumeReplicationClass: rbd-volumereplicationclass
  replicationState: primary
  dataSource:
    apiGroup: ""
    kind: PersistentVolumeClaim
    name: <pvc-name>
EOF
  1. <vr-name>: The name of the VolumeReplication object, recommended to be the same as the PVC name.
  2. <namespace>: The namespace to which the VolumeReplication belongs, which must be the same as the PVC namespace.
  3. <pvc-name>: The name of the PVC for which Mirror needs to be enabled.

Note After enabling, the RBD image in the Secondary cluster becomes read-only.

Failover

When the Primary cluster fails, it is necessary to switch the primary-backup relationship of the RBD image.

Prerequisites

  • The Kubernetes resources of the Primary cluster have been backed up and restored to the Secondary cluster, including PVCs, PVs, application workloads, etc.

Procedures

创建 VolumeReplication

Execute the following command on the Secondary cluster's Control node:

cat << EOF | kubectl apply -f -
apiVersion: replication.storage.openshift.io/v1alpha1
kind: VolumeReplication
metadata:
  name: <vr-name>
  namespace: <namespace>
spec:
  autoResync: false
  dataSource:
    apiGroup: ""
    kind: PersistentVolumeClaim
    name: <mirror-pvc-name>
  replicationHandle: ""
  replicationState: primary
  volumeReplicationClass: rbd-volumereplicationclass
EOF
  1. <vr-name>: VolumeReplication name.
  2. <namespace>: PVC namespace.
  3. <mirror-pvc-name>: The name of the PVC.

Note After creation, the RBD image on the Secondary cluster becomes primary and is writable.