Disaster Recovery in Ceph with cephadm, Ceph-CSI, and RBD Mirror
Introduction
Ceph is a highly available, scalable, and resilient storage solution widely used in cloud and enterprise environments. However, even with its built-in redundancy, disaster recovery (DR) strategies are essential to ensure business continuity in case of data center failures, network outages, or hardware failures. Ceph provides robust disaster recovery options, including RBD mirroring, to replicate block storage volumes across geographically separated Ceph clusters.
With the introduction of cephadm, Ceph cluster management has become more straightforward, making it easier to deploy and maintain disaster recovery setups. Additionally, Ceph-CSI enables Kubernetes clusters to consume Ceph storage efficiently. In this article, we will explore how to set up disaster recovery in Ceph using cephadm, Ceph-CSI, and RBD Mirror to protect RBD volumes used by Kubernetes clusters deployed across two data centers.
Disaster Recovery Architecture
We have two geographically separated data centers:
- Primary Data Center (Production): Hosts a Kubernetes cluster and a Ceph cluster.
- Secondary Data Center (Disaster Recovery – DR): Hosts another Kubernetes cluster and a Ceph cluster where data is replicated using RBD mirroring.
Kubernetes workloads in the primary DC store their persistent data in Ceph RBD volumes via Ceph-CSI. These volumes are mirrored asynchronously to the secondary DC using RBD mirroring, ensuring data availability in case of a failure in the primary DC.
Deploying Ceph with cephadm in Both Data Centers
Bootstrap the Ceph Cluster
On each Ceph cluster (Primary and Secondary):
cephadm bootstrap --mon-ip Add Additional Nodes
cephadm shell -- ceph orch host add Deploy Required Services
ceph orch apply mon
ceph orch apply mgr
ceph orch apply osd --all-available-devices
ceph orch apply rbd-mirrorEnsure that the rbd-mirror daemon is running on both clusters:
ceph orch ps | grep rbd-mirrorConfigure RBD Mirroring
On the primary Ceph cluster:
rbd mirror pool enable snapshotExport and import the authentication key:
ceph auth get-key client.rbd-mirror > rbd-mirror.key
scp rbd-mirror.key
ssh 'ceph auth import -i rbd-mirror.key'On the secondary Ceph cluster, add a peer connection:
rbd mirror pool peer add client.rbd-mirror@Verify peering status:
rbd mirror pool status Installing Ceph-CSI in Kubernetes Clusters
Now we have ceph cluster ready and we can deploy ceph-csi on our k8s clusters. We need to deploy ceph-csi in both locations, but
Deploy Ceph-CSI Driver
kubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/devel/deploy/csi-rbdplugin.yaml
kubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/devel/deploy/csi-rbdplugin-provisioner.yaml Enable RBD Mirroring on the Pool
rbd mirror pool enable snapshotConfigure StorageClass to Use the Mirrored Pool
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ceph-rbd-mirrored
provisioner: rbd.csi.ceph.com
parameters:
clusterID:
pool:
imageFormat: "2"
imageFeatures: layering
csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
csi.storage.k8s.io/provisioner-secret-namespace: default
csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
csi.storage.k8s.io/node-stage-secret-namespace: default
reclaimPolicy: Delete
allowVolumeExpansion: trueApply this StorageClass:
kubectl apply -f storageclass.yamlCreate a PersistentVolumeClaim (PVC)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ceph-rbd-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: ceph-rbd-mirroredFailover Process (Switching to the Secondary Data Center)
Promote the Secondary Ceph Cluster
rbd mirror pool promote Update ClusterID and PoolID Mappings
Ensure that the Kubernetes cluster in the DR site correctly maps the Ceph cluster’s ClusterID and PoolID using the predefined mapping.
apiVersion: v1
kind: ConfigMap
metadata:
name: ceph-csi-config
data:
cluster-mapping.json: |-
[
{
"clusterIDMapping": {
"primary-cluster-id": "secondary-cluster-id"
},
"RBDPoolIDMapping": [
{
"1": "2"
},
{
"11": "12"
}
]
}
]
Apply this updated mapping:
kubectl apply -f ceph-csi-config.yamlModify Ceph-CSI Config to Update Monitor Addresses on Secondary Cluster
To use a mirrored and promoted RBD image on a secondary site during a failover, you need to replace the primary monitor addresses with the IP addresses of the secondary cluster in ceph-csi-config. Otherwise, Ceph-CSI won’t be able to use the volumes, and application pods will become stuck in the ContainerCreating state. Thus, during failover, both clusters will have the same monitor IP addresses in csi-config on secondary site.
apiVersion: v1
kind: ConfigMap
metadata:
name: ceph-csi-config
data:
config.json: |-
[
{
"clusterID": "ceph1",
"rbd": {
"radosNamespace": "",
},
"monitors": [
"192.168.39.82:6789"
],
"cephFS": {
"subvolumeGroup": ""
}
},
{
"clusterID": "ceph2",
"rbd": {
"radosNamespace": "",
},
"monitors": [
"192.168.39.82:6789"
],
"cephFS": {
"subvolumeGroup": ""
}
}
]
Apply the updated configuration
kubectl apply -f ceph-csi-config.yamlModify StorageClass to Point to the Secondary Cluster
parameters:
clusterID: secondary-cluster-idApply the modified StorageClass:
kubectl apply -f storageclass.yamlRestart Affected Workloads
kubectl rollout restart deployment Validate Data Accessibility
Ensure the applications can access data stored in the secondary Ceph cluster.
Failback Process (Restoring to the Primary Data Center)
Demote the Secondary Cluster and Re-enable Mirroring
rbd mirror pool demote Update ClusterID and PoolID Mappings Back to Primary
apiVersion: v1
kind: ConfigMap
metadata:
name: ceph-csi-configModify StorageClass to Point Back to the Primary Cluster
parameters:
clusterID: primary-cluster-idRestart Workloads to Use the Primary Cluster
kubectl rollout restart deployment Verify Mirroring and Data Integrity
rbd mirror pool status Conclusion
By configuring ClusterID and PoolID mappings and ensuring proper Ceph monitor address updates during failover, you enable seamless disaster recovery for Kubernetes workloads using Ceph-CSI. This approach maintains data accessibility and consistency, facilitating a smoother failover and failback process. Using cephadm, deploying and managing mirroring has become significantly easier, enabling organizations to set up failover mechanisms efficiently. By following the above steps, you can ensure data integrity, minimize downtime, and enhance business continuity in the event of a disaster.
Author
Kamil Madáč
Grow2FIT Cloud&DevOps Consultant
Kamil is a Senior Cloud / Infrastructure consultant with 20+ years of experience and strong know-how in designing, implementing, and administering private cloud solutions (primarily built on OpenSource solutions such as OpenStack). He has many years of experience with application development in Python and currently also with development in Go. Kamil has substantial know-how in SDS (Software-defined storage), SDN (Software-defined networking), Data storage (Ceph, NetApp), administration of Linux servers and operation of deployed solutions.
Kamil regularly contributes to OpenSource projects (OpenStack, Kuryr, Requests Lib – Python).
The entire Grow2FIT consulting team: Our Team
