Ramen

User quick start guide for testing application DR

Mon, 01 Jan 0001 00:00:00 +0000

User quick start guide for testing application DR

This page will help you to set up an environment for testing Regional-DR basic tests.

This video will demonstrate how to test application DR after completing the configuration.

What you’ll need

To set up a Ramen development environment you will need to run 3 Kubernetes clusters. Ramen makes this easy using minikube, but you need enough resources:

Ramen developer quick start guide

Mon, 01 Jan 0001 00:00:00 +0000

Ramen developer quick start guide

Contribution Flow

Fork the repository on GitHub
Create a branch from where you want to base your work (usually main).
Make your changes and arrange them in readable commits.
Make sure your commit messages are in the proper format.
Make sure all tests pass, and add any new tests
Push your changes to the branch in your fork of the repository. as appropriate.
Submit a pull request to the original repository.

Getting the source

Fork the Ramen repository on github

Clone the source locally

git clone git@github.com:my-github-user/ramen.git

Add an upstream remote, to make it easy to sync your repo from upstream.
```
git remote add upstream https://github.com/RamenDR/ramen.git
```
Add the commit-msg hook to sign off your commits

Ramen requires a Signed-off-by: My Name <myemail@example.com> footer in the commit message. To add it automatically to all commits, add this hook:
```
cp hack/commit-msg .git/hooks/
```

Installing Ramen

Mon, 01 Jan 0001 00:00:00 +0000

Installing Ramen

This guide describes how to install Ramen’s disaster recovery operators on your cluster environment.

Overview

Ramen consists of two operators that work together to provide disaster recovery capabilities:

Ramen Hub Operator - Installed on the OCM hub cluster, orchestrates DR operations across managed clusters
Ramen DR Cluster Operator - Installed on each OCM managed cluster, manages local volume replication and workload protection

Prerequisites

Before installing Ramen, ensure the following requirements are met:

Configure Ramen

Mon, 01 Jan 0001 00:00:00 +0000

Configure Ramen

This guide describes how to configure Ramen for disaster recovery operations after installing the Ramen operators. Configuration involves setting up S3 storage profiles, creating DRCluster resources, and establishing DRPolicy resources to define replication relationships.

Prerequisites

Before configuring Ramen, ensure you have completed the installation steps described in install.md . You should have:

OCM (Open Cluster Management) with at least two managed clusters registered
Ramen hub operator installed on the OCM hub cluster
Ramen catalog installed on each managed cluster participating in DR

Gather Required Information

Before starting configuration, collect the following information:

Workload Management Using Ramen

Mon, 01 Jan 0001 00:00:00 +0000

Workload Management Using Ramen

This guide describes how to protect and manage workloads using Ramen’s disaster recovery capabilities.

Overview

Ramen supports protecting applications across three different deployment types:

GitOps Applications (Recommended) - For applications deployed from Git repositories via ArgoCD ApplicationSets
Discovered Applications - For existing applications deployed via kubectl, Helm, or other methods
Applications with Recipes - For complex stateful applications requiring custom capture/recovery workflows

All methods support the same DR operations:

Using Recipes for Disaster Recovery (DR)

Mon, 01 Jan 0001 00:00:00 +0000

Using Recipes for Disaster Recovery (DR)

Introduction

A recipe is a Kubernetes Custom Resource that defines custom workflows for capturing and recovering application resources and volumes during disaster recovery (DR) operations. Recipes are part of Ramen’s Kubernetes-native Disaster Recovery ecosystem, providing vendor-supplied or custom disaster recovery specifications for complex stateful applications. While Ramen supports multiple workload protection methods (GitOps, discovered applications), Recipes are specifically designed for applications that require:

Logging

Mon, 01 Jan 0001 00:00:00 +0000

Logging

This document outlines the logging standards and best practices for Ramen.

General Principles

Consistency: All controllers should follow the same logging conventions.
Performance Awareness: Logging should not significantly impact the performance. Be mindful of the logging levels and the amount of data logged.

Logger Configuration

Each controller should create its logger instance with a name specific to the controller for easy identification. For example, if the controller is named ProtectedVolumeReplicationGroupList, the logger name could be a shortened form of the controller like pvrgl.
```
logger := ctrl.Log.WithName("pvrgl")
```
Loggers should be enriched with key contextual information to make the logs more informative. This includes:
- The name of the custom resource.
- Then namespace of the custom resource, if applicable.
- The UID of the custom resource.
- The version of the custom resource.
```
logger = logger.WithValues("name", req.NamespacedName.Name, "rid", instance.ObjectMeta.UID, "version", instance.ObjectMeta.ResourceVersion)
```
The context passed to functions should be augmented with the logger instance:
```
ctx = ctrl.LoggerInto(ctx, logger)
```
This will be useful when we call functions that are outside of the controller scope and the instance.log object is not available to them.

drenv metrics

Mon, 01 Jan 0001 00:00:00 +0000

drenv metrics

The drenv stress-test report command automatically collects system metrics from Netdata if it is running. This is important to correlate test failures to system load.

Installing Netdata

macOS:

brew install netdata

Linux:

curl --fail --silent --show-error --location --output /tmp/netdata-kickstart.sh \
 https://get.netdata.cloud/kickstart.sh
bash /tmp/netdata-kickstart.sh --no-updates --stable-channel

Configuring netdata

For stress testing, configure Netdata to use minimal resources and store data on disk (persists across restarts, uses less memory).

Metrics

Mon, 01 Jan 0001 00:00:00 +0000

Metrics

Metrics are collected using Prometheus, and registered with its global metrics registry in each controller. There are two ways where you can look at the metrics in ramen. One way is use prometheus stack(recommended) and the other way is to use curl or postman. More details on each of these in the below sections.

More information on metrics is here

1. Using Prometheus Stack

We recommend to use kube-prometheus . Installing this will give you a containerized stack that includes Prometheus, AlertManager and Grafana.

Testing

Mon, 01 Jan 0001 00:00:00 +0000

Testing

Unit tests and Integration tests

The unit and integration tests for the controller code are written using the ginkgo and gomega testing frameworks. The tests are written in the *_test.go files in the controllers directory.

By default, the tests are run against a single kubernetes api server(https://github.com/kubernetes-sigs/controller-runtime/tree/main/tools/setup-envtest ) provided by the controller-runtime project.

The test framework uses mock interfaces for the components that aren’t installed in the test environment, like ocm, s3 and csi. These mock interfaces are used to fake a success or failure from the components to test the controller code.

End to End testing

Mon, 01 Jan 0001 00:00:00 +0000

End to End testing

RamenDR end-to-end (e2e) tests validate various scenarios for regional DR using predefined workloads and deployment methods.

Running End to End tests

[!IMPORTANT] All commands must be ran from the e2e directory.

Preparing a `config.yaml` file

For drenv environment

If drenv was used to configure RDR clusters, easily copy config.yaml.sample and add the clusters kubeconfig paths into config.yaml using:

Building ramen

Mon, 01 Jan 0001 00:00:00 +0000

Building ramen

Go toolchain

Ramen uses Go’s toolchain management to specify both the minimum required Go version and the recommended toolchain version in go.mod:

The go directive specifies the minimum Go version required to build Ramen.
The toolchain directive specifies the recommended Go toolchain for development. By default, Go automatically downloads this version when a newer toolchain is needed.

Building with an older Go toolchain

If you are building in a disconnected environment or with a Go version older than the toolchain directive, set GOTOOLCHAIN to use the locally installed Go toolchain:

DRPolicy CRD

Mon, 01 Jan 0001 00:00:00 +0000

DRPolicy CRD

Overview

The DRPolicy custom resource defines the disaster recovery topology and replication configuration between peer clusters. It is a cluster-scoped resource created by administrators on the OCM hub cluster that establishes the DR relationship between two managed clusters.

A DRPolicy specifies:

Which two clusters participate in DR
The replication schedule for Async (Regional DR)
Storage class selectors for volume replication
Whether to use async (Regional DR) or sync (Metro DR)

Lifecycle: DRPolicy resources are created once during DR setup and are typically long-lived.

DRCluster CRD

Mon, 01 Jan 0001 00:00:00 +0000

DRCluster CRD

Overview

The DRCluster custom resource represents a managed cluster registered for disaster recovery operations. It is a cluster-scoped resource created by administrators on the OCM hub cluster that provides:

S3 configuration for storing PV metadata and cluster state
Network fencing configuration for Sync (Metro)
Cluster availability and fencing status

DRClusters are referenced by DRPolicy resources to define which clusters participate in DR relationships. Each managed cluster that participates in DR must have a corresponding DRCluster resource on the hub.

DRClusterConfig CRD

Mon, 01 Jan 0001 00:00:00 +0000

DRClusterConfig CRD

Overview

The DRClusterConfig custom resource provides cluster-specific disaster recovery configuration on managed clusters. It is a cluster-scoped resource that exists on each managed cluster (not on the hub) and serves two primary purposes:

Configuration Communication: Conveys the cluster identity and desired replication schedules from the hub to the managed cluster
Capability Discovery: Scans the cluster for storage-related classes (StorageClass, VolumeSnapshotClass, VolumeReplicationClass, etc.) that have the required Ramen labels/annotations and reports them in its status. See the Status Fields section below for the specific label/annotation requirements for each class type.

Note: DRClusterConfig does not create these classes. Storage providers or administrators must create and label/annotate the classes appropriately. DRClusterConfig only discovers and reports them in its status.

DRPlacementControl CRD

Mon, 01 Jan 0001 00:00:00 +0000

DRPlacementControl CRD

Overview

The DRPlacementControl (DRPC) custom resource is the primary interface for protecting and managing applications with disaster recovery capabilities. Created by users in the application namespace on the OCM hub cluster, it orchestrates:

Initial DR protection of applications deployed on the preferred cluster
Failover operations to peer clusters during disasters
Planned relocation between clusters

The DRPC is the control plane for application DR - it creates and manages VolumeReplicationGroup (VRG) resources on managed clusters via ManifestWork, coordinates with OCM Placement, and tracks DR state.

VolumeReplicationGroup CRD

Mon, 01 Jan 0001 00:00:00 +0000

VolumeReplicationGroup CRD

Overview

The VolumeReplicationGroup (VRG) custom resource manages volume replication and Kubernetes object protection for an application on a managed cluster. VRGs are not directly created by users - they are automatically created and managed by the DRPlacementControl (DRPC) on the hub cluster via ManifestWork.

A VRG controls:

Volume replication state (Primary/Secondary) for all PVCs matching the selector
VolumeReplication or VolSync resource creation and management for data replication
PV metadata storage in S3 for cross-cluster recovery
Kubernetes object capture and recovery (via Velero and Recipe)

Lifecycle: Created by DRPC when enabling DR protection. Exists on the active cluster as Primary and on the peer cluster as Secondary. Deleted when DRPC is removed.

MaintenanceMode CRD

Mon, 01 Jan 0001 00:00:00 +0000

MaintenanceMode CRD

Overview

The MaintenanceMode custom resource enables coordination between Ramen and storage backends during disaster recovery operations. It is a cluster-scoped resource created by Ramen on managed clusters to signal storage providers that specific maintenance actions are required before proceeding with DR operations.

Storage backends use MaintenanceMode to:

Prepare storage systems for failover operations
Perform necessary cleanup or synchronization
Report readiness back to Ramen

This allows storage providers to integrate custom logic into Ramen’s DR workflows without Ramen needing to know storage-specific implementation details.

CephFS Regional DR Consistency Group

Mon, 01 Jan 0001 00:00:00 +0000

CephFS Regional DR Consistency Group

Summary

Currently, Ramen only supports non-consistency-group CephFS RDR (Regional Disaster Recovery). However, with the upcoming support for consistency groups in CephFS, Ramen can expand its capabilities to include consistency-group RDR for CephFS.

Motivation

Consistency Groups are pivotal in maintaining data integrity during synchronizing related datasets across regions, mitigating the risk of data corruption or loss during failover events. They constitute essential components of Regional Disaster Recovery, enabling organizations to minimize risks and ensure the availability and integrity of their data across geographical locations.

drenv stress test

Mon, 01 Jan 0001 00:00:00 +0000

drenv stress test

The drenv stress test evaluates drenv start robustness and helps debug failed runs.

Setup

Before running stress tests install Netdata on the host to allow correlating test failures with system load. See metrics for more info.

Running stress test

In this example we run 100 runs starting the regional-dr environment. If a run fails, we delete the clusters and continue. This is useful for understanding what are the most common failures.

Global VolumeGroupReplication Design Document

Mon, 01 Jan 0001 00:00:00 +0000

Global VolumeGroupReplication Design Document

1. Design

Some storage backends manage replication at the LUN or filesystem level, where all applications sharing the same storage unit belong to a single replication group and must transition state together. Global VGR builds on top of Ramen’s offloaded storage support to coordinate a shared VGR across all VRGs in such a replication group, with multi-level consensus, label propagation, status validation, watchers, and observability through conditions, metrics, and alerts.

Handling CVEs in the project

Mon, 01 Jan 0001 00:00:00 +0000

Handling CVEs in the project

Introduction

CVE, short for Common Vulnerabilities and Exposures, is a list of publicly disclosed computer security flaws. When someone refers to a CVE, they mean a security flaw that’s been assigned a CVE ID number.

CVEs filed on dependencies

Checking if a CVE applies to Ramen

Note: Ramen has two go modules, one for the main project and another for the api. You need to perform the check below for both the modules.

Incremental Sync for Consistency Groups

Mon, 01 Jan 0001 00:00:00 +0000

Incremental Sync for Consistency Groups

Extends Cephfs-RDR-ConsistencyGroup.md .

Summary

When drplacementcontrol.ramendr.openshift.io/enable-diff: "true" is set on a DRPC, Ramen preserves VolumeGroupSnapshots across sync cycles and uses the ceph-volsync-plugin External mover to transfer only changed blocks between consecutive snapshots.

Annotation Propagation

DRPC → VRG → RGS (ReplicationGroupSource)
 → RGD (ReplicationGroupDestination)

Follows the same flow as UseVolSyncAnnotation: the user sets the annotation on the DRPC instance, which propagates it to the VRG via ManifestWork. The annotation is whitelisted in constructVRGFromView so it survives hub round-trips. DRPC annotation changes trigger reconcile automatically via the default For(&DRPlacementControl{}) predicate.

VRG reconciler workflow

Mon, 01 Jan 0001 00:00:00 +0000

VRG reconciler workflow

This design document covers the VRG CRD and its reconciliation workflow.

Terminology

VRG: VolumeReplicationGroup
VR: VolumeReplication
PVC: PersistentVolumeClaim
PV: PersistentVolume

VRG operator functionality

VRG resource exists to manage a group PVCs that need replication. It is a namespace level resource that ensures the following:

For each existing or dynamically created PVC:
- Ensures PVCs are protected from deletion, prior to replicating them
- Ensures PVCs are bound before enabling replication for them
- Ensures only PVCs that are NOT being deleted are protected, when reconciling them
- Ensures PVs backing the PVCs are retained for the duration that they are protected
- Backs up the PV cluster data to an external S3 store, to enable restoring them on a remote peer cluster
Creates and manages VR resource for PVCs that need to be protected
Manages state changes in VRG by ensuring VRs reach the same desired state, coordinating with PVC state and ensuring:
- PVCs are not in use when being marked Secondary
- PVCs are marked for deletion when being marked Secondary
Manages VRG deletion, by ensuring VRs are in the desired state prior to deletion

VRG CRD

Spec

apiVersion: ramendr.openshift.io/v1alpha1
kind: VolumeReplicationGroup
metadata:
 name: vrg-sample
 generation: <generation>
spec:
 pvcSelector:
 matchLabels:
 <key>: <value>
 volumeReplicationClass: <string>
 replicationState: "Primary"|"Secondary"
 s3Endpoint: <url-string>
 s3SecretName: <string>

Fields:
- PVCSelector
  - Label selector to identify all the PVCs that are belong to this VRG and need to be replicated
- VolumeReplicationClass
  - ReplicationClass for all volumes in this replication group; this value is propagated to created VolumeReplication CRs
- ReplicationState
  - Desired state of all volumes [primary or secondary] in this replication group; this value is propagated to created VolumeReplication CRs
- S3Profiles
  - S3 Profiles to replicate PV cluster data

Status

status:
 protectedPVCs: [
 pvcName: <string>
 conditions: [
 type: ["Available"]
 status: [true|false|unknown]
 observedGeneration: <generation>
 lastTransitionTime: <time>
 reason: "Error"|"Progressing"|"Replicating"
 message: <human readable string>
 ]
 ]
 conditions: [
 type: ["Available"]
 status: [true|false|unknown]
 observedGeneration: <generation>
 lastTransitionTime: <time>
 reason: ["Replicating"|"Progressing"|"Error"]
 message: <human readable string>
 ]

status.conditions:
- type “Available” is to denote availability status of the VRG resource
  - NOTE: status may change back to “Progressing” in case there are new PVCs to deal with for the same CRD generation
- status
  - “Unknown” if VRG is not yet picked up for reconciliation
  - “True” if reconciliation is completed
    - reason is “Replicating”
  - “False” if reconciliation is progressing or has errors
    - reason: Progressing|Error
status.protectedPVCs:
- pvcName: Name of the PVC for which conditions are represented
- conditions:
  - type “Available” is to denote availability status for PVC protection
  - status
    - “True” if reason is “Replicating”
    - “False” if reason is one of “Error” or “Progressing”, to denote any hard errors or that reconciliation is in progress
  - NOTE(s):
    - Ideally if status.conditions.[Available].status is True, the all PVC status should be true, with the reason as “Replicating”
    - An observedGeneration number for when the PVC was protected may not be required, as once it is protected it stays protected. It may prove useful though if any straggling PVCs were protected during a cluster heal or related operations, to detect prior failover PVCs with the current generation

Notes

Status may change for the same observed generation, as PVCs are created
Should there be a per PVC condition that it’s backing PV was backed up? To denote that replication at storage maybe missing, but PV cluster data may exist?

VRG Example

---
apiVersion: ramendr.openshift.io/v1alpha1
kind: VolumeReplicationGroup
metadata:
 name: vrg-sample
spec:
 pvcSelector:
 matchLabels:
 appname: myapp
 volumeReplicationClass: "vr-class-sample"
 replicationState: "Primary"
 s3Profiles:
 s3ProfileEast
 s3ProfileWest

VRG reconciliation

PVC preparation and protection for all states

All reconciliation state changes need to ensure a PVC is in the right state and protected prior to proceeding with the state change. The following steps are taken to ensure the same:

VRG Type Sequence

Mon, 01 Jan 0001 00:00:00 +0000

VRG Type Sequence

Overview

RamenDR can use Velero to capture Kubernetes object information. By default, Velero will back up all object types in arbitrary order. However, many applications have internal dependencies that require specific ordering by object type. The VRG Type Sequencer attempts to address this deficiency by taking multiple partial backups, with the order defined a the user.

Example Use Case: Backup

Backup Overview

Take an example Backup that requires the following sequence:

Ramen

User quick start guide for testing application DR

User quick start guide for testing application DR

What you’ll need

Ramen developer quick start guide

Ramen developer quick start guide

Contribution Flow

Getting the source

Installing Ramen

Installing Ramen

Overview

Prerequisites

Configure Ramen

Configure Ramen

Prerequisites

Gather Required Information

Workload Management Using Ramen

Workload Management Using Ramen

Overview

Using Recipes for Disaster Recovery (DR)

Using Recipes for Disaster Recovery (DR)

Introduction

Logging

Logging

General Principles

Logger Configuration

drenv metrics

drenv metrics

Installing Netdata

Configuring netdata

Metrics

Metrics

1. Using Prometheus Stack

Testing

Testing

Unit tests and Integration tests

End to End testing

End to End testing

Running End to End tests

Preparing a config.yaml file

For drenv environment

Building ramen

Building ramen

Go toolchain

Building with an older Go toolchain

DRPolicy CRD

DRPolicy CRD

Overview

DRCluster CRD

DRCluster CRD

Overview

DRClusterConfig CRD

DRClusterConfig CRD

Overview

DRPlacementControl CRD

DRPlacementControl CRD

Overview

VolumeReplicationGroup CRD

VolumeReplicationGroup CRD

Overview

MaintenanceMode CRD

MaintenanceMode CRD

Overview

CephFS Regional DR Consistency Group

CephFS Regional DR Consistency Group

Summary

Motivation

drenv stress test

drenv stress test

Setup

Running stress test

Global VolumeGroupReplication Design Document

Global VolumeGroupReplication Design Document

1. Design

Handling CVEs in the project

Handling CVEs in the project

Introduction

CVEs filed on dependencies

Checking if a CVE applies to Ramen

Incremental Sync for Consistency Groups

Preparing a `config.yaml` file