End to End testing

End to End testing

RamenDR end-to-end (e2e) tests validate various scenarios for regional DR using predefined workloads and deployment methods.

Running End to End tests

[!IMPORTANT] All commands must be ran from the e2e directory.

Preparing a config.yaml file

For drenv environment

If drenv was used to configure RDR clusters, easily copy config.yaml.sample and add the clusters kubeconfig paths into config.yaml using:

cat config.yaml.sample ~/.config/drenv/rdr/config.yaml > config.yaml

If the drenv tool was used to configure RDR clusters with the KubeVirt addon for deploying VirtualMachine workloads using the configuration file located at envs/regional-dr-kubevirt.yaml, follow these steps to complete the e2e setup for VM workload testing:

cat config-vm.yaml.sample ~/.config/drenv/rdr-kubevirt/config.yaml > config.yaml

To test all recipe configurations use the config-recipe.yaml.sample:

cat config-recipe.yaml.sample ~/.config/drenv/rdr/config.yaml > config.yaml

For real cluster

Create a config.yaml file by copying the config.yaml.sample template:

cp config.yaml.sample config.yaml

Update config.yaml by uncommenting and adding cluster kubeconfig paths for the hub and managed clusters.

Clusters:
  hub:
    kubeconfigpath: /path/to/kubeconfig/hub
  c1:
    kubeconfigpath: /path/to/kubeconfig/c1
  c2:
    kubeconfigpath: /path/to/kubeconfig/c2

Validating the clusters

Before running tests it is useful to validate that the clusters are accessible and ready for testing. You can verify it using:

./run.sh -test.run TestValidation

Example output:

2025-03-06T16:52:54.122+0200    INFO    Using config file "config.yaml"
2025-03-06T16:52:54.122+0200    INFO    Using log file "ramen-e2e.log"
2025-03-06T16:52:54.127+0200    INFO    Using Timeout: 10m0s
2025-03-06T16:52:54.127+0200    INFO    Using RetryInterval: 5s
...
2025-03-06T16:52:54.149+0200    INFO    Ramen hub operator pod "ramen-hub-operator-865bdf6799-bgxkn" is running in cluster "hub"
2025-03-06T16:52:54.152+0200    INFO    Ramen dr cluster operator pod "ramen-dr-cluster-operator-67dff877f5-vntt7" is running in cluster "dr1"
2025-03-06T16:52:54.152+0200    INFO    Ramen dr cluster operator pod "ramen-dr-cluster-operator-67dff877f5-6v5sh" is running in cluster "dr2"
--- PASS: TestValidation (0.00s)
    --- PASS: TestValidation/hub (0.02s)
    --- PASS: TestValidation/c1 (0.03s)
    --- PASS: TestValidation/c2 (0.03s)
PASS

Our clusters are ready for testing!

Running DR tests

To run all the DR tests run the TestDR test:

./run.sh -test.run TestDR

The test perform a full DR flow with a tiny workload with multiple deployemnet methods and storage configurations.

[!TIP] The tests typically complete in 10 minutes, depending the machine running the tests.

When all tests complete we will see a test summary showing the status of all tests and the time to complete every step:

--- PASS: TestDR (6.13s)
    --- PASS: TestDR/subscr-deploy-rbd (497.76s)
        --- PASS: TestDR/subscr-deploy-rbd/Deploy (10.21s)
        --- PASS: TestDR/subscr-deploy-rbd/Enable (90.24s)
        --- PASS: TestDR/subscr-deploy-rbd/Failover (185.35s)
        --- PASS: TestDR/subscr-deploy-rbd/Relocate (150.29s)
        --- PASS: TestDR/subscr-deploy-rbd/Disable (55.45s)
        --- PASS: TestDR/subscr-deploy-rbd/Undeploy (6.22s)
    --- PASS: TestDR/disapp-deploy-rbd (558.78s)
        --- PASS: TestDR/disapp-deploy-rbd/Deploy (7.75s)
        --- PASS: TestDR/disapp-deploy-rbd/Enable (90.19s)
        --- PASS: TestDR/disapp-deploy-rbd/Failover (214.69s)
        --- PASS: TestDR/disapp-deploy-rbd/Relocate (172.57s)
        --- PASS: TestDR/disapp-deploy-rbd/Disable (62.45s)
        --- PASS: TestDR/disapp-deploy-rbd/Undeploy (11.13s)
    --- PASS: TestDR/disapp-recipe-check-exec-deploy-rbd (560.01s)
        --- PASS: TestDR/disapp-recipe-check-exec-deploy-rbd/Deploy (7.88s)
        --- PASS: TestDR/disapp-recipe-check-exec-deploy-rbd/Enable (95.19s)
        --- PASS: TestDR/disapp-recipe-check-exec-deploy-rbd/Failover (209.65s)
        --- PASS: TestDR/disapp-recipe-check-exec-deploy-rbd/Relocate (172.89s)
        --- PASS: TestDR/disapp-recipe-check-exec-deploy-rbd/Disable (63.50s)
        --- PASS: TestDR/disapp-recipe-check-exec-deploy-rbd/Undeploy (10.89s)
    --- PASS: TestDR/disapp-deploy-cephfs (653.84s)
        --- PASS: TestDR/disapp-deploy-cephfs/Deploy (7.86s)
        --- PASS: TestDR/disapp-deploy-cephfs/Enable (95.19s)
        --- PASS: TestDR/disapp-deploy-cephfs/Failover (210.47s)
        --- PASS: TestDR/disapp-deploy-cephfs/Relocate (268.03s)
        --- PASS: TestDR/disapp-deploy-cephfs/Disable (57.52s)
        --- PASS: TestDR/disapp-deploy-cephfs/Undeploy (14.77s)
    --- PASS: TestDR/subscr-deploy-cephfs (738.26s)
        --- PASS: TestDR/subscr-deploy-cephfs/Deploy (10.19s)
        --- PASS: TestDR/subscr-deploy-cephfs/Enable (90.26s)
        --- PASS: TestDR/subscr-deploy-cephfs/Failover (305.74s)
        --- PASS: TestDR/subscr-deploy-cephfs/Relocate (265.41s)
        --- PASS: TestDR/subscr-deploy-cephfs/Disable (59.50s)
        --- PASS: TestDR/subscr-deploy-cephfs/Undeploy (7.15s)
    --- PASS: TestDR/appset-deploy-rbd (777.94s)
        --- PASS: TestDR/appset-deploy-rbd/Deploy (20.13s)
        --- PASS: TestDR/appset-deploy-rbd/Enable (95.23s)
        --- PASS: TestDR/appset-deploy-rbd/Failover (360.73s)
        --- PASS: TestDR/appset-deploy-rbd/Relocate (255.39s)
        --- PASS: TestDR/appset-deploy-rbd/Disable (40.37s)
        --- PASS: TestDR/appset-deploy-rbd/Undeploy (6.10s)
    --- PASS: TestDR/appset-deploy-cephfs (1139.62s)
        --- PASS: TestDR/appset-deploy-cephfs/Deploy (20.13s)
        --- PASS: TestDR/appset-deploy-cephfs/Enable (95.23s)
        --- PASS: TestDR/appset-deploy-cephfs/Failover (420.80s)
        --- PASS: TestDR/appset-deploy-cephfs/Relocate (555.94s)
        --- PASS: TestDR/appset-deploy-cephfs/Disable (40.40s)
        --- PASS: TestDR/appset-deploy-cephfs/Undeploy (7.11s)
PASS

All tests completed successfully!

Tests configuration

The tests are defined in the configuration file. Each test specifies a deployer name, workload name, and PVCSpec. The PVCSpec and Deployer names should match a name in the PVCSpecs and Deployers sections of the configuration file.:

tests:
  - deployer: appset
    workload: deploy
    pvcspec: rbd
  ...

By default, the test name is generated from the configuration as “TestDR/{deployer}-{workload}-{pvcspec}”. See Running DR tests section for complete test list.

Deployers

The deployers section defines the available deployment methods. Each deployer has a name, type, and description. The type is used to identify the deployer implementation. There are 3 types available, appset, subscr, and disapp. For disapp, optional fields for recipe are added. The description provides additional context about the deployer.

Testing multiple applications

By default, each combination of deployer, workload, and pvcSpec must be unique. To run multiple applications with the same configuration, add an explicit name to each test. This is useful for stress testing storage by running multiple applications concurrently on the same storage class.

For example, testing 4 CephFS applications concurrently using the discovered application deployer:

tests:
  - name: cephfs-1
    deployer: disapp
    workload: deploy
    pvcspec: cephfs
  - name: cephfs-2
    deployer: disapp
    workload: deploy
    pvcspec: cephfs
  - name: cephfs-3
    deployer: disapp
    workload: deploy
    pvcspec: cephfs
  - name: cephfs-4
    deployer: disapp
    workload: deploy
    pvcspec: cephfs

Each test gets its own namespace (test-cephfs-1, test-cephfs-2, etc.) and independent DR resources. The name must be a valid DNS label (lowercase alphanumeric and hyphens, max 63 characters including the namespace prefix).

Run specific DR tests

Running specific tests is commonly used when debugging a failing test, developing a new test, or working on a new deployer or workload. It allows to selectively execute specific tests by matching full test names using regular expressions, making it easier to focus on specific scenarios.

Run a single DR test

Example:

./run.sh -test.run TestDR/subscr-deploy-rbd

This command runs the specific test for subscription based RBD busybox application.

Run DR tests using a specific deployer

Example:

./run.sh -test.run TestDR/appset

This command runs all DR tests related to ApplicationSet, covering both RBD and CephFS PVC based applications. Useful when focusing on a specific deployer.

Run DR tests using a specific storage

Example:

./run.sh -test.run TestDR/rbd

This command runs all DR tests related to RBD PVCs across all deployers. Ideal for verifying functionality specific to a storage type.

Run individual DR steps

For manual testing and debugging, you can run individual DR operations step by step:

Deploy an appset based test application:

./run.sh -test.run TestDR/appset-deploy-rbd/Deploy

Protect the application:

./run.sh -test.run TestDR/appset-deploy-rbd/Enable

Failover to the secondary cluster:

./run.sh -test.run TestDR/appset-deploy-rbd/Failover

Relocate back to the primary cluster:

./run.sh -test.run TestDR/appset-deploy-rbd/Relocate

Disable DR protection:

./run.sh -test.run TestDR/appset-deploy-rbd/Disable

Undeploy the application:

./run.sh -test.run TestDR/appset-deploy-rbd/Undeploy

Using multiple config files

Use this option if you want to maintain multiple configuration files and run tests using a specific one. Example usage:

./run.sh -config my_config.yaml

Stress testing

Stress test configurations run multiple instances of the same application concurrently to investigate storage performance under load. This is useful for detecting issues like CephFS quiesce serialization during failover.

Available stress test configurations:

  • config-stress-cephfs.yaml.sample - 8 CephFS applications
  • config-stress-rbd.yaml.sample - 8 RBD applications

To run a CephFS stress test:

cat config-stress-cephfs.yaml.sample ~/.config/drenv/rdr/config.yaml > config.yaml
./run.sh -test.run TestDR

To run an RBD stress test:

cat config-stress-rbd.yaml.sample ~/.config/drenv/rdr/config.yaml > config.yaml
./run.sh -test.run TestDR