seikaikyo/testing-ransomware-recovery-procedures

Testing Ransomware Recovery Procedures

When to Use

Use this skill when:

• Validating that ransomware recovery plans actually work under realistic conditions
• Measuring RTO (Recovery Time Objective) and RPO (Recovery Point Objective) against business requirements
• Testing backup restore operations to confirm data integrity and completeness after simulated encryption
• Conducting tabletop exercises or live recovery drills for ransomware scenarios
• Auditing disaster recovery readiness as part of compliance or cyber insurance requirements

Do not use for active incident response during a live ransomware attack. Use dedicated IR playbooks instead.

Prerequisites

• Isolated recovery test environment (air-gapped or network-segmented lab)
• Access to backup infrastructure (Veeam, Commvault, Rubrik, AWS Backup, Azure Backup)
• Documented RTO/RPO targets per application tier from business impact analysis
• Backup copies available for restore testing (production replicas or test snapshots)
• Recovery runbooks with step-by-step procedures for each critical system

Workflow

Step 1: Define Recovery Test Scope

Identify critical systems and their tiered recovery targets:

| Tier | System Type | RTO Target | RPO Target | Example | |------|------------|------------|------------|---------| | Tier 1 | Mission-critical | < 1 hour | < 15 min | Active Directory, core database | | Tier 2 | Business-critical | < 4 hours | < 1 hour | ERP, email, CRM | | Tier 3 | Business-operational | < 24 hours | < 4 hours | File shares, internal apps | | Tier 4 | Non-critical | < 72 hours | < 24 hours | Dev/test, analytics |

Step 2: Prepare Test Environment

# Verify isolated recovery network is segmented
# No routes to production should exist
ip route show | grep -v "192.168.100.0/24"  # recovery VLAN only

# Verify backup catalog is accessible
restic snapshots --repo s3:s3.amazonaws.com/backup-bucket --password-file /etc/restic/pw
# Or for Veeam:
# Get-VBRBackup | Where-Object {$_.JobType -eq "Backup"} | Select Name, LastPointCreationTime

Step 3: Execute Restore and Measure RTO

For each tiered system, measure the full recovery timeline:

1. Detection to Decision - Time from simulated alert to restore decision
2. Backup Locate - Time to identify and select the correct clean restore point
3. Restore Execution - Time to restore data/VM/application from backup
4. Validation - Time to verify data integrity and application functionality
5. Service Restoration - Time until the system is fully operational

Recovery Timeline Measurement:
  T0: Incident declared (simulated ransomware detection)
  T1: Recovery team assembled and backup identified
  T2: Restore initiated from clean backup
  T3: Restore completed, integrity checks passed
  T4: Application validated and service restored

  Actual RTO = T4 - T0
  Actual RPO = T0 - backup_timestamp

Step 4: Validate Data Integrity Post-Restore

# Compare file counts between backup manifest and restored data
find /restored/data -type f | wc -l
# Compare against pre-backup manifest

# Verify database consistency after restore
pg_isready -h localhost -p 5432
psql -c "SELECT count(*) FROM critical_table;" -d restored_db

# Hash verification of critical files
sha256sum /restored/data/critical_config.xml
# Compare against known-good hash from backup manifest

Step 5: Test Credential Rotation and Security Hardening

After restore, validate that security controls are re-established:

1. Rotate all service account passwords and API keys
2. Verify MFA is enabled on all administrative accounts
3. Confirm EDR/AV agents are running and reporting to management console
4. Validate firewall rules block known C2 indicators
5. Check that restored systems have latest security patches

Step 6: Document Results and Calculate Gap

Recovery Test Report:
  System: [Name]
  Tier: [1-4]
  RTO Target: [target]    Actual RTO: [measured]    Gap: [delta]
  RPO Target: [target]    Actual RPO: [measured]    Gap: [delta]
  Data Integrity: [PASS/FAIL]
  Application Validation: [PASS/FAIL]
  Security Controls Restored: [PASS/FAIL]

  Status: [MEETS TARGET / EXCEEDS TARGET / FAILS TARGET]
  Remediation Required: [description if FAILS]

Key Concepts

| Term | Definition | |------|-----------| | RTO | Recovery Time Objective: maximum acceptable downtime for a system after a disaster | | RPO | Recovery Point Objective: maximum acceptable data loss measured in time | | WRT | Work Recovery Time: time to verify system integrity after restore completes | | MTD | Maximum Tolerable Downtime: absolute limit before unacceptable business impact | | Clean Restore Point | A backup verified to be free of ransomware artifacts or encryption | | Recovery Sequencing | The order in which interdependent systems must be restored | | Air-Gapped Backup | Backup stored on media physically disconnected from the network |

Tools & Systems

| Tool | Purpose | |------|---------| | Veeam Backup & Replication | VM and physical server backup and restore | | Commvault | Enterprise data protection and recovery orchestration | | Rubrik | Cloud-native backup with ransomware recovery SLA | | AWS Backup | Centralized backup for AWS services | | Azure Backup | Microsoft cloud backup with immutable vault | | Restic | Open-source encrypted backup tool | | Velero | Kubernetes cluster backup and restore |

Common Pitfalls

• Not testing restores regularly: Backups that are never tested often fail when needed. Test quarterly at minimum.
• Ignoring recovery sequencing: Restoring an application before its database dependency causes cascading failures.
• Skipping credential rotation: Restored systems may contain compromised credentials that allow re-infection.
• Using production network for testing: Recovery tests on production networks risk spreading simulated or real infections.
• Measuring RTO without WRT: Restore completion is not recovery completion. Include validation and hardening time.
• No immutable backups: If ransomware can encrypt or delete backups, recovery is impossible. Use air-gapped or immutable storage.

References

• NIST SP 800-184: Guide for Cybersecurity Event Recovery
• CISA Ransomware Guide: https://www.cisa.gov/stopransomware
• Veeam RTO/RPO Best Practices: https://www.veeam.com/blog/recovery-time-recovery-point-objectives.html
• NIST CSF 2.0 RC.RP (Recovery Planning)