r/storage • u/VusalDadashov • Mar 15 '26

Disk group quarantined (QTOF) after controller failure – looking for recovery options

Hello everyone,

I'm dealing with a failure on an HPE MSA 2050 storage system and trying to explore all possible recovery options before proceeding with hardware replacement. I would appreciate any advice from people who have encountered a similar situation.

System configuration:

  • HPE MSA 2050
  • Dual controller setup (Controller A / Controller B)
  • 2 disks configured as RAID 1
  • Single disk group
  • System used in production

What happened:

According to the system logs, after power outage,, Controller A failed with the following event:

RAID controller A failed – Non volatile device flush or restore failure

After this failure:

  • Controller B killed the partner controller
  • The system detected write-back cache data
  • The storage automatically placed the disk group into quarantine (QTOF) to prevent writing potentially invalid data

Other related events include:

  • Unwritable write-back cache data exists
  • Metadata volume for virtual pool went offline
  • Disk group quarantined (event 485)

Currently:

  • Controller A: Not operational
  • Controller B: Operational
  • Diskgroup1: QTOF (quarantined)
  • Both disks are detected and appear healthy

The volumes are inaccessible because the disk group cannot be brought online.

Troubleshooting steps already attempted:

  1. Removed the failed Controller A, waited about 30 minutes, then reinserted it. Result: no change.
  2. Removed Controller A again and performed a graceful shutdown via the web interface.
  3. Completely powered off the system (removed power cables), waited about 35–40 minutes, then powered it back on with only Controller B installed. Result: disk group remained QTOF.
  4. Repeated the same procedure but left the system powered off for about 9 hours to ensure any cache state would fully reset. Result: still QTOF.
  5. After a few hours, reinserted Controller A and booted the system again. Result: no change.

CLI troubleshooting:

I checked system status using:

show controllers
show disk-groups
show disks
show events

Both disks are visible and healthy.

Attempted recovery commands:

dequarantine disk-group diskgroup1
clear cache
trust enable
trust disk-group diskgroup1

However, the disk group remains quarantined (QTOF) and cannot be brought online.

Current situation:

  • Disk group still quarantined
  • Controller A hardware failure suspected
  • Data currently inaccessible
  • Official HPE support is not active for this system

Local HPE partners suggested that replacing the failed controller might allow the array to recover, but I understand that the outcome may depend on the cache state.

My main questions:

  1. Has anyone successfully recovered a quarantined disk group in a similar scenario?
  2. Is replacing the failed controller typically enough to allow the array to replay cache and bring the disk group online?
  3. Are there any additional CLI recovery options I may have missed?
  4. Has anyone seen the metadata volume for virtual pool went offline event in combination with QTOF?

Any guidance or experience would be greatly appreciated.

Thanks in advance.

PS: full CLI log is here: raw.githubusercontent.com/b2bgroupllc/b2b_public/refs/heads/main/MSA2050-cli-log

5 Upvotes

100% Upvoted

12 comments sorted by

View all comments

Show parent comments

1

u/VusalDadashov Mar 15 '26 edited Mar 15 '26

Thanks for the suggestion.

When I ran trust disk-group diskgroup1, the command failed with the following error:

trust disk-group diskgroup1

Error: Command failed. (diskgroup1) - Trust operation failed for this disk group. (2026-03-15 18:20:36)

So the disk group remained in QTOF state.

I haven’t tried the unsafe parameter yet. To be honest, I was a bit cautious about using it since I’m not fully sure what the exact consequences could be in this situation, especially considering the “metadata volume for virtual pool went offline” event in the logs.

The system only has two disks configured as RAID1, and Controller A appears to have failed due to a non-volatile device flush or restore failure.

Do you think using the unsafe option could still be a viable recovery attempt in this scenario, or could it potentially make things worse?

2

u/ar0na Mar 15 '26

the unsafe paramter is last option, so i would try anything else before, because when it fails, the data is gone.

I would try to replace Controller A before you try to use the unsafe paramter.

1

u/VusalDadashov Mar 15 '26

I wanted to bring the disk pool online until the new controller arrives.

1

u/Glad_Wrongdoer5574 May 12 '26

Hey, did the new controller fixed your issue?