Backup RAID Rebuild Saturday 2nd December 2023 18:15:00

Our backup server's RAID is currently undergoing enhancements to increase performance and accommodate increased storage demands. We have expanded the array by adding two 18TB datacenter SAS Drives.

All existing backups can still be restored anytime.

During this process, VM backups are temporarily paused. We appreciate your understanding as we work to complete the upgrades swiftly.

Unfortunately all data stored using the backup service is lost. The system has been redeployed and is back in operation since 03.02.2024 15:08:26 UTC

To see the full incident report see the below Link: https://nextcloud.prohosting24.de/s/PADsmGjTtA3xwS8

Posted 3 months ago by admin

At the time being Backups also can not be restored anymore. Unfortunately the array went into a failed state and is going to recover first after the reshape has finished.

Until then it is not safe to use the array, therefore we have deactivated the backup service until further notice. The current reshape process is estimated to finish 31.01.2024 17:42:38 UTC.

So far no data has been corrupted or lost.

Posted 3 months ago by admin

The reshape in the course of the added hard disks creates a load that impairs the recovery of the failed hard disk. We do not accept the risk of possible data loss and therefore pause the backups again for at least 24 hours.

The backups of the last few days can still be restored as usual from the customer area.

Posted 3 months ago by admin

The backup service is currently undergoing a recovery process in response to a failed drive in the RAID array, leading to performance issues. Concurrently, we are expanding the array by incorporating two additional 18TB enterprise drives.

Posted 3 months ago by admin

The issue didn't occur so far anymore.

Posted 4 months ago by admin

Backups can be written again, we are still watching the situation.

Posted 4 months ago by admin

Unfortunately we have to continue our maintenance on the Backup Service by extending it again by two 18TB drives. We will keep you updated, we have been able to store new Backups of all VMs since our last maintenance.

It is again still possible to restore your existing backups anytime.

Posted 4 months ago by admin

Backups can be written again.

Posted 5 months ago by admin