[osuosl-openpower] Possible service disruption caused backend storage

Lance Albertson lance at osuosl.org
Tue Nov 16 02:19:26 UTC 2021


All,

Some of you might be experiencing issues with your VMs or CI jobs on
Power-CI due to an issue we have with our Ceph storage backend. We're in
the process of replacing some failed hard drives and ran into an issue
where some of the data is "stuck" while replication catches up.

We made a mistake and tried replacing one too many drives at a time which
caused this issue. This shouldn't cause any data loss, however any affected
VM should be rebooted. Once the data sync has been completed, we'll be
checking each VM and doing a reboot if we notice they did have I/O issues.

Sorry for the issues that our mistake has caused!

Thanks for your patience-

-- 
Lance Albertson
Director
Oregon State University | Open Source Lab
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osuosl.org/pipermail/openpower/attachments/20211115/92c0d292/attachment.html>


More information about the openpower mailing list