From lance at osuosl.org  Mon Jul 26 22:06:18 2021
From: lance at osuosl.org (Lance Albertson)
Date: Mon, 26 Jul 2021 15:06:18 -0700
Subject: [osuosl-openpower] Backend storage issues
Message-ID: <CAORfjXQvURNcM8WV_+id9LCarTRG8TNrDm=cJrXfuS+AshjN=g@mail.gmail.com>

All,

We encountered a momentary issue with the backend storage while rebooting
some of the nodes. This has caused several VMs to be stuck in an unstable
state. We're going through all of the affected VMs and forcing a reboot on
them to resolve the issue.

Apologies for outages this may have caused.

Thanks!

-- 
Lance Albertson
Director
Oregon State University | Open Source Lab
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osuosl.org/pipermail/openpower/attachments/20210726/03318f68/attachment.html>

From lance at osuosl.org  Tue Jul 27 00:28:48 2021
From: lance at osuosl.org (Lance Albertson)
Date: Mon, 26 Jul 2021 17:28:48 -0700
Subject: [osuosl-openpower] Backend storage issues
In-Reply-To: <CAORfjXQvURNcM8WV_+id9LCarTRG8TNrDm=cJrXfuS+AshjN=g@mail.gmail.com>
References: <CAORfjXQvURNcM8WV_+id9LCarTRG8TNrDm=cJrXfuS+AshjN=g@mail.gmail.com>
Message-ID: <CAORfjXQjp_k731rOG8TO4d3ccFhEDZsDXMzhi9dsh39O6QKiEg@mail.gmail.com>

It looks as though all of the VMs were affected by this issue and will
require a hard reset. Since I need to reboot the hypervisors anyway, I'm
going to go ahead and do that on each node and let it shutdown the VMs
instead of doing a live migrations. This will impact every VM on the
cluster. I will try and spot check the VMs to ensure they're back online
after the reboot.

Again, sorry for the issues this is causing.

Thanks-

On Mon, Jul 26, 2021 at 3:06 PM Lance Albertson <lance at osuosl.org> wrote:

> All,
>
> We encountered a momentary issue with the backend storage while rebooting
> some of the nodes. This has caused several VMs to be stuck in an unstable
> state. We're going through all of the affected VMs and forcing a reboot on
> them to resolve the issue.
>
> Apologies for outages this may have caused.
>
> Thanks!
>
> --
> Lance Albertson
> Director
> Oregon State University | Open Source Lab
>


-- 
Lance Albertson
Director
Oregon State University | Open Source Lab
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osuosl.org/pipermail/openpower/attachments/20210726/2d77d404/attachment.html>

From lance at osuosl.org  Tue Jul 27 03:13:06 2021
From: lance at osuosl.org (Lance Albertson)
Date: Mon, 26 Jul 2021 20:13:06 -0700
Subject: [osuosl-openpower] Backend storage issues
In-Reply-To: <CAORfjXQjp_k731rOG8TO4d3ccFhEDZsDXMzhi9dsh39O6QKiEg@mail.gmail.com>
References: <CAORfjXQvURNcM8WV_+id9LCarTRG8TNrDm=cJrXfuS+AshjN=g@mail.gmail.com>
 <CAORfjXQjp_k731rOG8TO4d3ccFhEDZsDXMzhi9dsh39O6QKiEg@mail.gmail.com>
Message-ID: <CAORfjXSBagUGs0MyvSouK7_5dFXfQOVso3KGqiKj1H-i2TbA0A@mail.gmail.com>

I just completed the reboots. All the systems should be back online. Please
let me know if you have any issue with your VM(s).

Thanks!

On Mon, Jul 26, 2021 at 5:28 PM Lance Albertson <lance at osuosl.org> wrote:

> It looks as though all of the VMs were affected by this issue and will
> require a hard reset. Since I need to reboot the hypervisors anyway, I'm
> going to go ahead and do that on each node and let it shutdown the VMs
> instead of doing a live migrations. This will impact every VM on the
> cluster. I will try and spot check the VMs to ensure they're back online
> after the reboot.
>
> Again, sorry for the issues this is causing.
>
> Thanks-
>
> On Mon, Jul 26, 2021 at 3:06 PM Lance Albertson <lance at osuosl.org> wrote:
>
>> All,
>>
>> We encountered a momentary issue with the backend storage while rebooting
>> some of the nodes. This has caused several VMs to be stuck in an unstable
>> state. We're going through all of the affected VMs and forcing a reboot on
>> them to resolve the issue.
>>
>> Apologies for outages this may have caused.
>>
>> Thanks!
>>
>> --
>> Lance Albertson
>> Director
>> Oregon State University | Open Source Lab
>>
>
>
> --
> Lance Albertson
> Director
> Oregon State University | Open Source Lab
>


-- 
Lance Albertson
Director
Oregon State University | Open Source Lab
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osuosl.org/pipermail/openpower/attachments/20210726/30c419cf/attachment.html>