From lance at osuosl.org  Tue May 21 19:16:16 2019
From: lance at osuosl.org (Lance Albertson)
Date: Tue, 21 May 2019 12:16:16 -0700
Subject: [openstack-announce] Ongoing VM network connectivity issues
 since Pike upgrade
In-Reply-To: <CAORfjXQ_GUgTw_p1KL3Qr+PTOf3gpEGys1sUiJQQCO7TGZt31Q@mail.gmail.com>
References: <CAORfjXQ_GUgTw_p1KL3Qr+PTOf3gpEGys1sUiJQQCO7TGZt31Q@mail.gmail.com>
Message-ID: <CAORfjXT6roac+VGPyPWtbeTjUvV3193QfH8dfg9HRx53tOtPGQ@mail.gmail.com>

All,

I wanted to send you an update on where we are at on this issue. So far
I've narrowed down the problem to happening when a VM using a private
network is removed causing certain iptable rules on the hypervisor to get
out of order. It only seems to effect inbound connections to the VM as
outbound seems to still work. I haven't been able to easily reproduce the
issue unfortunately which makes it difficult to troubleshoot. I've looked
through the source code and also looked online to see if anyone else had
run into this without success.

I've rebooted all of the hypervisors on our x86 cluster and two on our ppc
cluster (which was needed for the MDS updates). So far on the nodes that
have been rebooted we haven't seen any issues, but I need to let those run
for a few days to verify that theory. These machines were also due for a
reboot also because of the CentOS 7.5 -> 7.6 upgrade so perhaps it's
related to that.

At any rate, I've deployed a temporary cronjob on the nodes that haven't
been rebooted which should "fix" the networking issue. I have it set to run
every minute so that the downtime should be minimal.

I'll send another update as I have one.

Thanks-

On Thu, May 16, 2019 at 8:58 AM Lance Albertson <lance at osuosl.org> wrote:

> All,
>
> Since the upgrade to Pike we've noticed virtual machines suddenly losing
> network connectivity. This issue seems to sometimes fix itself or when we
> restart the  neutron-linuxbridge-agent service on the hypervisors. We are
> doing our best to track down why this is happening and how to fix it. Since
> we're not monitoring every host on the cluster, it's difficult for us to
> know when it happens so if you do have a problem with one of your VMs,
> please let us know either via IRC in #osuosl on Freenode, or via a support
> email.
>
> I'll be sending further updates as we have them.
>
> Thanks for your patience!
>
> --
> Lance Albertson
> Director
> Oregon State University | Open Source Lab
>


-- 
Lance Albertson
Director
Oregon State University | Open Source Lab
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osuosl.org/pipermail/openstack-announce/attachments/20190521/e97a26e7/attachment.html>