From lance at osuosl.org Wed Jan 3 01:07:48 2024 From: lance at osuosl.org (Lance Albertson) Date: Tue, 2 Jan 2024 17:07:48 -0800 Subject: [osuosl-openpower] Controller and Hypervisor OS Upgrades Message-ID: All, This month we will be going through all of our controller and hypervisor nodes and upgrading them to AlmaLinux 8 from CentOS 7. Starting tomorrow, we will be migrating to a new controller node. I expect there to be a short outage (10 minutes or less hopefully) for controller services (API, web interface and private project networks) while we migrate the services over to the new nodes for each cluster. After the controllers have been migrated, we will start doing upgrades on all the hypervisor nodes. We will attempt to use live migration on VMs to minimize impact, however this may not work since we'll be jumping versions of KVM. If live migration does not work, we will need to do a "cold" migration which requires shutting down VMs gracefully and starting them back up. This will happen whenever we start an upgrade on a hypervisor. The one exception to this will be the nodes which are using local storage. These will need to do an in-place upgrade and will require more downtime unfortunately. We will send out a more precise window for when we do those upgrades later this month. I will send out periodic updates if needed. I expect this to last a few weeks between all the various nodes we need to do this on. If you have any questions, please let me know. -- Lance Albertson Director Oregon State University | Open Source Lab -------------- next part -------------- An HTML attachment was scrubbed... URL: From lance at osuosl.org Thu Jan 4 16:15:32 2024 From: lance at osuosl.org (Lance Albertson) Date: Thu, 4 Jan 2024 08:15:32 -0800 Subject: [osuosl-openpower] Controller and Hypervisor OS Upgrades In-Reply-To: References: Message-ID: The controller migration happened yesterday with only a few minor issues that lingered throughout the day that were eventually resolved. The previous controller node is offline so if you were still using the old API endpoints, they will be failing now. We will be starting upgrades on hypervisors next. If there are any lingering issues, please let me know. On Tue, Jan 2, 2024 at 5:07?PM Lance Albertson wrote: > All, > > This month we will be going through all of our controller and hypervisor > nodes and upgrading them to AlmaLinux 8 from CentOS 7. Starting tomorrow, > we will be migrating to a new controller node. I expect there to be a short > outage (10 minutes or less hopefully) for controller services (API, web > interface and private project networks) while we migrate the services over > to the new nodes for each cluster. > > After the controllers have been migrated, we will start doing upgrades on > all the hypervisor nodes. We will attempt to use live migration on VMs to > minimize impact, however this may not work since we'll be jumping versions > of KVM. If live migration does not work, we will need to do a "cold" > migration which requires shutting down VMs gracefully and starting them > back up. This will happen whenever we start an upgrade on a hypervisor. > > The one exception to this will be the nodes which are using local storage. > These will need to do an in-place upgrade and will require more downtime > unfortunately. We will send out a more precise window for when we do those > upgrades later this month. > > I will send out periodic updates if needed. I expect this to last a few > weeks between all the various nodes we need to do this on. If you have any > questions, please let me know. > > -- > Lance Albertson > Director > Oregon State University | Open Source Lab > -- Lance Albertson Director Oregon State University | Open Source Lab -------------- next part -------------- An HTML attachment was scrubbed... URL: