[powerdev-hosting] UNPLANNED: powerdev1 hard disk failure - Jul 22, 2013 2:00pm PST (Jul 22 2100 UTC)

[Simonis, Volker]

Hi Lance,

for me everything works again like before.

Thanks for the fast recovery,
Volker

________________________________
From: powerdev-hosting-bounces at osuosl.org [powerdev-hosting-bounces at osuosl.org] on behalf of Lance Albertson [lance at osuosl.org]
Sent: Tuesday, July 23, 2013 12:52 AM
To: powerdev-hosting at osuosl.org
Subject: Re: [powerdev-hosting] UNPLANNED: powerdev1 hard disk failure - Jul 22, 2013 2:00pm PST (Jul 22 2100 UTC)

This maintenance has been completed. Luckily all the systems were RAID1 that would have been affected so all the systems are back online. I've initiated a rebuild of the RAID1 on all but the debian system which I had access to. You may notice some slow I/O for the next hour or two until the syncs have been completed.

I have also finally documented this whole silly process so it should go more smoothly next time!

Thanks!


On Mon, Jul 22, 2013 at 8:42 AM, Lance Albertson <lance at osuosl.org<mailto:lance at osuosl.org>> wrote:
Service(s) affected: The following LPAR's hosted on powerdev1

ajivana.pg<http://ajivana.pg>
builder1.centos
buildfarm.pg<http://buildfarm.pg>
chinook.fedora
partch.debian
lfdev-build-power64
openjdk
powerdev1-vios
ppcllvm
vac.pg<http://vac.pg>

Outage Window:
Start:   Tue, Jul 22, 2:00PM PST (Tue Jul 22 2100 UTC)

End:    Tue, Jul 22, 5:00PM PST (Wed Jul 23 0000 UTC)

Reason for outage:

We received a service call this morning from IBM that disk #5 in powerdev1 has failed. That means we will need to take all of the LPARS hosted on that machine offline today to repair the disk. Normally this wouldn't be a big deal but because of how these machines are configured, we will have to manually fix each LPAR. Any LPAR that is not using raid1 may have to be rebuilt.

Please copy any data you wish to save and shutdown your LPAR as soon as you can. If your LPAR is still active at 2pm, we will gracefully shut it down for you.

A special note: If we cannot complete the repairs today it might not be until later in the week until we fully fix the LPAR's. Most of us will be attending OSCON however we will do our best to bring LPAR's back online as soon as we can.

Projects affected:

Postgres
CentOS
Fedora
Debian
Linux Foundation
OpenJDK
LLVM

--
Lance Albertson
Director
Oregon State University | Open Source Lab



--
Lance Albertson
Director
Oregon State University | Open Source Lab
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osuosl.org/pipermail/powerdev-hosting/attachments/20130723/4b75a3ba/attachment.html>