[powerdev-hosting] UNPLANNED: powerdev1 hard disk failure - Jul 22, 2013 2:00pm PST (Jul 22 2100 UTC)

[Lance Albertson]

This maintenance has been completed. Luckily all the systems were RAID1
that would have been affected so all the systems are back online. I've
initiated a rebuild of the RAID1 on all but the debian system which I had
access to. You may notice some slow I/O for the next hour or two until the
syncs have been completed.

I have also finally documented this whole silly process so it should go
more smoothly next time!

Thanks!


On Mon, Jul 22, 2013 at 8:42 AM, Lance Albertson <lance at osuosl.org> wrote:

> Service(s) affected: The following LPAR's hosted on powerdev1
>
> ajivana.pg
> builder1.centos
> buildfarm.pg
> chinook.fedora
> partch.debian
> lfdev-build-power64
> openjdk
> powerdev1-vios
> ppcllvm
> vac.pg
>
> Outage Window:
> Start:   Tue, Jul 22, 2:00PM PST (Tue Jul 22 2100 UTC)
>
> End:    Tue, Jul 22, 5:00PM PST (Wed Jul 23 0000 UTC)
>
> Reason for outage:
>
> We received a service call this morning from IBM that disk #5 in powerdev1
> has failed. That means we will need to take all of the LPARS hosted on that
> machine offline today to repair the disk. Normally this wouldn't be a big
> deal but because of how these machines are configured, we will have to
> manually fix each LPAR. Any LPAR that is not using raid1 may have to be
> rebuilt.
>
> Please copy any data you wish to save and shutdown your LPAR as soon as
> you can. If your LPAR is still active at 2pm, we will gracefully shut it
> down for you.
>
> A special note: If we cannot complete the repairs today it might not be
> until later in the week until we fully fix the LPAR's. Most of us will be
> attending OSCON however we will do our best to bring LPAR's back online as
> soon as we can.
>
> Projects affected:
>
> Postgres
> CentOS
> Fedora
> Debian
> Linux Foundation
> OpenJDK
> LLVM
>
> --
> Lance Albertson
> Director
> Oregon State University | Open Source Lab
>



-- 
Lance Albertson
Director
Oregon State University | Open Source Lab
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osuosl.org/pipermail/powerdev-hosting/attachments/20130722/91cce24a/attachment-0001.html>