Seti is down again

[arkayn]

 Active
Unscheduled Outage – Emergency Shutdown Data Center Servers

Outage Type: UNSCHEDULED OUTAGE
Date Submitted: September 30, 2013
Outage Start/End Time: 18:15 – TBD
Groups Impacted: campus
Equipment: Data Center Servers

Description: UPDATE 19:15 – Power is returning to the Data Center but systems remain shutdown, temperature is slowly dropping.  Servers continue to be monitored.

Due to the campus widespread power outage the Data Center servers are experiencing extreme heat impacting central servers residing in the Data Center.  With temperatures rising over 100 degrees , all servers are gracefully being shutdown.

Services Impacted:

Calmail

Bconnected

Applications hosted on VM

All Database Systems

Telecom Gateways impacting Voice Services

BSpace

BCourses

CalPlanning

BAIRS

[msattler]

Well, let's hope the guys at the colo haven't lost their cheat sheets on what order in which to restart the whole Seti shebang.

[BANZAI56]

Warm servers -  not good.

[msattler]

Last update was that power has been restored, things are cool enough to restart servers.
Of course, I am sure that servers controlling campus based servers have priority over getting Seti back online.
Hopefully in the next hour or two.

[msattler]

Well....something's coming up.
Uploads are going through.  But no web pages or other server comms.
Cannot fetch scheduler list or make any scheduler requests.
Dunno if it's the colo guys trying to start things up, or if the boyz in da lab are getting their fingers into the servers.
Hope nothing got damaged due to overheat.

EDIT....
Just as I posted the above, the server status and web pages came back......down for maintenance.
As it's the usual Tuesday maintenance window, I suppose they'll tackle that and check everything before coming back up.

[arkayn]

http://www.sfgate.com/bayarea/article/UC-Berkeley-campus-evacuated-power-outage-4858201.php

[arkayn]

Beta is up, but main is still down.

[Richard Haselgrove]

Beta is up, but main is still down.

And now, all seems to be working. Reported completed work, and received new work in return.

[RottenMutt]

67 pages, i see things have not gotten better during my summer hiatus! 

[Claggy]

67 pages, i see things have not gotten better during my summer hiatus! 

What are you talking about?

Only ONE further page over the summer, and only three outages, one for loss of Network connectivity, one for loss of Power, and one unspecified, a definite improvement,

If you want guarantied 24/7/365 uptime, then you'll need to donate the tens or hundreds of millions of dollars for the redundant power supplies, redundant internet connections, and redundant servers,
then spread them across the world, and pay for the people to look after them.

Claggy

[arkayn]

Looks like muarae1 might be having problems as it usually is not down during the scheduled maintenance, and the fact that it is only 0730 in Berkeley.

[edit]
Looks like it happened around 0630 Berkeley time as I got work at 0550 just fine.

My current time is MST, 1 hour off Berkeley.

Code: [Select]

31/12/2013 07:29:52 | SETI@home | Started upload of 12mr13ad.3117.1396.438086664197.12.87_0_0
31/12/2013 07:29:57 | SETI@home | Finished upload of 12mr13ad.3117.1396.438086664197.12.87_0_0
31/12/2013 07:31:18 | SETI@home | Computation for task 12mr13aa.1426.1703.438086664204.12.83_0 finished
31/12/2013 07:31:18 | SETI@home | Starting task 12mr13ad.3117.1396.438086664197.12.100_0 using setiathome_v7 version 700 (cuda50) in slot 2
31/12/2013 07:31:21 | SETI@home | Started upload of 12mr13aa.1426.1703.438086664204.12.83_0_0
31/12/2013 07:31:43 |  | Project communication failed: attempting access to reference site
31/12/2013 07:31:43 | SETI@home | Temporarily failed upload of 12mr13aa.1426.1703.438086664204.12.83_0_0: connect() failed
31/12/2013 07:31:43 | SETI@home | Backing off 00:03:30 on upload of 12mr13aa.1426.1703.438086664204.12.83_0_0
31/12/2013 07:31:45 |  | Internet access OK - project servers may be temporarily down.
31/12/2013 07:33:48 | SETI@home | [sched_op] Starting scheduler request
31/12/2013 07:33:48 | SETI@home | Sending scheduler request: To fetch work.
31/12/2013 07:33:48 | SETI@home | Reporting 15 completed tasks
31/12/2013 07:33:48 | SETI@home | Requesting new tasks for NVIDIA
31/12/2013 07:33:48 | SETI@home | [sched_op] CPU work request: 0.00 seconds; 0.00 devices
31/12/2013 07:33:48 | SETI@home | [sched_op] NVIDIA work request: 517862.88 seconds; 0.00 devices
31/12/2013 07:33:50 | SETI@home | Scheduler request completed: got 0 new tasks
31/12/2013 07:33:50 | SETI@home | Project is temporarily shut down for maintenance
31/12/2013 07:33:50 | SETI@home | Project requested delay of 3600 seconds
31/12/2013 07:33:50 | SETI@home | [sched_op] Deferring communication for 01:00:00
31/12/2013 07:33:50 | SETI@home | [sched_op] Reason: project is down


[Mike]

Yes, its down since 6:30 Berkeley time.

[arkayn]

Looks like muarae1 up and crashed again as the site does not load at all, but we can still get work just fine.

[Dirk]

From the Cricket graph is looks like since ~ 7 hours no U/L possible.
My BOINC have too much backlogged U/Ls, so no new work request.
But my BOINC have still WUs for calculation for the next ~ 2 1/2 days ..

[RottenMutt]

gpu work only lasted for a few hours.

57 degrees F inside and getting colder, 22.9 degrees F outside.  may have to turn on the heat....