-
Sunday, 3rd of Mai, 0.40 hour (MEZ)
Seti is down, see Cricket Graph (http://fragment1.berkeley.edu/newcricket/grapher.cgi?target=%2Frouter-interfaces%2Finr-250%2Fgigabitethernet2_3;view=Octets;ranges=d)
heinz
-
Looks like a full crash again as they took the database down.
Uploads are fine and downloads would be too if the scheduler wasn't offline.
-
Looks like a full crash again as they took the database down.
Uploads are fine and downloads would be too if the scheduler wasn't offline.
DB is up. I got one task, downloaded fine. Uploads are still giving http/connect errors
-
Cannot, @ the moment, even reach SETI@home ???
Anyone noticing this too?
[EDIT] CAN reach the website now :)
C:\Documents and Settings\Fred Verster>tracert www.berkeley.edu
Busy tracing www.w3.berkeley.edu [169.229.131.81]
via maximaal 30 hops:
1 1 ms 1 ms 1 ms 192.168.2.1
2 15 ms 14 ms 14 ms 195.190.249.32
3 17 ms 16 ms 84 ms 213.75.64.117
4 17 ms 17 ms 17 ms 213.75.64.166
5 * * * Time-out bij opdracht.
6 27 ms 27 ms 27 ms nntr-s1-rou-1022.FR.eurorings.net [134.222.231.182]
7 109 ms 106 ms 169 ms nyk-s2-rou-1021.US.eurorings.net [134.222.226.161]
8 125 ms 125 ms 125 ms nyk-s2-rou-1001.US.eurorings.net [134.222.226.37]
9 126 ms 125 ms 126 ms chg-s1-rou-1001.US.eurorings.net [134.222.226.149]
10 126 ms 126 ms 126 ms eq-exchange.tr01-chcgil01.transitrail.net [206.223.119.116]
11 179 ms 179 ms 179 ms te4-1.tr01-sttlwa01.transitrail.net [137.164.129.2]
12 197 ms 198 ms 197 ms te4-1--260.tr01-plalca01.transitrail.net [137.164.129.34]
13 198 ms 197 ms 198 ms calren2-cust.plalca01.transitrail.net [137.164.131.238]
14 200 ms 200 ms 198 ms dc-svl-core1--svl-peer1-10ge.cenic.net [137.164.46.204]
15 200 ms 200 ms 199 ms dc-oak-core1--svl-core1-ge-1.cenic.net [137.164.46.213]
16 199 ms 199 ms 199 ms dc-oak-agg2--oak-core1-10ge.cenic.net [137.164.47.116]
17 200 ms 200 ms 199 ms ucb--oak-dc2-ge.cenic.net [137.164.23.30]
18 200 ms 200 ms 200 ms t2-3.inr-202-reccev.Berkeley.EDU [128.32.0.39]
19 203 ms 207 ms 213 ms t1-1.inr-211-srb.Berkeley.EDU [128.32.255.43]
20 200 ms 200 ms 200 ms webfarm.Berkeley.EDU [169.229.131.81]
De trace is completed.
Main DATABASE is down?!
-
Yep :(
-
Hoard of work trying to upload for me, at least my cache is still good.
-
Hoard of work trying to upload for me, at least my cache is still good.
They will recover, and you are lucky, i already got a i7 + gtx275 our of a job ;)
-
No more work. Everything I had something new no more work.
I got about 8 work units now.
-
Hi guys and dolls , trying to reach S@H, noticed almost everything is DISABLED or NOT RUNNING?! :'(
What is the matter?
-
"Unable to connect to database - please try again later Error: 2013Lost connection to MySQL server at 'reading initial communication packet', system error: 111"
on front page. It's something new. It seems their DB died completely :-\
-
"Unable to connect to database - please try again later Error: 2013Lost connection to MySQL server at 'reading initial communication packet', system error: 111"
on front page. It's something new. It seems their DB died completely :-\
They tried to got AP WU's out for the weekend, that's always fun.
-
The site was running last time I checked, but this speaks volumes.
(http://fragment1.berkeley.edu/newcricket/mini-graph.cgi?type=png;target=%2Frouter-interfaces%2Finr-250%2Fgigabitethernet2_3;inst=0;dslist=ifInOctets%2CifOutOctets;range=151200;rand=317)
-
yep. I got 10 new tasks but have only been able to download one.
-
Posting on the seti board: Shutdown Boinc and restart.
Downloads complete fine now.
-
Posting on the seti board: Shutdown Boinc and restart.
Downloads complete fine now.
Just waiting does the same. You just need some time to get through. I get just enough downloaded to get things going.
Heard about some DNS problems, but one computer even got 1000 WU and another on the same net, gets barely enough. It's just a bit busy.
First getting them out and next getting the uploaded again. There are a lot of short ones, 1 - 2 minutes so that's a lot of data.
And there are not a lot of AP 5.05 around.
-
Anyone see what was happening with the S@h forums while I was at school? Web server explode or something? [All other server contact seems fine here.]
-
Every Tuesday morning (Pacific time) we normally have a 3-4 hour outage for database maintenance. The upload/download servers and some web pages are offline during this time. This week took a particularly long time, and we'll be starting up the data servers shortly. Still, expect delays connecting to these servers for a while, and we'll be keeping many database intensive websites offline (like the forums) until some logjams have cleared.
-
Cheers, I'll read the front page news next time 8)
-
Cheers, I'll read the front page news next time 8)
Normally they don't tell anything, this will be a long wait.
-
not sure if they were doing some DB tweaking or what exactly. the portion on the Main page reads
This week took a particularly long time, and we'll be starting up the data servers shortly. Still, expect delays connecting to these servers for a while, and we'll be keeping many database intensive websites offline (like the forums) until some logjams have cleared.
So as I am getting no work (and the splitters for both MB and AP are running). the Boinc Replica is Offline and the query rate on the master DB is fairly high. I would say they are doing some DB work. Couple this with Matt's Tech News about cranking up NITPCKR.
Sepculation, AS Eric had told me the first attack for NTPCKER would be live returning Results and then starting to read backwards through the DB. At First I would suspect that the connection to either the Boinc Master/Replica would be very expensive. The other part that I have been subtly watching for is introduction of the QPIX number in the WU.
-
So as I am getting no work (and the splitters for both MB and AP are running). the Boinc Replica is Offline and the query rate on the master DB is fairly high. I would say they are doing some DB work. Couple this with Matt's Tech News about cranking up NITPCKR.
The query rate is old (as of 23h).
Another thing that strikes me as odd is the splitter status. Yesterday they added some new tapes and the "channels to do" for multibeam were over 300 channels if I remember correctly. Now they are 139. Some multibeam (along the astropulse) shortage on the way?
-
So as I am getting no work (and the splitters for both MB and AP are running). the Boinc Replica is Offline and the query rate on the master DB is fairly high. I would say they are doing some DB work. Couple this with Matt's Tech News about cranking up NITPCKR.
The query rate is old (as of 23h).
Another thing that strikes me as odd is the splitter status. Yesterday they added some new tapes and the "channels to do" for multibeam were over 300 channels if I remember correctly. Now they are 139. Some multibeam (along the astropulse) shortage on the way?
There are some WU coming in, I see them on 3 computers. So maybe there is some work out there.
-
I got about 50 'shorties' in three hours earlier today.
I think it's just going out as fast as they split it - remember we're still in an outage recovery phase.
-
Server Status shows the replica running and catching up. Cricket shows a steady state of Uploads and small increase in downloads.. So Richards statement is proving...
I have one machine that has managed to keep working and the others are starved and doing backup projects
Edit: If you believe the splitter status there should be AP being split
-
Currently, server status shows wu's being split, but 'results in field' not rising.
Huh?
-
Also AP wu creation rate practically nil although splitter working and file progression moving.
-
this weekend seti is about to run out of multibeam and astropulse completely according to matt. better up those queues to 10 days fellas!
-
Today is the 2nd time in as many weeks that my main 'crunching' machine (Q6600 running the optimized clients) totally ran out of work and is just idling there. I had already set the caches to 5.0/5.0 days, and before the machine ran dry, I upped that 5.0/10.0 days, but no luck in getting new work. Also, manually 'updating' and requesting new work gets me nothing.
I suppose I should look around for a secondary project to crunch when Seti is out of work again. *sigh*
Do the people here have a worthy and/or related project, which also does not clog up the machine for too long a time?
Reason: If/when Seti has work available again, I want to get back to crunching that again, without having to clear several days worth of cache of the 'other' project first.
Can I configure things like that, and if so, how?
Regards,
Patrick.
-
SETI/SETI beta/Einstein/MW here.
Also Aqua and Yoyo for idle CUDA GPU sometimes.
-
I added Einstein back onto my Quad last night, I used to crunch for it a couple of years ago.
Quad: SETI, Einstein, Milkyway(ATI)
Dell: SETI, Milkyway, Collatz(ATI)
iMac: SETI, Milkyway
The iMac just got 40 units from SETI today though.
-
SETI/SETI beta/Einstein/MW here.
Also Aqua and Yoyo for idle CUDA GPU sometimes.
Thanks both, I added Einstein@Home yesterday-evening at a workload of '10' (Seti has 100). I immediately got some work from Einstein@Home, so the machine is now happily crunching away again...
Let's hope the Seti@Home folks get some work ready to shove out of the door soon...
Regards,
Patrick.
-
It's all my fault ;D
I was sitting looking at a screen saying 9,999,821 credits, and I pressed 'update' just one more time.......
Now the whole database has fallen over.
Get a life, Richard, get a life!
-
LoL, I was asleep ... maybe I was snoring a bit loud :o
-
Hi, maybe Mork is missin Mindy . . . ::)
-
that is why i cache 10 day work ahead ;D
-
that is why i cache 10 day work ahead ;D
It's ironic.. a couple weeks ago when it was mentioned that Arecibo would go silent until November, there was plenty of work available and the bandwidth was around 60mbit, so I changed from the 4-day cache I have always had over to a 10-day, just to be safe.
Then Matt made significant progress with the software blanker, and I reverted. Finally caught up and started requesting work again yesterday.. and now broken.
For the record Richard.. I am holding you responsible. :p
-
Splitter status on Server status page shows all tapes err out, strange as no splitters are working. Wondering how major this crash is.
Anyway, good luck to the crew to get things working again.
-
Still no sign of activity, yet. Lucky got still some work from yesterdays flow of WU's
Hope the 'crew', gets it runing again. No trouble uploading, though.
-
Project announcement:
"Our database machine crashed. I can't get in to campus. Matt's nowhere to be found. Jeff's in Nepal. Dan's in South Africa. Please spread the word that we know that there's a problem."
(posted by Nicolas on the BOINC message boards)
-
What did you do this time Richard?
-
What did you do this time Richard?
Moi? Me, I know nuffink' :P. Never touched it, squire.
I'm busy trashing CPDN Beta tasks - every time I download one, BOINC creates 56 more.
Come to think of it, if that's a new server code bug, it would cripple SETI's database pretty effectively. And they updated the server code with the new task list format just before it started crashing ......
-
trying to deflect blame onto DA again?
-
If the cap fits. You got a better idea?
-
Well you two could be in cahoots, in some conspiracy ... plausible? I think so! ;D
-
You mean my fiendish plot to reach 10,000,000 credits and then lock everyone else out? :-X. Rats, found out again.
-
Come to think of it, if that's a new server code bug, it would cripple SETI's database pretty effectively. And they updated the server code with the new task list format just before it started crashing ......
Can't be the server code, otherwise Collatz would be crashing as well.
-
I don't know, for what they made the new column (stock or optimized app).. ::)
I don't need it.. :P
..so they could disable it.. ;) ;D
-
And here I thought I was responsible for upping the OC on the Frozen Nehi shortly before the crash.....
-
*twiddles thumbs*
-
*twiddles thumbs*
Do you want to buy a package filled with "patience" (for private usage only) ? (29.99) :P
-
Do you want to buy a package filled with "patience" (for private usage only) ? (29.99) :P
That's a pretty good price for cigarettes & beer. Got a pretty good supply at the moment, when I run out I'll checkout your eBay storefront ;D ... for now, off to do something constructive, gardening, & clean the car perhaps :-\ :o
-
Lol! :D That depends on the currency someone interprets into that numbers. :)
-
I love the server status page at the moment, it only lists 4 splitters, all on Bambi.
Nothing else.
-
.... actually, the series of tapes that were being split (before everything collapsed) was interesting. Some older stuff, including one from 2008 ... perhaps Radar blanking is looking good, ergo, perhaps no dry spell in November (albeit, these weekend outages are clubbing my RAC and an unrelated hiccup aborted 400 WU's that I never got to replenish before everything tanked, so it's been a dry weekend for me).
-
... perhaps Radar blanking is looking good, ergo, perhaps no dry spell in November
Let's hope radar blanking isn't the cause of the crash ;)
-
Uploads are going through again!!! :)
-
and you can report them as well. :)
Claggy
-
And I've got some new tasks (all resends, so far).
-
Just got one (1) with a _0 at the end, so it seems some splitters are working.
[edit]
Seems something is still broken. I get the following:
5-10-2009 21:28:42 SETI@home Computation for task 11au09ac.21878.3344.14.10.200_0 finished
5-10-2009 21:28:42 SETI@home Starting 11au09ac.21878.3344.14.10.223_0
5-10-2009 21:28:42 SETI@home Starting task 11au09ac.21878.3344.14.10.223_0 using setiathome_enhanced version 608
5-10-2009 21:28:45 SETI@home Sending scheduler request: To fetch work.
5-10-2009 21:28:45 SETI@home Reporting 1 completed tasks, requesting new tasks for GPU
5-10-2009 21:28:55 SETI@home Scheduler request completed: got 0 new tasks
5-10-2009 21:28:55 SETI@home Message from server: Completed result 11au09ac.21878.3344.14.10.200_0 refused: result already reported as success
More people with the same message or any ideas??
-
>> 5-10-2009 21:28:55 SETI@home Message from server: Completed result 11au09ac.21878.3344.14.10.200_0 refused: result already reported as success
Bummer, you got to re-chew someone else's WUrst. Despite the 0 (or 1), it was still a re-issue. Not surprised there could be some interesting overlap cases (which is far better than gap cases) as things are brought back up.
Out of curiosity, what was the Wingman's ISSUE date for that WU? Can you give us the result link.
-
No, not a problem at all - just typical congestion.
All it means is that you already tried to contact the scheduler to report that result. The scheduler got the message and acted on it, then sent a message back to acknowledge the report.
You never got that acknowledgement (fell down some wormhole in the internet). So BOINC didn't know the report had got through, and reported it again, just to be on the safe side: that's when you get the rather tetchy "I know, I know - been there and done that. Don't bother me with it again." message.
It's the same process as creates ghost results - you ask, it allocates, but the reply message disappears down that wormhole.
-
My results are being acknowledged ok.
-
My results are being acknowledged ok.
Sorry, wasn't meaning you - I'm sure most of them are.
It was a comment related to Hans Kramer's error message, and 52 aces' - wrong - explanation of it. I han't noticed we'd reached a page break, or I would have quoted one or both.
-
My results are being acknowledged ok.
Sorry, wasn't meaning you - I'm sure most of them are.
It was a comment related to Hans Kramer's error message, and 52 aces' - wrong - explanation of it. I han't noticed we'd reached a page break, or I would have quoted one or both.
I wasn't referring to you either :D
My post was an answer to those two too. Your post was more thorough though :)
-
Serve ok but I cannot see my tasks
This feature is turned off temporarily
-
Serve ok but I cannot see my tasks
This feature is turned off temporarily
As those are very high cost queries to the Boinc Database they were moved to the Replica Database. So even when the Replica is running it may be a few seconds behind reality.
Anything with very high detail about a "User and his/her computers" is a high cost query.
-
Seti went dark a couple of hours ago........
No forums, no nothing. Total crash, cannot even reach the servers.
Either the servers went down hard, or there could be a total network failure up to the hill.
Cricket graphs are still accessible, so the campus is reachable.
Hopefully it is not too serious, and they can get things back up in a couple of hours when they boyz get into the lab this morning.
-
Even http://seti.berkeley.edu/ (which covers all the SETI activity at Berkeley, like Serendip, as well as sah@home) is unreachable, which is worrying - maybe the whole of SSL is down, as Mark suggests. Perhaps the brakes on that shuttle bus failed again, and it plowed into a fibre junction box? Hope not, though there shouldn't be too many people around at this time in the morning.
-
Hey, here is also (already) a Berkeley OFFLINE thread.. ;)
-
Hey, here is also (already) a Berkeley OFFLINE thread.. ;)
Do you mean this thread, which has been used as necessary since May 2009, or the competitor SETI is down... (http://lunatics.kwsn.net/discussion-forum/seti-is-down.0.html) which was only started today?
-
Do you mean this thread, which has been used as necessary since May 2009, or the competitor SETI is down... (http://lunatics.kwsn.net/discussion-forum/seti-is-down.0.html) which was only started today?
It was only a little hint because of the other new thread.. ;) ;D
-
Even http://seti.berkeley.edu/ (which covers all the SETI activity at Berkeley, like Serendip, as well as sah@home) is unreachable, which is worrying - maybe the whole of SSL is down, as Mark suggests. Perhaps the brakes on that shuttle bus failed again, and it plowed into a fibre junction box? Hope not, though there shouldn't be too many people around at this time in the morning.
Or maybe a janitor plugged his vac into the wrong socket and kicked a router offline....LOL.
-
Something took Seti completely offline again...........
As of a couple of hours ago they went completely dark. Like the last time, the Cricket graphs are still accessible, but not a thing from the hill.
-
And we are back online!!!!!!!
Meowwwwwwwwwww!
-
Another weekend, another shutdown... :-(
Kick them damn Seti servers, somebody!
Everything seemed to be going so well.
regards, Gizbar.
-
Looks like somebody applied some steel toes. It seems both AP and MB work arte at least temporarily available. ;)
Joe
-
11/13/2009 8:44:33 PM Project communication failed: attempting access to reference site
11/13/2009 8:44:33 PM SETI@home Scheduler request failed: Couldn't connect to server
11/13/2009 8:44:34 PM Internet access OK - project servers may be temporarily down.
Weekend?
SETI@home down!
Nothing new.. ::) :(
http://setiathome.berkeley.edu/sah_status.html
DB offline.
I missed something, we have now a mentioned maintenance?
-
Well, per earlier posts in the Tech forum, MORK has become a growing concern. So that's probably the culprit today. But I could be wrong.
But when Matt and/or Erik say something is worrying, to the point their public faith in Mork's stability, reliability, availability is in question, we should probably take it at face value and anticipate unannounced outages, of potentially lengthy durations..., and with a possibility one day of an actual data loss (the one line we've been fortunate to not cross in a long long long time).
-
And we're back!! Yes, one of the messages I got when it first went down was something about mysql on Mork so it must have been either it went out or they tried something to fix it.
-
Ayone knows what maintenance SETI performs now, during weekend ?
-
Ayone knows what maintenance SETI performs now, during weekend ?
Well, this was just posted today: http://seti.berkeley.edu/?q=node/35
Maybe they're grabbing a spare snapshot of the data for scientific processing. With the new Nasa grant and the bad Nez PR, there could be pressure to show some legitimate value (all just speculation).
-
I don't think this outage was planned. We broke something.....
From front page...
Temporary Outage
Our servers are down temporarily while we make some database reconfigurations.
-
http://setiathome.berkeley.edu
Weekend maintenance? ..and so long?
Hmm.. strange.. :-\
The Berkeley crew search now all WUs of NEZ, for to resend them? ;)
-
If this outage now will help stabilize the system over the Christmas and New Year's holidays, it's fine by me. Several of the recent updates have described recurring issues that needed to be addressed. I know everyone would prefer to be out in front of a major problem rather than scrambling to fix it after a failure. The big breakdowns tend to occur over holidays. It would be nice to get to January without one.
-
24 hours are over.. ;)
But no servers online.
Maybe they will be offline for one week? Like Matt thought loud about in the forum?
Because in one week they will have again the same power outage.
-
That's not true, Boinc Site and Forum came up 15 minutes ago,
Give them time.
Claggy
I don't care about the BOINC site. ;)
The project SETI@home (http://setiathome.berkeley.edu) is still down.
;D
BTW.
AFAIK, in one week we'll have the same procedure!
-
That's not true, Boinc Site and Forum came up 15 minutes ago,
Give them time.
Blaggy
I don't care about the BOINC site. ;)
The project SETI@home (http://setiathome.berkeley.edu) is still down.
;D
BTW.
AFAIK, in one week we'll have the same procedure!
Seti's site now up, only 45mins after getting Boinc's site up
Claggy
-
That's not tvwe, Boinc Site and Forum came up 15 minutes ago,
Give them time.
Claggy
I don't care about the BOINC site. ;)
The project SETI@home (http://setiathome.berkeley.edu) is still down.
;D
BTW.
AFAIK, in one week we'll have the same procedure!
They have TWO Major Routers to get up and Running FIRST, it does no good if the network is not stable!
Then you have Servers that have been running WARM for Ages (now cold) need to start is a precise sequence. It not like turning your PC...
I turly hope you are not pushing to get work uploaded with your monster cache... The Little Guys should get it First!
Regards
Edit: Raistmer has asked you to do a couple of tests, you somehow feel that it is not important. While you may not feel that is important in your Credit Hunt, It is important that you do as he asëed. If you do then you can go chase Credit... Otherwise it is on you.
The only way you gain "advantage" is if YOU Help!
Posting it Seti Main is not helping. It is distracting. It does not solve the problem.
A Lot of people have helped You, it is about time you returned what you received.
-
Happy New Year!!!
And Seti is ringing in 2010 with a database crash....hope that isn't an omen of what the rest of the year is gonna be lilke...LOL.
Hope somebody is not too hung over to go in and kick things about tomorrow.
I wish the best in 2010 to the project and all who contribute to it.
Meow meow meow.
-
[size=20]Happy Newyear, 2010 . . . [/size]
Unable to connect to database - please try again later Error: 2013Lost connection to MySQL server at 'reading initial communication packet', system error: 113
Server status
Program Host Status
BOINC replica database jocelyn Running
BOINC master database mork Disabled
primary science database thumper Running
secondary science database bambi Disabled
data-driven web pages thinman Running
scheduling server anakin Disabled
scheduler process anakin Running
download server 1 bane Running
download server 2 vader Running
upload server bruno Running
The rest of the servers and programs are disabled or Not Running.
It is a 'sun' day, a day of, usually and tomorrow, weekend, maybe monday, unless someone dribes up to
the 'Lab'.
-
They have TWO Major Routers to get up and Running FIRST, it does no good if the network is not stable!
Then you have Servers that have been running WARM for Ages (now cold) need to start is a precise sequence. It not like turning your PC...
I turly hope you are not pushing to get work uploaded with your monster cache... The Little Guys should get it First!
Regards
Edit: Raistmer has asked you to do a couple of tests, you somehow feel that it is not important. While you may not feel that is important in your Credit Hunt, It is important that you do as he asked. If you do then you can go chase Credit... Otherwise it is on you.
The only way you gain "advantage" is if YOU Help!
Posting it Seti Main is not helping. It is distracting. It does not solve the problem.
A Lot of people have helped You, it is about time you returned what you received.
I don't know why you think to need to make me bad in public. >:(
You had a bad day?
To the first part of your message I didn't answerd, because I thouhgt - I don't understand what you mean.
But then after your edit, which I saw now - I thought I need to write something. ::)
Because of my two PCs have a RAC of ~ 80,000 , I should disable the network because of others should DL their WU cache first?
I don't understand this.
BOINC do his job automatically - I thought.
So people with high performance equipment should stop because other have right of way?
If so, then I need to shut down my equipment, my GPU cruncher is the current top_host_#2 at SETI@home.
So now because of your edit.
Ahh.. you think I don't support SETI@home, not the forum - help new members, advertisement for opt. apps, I never tested new software on my PCs, and so on..
I don't know, why you think this. :-\
I have ~ 2,900 posts in the forum of SETI@home.
Because of stupid unneeded messages, you think?
Credits? Yes, I lost ~ 400,000 Cr. because of unplanned server outages at SETI@home.
And, what I could do?
I lost RAC, because I tested 190.x , 191.x and 195.x nVIDIA_driver on my system for to publish my experiences in the SETI@home forum.
I done this only for me?
I made lot of tests.
For example - I said for around one year, it's not well to fix all GPUs only to one CPU-Core (_#0).
And this and this.. and so on and so on..
I guess you didn't followed all my messages in the forums.
This are only a few examples what I done.
And I should now ask for a medal for this?
SETI@home is a hobby for me. I think much more than only a hobby, with a RAC of ~ 80,000 and ~ 2,900 messages in the SETI@home forum.
You don't think?
I tested CUDA_V12b_x4 on my system.
I posted my experiences.
To now I didn't saw Raistmers question, that I should need to test CUDA_V12b_x4 without eFMer Priority.
I saw ~ 10 % slowdown compared to CUDA_V12, so what different result I should see without eFMer Priority?
Sure, I could make the test without eFMer Priority. No prob for me.
I asked Raistmer which forum he prefer. No answer. So I guess - both are well.
I don't know, the new year start very well after look here in this forum. :-X
I hope others had/have a better start than I in the new year.
BTW. For the future, maybe if you make an edit like this, it would be better to post a new message, that I have the possibility to see it.
It was now only 'good luck', maybe better 'bad luck'.
-
http://setiathome.berkeley.edu
From the current news:
Projects are down due to a database machine crash.
The machine that serves the BOINC database crashed last night. Until this is fixed we cannot send out work. 1 Jan 2010 16:27:45 UTC
http://setiathome.berkeley.edu/sah_status.html
-
Well........at least that means that somebody is on top of things, and if it can be fixed soon, it will.
And they were thoughtful enough to update that main page to let us know.
And the main page is accessible now.........LOL.
-
Seti Main and Beta are up, cricket graph is rising.
Claggy
Edit: Everything went very sluggish for a few minutes, now it's back again.
-
Well, it seems they are down again.
I cant acces their frontpage, and Boinc reports no contact to their servers.
One of my computers managed to upload its work, before the crash though.
-
OK, now they seem to be running again.
-
Now SETI@home is down again for ~ 24 hours.
CU.. ;)
-
Really odd that they couldn't figure out a redirect to a static page somewhere explaining they were down. Tons of people never look at their boards and probably assume they went away...
-
Berkeley is up again but the database is shut down for maintenance.
-
http://setiathome.berkeley.edu
Scheduled Power Outage - Update
We mostly recovered from our scheduled power outage this weekend, but due to minor problems we decided it would be best to keep the projects offline another day - however we're folding in the usual Tuesday outage today. So if all goes well we'll be fully up again by tomorrow (Tuesday) morning (if not sooner). 4 Jan 2010 22:19:26 UTC
-
Looks like they are down till Tuesday.
-
Looks like they are down till Tuesday.
I was 24 sec. faster.. ;)
-
Some of it is up, ie. web-pages and some database functions. However, if you look at the staus page, there are still big blocks of red.
Bit of patience, it'll be back soon.
regards, Gizbar.
-
Seti at a full dead stop as of a couple of hours ago....
Can't reach the servers at all.
Wonder if they lost power due to the quakes going on?
-
The BOINC site is still active, and from the evidence of the re-wiring weekends, they share power. But it might be another router/switch, judging by how abruptly it went down.
-
Well....they came back up for a couple hours. Just now shut back down for the weekly outage. Might make recovery after the outage just a little more congested.
Maybe we will get a note about what happened later when they come back up.
-
The Odd part was I could ping setiboinc.ssl.berkeley.edut and setiboincdata.berkeley.edu They were just not responding on port 80 it was almost as someone had disabled the deamons on the hosts.
-
Seti just went dark again........
Whazzup?
-
Cricket graph shows we had been downloading at max link speed for 10 hours solid - guess we blew a fuse in that dodgy database again....
Edit - maybe we blew the whole Berkeley campus - http://seti.berkeley.edu/ is down too.
http://downforeveryoneorjustme.com/%20http://seti.berkeley.edu/
-
They should go back to tubes........
They go soft, not down.....LOL.
Sigh.....back to bed then.
-
Looks like a hard crash at Seti, can't even bring up anything...
-
It was comprehensive and sudden, but BOINC (which lives in the same building) is still up, so it can't be a power outage - BOINC went dark too during the re-wiring weekends.
Best guess is a switch or router going down - that's happened before - in which case we can be hopeful that all the servers are still intact behind the blackout curtain.
-
Seti splitters started doing heavily some VHAR workunits, my PC started downloading a multitude of them, maybe a 1000 or more, for the past 12 hours. All this traffic from all seti participants, I guess must have something to do with the outage.
-
Doubt it. They're back, and straight up to full speed with no messing. Still think it was comms rather than db.
-
Looks like either they are working late or the web pages have a problem...Everything else seems to be working...
-
Replica is caught up, the only thing off is
data-driven web pages thinman Disabled
sah_assimilator1 vader Disabled
sah_assimilator2 vader Disabled
sah_assimilator3 vader Disabled
sah_assimilator4 vader Disabled
-
Looks like after limping for a day or two seti has crashed hard. The tuesday outage didn't seem to help...
-
Looks like it might be the router again as cricket still works.
http://boinc.berkeley.edu is down as well though.
Must just be the lab.
-
A little progress, finally got a "Project down for Maintenance" page.
From front page...
Projects are down due to a server closet air conditioning failure.
We have to power down most of our computers until this is fixed. 17 Feb 2010 2:36:55 UTC
--------------------------------------------------------------------------------
-
Thanks for the post hiamps.....
Just got home from work and saw that they are still down....wonder how soon they can get the AC repairman to get the chill on again.
One rig out of Cuda work....sigh.
-
Thanks for the post h)`mps.....
Just got home from work and saw that they are still down....wonder how soon they can get the AC repairman to get the chill on again.
One rig out of Cuda work....sigh.
Just got my 3rd Cuda card going and have a couple days worth.
-
Weird coincidence. I had my first ever overheat on my main cruncher yesterday (Tuesday). Took me a few minutes to figure out that it was an overheat and not my 21" CRT going out. Was weird, I could ping the rig, and access the shares and read/write data to the disks, but it wouldn't respond to a momentary press of the power button, nor the keyboard sequence for a reboot/shutdown. Ended up having to pull the plug.
Booted back up and both CPUs were over 60C, and that was after 3 minutes of nearly idle to boot back up. I have all the settings set for an automatic shutdown at 65C, apparently it didn't do that.
So for the time being, I had to scale back to 50% of CPUs and put a box fan in the window since i4 is 30F outside. Seems kind of...wrong to use a window A/C when it is 30F outside, you know?
-
Weird coincidence. I had my first ever overheat on my main cruncher yesterday (Tuesday). Took me a few minutes to figure out that it was an overheat and not my 21" CRT going out. Was weird, I could ping the rig, and access the shares and read/write data to the disks, but it wouldn't respond to a momentary press of the power button, nor the keyboard sequence for a reboot/shutdown. Ended up having to pull the plug.
Booted back up and both CPUs were over 60C, and that was after 3 minutes of nearly idle to boot back up. I have all the settings set for an automatic shutdown at 65A, apparently it didn't do that.
So for the time being, I had to scale back to 50% of CPUs and put a box fan in the window since it is 30F outside. Seems kind of...wrong to use a window A/C when it is 30F outside, you know?
May be a good time to clean dust bunnies...I blew my machine out yesterday with my air compressor. Running a lot cooler today.
-
I wouldn't mind if there was an update on seti, it's about 13 hrs since they announced the temp shutdown.
-
I wouldn't mind if there was an update on seti, it's about 13 hrs since they announced the temp shutdown.
On their front page it says their Air Conditioning is down, they won't know more until the A/C tech gets there....
-
Remember that this happened around 8 PM last night and it is just coming up on 8AM there currently.
-
Remember that this happened around 8 PM last night and it is just coming up on 8AM there currently.
Whatever happened to the Midnight AC dude? LOL.
Hopefully it's just another leak in the system that they can repair quickly. I am sure by now they have it being worked on.
-
My bet is tomorrow morning at the earliest. Even once they get it running it will need time to cool everything down enough to start up again.
-
We're back!!! Still not uploading but at least it's running on the message boards.
-
May be a good time to clean dust bunnies...I blew my machine out yesterday with my air compressor. Running a lot cooler today.
That is usually the first thing to look at, but I just did that about two weeks ago, and after 6 months of heavy use, there was almost no dust on the heatsinks at all. It was just the powdery kind of dust and not the hairy kind, even though I have two dogs and two cats. The filtered 5-1/4 bay covers take care of all the hair.
Problem is that I have been using a 28-inch box fan for about two years, and had it positioned to suck all the heat from the exhaust fans and blow it out into the hallway, which is where the return for the central heat/air is. Well that fan was about 12 years old and collectively over the years, I had put about 9 years of run-time on it, and the bushings started squeaking, so I took the motor apart and coated the surfaces with wheel bearing grease. It worked for about two days, started squeaking again, and then just locked up. So now I'm not pulling the heat out from behind the rig.
I was hoping having the window open with it being 30F outside would be enough, but apparently not.
-
Seti is up.......but still cannot upload or report a thing.
Cuda's running out of work....
Wish somebody could shed some light on this.
-
Wish somebody could shed some light on this.
Like updating status on the seti front page :)
-
While I am moving things to the wifes Work Apartment (in and out).
I have been emailing Debug Logs to the Seti Folks in an a attempt to sort what is happening.
What I surmise:
There is an issue with the Scheduler, not sure if it was an upgrade after the shutdown.
There is an issue with Bruno, uploads. My older Boinc Core cleared first. My newer Boinc Cores are dead in the water.
Somewhere in this libcurl rears its ugly head again.
Off to loading for the trip in the morning.
Regards
-
While I am moving things to the wifes Work Apartment (in and out).
I have been emailing Debug Logs to the Seti Folks in an a attempt to sort what is happening.
What I surmise:
There is an issue with the Scheduler, not sure if it was an upgrade after the shutdown.
There is an issue with Bruno, uploads. My older Boinc Core cleared first. My newer Boinc Cores are dead in the water.
Somewhere in this libcurl rears its ugly head again.
Off to loading for the trip in the morning.
Regards
Pappa,
So far as I can see, you are solo 'backchannel' on this one. Ozzfan is keeping the lid on the whiskey bottle, and that's about it.
There are some very bad things being said about Hurricane Electric - false, from my observations with Wireshark. They need stomping.
Could we get a project statement, please? Eric, or you quoting Eric at the very least. Some sort of preliminary diagnosis, and an indication of the sub-components under suspicion? There are some extraordinary things being claimed of pathping in Technical News, and the main thing they display is that the posters have NO IDEA of the difference between a home claas C and a full-blown datacentre LAN.
I repeat: this will start to get damaging unless the speculation and ignorance is replaced by project-derived facts asap.
-
Also off on some errands ('helping' a friend who lost a hard drive, give it a decent burial, come to terms with his loss, and change his habits ::) ). I do have underutilised cisco knowledge/tickets if you need any wireshark / other operational characteristics verified from the opposite side of the globe. On this side Packet loss varies from 0% per direct ping to Bruno, to 100%, which is just enough in combination to inhibit all uploads.
More speculation: The intermittent response reminds me of playing with frame relay congestion control (automatic discard, dependant on priorities and what bandwidth was paid for...) , which could indicate an 'exhausted' link somewhere in the paths. If the issue were to last, say, 'til the end of the month, then that's a possibility.
My p4 can attest that the problem is not merely with uploads though, since it did get some through last night, and was met with issues getting new work.
-
It is not the Network.
Every Lunatics User should know that. Anyone with Beta Test experience should also know that.
Things happen and then we dig to find out "why."
So if you see someone saying bad things about Hurricane, you should squelch that . When someone pays for a Gigabit Link, there is 24/7/365 support that insures the Network is Solid! That would leave UCB (the middleman up the hill which should have good support). If only Seti could afford to help users on that basis.
Realistically, the network is not the problem.
Regards
-
It is not the Network.
The fact that the forums are working, seemingly fine, should be enough to verify that (on some basic level anyway).
-
It is not the Network.
The fact that the forums are working, seemingly fine, should be enough to verify that (on some basic level anyway).
Forums go through Berkeley 128.xxx.xxx.xxx
Seti goes through Huricane 208.68.240.xxx
Seti Beta is the exception (down at the moment).
All get piped through Berkley.
Regards
-
Forums go through Berkeley 128.xxx.xxx.xxx
Seti goes through Huricane 208.68.240.xxx
...
OK, then then forum being operational puts nothing in the clear ;)... Only Richard receiving oddball RST packets puts network in the clear, and servers under suspicion... I'd prefer it wasn't their fault :'( . Going back to waiting for fermi release, since it is easier on the blood pressure :o
-
Maybe something is moving??
About half an hour ago my pc was able to report about 1600 completed workunits (scheduler request file of about 5.5 MB) and has better success with the rest of my uploads (more are getting through, since then it managed to upload more than 150).
-
Forums go through Berkeley 128.xxx.xxx.xxx
Seti goes through Huricane 208.68.240.xxx
...
OK, then then forum being operational puts nothing in the clear ;)... Only Richard receiving oddball RST packets puts network in the clear, and servers under suspicion... I'd prefer it wasn't their fault :'( . Going back to waiting for fermi release, since it is easier on the blood pressure :o
Jason so that you know and eveyone knows. Yes when the Forums are active, it means the Master Database is working and Berkeley has connectivity.
Other things I have ideas about but no real deifeintae topology map for.
-
Jason so that you know and eveyone knows. Yes when the Forums are active, it means the Master Database is working and Berkeley has connectivity.
Other things I have ideas about but no real deifeintae topology map for.
Gotya. I'm open minded at this stage. Unlike Sunu, no luck so far. (Signing off, I hope it's all fixed by the time I get back ;D )
-
Hi,
does I have a problem with francetelecom ? lost I 100%
can't send my work back since 4 days now...
any ideas ?
C:\Windows\system32>pathping 208.68.240.16 boinc2.ssl.berkeley.edu
Routenverfolgung zu boinc2.ssl.berkeley.edu [208.68.240.18]
über maximal 30 Abschnitte:
0 v8-sk01 [xxx.xxx.xxx.xxx]
1 xxx.xxx.xxx.xxx
2 AStrasbourg-xxx-x-xx-x.wxx-xx.abo.wanadoo.fr [xxx.xxx.xxx.xxx]
3 10.125.212.14
4 ge-1-0-0-0.ncstr102.Schiltigheim.francetelecom.net [193.253.91.90]
5 ge-0-0-0-0.ncstr101.Strasbourg.francetelecom.net [193.252.160.85]
6 xe-2-2-0-0.nistr101.Schiltigheim.francetelecom.net [193.252.163.182]
7 81.253.182.62
8 tengige0-4-4-0.ffttr1.FrankfurtAmMain.opentransit.net [193.251.132.125]
9 pos8-0-0-OC48.ar3.FRA3.gblx.net [64.208.110.225]
10 Hurrican-Electric-LLC.TenGigabitEthernet1-4.ar2.SJC2.gblx.net [64.214.174.246]
11 10gigabitethernet3-2.core1.pao1.he.net [72.52.92.69]
12 64.71.140.42
13 208.68.243.254
14 boinc2.ssl.berkeley.edu [208.68.240.18]
Berechnung der Statistiken dauert ca. 350 Sekunden...
Quelle zum Abs. Knoten/Verbindung
Abs. Zeit Verl./Ges.= % Verl./Ges.= % Adresse
0 v8-sk01 [xxx.xxx.xxx.xxx]
0/ 100 = 0% |
1 0ms 0/ 100 = 0% 0/ 100 = 0% xxx.xxx.xxx.xxx
0/ 100 = 0% |
2 60ms 0/ 100 = 0% 0/ 100 = 0% AStrasbourg-xxx-x-xx-x.wxx-xx.abo.wanadoo.fr [xxx.xxx.xxx.xxx]
0/ 100 = 0% |
3 --- 100/ 100 =100% 100/ 100 =100% 10.125.212.14
0/ 100 = 0% |
4 66ms 0/ 100 = 0% 0/ 100 = 0% ge-1-0-0-0.ncstr102.Schiltigheim.francetelecom.net [193.253.91.90]
0/ 100 = 0% |
5 --- 100/ 100 =100% 100/ 100 =100% ge-0-0-0-0.ncstr101.Strasbourg.francetelecom.net [193.252.160.85]
0/ 100 = 0% |
6 55ms 0/ 100 = 0% 0/ 100 = 0% xe-2-2-0-0.nistr101.Schiltigheim.francetelecom.net [193.252.163.182]
0/ 100 = 0% |
7 64ms 0/ 100 = 0% 0/ 100 = 0% 81.253.182.62
0/ 100 = 0% |
8 76ms 0/ 100 = 0% 0/ 100 = 0% tengige0-4-4-0.ffttr1.FrankfurtAmMain.opentransit.net [193.251.132.125]
0/ 100 = 0% |
9 187ms 2/ 100 = 2% 2/ 100 = 2% pos8-0-0-OC48.ar3.FRA3.gblx.net [64.208.110.225]
0/ 100 = 0% |
10 222ms 0/ 100 = 0% 0/ 100 = 0% Hurrican-Electric-LLC.TenGigabitEthernet1-4.ar2.SJC2.gblx.ne[64.214.174.246]
0/ 100 = 0% |
11 224ms 0/ 100 = 0% 0/ 100 = 0% 10gigabitethernet3-2.core1.pao1.he.net [72.52.92.69]
0/ 100 = 0% |
12 250ms 0/ 100 = 0% 0/ 100 = 0% 64.71.140.42
9/ 100 = 9% |
13 243ms 9/ 100 = 9% 0/ 100 = 0% 208.68.243.254
1/ 100 = 1% |
14 230ms 10/ 100 = 10% 0/ 100 = 0% boinc2.ssl.berkeley.edu [208.68.240.18]
Ablaufverfolgung beendet.
-
No, I don't think it is your ISP at all heinz. I believe it is project servers related, as we experience this around the globe.
Be a bit wary of how people are using this pathping command. pings can be ignored by routers ( I even do so in the sinbadsvn.dyndns.org domain ). So it's probably only useful to see who has responding to pings turned on and who doesn't. Ping'ing (not pathpinging) the server directly says it's 'alive' and 'attached' but that's about all i believe it can say.
-
Well....it's all dead in the water right now.
Can't reach the home page, server status page or the forums.
Dunno if somebody would possibly be working on the servers at this time on a Sunday night, or if things just folded up on their own accord.
EDIT....LOL, just as I posted this, they came back up.
EDIT II....and a few minutes later, locked up tight again.
-
And back up again.
-
And back up again.
But sadly, Eric's late night reboot did not have any effect on the comms difficulties. The kitties are still not able to phone home. Maybe today they will be able to sort it.
And just now, I am unable to reach the forums...
I am starting to wonder if there is something malicious going on.
-
Wow, talk about dedication, Eric is in and working on the problem already this morning. Expect SETI to be going up and down for awhile.
-
Wow, talk about dedication, Eric is in and working on the problem already this morning. Expect SETI to be going up and down for awhile.
It's Eric's baby....and it's very sick right now.
He must be tearing his hair out over this one.
-
Seems like things are starting to move again. Slooooooowly but getting better.
-
Looks like the web servers are turned off right now, uploads work fine though.
-
There's movement on gigabitethernet2_3 now, and just reported my completed Wu's,
but website's still down.
Claggy
-
After a reboot of my laptop, Seti's Website reports:
Tuesday, February 23, 2010 (4:30pm PST)
We're experiencing some problems with our web server, however the upload/download servers are up and running (and catching up from a long outage). We hope to have fully recovered sometime tomorrow (Wednesday).
Claggy
-
I know about rebooting, I just installed Win-7 Home to replace my RC install. At least my work was all still there after installing BOINC.
-
Seti Beta's Site and Forum is up, Seti Main still displaying a 'new' page.
Claggy
Edit: a couple of minutes later, and Seti Main's website came up.
-
Hi, can't reach SETI, :o , last down-- and uploads, were about 08:45.
Looks like it's DOWN :o
-
Looks like it's just the campus web-sites: http://downforeveryoneorjustme.com/%20http://seti.berkeley.edu/
Data (http://fragment1.berkeley.edu/newcricket/grapher.cgi?target=/router-interfaces/inr-250/gigabitethernet2_3&ranges=d%3Aw&view=Octets) is flowing more smoothly than it has for weeks ;D (Hooray for Hurricane Electric)
-
Hi Ricahrd, at least one positive thing, work-flow looks OK, for now. :)
-
Looks like it's just the campus web-sites: http://downforeveryoneorjustme.com/%20http://seti.berkeley.edu/
Data (http://fragment1.berkeley.edu/newcricket/grapher.cgi?target=/router-interfaces/inr-250/gigabitethernet2_3&ranges=d%3Aw&view=Octets) is flowing more smoothly than it has for weeks ;D (Hooray for Hurricane Electric)
Cricket graphs is one thing, updates and downloads is an entire different matter.
In my farm all but two hosts are only receving :
11/03/2010 11:44:54 Project communication failed: attempting access to reference site
11/03/2010 11:44:55 Internet access OK - project servers may be temporarily down.
This results in a 24 hr back-off, and no new tasks due to the inability to validate/update!
The questions is why only these two hosts? All are on same LAN.
What lies behind "Project communication failed: attempting access to reference site" ?
Current problem seems to affect at random on our side....
Morten
-
Well, hopefully if it's a campus thingy, they won't allow it to remain broken for very long.
And also hopefully, they have more data to load into the splitters. The last server status I was able to load this morning showed only 2 remaining datasets loaded, and only a couple of MB splitters still running, no ready to send cache left. Split and send mode.
-
They're baaaaaaaaaaaaaack....
Still in split and send mode, but all up.
-
Looks like something just crashed again as both the site is unreachable and cricket just dropped to nothing. I hope it is not the A/C again.
boinc.berkeley.edu is reachable though so it is not a campus wide problem.
(http://fragment1.berkeley.edu/newcricket/mini-graph.cgi?type=png;target=%2Frouter-interfaces%2Finr-250%2Fgigabitethernet2_3;inst=0;dslist=ifInOctets%2CifOutOctets;range=151200;rand=539)
-
:o And now it looks like SETI is down again .... :-\
Can't reach it since 2 minutes, now 03:11:30, hope nothing serious, well tomorrow, ehh now actually, hope somebody will come to the rescue ::)
-
http://boinc.berkeley.edu is reachable.
But http://seti.berkeley.edu isn't, so it looks like comms again - they can usually recover from that quite quickly, once Campus Networking get on the job.
-
Data communication has been restored to SETI, but the websites all seem still to be down.
-
Wish we could help them in their fight :-X
-
And again an unplanned server outage of SETI@home.. :(
1 1/2 hours 'submerged' traffic graph.
Cricket show current:
DL: 153.72 kbits/sec
UL: 591.02 kbits/sec
Much less than if we would have the weekly server maintenance.
-
So either the A/C went out again in the "closet" or something severed both connections.
-
And we're back.
-
May be back up but I'm still struggling to get work units - keeps backing off on both pc's
-
16 May 2010 3:57:37 UTC
Projects are down due to a database machine crash.
The machine that serves the BOINC database crashed this afternoon (Saturday). Until this is fixed we cannot send out work.
-
16 May 2010 3:57:37 UTC
Projects are down due to a database machine crash.
The machine that serves the BOINC database crashed this afternoon (Saturday). Until this is fixed we cannot send out work.
Meeeeeeouch. I hope it's something that can be patched up with a few spare parts and not a total meltdown.
Wishing Eric and crew the best of luck in getting it sorted.
-
At least Boinc can now seem to handle more work. Have had a 7 day setting for months and have had no communication problems with Boinc. Have enough work till Tuesday and beyond.
-
I always get amused with the weekend outages. How do these dang machines know that they are unattended.
BTW Seti@home is always looking for inexpensive(reads FREE) server hardware. So if you know someone that is getting rid of their newish server hardware give the kind folks at S@H a look :)
-
Just a reminder, SETI will be down again this evening for maintenance. Not sure how long they are planning to be down.
-
Just a reminder, SETI will be down again this evening for maintenance. Not sure how long they are planning to be down.
And remember that's (from the news item) Thursday evening California time - should be well after midnight UTC.
-
It has been posted on SETI main page that the times will be from 1800- 2200 California time. That works out to 0100-0500 UTC I believe.
-
And they actually got it back up by 2000 PDT. I have my fingers crossed that the mission is complete rather than needing to be rerun later.
Joe
-
They were talking about getting the new higher bandwidth line run up on the hill. I hope this is work on getting it installed and ready to go.
-
Seti down again, it's Friday so maybe they will get it fixed.
-
But if they get it fixed today what will we have to complain about over the weekend?
-
Looks like they lost the connections to the outside world once again.
Berkeley is reachable, but SETI is not.
-
Is this a test? Guess I'll get to see if my 3 day cache really is good for 3 days .
Ladies and gentlemen, please keep all arms and legs inside the car while ride is in motion! ;D
-
I have found with the newer Boinc versions a larger cache is OK now. I have a 7 day cache and never get the communication message I used to get. All is real smooth with larger cache.
-
In the US Monday is a free day.
So maybe we need to wait at least to after the Tuesday maintenance that everything is well again..
I have only a 3 day WU cache..
-
Yup.....ol' Cricket graph be flatter than a pancake. Nuthin' goin' in or out.......
Guess all the Seti staff, all of us, and some of the crunching rigs might as well enjoy the holiday (here in the US).
As usual, I hope it is something the boyz can make right easily on Tuesday when they are back in the lab.
-
What I'm seeing from long time ago is more or less that we'll have to say: "Seti is up again".
I hope that in the near future the problems could be mended and we'll work in a normal way.
-
Looks like very recently the cricket graph is no longer a pancake. Uploads are functional again, but it appears the website and scheduler are still down.
-
Website is up and running, uploads are going in but still no reporting or downloading of new work units.
-
Yes, the SETI@home forum is online, but it's again only slowly.
A well sign that there is something wrong.
Before the last unplanned outage, the forum was also slowly.
-
more signs of life from Seti Beta - was briefly able to download some new tasks :) Even an ap task for my new ATI card ;D!!
now back to same old "Couldn't Connect to the server" message
Looks like someone is in on their holiday trying to sort it for us though.
Got to give them credit for that, damn sure I wouldn't be doing work stuff on a holiday
-
Seems it's infective. Besides Seti with their database problem, Einstein had another problem with their filesystem server filling up, Collatz has/had internet provider problems, DNETC has server problems (scheduler) and Milkyway is just gone without a trace. Oh and apparently Spinhenge had an outage as well but has since recovered.
Stable client, unstable back-end. Task. ;)
Having said that, Seti's back. I just reported my work. Scheduler is running, web pages are running. Who knows for how long, though. ;)
-
Collatz is back as well, he lost power 3 times and the UPS' did not have a chance to recharge in between. Milkyway dropped off the air around 10 AM Eastern Daylight Time.
-
Totally gone again.
LOL, maybe my fault. I'd just done a checkout from their repository, then a second one failed. :P
Joe
-
Nah, 'twas Richard delivering a seminar on politics in Number Crunching forum (again)
-
Naw, it must be my fault, I was taking too many of their precious work units. :-\
-
Ears burning (again)? Those rednecks deserved it - and they got what they were asking for (fermis blocked from downloading uncrunchable work).
Joe, did you manage to get any of the current work-in-progress code before it went down? And did you get to see my theory that a host which has not reached quota gets its allowance reset at midnight, but a host which has reached quota stays blocked indefinitely - as if the over-quota rejection happens before the on-contact quota reset that David was talking about? That seems to be where part of the problem lies.
-
Janice (her handle was something else) reported in one of the news feeds that after receiving cuda23 6.09 work, she was now receiving cuda 6.08 work, which she hadn't seen before,
i did have a look at her cuda host, but because of the database display problem couldn't confirm that,
i'm wondering if the error people were getting is because they didn't have Cuda and Cuda23 in their app_info's (or even Cuda_fermi)
Claggy
-
I think we are suffering a "Murphy Attack" with 3 different issues
1) the Fermi Problem
2) Bugs in the new Server code or a crude stop gap effort to block the Fermi problem
3) A batch of bad work units that slipped through the checking process, possibly due to the noise detecting problem Matt refered to in his last post to the SAH fora.
After reading that V6.08/V6.09 users with non-Fermi cards were also getting -9 errors I checked back through my results. I found that about 1 in 4 CUDA tasks on all crunchers were finishing with this error. All the bad units were all VHAR's from the 24mr10ac.16936.20517.xxx, 24mar10ac.16936.19699.xxx and 24mr10ab.3036.25016.xxx series and .4 AR's from the 24mar10aa.3010.xxx series. I noticed units from these series were also erroring out when crunched on the CPU. Units from different series downloaded at the same time crunched OK on both CPU and GPU.
I also think Richard is correct about daily WU numbers not resetting as they should. Over the last few days the only machines of mine that have been able to get new work are the ones that crunch less than 100 units per day total. My "power crunchers" have gotten nothing for 3 or 4 days.
Brodo
-
...
Joe, did you manage to get any of the current work-in-progress code before it went down? And did you get to see my theory that a host which has not reached quota gets its allowance reset at midnight, but a host which has reached quota stays blocked indefinitely - as if the over-quota rejection happens before the on-contact quota reset that David was talking about? That seems to be where part of the problem lies.
Essentially I have all the available source code which might be pertinent. The checkout which failed was on a section of the S@H repository which is most probably unused for anything which would affect us, and hadn't been changed in more than 2 years.
However, BOINC supplies a generic set of sources for use by the projects, as well as the client side binaries we use. That's all under LGPL. S@H provides application binaries under GPL, so are obligated to have the sources available. They do not provide binaries of the BOINC server side code, and are free to modify the BOINC code in any way they want without providing sources. From observation in normal times, IMO they almost always use the generic BOINC code with whatever set of config.xml parameters seem appropriate. These aren't normal times and I think they're using patched server code which is someplace beween the "server stable" branch and the trunk of BOINC, and quite likely some code tweaks which are in neither. They certainly wouldn't want the broken credit granting at main, but much of the secondary related changes would help with the CUDA problems. It's a situation for Sherlock Holmes, not something which just requires reading the right document.
I did see your analysis on the quota situation. IIRC it makes sense. Whether I can spot anything in the code which would give that effect if only partiially active I don't know, D.A.'s brand of C++ is difficult for me to follow.
Joe
-
Hmm and my daily quota at SETI Bęta, was 1,264,937,360 WU's yesterday, but I didn't get them all :o
-
and its up.......
well nearly :D
-
and its up.......
well nearly :D
Nearly not.... Bruno is still down, so no upload. And no new work. There is only work for the "slower" computers with little uploads in waiting.
-
And if you can get downloads the 100 unit limit is still in force......
-
And if you can get downloads the 100 unit limit is still in force......
Better 100 than 0. :)
-
LoL .. 65 VLAR :D
-
I just downloaded 162 units and only 2 were VLAR, got lucky.
-
Progress! No complaints from scheduler about app version ;D
15/06/2010 4:21:28 AM SETI@home Requesting new tasks for GPU
15/06/2010 4:21:33 AM SETI@home Scheduler request completed: got 0 new tasks
15/06/2010 4:21:33 AM SETI@home Message from server: No work sent
15/06/2010 4:21:33 AM SETI@home Message from server: (reached daily quota of 100 tasks)
-
I'm still waiting for the 1000 WU to upload, slooooowly. Maybe before they shutdown again tomorrow. ;D
-
Progress! No complaints from scheduler about app version ;D
15/06/2010 4:21:28 AM SETI@home Requesting new tasks for GPU
15/06/2010 4:21:33 AM SETI@home Scheduler request completed: got 0 new tasks
15/06/2010 4:21:33 AM SETI@home Message from server: No work sent
15/06/2010 4:21:33 AM SETI@home Message from server: (reached daily quota of 100 tasks)
How do you do that :o LoL
If I use an app_info I get both the quota message and the wrong app version ???
Without the app_info all works perfectly
maybe its just me and my bad day at work is just continuing....... :P
-
How do you do that :o LoL
Dunno, just fixed itself I guess.
-
How do you do that :o LoL
Dunno, just fixed itself I guess.
Ohhhh - hope it hurrys up and gets round to my turn for some more tasks then so I can see if it still doesn't like my app_info
-
Damn
still getting that error:
14/06/2010 20:36:37 SETI@home Message from server: No work sent
14/06/2010 20:36:37 SETI@home Message from server: No work is available for Astropulse v5
14/06/2010 20:36:37 SETI@home Message from server: Your app_info.xml file doesn't have a usable version of SETI@home Enhanced.
14/06/2010 20:36:37 SETI@home Message from server: (reached daily quota of 100 tasks)
Oh well, got enough tasks to see me through a little while may just have to wait it out
-
I am getting the same messages. I am happy that I was able to upload my 29 remaining units, but it would have been nice if it let me download a few. It is nice to see that I am not the only one getting these messages. My last download was Friday, and I got 13. There were 140 the day before that. I'ts strange sitting at the helm of this rig with nothing for it to do. :)
Steve
-
It was me. I need to go out and get some lunch, so I pulled the plug to make sure nobody did anything stupid while I was away.
-
It was me. I need to go out and get some lunch, so I pulled the plug to make sure nobody did anything stupid while I was away.
Next time, take your the lunch from home.
-
Late night in Berkeley tonight. Already 6PM and still down. Hope they get everything running.
-
not looking healthy
Seti Beta has now changed to Set main again :'(
16/06/2010 18:48:58 SETI@home Beta Test [wfd] request: 0.00 sec CPU (0.00 sec, 0.00) NVIDIA GPU (0.00 sec, 0.00) ATI GPU (0.00 sec, 0.00)
16/06/2010 18:48:58 SETI@home Beta Test Sending scheduler request: To report completed tasks.
16/06/2010 18:48:58 SETI@home Beta Test Reporting 54 completed tasks, not requesting new tasks
16/06/2010 18:49:19 SETI@home Scheduler request completed
16/06/2010 18:49:19 SETI@home You used the wrong URL for this project
16/06/2010 18:49:19 SETI@home The correct URL is http://setiathome.berkeley.edu/
16/06/2010 18:49:19 SETI@home You seem to be attached to this project twice
16/06/2010 18:49:19 SETI@home We suggest that you detach projects named SETI@home,
16/06/2010 18:49:19 SETI@home then reattach to http://setiathome.berkeley.edu/
16/06/2010 18:49:19 SETI@home Already attached to a project named SETI@home (possibly with wrong URL)
16/06/2010 18:49:19 SETI@home Consider detaching this project, then trying again
16/06/2010 18:49:19 SETI@home Message from server: Invalid or missing account key. To fix, detach and reattach to this project.
-
Yes, I've got the same, :(
Claggy
Edit: Corrected it by editing my client_state.xml file so Beta's project name reads 'SETI@home Beta Test' instead of 'SETI@home':
<project>
<master_url>http://setiweb.ssl.berkeley.edu/beta/</master_url>
<project_name>SETI@home Beta Test</project_name>
make sure you edit the right one, and do it with Boinc shut down.
Edit 2: it's switched back once Boinc managed to reach the Beta Site, i'll have to edit it again.
-
thanks Claggy
I still haven't been able to contact Beta yet, bad timing I suspect as main also seems to be a bit hit and miss on when you can connect
I may just wait this out, get a beer and relax until Matt and Eric etc get this resolved
-
So far Seti has kept the upload servers working, hope they continue. Could make thing much less messy on friday if reporting is all that has to be worried about. Should make the limits not last so long...I hope.
-
You are right, uploads still going through. And here I was thinking, the server status was broken ;D - but bandwidth is a bit high for that especially since the boards are down. Well we shall see if/how long they keep them up.
-
I'm far away home with a real narrow band connection, but I've seen since yesterday that in my SETI crunching is not growing, so with my farm working at home, the server should be stopped as scheduled. :'(
BTW I'm working in the middle of the Mediterranean sea, and with only GPRS connection now. ;)
-
BTW I'm working in the middle of the Mediterranean sea, and with only GPRS connection now. ;)
Looking at your avatar picture I'd refrain from seiling in those waters then, LoL ;D ;D ;D
-
It certainly looks that way : Unable to connect to database - please try again later Error: 2003Can't connect to MySQL server on 'mork' (4).
On my wireless (WLAN) connection and
Unable to connect to database - please try again later Error: 2013Lost connection to MySQL server at 'reading initial communication packet', system error: 113 ,
and this on my cable connection. Hope they can do something about it, f.i. kick it ::)
Noticed UPload troubles since wednesday, 28 july.
-
I assume they just kept the upload/download on a bit longer this time round, and took them down on Wednesday.
To me completelt untutored eye this looks like a database crash (or the respective server hasn't rebooted or something)
An outage within an outage...
it was down already at 5 UTC, and i doubt we see any fixes before they come in this morning (ok that's like 15 UTC usually...) - and then they have to figure out the problem first...
Edit: I'm getting both error messages on the same connetction ;)
-
You can say, an outage whithin an outage, for sure, remember a post made by Matt, I think, Mork had some
trouble with 'this' before.
And quite sure that it is a Database Crash, after reading those messages. :o
There should be an exception to Moore's law, every 18 month's, hardware is prone to 'whatever failiar is possible' and
thus will happen.
(One of my MoBo's fails, it looks like the Voltage Regulator isn't working, -12V is missing) PSU is O.K.
-
If this page (http://twitter.com/setiathome) is official I wouldn't mind a short twitt of the problem :)
-
OK, they have to get to work first (always forget about the time difference :) ). Seems things are worked on now.
-
Whats up on seti-Server?
The normal outage isn't it ..
-
Afraid so. I think they had to hide the new outage schedule thread (was slowing everything down), but there is still a note on the homepage.
Anyway for the forseeable future SETI will be down Tuesday morning to Thursday afternoon (or Friday morning more likely) (PDT of course) to alllow them to use the machines for science - can't effectively access the database while it's being filled!
-
Well, there I was......
Trying to explain 'Springtime for Hitler' to a bunch of close-minded neophites.......
And they whacked me again.
Mel was just really having a go at the Hitler youth. It WAS funny......
He was not praising him for what he did.
So, after my $7000.00 fund raising drive last night, I am locked out again.
You would think I would learn something from this abuse................
Pfffffffffffft.
-
Mark.....................
You are the single most successful fund raiser for Seti that is NOT directly employed by the project. You should be proud of the last effort, it really was very successful.
But..........I know that there are a lot of members that hate to see you locked out all the time, including myself. You contribute so much to the project. You are certainly old enough to know when you are being antagonistic on the boards and old enough to know better. And that's all I'll say on the matter.
-
Hope it's not a long vacation Mark, we do miss you. Hurry back.
-
LoL, Sorry to laugh, but it all just reminded me of the fairly recent occasion I was modded & admonished by PM for using the word 'bastard' in a jocular context [Bastard : term of endearment in Australian slang ::) as in 'How's it goin' ya bastard?']. Didn't make much sense to me then, or now. ['Fair suck of the sav']
The wonderfully colourful local dialect of English/Australian ['Ocker Lingo'] I do acknowledge isn't appropriate for the boards, but that particular word is pretty much the self censored replacement for about two thirds of my vocabulary, so it was a bit like being told I wasn't wanted on the forums - which certainly brought me unhinged for a day or so, I got over it eventually. [ 'Was pretty ropeable, chucked a bit of a wobbly, went off like a frog in a sock']
I wouldn't place too much behind the apparent encroachment of uptight intolerance ['stinkin' yobboes, dry as a dead dingo's donger'] into an institution renowned for being so liberal ['Rip snorter of a joint']. It seems to be some character trait needed for a role that nobody wants for long, some kind of irrepressible overeagerness to enforce one's own interpretation of 'the rules' over and above consensus. ['Kangaroos in the top Paddock' if ya asked me.]
Well done with the server thingy ;D
['Ripper effort on the doovalaky moight']
Jason
[Edit:] Localised translations added [].
-
Thanks for the thoughts, Jason.
I'll try not to go off on a wobbly myself....LOL.
-
But..........I know that there are a lot of members that hate to see you locked out all the time, including myself. You contribute so much to the project. You are certainly old enough to know when you are being antagonistic on the boards and old enough to know better.
Hearty agreement to the first half, but for the latter part: Is it Mark or the folks that wish to impose/perpetrate the hear no evil, see no evil (and God forbid), speak no evil mantra?
Not like it’s a group of adults, from all walks of life, that are used to questioning/talking about something if it doesn’t seem exactly right. Most of our jobs depend on the ability to assess a situation (ie..ask questions) and then rapidly develop a plausible solution. Mark often has the stones (& RAC) to quickly call a Spade a Spade. He also appears to be on an extra short string for those mantra violations.
Then again, maybe I’m just an odd ball because I don’t have such politically correct thin skin…
-
Hope it's not a long vacation Mark, we do miss you. Hurry back.
Hopefully, the formula for resolving forum bans is more forgiving than that of DCF. :-\
-
Not to change the subject but, did someone forget to throw the switch over at SETI ? I see the web pages are still down today.
-
Not to change the subject but, did someone forget to throw the switch over at SETI ? I see the web pages are still down today.
Either forgot or some server shuffling has started and the BOINC database isn't in shape to be used. With most of the database status showing "as of 21h" the latter seems fairly likely. I suppose in another hour or three somebody should be in the lab and checking how things went overnight.
Joe
-
SETI is out 3 days a week Tue, Wed, Thur. Check back Fri.
Every Tuesday morning (Pacific time) we begin a 3 day data distribution outage to focus on science processing and development plus any needed systems maintenance. The upload/download servers will be offline during this time. The web site (including the forums) will only be offline during the database maintenance and backup portion of the outage. On Friday, you may experience connectivity issues as the servers catch up with demand. 6 Aug 2010 19:07:50 UTC
-
Forum's down again,
Claggy
Edit: 16 hours later, and they are back up, then down again within minutes, then up again after another few minutes.
Edt: 2: half an hour later and everthings down again,
-
And down - with a database error. ouch. looks familiar. wasn't that the reboot problem?
-
Thinman is showing as up but Mork, the master database server is down. Wonder if the boys went on a buying spree with all the money Mark raised and are swapping out a new server? I know it's wishful thinking but it sure would be nice! ;D
-
This entire outage has been up and down like a yo-yo.
-
Unable to connect to database - please try again later Error: 2013Lost connection to MySQL server at 'reading initial communication packet', system error: 111
Serious error or just some temporary thing while doing whatever they're doing?
-
BOINC replica database jocelyn Running
BOINC master database mork Disabled
SETI@home science database thumper Running
Astropulse science database bambi Running
data-driven web pages thinman Running
Looks like they have deliberately taken mork down for the time being.
-
Well, at least the message boards are up again now. 8)
-
Well, most of the servers are back up........I am still being dissed.........
Have requested remediation of my situation, but somehow doubt that it shall be forthcoming, regardless of the fact that I just donated $1000.00 for the new server.........
I can only hope that some sense prevails.
Otherwise I have a few decisions to make between now and September 6th.
I might shut everything down, but I am committed to my KWSN brothers and sisters who paid for the kitty auction rights, and to whom I am beholden.
NI.....NI.......NIIIIIIIIIIIIIIIIII.
And meow meow.
-
Gee, that was fun. I managed to get all my uploads in but couldn't report. Now SETI is down again. Hope my uploads got through ok.
-
Gee, that was fun. I managed to get all my uploads in but couldn't report. Now SETI is down again. Hope my uploads got through ok.
All my T8100's Astropulse Wu's went through, and reported O.K, and some of my E8500's reported O.K, other's unreported for now,
Claggy
-
Just checked again, everything has reported now. Guess we're running again.
Edit: I see now Thinman is down. Wonder what happened there?
-
Jeff posted that everything is coming off the BOINC master database, as the Replicia is offline, and he'd see how it copes, but within half an hour the forum had become unresponsive,
i expect thinman has been taken offline to lessen the database load until the initial upload and reporting has died out,
Claggy
Edit: i must have been wrong, forums are up again. :D
-
Thinman is back up now.
-
The boatds and everything else are up......
My donation and all elses seem to be down...........
S.OK........the fiirst 2 min of Mad Max says it all
You all have a nice day, now...............#$^^%^&^%#@.
-
OF course, his satire of Jews might have the same effedt.
Or various other Hitler jokes...........
Zig, !#@$$#%@.
-
...
S.OK........the fiirst 2 min of Mad Max says it all
...
Ever listen to the first 10 minutes of the Australian movie 'Gallipoli' with the picture turned off ?
-
And pardon my sorry ass,
But NIGGERs can sing.......so don't worry my bad ass about using the tern..........
I LOVE niiggers............
They got the grooooooove, man.
http://www.youtube.com/watch?v=A1FK620bS7A&feature=related
Enough of this shit about semantics, and what word works or does not.
Niggers be niggers.........it's all good. If a black man be offended by it, please talk to me about it.
Seriously. Most of the music that I love was made by darkies. And the white men who made their money out of playing it.
I do not disrespect them. My use of early 50's terms is not appreciated on the Seti boards......they can be such whacked assholes sometimes........
-
You want pure Nigger?
My favorite guitarist of all time........Stevie Ray Vaughan......played pure nigger.........
Don't know where he got it from down in Texas..........but......it was pure nigger shit.
So don't try to tell me I don't know where it came from or what to call it.
You watch this, and if you ain't got religion by the time the cut has ended, then you and I have nothing to talk about. I was at the man's last concert at Alpine Valley.........
http://www.youtube.com/watch?v=tWLw7nozO_U
Edit.......there are hundreds of cuts left of this master of the telecaster in the ether mist.
Even though he be my friend, may he be yours too. His younger brother Jimmie Lee tried to make the cut, but he just never got it the way Stevie did.
And if you Texas niggers don't get my drift.......another old ter.........I guess you never will. When Stevie gets to the bottom of it all.........oh yeah.
-
...
S.OK........the fiirst 2 min of Mad Max says it all
...
Ever listen to the first 10 minutes of the Australian movie 'Gallipoli' with the picture turned off ?
No.........please enlighten me.
-
Well if you ask me, 'political correctedness' outght to be banned anyway. (did you know you can't do 'little red riding hood' in nurseries any more? why? because th wolf eats them! nuts. just nuts.)
It's not the wording it's the attidude .
The problem on a public board is, that you can not easily see whether the attitude behind the wording is 'wrong'. So going for cautious and lean hard on the wording for fear of offence (or disgusting attitude) is not - well...
And as for banishment and funding drive - do you really want people to think you bought your way out?! We know it's not like that (timeline is wrong for startes) but sometimes it's all about appareances.
-
I know that, Hon.
But I kinda wish that $1000.00 woulda bought my bail, eh?
I seem a bit dissed here.
Eric appreciates me.........god knows the new server might.
Admin does not give a shit. HIs bre is butterd regarldess.
He does not know nor care where it comes from.
And the soul of the man the gave **** a good name lives on...........
I still miss him.......
http://www.youtube.com/watch?v=zAG-kX_IlUw
-
Bitch wans another take.............
If you need 15 takes.........you take what you need......babe.
I am done here........few appreciate my views on life anyway.
You want more..........guess you better bust the bricks and get me out of my slumber. In person.......
Just shout first so I don't shoot and ask questions later.............
-
There do exist a few bootlegs of the final, fateful Stevie Ray concert at which I was present.........
Have not found any live footage.........but here is a taste of what we all lost that day.
Had this running through my head as I made my way through the fog on highway 41 headed northbound homeward. It was the next day at work, little sleep, that I hit the floor when I heard the news........
http://www.youtube.com/watch?v=Edyr48B9XZ8
And if this is indeed a recording of his last jam............is it by far once of the best examples of his work. Must have been taken right offa the main board. Presence is astounding.
http://www.youtube.com/watch?v=_pcluU1UHH4&feature=related
Some claim that he saw it coming...........maybe an urban legend. But the Man was a legend in himself. You will have to decide if it was true........ I personally was lost for almost half a year upon his departure.
http://www.youtube.com/watch?v=pmld5LeqLVE&NR=1
-
A moment's notice..............
That's all we are..........dust in the wind.
And all your money won't another minut
The search is long.....my life is not.
I trust you will keep the search up when I am gone.
Please DO try to send me a message if you find them.
-
Yes, the kittyman dreams too...........
http://www.youtube.com/watch?v=I5T6gyCPDvM
And that's where he lives.........your loojing glass may find me there one day.
Don't look too long, I don't tarry.
So........are you going?
http://www.youtube.com/watch?v=_45W-Lq7ftw&NR=1
-
did you know you can't do 'little red riding hood' in nurseries any more?
I know how you feel Miep, I used to like the "Little Black Sambo" cartoons but for some reason they didn't quite make the politically correct standard! :D
Dream on Kittyman!
-
Oh crap.........
Now I'm in love with 2 at once..........
Kinda favor the long haired gurly. But my, oh my.
http://www.youtube.com/watch?v=ywMPtwM8avs&feature=related
-
OK........
Here we go again.
Best car chase ever........with Lalo's score to boot.
Fasten your seatbelts boyz, .....it's gonna be a bumpy night.
http://www.youtube.com/watch?v=lcW-T-thdoE&feature=fvw
-
Looks like somebody tripped over the cord on the way out again. ::)
-
Must have plugged it back in, I just got buried in tasks.
-
SETI message boards just came back up!
-
Seti just ditched again........
Forums and all.
Hopefully somebody is rebooting to try to fix the nasty download problems that have existed since coming back up after the outage.
-
Oh dear, Seti website and forum seems to be unreachable, at least schedule requests are still going through,
Claggy
-
Now everything is down including the main page. Can't report work either.
-
Now everything is down including the main page. Can't report work either.
Let's just hope it's a planned outage......
-
Now everything is down including the main page. Can't report work either.
Seti and Seti Beta's Websites are up now, but projects are down for maintenance,
Claggy
-
Now everything is down including the main page. Can't report work either.
Seti and Seti Beta's Websites are up now, but projects are down for maintenance,
Claggy
That's a good sign, I think.
Probably means that somebody is in the lab pushing the buttons.
-
Well, I can get to the main page and the server status page again. Doesn't tell much except they are down. On the server status page it shows everything either not running or disabled except the Astropulse science database on Bambi.
-
Main page now says the Aircon has failed, project will stay down until they've assessed the damage,
Claggy
-
The air conditioning in our server closet failed this morning, causing several of our important systems to overheat. The servers have since been shut down, but until we survey the damage and address these cooling issues all parts of the project will remain offline.
Anyone want to start a new fund raiser for air conditioning?
-
I wonder how long things will be down. I don't really have any pressing need to contact the servers, but I'm sure several hundred (or thousands) of people do.
-
Probably at least through tonight and well into tomorrow. That is if they can get someone out on a Sunday to work on it. Hmmm, "until we survey the damage and address these cooling issues" Sounds bad, could take even longer than tomorrow. Anybody remember how long they were down last time the AC went out? May be a one day fill-up again.
-
We're not the only ones with a problem with the heat.
Cheer leading team is banned for being to damned HOT. What's wrong with you ruskies?
http://englishrussia.com/index.php/2010/09/04/banned-beauties/ Hey Raistmer, got an extra cot?
-
I wonder how long things will be down. I don't really have any pressing need to contact the servers, but I'm sure several hundred (or thousands) of people do.
I was wondering too. I think that the campus air conditioning tech will probably reset the air conditioner soon, but bringing that many servers up and testing for damage is likely to take considerable time. I'd guess Monday sometime unless one or more of the staff can defer whatever else they had planned for the weekend.
I also wonder how many of the downloads started during the ~24 hours work was being delivered actually completed. BOINC getting a little bit of all downloads rather than trying to get at least a few completed was somewhat modified by the "project backoff" mechanism, but still I bet a lot of users have more partial transfers than completed ones.
Joe
-
Managed to get my tasks UPLoaded and got a lot of work D-Loaded, some of them took 2 hour to download.
Just got my first results from my GTX480, doing OK running 3 (0.04CPU+0.33GPU) SETI CUDA and 4 on CPU.
To get it running, I had to use 1x8 pin 17Amp, 1x6 pin 17Amp and a additional 3- 8 AMP from the Mobo.
And gets very hot, 88-92C is 'quite normal' , you can't even toch the copper pipes.
Hope they do get SETI, back on line.
-
We're not the only ones with a problem with the heat. Cheer leading team is banned for being to damned HOT. What's wrong with you ruskies?
http://englishrussia.com/index.php/2010/09/04/banned-beauties/ Hey Raistmer, got an extra cot?
Well the Russians isn't to blame...
Not surprisingly....
where Muslim countries will take part
-
Id keep the cheerleaders and send the Muslims home if they didn't like it. :P
-
Ouch.............
I hope there is no real damage to the servers.
That dang AC has been a problem child for some time now.
I think they need an alarm of some kind that would dial a handful of phone numbers to alert somebody when it goes on the fritz so they could shut things down before any real damage occurs.
-
We're not the only ones with a problem with the heat. Cheer leading team is banned for being to damned HOT. What's wrong with you ruskies?
http://englishrussia.com/index.php/2010/09/04/banned-beauties/ Hey Raistmer, got an extra cot?
ROFL ;D ;D ;D
Just FYI: Russia is not Muslim country :o Dominating religion here is Orthodoxy ;)
-
I also wonder how many of the downloads started during the ~24 hours work was being delivered actually completed. BOINC getting a little bit of all downloads rather than trying to get at least a few completed was somewhat modified by the "project backoff" mechanism, but still I bet a lot of users have more partial transfers than completed ones.
Joe
Yeah, I had >20 ATi MB tasks and 2 ATi AP tasks in download queue but nothing was downloaded for ~6-8 hours :( Now ATi GPU busy with MW instead of SETI...
-
Not only UPLoad's are stuck but DLoad's too and indeed a lot of them are 'partially' UP- or D'Loaded.
This could give a lot of "Ghosts", I take it.
Suspending network isn't an option, I think. This can go on till tomorrow, unless somebody is going to
'kick' the UPLoad & D'Load servers and start the AC first, an alarm should be in place, for this, cause
this can give real damage........... :o
7-9-2010 15:24:14 SETI@home Sending scheduler request: To fetch work.
7-9-2010 15:24:14 SETI@home Reporting 1 completed tasks, requesting new tasks
7-9-2010 15:24:15 SETI@home Started upload of 29mr10ac.23536.25021.11.10.172_1_0
7-9-2010 15:24:16 SETI@home Scheduler request completed: got 1 new tasks
7-9-2010 15:24:18 SETI@home Finished upload of 29mr10ac.23536.25021.11.10.172_1_0
7-9-2010 15:24:18 SETI@home Started download of 29mr10ac.4960.25021.13.10.96
7-9-2010 15:24:24 SETI@home Finished download of 29mr10ac.4960.25021.13.10.96
7-9-2010 15:40:32 Resuming computation
7-9-2010 17:55:30 SETI@home Started upload of 29mr10ac.1266.4980.12.10.54_1_0
7-9-2010 17:55:33 Project communication failed: attempting access to reference site
7-9-2010 17:55:33 SETI@home Temporarily failed upload of 29mr10ac.1266.4980.12.10.54_1_0: connect() failed
7-9-2010 17:55:33 SETI@home Backing off 1 min 0 sec on upload of 29mr10ac.1266.4980.12.10.54_1_0
7-9-2010 17:55:33 SETI@home Sending scheduler request: To fetch work.
7-9-2010 17:55:33 SETI@home Reporting 1 completed tasks, requesting new tasks
7-9-2010 17:55:34 Internet access OK - project servers may be temporarily down.
11-9-2010 14:42:41 SETI@home Started upload of 05ap10ab.9224.21917.7.10.170.vlar_0_0
11-9-2010 14:42:42 SETI@home Sending scheduler request: To fetch work.
11-9-2010 14:42:42 SETI@home Reporting 1 completed tasks, requesting new tasks
11-9-2010 14:42:44 SETI@home Scheduler request completed: got 1 new tasks
11-9-2010 14:42:46 SETI@home Started download of 05ap10ab.27177.5966.8.10.69
11-9-2010 14:43:02 SETI@home Finished upload of 05ap10ab.9224.21917.7.10.170.vlar_0_0
11-9-2010 14:43:17 SETI@home Finished download of 05ap10ab.27177.5966.8.10.69
12-9-2010 12:23:31 climateprediction.net Sending scheduler request: To send trickle-up message.
12-9-2010 12:23:33 climateprediction.net Scheduler request completed
12-9-2010 12:28:21 SETI@home Sending scheduler request: To fetch work.
12-9-2010 12:28:21 SETI@home Reporting 1 completed tasks, requesting new tasks
12-9-2010 12:28:42 Project communication failed: attempting access to reference site
12-9-2010 12:28:42 SETI@home Scheduler request failed: Couldn't connect to server
12-9-2010 12:28:43 Internet access OK - project servers may be temporarily down.
UP-- & D'Load have been intermittant, last 36- 48 hours.
-
Well the problem is, it's probably not an issue that nobody in the lab wants to go up there on a Sunday, but the AC is Berkeley's problem, and whoever needs to kick the AC either can't or won't go in on a Sunday, so they'll likely come in Monday morning, and after they get it working, then the lab can start bringing servers up..and then bring them down less than a day later for maintenance.
-
I also wonder how many of the downloads started during the ~24 hours work was being delivered actually completed. BOINC getting a little bit of all downloads rather than trying to get at least a few completed was somewhat modified by the "project backoff" mechanism, but still I bet a lot of users have more partial transfers than completed ones.
Joe
Yeah, I had >20 ATi MB tasks and 2 ATi AP tasks in download queue but nothing was downloaded for ~6-8 hours :( Now ATi GPU busy with MW instead of SETI...
I bet you not any more as the assimulator crashed at MW. Right now I am crunching DNETC work on my 5830.
-
Well, something's changed...........
Home page is gone again now.
Whether that's a good or bad sign I am not sure.
-
Homepage is served out of the same closet as the rest of the project... If the heat got to be too much, the team would have to shut it down also... I think I'd set up a laptop or something else that runs cool that can just serve simple text only 'Project down due to cooling issues' messages to homepage requests.
-
I suspect that it is going to be down for a while. There is also the issue of the 3 day out and if they will still have that and, if they do, what happens afterwards with volumes.
I myself was running 1 of my boxes dry to change GPUs to Fermi and got caught out. That's going to put a dint in the old run rate, oh well, s^^t happens.
VP
-
Straining my memory here but if I remember correctly the homepage is run off the Berkeley servers separate of the SETI servers. That's why most of the time we can see the homepage when all the SETI servers are down for maint.
-
Straining my memory here but if I remember correctly the homepage is run off the Berkeley servers separate of the SETI servers. That's why most of the time we can see the homepage when all the SETI servers are down for maint.
I believe this is the case. I know the homepage uses the campus ISP, but I can't remember if it is in a different closet/building or not. If everything was up, a few forum searches would find the answer.
-
I think I've figured it out! SABOTAGE!!! The campus IT guys just found out SETI is getting a shiny new server and got jealous! They broke the AC and then shut down the SETI Front page. :o ;D
-
Homepage are still online, right know!
Database-Server offline ...
-
I am hoping that, given that magnitude of the downtime and the fact that things never had time to recover after last week's outage, the boyz might get things sorted before coming back online and forgo the 3 day outage this week. Hopefully that would give things time to recover whilst they are in the lab and able to monitor the servers should problems arise.
This, of course, is assuming that no critical damage has been sustained to the servers from the AC failure and subsequent overheating. But there is also the possibility of hidden damage or partial damage that can rear it's head after such an event.
Wishing the crew the best of luck in sorting things this week.
Meow meow.
-
Talking about damage, I assume they have backups but could it ever be so bad that stuff like user stats and credits is lost?
-
Talking about damage, I assume they have backups but could it ever be so bad that stuff like user stats and credits is lost?
Depends where they keep the data. If it's all in the same location/building definitely.
-
Monday Update: The air conditioning in our server closet failed on Saturday, causing several of our important systems to overheat. Most servers and services were shut down and will remain down as we continue to wait for the air conditioning to be fixed, which will be Tuesday at the earliest.
They changed the message on the front page. It's gonna be close but I might just make it if they get it fixed early enough.
-
Homepage doesn't respond, and cricket is flatlined. 5:15pm PDT.. if they're not still in the lab, I would venture to guess at this point that we roll right into the scheduled downtime for the week. Just a theory.
-
Glad I saw and copied the new message before it went away again. Hard to tell what they will do about the regular outage after this one though.
-
With the 3-day outage, here's what I don't understand - working/developing on a production environment? That's bad practice. Isn't seti beta suppose to handle that; I guess its just for frontend(science app). There must be something wrong with their methodology.
-
With the 3-day outage, here's what I don't understand - working/developing on a production environment? That's bad practice. Isn't seti beta suppose to handle that; I guess its just for frontend(science app). There must be something wrong with their methodology.
Beta runs off the same servers as SETI, they have the downtime so they can analyze the data that has come in in the last 10 years. They tried it with all the servers up and it just crashed the systems from the strain.
-
If I recall, the main webserver is actually in the same closet as the other servers, but you're correct that it is fed from the campus ISP instead of the separate high bandwidth one for S@H
-
Monday Update: The air conditioning in our server closet failed on Saturday, causing several of our important systems to overheat. Most servers and services were shut down and will remain down as we continue to wait for the air conditioning to be fixed, which will be Tuesday at the earliest.
They changed the message on the front page. It's gonna be close but I might just make it if they get it fixed early enough.
Thanks for saving that notice while I was gone to work.
Bummer........wonder if they couldn't get anybody in to look at it or if they are waiting on parts......
Sigh.......maybe tomorrow.
-
Well, all my work completed overnight. Got 4 ready to report that uploaded just before we went down the rest waiting to upload. Should be interesting to see what they are going to do today. :-\
-
I ran out of ATI Astropulse tasks a couple of hours ago, now down to my last 6 Cuda tasks on Seti Main, still got 20 Normal AR Cuda tasks on Beta, and 20 Cuda VLARS, they'll keep me going for almost 2 days, :D
Claggy
Edit: The website is up again, and according to the Status page, the download servers are online, and cricket shows some uploads and downloads too, (but not for very long)
-
Sure glad I got my 10 day cache, should still have a couple days left...Hope they didn't lose much.
-
Sure glad I got my 10 day cache, should still have a couple days left...Hope they didn't lose much.
Still got some left here as well......
The home page is back up, with the usual Tuesday-Thursday outage message.
Wondering now if that message is automated, or if they really intend to be down until Friday.
Meow, Meow, Meeeeeouch.
-
I just refreshed the home page and they have now taken down the outage notice.....
Maybe some hope that they will not leave it down all week?
-
Still, just Bane, Bambi and Thumper up. They are probably just bringing them up to access the damage. With the upload server still down we aren't going to get any downloads. We'll just have to wait and pray all the servers check out. Even if the AC is fixed they will have to wait while the server closet cools back down to where they can get started back up.
-
The download server was up for a bit, and I got some tasks downloaded that were hung when the outage occurred.....whether they intend to leave it up or were just testing I dunno.
Cricket graph shows a spike, but it has since dropped back down and getting HTTP errors on current download attempts. Hope that doesn't indicate that the download server crashed on it's own.
-
Downloads are back up.......
Hit the button if you still got any that are hung from last week. I have some going through now.
-
It may be completing downloads that were started before but it doesn't seem to be giving any new work. I tried an update and got this reply...
9/14/2010 2:25:29 PM SETI@home update requested by user
9/14/2010 2:25:33 PM SETI@home Fetching scheduler list
9/14/2010 2:25:34 PM SETI@home Master file download succeeded
9/14/2010 2:25:39 PM SETI@home Sending scheduler request: Requested by user.
9/14/2010 2:25:39 PM SETI@home Reporting 4 completed tasks, not requesting new tasks
9/14/2010 2:25:41 PM Project communication failed: attempting access to reference site
9/14/2010 2:25:41 PM SETI@home Scheduler request failed: Couldn't connect to server
9/14/2010 2:25:42 PM Internet access OK - project servers may be temporarily down.
-
Downloads are back up.......
Hit the button if you still got any that are hung from last week. I have some going through now.
At last, at least thats one of my rigs with enough tasks to see it through another couple of days
-
Well, homepage up is a good sign that the lab hasn't melted...
-
Not looking good, from the fron page:
News
Air Conditioning Update (Tuesday): The air conditioning in our server closet failed on Saturday, causing several of our important systems to overheat. Most servers and services were shut down to prevent damage and remained down until the air conditioner was finally checked out Monday afternoon. We tested the system under full load this morning and it failed again, so full production is delayed at least another day. Sorry for the confusion/inconvenience.
14 Sep 2010 21:31:43 UTC · Comment
-
All my incomplete downloads are finished now. That is , have few days of CPU and GPU work now again
-
You guys are just trying to make me feel bad cause I got all my downloads finished before this happened and now I'm out of work! >:(
-
You guys are just trying to make me feel bad cause I got all my downloads finished before this happened and now I'm out of work! >:(
Of course, those who got their downloads were the ones making it impossible for the rest of us ;D
Joe
-
My iMac has finished all of its work, the Q8200 still has a long list of WU waiting to download but nothing to crunch and the X4 630 managed to get everything downloaded and still has SETI work.
The iMac is working on Milkyway, the Q8200 is working on Einstein and Milkyway and the X4 630 is working on SETI and Collatz.
Now I just need to get that Nvidia 460-768 ordered.
-
I've got plenty of work to last until probably Saturday on the main cruncher. The single-core machine still has about 12 days of work. I think the last three APs in that cache are going to miss the deadline.
-
Things are looking up. Message boards are back, most of the servers are working again. Only one download server is up and the upload server is still down though. Won't be as long as it has been, we might just get back up and running today after all.
-
Things are looking up. Message boards are back, most of the servers are working again. Only one download server is up and the upload server is still down though. Won't be as long as it has been, we might just get back up and running today after all.
Dunno about that.......here's the latest update on the home page.
Air Conditioning Update (Wednesday): The air conditioning in our server closet failed on Saturday, causing several of our important machines to overheat. Most servers and services were immediately shut down to prevent damage and mostly remained down as initial attempts to fix the air conditioning system failed and new parts are being ordered. We just started the project up to help clear some pipes but some services will remain off, and the rest may go off again at any time. Sorry for the confusion/inconvenience.
Just checked and all up and download servers are shown as offline and Cricket graphs went flat.
The kitties give another collective sigh.
-
Just managed to upload and report all my Seti tasks ;D ;D ;D ;D
Just Beta left to do, but I guess they'll leave it down till Seti is up and running safely
-
Uploads are running for me now but download servers appear to be off.
-
My 8-core Xeon server is off since seti is down. I have a lot results to upload. I will switch on when upload and download servers are running normal.
No panic, I have time.
-
Bet most of those that didn't will be increasing to 10 days. Still running.
-
Well, I wonder what the boys are up to? Almost noon there and the web pages are still down. Starting to get withdrawal symptoms! :'(
-
Well, I wonder what the boys are up to? Almost noon there and the web pages are still down. Starting to get withdrawal symptoms! :'(
Yesterday was worse, no Seti , Seti Beta or Lunatics.
Claggy
-
Anyone know what was with Lunatics site?
-
Anyone know what was with Lunatics site?
Yes,
I emailed Toby Murray of kwsn yesterday, his replies were:
>>Yeah this is weird. It is complaining that one of the PHP functions
>> used in formatting the date/time strings for display is deprecated in
>> the version of PHP we are running now. I have no clue why it just
>> started throwing this error as we have been using the same version of
>> PHP since we moved to the new server in the middle of August.
>>
>> I can probably patch that function by hand here in a couple of hours
>> but I also kind of want to figure out why it suddenly flipped out.
>>
>> Toby
and:
>Well after looking at what the code was doing I fixed it by just
>changing the SMF code to call a different PHP function that performs
>the same operation. The website is back up but I'm still not sure why
>it broke and it is kind of freaking me out to be honest... Do you know
>if any SMF settings were changed by admins recently?
>
>Toby
Claggy
-
And this from Rom Walton on BOINC Alpha mailing list...
Over the last few days we have been experiencing a website overload,
luckily we have a new replacement server available.
5pm EST we will be bringing down the Trac, MediaWiki, and Pootle
websites and begin migrating over all the content that is hosted on
http://boinc.berkeley.edu/ to the new server. All the various website
operations will be restored over the next couple of days.
Thanks for your understanding.
----- Rom
Better not be SETI's new server!!!! ;)
-
...
Better not be SETI's new server!!!! ;)
LOL, that would really cause a major stir! No chance, that new server probably hasn't been ordered yet. Arielle is back from a semester away so the "weekly" blog giving her interpretation of staff meetings is back at http://seti.berkeley.edu/?q=blog, and seems to indicate as of yesterday they were still considering details.
Meanwhile, they at least have mork back in running status so there would be data to support forums and such when they enable thinman.
Joe
-
Seti still offline tonight........
Still awaiting the boards to come back up to see if there is any response on my request for updated server specs...
Seems like Matt was kinda waffling on adding some of the things I had in my donation drive.
I understand that the staff is the final judge on what they think they do and don't need, but I hope that they take into account that some donations were based on the upgrades that I proposed.
We shall see....
Meow meow.
-
Since all this downtime started I have noticed I look less and less often. At this rate I may not be looking at all in a month or so....This can't be good for keeping users.
-
Since all this downtime started I have noticed I look less and less often. At this rate I may not be looking at all in a month or so....This can't be good for keeping users.
Well, keep the faith, fellow Setizens.....
I AM hoping that the new server will add enough horsepower to the server stables that maybe at some point these 3 day outages will not be necessary to further the science. Until the new server is in place and fully commissioned, with the additional server shuffling that will happen at that point, I expect we will have to continue to endure the outages.
-
And the forums are back ONLINE!!!
Meow!
-
I was just wondering if the local code wizards (Joe, Jason, Raistmer, ...) ever considered contributing to setiquest.org ?
AFAIK they provide only the radio telescope data - the users are free to analyze this data with whatever algorithm they want. This would mean much less restrictions than what we have with MB and AP applications. No?
I am sure there would be a huge community willing to run KWSN applications crunching setiquest.org data.
-
Hm, correct data analysis requires some appropriate knowledge and education first. Interpretation of recived result - too. So I don't think that "process data as you wish" will give anything good unless professional radio-astronom will guide algorithm development.
[And while I'm interesting astronomy I'm not astronom, think others too ;) ]
-
Unfortunately it's not working the other way round either (only scientists - no IT guys). We are getting close to a 24/7 downtime with S@H (3 days planned outage + 4 days unplanned outage) ;D
-
Looks like SETI is taking a long weekend off! :'( This from Jeff Cobb in the News...
Reality got a little ahead of us on this one. We were days, a week tops, away from migrating
upload service from bruno to bambi. This will double our upload space and allow us to turn
off bruno.
We will now move this up and make it top priority come Monday. We need to reconfigure
the raid on bambi and then let the raid sync. At that point we can both turn the projects
on and start migrating the results from bruno to bambi. We hope that this will be early in
the week. We'll then leave the projects on through next weekend, ie no normal 3 day outage.
-
Is it just me, or is anybody else really pumped up about the future of the Seti project?
They are reorganizing their server structure to move forward with the introduction of Oscar, and things are just moving in SUCH a positive direction lately.
A few more weeks, and this is gonna be a whole new Seti.
The kitties and I are sooooooooooo excited about this, I just can't tell ya.
Meow meow meow!!
-
Somebody must be in the lab playing around!! I just now got this message
Unable to connect to database - please try again later Error: 2013Lost connection to MySQL server at 'reading initial communication packet', system error: 113
And Yes Mark, I really am excited to see all the new things going on.
-
Somebody must be in the lab playing around!! I just now got this message Unable to connect to database - please try again later Error: 2013Lost connection to MySQL server at 'reading initial communication packet', system error: 113
And Yes Mark, I really am excited to see all the new things going on.
Just got the 'project is down' page when trying the forums......
So something is afoot.
-
Hate when they do that just as I am ready to hit the reply button in a thread. :D
-
Been trying to get work for my updated rig all weekend. Updated it to Win7-64HP, same as my main rig, updated Boinc version to 6.10.58, and then to the latest Lunatic's build. First it was telling me that it wasn't connected to a project, but that it was using the preferences from Seti anyway. In the end I had to blank the partition that Boinc sits on, and re-install from scratch. I suspect that I'll get a whole new machine ID for it, when I can get some work for it. Then just now went to check on Seti on my main rig, and found that the whole lot seems to be down. I know it'll be worth it in the end, but I'm just a bit frustrated as I've got a whole rig sitting doing nothing at the moment.
P.S. I didn't trash any work either. I ran the cache dry before the re-install.
regards, Giz.
-
Unable to connect to database - please try again later Error: 2013Lost connection to MySQL server at 'reading initial communication packet', system error: 113
Not only Spare CPU cycles, but this is the other way around, IMO, whitout the last Donation Drive, SERVER problems would keep the project from funtioning as it should...........
But it appears to be much worse, more Down -Time then UP-Time, last week !? IMO, SETI lacks serious steady funding, but that's also a
political matter and thus not an easy one to 'solve'.
Are there just too much (heavy=CUDA/CAL) crunchers, or just too much computers hammering on the continued (DDOS-like) overloaded SERVER's.
Doesn't SETI have more then a MILLION hosts connected, or is that BOINC all together, with over 2 million hosts.
Looks like, at least SETI, has become too big for it's 'clothing', it has grown and needs a new 'jacket' ..............
Well, just my 2 (Euro) cents.
-
If the info posted in the forums while they were still up holds true, it's looking like work is not gonna be available until Tuesday or so.
Unless the server shuffle goes better than they expected, but Jeff indicated it was going to be a lengthy process.
A bit of a pain in the behind, but a necessary process to get the servers reassigned.
-
That is a pain :(
Will probably be out of work on both rigs by then.
Those servers are gonna get hit so hard when they finally bring them back on line.
Hmmm, I love the smell of burned ethernet in the morning!
Giz
-
Two rigs (Q6600's) are already out of SETI work, 1 still tries to UPLoad to SETI Bęta, but it's down, too.
Only my XP64 has a few MB and 1 AstroPulse task, looks like they are swapping SERVERs around or the
A/C-Unit, is on 'strike'..............
Could indeed be tuesday or with some luck this afternoon, Berkeley time....... ::)
-
They may have more work than they expected today. The MYSQL server (Mork) went down yesterday too. That's why we can't get to the message boards. I don't know how that is going to effect things as far as getting it up and running but I still don't hold out much hope for today. I still think we will be lucky if they get it up by tomorrow.
-
I wonder if things would be better if we had planned outages every other week? Would the science suffer in this case?
-
I wonder if things would be better if we had planned outages every other week? Would the science suffer in this case?
Well, the whole setup is gonna change over the next month or so as they shuffle and retire some of the troublesome servers, as well as the addition of the new, more powerful server that is coming in the next few weeks.
Once all that is in place, we will all have to cross our fingers and hold our collective breaths that we will have a stronger, more reliable Seti project.
-
I would argue that the outages, while they do cause some issues, aren't really that bad... This issue with the upload disks choking wasn't so much due to the outage, as it was the A/C failure right before... A/C failure can and will take down even the best servers, and all but the best of datacenters.
Given their shoestring budget and the scale of the userbase, SETI is doing a phenomenal job of keeping the lights on, even if the uptime is currently hovering around 50%. The BOINC platform itself was built to allow for a project with frequent outages or work shortages, and as such they highly encourage you to enroll in multiple (non-related - ie. not SETI and SETI beta) projects to keep the pipes moving.
-
Well, some of the problems with the system choking are suspected to be related to the accidental creation of ghost tasks, which seem to be inflating the size of the database over the last couple of months.
You can get a little more detail of some of the things that are being dealt with from the technical news link on the home page....which is currently available, even though the forum itself is not.
http://setiathome.berkeley.edu/tech_news.php
-
I used to run SETI quite a bit, but I'm sitting on the sidelines for the moment (doing other projects) hoping they can find and implement a more predictable solution.
I hate it when I change something and they change something. I can't figure out who to blame when it doesn't work :P
-
Looks like someone's in the lab early today
Download service on Vader's just kicked in :D
-
Looks like someone's in the lab early today
Download service on Vader's just kicked in :D
But what good does that do if everyone is full of uploads? Wonder why they didn't start upload server first? Just down to a few VLars on my CPU's.
-
Looks like someone's in the lab early today
Download service on Vader's just kicked in :D
But what good does that do if everyone is full of uploads? Wonder why they didn't start upload server first? Just down to a few VLars on my CPU's.
Well....nothing is showing up on the Cricket graphs yet. But let's hope it is a sign that things are going along OK and they are getting ready to come back online today.
And there may be a bunch of stuck downloads out there......but I think mine are mainly all uploads too.
I got the kitties crossing all their little toes for luck..........
-
And the forums just came back online!!!
Meow meow!
-
But still nothing going in, they must still be syncing the raid on the new upload server.
-
All my machines have been able to upload all their results, some of these have a report deadline of 01-Oct-2010, so things are getting a bit tight to say the least.
However, I cannot report the results("Couldn't connect to the server."), and also no new work is being sent out apparently.
Let's hope they get the rest running real soon now...
Regards, Patrick.
-
All is purrrrrrrrring along pretty well right now.
Most important server services are online. All my rigs have reported their week's worth of backed-up completed results. The kitties give a collective sigh of relief, and are sniffing for new work.
It is going out as the bandwidth allows them to connect.
Cricket graphs show the bandwidth maxxed out, so work is going out as fast as the infrastructure will allow.
Hope it can hold together better now under the new server configuration.
Meow meow.
-
Oh, crap.,...
Sunday mornin' comin' down.....
And so is Seti..
Not even the boards are up.
Such a bummer.
-
Guess they are still there.......
After about 10 minutes, the forums started to respond again...sort of.
-
In my Boinc Notices Tab it says
Our upload (Results) storage area is full
the projects are down until we resolve this in the next few days
10/1/2010 11:00:08 AM
-
Yeah, they just haven't changed that message HiAmps this is something else again. The website seems to be going up and down a lot. I was trying to send a reply to a post when it started this time, hope it isn't resending the post each time I retry to get to the website.
-
LOL, missed that... :o
-
The site is not working again, although in the morning everything was ok ... ???
-
At this moment, I can notreach SETI@home or SETI-Bęta, it looks like they are down, again :-\
(12:08 UTC+2)
Try again in a few minutes ::)
Well looks like I was wrong, it took some time, but it's UP & Runnin............
-
I've got a partial Forum page open, and my E8500 has just had 20 former Ghosts resent to it, so it is up, but under heavy load,
Claggy
-
and here we go again :'(
-
and here we go again :'(
The Forum might be down for Maintenance now, but the scheduler is still accepting requests, still not getting any work often, maybe that'll change, :)
Claggy
-
Yeah, just when I decided to try your trick for getting it to resend lost tasks! :-( Now I'll have to wait another 20 minutes till I get another task completed to try again if I have to unless it tries to request more work.
-
From the Seti home page..................
The database used for web requests is down, and the main database cannot handle both web requests and work distribution. We have disabled web access to allow work distribution to occur smoothly.
3 Oct 2010 18:29:16 UTC
So it looks like no forums until sometime Monday.
-
Yeah, just when I decided to try your trick for getting it to resend lost tasks! :-( Now I'll have to wait another 20 minutes till I get another task completed to try again if I have to unless it tries to request more work.
I was doing the same but I had to read it several times to try to understand the resend trick, didn't get it...
Do you have an easy to understand manual? ;D
-
From the front page:
The database used for web requests is down, and the main database cannot handle both web requests and work distribution. We have disabled web access to allow work distribution to occur smoothly.
3 Oct 2010 18:29:16 UTC · Comment
-
Forum's are up again.
Claggy
-
Forum's are up again.
Claggy
And just what that was all about, I haven't a clue........take 'em offline because the master DB could not handle it, and then right back up again?
Maybe they found some other problem. But the replica still shows as being offline.
-
Well the status page has up dated with fresh figures now,
807,174 MB ready to send, 7,187 Astropulse ready to send,
it's just getting them to or from the feeder that seems to be the problem,
Claggy
-
And we're gone again!!
-
and the Forum's are down again,
Claggy
-
This is starting to read like a play-by-play of some of today's football games............
-
Disabled web access only. Everything else seems to work fine.
-
Disabled web access only. Everything else seems to work fine.
Disabling the web seems to have done the trick, I got my 5 day cache filled quickly.
-
Same here, had a problem with a crash yesterday and had to set up a new Boinc installation. Couldn't get any work all day (needed a new machine id), then after I gave up and went to bed, they've disabled the web access and my machine has got it's id and filled up with work.
Just checked it now, I have plenty of work (5.40am in the UK), but cannot access the forums still. I guess that the boys will sort it out when they come in. Maybe the server reconfig hasn't gone as well as planned.
Giz.
-
I wonder what prevents them to engage Jocelyn (replica). Her status shows she's running, but for what purpose when she's offline? I mean, replica has been more than usefull in situations like this before.
-
I wonder of Thinman was affected when the A/C went down.
-
Their actions are sometimes difficult to understand. Simple things they are trying to solve a very strange way ...
-
Their actions are sometimes difficult to understand. Simple things they are trying to solve a very strange way ...
They are indeed. But right now I'd suggest they should loosen the limits a bit if they're planning another 3 day science fair this week. Cricket (http://fragment1.berkeley.edu/newcricket/grapher.cgi?target=%2Frouter-interfaces%2Finr-250%2Fgigabitethernet2_3;view=Octets;ranges=d%3Aw) confirms it.
-
Well my cache should be good for about 12 days now. Woke up to find that I was able to pick up about 20 APs during the night.. 13 in one request.
Now if I could just pull up the tasks page so that I can update my spreadsheet for APs... :P
-
Hey cool... BOINC master database is now disabled, and there's a whoooole lot of red.
-
I think someone is trying to get it back in operation, some status changed between the 1:40 and 1:50 UTC status page updates.
Joe
-
Projects are down due to a database machine crash.
The machine that serves the BOINC database crashed. The projects are down until the recovery is complete.
That answers it.
-
I think someone is trying to get it back in operation, some status changed between the 1:40 and 1:50 UTC status page updates.
Joe
Uploads still working tho...
In any case, I give them credit because they were there on Monday evening. Even if it was remote controlled :)
-
Apparently mork has been throwing tantrums and crashing on a regular basis. When that happens, the replica (thinman) has to be reloaded from a new backup copy. And according to Eric, mork has been crashing so often that the exercise is futile.
So the problem is really not with the replica, thinman.....
They are going to try to swap some different RAM into mork tomorrow in an effort to find the problem.
Let's keep our fingers crossed for them.
-
Let's keep our fingers crossed for them.
Aye aye...
-
Hopefully they get some of the new computers on-line sooner than later.
-
Hopefully they get some of the new computers on-line sooner than later.
Well, supposedly bruno is retired now, with bambi taking over upload duties.
I am not sure if they have finalized specs and actually placed the order for Oscar, the brand new powerful server which the Seti folk donated so generously for. It certainly is not online yet, and probably won't be for a few weeks at best. So I hope they can get mork stabilized between now and then.
-
Hopefully they get some of the new computers on-line sooner than later.
Well, supposedly bruno is retired now, with bambi taking over upload duties.
I am not sure if they have finalized specs and actually placed the order for Oscar, the brand new powerful server which the Seti folk donated so generously for. It certainly is not online yet, and probably won't be for a few weeks at best. So I hope they can get mork stabilized between now and then.
Can't check as the website still says bruno is running...
Just as one thing gets fixed, another rears it's ugly head! Boo! Just got one machine up and running after problems with it crashing over the weekend, now the other has run out of work! Has anybody suggested a flaky power supply possibly for some of the problems besieging mork? I know that Mark said that they're looking at changing out some ram on it today. There must be a reason why it keeps crashing, and it seems to be getting worse and more frequent...
Giz.
-
Looks like they are going ahead with the scheduled 3 day outage....
Weekly Outage and Initial Catch Up
Every Tuesday morning (Pacific time) we begin a 3 day data distribution outage to focus on science processing and development plus any needed systems maintenance. The upload/download servers will be offline during this time. The web site (including the forums) will only be offline during the database maintenance and backup portion of the outage. On Friday, you may experience connectivity issues as the servers catch up with demand. 5 Oct 2010 16:34:36 UTC
-
Looks like they are going ahead with the scheduled 3 day outage....
Weekly Outage and Initial Catch Up
Every Tuesday morning (Pacific time) we begin a 3 day data distribution outage to focus on science processing and development plus any needed systems maintenance. The upload/download servers will be offline during this time. The web site (including the forums) will only be offline during the database maintenance and backup portion of the outage. On Friday, you may experience connectivity issues as the servers catch up with demand. 5 Oct 2010 16:34:36 UTC
That is an automated message........
It is unclear at this time whether they will actually have an extended outage or not. I suspect if they can get mork back online, they would want to restart data distribution ASAP....due to all the recent downtime. But that is purely my conjecture.
And Oscar is slated to replace thumper as the master science database....so they really still need to get mork sorted and stable.
-
Yeah but every time I look at the server status page it's different. Guess they are playing with everything while trying to get things working.
-
And Oscar is slated to replace thumper as the master science database....so they really still need to get mork sorted and stable.
I remember that Matt talked about the three problematic server. But I do not remember that Thumper was the one who recently made the most problems. I believe that it was Mork from the beginning, and perhaps it should be replaced first?
OTOH, Mork is their only server capable of handling the mysql database... I can only guess how it's like being in their shoes.
-
So SETI provided us with more varied ways to search the database. Badly needed improvements I might add. Then surprise, surprise..........servers cannot deal with the increased database search requests.
Did anyone actually expect that the increased searches could be handled without problems? Right now SETI cannot even keep up with uploads and downloads at the same time. Users are simply overwhelming the system.
So one new server, paid for by some very generous crunchers. Anyone think that will solve all the problems?
-
And Oscar is slated to replace thumper as the master science database....so they really still need to get mork sorted and stable.
I remember that Matt talked about the three problematic server. But I do not remember that Thumper was the one who recently made the most problems. I believe that it was Mork from the beginning, and perhaps it should be replaced first?
OTOH, Mork is their only server capable of handling the mysql database... I can only guess how it's like being in their shoes.
I think you're right. Mork has been a little unstable on and off since it was installed. The problem is that it is the most powerful server they have at the moment, and as such is relied upon to do much of the heavy work. It seems to be that as time goes on, it's flaws are being revealed more and more. I'm not sure how they can track the fault down on it, Mark said they are going to try to change the ram, and I've mentioned the power supply. I'm assuming that it has a redundant power supply, or the ability to hot swap it.
I know what you mean. 150,000 users all crying out for work, running round like chickens with their heads' cut off. Matt and co must be tearing their hair out trying to sort out the recurrent problems they've got. I've shut a machine down for the moment, as it has no work, and I'm trying to be patient. I don't crunch anything except Seti, and I still haven't been tempted enough to try an additional project.
Let's hope they can crack it, and if they do, maybe open the taps and let a bit of work out.
So SETI provided us with more varied ways to search the database. Badly needed improvements I might add. Then surprise, surprise..........servers cannot deal with the increased database search requests.
Did anyone actually expect that the increased searches could be handled without problems? Right now SETI cannot even keep up with uploads and downloads at the same time. Users are simply overwhelming the system.
So one new server, paid for by some very generous crunchers. Anyone think that will solve all the problems?
Maybe not all of the problems, but it will hopefully provide a strong reliable server to start re-organising the rest of the servers and give a stable base to work from. It makes you wonder how they get any science done at all, when they're fire-fighting servers all the time. It might give them some time to pinpoint the most unreliable servers and work from there.
Giz.
-
Hopefully they get some of the new computers on-line sooner than later.
Well, supposedly bruno is retired now, with bambi taking over upload duties.
I am not sure if they have finalized specs and actually placed the order for Oscar, the brand new powerful server which the Seti folk donated so generously for. It certainly is not online yet, and probably won't be for a few weeks at best. So I hope they can get mork stabilized between now and then.
Can't check as the website still says bruno is running...
Just as one thing gets fixed, another rears it's ugly head! Boo! Just got one machine up and running after problems with it crashing over the weekend, now the other has run out of work! Has anybody suggested a flaky power supply possibly for some of the problems besieging mork? I know that Mark said that they're looking at changing out some ram on it today. There must be a reason why it keeps crashing, and it seems to be getting worse and more frequent...
Giz.
Eric says that subbing the power supplies is next on the list if the RAM swap does not make mork happy. There are 2 power supplies, they would try subbing one, and then the other. And he says they do have spares.
-
yep the drought is certainly getting deeper...apart from the conversation above is there any view ecking out on when its going to come back up and, i suppose more importantly, whether they will open the gates to allow the waters to flow to quench our parched fields ??
-
Not certain.......
It depends on their luck with getting mork sorted.
If I can get any more news tomorrow morning I will post it here.
-
thnx very much
vp
-
Eric says that subbing the power supplies is next on the list if the RAM swap does not make mork happy. There are 2 power supplies, they would try subbing one, and then the other. And he says they do have spares.
Thanks for the info, Mark.
I don't know much about servers to suggest much more. Never come across a faulty cpu after the Pentium IV problem that couldn't add up. Only thing left after that is the motherboard, or the server software. Even the ECC ram I wouldn't have expected to fail, but it's worth swapping if they have some spares. Hardware I know a bit about, server software I know zilch.
Giz.
-
Eric replied this morning that there was simply too much to do in one day trying to get things up and running.
He says he is hopeful for today......
The kitties are too, as I have rigs going idle one by one as they run out of kibble to crunch on.
-
Eric replied this morning that there was simply too much to do in one day trying to get things up and running.
He says he is hopeful for today......
The kitties are too, as I have rigs going idle one by one as they run out of kibble to crunch on.
Good news and it's good to know that we have an insider to keep us informed :)
-
I'm not worried so much about work as I've still got a couple of days worth on my machine but I sure do miss the forums! ;D
-
I'm not worried so much about work as I've still got a couple of days worth on my machine but I sure do miss the forums! ;D
These are prety good forums too. Only thing that's missing is the voice of James Earl Jones in my head sayin' "Join me ... and recieve a free toaster oven" ;D
-
Yipee! The forums are back, account pages too but with some restrictions. (ie. pendings, task list)
Looks like we're finaly moving 8)
-
Which would indicate that they are still running without the replica online...trying to minimize load on the database as much as practical.
Lets hope mork is now up to the task........
-
I Can't log into my account...............and can't make any post on the forums.
ver mind.........I got logged into my account now.
-
I Can't log into my account...............and can't make any post on the forums.
ver mind.........I got logged into my account now.
Might have been things just getting resynched.....
-
Jeff just posted the following in the Technical News thread.........
"It's been a painful week, but with some progress.
The server run before last was cut short by our upload space filling up. That was fixed by the bruno migration and we started the last server run a bit early.
But a crash of our primary boinc db machine, mork, got the secondary db server, jocelyn, out of sync. That meant that all of the read only queries had to go to mork instead of jocelyn. This overwhelmed mork and I turned off web access just so the server run could continue. Then mork crashed again Monday evening. Ouch.
Yesterday, we did our normal backup of mork and are recovering jocelyn from that today. The forums are up, but result viewing is disabled at the moment. We need to clear the back end queues ahead of the next server run and mork resources are needed for that.
Mork's tendency to crash seems to have accelerated. Perhaps this is secondary to the cooling crisis we had a couple of weeks ago. Actually, "crash" is not the correct term. It simply hangs and requires a power cycle to boot. Fortunately, we have mork on a networked power strip and can power cycle it remotely. Upon boot, there are no footprints whatsoever as to the cause of the hang. This sounds like hardware. So today we are going to bring mork down to swap out all of the memory and remove a couple of unused components in a desperate attempt to fix the problem. The forums of course will be down during this operation."
So, be prepared for some more downtime this afternoon. Hopefully all goes well with the mork RAM transplant.
Meow meow for now.
-
I've actually been thinking about this for a little while now, but I think it is confirmed that there is in fact gigabit going up the hill to SSL now (don't know if that includes the Hurricane Electronics link as well), but SAH is still limiting their usage to 100mbit? I'm thinking if maybe they could increase it to say.. 150 or 200, downloads would end up going through faster and therefore not have the pipe maxed out for as long. Or possibly putting the scheduler on a separate VLAN so there are no packet contention issues when the pipe is maxed out for downloads. Observations over the past few years have shown that most of the problems with ghosts or just failed transfers happens when there are a ton of downloads taking place.
Maybe one possible idea is to cycle the scheduler and download processes. Just do something like 15 or 20 minutes with the scheduler enabled, but downloads disabled, so you get work assigned, but can't download it, but then turn the scheduler off and let downloads run for an hour or two, and repeat until the pipe is no longer maxed out. It's a thought.
My area of expertise is in networking, and it seems like there are several ways to probably fix, or at the very least, smooth out some of the rough edges.
No doubt that Oscar will make a marked/noticeable improvement over the current setup, but it seems unlikely that one server will fix everything, especially since most of the problems are more network-related than software-related.
Just my opinion.
-
Bandwidth is NOT the problem right now.....
It's server stability......most notably mork's lack of it.
If everything hangs together on the server side, we can live with the bandwidth currently available.
The problem has been with the servers crashing Seti-side, not the bandwidth. It may contribute to the ghost problem, but until the servers are all stable, that cannot be established.
Hang in there folks, it's gonna be a bumpy ride. LOL.
-
Dan has contacted HP and Sun to see if they can give us a deep discount on a machine that could replace mork, hopefully deep enough that we can purchase it on what's remaining from the donations made in the Number Crunching threads.
Good luck Dan, that would be great, sort of like two for the price of one. I hope it doesn't cut into the amount of RAM and harddrives you guys were going to get with Oscar though. At least not too much. If it does let us know so we can wake up Kittyman to start another drive.
____________
Eric posted the first part of this and the second was my reply in tech news. Sure would be great to have two new servers, they sure are needed.
As to the bandwidth, the line they ran up was for the whole lab, guess they didn't run the Hurricane link while they were at it. That's another thing that would really help.
-
I have started a new Seti server fund drive in the NC........
Please participate if you can.
The project needs your help.
Meow meow.
-
I have started a new Seti server fund drive in the NC........
Please participate if you can.
The project needs your help.
Meow meow.
Good job Mark ! There are 5K+$ already raised. I'll see what I can do, without robbing the bank. ;D
-
I wish I could rob the bank, not that they have got much money at the moment!
Someone in the UK won our Euromillions lottery last night, and will wake up Ł113 million richer this morning.
I'm just sorry to say it wasn't me.
The servers really wouldn't have been a problem then...
-
The response, again......has been simply amazing.
Simply amazing.
Those that luv this project are wonderful folks.
You, of course, are too.
This little planet and the hope that we all cling to make it worth the trip.
Meow, meow.
-
They kicked me......must have been too verbose for their tastes/
But, I'll leave you with this....
http://www.youtube.com/watch?v=alGx-O3MBzY&feature=related
Tastyest guitar I have found in ages.
-
There are some things that you can do with a strat that you cna't do in bed.
Or a sax, for that matter.
http://www.youtube.com/watch?v=qeksR8ulBPM&feature=related
-
Not only some of the best strat action ever recorded, but some of the nastiest drim work too.
http://www.youtube.com/watch?v=HRMSNiSaP7s&feature=related
-
the '66 concert in Japan............
The apex of axe art.
http://www.youtube.com/watch?v=K40ryWp3Ro4&NR=1
-
Just i know why seti is so frequently down -> youtube
-
For Mark and all other faithfull setizens, pretty green glow of the cricket graph.
(http://img251.imageshack.us/img251/8118/minigraphcaam7tfc.png)
-
Looks like we broke SETI again. Can't even get to the homepage. Been like that for about 15 minutes. Seems a bit early for the (ab)normal outage.
-
Looks like we broke SETI again. Can't even get to the homepage. Been like that for about 15 minutes. Seems a bit early for the (ab)normal outage.
Everything dropped at about 6:30 am local, first day back after the long weekend. It's that relief Janitor again, using the wrong circuit for his hoover....
-
Looks like we broke SETI again. Can't even get to the homepage. Been like that for about 15 minutes. Seems a bit early for the (ab)normal outage.
Everything dropped at about 6:30 am local, first day back after the long weekend. It's that relief Janitor again, using the wrong circuit for his hoover....
More like someone unplugged the router from the UPS for their coffee maker. :D If the power goes out, coffee can still be made.
-
The forums, uploading and reporting just came back online......dunno for how long, but the are there now.
EDIT...Downloads too!
-
Jeff just posted an explanation in the Seti Technical News forum......
"When I looked at the cricket graph this morning, I thought, "oh no, mork is hung again". But nope, not this time. It was the UPS supporting our internal file server - the one with all of our home accounts on it. That UPS is now completely dead. We took it out of the loop and everything came up OK. We'll be starting the outage soon."
Dunno yet if it's gonna be 1 or 3 days......
-
And the outage has now started.
Eric says it will probably be a 3 day outage due to more work that Matt and Jeff need for further work on testing mork and additional tuning of the servers.
And Jeff has indicated that when they come back up, he will give up higher limits this week.
Meow meow meow for now.
-
I'm a bit concerned that the symptoms described for Mork sound like tired mobo caps & 'dendritic growths' on the PCB. Wish I could teleport there, I'd have it fixed pretty quickly.
(Sadly not a common skillset)
-
And the outage has now started.
Eric says it will probably be a 3 day outage
<snip>
Ugh..and it looks like I loaded up on a lot of shorties.
:(
-
Seti had more problems today:
We had another mysql database crash (and recovery) this afternoon which delayed our usual outage chores - the forums will remain offline for at least a few more hours (maybe all night).
12 Oct 2010 23:02:55 UTC
Claggy
-
Liebe Seti-Leute, so langsam wird das mit den ständigen Server-Downs echt peinlich!
Bei kaum einem der anderen BOINC-Projekten ist so oft die Server nicht erreichbar.
Klar, Server Hard-/Software kosten Geld, und vor Ausfällen und Defekten ist niemand wirklich geschützt,
aber ihr seit die Vorreiter, die alten Hasen!
Kriegt das mal in den Griff, BITTE
-
It is too much at them problems lately.
:D There can be they lose qualification? ;D
-
It is too much at them problems lately.
:D There can be they lose qualification? ;D
Ha! you learn Australian humour fast :P
-
;D I try not to lag behind ::)
-
The project's database server is down. Please check back in a few hours.
Well ghosts and cache again I guess...
-
Well ghosts and cache again I guess...
LoL, I saw that funny exchange on NC just before it went down, and had a chuckle.
IMO More like try the sum total of:
- Clients built to perform Distributed denial of service atatcks on their servers,
- Timing induced by extended outages
- Moore's Law
- Limited download pipe
- probably more variables I haven't thought of
Symptoms don't count as causes:
- Ghosts are a symptom of pressure on the system in one or more places
- People increasing their client cache sizes is a symptomatic response
- Both may be subsequent cotnribtors to storage space filling up. With the recent increase in size allocated there I don't see how that could be a problem aready, and IIRC the numbers , Results out in the field has dropped substantially over the last week.
The Stepping up of quotas has shown to improved the equitability of work distribution, but IMO the stepping up has been too rapid ... Perhasps as a stress test ?
- More uptime seems to help, but with problematic servers for the time being that one's out
- addressing the other causes I think would take bucket loads of cash & time for the project. I hope the recent donations help some of those.
Don't know what to do about Moore's law though :D Maybe we should all go back to Commodore 64s , Darn sight easier to program anyway, and no fans to clean out.
Jason
-
Maybe we should all go back to Commodore 64s , Darn sight easier to program anyway, and no fans to clean out.
Jason
Would solve a lot for Seti and you get to listen to the cool whining noise as a game loads from cassette.
Still got mine somewhere - may have to dig it out and have a game of Chuckie Egg soon ;D
-
Don't have a Commodore 64 anymore but do have an Amiga 500, if it still works I don't know, forgot how many years since I tried it last.
-
Seems like mork is still a sick puppy......
I suspect they shall order a replacement very soon.
-
The forums are back up,
Claggy
Edit: looks as if the rest of the project might be coming up too, there's quite a lot of green on the Status page.
-
And just like that, it crashed completely.
-
It's moving again now....but for how long?
-
How about till right about now? It's down again!
-
How about till right about now? It's down again!
Yeah, it's all dead in the water now......they're probably using the defibrillator on mork.
-
Man, das ist doch Shice!
Die seti-Server laufen nicht mal zwei Tage, und schon sind die wieder down ..
-
Man, das ist doch Shice!
Die seti-Server laufen nicht mal zwei Tage, und schon sind die wieder down ..
Ja ist es schlecht ...... Aber ich habe Erhöhung viele Abgaben zum Projekt geholfen, und zwei neue Bediener werden bestellt. Fürs Erste müssen wir geduldig sein.
-
Well.......
It ain't comin' back up tonight, folks.
Best I get some sleep and check up on it in the morning.
No matter what some naysayers would have you believe, better days are coming.
Long live Seti.
Meow meow meow.
Mark
-
this new process is claerly not working well at all...if we are lucky its up for two days before disappearing in heap...
why don't they just admit that they have done the wrong thing and go back to what they were doing before...
vp
-
this new process is claerly not working well at all...if we are lucky its up for two days before disappearing in heap...
why don't they just admit that they have done the wrong thing and go back to what they were doing before...
vp
It's a rouge server that is dogging them.........not anything else they have done.
The master database server, mork....has stepped to the edge and looked down, and unfortunately, has jumped off.
There is nothing wrong with the project, it's goals, or it's means of achieving them. It's just the freakin' hardware. Which should be remedied as soon as they can get the new servers we have funded into the stable.
Keep the faitth, my friends, just keep the faith.......
It may still take some weeks before the new servers are online........
Please have faith.......it will come.
-
Thanks für Information!
And sorry for the German, but my english not the best.
-
Thanks für Information!
And sorry for the German, but my english not the best.
LOL...neither is my German......although it is the land of my daddy's mother and father.
Hope the Babelfish translator didn't mangle things too badly....LOL.
Eric confirmed to me this morning that mork is officially on the way out. But they are rebuilding drive tables, resynching, and trying to get things back on the road as soon as they are able.
I wished him the best of luck with the bailing wire and duct tape until they can obtain a new server to replace mork.
Please, everybody. Try to keep the faith whilst this major transition occurs. There is gonna be a lotta pissing and moaning on the Seti boards when they come back up.
The folks here, I think, have a better grasp of the reality of the situation and a more positive attitude than some Seti participants do. Could you please try to reflect that in your posts at Seti for the next few weeks?
It might help smooth some feathers during these rough times.
Thanks, and meow meow.
Mark and the kitties.
-
I thought i would give youall a dish on the newest kitty poem.........
Not yet published on Seti.
I saw a little kitty just the other day.......
She was walking across my driveway. Going her way.
I tried to coax me up to where I was....
but she was set in her ways,
wouldn't let me near.
She was not a feral, that I know.
For she would not have even let me approach her so.
I put some food out, hoping the she
might come back the next day
and consider approaching me.
Next day came and went and I did not see
my little kitty friend come back
to say hello to me.
Couple of days later, and a mile or two away,
I was driving along....
and in the middle of the night,
A kitty in the middle of the road.
You know, those eyes that glow in the headlights.
I stopped, and stared......now could it be?
The same little kitten in my driveway that was looking at me?
Wheeled around and went back to the parking lot where
I had last seen her run.
But the lot was quite bare.
I started to search, with my car standing by,
and then another car parked alongside mine, Oh, my.
He took one look at myself crawling there
in the bushes.....oh, Lord, what is he doing there?
Turns out that the security system had eyes only for me,
and not the kitty that I was seeking could they see.
It was hard to explain that I was trespassing so
Only to seek out a kitten that would make my heart glow.
And then the lot turned quite bright, for you see.......
The cops where looking for not the kitty, but me.
Grilled me for about an hour or so.......maybe it was two.
Did not believe that I was just a simple kitty soul.
I am not sure t(dy really believed me.....but they had nothing else to do that night.
Kitty has not been seen since that day........
Frickin' cops, they have gone their own way.
The local gendarms finally let me go when they figured out I was not trying to loot the
place for kibble.
Sent a couple of prayers up to the kitty Gods.....
Asking for the return of the lost one I sought.
The prayer's not been quite answered of yet.
But I'm not sure tha Lord can recieve all he's sent.
But my history's been writ on the cop's local glow........
"This man is the kittyman......
Just let his ass go."
MSS.......10/18/10
-
Ok.......I am suffering fron Seti withdrawl
Got a couple of things to say............
I saw a little kitty just the other day.......
She was walking across my driveway. Going her way.
I tried to coax me up to where I was....
but she was set in her ways,
wouldn't let me near.
She was not a feral, that I know.
For she would not have even let me approach her so.
I put some food out, hoping the she
might come back the next day
and consider approaching me.
Next day came and went and I did not see
my little kitty friend come back
to say hello to me.
Couple of days later, and a mile or two away,
I was driving along....
and in the middle of the night,
A kitty in the middle of the road.
You know, those eyes that glow in the headlights.
I stopped, and stared......now could it be?
The same little kitten in my driveway that was looking at me?
Wheeled around and went back to the parking lot where
I had last seen her run.
But the lot was quite bare.
I started to search, with my car standing by,
and then another car parked alongside mine, Oh, my.
He took one look at myself crawling there
in the bushes.....oh, Lord, what is he doing there?
Turns out that the security system had eyes only for me,
and not the kitty that I was seeking could they see.
It was hard to explain that I was trespassing so
Only to seek out a kitten that would make my heart glow.
And then the lot turned quite bright, for you see.......
The cops where looking for not the kitty, but me.
Grilled me for about an hour or so.......maybe it was two.
Did not believe that I was just a simple kitty soul.
I am not sure they really believed me.....but they had nothing else to do that night.
Kitty has not been seen since that day........
Frickin' cops, they have gone their own way.
The local gendarms finally let me go when they figured out I was not trying to loot the
place for kibble.
Sent a couple of prayers up to the kitty Gods.....
Asking for the return of the lost one I sought.
The prayer's not been quite answered of yet.
But I'm not sure the Lord can recieve all he's sent.
But my history's been writ on the cop's local glow........
"This man is the kittyman......
Just let his ass go."
MSS.......10/18/10
And then this.........
Meet me in the middle of my dreams.....the we can talk.
Hammond B52 organ in the background.......kitties sighing for their lost soulds.
http://www.youtube.com/watch?v=hSq4B_zHqPM&ob=av3e
-
How long does the database recovery normally take? Not that I need WUs ;D
-
It takes until it takes.........
Meanwhile, back at the kitty ranch......I was just thinkin'.........
Bought that n word I got banished for a while back.
And I might say.........fuck anybody who cannot see the humour that Mel Brooks intended in this.l
Niggers excluded, of course. I suspect there are not any nigges monitoring thiese threds right now who might take offense.
And any persons of dark skin,,,,,,,,,,thiis was not directed at you....it was Mel,s poke at our sometimes bad hope of liberty at your expense.
-
Projects are down due to a database machine crash.
The machine that serves the BOINC database crashed. The projects are down until the recovery is complete. The current database machine has become too unreliable to continue as the primary DB server. We are going to move this functionality to the machine that is currently acting as the replica server. This will be a temporary fix, as this move will leave us with reduced throughput capacity. We are ordering a new replacement sever. This purchase is made possible by generous donations from SETI@Home participants.
18 Oct 2010 21:36:31 UTC
"reduced throughput capacity" means getting (even) harder downloading WUs I guess...
-
They are doing the best that they can do to patch up mork.....
They will try to do their best until kitty Holiday shows up to fis it all........
This is a different take on Doc's little difference of opinion.......
http://www.youtube.com/watch?v=QmdfD2byTuY
-
"reduced throughput capacity" means getting (even) harder downloading WUs I guess...
I think it mostly means that it will be a less powerful machine.
jocelyn: Sun V40z (4 x 2.2GHz Opteron, 28 GB RAM) - Replica (Either 4 or 8 cores)
mork: Intel Server (4 x six-core 2.13GHz Xeon, 64 GB RAM) - Master (24 cores)
Less cores and less RAM until they can get the new servers in place.
-
Asilde forn the server isuue, I have a social isssul that I must addrase.......
The n word. 'they are what they are, and choos to present thensvles to te.
If anybody pesent wishes to present themselves......please do so now.
And I may stand down........or not.
I realize that all men were created equal..........
just an irrellevet movie clip............
White gurl with dirty blond hair......
No.......you aint right yet.
http://www.youtube.com/watch?v=cHhxW6wDHXs&feature=fvst
-
A little sidetrack for you all........
One of my favoarite flicks.......
Young Frankenstein.
Stay close to the candles...............
http://www.youtube.com/watch?v=S6Ec43f1xdU
http://www.youtube.com/watch?v=gssf-r3AFAU&feature=related
-
"reduced throughput capacity" means getting (even) harder downloading WUs I guess...
I think it mostly means that it will be a less powerful machine.
jocelyn: Sun V40z (4 x 2.2GHz Opteron, 28 GB RAM) - Replica (Either 4 or 8 cores)
mork: Intel Server (4 x six-core 2.13GHz Xeon, 64 GB RAM) - Master (24 cores)
Less cores and less RAM until they can get the new servers in place.
I am so glad Mark had those two fund raisers. At least they have some cash to attempt a solid fix.
Steve
-
Just keeping myself amused whilst the crunchers do their thing........
Was perusing youtube.....
Got into a loop about the gay boy from Milwaukee......you know, the town that made beer famous, or was it the other way around?
Anyways, there was this young kid from the suburbs named Liberace......(as the clip goes, first name and last)...
I remember as a kid watching his show on TV. We all waited for the highlight number of the week, and he never disappointed. I have bookmarked about a dozen of them, which I would rather wait and share on Seti rather than duplicate them here.
But this one is kinda special...so I will.
A cliip he did on the Smothers Brothers show...... (http://www.youtube.com/watch?v=R2BdoPTGc64&feature=related) "Do yo know how fast you were playing???"
-
Still one of my favorite movies of all time........and one of my favorite actors.
I guess hes has directed another movie coming out soon.....
But his career is so long and storied that I can't even begin to discuss it here.
Clint Eastwood.
From Dirty Harry hell and back, he has been there. And with class. I guess the scuttlebutt around the set is that he is as wonderful a man as a director as he was an actor.
You go, Clint.
http://www.youtube.com/watch?v=5SO5VO2ixWY Unforgiven
http://www.youtube.com/watch?v=aM8iT1UHnjI Gran Torino spook scene
-
Asilde forn the server isuue, I have a social isssul that I must addrase.......
The n word. 'they are what they are, and choos to present thensvles to te.
If anybody pesent wishes to present themselves......please do so now.
And I may stand down........or not.
I realize that all men were created equal..........
just an irrellevet movie clip............
White gurl with dirty blond hair......
No.......you aint right yet.
http://www.youtube.com/watch?v=cHhxW6wDHXs&feature=fvst
Huh! Weird we cant view this in UK - says
This video contains content from Vevo, who has blocked it in your country on copyright grounds.
-
A little sidetrack for you all........
One of my favoarite flicks.......
Young Frankenstein.
Stay close to the candles...............
http://www.youtube.com/watch?v=S6Ec43f1xdU
http://www.youtube.com/watch?v=gssf-r3AFAU&feature=related
Can't fault you on your choice of possibly the best film ever made.
To borrow from this and imaging Eric working late in the lab pinned to the wall by Mork and saying to Matt ......
'Whatever you do, don't put the RAM back ......"
"Now push with all your might ..........."
Of course by now some Setizens are probably saying "A riot is an ugly thing and it is about time we had one!!"
Look where thats getting the French!
Live long and prosper ............
John.
-
A little sidetrack for you all........
One of my favoarite flicks.......
Young Frankenstein.
Stay close to the candles...............
http://www.youtube.com/watch?v=S6Ec43f1xdU
http://www.youtube.com/watch?v=gssf-r3AFAU&feature=related
Can't fault you on your choice of possibly the best film ever made.
To borrow from this and imaging Eric working late in the lab pinned to the wall by Mork and saying to Matt ......
'Whatever you do, don't put the RAM back ......"
"Now push with all your might ..........."
Of course by now some Setizens are probably saying "A riot is an ugly thing and it is about time we had one!!"
Look where thats getting the French!
Live long and prosper ............
John.
LOL....seems that mork did not respond........
Must have got his RAM fom Abby Normal........
And we all know what the French got.......toast, of course.
-
Forums are back up and Jeff has posted this in Tech news:
The good news is that forums are up and the projects will be up soon.
The bad news is that the work limits will be quite restrictive for a while. This is because we swapped the boinc mysql db master and replica servers. The master had been mork and mork has just become too unreliable The new master is jocelyn and it has less than half the memory of mork.
The good news is that the bad news is temporary, because yesterday we ordered a new mork! It should be here in a couple of weeks. Details on this will follow. BTW, we also ordered the new science db server!
We'll have to feel out the outage schedule over the next few weeks. Thank you for your amazing support and your patience.
-
Well.......
It looks like we've crashed jocelyn.
From the home page......
Unable to connect to database - please try again laterError: 2013Lost connection to MySQL server at 'reading initial communication packet', system error: 111
From Boinc messages......
10/21/2010 1:18:00 AM SETI@home Sending scheduler request: To report completed tasks.
10/21/2010 1:18:00 AM SETI@home Reporting 9 completed tasks, not requesting new tasks
10/21/2010 1:18:08 AM SETI@home Scheduler request completed
10/21/2010 1:18:08 AM SETI@home Message from server: Server can't open database
-
Well.......
It looks like we've crashed jocelyn.
Given the present circumstances and huge demand, it wasn't that hard to break her down. Even without 3+ day outages, 14 months ago, she was barely up to the job. It is sad that this happened without any downloads and can only hope that the problem is a flaw in the current configuration of the server settings.
-
The seti-server aktivity is very binary!
0 = crash
1 = working
0 = crash
1 = working
If Moore's Law right, the frequency will be very fast in the next two years ...
-
fyi...you can upload wus at the moment...no wus down though...
-
fyi...you can upload wus at the moment...no wus down though...
Judging from the Cricket graphs, uploads have been going on all along......but no reporting or downloads. I have stranded downloads waiting on almost all rigs. Gotta get jocy back up....not sure if it will happen yet tonight or not until tomorrow.
-
uploads were down earlier in the week...boxes going dry (1 empty now, others not far off)...
the problem is going to be getting wus...everyones machines will hammer the servers...and there will be ghosts galore...
i can only see the ghost problem getting worse at the moment...
-
uploads were down earlier in the week...boxes going dry (1 empty now, others not far off)...
the problem is going to be getting wus...everyones machines will hammer the rervers...and there will be ghosts galore...
i can only see the ghost problem getting worse at the moment...
Probably.......
If server constipation makes ghosts, having jocelyn on duty can only make it worse.
We can only hope to hold out for a few more weeks until the masked man arrives with Tonto..........
-
Why not just detach for a month or two? Until the new HP boxes arrive. This is what I will do ... I'm no masochist ;)
Quoting Papa:
"I have set No New Tasks. ... So the next time you push that Update Button, ask yourself if you are going to be the straw that broke the camels back?"
http://setiathome.berkeley.edu/forum_thread.php?id=61843#1043500
-
This trend is self-destructing. There will be no harm to servers if there will be no participants. But there will be no data processing as well. In short - "no brain no pain" ;)
IMO not a way to go.
-
Why not just detach for a month or two? Until the new HP boxes arrive. This is what I will do ... I'm no masochist ;)
Quoting Papa:
"I have set No New Tasks. ... So the next time you push that Update Button, ask yourself if you are going to be the straw that broke the camels back?"
http://setiathome.berkele{/edu/forum_thread.php?id=61843#1043500
Pappa has been a bit too critical in the last few weeks........
Just let the boxes idle and keep sniffing.
Like the kitties do. Sniff, sniff, nothing there, sniff again in a few hours.
Sniff, sniff.........and sooner or later, there will be kibble.
Sniff. The kitties know.
Meoooooooooowww. Sniff....sniff.......LOL sniff sniff........
-
There will be no harm to servers if there will be no participants.
Yes, I have to admit while scanning over Papas posting this is what I was getting ("without you users we wouldn't have this problems").
I know this will piss many power crunchers off - but why not just giving out the number of work units the project can really[/k] handle?
Anyway ... maybe in the end this all is just a huge experiment in Crowd Psychology - and Self Adaptation ;)
P.S.
Shame that other, very-well funded, projects like for example Einstein@Home, don't allow users/developers to write optimized apps. Maybe setiquest.org well be an alternative ... one day.
-
Like the kitties do. Sniff, sniff, nothing there, sniff again in a few hours.
I have no problem with the kitties sniffing every few hours.
But have you seen what a cuda box will do while caching up? It sniffs about every twenty seconds. I suggest that might well be a problem (because a 'sniff' is an incredible amount for work for the database server), and - yes - it would be better in the short term while we are resource-constrained to voluntarily limit those hosts which are in the heavy-hitting phase of their task allocations.
-
P.S.
Shame that other, very-well funded, projects like for example Einstein@Home, don't allow users/developers to write optimized apps. Maybe setiquest.org well be an alternative ... one day.
Did you see my reply on the Einstein message board? The source is available - linked from the front page, even, and there are instructions for compilation. The only request - and it is that, a request - is that you using a version number of 0.99 or lower when reporting results from a self-compiled application.
The main reason Einstein hasn't spawned an optimisation community is that the apps are already optimised in-house, so everybody gets an optimised app, not just the self-selected few. But I'm sure there's still scope for more optimisation - if you think you can do better, I'm sure you will be welcomed with open arms. You might even get a consultancy out of it, as Akos Fekete did.
-
Like the kitties do. Sniff, sniff, nothing there, sniff again in a few hours.
I have no problem with the kitties sniffing every few hours.
But have you seen what a cuda box will do while caching up? It sniffs about every twenty seconds. I suggest that might well be a problem (because a 'sniff' is an incredible amount for work for the database server), and - yes - it would be better in the short term while we are resource-constrained to voluntarily limit those hosts which are in the heavy-hitting phase of their task allocations.
Richard.......I respect your opinions greatly.......
But this is just something that does not work for me.........
If the servers are working correctly, the 'sniffers' get their work or allocations quickly and are silenced accordingly. most Boinc versions do that. And then they sniff out more, when allowed.
The small percentage of folks like me that sniff out work all the time is such a small percentage of the user base that I refuse to accept that it has an impact.......
It's the large user base that runs out of work and Boinc sniffs..........because Boinc is TRAINED to sniff.
Even the new 20 hour backoff kitties will have to sniff once in a while, and it will add up. Eventually, they are ALL sniffing. Not just MY kitties.
My kitties are just trained to sniff more often.....and the few sniffs they give the servers don't bother them a bit, trust me. It's the mulit-sniffs from the multitudes that are the problem. And that cannot be avoided. Even with your buddy DA's 20 hour frickin' backoffs.
I will never load a Boinc client with such backoffs.........unless I have somebody recompile it for me without.
Verstai?
-
The only request - and it is that, a request - is that you using a version number of 0.99 or lower when reporting results from a self-compiled application.
Nice :) Thanks for the update.
The main reason Einstein hasn't spawned an optimisation community is that the apps are already optimised in-house, ...
AFAIK only FFT is done by the GPU at the moment - so the GPU is utilized only at about 20% (like Raistmers app in the good old days *g*). I am sure there is much space for optimization ...
-
P.S.
Shame that other, very-well funded, projects like for example Einstein@Home, don't allow users/developers to write optimized apps. Maybe setiquest.org well be an alternative ... one day.
Did you see my reply on the Einstein message board? The source is available - linked from the front page, even, and there are instructions for compilation. The only request - and it is that, a request - is that you using a version number of 0.99 or lower when reporting results from a self-compiled application.
The main reason Einstein hasn't spawned an optimisation community is that the apps are already optimised in-house, so everybody gets an optimised app, not just the self-selected few. But I'm sure there's still scope for more optimisation - if you think you can do better, I'm sure you will be welcomed with open arms. You might even get a consultancy out of it, as Akos Fekete did.
Anybody wanna stop opti's on Seti????????
OMFG.......if you think that is an answer to the project's problems.......
I better just stop now.......otherwise the kitties are just going to go berserk.
Grrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr.
-
OMFG.......if you think that is an answer to the project's problems.......
I better just stop now.......otherwise the kitties are just going to go berserk.
Grrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr.
Better stop drinking - and start reading. Properly. Nobody want's to stop optimized apps ... ::)
-
Now Now people. Keep it nice please, tensions are high enough with all the bone pointing going on about server load.
-
I will simply spit if anybody suggests again that limiting the big contributors to the project is the answer to it's problems........
You (expletive deleted for clarity) have muddied the issue for faaaaaaaar to long now.
The few hosts who have the capacity to make a few more work requests to Seti are such a small percentage of the total that they are swamped by the rest.......do you actually think that anything I could do on my end including shutting them all down would make the servers stay up??????????
Eric knows this.........that's why the limits are put in place each week.........it limits the total amount of work requested.........not by the big users, but by everybody.
And it is more the 'everybody' that overloads the servers than the big users......we are just more visible for you barbs..........
So stop jousting at us........
-
OMFG.......if you think that is an answer to the project's problems.......
I better just stop now.......otherwise the kitties are just going to go berserk.
Grrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr.
Better stop drinking - and start reading. Properly. Nobody want's to stop optimized apps ... ::)
Who said I was drinking........I'll tell you when I an f$$$$ing drinking.........not nice of you to presume.
-
... and I already have a migraine...
So Right first up:
Nobody suggested that only big users should be considerate of the current servers reduced capacities.
Nothing we can do about the crowd, that doesn't read forums etc. or doesn't even remember what that blue-yellow symbol in their taskbar is. And that is almost certainly the majority of the user base.
Those with eyes to read however might want to consider what they can collectively do to reduce server load.
Yes, Mark, we don't gain much if just the handfull of big ones take half a step back - it will only have an effect if everybody contributes. And of course NNT on an empty rig is not really an option [well unless you run several other projects and can switch to them to give SETI a breather]
What else? NNT and manually update once in a while. Set small cache until the worst has passed. Don't keep hitting the update button (especially not in the first half hour when everybody is homing in) I'm sure there's more tweaking options (i.e. spreading with backoffs) that can be used to 'take half a step back'.
As Richard has already pointed out, while the limits help to distribute work more evenly they increase serverload because the starved GPUs keep asking for more only to be told they already got some...
It would probably help to implement a 'you've reached a limit'-backoff, so there won't be essentially empty requests. Dripfeeding, while keeping the rigs at the limit, also doesn't help with serverload.
Effectively the 'report one, get one' is the same as if you were set to report after each task - and it makes a vast difference to serverload if you do 10x 1 or 1x 10.
So the best suggestion I can currently come up with is to intermittently set NNT: Wait till you have some work [well when the system is finally up to it] by letting boinc run it's usual backoff routine. Fill up a bit - maybe half a day worth, maybe to the limits - then go to NNT. After a few hours (i.e. while you still have some left, so you won't run dry but so that you will be asking for a reasonable amount) switch back on. fill up some, switch off. redo from start...
I've no idea how well that works, or if it is feasible for larger systems. It does require some babysitting, but it will help to reduce serverload. As to ghost creation I wouldn't think there's a different chance of a single request getting lost than of a bunch - statistically the single requests might result in less ghosts per system, but at the cost of increasing serverload.
So bottomline: If everybody could try to be a little less emotional and egoistic, we might be seeing some work before the new servers are in place instead of one crash after the other.
And as always, talking - sorry writing - to the wrong audience ::)
-
Filling cache then go to disabled network has another good side:
You will accumulate lot of results to scan for elepsed times. Then you will have very nice picture of host true performance (for beta apps, copying whole project directory before switch network ON again will provide good backup for offline checking tasks that can produce invalid results online.
So, switching ON, loading cache, then switching OFF network completely - is the way I run my ATI box many weeks already.
-
BTW, on my good old Athlon 64 I have 3 SETI MB tasks in download state now ... and absolutely idle CPU :(
Einstein backup refuse to ask for work. BOINC thinks that SETI tasks should be run instead (but they not downloaded and will not be downloaded any time soon).
Obviously it's BOINC bug. Anyone knows is it fixed already and in what BOINC version?
-
BTW, on my good old Athlon 64 I have 3 SETI MB tasks in download state now ...
Ahhh, so you're the one blowing up the servers. ::)
Obviously it's BOINC bug. Anyone knows is it fixed already and in what BOINC version?
Not sure which Boinc version it was introduced in, fairly recent I think.
[Edit:] http://boinc.berkeley.edu/trac/wiki/VersionHistory says 6.10.19-56 , so probably buggy I suppose. Will see if I can see anything obvious in my modded 6.10.58 this weekend.
[Later:] Check your debts, and also whether those 3 tasks in the download queue are blocking all requests for work (Shouldn't be :-\ any uploads ?).
-
BTW, on my good old Athlon 64 I have 3 SETI MB tasks in download state now ... and absolutely idle CPU :(
Einstein backup refuse to ask for work. BOINC thinks that SETI tasks should be run instead (but they not downloaded and will not be downloaded any time soon).
Obviously it's BOINC bug. Anyone knows is it fixed already and in what BOINC version?
If you haven't any SETI left, you could try suspending it altogether from the projects tab. Else suspending the no yet downloaded tasks from the tasks tab should work.
-
BTW, on my good old Athlon 64 I have 3 SETI MB tasks in download state now ... and absolutely idle CPU :(
Einstein backup refuse to ask for work. BOINC thinks that SETI tasks should be run instead (but they not downloaded and will not be downloaded any time soon).
Obviously it's BOINC bug. Anyone knows is it fixed already and in what BOINC version?
I reported it in early July, and David fixed it the same day - http://boinc.berkeley.edu/trac/changeset/21877
But unfortunately that was too late for any of the BOINC v6.10.xx line - the last (.58) was released five days earlier.
It should be in all the v6.11/12 test builds: v6.12.4 (yesterday) should have non-FIFO GPU scheduling too - but I need a second viabale CUDA project (apart from GPUGrid) to test that.
And as Carola says, suspending the individual SETI tasks that can't download released the logjam for me.
-
That's a good one line fix. I like those those kind :D. Any more I should look for ?
-
hehe, yesss :)
I look at this PC queue good if once per month or rare now, too busy with GPU-enabled once ;)
And her 3 not coming tasks definetely was that killed the Pappa's Camel ;)
-
BTW, on my good old Athlon 64 I have 3 SETI MB tasks in download state now ... and absolutely idle CPU :(
Einstein backup refuse to ask for work. BOINC thinks that SETI tasks should be run instead (but they not downloaded and will not be downloaded any time soon).
Obviously it's BOINC bug. Anyone knows is it fixed already and in what BOINC version?
If you haven't any SETI left, you could try suspending it altogether from the projects tab. Else suspending the no yet downloaded tasks from the tasks tab should work.
Yes, I did so. But this required manual intervention. And as I said already I look into that host only once per month...
-
That's a good one line fix. I like those those kind :D. Any more I should look for ?
How about this one: http://boinc.berkeley.edu/trac/changeset/22206 (http://boinc.berkeley.edu/trac/changeset/22206)
It's a response to a problem i reported in early August, and is incorporated in the 6.11 and 6.12 series,
Claggy
-
LoL, that's a pretty bad one actually. Will see what I can do for Mod12. [Edit:] Yes, reading it I remember that one now, I'll include that fix also.
[Done]
-
Looks like we are on our way again.
-
Did they managed to fix the FIFO bug for GPUs in 6.12.x?
-
Did they managed to fix the FIFO bug for GPUs in 6.12.x?
Yep, 6.12.4 has this fix:
•client: use project STD, rather than arrival time,
as the major criterion in choosing non-EDF GPU jobs.
GPU scheduling now respects resource share,
and as a result STD should no longer diverge.
I'm running 6.12.4 on my T8100 Laptop, but since it can only run one Cuda project, can't tell if it's fixed.
Claggy
-
I'm running 6.12.4 on my T8100 Laptop, but since it can only run one Cuda project, can't tell if it's fixed.
I've loaded v6.12.4 on my Fermi, and zeroed debts on the way in so everything starts level.
The good news: yes, task startup by debt is working. Starting from zero, it finished off the SETI task in progress, building up a small amount of debt. When that finished, it pulled a GPUGrid task from well down the page, and started that next.
The bad news: it's overly agressive on task switching. 8 minutes later (triggered by a SETI work allocation), it preempted the GPUGrid task, and started running SETI again. Later on, it preempted SETI, and went back to GPUGrid.
And I'm still getting 'leakage' into CPU STD for projects which should be ineligible.
So, test with caution: this isn't the end of the story. Credit hounds may wish to avoid this version - all that task switchinbg, especially given the heavy SETI start-up penalty, isn't going to be efficient.
-
I'm running 6.12.4 on my T8100 Laptop, but since it can only run one Cuda project, can't tell if it's fixed.
I've loaded v6.12.4 on my Fermi, and zeroed debts on the way in so everything starts level.
The good news: yes, task startup by debt is working. Starting from zero, it finished off the SETI task in progress, building up a small amount of debt. When that finished, it pulled a GPUGrid task from well down the page, and started that next.
The bad news: it's overly agressive on task switching. 8 minutes later (triggered by a SETI work allocation), it preempted the GPUGrid task, and started running SETI again. Later on, it preempted SETI, and went back to GPUGrid.
And I'm still getting 'leakage' into CPU STD for projects which should be ineligible.
So, test with caution: this isn't the end of the story. Credit hounds may wish to avoid this version - all that task switchinbg, especially given the heavy SETI start-up penalty, isn't going to be efficient.
I thought the task switching rate was a user configurable option.......switch between tasks every......
Does that not work here?
-
The bad news: it's overly agressive on task switching. 8 minutes later (triggered by a SETI work allocation), it preempted the GPUGrid task, and started running SETI again. Later on, it preempted SETI, and went back to GPUGrid.
And I'm still getting 'leakage' into CPU STD for projects which should be ineligible.
So, test with caution: this isn't the end of the story. Credit hounds may wish to avoid this version - all that task switchinbg, especially given the heavy SETI start-up penalty, isn't going to be efficient.
IMO not only "credit hounds" but all who take care about their host performance. GPU task switching should be at minimum even stronger than CPU task switching (any task switching is bad enough, but GPU is just worst).
Interesting, will this version do preemting of MW GPU tasks that don't checkpoint at all ?
I've seen foolish pre-emting of 3min-only ATI MW tasks in prev builds (for example, when another GPU task from another project un-suspended and BOINC decides to continue it immediately (and no, no high priority mode and far-far from deadline) ).
Very nature of GPU apps implies much bigger startup overhead, for different projects this overhead will be different of course, but in all cases it will be much bigger than corresponding CPU overhead. So switching GPU task in progress w/o absolute need is counterproductive.
-
I thought the task switching rate was a user configurable option.......switch between tasks every......
Does that not work here?
LoL, it works ~ in same extent as user-configurable project shares in all prev builds before "backup project" was implemented properly. In short, it almost doesn't work.
[
BTW, if BOINC would respect user settings more and switch to another task only after user-specified number of minutes (if no high-priority mode involved) it could complete GPU task and do switch on task boundary cause most GPU tasks (maybe except GPU grid with their long GPU tasks) will finish earlier than even default of 60 min.
]
-
Agree totally with you both - but this is an early, TEST build. I'll be filing a bug report, of course, with logs.
But I've no experience with MilkyWay - I'm not attached. Someone else had better do that one.
Let's see if we can get the bugs out of this one before they move on to the v6.13.xx line :P
-
Let's see if we can get the bugs out of this one before they move on to the v6.13.xx line :P
Hehe, yeah, bug-hunting always behind feature implementing. It's common problem, Windows itself is ill with the same disease.
-
LoL, it works ~ in same extent as user-configurable project shares in all prev builds before "backup project" was implemented properly. In short, it almost doesn't work.
No, for CUDA tasks it doesn't work - at all. It's a bug. I think the TSI test slipped out of (that bit of) the code along the way.
-
Oh dear, website is unreachable, scheduler requests fail, and cricket graph has taken a dive, :(
Claggy
-
Oh dear, website is unreachable, scheduler requests fail, and cricket graph has taken a dive, :(
But uploads are still working ;D
(for the time being, touch wood.)
-
Yup.......
Looks like jocelyn got tangled up again.
Unable to connect to database - please try again laterError: 1040Too many connections
-
Jocelyn is disabled. I can still get to the server status page but that's about it. Looks like uploads are getting through but not reporting. Oh well ! ::)
-
Unable to connect to database - please try again laterError: 1040Too many connections, is what I get, trying to login, will be tomorrow
I think.
The message is clear, overload!
-
The scheduling server, (Bane) is now showing as disabled too. Guess somebody is doing something. Hopefully they will get us going again before too long. http://setiathome.berkeley.edu/sah_status.html
-
Looks like somebody might have their fingers on the buttons.......
Almost everything is down now.
Last time jocelyn crashed, it took the better part of a day to resynch the drive tables though.
Depends on just what happened, I guess.
-
Forums just came back up......hope that's a good sign.
-
And we're down again. Wonder how long this time?
-
Well.....one thing or another lately.
Apparently it was thumper that messed things up this time.
And it looks like we are down until tomorrow morning.
See ya then......
Meow meow.
-
Thumper huh? Silly rabbit!!
Well, I've got plenty of work to keep me busy but I sure do miss the forums.
-
I just shut the Frozen 920 down.....ran itself dry.
I have varying amounts of work left on most of the rigs.
Should keep things warm enough until tomorrow.
-
Nearly out of work here too. One ran out of work yesterday, and this one is only still going because it has 2 AP units that it is crunching.
My team - GPU Users group - has offered a brand new server board with a couple of Xeon processors. We would purchase a rackmount case for it, and the RAM, leaving Seti to fit it with the HDD's. Depending on the spec required, there may be some drives in it for them too. This is on top of the 2 ordered servers that Mark drove the fund raisers for. (And I did make a donation to that as well). It wouldn't be as powerful as the ordered ones, but would make a good second line machine. Apart from the CPU's, this will be all brand new kit.
Unfortunately, despite repeated emails, we have had no response from the top brass at Seti. Trying to help with the current woes, but seemingly unable to.
Sometimes it makes it so hard to keep the faith.
Giz.
-
Unfortunately, despite repeated emails, we have had no response from the top brass at Seti. Trying to help with the current woes, but seemingly unable to.
Sometimes it makes it so hard to keep the faith.
Giz.
I will try to check into this a bit.......
-
Thanks Mark.
You seem to have a good line of communication with the guys at the moment, and it would be a shame to see this kit wasted.
Regards, Giz.
-
I have sent you an email on the subject.....
Must get some more sleep with the kitties now.
And hopefully our favorite project can get things moving a little bit in the morning.
Meow meow.
-
The first campus shuttle bus should arrive at the lab in about an hour. If Jeff's usual early arrival happens, things which couldn't be handled remotely will be possible. I just hope they get the server status page updating fairly promptly.
Joe
-
The first campus shuttle bus should arrive at the lab in about an hour. If Jeff's usual early arrival happens, things which couldn't be handled remotely will be possible. I just hope they get the server status page updating fairly promptly.
Joe
Well....
Parts of the server status page have updated, so some activity is taking place. But some of it is still iffy. Even though the upload server still shows as online, they are not working yet.
-
Giz, if memory serves correctly, I believe they are trying to avoid "whiteboxes" since there is no real support for them. If you spend US$10K on a server from a major brand, they provide support to keep you happy so that hopefully, you'll buy more from them. With whiteboxes, you're on your own, basically.
That was discussed in the one forum thread that had basically a wishlist, and one of the servers they wanted was pretty simple, but through a major brand, was more than five times the cost of going to newegg and getting all the parts, and the latter option was turned down.
Now maybe these days, with desperation at its peak, they might be willing to go with a whitebox, but if Oscar is going to be as good as we're all hoping, maybe not. We'll just have to wait and see.
-
Aside from from the Server issues, how about the BandWith, which is an isue, too.
UPDate to GigaBit Link, that also is necessary and has been Discussed many, many times
times before, I presume, the new SERVER, has to have 1, better 2 or more, gigabit
nic's, but also have manage the INcreased DataFlow per second.
(Which puts more strain on the Servers, then ever!)
Quite a change anyway, cause MODEM-ROUTER and everything between this end SERVER,
has to be UPDated, too. In most cases, there isn't much, between these 2, a (cat5) cabel.
(Twisted Pair)
Is there any news, about this, remenbered this?
I was called if I was interrested in VDSL and Fiber? Still expensive, but incredible fast!
AsynchronousDSL*, is the most used connection, I think, WiFi is gaining strong and has
many advantages, especially in house. Not suitable for longer distances then 1 Mile (Visible, but 0.5 mile is a more realistic distance)
* Has limitations, except ADSL 2 is fater then ADSL (1), but the speed, strongly depends on the Distance between 'Home' and 'Central' or a (MODEM and Amplifiers Central)
Sorry, I'm drifting (away)
On Topic.
Digital Subscriber Line, VDSL. (http://nl.wikipedia.org/wiki/VDSL)
VDSL. (http://en.wikipedia.org/wiki/Very-high-bitrate_digital_subscriber_line)
-
Upgrading to gigabit would in fact make data transfers go much faster, and my understanding is that the lab is already gigabit internally, including the router that is hooked up to the line that runs down the hill.. it's the equipment down on campus that needs to be upgraded, as well as the medium itself.
However, making a bigger pipe will only increase the strain on the servers due to allowing more connections to successfully reach through the saturated link. It's one of those Catch 22 situations, but as others and I have said so far.. we'll have to wait and see just how good Oscar really is before theorizing new things to change/upgrade/replace.
-
UPDated, my 'previous post' and added the remark on more load on the Down Load SERVER, the servers end should be able to handle that added traffic!
Seti isn't the only project with the most people, but also in Serving Hosts
-
Okay, now we're testing my memory here. They just ran one fiber cable up the hill but it is for the entire SSL, hence SETI only gets 100MB out of it. The memory check comes in that as I remember, Hurricane Electric is already giving SETI a Gig for the price of 100GBs. The problem was in that the college IT department wanted something like $30,000 to run the dedicated fiber up the hill to them. My guess is that at the time they ran the new fiber for the lab SETI couldn't afford the price of the fiber so that it could be run up at the same time. Again, if I remember correctly, the college won't allow an outside company to run anything on campus, it must go through the college IT, even though Hurricane would probably do it for SETI at a greatly reduced price.
Somebody please correct me if I'm wrong here.
-
now i know nothing about the server side, but anyone other then me starting to wonder if this is a software problem?
-
now i know nothing about how the server side, but anyone other then me starting to wonder if this is a software problem?
I did....I brought it up and Eric was gonna ask Jeff to update reload mysql, which apparently they tried on jocelyn with no improvement. Don't know if they tried it on mork or thumper.
Eric was going to peruse the mysql bug reports to see if there were any clues there...
Not sure why they were not able to get things running today.....I'll probably hear a bit more in the morning.
-
not just the database, but in the basic programming/logic of the whole thing. shouldn't the results out in the field be approaching zero?!?
-
not just the database, but in the basic programming/logic of the whole thing. shouldn't the results out in the field be approaching zero?!?
No?
Some 38 hours ago there were about 4 million results in progress, with at least half a million issued in the last reporting/DL window.
That number can only drop if results are reported or time out (and only with running validator/transitioner).
If the processing was running with no UL/DL you would see the decay with timeout [asuming 'out in the field' does not include tasks waiting for resend due to timeout] - this would reach zero after about 6 weeks
If everything was up but no new WU were being handed out you would see a fast reduction from quick hosts followed by slow reduction while slower hosts hand their work back in - possibly asymptotic to the number of tasks not really being crunched. In six week that fallout times out, gets reissued and you get another asymptotic decay to a smaller positive number. Which times out another 6 weeks later and get reiessued. Thankfully we are dealing with natural numbers (and finite resends) or that number couldn't reach zero in your life time ;) With a six week deadline you'd be looking at some 42 weeks of reissues.
Look at the bright side - if a child was concieved the day SETI stopped handing out new work and just waited till everything was back, that child would see the light of day before the last WU was crunched.
-
so you think the last work unit will need to be reissued 9 times before credit is given!!!???!!!
-
so you think the last work unit will need to be reissued 9 times before credit is given!!!???!!!
Yes, at least that many times!
Have a look at the Einstein server status page (http://einstein.phys.uwm.edu/server_status.html). The current run (S5GC1) has been ongoing for the last 172 days, but there are still 206 tasks (64 workunits) from the previous S5GCE run still outstanding. Einstein tasks have 14 day deadlines, but even allowing for a typical 6-day delay between task (re-)generation and issue to a volunteer, 172 days is still at least 8 time-out/resend cycles.
I think those of us who pride ourselves on efficient and timely processing of the work projects ask us to compute for them, forget that there is a far larger number of non-obsessives who receive, but do not return, that work. Given the numbers involved, it really does take that long to eliminate the asymptotic tail.
-
Just posted on the home page.......a very lengthy process.
Would be surprised if they come back up today.
Status, Oct 26, 2010
We are now doing very careful (and time consuming) table checks on the entire database. 26 Oct 2010 18:00:02 UTC
-
You're doing better than me then, Mark.
I can only get the 'Project is down for maintenance' page on the website...
-
Giz,
They posted that on the home page. I can also still get to the server status page but I don't know how up to date it is. Master data base shows as 49 hours ago and some of the others are showing 50 hours ago. I agree with Mark, it is very doubtful that they will be up today. I'm not really sure we will even see them tomorrow. I've still got 26 GPU tasks to finish but a lot of them are shorties so I will be completely dry tonight. I'm already out of CPU tasks but I figure I wont reschedule any as I would just run out that much sooner. I'll just let my computer rest until they send me something. I hope they start off with a ghost resend, I've got about 30 of them I'd like to get rid of.
-
Hi Perryjay,
I've had nothing since the weekend. Waiting patiently for anything to happen. I flushed the DNS cache, and can still only get the Project is down page. No home page or server status page.
I guess with Mork being so unreliable, Jocelyn will take 2-3 times as long to process the database table checks.
Thanks for the info anyway.
I guess I'll have to go back and kill a few more mutants in 'Shadow Of Chernobyl', and shut it down when I go to bed - I'll save a bit of Electricity instead.
Giz.
-
Must be mad, I'm replying to myself :D
Finally got the home page up and the server status page, not that they mean much at the moment.
Just keep waitin' guys...
-
Well......
At the end of their day today, the boyz will still waiting for some database recovery operations to complete.
Untill they do, a complete backup cannot be done. Hopefully the database finishes sorting overnight, and they can backup and restart tomorrow.
But, you know how things have been going lately.
Wish 'em luck, my friends.
Meow meow.
-
Kinda funny side note........
Have shut down all but a few of the slower rigs that still have a bit of work to crunch.
Kinda cool and quiet in the crunching den.
One of my kitties, Tigger II, has been slowly pacing back and forth listening to some of the sounds the remaining rigs are making. Sounds which are covered up by the general din when everything is going full bore.
I always told you the kitties help me keep an eye on things........LOL.
Meow meow.
-
i doubt both servers (hardware) are bad and am glad they are starting to at the database. my estimated estimated up time is Friday, down time is Saturday; hope i'm wrong!
the Powers that be (http://en.wikipedia.org/wiki/The_powers_that_be_%28phrase%29) said they ran three or so loops of memtest, which i don't find as thorough as vistas memory checker. now don't get me wrong, if memtest finds a error it is a error; it just don't push as hard as vista's memory checker and vista's memory checker will find problems when memtest don't. how about running pi or something, the funny thing is i usually use seti to do my stability tests:P
-
They're using Linux distros.. not windows. That's why they can't use Vista's memory checker..or super pi and the like. Granted, they could probably use wine, but realistically, memtest does well enough.
However, as I just experience with one of my rigs two weeks ago.. running memtest was not able to diagnose a problem I was having. It would run memtest for hours and hours and be just fine, but as soon as you put a load on the CPU..instant power-off. Five bad caps along the side of the CPU socket. $4 ebay auction and a soldering iron and problem solved.
-
Didn't realise that there were still motherboards out there suffering from 'BAD CAPS' still. Last board I had fail with that was an old Gigabyte one, back when I was running an Athlon XP CPU.
Unfortunately I didn't have the equipment at home to replace them, and it gave me an excuse to upgrade the system.
Glad you got yours sorted though! :)
[edit] They probably wouldn't go to the trouble to install/setup wine, just to run vista memory checker, even if they have a copy of vista lying around, which they probably don't! I've never had a problem with memtest, but I normally use memtest+ now, which can be run off a bootable cd/usb stick. I've never tried to run it off a bootable memstick though. [/edit]
-
A front page update, for those suffering withdrawal symptoms:
Status, Oct 27, 2010
The database is being backed up now. When that has completed, we can bring the web site and forums up.
27 Oct 2010 19:32:23 UTC
-
Didn't realise that there were still motherboards out there suffering from 'BAD CAPS' still. Last board I had fail with that was an old Gigabyte one, back when I was running an Athlon XP CPU.
Well it's not a ridiculously old board.. only two generations old. It's a K8, but it's a socket 754 board. I got it almost four years ago just before newegg quit selling s754 boards. Needed an office workstation, so $300 in parts built an entire machine.
Newer boards still have caps, but they're the small compact ones, and the high quality Japanese caps that basically don't ever go bad, so "bad cap" replacement is likely nowhere near common anymore.
-
Forums online now :)
-
I like the thread about posts getting lost.... that has no posts in it :D
-
Forums online now :)
Was that 5 mins uptime? All seems down now...
-
Dang it, I must have blinked!! I missed it.
-
Dang it, I must have blinked!! I missed it.
Quick it's up again ;D
-
Got it this time. Jeff Cobb posted over in the Tech news Forum
Just a quick note. Obviously, jocelyn is up. Mork is recovering.
The purchase orders for both oscar and the new mork went out late today or will go out early tomorrow. It takes a while for these things to work their way through the purchasing pipeline.
We decided to go with HP for these machines. They gave us a very good deal. We are getting two identical (oscar class) machines. I'll post the specs in another note. We hope to have them on hand in about 2 weeks.
At this point, we are discussing what we will do between now and when the new servers are on line.
-
I hope you can all appreciate the irony that people are actually asking not to get work ;D
-
Well it's not a ridiculously old board.. only two generations old. It's a K8, but it's a socket 754 board. I got it almost four years ago just before newegg quit selling s754 boards. Needed an office workstation, so $300 in parts built an entire machine.
Newer boards still have caps, but they're the small compact ones, and the high quality Japanese caps that basically don't ever go bad, so "bad cap" replacement is likely nowhere near common anymore.
Yes, the newer 'solid state' caps that they're generally called, seem to be a lot more reliable. Not that they are 'solid state'. Never had one of those 754's, went from socket 462 to 939, then on to AM2/AM3.
-
I hope you can all appreciate the irony that people are actually asking not to get work ;D
Well I still want work! It's getting cold in here! ;D
-
Einstein@home (CPU), MilkyWay (ATI) and Collatz (NV) are keeping my room warm ;)
[Actually I have lot of ATI work for SETI still, just can't report it. Hope this time "hour of MW" will not come ;) ]
-
Einstein@home (CPU), MilkyWay (ATI) and Collatz (NV) are keeping my room warm ;)
[Actually I have lot of ATI work for SETI still, just can't report it. Hope this time "hour of MW" will not come ;) ]
I just can't bring myself to sign up for another project, seeing as Seti was all I ever wanted to do... Don't really want to sign up to a project that will get abandoned as soon as Seti sorts itself out!
Just got 2 AP left to report, everything else finally went through.
Down to playing games now, but that just doesn't make the computer as warm as 6 seti wu's! ;D
Not to mention one rig shut down for days now! I might even have to put the heating on soon! ;D
-
No shame in attaching to a project for a short time, and detaching again afterwards - just please allow any work in progress to flush through (or at least report the work as aborted) before you leave. Probably best not to choose CPDN, then ;D
In the meantime,
I added a new config option, <dont_send_jobs>.
When set, the scheduler doesn't send jobs,
and the feeder doesn't enumerate them.
Set this during periods when you want to let your server
catch up handling completed jobs, and not issue new ones.
I wonder which project might be feeling the need to let its servers "catch up handling completed jobs", LOL :P
-
No shame in attaching to a project for a short time, and detaching again afterwards - just please allow any work in progress to flush through (or at least report the work as aborted) before you leave. Probably best not to choose CPDN, then ;D
In the meantime,
I added a new config option, <dont_send_jobs>.
When set, the scheduler doesn't send jobs,
and the feeder doesn't enumerate them.
Set this during periods when you want to let your server
catch up handling completed jobs, and not issue new ones.
I wonder which project might be feeling the need to let its servers "catch up handling completed jobs", LOL :P
I was wondering that too, i hope resend lost tasks still works with the option enabled,
Claggy
Edit: I've been busy doing CPDN, Einstein and Collatz on my E8500/9800GTX+/HD5770 machine, Astropulse and CPDN on my T8100 Laptop, and Einstein on my Atom Netbook,
i always tend to keep a couple of Suspended CPDN Wu's on the E8500 and T8100 for extended server outages, with Resourse share set to 15% of Seti's.
-
Jeff has posted this:
We've decided to keep the project down until the new servers are up and running and the databases migrated to them.
The forums will stay up.
The back end and the upload server will stay up until we clear the outstanding results.
The time line we are looking at is about one month - two weeks for the servers to arrive and another two to get them going. We'll see as time goes on whether or not that's too aggressive.
The down time will be used for preparing the databases for migration. For example, on the science side, we can finally finish a big merge of the spike table and drop the old spike table. This will make the database smaller and easier to migrate.
We will also use the time for science processing and analysis.
More later...
Claggy
-
I added a new config option, <dont_send_jobs>.
When set, the scheduler doesn't send jobs,
and the feeder doesn't enumerate them.
Set this during periods when you want to let your server
catch up handling completed jobs, and not issue new ones.
... i hope resend lost tasks still works with the option enabled,
Claggy
Sorry, no such luck.
Joe
-
I decided to let my 2 Quads run some Aqua for a month or so, the iMac will just get by with Collatz.
-
I just cut the Frozen One over to MW.......makes me a little sick having to do so, but otherwise I would have to start the furnace to keep the kitties warm.
-
BTW, I think the best name for the new server would be, lest it be 'Oscar II', should be Phoenix.
A rising from the ashes, so it were.
-
Phoenix? I thought you didn't like it when I suggested it over on the NC forum! ;D Well, this time I decided to go with Einstein. Would be nice to know what I was doing though. Strange that their "GPU" app takes a whole CPU at the same time as the GPU. Seems a waste.
-
I think you all should know this...........
http://www.youtube.com/watch?v=IHH8bfPhusM&feature=related
I have been wronged...........they put in MY place, for sure.
But I sure as hell am not ready to 'make nice'.
-
BTW, I think the best name for the new server would be, lest it be 'Oscar II', should be Phoenix.
A rising from the ashes, so it were.
How about Felix and Oscar instead.
-
One rig walkin' , walkin, wrong.
Call me Phoenix, baby...........or I am gone.
-
Seti's Database is acting up, couldn't get the Donations thread to change to having been read,
then noticed the title at the bottom of the thread has 'Database Error' at the end of the thread's title too,
couldn't post about it eithier,
and now the project's Down for Maintenance,
Claggy
Edit: and Einstein's down too.
-
Edit: and Einstein's down too.
Einstein pre-announced that they were going to be relocating servers within their Milwaukee facility sometime this afternoon - not expected to be a long job.
-
Einstein should get good bost these days :) This my secondary project cause it does most close things to SETI.
-
Edit: and Einstein's down too.
Einstein pre-announced that they were going to be relocating servers within their Milwaukee facility sometime this afternoon - not expected to be a long job.
I did read about that, just didn't have the date and time fixed in my head, so forgot about it,
Claggy
-
And they're back, quick as advertised.
-
I think SETI community should quickly finish both current Einstein's searches before new servers arrive to be free to new work :)
-
Only 1,2M of tasks and ~20% of beams remains - on one teeth for SETI community ;D
[Pity that Einstein's GPU support on infant stage]
-
So what are you waiting for? ;)
http://einsteinathome.org/forum_thread.php?id=8487
-
LoL.
Maybe they communicate more actively with third party optimizers than SETI staff (still awaiting answer from Eric for last GPU AP code/binary offer and proposal to test them on beta), but to do optimization w/o direct possibility to test it looks like too big pain in the .ss to go that way ;)
-
But then again, read
http://einstein.phys.uwm.edu/forum_thread.php?id=8525
-
Yeah, Einstein's CPU code should be pretty efficient one. I remember that amazing time when Akos released his binary patched (!) build. AFAIK he had no source code, just reverse engeneered binary. True Mage degree IMHO :) So I complain exclusively about GPU infancy, not CPU ;)
-
SETI's gone again - but this time, it's taken BOINC, and the whole of the SSL with it :o
Or, more likely, some Campus event has taken down all three together. With any luck, the servers are still plodding away behind their unintended firewall.
-
SETI's gone again - but this time, it's taken BOINC, and the whole of the SSL with it :o
Or, more likely, some Campus event has taken down all three together. With any luck, the servers are still plodding away behind their unintended firewall.
Well, cricket is alive and kicking - and my last task uploaded and reported 10 minutes ago, so at least that part of the servers is alive.
edit: still showing as ready to report, though I'm getting scheduler request completed, project down for maintanace messages ::)
-
The last 3 hops on the traceroutes show:
13 212 13 128.32.0.58 g3-19.inr-201-sut.Berkeley.EDU
14 213 1 128.32.255.110 g6-1.inr-230-spr.Berkeley.EDU
15 * * timed out
16 231 18 128.32.18.189 boinc.SSL.berkeley.edu
host reached
13 211 17 128.32.0.58 g3-19.inr-201-sut.Berkeley.EDU
14 212 1 128.32.255.110 g6-1.inr-230-spr.Berkeley.EDU
15 * * timed out
16 236 24 128.32.18.150 setiathome.SSL.berkeley.edu
host reached
In the inr-230 interfaces, there's "gigabitethernet6_1: 128.32.255.110: Tier2 side A to inr-201", for which the daily octets graph is at http://fragment1.berkeley.edu/newcricket/grapher.cgi?target=%2Frouter-interfaces%2Finr-230%2Fgigabitethernet6_1;ranges=d;view=Octets. That clearly shows the down time, but many of the other inr-230 interfaces did too. I'm tentatively thinking that graph may be mostly traffic on the new fiber optic link to SSL.
Joe
-
Downloads have been reenabled (splitters are down, so previously split work and resends only)
See also Matt's post (http://setiathome.berkeley.edu/forum_thread.php?id=61943&nowrap=true#1046035)
And of course we are maxed out on bandwidth ;D Expect some http errors before you get hold of those shiny WUs.
-
yep, seti is spitting out a little kibble, get it while it's hot!
-
Meow meow!!!
Da kitties got dem some Seti kibbles!!!
Purr purr purr.
-
Set all the rigs to NNG......(no new ghosts).
A hard to access button behind the buttons......
Which the kitties are unashamedly abusing right now.........LOL.
-
You may not know the movie quote to which I am referring.......but.
I can fill you in if necessary.....
"Excessive button pushing by the kitties in search of kibble..........has been approved....LOL.
-
Eric, Matt, Jeff......................
Thanks for the burst of kibble.........it has kept the kitty cradle warm during the coldest burst of this season......
Meow meow.
-
I've acquired a bunch of Einstein@Home work, and then disabled the fetching of more work from Einstein@Home.
The machines are now slowly emptying the pool of Einstein work.
Does anyone here have any idea when Seti will start sending out work again? Or should I fetch another load of Einstein work to keep the iron warm?
Regards, Patrick.
-
With idle SETI users Einstein's work supply will be empty soon so hurry up to stock some work from Einstein ;D
-
I've acquired a bunch of Einstein@Home work, and then disabled the fetching of more work from Einstein@Home.
The machines are now slowly emptying the pool of Einstein work.
Does anyone here have any idea when Seti will start sending out work again? Or should I fetch another load of Einstein work to keep the iron warm?
Regards, Patrick.
The general chatter coming from Seti forums seems to point to the end of November before they start distributing work units.
-
I've acquired a bunch of Einstein@Home work, and then disabled the fetching of more work from Einstein@Home.
The machines are now slowly emptying the pool of Einstein work.
Does anyone here have any idea when Seti will start sending out work again? Or should I fetch another load of Einstein work to keep the iron warm?
Regards, Patrick.
The general chatter coming from Seti forums seems to point to the end of November before they start distributing work units.
OK. I've re-enabled Einstein work fetching on the machines which can handle it, and I'll disable it when the first Seti units are distributed again (or I read about the servers coming up and work going out here ;)).
Thanks for the info!
Regards, Patrick.
-
Both SETI and BETA are unreachable, the download servers do respond though.
Looks like mysql crashed again.
-
Both SETI and BETA are unreachable, the download servers do respond though.
Looks like mysql crashed again.
Something certainly failed, but one of my hosts was assigned a few reissues about 25 minutes after your post so the database was active then. Another host couldn't get the master file two hours earlier and again very recently, so it may just be whatever server has the non-data-dependent web duties.
Joe
-
Both SETI and BETA are unreachable, the download servers do respond though.
Looks like mysql crashed again.
Something certainly failed, but one of my hosts was assigned a few reissues about 25 minutes after your post so the database was active then. Another host couldn't get the master file two hours earlier and again very recently, so it may just be whatever server has the non-data-dependent web duties.
Joe
And still not a single sign, maybe swapping hardware already............?
-
And still not a single sign, maybe swapping hardware already............?
That was my first thought when everything disappeared. My money is on rampant bit rot though :)
-
And still not a single sign, maybe swapping hardware already............?
That was my first thought when everything disappeared. My money is on rampant bit rot though :)
Got to agree with purple rabbit on that one. I really doubt anyone was in swapping things around on a Sunday night. Though whatever broke just might get replaced today. Things keep breaking this way we may get the new servers running a lot sooner than planned out of pure desperation.
-
New message from Matt over in tech news.. http://setiathome.berkeley.edu/forum_thread.php?id=62094 Thumper and Marvin decided to take some time off. Mork has now been turned off, maybe for good and Carolyn is now the only MySQL replica. They are getting Oscar ready but it looks like it will be after Thanksgiving before it gets going. Won't be long now!
-
OK. I've re-enabled Einstein work fetching on the machines which can handle it, and I'll disable it when the first Seti units are distributed again (or I read about the servers coming up and work going out here ;)).
Thanks for the info!
Regards, Patrick.
You might be able to automate the process. Try setting Einstein resource share to zero, SETI to any number but zero. BOINC says the result is Einstein will only request work if projects set to non-zero run out of work. Like SETI right now. I'm using zero on PrimeGrid. Seems to work. Both SETI and MW@H had/have trouble getting WU out. PrimeGrid is filling in.
-
OK. I've re-enabled Einstein work fetching on the machines which can handle it, and I'll disable it when the first Seti units are distributed again (or I read about the servers coming up and work going out here ;)).
Thanks for the info!
Regards, Patrick.
You might be able to automate the process. Try setting Einstein resource share to zero, SETI to any number but zero. BOINC says the result is Einstein will only request work if projects set to non-zero run out of work. Like SETI right now. I'm using zero on PrimeGrid. Seems to work. Both SETI and MW@H had/have trouble getting WU out. PrimeGrid is filling in.
Interesting, didn't know about that. I have set the workshare distribution to 100 for seti, and 10 for Einstein, so Einstein isn't getting too much work, even if seti does nothing.
But this solution certainly sounds very nifty.
Regards, Patrick.
-
Do we have usual database backup now or something goes wrong with splitters enabling?
-
It seems the project comes to life :)
-
I'm sure everybody already noticed this a long time ago, but my main rig just downloaded a boatload of WUs for Seti@Home...
Seems the project's back!?
Regards, Patrick.
-
OH YEAH!!!
Back in a big way with the new servers Oscar and Carolyn on duty.
Matt is playing with the settings getting things ramped up to full speed.
So far, so good.
Meowza!
-
But the question remains, will these two beasties be sufficient for guys in the lab to drop the 3-day outages?
-
only time will tell ... personally i think the 3 day outage was their biggest mistake yet
-
only time will tell ... personally i think the 3 day outage was their biggest mistake yet
I don't think it was a mistake........
Some folks have been hammering them for a long time to show and tell some real scientific results from the work we have all been donating for years.
And with the hardware they had to work with, along with the problems they were having just keeping daily opps going because of it, they simply needed to extend the downtime to have enough computer resources available to do some of the scientific work they wanted to accomplish.
I am hoping when the new servers are fully configured and tweaked, this may no longer be necessary.
Meow.
-
only time will tell ... personally i think the 3 day outage was their biggest mistake yet
I would agree! I believe the three day outage caused too much server/network congestion and was causing work units to get lost and as a result the size of the database got out of had. This was probably made worse since they don't know what they are doing and can’t afford to hire a real engineer.
The servers which were replaced tested to be fine, they should configure them to run nitpicker 24/7.
I’m sure you all will come running to their defense and flame me, but this is what I believe. I've also reduced my cache to 5 days.
-
and again
-
From the front page, Good Idea:
Weekly Outage early this week
Since we're recovering from a failed disk over the weekend, we might as well stop the project and do our usual Tuesday outage chores today. 13 Dec 2010 19:49:12 UTC
Claggy
-
From the front page, Good Idea:
Weekly Outage early this week
Since we're recovering from a failed disk over the weekend, we might as well stop the project and do our usual Tuesday outage chores today. 13 Dec 2010 19:49:12 UTC
Claggy
Wonderful idea!!!
And good to know, as that message is no longer on the home page...
The automated Tuesday outage message is there instead.
Happy to know this, as the rigs are struggling with the traffic to download work to get them going at full speed again.
Thank for posting it.
Meow!!!
-
Something's come unglued at Seti. Or with the campus network.
Completely unreachable.
-
Strike that last post....something went on earlier last night.
I just rebooted 1 rig that had crashed, and it is connecting.
Must be some kind of DNS issue again.
-
Strike that last post....something went on earlier last night.
I just rebooted 1 rig that had crashed, and it is connecting.
Must be some kind of DNS issue again.
Both Seti and Boinc's Sites were unreachable here too, but scheduler requests were still going through,
Claggy
-
Strike that last post....something went on earlier last night.
I just rebooted 1 rig that had crashed, and it is connecting.
Must be some kind of DNS issue again.
Both Seti and Boinc's Sites were unreachable here too, but scheduler requests were still going through,
Claggy
I had to reboot 3 rigs to get them to connect to Seti.....the other 5 were not affected.
One of the ones I had to reboot was my daily driver, which could not connect to Seti with Boinc or connect to the website with the browser. But when I checked the Cricket graph and saw that they came back up with full bandwidth after that, I knew something was afoot.
Very strange.
-
I would like tho here from thos thos wither do or dont. like yto here forom mer.
I have some life chanhgeing decision to m ake based on this point.....
LI ca't wait for answers forever.
-
What'd I miss ? Victor & the KittyMan in a brawl ? I'd pay to see that :o
-
What'd I miss ? Victor & the KittyMan in a brawl ? I'd pay to see that :o
Not my brawl.......
I just finally decided to call him out.
Many others have hinted at it......I just got tiired of it.
I bought him a kitty.......out of pity,
Now he goes on and on and on about the computer toys he is gonna buy, all whilst posting about how the 'system' is gonna buy him a house.
Sorry, but it made me so sick I had to call him on it.
As Doc Holliday said.......
I may be deranged, but I just have to call.........
I called a spade a spade.
If some think I am nasty for doing so,,,,,,,,,,fuck them.
I called the hand. It's up to Vic to lay down.
I think he is going to be very quiet for a while.......just my gamble.
But you know........poker is an honest game.
And I have learned enough in life that I no longer tolerate cheaters.
-
All of a sudden, everything went dead at setiland. Let's hope it's just a minor glitch. :o
-
Yeah, I don't think it's a server problem. Cricket is showing 230kbit/sec. Seems like the entire network went down. So much for a trouble-free weekend. So close..
-
Cleaning crew must have come in early. Can't even get to the main page. Maybe the powers that be at Berkeley figured we were having too much fun! :D
-
It is back up once again.
-
Uploads are now going in. Haven't got any new downloads yet but that could be just everyone hitting them again. Guess the guys finished their coffee and got to work. ;D
-
Well, now I'm getting the down for maintainance message. Wonder how long this is going to last?
-
I would gyess that they would use this down opportunity to do the weekly maintanence backups, so I expect to see it back up in a few hours. They may slide this into a three day outage, depending on what their needs are.
Steve
-
I dunno what the devil is going on.....
Up and down, up and down.
Not a normal outage scenario.
Connectivity to the forums has been very sporadic as well.
-
Matt just posted in tech news.
I'll repost it here because you can get to the forums one minute and not the next......
Ah, the few days back at the lab between Xmas and New Year's... The university assumes nobody works at this time, so the buses aren't running, and so I have to drive into the lab. But of course they are still handing out parking tickets to people without regular parking permits (like myself, who rarely drive to the lab). So I gotta park elsewhere. Anyway...
Except for bruno (the upload server) having fits we were pretty much running smoothly all weekend. However bruno is also the main BOINC back-end administrative server for the SETI@home/Astropulse project, so when it has fits, everything kinda gums up. We couldn't get into bruno remotely (full process table?) so it waited until this morning when Jeff got in and rebooted it.
There was some cleanup after that, and we seemed out of the woods, but we're still having these mysql issues where the database enters these long periods of flushing pages to disk. We all agree that this is largely due to the increased demand (after all the long/short outages over the past two months, and perhaps a bout of short runners). Increased demand means more deltas, which in turn means more fragmented pages. We have these weekly outages to defragment the database, but given the load it's like 3-4 weeks of fragmentation within one week. We're thinking the outage tomorrow will largely fix this, but we're still tuning other stuff in the meantime. We already gave mysql access to more memory, but Bob predicted this wouldn't help, and he was right. He's trying other stuff now.
So the plan is to hang on have just the normal outage tomorrow, then be up (as best we can) the rest of the week and throughout the New Year's Eve weekend. Then in the new year we can really start squeezing these new servers and see what they got.
Oh yeah - I turned off the "resend lost results" for now to reduce the load on mysql. This is temporary.
- Matt
-
I thought this was normal for SETI, have you not been paying attention for the last few months !!!! ;D
(Ducks and runs for cover ....)
I dunno what the devil is going on.....
Up and down, up and down.
Not a normal outage scenario.
Connectivity to the forums has been very sporadic as well.
-
Can you reach berkeley?
I can upload and contact the scheduler, but I can neither download nor view the site.
-
everything seems fine for me - but I've not need to download any new tasks can reach the site though
-
All OK here too. I'm just breaking in a new host, and it passed the magic 10 validations for CPU tasks this morning - so estimates are now 2 hours instead of 20 hours ;D
In the last hour (15:00 - 16:00), it requested and received new work the maximum possible 12 times. Only problem today has been congestion when both MB shorties and AP are in the feeder mix - some download retries, but all cleared in the end.
You're not trying to reach the US via Egypt, are you, sunu? ;)
-
I'm not having any problems here in Florida. Seems like uploads and downloads are doing fine. No problem reaching the webpages either.
-
You're not trying to reach the US via Egypt, are you, sunu? ;)
Even though we shouldn't be laughing :D :D
Everything seems fine now.
-
Anybody know whazzup with Seti?
According to the Cricket graphs, all outbound traffic came to a dead stop a couple of hours ago.
Can't reach setiathome.berkeley.edu at all.
But what's strange is it seems that uploads are still going through.
???
-
I think this was a planned outage at the university, I think Matt said something about it in his Monday post.
-
I know what you are referring to, but I thought that was supposed to happen Wednesday morning.
Maybe I am mistaken, or it was rescheduled.
If true, hopefully it will be back up soon, as Matt's post indicated.
Meow.
-
Looking at the weekly graph, I only see the Tuesday outage. Wednesday looks perfectly normal.
-
Looking at the weekly graph, I only see the Tuesday outage. Wednesday looks perfectly normal.
You are right, I see that now.
It was probably postponed.
They'll be up later tonight, or tomorrow morning if the boyz have to reset some things to get them going again.
Kinda handy that the uploads were not affected so far, should lessen the backlog when things get back on track.
-
That planned Wednesday outage was supposed to last about 1.5 hours, this one is getting close to 3 hours. And although the downloads fell quickly they didn't just cut off so my guess is the download servers are also working but are idle because nobody can request work from the Scheduler. And the BOINC home page, etc. are working, so I think the hope this is a reschedule of the Wednesday outage is ruled out.
Jeff's an early riser, if it can be fixed by a remote kick I'd expect either someone will do so soon or Jeff will when he gets up. Then the first campus shuttle bus arrives at the lab before 8 am...
Joe
-
That planned Wednesday outage was supposed to last about 1.5 hours, this one is getting close to 3 hours. And although the downloads fell quickly they didn't just cut off so my guess is the download servers are also working but are idle because nobody can request work from the Scheduler. And the BOINC home page, etc. are working, so I think the hope this is a reschedule of the Wednesday outage is ruled out.
Jeff's an early riser, if it can be fixed by a remote kick I'd expect either someone will do so soon or Jeff will when he gets up. Then the first campus shuttle bus arrives at the lab before 8 am...
Joe
Well, we wait 'till the 'morrow then, eh?
Meow meow.
-
That planned Wednesday outage was supposed to last about 1.5 hours, this one is getting close to 3 hours. And although the downloads fell quickly they didn't just cut off so my guess is the download servers are also working but are idle because nobody can request work from the Scheduler...
Joe
When I first looked this morning, every machine was backed up on uploads, so I assumed they were included in the general breakage. But the upload line on Cricket is normal, so I retried - and all uploads went through at the first attempt. Must have been a transient for that server, at least.
But the drop-off in downloads was much, much sharper than it was when the scheduler was switched off for Tuesday's maintenance, and I have two downloads stuck, not responding to retries. So I suspect some common factor has taken out the website, scheduler and download server - but it's server, not comms, related because of the uploads working.
Correction - the scheduler's OK, those uploads have just reported (though no new tasks - "no tasks available"). So it's just the website and downloads broken (plus an unknown amount of back-end, of course)
-
I've got 3 Astropulse resends since the outage started @ 08:41:46 and 08:46:57, but can't download, my reported work results increases my total credit,
Seti Beta uploads aren't going through eithier, althrough the scheduler is working there,
Claggy
-
And now I've got a MB task, scheduled but not downloaded. Just one, and likewise a resend. People were moaning about the splitters being off last night, and at one stage I saw, not only were the splitters red (ran out of tapes), but most of the back end was orange ('men at work'). Maybe they forget to clip something back on when they'd finished? (or only half clipped it, since it lasted until 8 pm before falling off ;D)
-
No problems with uploading tasks, last was uploaded 15 minutes ago, probably the only thing that works OK, since I've a boat-load of MB 6.03 and CUDA,
also AP WU's, downloaded just 30 minutes ahead of the UPLoads!
Only I can't reach the SETI Main Page, so Data Driven Webpages are offline, too. Hope it can be fixed if some one has the time to come in and look
what's going on . Or isn't :-\
-
... Only I can't reach the SETI Main Page, so Data Driven Webpages are offline, too. Hope it can be fixed if some one has the time to come in and look
what's going on . Or isn't :-\
SSL site-wide planned outage for cable work, beyond s@h's control. Supposed to be just an hour or so outage. Then again, for external work, who knows?...
Cheers,
Martin
-
SSL site-wide planned outage for cable work, beyond s@h's control. Supposed to be just an hour or so outage. Then again, for external work, who knows?...
Then why would uploads be working? And the whole of BOINC - message boards and Wiki? That's at SSL, too.
No, you'll have to look for an alternative explanation, Martin - sorry :'(
-
New info at BOINC / projects (http://boinc.berkeley.edu/dev/forum_thread.php?id=6318&nowrap=true#36721):
Seti Down, likely through the weekend. Gowron d/l fileserver has failed.
IIRC, Gowron is the big Network Attached Storage device holding all six million workunit files for the work in progress. And also IIRC, it's a bugger to reboot and resync after a failure. "Through the weekend" sounds realistic.
-
Appologies for being off topic.......................
Anyone have a place I can download BoincView v1.4.2???
-
Appologies for being off topic.......................
Anyone have a place I can download BoincView v1.4.2???
I still have the original .zip download - PM me an email address, and I'll send it over. Unless the powers-that-be are prepared to sanction a 936 KB attachment here?
-
Seti germany hold a copy at http://www.seti-germany.de/wiki/BoincView - google is your friend ;) [especially if it's set to Germany ;D]
-
Appologies for being off topic.......................
Anyone have a place I can download BoincView v1.4.2???
I found it on some site called the core. Rehosting on my site.
http://www.arkayn.us/seti/boincview_1_4_2.zip
-
rac is dropping quick even though i've been able to upload and report, seems something is wrong with the rig which grants credit.
-
Oh my.........
Still down for the count.
Well, it's nigh on 8:30am in Berkeley. One of the boyz should be ready for coffee any time now.
-
Got it.............................
Many thanks!!
-
Something must still be amiss........
It's well on 11.00am Berkeley time.......
Things are still crashed.
Wonder what's up this time, bunny........
Meow.
-
I've just had it confirmed that the news I passed on from Gary in message 35527 (http://lunatics.kwsn.net/1-discussion-forum/seti-is-down-again.msg35527.html#msg35527) is official - please feel free to pass on to other BOINC-related boards.
Or more specifically, Gowron has "a" bad disk, and the project "may" be down all weekend.
-
Sigh........with hundreds of HD's online.........this might always be a risk.
Even wth RAID.
Gonna go on.
-
Web pages are live !
-
Anyone else wondering when the "down for maintenance" message is going to go back to "for a few hours, maybe all day" instead of "3-day maintenance downtime"? It is just cosmetic and all, and us regulars know the 3-day outages are pretty much over, but the forum irregulars most likely don't know that.
-
Looking at the cricket graphs I noticed we have been down for five days! To bad I reduced my cache to five days after the great outage...
-
Oh, and also the webpages don't work now. They did a few hours ago, but no worky now.
-
They just came back up in the last half hour.....nothing much new on the Cricket graphs yet, however.
-
FYI: Gowron has crashed again. But a friendly soul rebooted the webserver, so people can complain about it :P
-
All down again this morning....
Some of the boyz must be in the lab trying to bring things back to life.
Best of luck to 'em.
Meow.
-
Somebody knows, what has happened? :-\
-
:-[ Here the answer... :(
"A disk failed on our workunit storage server, so downloads won't be working until the system recovers (quite probably not until Monday)."
-
Yeah, that was the situation over the weekend......
They tried to bring it back up since then, and suffered another drive failure.
They are now in their 3rd attempt at bringing things back up, if I am not mistaken.
Bad run of Karma.
-
boards are back
-
boards are back
Looks like they've messed up a style sheet, and failed to load an icon - but who's counting, when they work? ;D
-
looks like the project is back, there is a spike of activity on cricket graphs.
-
Download servers are running, but the scheduler is not.
I am working on getting the work that was assigned last week downloaded now.
-
Cricket just took a nose-dive from maxed out to the floor. And the stylesheet issues are back again, it seems. Something is moody.
[edit: okay, so the stylesheet issues are some visual upgrades that are in progress, one element at a time.]
-
i was still able to dl my queued up work even thou the cricket graphs fell to the floor!!!
-
i was still able to dl my queued up work even thou the cricket graphs fell to the floor!!!
The part that fell to the floor was the downloads.......
Uploads are still being accepted.
Gosh, I do hope they are able to get things sorted today.
Matt is supposed to be back in the lab. I think he really needed the day off.
Wish 'em luck, folks.
-
Of course there is still one major bug in BOINC, if you have work assigned to the machine but not downloaded because of server issues, the manager will not request work from your backup projects because it thinks there is plenty of work onboard already.
I ended up up putting the queued up DL on suspend so I could get something for the machine to do.
My CUDA machine now has SETI CUDA work but the CPU is working on Aqua once again, the ATI machine is doing Aqua and Collatz.
-
Which version of BOINC are you using? I thought we'd cured that one - though the cure may be in one of the v6.12.xx line, still stuck in testing.
I'll try and find when it ought to have come in - unfortunately I've already downloaded all my backlog, so can't test again just yet ;D
-
6.12.12
And yes I am on the list.
I should also specify that my backups are all using the "0" share setting.
-
Well, it seems that Seti might be down until early next week........
Apperently Gowron is having major problems with it's RAID setup.
Matt says they are getting wonderful support from Overland, but the fix may require updating the ancient OS on the box and rebuilding the array from scratch. They fly in that ointment is that it requires transferring all of the data (2 terrabytes worth) to another server first (the newly updated Thumper), doing the upgrade, and then transferring the data all back to Gowron. A lengthy process.
They are winning battles, one at a time, but it seems lately that when one battle is won, another comes forth to take it's place.
I think Matt (and all the boyz in the lab are doing a wonderful job.
Something about the 'patience of Job' comes to mind.
Meow.
-
UP ... DOWN ... UP ... DOWN ... UP ... DOWN ... DOWN ... DOWN ... DOWN ....
WUHAAA
Thats pure horror
-
I really don't know what happens now, but this downtime looks like quite serious.
hope will be up during w-end.
-
I really don't know what happens now, but this downtime looks like quite serious.
Boards are up, you can read for yourself. http://setiathome.berkeley.edu/forum_thread.php?id=63138&nowrap=true#1075931
-
I really don't know what happens now, but this downtime looks like quite serious.
Boards are up, you can read for yourself. http://setiathome.berkeley.edu/forum_thread.php?id=63138&nowrap=true#1075931
thanks for the update, but at sea with a GSM connection, don´t have to much chance to read and go where I do like ;)
-
Hmmmmmmm....
Entire Berk campus down?
Can't reach anything, including the Cricket graphs.
-
Can't reach SETI , too, made a post in a new 'thread' :o . Shouldn't have done so :-\
Well, I deleted it ::)
I made 1 post at SETI and when trying to load another page, NC Forum, it appeared to be totally down, even the frontpage is
UNreachable. A POWER out?
Last UPLoad was on 12:01 UTC, after that only a message :Project communications failed and a few other messages.
-
Dunno.....
As I said, I can't even reach the Cricket graphs, which though on Berkeley campus, are not tied at all to the Seti servers.
So it's affecting more than just Seti.
-
Dunno.....
As I said, I can't even reach the Cricket graphs, which though on Berkeley campus, are not tied at all to the Seti servers.
So it's affecting more than just Seti.
I can't reach the Boinc forum eithier, or Cricket, Seti, or Seti Beta on my Laptop or Netbook,
but when i switch to my Desktop, which is on a different ISP, i can, posting in Panic Thread,
Claggy
Edit: changed the Cricket image to a later image, as things start to recover.
-
You dog!!!
The kitties can't reach a thing.....I can change my DNS, but not my ISP. SBC Global/ATT.
No problem reaching anything else I have tried other than Berk.
-
Somebody just put the plug back in the socket..........
Connecting right up now.
-
My Laptop managed to reach Berkeley again too,
and a few minutes later the Netbook managed to reach Berkeley, (same ISP as the Laptop, but different Router)
Claggy
-
[As of 26 Mar 2011 | 14:20:08 UTC] From the SETI Berkeley Frontpage.
Someone gave 'it' a kick, as I can reach SETI now. ;D
-
Well, looks like SETI is gone again. Uploads seem to get through but can't report and can't reach the webpages this morning.
-
Well, looks like SETI is gone again. Uploads seem to get through but can't report and can't reach the webpages this morning.
Reporting (well, all scheduler functions) died about 02:00 UTC, but the website was OK until mid-morning. Now occasionally there, but very, very slow. I have a server status page from about 20 minutes ago showing 62 master database queries a second - it's usually 400-plus, even when things are quiet.
-
Well, looks like SETI is gone again. Uploads seem to get through but can't report and can't reach the webpages this morning.
Reporting went down last night, but the forums were still flowing OK.
This morning, they are difficult to reach at best. Can connect once in a slow while.
Betting that one of the servers locked up, and is slowly tying everything else up that interconnects with it.
Suspect that a bit of rebooting action in a few hours when the boyz get back in the lab may set things straight again.
Unless they are forced to go through another one of those dang raid resynchs.
-
Well, looks like SETI is gone again. Uploads seem to get through but can't report and can't reach the webpages this morning.
webpages are still there, but verrrry veeeerrrrrry slow. If I get something then with partial graphics. paticience and persistence helps :D
-
as per stats in update projects, doesn't work by now.
-
Hope we're running again when the guys gets into the lab :) (And not Monday or something ;D )
-
Everthing appeares to be down, Scheduling-Server was already down :-\
I do hope someone can fix this, before the weekends begins......If somebody is going to the lab?
-
Something's coming to life.......Cricket graph shows a bounce.
Reporting going through.
Website sill unreachable....should be back soon, I suspect.
-
And...........we're baaaaaaaaaaaaaaaaaack!!!
Meow!
-
Oh, sigh.........
Anybody know what's up with the Seti servers this time?
Uploads and reporting completed work seem to be online, but no downloads or webpages.
Hmmmmm...
-
Forums are back....still no downloads, even though the status page is updating and showing nothing odd.
'Nuther network oddity?
-
The grapevine says 'Gowron failed a drive and is hung. It will probably be 3-4 days to re-sync the RAID array, so no work until then.'
-
I woudn't say it's actually down, but when I try to read a forum thread I get a nice page that is empty after the thread title ;D Oh and the (mod) buttons. and you can still post to a thread, and then you see the posts you are replying to :)
-
I woudn't say it's actually down, but when I try to read a forum thread I get a nice page that is empty after the thread title ;D Oh and the (mod) buttons. and you can still post to a thread, and then you see the posts you are replying to :)
OK......
Glad to know it's not just me.....hope it's nothing too serious.
-
That happened a couple of weeks ago.
-
Looks like SETI beta validators are offline. Any news on this?
-
Looks like SETI beta validators are offline. Any news on this?
Posted by Eric himself on this very board:
http://lunatics.kwsn.net/4-linux/setihome-v7-r1000-vs-r1018.msg38863.html#msg38863
Sorry, that's in the restricted development area, but to summarise for the general readership -
Eric, with the help of Joe Segur and Urs Echternacht of this parish, has been working on making the Beta/V7 apps take advantage of the new Sandybridge/AVX technology. There were bugs in the first attempt (unsurprising - that's computer programming for you). So the early test results returned by the AVX test hosts will be bound to fail validation. Eric thinks that the bug is now fixed, but it's now the weekend. He wants to catch his breath, get the AVX app sorted out and deployed on Monday, and deal in a considerate way with the test hosts/users which returned invalid results through no fault of their own.
-
Four workunits wasted:
http://setiathome.berkeley.edu/workunit.php?wuid=758099617
http://setiathome.berkeley.edu/workunit.php?wuid=751042293
http://setiathome.berkeley.edu/workunit.php?wuid=751042281
http://setiathome.berkeley.edu/workunit.php?wuid=751042325
Edit: Well, from invalids, they validated :)
-
Is it just me or is the whole campus down? All of a sudden I can't reach the message boards or the cricket graph. Very strange.
-
Website is reachable. Uploads stopped around 7:00 UTC.
-
Okay, whatever it was is gone now. I'm used to the message boards going down now and then but it worried me when the cricket graph was out too. Everything seems to be working again now.
-
Downloads still worked.
I didn´t notice my host reported at 11:39 AM UTC.
But validators are off also.
-
No download works for me since a day, a lot of starting downloads hunging aroun and no progress since 12 hours.
Although the Graphs for gigabitethernet2_3 (http://fragment1.berkeley.edu/newcricket/grapher.cgi?target=/router-interfaces/inr-250/gigabitethernet2_3&ranges=d%3Aw&view=Octets) shows 95%
-
Works fine here.
Downloaded 100 APs in 2 days.
-
Works fine here.
Downloaded 100 APs in 2 days.
So it's you that s getting them all then? ;D
I'm only getting one AP every four or five attempts
-
Works fine here.
Downloaded 100 APs in 2 days.
So it's you that s getting them all then? ;D
I'm only getting one AP every four or five attempts
I´m sorry, i´m a bad boy i know. ;D
-
This time it looks like it went completely down. No websites reachable, no connecting to servers, no pings succeed. It is gone! (from here) How long this time , who knows ?
-
Yes, website is down. Uploads working though.
-
Yes, website is down. Uploads working though.
Not from here, but if you say that it works for you then there is hope that "they" are aware of the problem.
-
I can contact the upload server, but both download servers and the sites are timing out.
The BOINC site it up though.
-
Same here.
Upöoads and reports are working.
-
I can upload and report to Seti Main, and can report to Seti Beta, but can't upload to Beta,
The status page on Seti Main is stuck at: [As of 10 Jul 2011 | 22:10:08 UTC] (But that might be cached somewhere)
Claggy
-
I can upload and report to Seti Main, and can report to Seti Beta, but can't upload to Beta,
The status page on Seti Main is stuck at: [As of 10 Jul 2011 | 22:10:08 UTC] (But that might be cached somewhere)
Claggy
The near legendary instability of the Seti servers is all part of the white knuckle ride !!! It's amazing how the guys at Seti keep pulling rabbits out of the hat given the shoestring budget they operate on.
-
I can contact the upload server, but both download servers and the sites are timing out.
The BOINC site it up though.
Same here. Hope that guys will fix it soon, after the morning cup of cofee.
-
As of about 4:30 am Berkeley time the Hurricane Electric link went totally dead. Cross your fingers and hope it will be easily restored.
Edit: That was a short one, back in operation in about half an hour.
Joe
-
I'm seeing some strange behavior on all BOINC projects. My PC's are all busy, message queue is reporting server contact, but no results are shown on the individual sites. Is this because of the "new credit" imbroglio? anyone else affected?
I'm processing SETI, Constellation, MW, Collatz.
Thanks Art
-
I'm seeing some strange behavior on all BOINC projects. My PC's are all busy, message queue is reporting server contact, but no results are shown on the individual sites. Is this because of the "new credit" imbroglio? anyone else affected?
I'm processing SETI, Constellation, MW, Collatz.
Thanks Art
No probs here.
It sure has nothing to do with CreditNew.
-
Well SETI has database problems.
-
Uploads are going well, but just not able to report 'em.
-
Unable to connect to database - please try again laterError: 2013Lost connection to MySQL server at 'reading initial communication packet', system error: 111
Whatever that means.... :D .... it's not dead anymore.
-
Well, I guess the boys are in. The message has changed to project down for maintenance.
-
Has Berkeley fallen off the internet map? I can't even reach the Cricket graphs....
-
Has Berkeley fallen off the internet map? I can't even reach the Cricket graphs....
Nope Just you.
-
can't connect too
-
Well, something's gone away between here and there. Not my connection. I can browse just about anything else, but none of my rigs can connect to Seti.
-
My computers cant connect either.
-
Loading for me, even Cricket.
But then I am fairly close.
-
Boinc says "can't resolve host name".....so something's afoot.
And it just ties Boinc in a knot while it is trying. On XP or 7.
-
.18 is still down, but both .13 and the upload server are accessible to me.
-
.18 is still down, but both .13 and the upload server are accessible to me.
I am unable to reach the servers, the forums, or the Cricket graph.
-
It all came back about 9 minutes ago........all clear now.
-
From Matt Lebofsky
Posted: 14 Feb 2012 | 22:10:15 UTC
We hit a storage limitation in our science database. Everything is okay but we will remain down, at least overnight, to run various tests.
-
Good to See Matt is back, nobody else tells us anything !!
From Matt Lebofsky
Posted: 14 Feb 2012 | 22:10:15 UTC
We hit a storage limitation in our science database. Everything is okay but we will remain down, at least overnight, to run various tests.
-
Looks like the space sciences lab is down once again.
-
Yep.
-
I'm not much of a limit supporter, but I wonder whether the sudden opening of flood gates was very wise? ::)
-
;D It's Diablo 3 fans began to connect to all available servers because of hopelessness and put SETI down.. ;D :D ;)
-
Looks like the space sciences lab is down once again.
Agreed, all of it including BOINC and other non-SETI research projects. Maybe a power outage for the building...
Joe
-
I thought they fired that cleaning guy! :)
-
Definitely unscheduled.
http://ucbsystems.org/category/active/
-
Got some authoritative news.
It is a short at a substation causing a power outage to the hill area where the Space Sciences lab is located. Traffic signals are out on the hill, all the homes in that area have no power.
That was posted as 'current' about half an hour ago. Since the cricket graph flatlined well over 12 hours ago, they obviously need to do a lot more than just reset a breaker. Even once power is available, there'll be a lot of checking to do as the servers spin up again. Don't expect much SETI action today.
-
Thanks, Richard, for that bit of news.
And thanks for that link, arkayn. First time I saw that one.
-
I got it from barry over on my site.
-
I got it from barry over on my site.
Ooops, misunderstood what you'd got. Forget I posted that.
-
With all those raccoons hanging around the Korpela estate, you'd think they would set up some little treadmills and get them to generate some juice for the lab....
-
I hope somebody didn't try to steal the wire from the active substation. This guy in the article was unlucky.
The Article.(different substation in CA.)
News Article Link (http://abclocal.go.com/kabc/story?section=news/local/inland_empire&id=8649657)
Kevin
-
and where did he think he'd be able to sell that wire had he not been burned beyond recognition.
Plans should have a beginning middle and end. You should have considered the end at the beginning. Now he's near death, and most likely will not make a complete recovery. If he does he's another waste of time headed into a court room.
BTW San Bernadino is near Los Angeles so it is unlikely this had anything to do with the Berkeley incident
-
Maybe the wrong place to say this ;) but I miss the SETI forum :)
-
Update: Wednesday, May 16, 2012 4:00 pm: Service desk has been informed that the new ETA for restoration of service is Thursday afternoon.
I think this is ridiculous service for PGE, almost 48 hours to fix this in non-critical weather.
-
Well then. I guess that is a reasonable explanation for what's going on. This morning on both of my machines, I suspended network comms. Yay for full caches before issues arise.
-
It is also not listed on the PGE outage map, makes you wonder if they think it is important.
http://pge.com/myhome/customerservice/outages/
-
Plans should have a beginning middle and end. You should have considered the end at the beginning. Now he's near death, and most likely will not make a complete recovery. If he does he's another waste of time headed into a court room.
what a terrible way to die. and people poor lighter fluid on dogs and burn then, i guess this is payback to humans...
Would-Be Copper Thief ‘Extremely Grave’
After He’s Shocked And Catches On Fire
May 5, 2012
SAN BERNARDINO (CBS) — A would-be copper thief was gravely injured early Saturday morning when he attempted to steal copper wiring from a live Edison vault, according to authorities.
Witnesses told CBS2 and KCAL9 reporter Melissa Maynarich that the man could be heard screaming in agony.
Jeff Grammatico was across the street from the substation when he said, “I heard a boom.” He added, “It made me jump from one side of the room to the other.”
He ran to the fence and saw a man burning alive. “It was scary. I was trying to cut the gate to get in. It was horrifying to watch him burn.”
Grammatico says he ran back to his home to get a fire extinguisher while another passerby called 911. “He kept standing and screaming, screaming, screaming. And I couldn’t get in. I had a fire extinguisher. But he wouldn’t
come over. But he started to roll.”
Officials say the suspect is “extremely grave” and that it’s likely he was burned by 33,000 volts of electricity.
The incident occurred on the corner of Rialto and Stoddard about 2 a.m.
Southern California Edison said about 14,000 customers lost power. Service was fully restored by Saturday afternoon.
The man is described as a man in his 20s. Police have not released the man’s name. Witnesses said he was so badly burned there was no way for him to be identified. JR Velasquez said it was hard to witness the horror of the man burning. “It was something I really never seen n my life. Never happened to any animal or anybody.”
The man has not yet been charged with a crime. Police are trying to locate his family.
Would-Be Copper Thief ‘Extremely Grave’ After He’s Shocked And Catches On Fire « CBS Los Angeles
this thread should be moved to it's own sub forum with the same title...
-
It is also not listed on the PGE outage map, makes you wonder if they think it is important.
http://pge.com/myhome/customerservice/outages/
I noticed that this morning.....wondered why it was not listed.
And yes, I think it's kinda crazy that they could not have patched in a another feed while they are trying to fix the original problem.
-
;D It's Diablo 3 fans began to connect to all available servers because of hopelessness and put SETI down.. ;D :D ;)
What? Another Error 37? :P
(*&&^ %@#!@& (^#$@ %#$^*
-
It is also not listed on the PGE outage map, makes you wonder if they think it is important.
http://pge.com/myhome/customerservice/outages/
There's some news on the UCB website now:
http://newscenter.berkeley.edu/2012/05/16/power-outage-darkens-lawrence-hall-rest-of-campus-hill-area/
That suggests that the fault may be in a private line owned by the Campus, which would explain why it doesn't appear on the PG&E map.
-
And thank you for that new bit of information, Richard.
If campus personnel are involved in finding the fault, you could well be correct.
-
Once Seti does come back up, there will be much congestion of the bandwidth of course.
My concern, what with all the server tweaks and changes as of late, is whether there is going to be a problem with reporting large numbers of completed WUs.
In the past this has sometimes resulted in HTTP errors because the server could not cope with the large reporting lists.
There is a work around involving a CC config entry I believe, that limits the amount of work reported in a scheduler request. I think at one time, Seti also had to change some server settings to fix the problem.
We shall see.......
But be prepared for this bug to possibly raise it's ugly little head.
-
Right now, both of my machines are configured to report 50 tasks at a time, that usually works just fine.
I kind of hope they turn on the upload server first and let us do a mass upload before they turn on the scheduler and finally the download servers.
-
Right now, both of my machines are configured to report 50 tasks at a time, that usually works just fine.
I kind of hope they turn on the upload server first and let us do a mass upload before they turn on the scheduler and finally the download servers.
Please refresh our memory.
If it is needed, what was that tweak again?
-
Right now, both of my machines are configured to report 50 tasks at a time, that usually works just fine.
I kind of hope they turn on the upload server first and let us do a mass upload before they turn on the scheduler and finally the download servers.
Please refresh our memory.
If it is needed, what was that tweak again?
Just use the following in your cc_config.xml (for Jason's mod 6.10.58 and Boinc 6.12.x and later):
<cc_config>
<log_flags>
</log_flags>
<options>
<max_tasks_reported>50</max_tasks_reported>
</options>
</cc_config>
Claggy
-
Thanks....I remember now.
I had used it once in the past as a work around, but don't think I left it active.
Will wait to see what happens, and add it again if required.
Meow!
-
I know the guys in the lab have a lot of other things to worry about, but has anyone suggested that they use something like Twitter to fill us all in on this type of catastrophic outage?
Referencing the Diablo3 issues, even Blizzard was using Twitter as a means of telling everyone when/how long they would be down as it was faster than their own forums.
-
Definitely unscheduled.
http://ucbsystems.org/category/active/
There's an update now:
Update: Thursday, May 17, 2012 11:30 am: Service desk has been informed that it is still not known when full power will be restored to these locations. However, partial network connectivity to SSL has been restored using generated power.
Claggy
-
Not much point in having a network if there isn't a live server on the other end of it ;D (and if they were my servers, I don't think I'd try to run them off a genny)
Still, the AirBears will be happy...
-
Not much point in having a network if there isn't a live server on the other end of it ;D (and if they were my servers, I don't think I'd try to run them off a genny)
Still, the AirBears will be happy...
Agreed,
Brad has posted this at the GPUUG: Update on SETI @ Home Outage (http://www.gpuug.org/content/update-seti-home-outage)
Here's a quick update on the outage for interested parties:
A 15kV underground power line heading from the campus power substation to the hill that houses the SETI at Home lab shorted out around 2030 on Tuesday night. PG&E, the local power company, has spent the last two days crawling the line from manhole to manhole looking for the problem. Apparently while looking for the first short, they managed to cause another short in the line.
PG&E has suggested power should be back working this (Thursday) afternoon however understandably this may not happen on schedule. Please also keep in mind that it may take some time for the lab to get back into full swing due to this unexpected shutdown.
Thanks go to Dr. Eric Korpela for the above information and for taking the time to fill us in.
Warmly,
Brad Deauman.
I've started the donations on the SETI@Home - Green Bank Data Transport Package and the SETI@Home - HDD Transport Cases, feel free to donate too,
Claggy
-
After the power is back on, I'm willing to bet that there will be at least 15 threads started in Q&A and NC because people couldn't
1) return work
2) upload work
3) get new work
4) I started using BOINC but nothing is happening.
To those future people I'll say this...Really?
-
Oh, and here's another good thought for everyone to think about, too. IF it comes back up this afternoon/evening (Thursday), somebody will have to go into the lab after-hours and bring everything back up and make sure it is all coming up gracefully.
Or.. wait until the morning to bring everything back up, spend most of the day doing integrity checks, and then two minutes after they walk out the door on a Friday afternoon.. well we all know that routine.
It might be Monday afternoon before things are working semi-smoothly, and then it's time for the Tuesday maintenance only a few hours later.
-
After the power is back on, I'm willing to bet that there will be at least 15 threads started in Q&A and NC because people couldn't
1) return work
2) upload work
3) get new work
4) I started using BOINC but nothing is happening.
To those future people I'll say this...Really?
I won't take that Bet...Beginning to look like it will be Friday or Later. :P
-
Thursday next week, 5/24/2012 at 4pm pacific time.
scooby doo where are you!
-
Just wondering. Does the GPUUG has enough in the kitty for a Looooong extension lead ? ::)
T.A.
-
then it's time for the Tuesday maintenance only a few hours later.
Seeing as the crash happened on Tuesday night will they need one ? I suppose it all depends on whether the UPS's lasted long enough for the system to have a soft shutdown.
Just use the following in your cc_config.xml (for Jason's mod 6.10.58 and Boinc 6.12.x and later):
Where can I find the modified 6.10.58 ?
T.A.
-
Cricket is on again.
There is some hope.
http://fragment1.berkeley.edu/newcricket/grapher.cgi?target=%2Frouter-interfaces%2Finr-250%2Fgigabitethernet2_3;ranges=d;view=Octets
-
What you are seeing there Mike is bits per second instead of Mbits, if you go out to weekly it is barren.
http://fragment1.berkeley.edu/newcricket/grapher.cgi?target=%2Frouter-interfaces%2Finr-250%2Fgigabitethernet2_3;view=Octets;ranges=w
-
There's another update: http://ucbsystems.org/category/active/unscheduled-outage/
Update: Thursday, May 17, 2012 11:45 pm: IST Service Desk has been informed that the power has now been restored.
I'm bit confused about the time stamp tho. Did they use EDT coz right now it is 21:58 at Berkeley? ::)
-
Tomorrow morning they should start getting the project up then.
-
Tomorrow morning they should start getting the project up then.
More news:
Eric, Matt and Jeff are all meeting at the lab at 7am tomorrow morning to power everything back on.
(that'll be 14:00 UTC) How long it all takes will depend on what collateral damage they find when they start work on it, but I suspect an emergency beer shipment around lunchtime would be appreciated.
-
Tomorrow morning they should start getting the project up then.
More news:
Eric, Matt and Jeff are all meeting at the lab at 7am tomorrow morning to power everything back on.
(that'll be 14:00 UTC) How long it all takes will depend on what collateral damage they find when they start work on it, but I suspect an emergency beer shipment around lunchtime would be appreciated.
And what a motley looking crew that would be at that time of the morning in Berkeley, eh?
LOL.
-
What you are seeing there Mike is bits per second instead of Mbits, if you go out to weekly it is barren.
http://fragment1.berkeley.edu/newcricket/grapher.cgi?target=%2Frouter-interfaces%2Finr-250%2Fgigabitethernet2_3;view=Octets;ranges=w
OOoops, it was 6 AM so i probably wasn´t awake.
-
I was beginning to think it was time to get backup projects rolling. I'm getting mighty low on work. With the glut that is sure to occur the second the servers are back on it might not be a bad idea to get some backups rolling anyway
-
Well, it's 7AM. Wonder which one overslept and isn't there yet?
-
Upload traffic has commenced.
Let's hope there's not too much trouble bringing all the rest of it back up from cold iron.
-
Upload traffic has commenced.
Slow going through, Seti and Boinc websites are up now, but forums are down for maintenance,
Claggy
-
So long its working, i´m fine.
-
Message Boards are up
Claggy
-
This is the first Friday that Matt has worked in the SSL in how long?
-
Right now, both of my machines are configured to report 50 tasks at a time, that usually works just fine.
I kind of hope they turn on the upload server first and let us do a mass upload before they turn on the scheduler and finally the download servers.
Please refresh our memory.
If it is needed, what was that tweak again?
Just use the following in your cc_config.xml (for Jason's mod 6.10.58 and Boinc 6.12.x and later):
<cc_config>
<log_flags>
</log_flags>
<options>
<max_tasks_reported>50</max_tasks_reported>
</options>
</cc_config>
Claggy
Yup, had to go there.
Trying to report 1800 to 2400 tasks just resulted in HTTP internal server errors.
Set it at 100, and all 3 top rigs are now reporting their completed work in more manageable packets.
I'll just leave it in place and active....no reason not to.
-
looks like upload server is full or down. time to move this thread back to the top where it is always!!!
-
It's baaaack! Things are going and coming as normal? .
-
From the front page at SETI@Home...
Planned Power Outage
Repairs are necessary after the short circuit that caused power failures in several labs here on campus two weeks ago. These repairs require a planned power outage, which starts this afternoon (Tuesday 22:00 UTC). We'll be back on line Thursday morning (roughly Thursday 17:00 UTC). Our websites and data servers will be offline throughout this outage. 29 May 2012 | 17:11:12 UTC
-
Thursday morning??? OUCH!!!
-
Owie.....
But thanks for the info, arkayn.
I had not seen that notice.
Better than if the servers had crashed, I guess.
Meow.
-
This time, the BOINC servers and website will be able to stay up at least part of the time. Their outage (times in PDT) was posted as
Due to a planned power outage, the BOINC web site will be down tomorrow (Wed 30 May) from:
3:45 AM to 11:00 AM
5:00 PM to 8:00 AM the next day.
Two reasons for the difference:
1) SETI is a complicated beast with lots of interconnected servers. It's best if they're powered down carefully (and in the right order) while the staff are on-site, and powered back on again the same way.
2) "Two big temporary generators got moved into the SSL parking lot today. The lab has enough back-up generator power for Isaac [the BOINC server] and for the lab routers and switches, but not enough to keep seti@home up."
-
so the servers will be back on line FRIDAY!!!
-
so the servers will be back on line FRIDAY!!!
It depends what time-zone you're in, and what your operational hours are. Here in the UK, I'm expecting them to be available for six hours on Thursday evening - and since I crunch at home, that'll be useful. If I crunched in an office, it would indeed be Friday morning before I saw them online - but that would be because of my timetable, not theirs.
Most Americans will see them online by lunchtime Thursday, subject to Murphy's Law.
-
Here on the US East Coast we are three hours ahead of Berkeley so we will be looking for SETI to come back online sometime after 1400 (2PM) tomorrow afternoon.
I'm out of CPU work already but should have enough GPU work to last me through tonight if I'm lucky. Don't think I will make it all the way to tomorrow afternoon though. Oh well, my poor little computer needs a rest anyway. :)
-
I still have enough of my cache left to last three more days on the main rig (I ask for a 10-day cache, but it was sitting on about 5-6 at the time). Single-core MB-only machine has about 24 hours left (2.5-day cache). Before it went down, I needed ~1.6M seconds of work to top the main off, but AP was in short supply, so I didn't get all that I needed. I'll be alright.
-
My guess from reading between the lines of the various official statements is that the forecast time for full power restoration is before 8 am Berkeley time. I think just possibly with good luck the project may be up before 17:00 UTC (10 am Berkeley time).
Joe
-
That 'll be 19:00 h., in the Netherlands (UTC+2), I do hope .
-
And we are back online!!!
Hope the servers can handle the ruckus.
Full meow ahead!
-
As of 13:50 or 11:50 UTC, SETI appears to be down :-\
-
As of 13:50 or 11:50 UTC, SETI appears to be down :-\
The SETI project went offline round about then, but within the last 30 minutes the last remaining data traffic on Cricket has disappeared, and the BOINC domain has gone down as well. That's more like the signs we were getting during the power outages.....
Going off to watch the footie instead. ;)
-
Going off to watch the footie instead. ;)
Go England :)
EDIT: 3 secs later England scores!
-
Got word that it is indeed another power outage.
Duration and cause unknown at present.
-
Apparently, the outage is limited to the building the Seti lab is in, and not directly related to the high voltage lines that shorted previously.
-
But it's up again, I just checked :D
-
Seti website has crashed......no connect.
The forums were getting really, really laggy. And now nothing.
I sent Eric an email shortly before the crash, don't know if he will receive it.
So far, the Cricket graphs seem unaffected.
And 10 minutes later, back up.......but still hanging badly......painfully so.
-
Eric says it was a freaking spider scan trussing up thingys.
Should have it blocked now.
-
I already received 55 work units an no problem at the moment with updates either uploads ;-)
-
I guess I'll be the first...
I think Seti is down again. On a Saturday night/Sunday AM nonetheless
:-(
6:30 UTC
-
It just came up, so false alarm... hmm. It was inaccessible by me for about an hour
-
Message from today's Tuesday outage:
Science Database Updates
We are attempting some updates to the master science database. This may mean a workunit shortage over the next day or so after we come back on line later this afternoon from our usual Tuesday outage. 10 Jul 2012 | 16:03:33 UTC
Claggy
-
Message from today's Tuesday outage:
Science Database Updates
We are attempting some updates to the master science database. This may mean a workunit shortage over the next day or so after we come back on line later this afternoon from our usual Tuesday outage. 10 Jul 2012 | 16:03:33 UTC
Claggy
I did notice a shortage of work and SETI@home appears unaccesable for now?!
Maybe a few hours later? Well certainly do hope so :-\
Replica seconds behind master 156,883 1m
So, SERVER-PAGE makes no sense atm..
Well, SETI is up again1
-
Another power outage?
berkeley.edu is down!
-
Another power outage?
berkeley.edu is down!
All OK from here.
-
And from Arizona as well.
-
Tracert is OK for 22 hops - the last being ae-62-62.ebr2.newyork1.level3.net - after that - dead!
Anyone else from Europe?
-
Depends if you count the UK as being in Europe...
Mine took a stopover in New York as well, but got bored of sightseeing and carried on to Berkeley:
13 27 ms 27 ms 27 ms vlan103.ebr2.London1.Level3.net [4.69.143.94]
14 93 ms 101 ms 95 ms ae-44-44.ebr1.NewYork1.Level3.net [4.69.137.78]
15 95 ms 100 ms 104 ms ae-61-61.csw1.NewYork1.Level3.net [4.69.134.66]
16 98 ms 99 ms 100 ms ae-62-62.ebr2.NewYork1.Level3.net [4.69.148.33]
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
21 178 ms 178 ms 178 ms dc-svl-core1--svl-isp1-10ge.cenic.net [137.164.47.133]
22 187 ms 187 ms 187 ms dc-svl-agg1--svl-core1-10ge.cenic.net [137.164.47.120]
23 180 ms 179 ms 180 ms dc-ucb--svl-agg1-10ge.cenic.net [137.164.50.19]
24 205 ms 188 ms 188 ms t1-3.inr-201-sut.Berkeley.EDU [128.32.0.65]
25 181 ms 180 ms 180 ms g5-1.inr-204-sut.Berkeley.EDU [128.32.255.105]
26 181 ms 180 ms 181 ms ssl-f1.SSL.Berkeley.EDU [169.229.0.218]
27 182 ms 181 ms 181 ms isaac.ssl.berkeley.edu [128.32.18.189]
-
All seems OK here in Wisconsin, USA.....
Have been on the forums all morning lurking.
And the crickets are still singing full note.
-
All seems OK here in Wisconsin, USA.....
Have been on the forums all morning lurking.
And the crickets are still singing full note.
I know you're UK based, Richard - that's why I wrote "Anyone else from Europe" :-)
Here's my log:
12 39 ms 15 ms 24 ms level3-ic-130907-s-b1.c.telia.net [213.248.77.198]
13 * * * Request timed out.
14 40 ms 41 ms 42 ms ae-0-10.bar1.Copenhagen1.Level3.net [4.69.137.153]
15 43 ms 51 ms 50 ms ae-7-7.ebr1.dusseldorf1.level3.net [4.69.142.170]
16 46 ms 50 ms 49 ms ae-22-22.ebr2.dusseldorf1.level3.net [4.69.143.186]
17 32 ms 31 ms 35 ms ae-48-48.ebr1.amsterdam1.level3.net [4.69.143.209]
18 36 ms 35 ms 34 ms ae-1-100.ebr2.amsterdam1.level3.net [4.69.141.170]
19 41 ms 42 ms 42 ms ae-46-46.ebr2.London1.Level3.net [4.69.143.74]
20 111 ms 112 ms 116 ms ae-41-41.ebr1.newyork1.level3.net [4.69.137.66]
21 113 ms 118 ms 120 ms ae-61-61.csw1.newyork1.level3.net [4.69.134.66]
22 114 ms 111 ms 110 ms ae-62-62.ebr2.newyork1.level3.net [4.69.148.33]
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.
31 * * * Request timed out.
32 * * * Request timed out.
33 * * * Request timed out.
34 * * * Request timed out.
35 * * * Request timed out.
36 * * * Request timed out.
37 * * * Request timed out.
38 * * * Request timed out.
39 * * * Request timed out.
40 * * * Request timed out.
Trace complete.
It' really weird that we're getting different behaviour from seemingly the same route.
-
What's the IP-address you get for berkeley.edu at your end?
-
Finally the route is OK, and berkeley.edu available!
-
The Seti forums are simply stuffed right now.
Drop dead stuffed.
-
Uhh......
It's dead, Captain.
-
Been up and down for the last few hours.
Cricket is OK.
Another serious bot attack, or a real DNS attack?
Dunno, but things are not good.
-
I come home from work in the middle of the night and find seti@home is down. Curious if it ever came back online today after their planned outage or if it's been down all day and night? Or did I miss another memo about a longer planned outage, as I've been very busy?
-Dave
[EDIT]
From BOINC forums (and from seti page apparently):
This message was posted on the home page of the SETI@home project :
Database Issues
The BOINC mysql database is currently unresponsive. The project will be offline for the next 12 hours or so. 25 Jul 2012 | 2:08:54 UTC
This stuff with the outages, ALWAYS happens while I'm in the middle of testing/debugging/changing Boinc versions and setups on my machines....
I just changed some stuff on my server, and an OS update changed some things about the way I (used to) run BOINC. So in the process of getting it figured out the SETI site goes offline for its maintenance so I cannot verify my work is good, nor can I add the Seti project to the other machine I've had to make changes to. Same thing now when I get home tonight and wanted to finish my changes and be done with the other box. :-(
And all this while working 65-70 hours this week and last.
Perhaps tomorrow at 4am or so I'll get things finished.
-
If you check the Cricket graphs, (http://fragment1.berkeley.edu/newcricket/grapher.cgi?target=/router-interfaces/inr-250/gigabitethernet2_3&ranges=d%3Aw&view=Octets) it looks like they came back up for a couple of hours, started to falter, and then shut down.
-
If you check the Cricket graphs, (http://fragment1.berkeley.edu/newcricket/grapher.cgi?target=/router-interfaces/inr-250/gigabitethernet2_3&ranges=d%3Aw&view=Octets) it looks like they came back up for a couple of hours, started to falter, and then shut down.
Database Issues
The BOINC mysql database is currently unresponsive. The project will be offline for the next 12 hours or so. 25 Jul 2012 | 2:08:54 UTC
From SETI@home Main Page.
-
The graph is showing activity again for at least the last few hours. I wonder if they are getting work processing parts up and running before getting the other stuff up
-
Carolyn is running again, must be getting close to turning on the the rest.
-
Carolyn is running again, must be getting close to turning on the the rest.
Yes, to tie off the loose end (even at risk of invoking Murphy), all functions are active again - though downloads are, as usual, a little reluctant to leave the nest.
Ah - edit - except Beta.
-
The Scheduler, Uploads and Downloads are working at Beta. Seems to be just the web-based stuff missing.
Joe
-
Looks like Carolyn locked up again, site is not loading except for a yellow box.
-
The Scheduler, Uploads and Downloads are working at Beta. Seems to be just the web-based stuff missing.
Joe
Do 26 Jul 2012 23:56:56 CEST SETI@home Beta Test update requested by user
Do 26 Jul 2012 23:56:56 CEST SETI@home Beta Test Fetching scheduler list
Do 26 Jul 2012 23:57:03 CEST SETI@home Beta Test [error] No scheduler URLs found in master file
no.
-
The Scheduler, Uploads and Downloads are working at Beta. Seems to be just the web-based stuff missing.
Joe
Do 26 Jul 2012 23:56:56 CEST SETI@home Beta Test update requested by user
Do 26 Jul 2012 23:56:56 CEST SETI@home Beta Test Fetching scheduler list
Do 26 Jul 2012 23:57:03 CEST SETI@home Beta Test [error] No scheduler URLs found in master file
no.
The scheduler url isn't embeded in the Seti Beta 'Project down for maintenance page', you'll need to edit your client_state.xml to get Boinc to contact Beta (while it's in the semi down for maintenance state),
If you set the following for the Beta project, you should be able to connect again:
<nrpc_failures>0</nrpc_failures>
<master_fetch_failures>0</master_fetch_failures>
<scheduler_url>http://setiboinc.ssl.berkeley.edu/beta_cgi/cgi</scheduler_url>
Edit: never mind, Seti Beta is now fully up from maintenance.
Claggy
-
Due to a planned day long power outage, which starts early on Wednesday, all servers will be down from the beginning of Tuesday's normal outage until early on Thursday.
-
Good to know. I'll just disable network communications on both machines until Thursday.
-
Now the BOINC website is down as well - unsurprisingly, since they live in the same building, with the same power supply. It was pre-announced.
-
Now looks like that BOINC is up or at least the web page is working, but no DW/UP in WU neither crunched results for SETI.
:'(
-
Now looks like that BOINC is up or at least the web page is working, but no DW/UP in WU neither crunched results for SETI.
:'(
Well, that should mean that power has been restored.
Now we just have to await the arrival of the boyz in the lab tomorrow morning to set things alight.
-
There was a blip of activity on the Crickets. Looks like it's all down again though after only ~15 minutes. I'll fight for upload slots in 12+ hours.
-
Uploads are starting to happen, but a bit slow and hit-and-miss.
-
Missed it by that.......much. Finished my last work unit less than an hour before they came back up. Reported 272, requested new work and got one whole shorty. Took me all of four minutes to complete and report. Of course then I got no work available for a try or two but now have around twenty work units to crunch on until they get this mess cleared up. Could still use some for my CPUs but at least my GPU has something to do.
Well, while I was typing they were nice enough to send me another twenty tasks. All resend lost tasks. Didn't even know they were missing. :)
-
http://www.youtube.com/watch?v=oPwrodxghrw
-
My server comms were mundane at best. Allowed network comms, all uploads went through on the first try, all reported on the following work request. AP-only machine got no new work in response, MB-only machine got a full cache (2.5 days, ~15 WUs) of _0's in one request.
-
Oops...
Looks like Seti went dark about 2 hours ago.....forums and all at the moment.
Home page slooooooooowly loads with the following message....
Unable to connect to database - please try again laterError: 2003Can't connect to MySQL server on 'carolyn' (111)
I thought they had dumped MySQL........
Meouch.
-
Project is temporarily shut down for maintenance
At least one is working .. ;)
-
Still no connection possible ..
Scheduler request failed: Couldn't connect to server
Scheduler request failed: HTTP internal server error
Scheduler request failed: Timeout was reached
-
OK, S@h is again reachable .. - at least for my machine .. ;)
-
OK, S@h is again reachable .. - at least for my machine .. ;)
Front page news, 22:56 UTC last night:
We are recovering from a database crash
The main boinc database suffered a crash this morning. We are back up and running now and are catching up on work distribution.
-
Richard....
Just curious though we are back up and running......
I thought I read that they had converted away from MySQL.
Was that only on some of the servers?
-
Richard....
Just curious though we are back up and running......
I thought I read that they had converted away from MySQL.
Was that only on some of the servers?
They've always run two database systems - MySQL for BOINC (WUs/tasks/message boards), and IIRC Informix for the science databases and NTPCKR. So far as I know, there's been no change there - it would be a big job.
You might be thinking of the recent change from Apache to nginx - that's the webserver that handles comms (including uploads and downloads), not the database.
-
Richard....
Just curious though we are back up and running......
I thought I read that they had converted away from MySQL.
Was that only on some of the servers?
They've always run two database systems - MySQL for BOINC (WUs/tasks/message boards), and IIRC Informix for the science databases and NTPCKR. So far as I know, there's been no change there - it would be a big job.
You might be thinking of the recent change from Apache to nginx - that's the webserver that handles comms (including uploads and downloads), not the database.
Yes, that's what I was remembering reading.......
Why not convert the whole kit to nginx, as most have said it is superior to MySql?
-
Yes, that's what I was remembering reading.......
Why not convert the whole kit to nginx, as most have said it is superior to MySql?
Completely different animal - nginx isn't a database management system, it's just another app - a communications app - which runs on Linux.
-
Yes, that's what I was remembering reading.......
Why not convert the whole kit to nginx, as most have said it is superior to MySql?
Completely different animal - nginx isn't a database management system, it's just another app - a communications app - which runs on Linux.
Thank you Richard.....
I don't understand these things too well.
I thought nginx was replacing MySQL, which many have said Seti has about reached it's useful limitations on.
-
Looks like the space science lab just dropped off the grid again. Unable to get to Main, beta or BOINC sites.
-
Yep, same here.
-
Looks like the space science lab just dropped off the grid again. Unable to get to Main, beta or BOINC sites.
Yup...
Stone cold dead. Forums and all. The Cricket graphs are still up, but they've dropped dead in their tracks.
-
The last responding hop for a treceroute to boinc.berkeley.edu or setiathome.berkeley.edu is ssl-f1.SSL.Berkeley.EDU [169.229.0.218]. I guess that means at least something in the SSL building is alive, though it doesn't rule out a power failure for the building with only a few things switched over to a backup generator.
I did note on the Cricket graphs that some downloads continued for about 10 minutes after uploads stopped, perhaps UPS time.
{edit} And a few minutes later things seem to be coming back. Both BOINC and SETI@home web pages are reachable, Cricket showing some signs...
{edit2} Coming and going. IST service status (http://systemstatus.berkeley.edu/) says:
IST Service Desk is receiving reports of a temporary power outage at various campus locations. This outage is apparently affecting phones, networks and electrical power. Power seem to be coming back up and IST will update this announcement as more information is known.
Joe
-
Well, I just hope it wasn't another server-damaging power event this time!
-
Seti Beta page now shows :
Unable to connect to database - please try again laterError: 2003Can't connect to MySQL server on 'oscar' (111)
-
Seti Beta page now shows :
Unable to connect to database - please try again laterError: 2003Can't connect to MySQL server on 'oscar' (111)
Yeah, server status is updating and shows DBs disabled, along with most other thingys.
-
And.....from the Seti main page.
"Unscheduled Power Outage
Power to the north east side of the Berkeley campus and the Space Science Lab was interruped this morning at about 16:30 UTC until about 17:30. The cause of the outage has not yet been determined. We're working to get our databased back in working order. We don't yet have an ETA for when the BOINC server will be back online. 29 Nov 2012 | 18:24:52 UTC "
-
:-(
And I actually had some time off today.
-
Carumba!!!
The main page message now reads thusly...
"Unscheduled Power Outage
Power to the north east side of the Berkeley campus and the Space Science Lab was interruped this morning at about 16:30 UTC until about 17:30. The cause of the outage has not yet been determined. We're working to get our databased back in working order. Current estimate is that we'll be back online on Tuesday, Dec 6. 30 Nov 2012 | 4:42:45 UTC "
Tuesday, Dec. 6th????? Tuesday is the 4th, the 6th is Thursday. In either case, a rather long outage.
I wonder if the power problem took out some hardware or if they have simply decided to do some database revamping as long as the servers went down ungracefully.
-
December the 6th. is a Tuesday - in 2016 :o
-
December the 6th. is a Tuesday - in 2016 :o
Oh gawd, Richard......
I hope we don't have to wait THAT long....LOL.
-
A side note to any and all GPUUG team members here....
Juan has now started a GPUUG team forum over at Einstein for us to converse in whilst Seti is down.
Please come visit.
Meow.
-
FYI...
I have word from Eric that the servers survived the power outage hardware-wise.
The problem is that the server that hosts the backups is now rebuilding the RAID array, and given it's size, that's gonna take quite some time to self-sort. And there's nothing they can do until that rebuild completes.
Meowch.
-
December the 6th. is a Tuesday - in 2016 :o
Or they could still be using last years calendars, time for a donation of new calendars i think, Space/Kitties/Raccons would be appropriate, ;) suggestions?
Edit: Donated $30 to the GPU General fund for any calendars suitable, and $30 to the new servers.
Claggy
-
<sigh> :(
-
December the 6th. is a Tuesday - in 2016 :o
Oh gawd, Richard......
I hope we don't have to wait THAT long....LOL.
LMAO....
Wow that is a looooong outage. Oh my. Always when work is slow. These past months when I was working 60+hrs a week there wasn't any real outages. Now that I'm slow, we're down for another half an age.
-
December the 6th. is a Tuesday - in 2016 :o
Or they could still be using last years calendars, time for a donation of new calendars i think, Space/Kitties/Raccons would be appropriate, ;) suggestions?
Edit: Donated $30 to the GPU General fund for any calendars suitable, and $30 to the new servers.
Claggy
Well done and thank you, Claggy!
-
What does the currently server problems mean for S@h-Beta?
My machine have still S@h-Beta WUs.
~ 160 WUs already ULed, ready for report.
~ 70 WUs still in the BOINC/Transfer overview (network suspended) - (the UL would work).
Still a lot WUs ready for calculation.
I should let calculate them?
The already ULed results could be lost now?
I should let suspended the network for to be sure that nothing will go lost?
Maybe if BOINC report the ULed results, the server will not find the ULed results?
What should I do now, network allow - or suspend?
Thanks.
-
Just let it run.
Nothing should be lost.
Mike
-
Due to lab-wide electrical repairs there will be a complete power outage from January 4th to January 6th, during which all our servers will be unreachable. We are also having our server closet air conditioner repaired, and will be completely off line from January 14th to January 15th.
-
The BOINC web site and other services will be offline on 5 January 2013 because of a planned power outage at Space Sciences Laboratory.
Your guess is as good as mine why Matt and David are characterizing the first outage so differently.
Joe
-
The BOINC web site and other services will be offline on 5 January 2013 because of a planned power outage at Space Sciences Laboratory.
Your guess is as good as mine why Matt and David are characterizing the first outage so differently.
Joe
My guess is they are going to bring the SETI servers down early so they can do it in the proper order.
-
My guess is DA, as usual, is being overoptimistic.
And Matt is being more realistic, or at least trying to prepare the users for the worst case.
Being an electrician, I know things don't always go as smoothly as planned or according to a preplanned timetable.
If the outage is as Matt says, everybody is forewarned.
If the outage is shorter than Matt says, everybody is happy.
I prefer Matt's approach.
Meow.
-
Matt Lebofsky wrote:
UPDATE: the outage scheduled for the weekend of the 4th-6th has been cancelled so normal operations shall continue. We still plan to have our air conditioner repaired and will be completely off line from January 14th to January 15th during that time.
Claggy
-
Latest news on the outage to repair AC:
Two day outage to repair our machine room air conditioner
Projects are down until the work is complete, which should be at some point on Tuesday, January 15.
14 Jan 2013, 17:00:36 UTC
Joe
-
At about 08:50 Berkeley time, the Cricket graph is showing activity.
Joe
-
At SETI Beta, attempting an upload I get:
[error] Error reported by file upload server: Server is out of disk space
The Cricket activity suggests the same at main, though I don't have any to report there.
Joe
-
All my uploads to Seti Main went through O.K, unlike Beta,
Edit: 5 minutes later, and Beta uploads are going through O.K.
Claggy
-
Uploads are functional. Upload your backlog now before things get too saturated. My two rigs are just waiting for the scheduler for reporting now. ~25 or so uploads went through from both machines in under 30 seconds total.
-
Someone can see the SAH site?
http://setiathome.berkeley.edu
I (@ 11:29 UTC) - not. :(
-
I cannot connect to the website either. The Cricket graphs are still solid green and my computers are still able to reach the project servers, but no web content at all.
And today is a holiday at Berkeley due to Washington's Birthday/Presidents' Day, so nobody would be scheduled to be in the lab. I'll send an email off to Eric, but one never knows when he may get it.
Good news is, it does appear that work is still going out. My best rig just got some new WUs after a number of work requests being told it had reached a limit of tasks in progress, so all is not down just yet.
-
Most likely, it sounds like the "new" Thinman crashed this time.
-
Most likely, it sounds like the "new" Thinman crashed this time.
Well, I dunno if thinman has any other duties, but if the web pages going down until tomorrow is the worst that happens, we can sure all live with that. (Sprinkles a little catnip around for good luck)
-
So, now we have project servers working, but no website? Usually it's the opposite. Ah well such is life.
Hope everyone has been well over in this corner of the web.
-Dave
-
Eric got my email and said he would look into the problem. Depends, I suppose on whether it can be patched up by remote or not.
-
My caches are full so not really a problem to me.
It seems at least one server needs a nap every week. ::)
-
After a few minutes, Eric replied.....
Thinman is down hard, I sent an "if you're there reboot it to Matt,
Jeff, and Dan.
-
Other problems may be brewing.
My rigs are reporting OK but I've been getting the "No Tasks Sent" message on all rigs since 23:30 UTC. The fastest rig is out of GPU work.
All has been good here since the last Planned Outage.
T.A.
-
I suppose things could be a lot worse, but the webserver being down is going to leave a lot of holes in my spreadsheet for not being able to record the results from probably 12-15 tasks (considering I burn through about 6/day, it's not devastating, but statistics don't like holes in data sets).
-
Maybe their rubber-band broke?
-
Thinman has been rebooted and the forums are back online.
-
Looks like thinman just crashed once again, main page and upload server unreachable but the scheduler and download server are functioning.
-
........ but the scheduler and download server are functioning.
Which is a lot of help if you cant upload in order to report and request new work :(
Oh Well, Jest one durn thing after another
T.A.
-
Again... :(
-
Just sent an EMail to David, Eric, Matt and Jeff.
-
Yep, it looks like thinman is down and bruno is hung. Unfortunately
this happened about half an hour after I left work.
-
And Seti is back up and running!!!
Meeeeeeeowza!
-
Not for long... It's down again :(
Hope I've not jinxed them by suggesting they conscript a few students to contribute pedal power dynamos to power the closet... :o
Here's hoping for a clean speedy recovery again...
Happy crunchin',
Martin
-
Up and running for me.
-
And, of course, a reminder that they are going down for power repairs in about 9 hours until sometime Sunday afternoon. I am sure most of you are aware, but just in case...........
Lab-wide Power Outage This Weekend (Feb 22-24)
Once again we are going offline due to electrical repairs in our entire building. We plan to come down Friday afternoon (15:00 local time) and be back up by Sunday afternoon (12:00 local time). All SETI servers will be unreachable during that span.
-
Yep. It's been down for a few hours now. I was trying to remember to pay attention to it and do some last-minute reporting and get the validated info before it went down, but got distracted with family stuff for the evening. Oh well. Just set both machines to "no network communications" until I see signs of life on the crickets sometime late Sunday.
-
This truly sux.
Just woke up........house is cold. Fired up the furnace.
We got 8 inches or more of snow I have to deal with now,
Cannot move the car.
Will winter not just go AWAY????
-
... Will winter not just go AWAY????
Hey! You just noticed the global warming?!
:(
Happy warm crunchin',
Martin
-
... Will winter not just go AWAY????
Hey! You just noticed the global warming?!
:(
Happy warm crunchin',
Martin
Yeah, I noticed....LOL.
A few more inches of the fluffy stuff and we are in the record books for the most snow in one year ever.
Just got back in from firing up the 'ol blower to deal with it.
-
global warming....
hopefully seti comes back up soon, so we can use more electric, which produces more carbon dioxide to thaw you out their...
-
Seti is online again.
-
Seti is back up, if a bit slow at the moment,
Claggy
-
Scheduled Outage Network and PBX at Botanical, LHS and SSL
Posted by CSS IT ~jgill
On 3/18/2013 at 3:41 pm PST
Posted in Scheduled Outage
Outage Type: SCHEDULED OUTAGE
Date Submitted: Tuesday, March 19th – Wednesday, March 20th, 2013
Outage Start/End Time: 1700 – 0800
Groups Impacted: Botanical, LHS and SSL
Equipment: Network and PBX
Description: As part of network upgrades for the Hill, IST staff will be doing an extensive reworking of the fiber optics serving Botanical Gardens, Lawrence Hall of Science and Space Sciences Lab. Due to the time intensive nature of the work, this is an overnight outage starting at 5:00 P.M. Tuesday, March 19th, and concluding no later than 8:00 A.M. Wednesday, March 20th. This outage will disable all network connectivity for these locations, as well as PBX phone service.
That means SETI...
-
Still up as of almost 1900 PDT.
Slow, but still there.
-
2300 PDT, some services are still responsive. Just noticed uploads are having issues, but the website is still working (very very slow.. need to hit refresh a few times to kick it). Just seems like tons more packet loss than usual, and Cricket is still showing a saturated pipe.
-
Yes....
Very strange. I thought we were to be in total blackout, yet the forums are still up.
So are the servers and everything else, but reaching them to transfer any data seems almost hopeless.
I don't for the life of me know what is still saturating the bandwidth as shown on the Cricket graphs, because my rigs surely can't seem to do much communicating.
And with uploads apparently almost nil, hardly anybody could be requesting new work, much less getting any issued.
Puzzling indeed.
-
Still reachable at 0445 PDT, barely.
-
They've either found some ancient 10 Mbit coax link up the hill to use as a backup, or somebody's patched their iPhone into the router...
-
They've either found some ancient 10 Mbit coax link up the hill to use as a backup, or somebody's patched their iPhone into the router...
They better have a hack of a data plan on that iPhone...LOL.
-
Somehow I'm reminded of the Grand Old Duke of York (http://en.wikipedia.org/wiki/The_Grand_Old_Duke_of_York)...
-
And we are back, folks.
Meow and away.
-
Yep, downloading already.
-
Just seen Beta side is up again. Project of course temporarily shut down.
-
Just seen Beta side is up again. Project of course temporarily shut down.
I think it just hasn't been taken fully down yet, might be because the Main & Beta website are hosted on different servers.
Claggy
-
They most likely left Thinman up there until they get the rest of the servers up and running first, then it will be moved down with the rest of them.
-
Whatever ? At least not all is dead at Berkeley/SSL.
-
all is dead, what is the deal. house is getting cold!
-
all is dead, what is the deal. house is getting cold!
Server Closet Migration Outage (April 1-3)
We are in the process of moving all our servers to a new location. All SETI data servers are currently unreachable. We hope to be back on line by the morning of Thursday, April 4th.
3 Apr 2013, 3:02:10 UTC
-
It really sucks that they can't stop the rationing of work units coming into outages or altogether for that matter. I have only 3 computers left running Seti at all and no point in putting any of my upgraded machines to Seti as they don't use the computer power available to them. Guess it has just become a software hobby as they don't seem to worried about the science. JMO
-
Seems they ration so we all can get a little work, from time to time.
The amount rationed out lasts less than a day on my machine.
For sure it would be nice to have enough to last a couple (more) days at times like this.
Maybe the server move to the co-lo will be a step toward more work being available, but it will probably just expose the next weak link in the supply chain/system.
-
all is dead, what is the deal. house is getting cold!
Server Closet Migration Outage (April 1-3)
We are in the process of moving all our servers to a new location. All SETI data servers are currently unreachable. We hope to be back on line by the morning of Thursday, April 4th.
3 Apr 2013, 3:02:10 UTC
do we get a better Ethernet connection then Fast Ethernet (100Mb/s)?
-
The home page and the server status page are back online from the new location this morning!!
-
The home page and the server status page are back online from the new location this morning!!
Thats good news. Lets hope they have not lost any hardware on the move.
-
Seti Beta Forums are up too now.
Edit: Just a bit later and the Main forum is up.
Claggy
-
And I'm uploading/reporting finished work!
Edit:
And I just got 3 AP WU's at "astronomical" speeds ;)
-
I have work from Beta, but the sight shows that they are down again.
Have only gotten some CPU work from main so far.
-
It would appear that the entire Seti@home website and WU servers have just lost connectivity with the outside world :(
-
After ~ 6 weeks no SAH - because in this time no DSL, just only '56k' - yesterday my PC crunched happy SAH WUs again. :)
~ 24 hours later, SAH is broken? :(
This happened the 1st time after the server relocation?
BTW. Someone have the URL to the new Cricket graph?
Thanks. :)
-
It's not the SETI project which has gone down, but the whole berkeley.edu internet domain - no further information yet.
The new Cricket graph would be
http://fragment1.berkeley.edu/newcricket/grapher.cgi?target=/router-interfaces/inr-211/gigabitethernet6_17&ranges=d%3Aw&view=Octets
but that's still a Berkeley address, so that's down too at the moment.
-
It seems to be back.
-
It seems to be back.
+1
Claggy
-
Looks as if berkeley is unreachable again.
Edit: Fifteen minutes later, it is back. (I susp%bt it might have just been my ISP)
Claggy
-
Three more of us saw that, and for longer than 15 minutes - most Berkeley addresses seemed to be unreachable from the UK (or possibly from our shared ISP - I think most of us will be with BT) between about 08:10 UTC and 08:55 UTC.
-
Three more of us saw that, and for longer than 15 minutes - most Berkeley addresses seemed to be unreachable from the UK (or possibly from our shared ISP - I think most of us will be with BT) between about 08:10 UTC and 08:55 UTC.
My connection is through a 'Three' 3G Dongle, i found i had no Berkeley connectivity, so pulled the Dongle, reinserted it, and reconnected it, there was no change, then about 15 minutes later it was working (at ~09:05 UTC)
Claggy
-
There seems to be problems, after initially being able to contact the schedulers at both the Main and Beta projects, then later being unable to, they're now taken both projects down for maintenance again.
Edit: that didn't last long.
Edit 2: down again.
Edit 3: and up again within a cpouple of minutes, if that.
Claggy
-
berkeley is out by the looks of it ...
-
The weekly maintenance yesterday at Tuesday was short, compared with others. IIRC ~ 2 hours or something.
Today the SETI@home crew will install the new 'SETI@home v7' apps - from the rumors.
A pity that noone announced it at the front page of SETI@home that this will happen. :(
Although it's not a big deal for stock BOINC/SETI@home user. Stock BOINC will D/L all new project apps automatically.
The app_info.XML users get trouble the next days if 'SETI@home Enhanced' tasks are not longer available - and they get no new 'SETI@home v7' tasks.
If this is new for someone, in the 'Number Crunching' forum at SETI@home 'msattler' made a thread, IIRC 'If v7 rolls out.....' or something like this. (Can't insert the URL, because the SETI@home forum is currently down.)
I would like to take the opportunity...
I would like to say 'Thank you!' to all Lunatics crew members, and to all others around which help them, for all their hard work in their free time. :)
My appreciation, respect and recognition! :)
They help the project SETI@home also behind the curtain, optimize the stock apps, support the admins, and so on.
It's look like next Monday they will release the new Lunatics Installer v0.41 with new inter alia opti 'SETI@home v7' apps.
Again, 'Thank you!'.
:)
-
'well, I am down now.
Idiot mods at Seti don't seem to have a sense of humour.
I have other things to do.
You are gonna miss me soneday.
(http://www.youtube.com/watch?feature=endscreen&v=cmSbXsFE3l8&NR=1)
-
I know things are busy with other tasks.....but pulllllllllleez pass this one.
I go now.
Pitch Perfect. (http://www.youtube.com/watch?feature=endscreen&v=26Uftr-jWz4&NR=1)
I know some of you are not into mucic, and are way over your hears with the installer thingy.
But, maybe you would enjoy a braak.
-
'well, I am down now.
Idiot mods at Seti don't seem to have a sense of humour.
I have other things to do.
You are gonna miss me soneday.
(http://www.youtube.com/watch?feature=endscreen&v=cmSbXsFE3l8&NR=1)
going to bed and noticed the curching room was to quite. same here, just have a load of AP for the cpu..
-
Hmmmmmmmmmm.
just where is that grand safety net the colo was supposed to provide now?
-
Just woke up, Saturday morning and the SAH server are down?
Looked to the Cricket graph, since ~ 2 hours.
Sent E-Mail to David, Eric, Matt and Jeff.
-
I, too, took a look at the Crickets and realized they were flat-lined. Decided to run a traceroute to see where the problem may lie, as I was thinking maybe an HE issue..
but scheduler server, upload, and both downloads are ping-able, so they are accessible, but they don't seem to be doing anything more than being network-aware.
-
From http://ucbsystems.org/
Network Connectivity issues
Outage Type: UNSCHEDULED OUTAGE
Date Submitted: Friday May 31st, 2013
Outage Start/End Time: 21:30 – Ongoing
Groups Impacted: Campus
Equipment:
Description: There is currently a network connectivity issue that involves CalNet Authentication Issues. This is affecting multiple services including sending of email. There is currently not an ETA for this issue currently.
Matt, Eric and the boyz can sleep tight: have to wait on the Campus elves for this one.
-
It's now fixed, uploads are going through, and the website is available.
Claggy
-
Seti is back online.
-
Seti is down, the website and scheduler are unreachable,
Claggy
-
Up for a little bit and then completely offline now.
Berkeley is still up though.
-
Active
Unscheduled Outage – Emergency Shutdown Data Center Servers
Outage Type: UNSCHEDULED OUTAGE
Date Submitted: September 30, 2013
Outage Start/End Time: 18:15 – TBD
Groups Impacted: campus
Equipment: Data Center Servers
Description: UPDATE 19:15 – Power is returning to the Data Center but systems remain shutdown, temperature is slowly dropping. Servers continue to be monitored.
Due to the campus widespread power outage the Data Center servers are experiencing extreme heat impacting central servers residing in the Data Center. With temperatures rising over 100 degrees , all servers are gracefully being shutdown.
Services Impacted:
Calmail
Bconnected
Applications hosted on VM
All Database Systems
Telecom Gateways impacting Voice Services
BSpace
BCourses
CalPlanning
BAIRS
-
Well, let's hope the guys at the colo haven't lost their cheat sheets on what order in which to restart the whole Seti shebang.
-
Warm servers - not good.
-
Last update was that power has been restored, things are cool enough to restart servers.
Of course, I am sure that servers controlling campus based servers have priority over getting Seti back online.
Hopefully in the next hour or two.
-
Well....something's coming up.
Uploads are going through. But no web pages or other server comms.
Cannot fetch scheduler list or make any scheduler requests.
Dunno if it's the colo guys trying to start things up, or if the boyz in da lab are getting their fingers into the servers.
Hope nothing got damaged due to overheat.
EDIT....
Just as I posted the above, the server status and web pages came back......down for maintenance.
As it's the usual Tuesday maintenance window, I suppose they'll tackle that and check everything before coming back up.
-
http://www.sfgate.com/bayarea/article/UC-Berkeley-campus-evacuated-power-outage-4858201.php
-
Beta is up, but main is still down.
-
Beta is up, but main is still down.
And now, all seems to be working. Reported completed work, and received new work in return.
-
67 pages, i see things have not gotten better during my summer hiatus!
-
67 pages, i see things have not gotten better during my summer hiatus!
What are you talking about?
Only ONE further page over the summer, and only three outages, one for loss of Network connectivity, one for loss of Power, and one unspecified, a definite improvement,
If you want guarantied 24/7/365 uptime, then you'll need to donate the tens or hundreds of millions of dollars for the redundant power supplies, redundant internet connections, and redundant servers,
then spread them across the world, and pay for the people to look after them.
Claggy
-
Looks like muarae1 might be having problems as it usually is not down during the scheduled maintenance, and the fact that it is only 0730 in Berkeley.
[edit]
Looks like it happened around 0630 Berkeley time as I got work at 0550 just fine.
My current time is MST, 1 hour off Berkeley.
31/12/2013 07:29:52 | SETI@home | Started upload of 12mr13ad.3117.1396.438086664197.12.87_0_0
31/12/2013 07:29:57 | SETI@home | Finished upload of 12mr13ad.3117.1396.438086664197.12.87_0_0
31/12/2013 07:31:18 | SETI@home | Computation for task 12mr13aa.1426.1703.438086664204.12.83_0 finished
31/12/2013 07:31:18 | SETI@home | Starting task 12mr13ad.3117.1396.438086664197.12.100_0 using setiathome_v7 version 700 (cuda50) in slot 2
31/12/2013 07:31:21 | SETI@home | Started upload of 12mr13aa.1426.1703.438086664204.12.83_0_0
31/12/2013 07:31:43 | | Project communication failed: attempting access to reference site
31/12/2013 07:31:43 | SETI@home | Temporarily failed upload of 12mr13aa.1426.1703.438086664204.12.83_0_0: connect() failed
31/12/2013 07:31:43 | SETI@home | Backing off 00:03:30 on upload of 12mr13aa.1426.1703.438086664204.12.83_0_0
31/12/2013 07:31:45 | | Internet access OK - project servers may be temporarily down.
31/12/2013 07:33:48 | SETI@home | [sched_op] Starting scheduler request
31/12/2013 07:33:48 | SETI@home | Sending scheduler request: To fetch work.
31/12/2013 07:33:48 | SETI@home | Reporting 15 completed tasks
31/12/2013 07:33:48 | SETI@home | Requesting new tasks for NVIDIA
31/12/2013 07:33:48 | SETI@home | [sched_op] CPU work request: 0.00 seconds; 0.00 devices
31/12/2013 07:33:48 | SETI@home | [sched_op] NVIDIA work request: 517862.88 seconds; 0.00 devices
31/12/2013 07:33:50 | SETI@home | Scheduler request completed: got 0 new tasks
31/12/2013 07:33:50 | SETI@home | Project is temporarily shut down for maintenance
31/12/2013 07:33:50 | SETI@home | Project requested delay of 3600 seconds
31/12/2013 07:33:50 | SETI@home | [sched_op] Deferring communication for 01:00:00
31/12/2013 07:33:50 | SETI@home | [sched_op] Reason: project is down
-
Yes, its down since 6:30 Berkeley time.
-
Looks like muarae1 up and crashed again as the site does not load at all, but we can still get work just fine.
-
From the Cricket graph is looks like since ~ 7 hours no U/L possible.
My BOINC have too much backlogged U/Ls, so no new work request.
But my BOINC have still WUs for calculation for the next ~ 2 1/2 days .. ;)
-
gpu work only lasted for a few hours.
57 degrees F inside and getting colder, 22.9 degrees F outside. may have to turn on the heat....
-
Furnace is on here....
Around -20f last night, coldest night of the winter so far, and Seti goes poof. That's right, MINUS 20f. All the CPUs have work to last a while, but the GPUs ran out long ago. And the CPUs don't generate but a fraction of the heat that the GPUs toss out.
But the furnace will keep the kitties warm until Seti comes back up.
Meowsigh.
-
Eric says that Bruno fell over last night. He says it's back up now and they are working on damage control.
Followed, I assume, by the weekly outage.
I brought the validate error/invalid task situation to his attention, and asked if it could be script-fixed.
We shall see on that front later, I guess.
Meowouch.
-
The SAH scheduler server (since ~ 17:40 UTC) and website is down.
Upload is OK.
FYI, I sent E-Mail to David, Eric, Matt & Jeff.
-
I got response from Eric, they are working to fix the problem.
-
It was up for a little while, but looks like it died once again.
No uploads currently.
-
or status page....
-
it is back now... :D
-
and the upload server is down again.
-
SETI hasn't run specially smooth lately, but well guess we will live with it :)
-
Hmmm....
My calendar says Monday, but it would appear we are having a Tuesday outage.
Unless something crashed hard. I saw no signs of impending trouble through.
Meowhazzup?
-
Project main page says:
News
Weekly Outage and Initial Catch Up
Because of staff away times, we are conducting the weekly maintenance
outage today (Monday, 05/12/2014)
12 May 2014, 14:21:26 UTC
Joe
-
Thanks for the heads-up, Joe.
-
The outage for utility power work at the UCB data center has begun. The project is shutting down gracefully of course, so front pages and server statuses are still alive. I presume when the power actually goes off those will disappear too.
Joe
-
Cricket just flatlined and both main and beta are not responding.
(http://fragment1.berkeley.edu/newcricket/mini-graph.cgi?type=png;target=%2Frouter-interfaces%2Finr-211%2Fgigabitethernet6_17;inst=0;dslist=ifInOctets%2CifOutOctets;range=151200;rand=280)
-
Could be something with the colo facility, but the BOINC home page is OK, so maybe it's SAH specific. This being Saturday, I hope someone who can fix it notices it.
Joe
-
Isn't the BOINC server still up in the SSL.
-
Interesting that it seemed to flatline right at noon (maybe 10-15 minutes prior). Seems almost like it was planned (in one way or another) and nobody was given a heads-up.
What I mean is.. maybe it wasn't intentional, but going down at noon sounds like maybe some kind of cron job gone awry.
Just a thought.
-
What is even weirder is that IST shows nothing for the outage.
http://ucbsystems.org/category/active/
-
Ebola???
-
As of 07:45 Berkeley time, it's up again. Just about a 20 hour outage, the recovery will take awhile.
Joe
-
Anyone know what happened this time? Everything's been down for 12 hours now.
edit: re-phrase: uploads still work, but website and scheduler have been down.
-
This is starting to get like the bad old days before the server relocation. :'(
-
Anyone know what happened this time? Everything's been down for 12 hours now.
edit: re-phrase: uploads still work, but website and scheduler have been down.
From the front page.
Database Issues
The BOINC mysql database is currently unresponsive. The project will be offline for the next 24 hours or so.
-
While they are transferring the servers, there is an IRC server set up so anyone who wants to chat, can.
76.101.50.133/6667
#setidown
-
While they are transferring the servers, there is an IRC server set up so anyone who wants to chat, can.
76.101.50.133/6667
#setidown
And the reason was:
SETI@home Network Outage (August 17-18)
Due to network and DNS upgrades/changes we will be shutting down the projects tonight (August 17th) and we will be offline until the next morning. 18 Aug 2015, 3:46:32 UTC
Edited: Edited my invisible post back in after correcting for my 'new' local time.
Claggy
-
I think it lost your post Claggy.
-
I had the forum time set at plus 4 hours (which used to be right), that gave a 9pm posting time for the above posts, not right.
I've changed it to plus one now, that should give a correct local time of 5pm.
And the text i post now is displayed now.
Claggy
-
Hmmmm...
Everything berkeley.edu seems to be offline.
Wonder if the colo blew up.
Meow?
EDIT....
The IST status page now indicates multiple campus outages, and campus IT is now working on it.
http://ucbsystems.org/category/active/
-
OMG.....
Last update says there was a fire in the data center......no word on how bad or involved it was.
Let's hope and pray the Seti servers are intact and it was only connectivity that was compromised.
The kitties are nervous!!!
-
OMG.....
Last update says there was a fire in the data center......no word on how bad or involved it was.
Let's hope and pray the Seti servers are intact and it was only connectivity that was compromised.
The kitties are nervous!!!
Doesn't sound good, hope it's not ET playing around.
-
OMG.....
Last update says there was a fire in the data center......no word on how bad or involved it was.
Let's hope and pray the Seti servers are intact and it was only connectivity that was compromised.
The kitties are nervous!!!
Doesn't sound good, hope it's not ET playing around.
No...LOL. Hopefully surveillance cameras don't show any high energy beams of any sort from the sky starting the fire.......
Anyway...the damage survey at the data center is supposed to start in about 5 minutes. It may take some hours before we get any news of their findings.
The kitties are keeping all their little toes crossed that it is minimal, and at very least, has not harmed Seti's servers.
Meow!
-
Latest news just posted by campus IST..........,.
UPDATE: Saturday 09/19/2015 07:45: We are currently restoring systems and bringing applications back online as they become available.
Please rest assured your email is still working. Even if you are unable to access your mail at this time, it will be there waiting for you when CalNet authentication is restored. Your phone and external email clients can still be used to access your account.
Once our systems are available, instructors are being asked to provide students with appropriate accommodations for possible missed assignments or other issues related to the outage.
-
I'm out for the day......
Here is the link to follow for news from campus IST as they sort things today........
http://ucbsystems.org/category/active/
Meow for now.
-
What some of you do not know or appreciate is this fact.
I am a savant.
My pain I have tried to relay to you at times.
I can be both quite sturdy at some times, and again just trashy at best.
My posts here and on Seti are my attempts at sharing what I have been through and what I have to live through day to day.
I drink and release my thoughts, but I am NOT a daily alch, much to the chagrin of some who think I am.........
My drinking is a crutch I use at times...most savants need a release or they simply die of thought overdose.
I am not a daily drinker. I binge heavily when I need to.
I hope one day, some of you shall come to realize that.
My only true peace shall come on the day I pass and get to meet my Lord and ask HIM for the answers I have sought for all of my years here on this earth. Because I simply have more questions than answers in this life.
And you do not have the answers for me either, much as some of you suppose to.
Meow for now,
Mark.
-
Looks like the entire projects fell of the net. Anybody know whats happened ?
-
Looks like the entire projects fell of the net. Anybody know whats happened ?
Everything went down yesterday afternoon except uploads. I sent my usual messages to Eric, Matt, and Jeff.
Was out for the evening. When I got home, I had a message from Eric that Oscar had crashed, and was back up.
Things were working again somewhat, but the SSP was not updating and the replica DB was offline. Not sure if any work was being generated or sent out, but I was not seeing any coming in.
A short while after I got home, things went down again, and currently uploads are not working either.
Hopefully things just got too tangled to fix by remote, and they will be able to sort it later this morning.
It's only 5:17AM in Berkeley as of this writing. 13:17 UTC.
-
Well, Seti's been down for over 11 hours now. They are working on it. And to complicate matters, when the guys at the colo try to reboot the Seti server, now it reboots the Boinc server which is in the same box. Eric thinks they may have a PSU issue now.
So, it could take a bit to get things back on track.
-
Well at least we can still report and get new WUs.
-
Well at least we can still report and get new WUs.
Yeah, so far that is hanging in there. Those servers must be in a different box in the rack than the one with the web servers having the difficulty. Apparently there are 4 servers in a 1U box that is having the trouble.
-
I'm glad to know that it is not just me having problems today. I did see on the Berkeley IT web site some maintenance going on but it did not look like it could affect access.
-
I'm glad to know that it is not just me having problems today. I did see on the Berkeley IT web site some maintenance going on but it did not look like it could affect access.
Yeah, I checked there also when Boinc went down after Seti did. Nothing going on there that would take us all down.
-
:'(
-
Lets get these 2 topics merged into one.
Might be a router problem as the site is just timing out while loading.
-
"Fresh from Eric Korpela, through email:
Carolyn (our BOINC database server) went down last night. Looks like a raid card problem. We're working on it. Will give an estimate shortly."
-
Thanks for posting Eric's statement here.
-
After only SETI being down last night, both SETI and BOINC are down currently.
-
I just checked my main cruncher and the queues are full at this point in time. So I'm not sure what is really down aside from the main site.
John
-
Is SETI@home really still in 'maintenance mode'? Or something at my end is broken? :-\
If I read the messages in BOINC correct, then since yesterday ~ 17:00 to now ~ 09:30 (MEST, UTC + 02:00), 16 1/2 hrs...
I switch on my dual Xeon with quad R9FuryX ((maybe?) just for the upcoming 'SETI.Germany Wow!-Event 2017' which start in ~ 5d:20hrs), and now since yesterday ~ 23:30 the GPUs, since ~ 05:00 the CPUs idle.
How can my PCs make much pending credits of results? :(
-
Yes, it is.
This happened before not to long ago.
We will have to wait til 5 PM or so i guess.
-
And it`s back up.
Jeff posted in the Panic mode thread that it was his fault.
No big deal, we are all humans.