Seti is down again

[perryjay]

Well, now I'm getting the down for maintainance message. Wonder how long this is going to last?

[SciManStev]

I would gyess that they would use this down opportunity to do the weekly maintanence backups, so I expect to see it back up in a few hours. They may slide this into a three day outage, depending on what their needs are.

Steve

[msattler]

I dunno what the devil is going on.....
Up and down, up and down.

Not a normal outage scenario.

Connectivity to the forums has been very sporadic as well.

[msattler]

Matt just posted in tech news.

I'll repost it here because you can get to the forums one minute and not the next......


Ah, the few days back at the lab between Xmas and New Year's... The university assumes nobody works at this time, so the buses aren't running, and so I have to drive into the lab. But of course they are still handing out parking tickets to people without regular parking permits (like myself, who rarely drive to the lab). So I gotta park elsewhere. Anyway...

Except for bruno (the upload server) having fits we were pretty much running smoothly all weekend. However bruno is also the main BOINC back-end administrative server for the SETI@home/Astropulse project, so when it has fits, everything kinda gums up. We couldn't get into bruno remotely (full process table?) so it waited until this morning when Jeff got in and rebooted it.

There was some cleanup after that, and we seemed out of the woods, but we're still having these mysql issues where the database enters these long periods of flushing pages to disk. We all agree that this is largely due to the increased demand (after all the long/short outages over the past two months, and perhaps a bout of short runners). Increased demand means more deltas, which in turn means more fragmented pages. We have these weekly outages to defragment the database, but given the load it's like 3-4 weeks of fragmentation within one week. We're thinking the outage tomorrow will largely fix this, but we're still tuning other stuff in the meantime. We already gave mysql access to more memory, but Bob predicted this wouldn't help, and he was right. He's trying other stuff now.

So the plan is to hang on have just the normal outage tomorrow, then be up (as best we can) the rest of the week and throughout the New Year's Eve weekend. Then in the new year we can really start squeezing these new servers and see what they got.

Oh yeah - I turned off the "resend lost results" for now to reduce the load on mysql. This is temporary.

- Matt

[Pizzadude]

I thought this was normal for SETI, have you not been paying attention for the last few months !!!!

(Ducks and runs for cover ....)

I dunno what the devil is going on.....
Up and down, up and down.

Not a normal outage scenario.

Connectivity to the forums has been very sporadic as well.

[sunu]

Can you reach berkeley?

I can upload and contact the scheduler, but I can neither download nor view the site.

[Ghost0210]

everything seems fine for me - but I've not need to download any new tasks can reach the site though

[Richard Haselgrove]

All OK here too. I'm just breaking in a new host, and it passed the magic 10 validations for CPU tasks this morning - so estimates are now 2 hours instead of 20 hours

In the last hour (15:00 - 16:00), it requested and received new work the maximum possible 12 times. Only problem today has been congestion when both MB shorties and AP are in the feeder mix - some download retries, but all cleared in the end.

You're not trying to reach the US via Egypt, are you, sunu?

[perryjay]

I'm not having any problems here in Florida. Seems like uploads and downloads are doing fine. No problem reaching the webpages either.

[sunu]

You're not trying to reach the US via Egypt, are you, sunu?

Even though we shouldn't be laughing 

Everything seems fine now.

[msattler]

Anybody know whazzup with Seti?
According to the Cricket graphs, all outbound traffic came to a dead stop a couple of hours ago.
Can't reach setiathome.berkeley.edu at all.
But what's strange is it seems that uploads are still going through.

[arkayn]

I think this was a planned outage at the university, I think Matt said something about it in his Monday post.

[msattler]

I know what you are referring to, but I thought that was supposed to happen Wednesday morning.
Maybe I am mistaken, or it was rescheduled.
If true, hopefully it will be back up soon, as Matt's post indicated.

Meow.

[arkayn]

Looking at the weekly graph, I only see the Tuesday outage. Wednesday looks perfectly normal.

[msattler]

Looking at the weekly graph, I only see the Tuesday outage. Wednesday looks perfectly normal.

You are right, I see that now.
It was probably postponed.

They'll be up later tonight, or tomorrow morning if the boyz have to reset some things to get them going again.

Kinda handy that the uploads were not affected so far, should lessen the backlog when things get back on track.