Seti is down again

[perryjay]

I'm not worried so much about work as I've still got a couple of days worth on my machine but I sure do miss the forums!   

[Zeus Fab3r]

I'm not worried so much about work as I've still got a couple of days worth on my machine but I sure do miss the forums!   

These are prety good forums too. Only thing that's missing is the voice of James Earl Jones in my head sayin' "Join me ... and recieve a free toaster oven" 

[Zeus Fab3r]

Yipee! The forums are back, account pages too but with some restrictions. (ie. pendings, task list)
Looks like we're finaly moving 

[msattler]

Which would indicate that they are still running without the replica online...trying to minimize load on the database as much as practical.
Lets hope mork is now up to the task........

[Geek@Play]

I Can't log into my account...............and can't make any post on the forums.

ver mind.........I got logged into my account now.

[msattler]

I Can't log into my account...............and can't make any post on the forums.

ver mind.........I got logged into my account now.

Might have been things just getting resynched.....

[msattler]

Jeff just posted the following in the Technical News thread.........

"It's been a painful week, but with some progress.

The server run before last was cut short by our upload space filling up. That was fixed by the bruno migration and we started the last server run a bit early.

But a crash of our primary boinc db machine, mork, got the secondary db server, jocelyn, out of sync. That meant that all of the read only queries had to go to mork instead of jocelyn. This overwhelmed mork and I turned off web access just so the server run could continue. Then mork crashed again Monday evening. Ouch.

Yesterday, we did our normal backup of mork and are recovering jocelyn from that today. The forums are up, but result viewing is disabled at the moment. We need to clear the back end queues ahead of the next server run and mork resources are needed for that.

Mork's tendency to crash seems to have accelerated. Perhaps this is secondary to the cooling crisis we had a couple of weeks ago. Actually, "crash" is not the correct term. It simply hangs and requires a power cycle to boot. Fortunately, we have mork on a networked power strip and can power cycle it remotely. Upon boot, there are no footprints whatsoever as to the cause of the hang. This sounds like hardware. So today we are going to bring mork down to swap out all of the memory and remove a couple of unused components in a desperate attempt to fix the problem. The forums of course will be down during this operation."


So, be prepared for some more downtime this afternoon.  Hopefully all goes well with the mork RAM transplant.

Meow meow for now.

[Cosmic_Ocean]

I've actually been thinking about this for a little while now, but I think it is confirmed that there is in fact gigabit going up the hill to SSL now (don't know if that includes the Hurricane Electronics link as well), but SAH is still limiting their usage to 100mbit?  I'm thinking if maybe they could increase it to say.. 150 or 200, downloads would end up going through faster and therefore not have the pipe maxed out for as long.  Or possibly putting the scheduler on a separate VLAN so there are no packet contention issues when the pipe is maxed out for downloads.  Observations over the past few years have shown that most of the problems with ghosts or just failed transfers happens when there are a ton of downloads taking place.

Maybe one possible idea is to cycle the scheduler and download processes.  Just do something like 15 or 20 minutes with the scheduler enabled, but downloads disabled, so you get work assigned, but can't download it, but then turn the scheduler off and let downloads run for an hour or two, and repeat until the pipe is no longer maxed out.  It's a thought.

My area of expertise is in networking, and it seems like there are several ways to probably fix, or at the very least, smooth out some of the rough edges.

No doubt that Oscar will make a marked/noticeable improvement over the current setup, but it seems unlikely that one server will fix everything, especially since most of the problems are more network-related than software-related.

Just my opinion.

[msattler]

Bandwidth is NOT the problem right now.....
It's server stability......most notably mork's lack of it.
If everything hangs together on the server side, we can live with the bandwidth currently available.

The problem has been with the servers crashing Seti-side, not the bandwidth.  It may contribute to the ghost problem, but until the servers are all stable, that cannot be established.

Hang in there folks, it's gonna be a bumpy ride.   LOL.

[perryjay]

    Dan has contacted HP and Sun to see if they can give us a deep discount on a machine that could replace mork, hopefully deep enough that we can purchase it on what's remaining from the donations made in the Number Crunching threads.



Good luck Dan, that would be great, sort of like two for the price of one. I hope it doesn't cut into the amount of RAM and harddrives you guys were going to get with Oscar though. At least not too much. If it does let us know so we can wake up Kittyman to start another drive.
____________

Eric posted the first part of this and the second was my reply in tech news. Sure would be great to have two new servers, they sure are needed.
As to the bandwidth, the line they ran up was for the whole lab, guess they didn't run the Hurricane link while they were at it. That's another thing that would really help.

[msattler]

I have started a new Seti server fund drive in the NC........
Please participate if you can.

The project needs your help.

Meow meow.

[Zeus Fab3r]

I have started a new Seti server fund drive in the NC........
Please participate if you can.

The project needs your help.

Meow meow.

Good job Mark ! There are 5K+$ already raised. I'll see what I can do, without robbing the bank.

[Gizbar]

I wish I could rob the bank, not that they have got much money at the moment!

Someone in the UK won our Euromillions lottery last night, and will wake up £113 million richer this morning.

I'm just sorry to say it wasn't me.

The servers really wouldn't have been a problem then...

[msattler]

The response, again......has been simply amazing.

Simply amazing.

Those that luv this project are wonderful folks.

You, of course, are too. 

This little planet and the hope that we all cling to make it worth the trip.

Meow, meow.

[msattler]

They kicked me......must have been too verbose for their tastes/

But, I'll leave you with this....

http://www.youtube.com/watch?v=alGx-O3MBzY&feature=related

Tastyest guitar I have found in ages.