Help required! Cuda on new system crashes!

[efmer (fred)]

Thanks for the info, Fred.

I'm just happy at the moment that it's running successfully without crashing. Never noticed it swapping wu before that's all. If it's normal behaviour I can live with it. My RAC is taking a dive with all this mucking around going on, but I need to find out what is going on.

As posted earlier I went back to version 6.6.38 from version 6.6.41, which I notice has now been pulled from Boinc downloads for Windows systems. Have you tried the 6.10.17 version yet?

Want to get it running properly before messing with another new Boinc version.

regards, Gizbar.

I will wait some time because 6.6.38 works without any problems for me. It's mainly for AMD/ATI users.

[Claggy]

Gizbar,
I'd make sure you have Boinc 6.6.37 /.38 minimum, or go to 6.10.17, which is very stable now,
6.10.x was originally only getting ATI support, but has had a lot of other fixes / enhancement's since,
the one i like the most is the 'Show active Task' only button, cuts down on a lot of the traffic between
the boinc client and boinc manager.
I'm running it on both my Desktop and Laptop no problem, along with the new Beta 195.39 drivers,
again no problem there eithier, I'd try Boinc 6.10.17 first, then later with the new drivers,
and have a look if there are any later chipset drivers available as well.

The problems with Boinc 6.6.36 are that if you have a largie-ish cache, and you ask for Seti work
and if you get lots of shorties, then Boinc will go EDF on the GPU, start a shortie on the GPU,
it might complete some of it before switching to another since that is in worse deadline pressure the the first shortie,
that's O.K in itself, but the problem is that if Boinc switches GPU Wu's before the first wu has checkpointed,
then Boinc doesn't free up the GPU memory, and every GPU wu after that runs in CPU fallback mode,
taking a whole core,  meaning you now have more CPU tasks than cores,
(I don't think that's your problem through)
6.6.37 fixes the problem of GPU tasks going into CPU fallback mode, but not the problem of the actual switching,
6.10.17 fixes the problem with switching GPU tasks, GPU tasks now run not quite FIFO order,
they run in received order by date/time,  and subdivided into report deadline order.

Claggy

Edited and added more thoughts

[efmer (fred)]

Gizbar,
I'd make sure you have Boinc 6.6.37 /.38 minimum, or go to 6.10.17, which very stable now,
6.10.x was originally only getting ATI support, but has had a lot of other fixes / enhancement's since.

The problems with Boinc 6.6.36 are that if you have a largie-ish cache, and you ask for Seti work
and if you get lots of shorties, then Boinc will go EDF on the GPU, start a shortie on the GPU,
it might complete some of it before switching to another since that is in worse deadline pressure the the first shortie,
that's O.K in itself, but the problem is that if Boinc switches GPU Wu's before the first wu has checkpointed,
then Boinc doesn't free up the GPU memory, and every GPU wu after that runs in CPU fallback mode,
taking a whole core,  meaning you now have more CPU tasks than cores,
(I don't think that's your problem through)
6.6.37 fixes the problem of GPU tasks going into CPU fallback mode, but not the problem of the actual switching,
6.10.17 fixes the problem with switching GPU tasks, GPU tasks now run not quite FIFO order,
they run in received order by date/time,  and subdivided into report deadline order.

Claggy

Thanks, but the rapid pullback or the previous release got me a bit scared.

[Gizbar]

Thanks for the replies.

I've just heard from MarkJ on the Seti forum and he has explained that this could happen with the earlier versions of Boinc numbered 6.6.xx, and has been resolved in some of the later versions and suggested I upgrade to 6.10.17 as well. It has to do with the 'Task Switching Interval', which would let Boinc start a new task instead of running to completion. Please be aware I'm just relaying the information...

I think I'm proving that the card is stable on XPPro-32, it has been running for at least 2.5 hours now without a glitch, freeze, or crash. I'll leave it a bit longer and then start to try to install Win7-64HP again and see where that gets me to.

regards, Gizbar.

[efmer (fred)]

Thanks for the replies.

I've just heard from MarkJ on the Seti forum and he has explained that this could happen with the earlier versions of Boinc numbered 6.6.xx, and has been resolved in some of the later versions and suggested I upgrade to 6.10.17 as well. It has to do with the 'Task Switching Interval', which would let Boinc start a new task instead of running to completion. Please be aware I'm just relaying the information...

I think I'm proving that the card is stable on XPPro-32, it has been running for at least 2.5 hours now without a glitch, freeze, or crash. I'll leave it a bit longer and then start to try to install Win7-64HP again and see where that gets me to.

regards, Gizbar.

The latest beta driver from nVidia and I got a kernell error after a few minutes. Haven't seen one of those for some time. They still have work to do.

[Gizbar]

Hi Fred, I don't tend to mess around with the Beta drivers much. I only used 191.07 because that was an official release and whql'd by microsoft.

I'm happy that the card is now stable and running well on XPPro-32. Had it running for about 6.5 hours now (got waylaid by the last Grand Prix of the season, lol!) and it hasn't crashed or frozen or glitched once.

Now to try it on Win7-64HP again, and see if I can do a better job of installing it all than I did last time.

Might be offline for a bit.

regards, Gizbar.

[Gizbar]

New update! The story so far...

Ran successfully on XPPro-32 for approx 6.5 hours, so think I pretty much proved the card was stable.

Installed Win7-64HP on system. Told me an update had failed, and to re-run setup, choosing 'Get latest installation updates from internet'.

Did this and then installed outstanding updates. Then installed my preferred AV, which in my case is Avast.

Installed Nvidia drivers 191.07 for Win7-64.

Installed new partition just to install Boinc on. (Thanx Brodo!)
Then copied my installation of Boinc to the partition and installed 6.10.17.  Installed Lunatics v0.2 32bit and cuda 2.3 dll's. Didn't work too well first off, so spent some time getting it to reread configs and data etc. Did work the first time with XPPro-32, but was still using 6.6.38 then.

Then ran reschedule to get some GPU work as no work is coming from the servers.

Currently have run 3-4 wu's without failing so far... fingers xxxxxd.

Installed Firefox and update to be able to post this. Don't like IE, only use it when I have to.

Haven't installed anything else yet, even my preferred firewall. Just relying on Windows one, the router, and my AV at the mo.

Testing In Progress..... (but looking promising so far!)

regards, Gizbar.

[Gizbar]

Ok. Still not perfect.

Had a blue screen overnight (1st blue screen ever on Win7!) but hadn't stopped it rebooting after an error, and can't tell what it's failed on yet. But it did go for over 4.5 hours this time which is unprecedented in this story so far.

Will carry on troubleshooting, but it might mean a voltage boost for something along the line. Have to work today, so won't be about 'til later.

regards, Gizbar.

[Pappa]

Gizbar

Monitor Temps the CPU's Win 7 seems to be a bit more sensitive to over temp... experience...

[Gizbar]

Update:- 12 more hours on, no crashes. Run 4 wu + 1 cuda units all day, and everything still cooking on gas when I got home. Still playing a wait and see game, but many thanks to all suggestions, ideas, and solutions.

@Pappa - Thanks for the suggestion, but I was monitoring temps pretty closely anyway. Room temp was about 22c (72f) while testing, cpu was sitting around 50-52c (o/c from 3.2 to 3.6) that's approx 122-126f, and the GPU was going from approx 73-78c (163-173f in old money).  Was using Coretemp64 for cpu, and GPU-Z for Gfx card. Have to clean the dust out of the filters every 2-3 weeks anyway. It makes a big difference, and does stop a lot of it getting into the pc. I've got an Antec 902 case and it does have good airflow. I can increase it as well if need be, as it's not running full speed yet. I've had a lot of trouble with overheating in the past, and the Antec 902 and another Antec 300 have been the best cases I've ever had. Not saying they're perfect, but the best I've had, and whatever cases I get in the future will always have a 'blowhole' fan to exhaust hot air out of the top of the case.

regards, Gizbar.

[Gizbar]

24 hours on, no blue screens, no crashes, and am slowly re-installing all my programs. Everything seems ok now.

regards Gizbar.

[MarkJ]

Thanks for the info, Fred.

I'm just happy at the moment that it's running successfully without crashing. Never noticed it swapping wu before that's all. If it's normal behaviour I can live with it. My RAC is taking a dive with all this mucking around going on, but I need to find out what is going on.

As posted earlier I went back to version 6.6.38 from version 6.6.41, which I notice has now been pulled from Boinc downloads for Windows systems. Have you tried the 6.10.17 version yet?

Want to get it running properly before messing with another new Boinc version.

regards, Gizbar.

I have 5 systems running Win7 x64 with BOINC 6.10.25 (64 bit). I am using Nvidia 195.62 drivers. All seem fine. I was running 191.07 drivers before without any issues. I don't OC my hardware as they run hot enough as it is.

If you don't want to live quite on the bleeding edge use 6.10.18 which is fairly good and won't preempt the cuda tasks. There is a 64 bit multi-beam app in the downloads area and you'll find its faster than the 32 bit app for CPU work. The cuda app is still 32 bit.

[Gizbar]

Thanks for the info, MarkJ. I have a stack of work ready to report/upload due to the enforced power outage at Berkeley that started yesterday.  I don't want to trash my upload queue by upgrading it shoddily. How can I upgrade without disturbing all this work?

regards, Gizbar.

[Gizbar]

Boinc Site is now back up after the power outage, and even though I'be selected 'all versions' on the home page, it is still only giving me version 6.10.18 as the recommended version and 6.10.24 as the development version... where, oh where is 6.10.25? 

[Claggy]

Boinc Site is now back up after the power outage, and even though I'be selected 'all versions' on the home page, it is still only giving me version 6.10.18 as the recommended version and 6.10.24 as the development version... where, oh where is 6.10.25? 

http://boinc.berkeley.edu/dl/?C=M;O=D

Claggy