SETI MB CUDA for Linux

[IanJ]

Guys,
 Forgive if this is not the place for posting questions, but from what I read here I think it is.
 I have installed the Crunchr CUDA app on my FedoraCore10 64bit machine. After a fair amount of grief with segfaults, today I finally managed to get my first result in. However two of my results this morning have a strange error. Could anyone elaborate on what the problem is and what I should do to fix it. The card is a 9600GT. Here is the output:-

<core_client_version>6.6.36</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)
</message>
<stderr_txt>

SETI@home MB CUDA_2.2 608 Linux 64bit SM 1.0 - r12 by Crunch3r :p
VLAR autokill mod

setiathome_CUDA: Found 1 CUDA device(s):
   Device 1 : GeForce 9600 GT
           totalGlobalMem = 536608768
           sharedMemPerBlock = 16384
           regsPerBlock = 8192
           warpSize = 32
           memPitch = 262144
           maxThreadsPerBlock = 512
           clockRate = 1625000
           totalConstMem = 65536
           major = 1
           minor = 1
           textureAlignment = 256
           deviceOverlap = 1
           multiProcessorCount = 8
setiathome_CUDA: CUDA Device 1 specified, checking...
   Device 1: GeForce 9600 GT is okay
SETI@home using CUDA accelerated device GeForce 9600 GT
setiathome_enhanced 6.01 Revision: 402 g++ (GCC) 4.2.1 (SUSE Linux)
libboinc: BOINC 6.7.0

Work Unit Info:
...............
WU true angle range is :  0.388520
Cuda error 'cudaAcc_CalcChirpData_kernel2' in file './cudaAcc_CalcChirpData.cu' in line 106 : unspecified launch failure.
cufft: ERROR: /root/cuda-stuff/sw/rel/gpgpu/toolkit/r2.1/cufft/src/execute.cu, line 1070
cufft: ERROR: CUFFT_EXEC_FAILED
cufft: ERROR: /root/cuda-stuff/sw/rel/gpgpu/toolkit/r2.1/cufft/src/execute.cu, line 1070
cufft: ERROR: CUFFT_EXEC_FAILED
cufft: ERROR: /root/cuda-stuff/sw/rel/gpgpu/toolkit/r2.1/cufft/src/cufft.cu, line 151
cufft: ERROR: CUFFT_EXEC_FAILED
CUFFT error in file './cudaAcc_fft.cu' in line 62.

</stderr_txt>
]]>

 Thanks
 Ian
 

[sunu]

Thanks!

Just yesterday I put the new ap 5.06 in, but I haven't got any astropulse workunits yet. So currently only MB.

My load averages are above 4 and usually below 5.5

A GTX285 should be able to do 10000-14000 RAC alone.

With all those stuff running in your desktop I don't know if it would be a good idea to buy a low non-CUDA capable card for X and have your other 2 cards dedicated to CUDA. Of course your motherboard would need 3 PCI-E slots.

EDIT:

...
SETI@home MB CUDA_2.2 608 Linux 64bit SM 1.0 - r12 by Crunch3r :p
VLAR autokill mod
...
cufft: ERROR: /root/cuda-stuff/sw/rel/gpgpu/toolkit/r2.1/cufft/src/execute.cu, line 1070
cufft: ERROR: CUFFT_EXEC_FAILED
cufft: ERROR: /root/cuda-stuff/sw/rel/gpgpu/toolkit/r2.1/cufft/src/execute.cu, line 1070
cufft: ERROR: CUFFT_EXEC_FAILED
cufft: ERROR: /root/cuda-stuff/sw/rel/gpgpu/toolkit/r2.1/cufft/src/cufft.cu, line 151
cufft: ERROR: CUFFT_EXEC_FAILED
...

You're probably running the 2.2 cuda app with 2.1 libraries. Get the newer 2.2 or even better the 2.3 cuda libraries. Also you'll have to upgrade your NVIDIA driver to a 2.2 (185.18.xx) or 2.3 (190.xx) compatible one.

[IanJ]

Sunu,
 I'll try with the 2.2. I've installed Cuda Driver 185.18.14 (2.2? from the nvidia website). Previously installed I had 185.18.36. I have now installed the Cuda Toolkit 2.2, set my PATH and amended ldconfig.
 I now await tasks from SETI, at the moment it's out of work.
 Ian

[riofl]

well problem is i am now spoiled.. i had an 8600gt 256mb card i used for my desktop and ran the tesla for cuda before i got the 285. the 285 is several orders of magnitude better in desktop performance. i think i would rather just replace the tesla with a 2nd 285 and let that one crunch full speed and let this one do as it can. would still be a large improvement over the tesla in the 2nd slot. either that or maybe buy a motherboard with 3 slots that can take 3 of these cards leaving room for them to breathe and get a gtx 260 to use for my desktop and minor cuda crunching and let both 285 have at it full steam. i expect the 260 should be up to the task for my desktops.

[sunu]

I've installed Cuda Driver 185.18.14 (2.2? from the nvidia website). Previously installed I had 185.18.36.

185.18.14 is older than 185.18.36, why rollback? Also try cuda 2.3 with 190.xx driver, it's faster than 2.2.

well problem is i am now spoiled.. i had an 8600gt 256mb card i used for my desktop and ran the tesla for cuda before i got the 285. the 285 is several orders of magnitude better in desktop performance. i think i would rather just replace the tesla with a 2nd 285 and let that one crunch full speed and let this one do as it can. would still be a large improvement over the tesla in the 2nd slot. either that or maybe buy a motherboard with 3 slots that can take 3 of these cards leaving room for them to breathe and get a gtx 260 to use for my desktop and minor cuda crunching and let both 285 have at it full steam. i expect the 260 should be up to the task for my desktops.

Or maybe get a GTX295 in place of tesla and no need for a new motherboard.

TO ALL
Please see this thread and take proper action (abort those workunits): I've lost quite a few credits because of this.

[riofl]

yeah a 295 is an option. then it can dual crunch away and let the 285 'limp' along .. an idea to consider.. i suppose i could take the $ for that from my savings for my new project next year.. would also give me some experience with that monster will have to look and see if my current psu can handle it and the 285 and all the other things i have going on. just got it 2 months ago... kinda hate to replace it already.

[riofl]

TO ALL
Please see this thread and take proper action (abort those workunits): I've lost quite a few credits because of this.

hmmm guess a bit of all the probs people have is the data supplied. shame, but then again with the massive amount of chopping and adjusting the master data i can imagine errors creep in.

will have to check this out. i do get a number of units aborting with computation errors, usually 3 or 4 a day, but i have attributed those to the flaky tesla.. havent checked them all just a few and then gave up on it.

[Richard Haselgrove]

Badly-prepared data is actually pretty rare at SETI - that's why I made such a point of drawing that set to Eric's attention.

The point Sunu was making is that those WUs don't error out while crunching: they run full duration, and then error out when the time comes to upload the results. That's why they're a waste of time.

[riofl]

Badly-prepared data is actually pretty rare at SETI - that's why I made such a point of drawing that set to Eric's attention.

The point Sunu was making is that those WUs don't error out while crunching: they run full duration, and then error out when the time comes to upload the results. That's why they're a waste of time.

ahh. yeah.

[riofl]

any preferences in brand on the 295?  i was thinking of going with xfx only because my 285 is an xfx black edition.. also looked at asus and evga

[sunu]

yeah a 295 is an option. then it can dual crunch away and let the 285 'limp' along .. an idea to consider.. i suppose i could take the $ for that from my savings for my new project next year..

Well if you want RAC right here, right now a GTX295 is probably your best choice with your current setup (as it was the case with mine). I was "forced" to upgrade because my GTX280 burned about 2 months ago.

You have a future project in mind so if I were you I would pursue that. End of 2009 beginning of 2010 we will have the update to Nehalem processors while NVIDIA is going to release its next generation of graphics cards (about Christmas time) with the next generation dual card probably in the 1st quarter of 2010.

any preferences in brand on the 295?  i was thinking of going with xfx only because my 285 is an xfx black edition.. also looked at asus and evga

My GTX280 that burned was EVGA but I stayed with them. Both my GTX285 and GTX295 are EVGA. XFX should be good and ASUS even more. If you take the plunge and buy now prefer someone who gives you a step-up option as you might catch the new NVIDIA cards when they will be released, so EVGA, BFG or XFX (don't know if XFX has a step-up program).

[riofl]

yeah a 295 is an option. then it can dual crunch away and let the 285 'limp' along .. an idea to consider.. i suppose i could take the $ for that from my savings for my new project next year..

Well if you want RAC right here, right now a GTX295 is probably your best choice with your current setup (as it was the case with mine). I was "forced" to upgrade because my GTX280 burned about 2 months ago.

You have a future project in mind so if I were you I would pursue that. End of 2009 beginning of 2010 we will have the update to Nehalem processors while NVIDIA is going to release its next generation of graphics cards (about Christmas time) with the next generation dual card probably in the 1st quarter of 2010.

the future project will take a while to get off the ground as well as gathering parts here and there as i can in addition to any bulk buys i can do. it wont happen in jan but will begin gathering parts and probably will begin assembly in march.

any preferences in brand on the 295?  i was thinking of going with xfx only because my 285 is an xfx black edition.. also looked at asus and evga

My GTX280 that burned was EVGA but I stayed with them. Both my GTX285 and GTX295 are EVGA. XFX should be good and ASUS even more. If you take the plunge and buy now prefer someone who gives you a step-up option as you might catch the new NVIDIA cards when they will be released, so EVGA, BFG or XFX (don't know if XFX has a step-up program).

don't know either but i most likely would stick with whatever i get now for this machine and will take a good hard look at all the new stuff before i commit on the new project. that one is hopefully going to give any cray supercomputers a run for their money. heh my family looks at me like i am insane.. ,instead of saving for retirement  since i'm the oldest of the 'kids', i'm spending on computers

[sunu]

that one is hopefully going to give any cray supercomputers a run for their money.

I hope we'll have a linux machine in the #1 spot of the top hosts list.

[riofl]

that one is hopefully going to give any cray supercomputers a run for their money.

I hope we'll have a linux machine in the #1 spot of the top hosts list.

that would be great!

ok i ran my 1 hr test with no activity in the desktop. i only entered 1 desktop and that was to keep an eye on the cpu/gpu temps. 1 strip of gkrellm is all i ran. nothing else open.

the way i did it was to set to not fetch work, then hit update to get rid of current finished units. then shut boinc down. rebooted computer to clean up memory and then with just that mentioned above, started boinc and let it sit for an hour. the units has approx the same completion times..... maybe the average might have been 1 or 2 min faster but mostly they were approximately the same. then i uploaded them, and went to find them in the task listings and could not... newest were sept 4. ill check again later in case they are delayed in posting, or maybe they dont post until a workday. not sure.

[riofl]

heh guess i will never comprehend boinc's logic it sets aside workunits due sept 21st in favor of completing workunits not due till oct 20th!  there must be some urgent flag the project sets in these that we dont  see. they are not processing in "high priority" mode. it seems to eventually get to the ones it sets aside, but it seems completely illogical to stop a wu that is 93% complete. makes no sense at all  to me.