SETI MB CUDA for Linux

[Raistmer]

In windows the difference in first and second PCI-E slots (if first has motitor attached and second not) is:
GPU that used by Windows for video output will subject of 3 or 2 seconds timeout, but secong GPU will not.
Don't know if this relevant to Linux though.

[sunu]

GPU that used by Windows for video output will subject of 3 or 2 seconds timeout, but secong GPU will not.
Don't know if this relevant to Linux though.

Well, if it is because of the first gpu also drawing the screen then it will probably also exist in linux. We don't have a big sample of seti cuda users with multi gpus in linux. Actually the sample is non-existent 

What Tye describes might be some faulty config, strange driver behavior, or some weird motherboard-gpu-gpu hardware incompatibility.

[Raistmer]

Not sure it exist in linux. It's not GPU feature, it's windows feature - it will kill driver (Vista) with more than 2 secs of "no answer" from it.
Don't know if Linux kerner implements such watchdog machanism or not.
GPUs that don't output video don't subject of this "driver hung" check and can run long kernels. That's why surely not all that work OK on Tesla will work OK on user's GPUs (even if newly GPUs slightly faster than first released Teslas IMHO)

[b0b3r]

Hello everyone

Came across an interesting error message in task 1294937260 while researching something else.

SETI@home MB CUDA 608 Linux 64bit SM 1.0 - r06 by Crunch3r :p

Error: API mismatch: the NVIDIA kernel module has version 180.29,
but this NVIDIA driver component has version 180.60. 
...

Something to watch for when fiddling about with Linux drivers and modules.

The anonymous owner of host 5011059 seems to be having a real problem getting his or her GTX 295 running under gentoo.

With this host I don't have any problems. It just happen during system upgrade.

I have a real problem with host 5018683. I don't have any idea what's wrong. It isn't over clocked, or overheating. GPU-s have about 75C~77C at full load (~52C idle). And other CUDA programs are working fine, but with SETI almost all end with:

cufft: ERROR: /root/cuda-stuff/sw/rel/gpgpu/toolkit/r2.2/cufft/src/execute.cu, line 1070
cufft: ERROR: CUFFT_EXEC_FAILED
cufft: ERROR: /root/cuda-stuff/sw/rel/gpgpu/toolkit/r2.2/cufft/src/execute.cu, line 1070
cufft: ERROR: CUFFT_EXEC_FAILED
cufft: ERROR: /root/cuda-stuff/sw/rel/gpgpu/toolkit/r2.2/cufft/src/cufft.cu, line 147
cufft: ERROR: CUFFT_EXEC_FAILED
Cuda error 'cufftExecC2C' in file './cudaAcc_fft.cu' in line 63 : unspecified launch failure.
Cuda error 'cudaAcc_GetPowerSpectrum_kernel' in file './cudaAcc_PowerSpectrum.cu' in line 56 : unspecified launch failure.
Cuda error 'cudaAcc_GetPowerSpectrum_kernel' in file './cudaAcc_PowerSpectrum.cu' in line 56 : unspecified launch failure.
Cuda error 'cudaAcc_summax32_kernel' in file './cudaAcc_summax.cu' in line 148 : unspecified launch failure.
Cuda error 'cudaAcc_summax32_kernel' in file './cudaAcc_summax.cu' in line 148 : unspecified launch failure.
Cuda error 'cudaMemcpy(PowerSpectrumSumMax, dev_PowerSpectrumSumMax, cudaAcc_NumDataPoints / fftlen * sizeof(*dev_PowerSpectrumSumMax), cudaMemcpyDeviceToHost)' in file './cudaAcc_summax.cu' in line 161 : unspecified launch failure.

I will be thankful for any idea what's wrong and how to solve it.

[Raistmer]

FFT lib kernel launch failed, most probably incompatibility between CUDA RT and video driver used.

[b0b3r]

I tried these combinations of drivers and cuda:
- drv 180.29 with cuda 2.1
- drv 180.60 with cuda 2.1
- drv 185.18.14 with cuda 2.1
- drv 185.18.14 with cuda 2.2

And they all give same results. Strange thing is that just a few weeks back (drv 180.29 cuda 2.1) everything works fine, maybe there is something wrong with those results unit.

[sunu]

b0b3r as a start do an ldd of the seti client and post here as well as your xorg.0.log

[b0b3r]

I don't use Xorg on this machine. Here is ldd output:

linux-vdso.so.1 =>  (0x00007fffeb1ff000)
libcufft.so.2 => /opt/cuda/lib/libcufft.so.2 (0x00007f38e2c23000)
libcudart.so.2 => /opt/cuda/lib/libcudart.so.2 (0x00007f38e29e5000)
libcuda.so.1 => /usr/lib/libcuda.so.1 (0x00007f38e2518000)
libstdc++.so.6 => /usr/lib/gcc/x86_64-pc-linux-gnu/4.3.2/libstdc++.so.6 (0x00007f38e220d000)
libm.so.6 => /lib/libm.so.6 (0x00007f38e1f88000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00007f38e1d6c000)
libc.so.6 => /lib/libc.so.6 (0x00007f38e19f9000)
libdl.so.2 => /lib/libdl.so.2 (0x00007f38e17f5000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007f38e15de000)
librt.so.1 => /lib/librt.so.1 (0x00007f38e13d5000)
libz.so.1 => /lib/libz.so.1 (0x00007f38e11bf000)
/lib64/ld-linux-x86-64.so.2 (0x00007f38e2f3e000)

[b0b3r]

I found some information on google that it may be problem with memory leak in cufft, here is example.

[sunu]

I don't use Xorg on this machine.

What do you use? An

ls /dev/nv*

what does it give you?

[b0b3r]

ls -al /dev/nv*
crw-rw-rw- 1 root root 195,   0 Jul 12 20:01 /dev/nvidia0
crw-rw-rw- 1 root root 195,   1 Jul 12 20:01 /dev/nvidia1
crw-rw-rw- 1 root root 195,   2 Jul 12 20:01 /dev/nvidia2
crw-rw-rw- 1 root root 195, 255 Jul 12 20:01 /dev/nvidiactl

[sunu]

Your system sees three devices.

In your host 5018683, boinc doesn't even see your graphics cards. Are you sure that you have intalled them correctly?

Also in both of your hosts upgrade boinc. 6.4.5 is too old.

[pp]

Came across an interesting error message in task 1294937260 while researching something else.

SETI@home MB CUDA 608 Linux 64bit SM 1.0 - r06 by Crunch3r :p

Error: API mismatch: the NVIDIA kernel module has version 180.29,
but this NVIDIA driver component has version 180.60.  Please make
sure that the kernel module and all NVIDIA driver components
have the same version.
setiathome_CUDA: Found 1 CUDA device(s):
Cuda error 'cudaGetDeviceProperties( &cDevProp, i )' in file './cudaAcceleration.cu' in line 138 : initialization error.

Something to watch for when fiddling about with Linux drivers and modules.

The anonymous owner of host 5011059 seems to be having a real problem getting his or her GTX 295 running under gentoo.

He hasn't installed the NVIDIA drivers properly.

Well, actually he has. But after installation of the new package he neither rebooted nor loaded the new module. He's still running his system with the old version in memory. Easy mistake to do in Gentoo - been there done that. 

[sunu]

Not sure it exist in linux. It's not GPU feature, it's windows feature - it will kill driver (Vista) with more than 2 secs of "no answer" from it.
Don't know if Linux kerner implements such watchdog machanism or not.
GPUs that don't output video don't subject of this "driver hung" check and can run long kernels. That's why surely not all that work OK on Tesla will work OK on user's GPUs (even if newly GPUs slightly faster than first released Teslas IMHO)

Raistmer do you mean something like this? From cuda 2.2 release notes:

o Individual GPU program launches are limited to a run time
  of less than 5 seconds on a GPU with a display attached.
  Exceeding this time limit causes a launch failure reported
  through the CUDA driver or the CUDA runtime. GPUs without
  a display attached are not subject to the 5 second run time
  restriction. For this reason it is recommended that CUDA is
  run on a GPU that is NOT attached to an X display.

So yes, it also exists in linux.


@b0b3r: The error you posted above with the "unspecified launch failure" messages might be because of that.

Curiously I've crunched tens of thousands of workunits with my GPU that also runs X with ever seeing that kind of error.

[Jason G]

....Individual GPU program launches are limited to a run time
  of less than 5 seconds on a GPU with a display attached.....

  Yes.  In embedded microcontroller system terminology, that's called a "Watchdog Timer".  Crazy people program GPUs to take longer than that, Lunatics try to fix it.