SETI MB CUDA for Linux

[Hefto99]

Here are my results....   my_science_app1 is setiathome-CUDA-6.08.x86_64-pc-linux-gnu

AMD Athlon X2 3800+ (2 GHz), GeForce 8600 GT, openSUSE 11.1 64-bit



[attachment deleted by admin]

[Raistmer]

And your system is? (CPU/GPU )

[Raistmer]

The same tasks with Windows app on Q9450/9600GSO:

AK_v8b_win_SSSE3x.exe -verb -st / testWU-1.wu :
Started at  : 17:00:17.618
Ended at    : 17:02:04.759
    107.125 secs Elapsed
    105.113 secs CPU time
MB_6.08_mod_CPU_team_CUDA.exe -verb -st / testWU-1.wu :
Started at  : 17:02:04.790
Ended at    : 17:03:07.923
     63.118 secs Elapsed
     18.159 secs CPU time

AK_v8b_win_SSSE3x.exe -verb -st / testWU-2.wu :
Started at  : 17:03:08.017
Ended at    : 17:05:13.909
    125.876 secs Elapsed
    123.849 secs CPU time
MB_6.08_mod_CPU_team_CUDA.exe -verb -st / testWU-2.wu :
Started at  : 17:05:13.940
Ended at    : 17:06:26.932
     72.977 secs Elapsed
     18.798 secs CPU time

AK_v8b_win_SSSE3x.exe -verb -st / testWU-3.wu :
Started at  : 17:06:27.026
Ended at    : 17:08:40.515
    133.474 secs Elapsed
    131.462 secs CPU time
MB_6.08_mod_CPU_team_CUDA.exe -verb -st / testWU-3.wu :
Started at  : 17:08:40.546
Ended at    : 17:09:59.591
     79.030 secs Elapsed
     20.015 secs CPU time
Speedup     : 84.78%
Ratio       : 6.57 x

AK_v8b_win_SSSE3x.exe -verb -st / testWU-4.wu :
Started at  : 17:09:59.685
Ended at    : 17:10:38.264
     38.563 secs Elapsed
     36.535 secs CPU time
MB_6.08_mod_CPU_team_CUDA.exe -verb -st / testWU-4.wu :
Started at  : 17:10:38.295
Ended at    : 17:11:14.175
     35.864 secs Elapsed
     15.865 secs CPU time

AK_v8b_win_SSSE3x.exe -verb -st / testWU-5.wu :
Started at  : 17:11:14.269
Ended at    : 17:13:21.112
    126.828 secs Elapsed
    124.816 secs CPU time
MB_6.08_mod_CPU_team_CUDA.exe -verb -st / testWU-5.wu :
Started at  : 17:13:21.143
Ended at    : 17:14:34.510
     73.351 secs Elapsed
     18.439 secs CPU time

AK_v8b_win_SSSE3x.exe -verb -st / testWU-7.wu :
Started at  : 17:16:58.498
Ended at    : 17:17:58.387
     59.857 secs Elapsed
     57.845 secs CPU time
MB_6.08_mod_CPU_team_CUDA.exe -verb -st / testWU-7.wu :
Started at  : 17:17:58.418
Ended at    : 17:18:39.898
     41.465 secs Elapsed
     16.021 secs CPU time

[Hefto99]

AMD Athlon X2 3800+ (2 GHz), GeForce 8600 GT, openSUSE 11.1 64-bit

[Hefto99]

Hi all,

is it possible to autokill VLAR WUs with this Linux application? My system is very laggy with these long Work Units... 

Thanks for info,
H99

[sunu]

Autokill unfortunately no. You can do it with a script or manually (make a search for <true_angle_range>0.01 or 0.00 and delete or abort them).

[rja]

I realize that the app_info.xml files are hints, but is there a reason for the MB file to differ from the MB+AP file?

Should the version_num, avg_ncpus, and max_ncpus match between the MB+AP app_info.xml and MB app_info.xml files in setiathome-CUDA-6.08.i686.tar.bz2?  Same for setiathome-CUDA-6.08.x86_64.tar.bz2?

The MB+AP app_info.xml has version_num of 607 while the MB app_info.xml has version_num of 608.

The version 603 MB+AP avg_ncpus, and max_ncpus are set to 1.0000 while the MB avg_ncpus, and max_ncpus are set to 0.040000.

[ML1]

My CPU is running on 2GHz (Athlon X2 3800+), GK is 8600 GT on default clocks, openSUSE 11.1 64-bit, here are some results:

[...]

CPU utilization is almost 100% for SETI

I see pretty much the same on my system for an AthlonXP 6400+ and 8600GT GPU (256 MB VRAM).

Is the CPU doing a busy-wait poll of the GPU? Or why the high CPU utilisation?

As an experiment I'm keeping the CPU priority down to nice 19 (instead of the default 10) to see if there is any slowdown for the CUDA processing. However, that only reduces the CPU load to between 75% and 90% for a core.

Is there any profiling that we can run to see what it is doing with the CPU time?

Happy crunchin',
Martin

[ML1]

I see pretty much the same [100% CPU] on my system for an AthlonXP 6400+ and 8600GT GPU (256 MB VRAM).

Is the CPU doing a busy-wait poll of the GPU? Or why the high CPU utilisation?

As an experiment I'm keeping the CPU priority down to nice 19 (instead of the default 10) to see if there is any slowdown for the CUDA processing. However, that only reduces the CPU load to between 75% and 90% for a core...

And for a brief comparison of a very few examples (sorting by AR):

04-Feb-2009 20:28:20 04-Feb-2009 20:43:07 2.7155224489909 19dc08ac.31914.13160.15.8.44
05-Feb-2009 17:48:16 05-Feb-2009 18:02:44 2.7155504718476 17dc08ae.20201.15207.8.8.172
07-Feb-2009 09:34:39 07-Feb-2009 09:53:13 2.7155603822925 17dc08ae.1228.13162.7.8.1
07-Feb-2009 11:07:38 07-Feb-2009 11:26:01 2.7155603822925 17dc08ae.1228.13162.7.8.143
07-Feb-2009 11:26:01 07-Feb-2009 11:43:35 2.7155603822925 17dc08ae.1228.13162.7.8.149
07-Feb-2009 10:11:51 07-Feb-2009 10:30:33 2.7155603822925 17dc08ae.1228.13162.7.8.7
03-Feb-2009 18:19:10 03-Feb-2009 18:35:18 2.7155918111126 20dc08ae.27884.20931.9.8.10
03-Feb-2009 19:23:17 03-Feb-2009 19:38:40 2.7155918111126 20dc08ae.27884.20931.9.8.11

07-Feb onwards is at nice 19 and 75% and 90% CPU on one core. So roughly, 15mins WUs go up to be about 19mins.

05-Feb-2009 23:24:24 05-Feb-2009 23:57:35 0.70474762854275 16dc08ad.2380.10706.11.8.3
05-Feb-2009 22:51:12 05-Feb-2009 23:24:24 0.7330587117685 16dc08ad.2380.11115.11.8.6
07-Feb-2009 04:03:45 07-Feb-2009 04:41:28 0.81850781128826 21dc08ab.5849.11933.10.8.173
06-Feb-2009 10:13:58 06-Feb-2009 10:43:48 0.82366885623122 21dc08ab.11148.20931.4.8.88
07-Feb-2009 02:27:27 07-Feb-2009 03:04:36 0.82505766342312 21dc08ab.5849.20931.10.8.31
05-Feb-2009 22:21:52 05-Feb-2009 22:51:12 0.84615758941882 16dc08ac.28940.4571.10.8.84
06-Feb-2009 05:54:03 06-Feb-2009 06:23:36 0.86971373346817 20dc08ab.30334.20931.11.8.13

And 33mins is pushed up to 38mins, & 30mins -> 37mins...

06-Feb-2009 19:11:13 06-Feb-2009 19:56:01 0.43305205747667 16no08ag.22317.25021.15.8.184
07-Feb-2009 03:04:36 07-Feb-2009 04:03:45 0.43362818142428 01dc08aa.19405.7025.3.8.251
07-Feb-2009 01:31:22 07-Feb-2009 02:27:27 0.43362879013513 01dc08aa.19405.4571.3.8.98
06-Feb-2009 18:24:59 06-Feb-2009 19:11:13 0.43418781810648 16dc08ac.14209.240437.9.8.152
06-Feb-2009 17:39:39 06-Feb-2009 18:24:59 0.43418793067813 16dc08ac.14209.238801.9.8.234
06-Feb-2009 22:16:11 06-Feb-2009 23:01:44 0.43422120716759 16dc08ac.28940.243300.10.8.199
07-Feb-2009 04:41:29 07-Feb-2009 05:40:32 0.43430233574049 16dc08ac.26815.244118.11.8.113

45mins -> 56-59mins...


Sooo... The slowdown looks to be roughly proportionate to the lower proportion of CPU used...

Are the nVidia Windows drivers so very much more efficient than their compiles for Linux?

Or are there very many frequent busy-waits for many small GPU steps?

Happy crunchin',
Martin

[Raistmer]

For my GPU -poll mod is useless:

http://setiathome.berkeley.edu/forum_thread.php?id=51712&nowrap=true#863155

[rja]

I tried the MB only version of setiathome-CUDA-6.08.i686.tar.bz2 and got the GaussFit_kernel errors that smurf mentioned earlier in « Reply #7 on: 21 Jan 2009, 05:53:48 pm » after about 14 minutes or processing.

It has an Nvidia 8800 GTS 640MB.  The GPU core temp is usually 68C, but went to 80C when processing a CUDA workunit.  There was a single cpu process of setiathome running at 100%.

Would it help to use a newer version of boinc than 6.4.5?

Fedora 9, nvidia-graphics180.22-kmdl-2.6.27.12-78.2.8.fc9-180.22-106.fc9.i686 from atrpms, boinc-client-6.4.5-2.20081217svn.fc9.i386 from Fedora, with the latest Nvidia cudatoolkit_2.1_linux32_fedora9.run, for this computer - http://setiathome.berkeley.edu/show_host_detail.php?hostid=4131779

Here is an example taskid http://setiathome.berkeley.edu/result.php?resultid=1154715237

I tried removing the cuda 2.1 libs and using the cudalibs (2.0?) that were in setiathome-CUDA-6.08.i686.tar.bz2, but got the same GaussFit_kernel errors.

This was in a different workunit's slots/0/stderr.txt before it was uploaded:

SETI@home MB CUDA 608 Linux 32bit SM 1.0 - r06 by Crunch3r :p

setiathome_CUDA: Found 1 CUDA device(s):
   Device 1 : GeForce 8800 GTS
           totalGlobalMem = 670760960
           sharedMemPerBlock = 16384
           regsPerBlock = 8192
           warpSize = 32
           memPitch = 262144
           maxThreadsPerBlock = 512
           clockRate = 1188000
           totalConstMem = 65536
           major = 1
           minor = 0
           textureAlignment = 256
           deviceOverlap = 0
           multiProcessorCount = 12
setiathome_CUDA: CUDA Device 1 specified, checking...
   Device 1: GeForce 8800 GTS is okay
SETI@home using CUDA accelerated device GeForce 8800 GTS
setiathome_enhanced 6.01 Revision: 402 g++ (GCC) 4.1.2 20070925 (Red Hat 4.1.2-33)
libboinc: BOINC 6.5.0

Work Unit Info:
...............
WU true angle range is :  0.447901
Optimal function choices:
-----------------------------------------------------
name               
-----------------------------------------------------
              v_BaseLineSmooth (no other)
  v_vGetPowerSpectrumUnrolled2 0.00010 0.00000
             sse1_ChirpData_ak 0.00814 0.00098
                 v_vTranspose4 0.00381 0.00000
                BH SSE folding 0.00144 0.00000
Cuda error 'GaussFit_kernel' in file './cudaAcc_gaussfit.cu' in line 506 : invalid configuration argument.
Cuda error 'GaussFit_kernel' in file './cudaAcc_gaussfit.cu' in line 506 : invalid configuration argument.
and lots more of these Cuda error 'GaussFit_kernel' lines

[zjones]

I have been trying to get the SETI MB CUDA client running and ever work unit so far immediately hits a computation error as soon as it trys to crunch.

The error log looks like this:

<core_client_version>6.4.5</core_client_version>
<![CDATA[
<message>
process exited with code 127 (0x7f, -129)
</message>
<stderr_txt>
setiathome-CUDA-6.08.x86_64-pc-linux-gnu: error while loading shared libraries: libcufft.so.2: invalid ELF header

</stderr_txt>
]]>


The machine has a AMD Opteron 248, with 2 Quadro 5600 FX. These are equivalent to GeForce 8800 GTXs and G80 chipsets.  I am using CentOS 5.2 (x86_64) with Linux Kernel 2.6.18-8.el5.  I am using NVIDIA drivers x86_64-180.22 and CUDA toolkit 2.1 64-bit.  I have tried BOINC clients 6.4.5 and 6.6.2.  I have tried using the CUDA toolkit libs and the ones provided in Crunch3r's package. 

Have any of you run into this problem and/or have any suggestions?

Thanks.

[ML1]

I have been trying to get the SETI MB CUDA client running and ever work unit so far immediately hits a computation error as soon as it trys to crunch.

The error log looks like this:

[...]
setiathome-CUDA-6.08.x86_64-pc-linux-gnu: error while loading shared libraries: libcufft.so.2: invalid ELF header
[...]

The machine has a AMD Opteron 248, with 2 Quadro 5600 FX. These are equivalent to GeForce 8800 GTXs and G80 chipsets.  I am using CentOS 5.2 (x86_64) with Linux Kernel 2.6.18-8.el5.  I am using NVIDIA drivers x86_64-180.22 and CUDA toolkit 2.1 64-bit.  I have tried BOINC clients 6.4.5 and 6.6.2.  I have tried using the CUDA toolkit libs and the ones provided in Crunch3r's package. 

Have any of you run into this problem and/or have any suggestions?

'Tis working fine here and for the same revision for the nVidia drivers.

Kernel 2.6.18 is from a while ago now... It could well be that Crunch3r has used a much more recent kernel and a more recent version of gcc for his compiles.

Can you try a more recent kernel/distro?

I'm using Mandriva 2.6.27.7-server-1mnb.

Good luck,
Martin

[CorranHorn]

It's the same problem on my computer.

http://setiathome.berkeley.edu/show_host_detail.php?hostid=4023395

Cuda is ok

chess@chess-desktop:~/Documents/setiathome-CUDA-6.08.x86_64$ ldd setiathome-CUDA-6.08.x86_64-pc-linux-gnu
   linux-vdso.so.1 =>  (0x00007ffff01ff000)
   libcufft.so.2 => /usr/lib/libcufft.so.2 (0x00007f80e7cbc000)
   libcudart.so.2 => /usr/lib/libcudart.so.2 (0x00007f80e7a7e000)
   libcuda.so.1 => /usr/lib/libcuda.so.1 (0x00007f80e7607000)
   libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007f80e72fa000)
   libm.so.6 => /lib/libm.so.6 (0x00007f80e7075000)
   libpthread.so.0 => /lib/libpthread.so.0 (0x00007f80e6e59000)
   libc.so.6 => /lib/libc.so.6 (0x00007f80e6ae7000)
   libdl.so.2 => /lib/libdl.so.2 (0x00007f80e68e3000)
   libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007f80e66cb000)
   librt.so.1 => /lib/librt.so.1 (0x00007f80e64c2000)
   libz.so.1 => /usr/lib/libz.so.1 (0x00007f80e62aa000)
   /lib64/ld-linux-x86-64.so.2 (0x00007f80e7fd6000)

OS

Operating System   Linux
2.6.27-11-generic

[dtiger]

I crunshed several units. One was already correctly validated. The others have error messages:
<core_client_version>6.4.5</core_client_version>
<![CDATA[
<stderr_txt>
 file './cudaAcc_gaussfit.cu' in line 506 : invalid configuration argument.
Cuda error 'GaussFit_kernel' in file './cudaAcc_gaussfit.cu' in line 506 : invalid configuration argument.
Cuda error 'GaussFit_kernel' in file './cudaAcc_gaussfit.cu' in line 506 : invalid configuration argument.
Cuda error 'GaussFit_kernel' in file './cudaAcc_gaussfit.cu' in line 506 : invalid configuration argument.
Cuda error 'GaussFit_kernel' in file './cudaAcc_gaussfit.cu' in line 506 : invalid configuration argument.
Cuda error 'GaussFit_kernel' in file './cudaAcc_gaussfit.cu' in line 506 : invalid configuration argument.
....

I have a GeForce 8800 GT. Whats wrong here ? 

Now I updated to 180.22. But same situation. The card is not overclocked and temperature is 62 C. There are two types of cases:
- small WUs with a time to completion of ~7 min are running fine without any errors and granted credit is the same as claimed credit (14-15)
- big WUs with a time to completion of ~25 min have the gausfit error message and the granted credit is smaller than the claimed credit (claimed ~50, granted ~40)

I have exactly the same situation.
Video card is 8600 GT 256 MB, NVIDIA 180.29 driver.
Exactly the same uncountable error messages in stderr out, the same clamed/granted credits.

http://setiathome.berkeley.edu/result.php?resultid=1170131986