SETI MB CUDA for Linux

[sunu]

pp, that's what I've been saying in all my posts to you, UPGRADE TO 185.18.14!!!

[pp]

 

I'm old school. I don't blindly upgrade everything to the latest version without thought. I need to understand what has changed and how it will affect the rest of my system. I tried 185.13 right now but got computation errors on every WU. Back to 180.60 for now. I'll have a look at 185.18.14 after some more research...

[pp]

185.19 also gives computation errors and it's the latest from nvidia's ftp. Guess my energy has to focus on the boinc version next... I'll just have to empty my queue before I continue experimenting.

[sunu]

No. The latest version is 185.18.14, trust me. But as I wrote in my other post the right driver is just the beginning. There is much more for a proper seti install.

I'll try to write a guide for seti cuda in linux and post it on a new thread.

[Richard Haselgrove]

185.19 also gives computation errors and it's the latest from nvidia's ftp. Guess my energy has to focus on the boinc version next... I'll just have to empty my queue before I continue experimenting.

The actual 'discovery post' is in a development area not generally accessible, but the facts seem clear enough:

Experiments with CUDA 2.2

I tried cuda 2.2 with 185.18.08 beta driver and our 64 bit linux app.

1. 185.18.08 driver isn't compatible with 2.1 cuda libraries. You have to install the 2.2 ones.
2. Current optimized app doesn't have any problem running with the new 2.2 libs.
3. With cuda 2.2, current linux app exhibits the same behavior with the windows app. It no longer uses a full core, only for the first few seconds and then cpu utilisation hovers around 0-2%. Now in linux we potentially (see #6 ) can also use a 4+1 config, not only 3+1 as it is now.
4. Computation time is better with 2.2. Using file creation/modification timestamps, as it is now impossible to get accurate computation times for 2.2 I got for a random 0.44 AR wu:
          2.1      9min 30sec
          2.2      8min 28sec
5. Results are strongly similar
6. While standalone it runs ok, under BOINC I couldn't make it run, I get instant computation errors.

So you need the v2.2 CUDA runtime and FFT library as well as the updated drivers. I don't know the answer to that point 6, though.

[pp]

Tried CUDA 2.2 with 185.18.14 and BOINC = computation error
Tried CUDA 2.2 with 180.60 and BOINC = doesn't use CUDA device despite being recognized

Did NOT try client standalone.

Me very tired now and back to CUDA 2.1 and 180.60 for now. I think I'll let other people to the experimenting for now. Thanks for your patience.

[s52d]

Huh, once every few days all CUDA WUs error out (libraries etc).
http://setiweb.ssl.berkeley.edu/result.php?resultid=1262888457

boinc restart, and all is OK.


libraries:

-rwxr-xr-x 1 iztok users 252680 2009-01-16 23:30 /usr/lib64/libcudart.so.2.1*
-rwxr-xr-x 1 iztok users 252680 2009-01-16 23:30 /home/iztok/boinc/projects/setiathome.berkeley.edu/libcudart.so.2*
-rwxr-xr-x 1 iztok users 252680 2009-01-16 23:30 /home/iztok/boinc/libcudart.so.2*
-rwxr-xr-x 1 iztok users 254912 2009-02-05 00:39 /home/iztok/boinc/BOINC64/libcudart.so*
lrwxrwxrwx 1 iztok users     16 2009-06-04 11:30 /usr/lib64/libcudart.so.2 -> libcudart.so.2.1*
lrwxrwxrwx 1 iztok users     14 2009-06-04 11:30 /usr/lib64/libcudart.so -> libcudart.so.2*

-rwxr-xr-x 1 iztok users 1150912 2009-01-16 23:30 /usr/lib64/libcufft.so.2.1*
-rwxr-xr-x 1 iztok users 1150912 2009-01-16 23:30 /home/iztok/boinc/projects/setiathome.berkeley.edu/libcufft.so.2*
-rwxr-xr-x 1 iztok users 1150912 2009-01-16 23:30 /home/iztok/boinc/libcufft.so.2*
lrwxrwxrwx 1 iztok users      15 2009-06-04 11:30 /usr/lib64/libcufft.so.2 -> libcufft.so.2.1*
lrwxrwxrwx 1 iztok users      13 2009-06-04 11:30 /usr/lib64/libcufft.so -> libcufft.so.2*
-rw-r--r-- 1 iztok users      76 2009-06-14 14:42 /home/iztok/boinc/slots/0/libcufft.so.2

are those two to be replaced with 2.2?
Where to get them?

configuration here: slackware64, NVIDIA 185.18.14, latest boinc ...

BR
Iztok

[sunu]

Huh, once every few days all CUDA WUs error out (libraries etc).
http://setiweb.ssl.berkeley.edu/result.php?resultid=1262888457
boinc restart, and all is OK.
Iztok

I haven't seen or heard something like this before. Might be the client goes out of memory but can't fall back to cpu crunching and throws all these errors? Just a guess. Look at the computing preferences in your account page and make sure that "Leave applications in memory while suspended?" is set to no.

[s52d]

Keep work in memory  is set to no: first to be checked a while ago when I started hunting errors.
It was allways set to NO.

BR
Iztok

[sunu]

Do you switch your wus back and forth between cpu and gpu? I catched these two wus:

http://setiweb.ssl.berkeley.edu/result.php?resultid=1260953059
http://setiweb.ssl.berkeley.edu/result.php?resultid=1260953058

In their stderr.txt they have messages from both AKv8 and cuda app. They are also both vlar.

Also I've read various bad things about 6.6.3x boinc versions. Maybe try a different boinc? I'm using 6.6.20.

[s52d]

Do you switch your wus back and forth between cpu and gpu? I catched these two wus:

http://setiweb.ssl.berkeley.edu/result.php?resultid=1260953059
http://setiweb.ssl.berkeley.edu/result.php?resultid=1260953058

I
Also I've read various bad things about 6.6.3x boinc versions. Maybe try a different boinc? I'm using 6.6.20.

Yes, I do. After failure (quick bunch of errors), I run out of cuda work, so perl script helps a bit.
I checked few, and they validate.

Now... every 20 minutes I check ps: if there are two setiathome-CUDA-6.08.x86_64-pc-linux-gnu running,
boinc is killed -TERM, and restarted after minute.

2.2 libraries?

BR
Iztok

log example, when my watchdog triggered.
boinc started two copies of 608 (cuda) on top of AK classic 603.

14-Jun-2009 22:43:42 [SETI@home] Computation for task 30mr09aa.25976.14387.11.8.3_1 finished
14-Jun-2009 22:43:42 [SETI@home] Starting 30mr09aa.25976.14387.11.8.0_1
14-Jun-2009 22:43:42 [SETI@home] Starting task 30mr09aa.25976.14387.11.8.0_1 using setiathome_enhanced version 608
14-Jun-2009 22:43:44 [SETI@home] Started upload of 30mr09aa.25976.14387.11.8.3_1_0
14-Jun-2009 22:43:52 [SETI@home] Finished upload of 30mr09aa.25976.14387.11.8.3_1_0
14-Jun-2009 22:45:26 [SETI@home] Computation for task 22mr09aa.12857.21340.12.8.191_1 finished
14-Jun-2009 22:45:26 [SETI@home] Starting 21mr09ac.19502.15201.14.8.227_1
14-Jun-2009 22:45:26 [SETI@home] Starting task 21mr09ac.19502.15201.14.8.227_1 using setiathome_enhanced version 608
14-Jun-2009 22:45:26 [SETI@home] Starting 22mr09aa.12210.890.14.8.72_0
14-Jun-2009 22:45:26 [SETI@home] Starting task 22mr09aa.12210.890.14.8.72_0 using setiathome_enhanced version 603
14-Jun-2009 22:45:28 [SETI@home] Started upload of 22mr09aa.12857.21340.12.8.191_1_0
14-Jun-2009 22:45:34 [SETI@home] Finished upload of 22mr09aa.12857.21340.12.8.191_1_0
14-Jun-2009 22:45:53 [SETI@home] Computation for task 21mr09ad.26644.132650.12.8.71_1 finished
14-Jun-2009 22:45:53 [SETI@home] Starting 22mr09aa.12210.890.14.8.81_0
14-Jun-2009 22:45:53 [SETI@home] Starting task 22mr09aa.12210.890.14.8.81_0 using setiathome_enhanced version 603
14-Jun-2009 22:45:55 [SETI@home] Started upload of 21mr09ad.26644.132650.12.8.71_1_0
14-Jun-2009 22:46:02 [SETI@home] Finished upload of 21mr09ad.26644.132650.12.8.71_1_0
14-Jun-2009 22:50:02 [---] Received signal 15
14-Jun-2009 22:50:02 [---] Exit requested by user

one minute might be to short: a bunch of errors after restart:

14-Jun-2009 22:51:15 [SETI@home] Starting task 14mr09ab.27226.7434.12.8.96_0 using setiathome_enhanced version 608
14-Jun-2009 22:51:18 [SETI@home] Computation for task 14mr09ab.27226.7434.12.8.96_0 finished
14-Jun-2009 22:51:18 [SETI@home] Starting 14mr09ab.27226.7434.12.8.85_0
14-Jun-2009 22:51:18 [SETI@home] Starting task 14mr09ab.27226.7434.12.8.85_0 using setiathome_enhanced version 608
14-Jun-2009 22:51:20 [SETI@home] Started upload of 14mr09ab.27226.7434.12.8.96_0_0
14-Jun-2009 22:51:21 [SETI@home] Computation for task 14mr09ab.27226.7434.12.8.85_0 finished
14-Jun-2009 22:51:21 [SETI@home] Starting 14mr09ab.27226.7434.12.8.57_0
14-Jun-2009 22:51:21 [SETI@home] Starting task 14mr09ab.27226.7434.12.8.57_0 using setiathome_enhanced version 608
14-Jun-2009 22:51:23 [SETI@home] Started upload of 14mr09ab.27226.7434.12.8.85_0_0
14-Jun-2009 22:51:24 [SETI@home] Computation for task 14mr09ab.27226.7434.12.8.57_0 finished
14-Jun-2009 22:51:24 [SETI@home] Starting 14mr09ab.27226.7434.12.8.91_0
14-Jun-2009 22:51:24 [SETI@home] Starting task 14mr09ab.27226.7434.12.8.91_0 using setiathome_enhanced version 608
14-Jun-2009 22:51:28 [SETI@home] Finished upload of 14mr09ab.27226.7434.12.8.96_0_0
14-Jun-2009 22:51:28 [SETI@home] Started upload of 14mr09ab.27226.7434.12.8.57_0_0
14-Jun-2009 22:51:28 [SETI@home] Computation for task 14mr09ab.27226.7434.12.8.91_0 finished
14-Jun-2009 22:51:28 [SETI@home] Starting 14mr09ab.27226.7434.12.8.56_0
14-Jun-2009 22:51:28 [SETI@home] Starting task 14mr09ab.27226.7434.12.8.56_0 using setiathome_enhanced version 608
14-Jun-2009 22:51:29 [SETI@home] Finished upload of 14mr09ab.27226.7434.12.8.85_0_0
14-Jun-2009 22:51:30 [SETI@home] Started upload of 14mr09ab.27226.7434.12.8.91_0_0
14-Jun-2009 22:51:31 [SETI@home] Computation for task 14mr09ab.27226.7434.12.8.56_0 finished
14-Jun-2009 22:51:31 [SETI@home] Starting 14mr09ab.27226.7434.12.8.67_0
14-Jun-2009 22:51:31 [SETI@home] Starting task 14mr09ab.27226.7434.12.8.67_0 using setiathome_enhanced version 608
14-Jun-2009 22:51:33 [SETI@home] Finished upload of 14mr09ab.27226.7434.12.8.57_0_0
14-Jun-2009 22:51:33 [SETI@home] Started upload of 14mr09ab.27226.7434.12.8.56_0_0
14-Jun-2009 22:51:34 [SETI@home] Finished upload of 14mr09ab.27226.7434.12.8.91_0_0
14-Jun-2009 22:51:34 [SETI@home] Computation for task 14mr09ab.27226.7434.12.8.67_0 finished

manual restart, a bit more waiting, seems OK

BR
Iztok

[sunu]

2.2 libraries?

What do you mean?

Also try a different boinc version.

[s52d]

2.2 libraries?

What do you mean?

Also try a different boinc version.

libraries:

-rwxr-xr-x 1 iztok users 252680 2009-01-16 23:30 /usr/lib64/libcudart.so.2.1*
-rwxr-xr-x 1 iztok users 252680 2009-01-16 23:30 /home/iztok/boinc/projects/setiathome.berkeley.edu/libcudart.so.2*
-rwxr-xr-x 1 iztok users 252680 2009-01-16 23:30 /home/iztok/boinc/libcudart.so.2*
-rwxr-xr-x 1 iztok users 254912 2009-02-05 00:39 /home/iztok/boinc/BOINC64/libcudart.so*
lrwxrwxrwx 1 iztok users     16 2009-06-04 11:30 /usr/lib64/libcudart.so.2 -> libcudart.so.2.1*
lrwxrwxrwx 1 iztok users     14 2009-06-04 11:30 /usr/lib64/libcudart.so -> libcudart.so.2*

-rwxr-xr-x 1 iztok users 1150912 2009-01-16 23:30 /usr/lib64/libcufft.so.2.1*
-rwxr-xr-x 1 iztok users 1150912 2009-01-16 23:30 /home/iztok/boinc/projects/setiathome.berkeley.edu/libcufft.so.2*
-rwxr-xr-x 1 iztok users 1150912 2009-01-16 23:30 /home/iztok/boinc/libcufft.so.2*
lrwxrwxrwx 1 iztok users      15 2009-06-04 11:30 /usr/lib64/libcufft.so.2 -> libcufft.so.2.1*
lrwxrwxrwx 1 iztok users      13 2009-06-04 11:30 /usr/lib64/libcufft.so -> libcufft.so.2*
-rw-r--r-- 1 iztok users      76 2009-06-14 14:42 /home/iztok/boinc/slots/0/libcufft.so.2

are those two to be replaced with 2.2?
Where to get them?

[sunu]

You can get the cuda 2.2 libraries from http://www.nvidia.com/object/cuda_get.html They are in the cuda toolkit.

Edit: From the boinc log you've posted above it seems more of a boinc problem. Try a different boinc version. Also can you post your app_info.xml?

[s52d]

ok, downgraded boinc to 6.4.5 (latest stable).

seems to worrk: 4 normal tasks and one CUDA, running with prioriti 10 and some low % of CPU time.

BR
Iztok

app_info: tried to hack 4 CPU and one CUDA.

<app_info>
<app>
<name>setiathome_enhanced</name>
</app>
<file_info>
<name>setiathome-CUDA-6.08.x86_64-pc-linux-gnu</name>
<executable/>
</file_info>
<file_info>
<name>libcudart.so.2</name>
<executable/>
</file_info>
<file_info>
<name>libcufft.so.2</name>
<executable/>
</file_info>
<file_info>
<name>AK_V8_linux64_ssse3</name>
<executable/>
</file_info>
<app_version>
<app_name>setiathome_enhanced</app_name>
<version_num>528</version_num>
<file_ref>
<file_name>AK_V8_linux64_ssse3</file_name>
<main_program/>
</file_ref>
</app_version>
<app_version>
<app_name>setiathome_enhanced</app_name>
<version_num>603</version_num>
<file_ref>
<file_name>AK_V8_linux64_ssse3</file_name>
<main_program/>
</file_ref>
</app_version>
<app_version>
<app_name>setiathome_enhanced</app_name>
<version_num>605</version_num>
<plan_class>cuda</plan_class>
<avg_ncpus>0.040000</avg_ncpus>
<max_ncpus>0.040000</max_ncpus>
<coproc>
<type>CUDA</type>
<count>1</count>
</coproc>
<file_ref>
<file_name>setiathome-CUDA-6.08.x86_64-pc-linux-gnu</file_name>
<main_program/>
</file_ref>
<file_ref>
<file_name>libcudart.so.2</file_name>
</file_ref>
<file_ref>
<file_name>libcufft.so.2</file_name>
</file_ref>
</app_version>
<app_version>
<app_name>setiathome_enhanced</app_name>
<version_num>606</version_num>
<plan_class>cuda</plan_class>
<avg_ncpus>0.040000</avg_ncpus>
<max_ncpus>0.040000</max_ncpus>
<coproc>
<type>CUDA</type>
<count>1</count>
</coproc>
<file_ref>
<file_name>setiathome-CUDA-6.08.x86_64-pc-linux-gnu</file_name>
<main_program/>
</file_ref>
<file_ref>
<file_name>libcudart.so.2</file_name>
</file_ref>
<file_ref>
<file_name>libcufft.so.2</file_name>
</file_ref>
</app_version>
<app_version>
<app_name>setiathome_enhanced</app_name>
<version_num>608</version_num>
<plan_class>cuda</plan_class>
<avg_ncpus>0.040000</avg_ncpus>
<max_ncpus>0.040000</max_ncpus>
<coproc>
<type>CUDA</type>
<count>1</count>
</coproc>
<file_ref>
<file_name>setiathome-CUDA-6.08.x86_64-pc-linux-gnu</file_name>
<main_program/>
</file_ref>
<file_ref>
<file_name>libcudart.so.2</file_name>
</file_ref>
<file_ref>
<file_name>libcufft.so.2</file_name>
</file_ref>
</app_version>
</app_info>