+- +-
Say hello if visiting :) by Gecko
11 Jan 2023, 07:43:05 pm

Seti is down again by Mike
09 Aug 2017, 10:02:44 am

Some considerations regarding OpenCL MultiBeam app tuning from algorithm view by Raistmer
11 Dec 2016, 06:30:56 am

Loading APU to the limit: performance considerations by Mike
05 Nov 2016, 06:49:26 am

Better sleep on Windows - new round by Raistmer
26 Aug 2016, 02:02:31 pm

Author Topic: SETI MB CUDA for Linux  (Read 503969 times)

Offline riofl

  • Knight o' The Round Table
  • ***
  • Posts: 240
Re: SETI MB CUDA for Linux
« Reply #705 on: 17 Aug 2010, 09:00:15 am »
ok. i am going to have to get down to some very serious diagnostics here. the gtx 295 exhibits the same trouble as the 285 does on port1 !!!!!! yet a gts250 does not... or on that one boot it didnt. i have asked my friend to bring it back when he can stay a while and i can try multiple reboots. funny thing is when the 285 is in alone, it always picks port1.

its the oddest behavior i have ever seen. i enabled the 3 monitor setup which works, but its a lottery which screen will become 1600x1200 with each reboot now. mostly it sticks to device 0 dfp1 , 2nd port... but it has done this to  all 3 ports and screens!! i'm baffled.. i am sending msgs off to everyone involved in the entire video display chain to see if anyone has some ideas.. i am including gigabyte in this too although i dont know how the mobo can affect it.

i put the 295 as the first device to use and the 285 in the 2nd slot.. its weird but the 295 is actually slower in video response than the 285 is. however all 3 gpu chips crank cuda out very nicely. the 285 is a bit faster there too. i guess they had to go with slower, cooler gtx275 setups in order to be successful in dual gpu in a single box. temps are good in this pupppy though. so far they have not exceeded 65c under heavy load! of course i have the fan set to 100% :P i think it was asus or acer that made a limited edition dual 285 card.. expensive tho. also only made 1000 of them.

looks like i have a while of brain burning diagnostics ahead of me... hehe i think a phd in astrophysics would probably be easier. at least that course is ordered and logical :)

« Last Edit: 17 Aug 2010, 09:03:52 am by riofl »

Offline sunu

  • Alpha Tester
  • Knight who says 'Ni!'
  • ***
  • Posts: 771
Re: SETI MB CUDA for Linux
« Reply #706 on: 17 Aug 2010, 09:54:10 am »
The xorg.log would be quite useful to your troubleshooting.

The 2 gpus used in a gtx295 are weaker than the gpu in gtx285. They are not even equal to a gtx275. They are downclocked gtx275s.

Offline riofl

  • Knight o' The Round Table
  • ***
  • Posts: 240
Re: SETI MB CUDA for Linux
« Reply #707 on: 17 Aug 2010, 11:56:59 am »
maybe i will switch them then and put the 295 in the 2nd slot. the slower screen updates and pausing are getting to me. didnt do that with the 285 and tesla.

Offline Metod, S56RKO

  • Alpha Tester
  • Knight o' The Realm
  • ***
  • Posts: 51
Re: SETI MB CUDA for Linux
« Reply #708 on: 23 Aug 2010, 05:10:33 am »
I crunshed several units. One was already correctly validated. The others have error messages:
<core_client_version>6.4.5</core_client_version>
<![CDATA[
<stderr_txt>
 file './cudaAcc_gaussfit.cu' in line 506 : invalid configuration argument.
Cuda error 'GaussFit_kernel' in file './cudaAcc_gaussfit.cu' in line 506 : invalid configuration argument.
Cuda error 'GaussFit_kernel' in file './cudaAcc_gaussfit.cu' in line 506 : invalid configuration argument.
Cuda error 'GaussFit_kernel' in file './cudaAcc_gaussfit.cu' in line 506 : invalid configuration argument.
Cuda error 'GaussFit_kernel' in file './cudaAcc_gaussfit.cu' in line 506 : invalid configuration argument.
Cuda error 'GaussFit_kernel' in file './cudaAcc_gaussfit.cu' in line 506 : invalid configuration argument.
....

I have a GeForce 8800 GT. Whats wrong here ?  :(



For windows version I would propose to update drivers. No idea if it applicable to Linux...

I've been recently bitten by this very same problem. Im my case it seems that I've got too recent NVIDIA drivers (version 256.35, cudatoolkit 3.1).

I'm running 32-bit ubuntu 10.04 on Q6600, 8GB RAM and GeForce 8600 GT. I'm not running graphical user interface (X server might be running though), BOINC 6.10.17 says
Code: [Select]
NVIDIA GPU 0: GeForce 8600 GT (driver version unknown, CUDA version 3010, compute capability 1.1, 255MB, 99 GFLOPS peak)

Any suggestion? 64-bit linux on this machine is out of question due to unrelated reasons.

Offline sunu

  • Alpha Tester
  • Knight who says 'Ni!'
  • ***
  • Posts: 771
Re: SETI MB CUDA for Linux
« Reply #709 on: 23 Aug 2010, 05:58:34 am »
What seti app do you use?

Also a link to your machine would be nice.

With 8 GB RAM may I ask why 64 bit is out of the question?

Offline Metod, S56RKO

  • Alpha Tester
  • Knight o' The Realm
  • ***
  • Posts: 51
Re: SETI MB CUDA for Linux
« Reply #710 on: 23 Aug 2010, 06:30:36 am »
Here's link: http://setiathome.berkeley.edu/show_host_detail.php?hostid=129668.

I'm using Crunch3r's setiathome-CUDA-6.08.i686.

For living this machine does some GIS stuff with interface to MS SQL server. We've found out that Unix ODBC is broken on 64-bit while 32-bit one is OK. Hence 32-bit OS on this machine. My heart bleeds, but one can not have everything. :(

Hummm ... on the other hand, it seems to proceed somehow. state.sah has this line in it:

Code: [Select]
<prog>0.20329889</prog>

My concern (not primary though) is about why BOINC doesn't run GPU task in addition to 4 CPU tasks (quad core processor).
« Last Edit: 23 Aug 2010, 06:36:01 am by Metod, S56RKO »

Offline Metod, S56RKO

  • Alpha Tester
  • Knight o' The Realm
  • ***
  • Posts: 51
Re: SETI MB CUDA for Linux
« Reply #711 on: 23 Aug 2010, 09:13:03 am »
Seems that those lines are somewhat benign for the application itself. GPU finished one WU: http://setiathome.berkeley.edu/result.php?resultid=1690649422. Run time is nothing special (half of normal CPU time). What remains is to see result verified. If it verifies, then those lines in stderr make major aesthetic problem.

If the above is fine, then the issue of BOINC not running GPU task concurrent with 4 CPU tasks becomes primary concern.

Offline sunu

  • Alpha Tester
  • Knight who says 'Ni!'
  • ***
  • Posts: 771
Re: SETI MB CUDA for Linux
« Reply #712 on: 23 Aug 2010, 12:38:18 pm »
I guess you tried a 32bit chroot for your work and didn't work?

For the seti part:
-256.44 is the latest nvidia driver. Get it.
- If you don't run a x server make sure that you initialize your card properly. This is from the cuda release notes:

o In order to run CUDA applications, the CUDA module must be
  loaded and the entries in /dev created.  This may be achieved
  by initializing X Windows, or by creating a script to load the
  kernel module and create the entries.

  An example script (to be run at boot time):

  #!/bin/bash

  /sbin/modprobe nvidia

  if [ "$?" -eq 0 ]; then

  # Count the number of NVIDIA controllers found.
  N3D=`/sbin/lspci | grep -i NVIDIA | grep "3D controller" | wc -l`
  NVGA=`/sbin/lspci | grep -i NVIDIA | grep "VGA compatible controller" | wc -l`

  N=`expr $N3D + $NVGA - 1`
  for i in `seq 0 $N`; do
  mknod -m 666 /dev/nvidia$i c 195 $i;
  done

  mknod -m 666 /dev/nvidiactl c 195 255

  else
  exit 1
  fi

- Latest boinc is 6.10.56. Get it.


Let's try to reinstall the whole thing:
- Close boinc and delete everything in projects/setiathome.berkeley.edu directory. Delete client_state.xml and client_state_prev.xml in your boinc directory. Delete everything in the slots directory.
- Get http://calbe.dw70.de/AKV8/REL/V4/AK_V8_SSSE3_LX32.tar.bz2 and put the executable in projects/setiathome.berkeley.edu directory
- Get http://calbe.dw70.de/cuda/rel/setiathome-CUDA_2.2_6.08.x86_vlarkill.tar.bz2 and put the executable in projects/setiathome.berkeley.edu directory.
- Get http://lunatics.kwsn.net/index.php?module=Downloads;sa=dlview;id=258 and put the executable in projects/setiathome.berkeley.edu
- Get the 2.3 toolkit (delete everything from the 3.1 you have) and put

libcudart.so
libcudart.so.2
libcudart.so.2.3
libcufft.so
libcufft.so.2
libcufft.so.2.3

in the projects/setiathome.berkeley.edu directory.

- Edit accordingly your ld.so.conf or the corresponding ld-something file of your distro with the above location of the cuda libs. Do a ldconfig to update the cache.
- Do a ldd... to the cuda app to see if all dependencies are met.
-Make a file named app_info.xml in the projects/setiathome.berkeley.edu directory and put the following:

<app_info>
    <app>
        <name>astropulse_v505</name>
    </app>
    <file_info>
        <name>ap_5.06r411_sse3_x86_64-beta4</name>
        <executable/>
    </file_info>
    <app_version>
        <app_name>astropulse_v505</app_name>
        <version_num>506</version_num>
        <flops>41430637443.0638666</flops>
        <avg_ncpus>1.0000</avg_ncpus>
        <max_ncpus>1.0000</max_ncpus>
        <file_ref>
            <file_name>ap_5.06r411_sse3_x86_64-beta4</file_name>
            <main_program/>
        </file_ref>
    </app_version>
    <app>
        <name>setiathome_enhanced</name>
    </app>
    <file_info>
        <name>AK_V8_linux64_ssse3</name>
        <executable/>
    </file_info>
    <file_info>
        <name>setiathome-6.08.CUDA_2.2_x86_64-pc-linux-gnu</name>
        <executable/>
    </file_info>
    <file_info>
        <name>libcudart.so.2</name>                                                                                                                                                 
        <executable/>                                                                                                                                                               
    </file_info>                                                                                                                                                                   
    <file_info>                                                                                                                                                                     
        <name>libcufft.so.2</name>                                                                                                                                                 
        <executable/>                                                                                                                                                               
    </file_info>                                                                                                                                                                   
    <app_version>                                                                                                                                                                   
        <app_name>setiathome_enhanced</app_name>                                                                                                                                   
        <version_num>603</version_num>                                                                                                                                             
        <flops>24463044489.9277371</flops>                                                                                                                                         
        <avg_ncpus>1.0000</avg_ncpus>                                                                                                                                               
        <max_ncpus>1.0000</max_ncpus>                                                                                                                                               
    <file_ref>                                                                                                                                                                     
        <file_name>AK_V8_linux64_ssse3</file_name>                                                                                                                                 
        <main_program/>                                                                                                                                                             
    </file_ref>                                                                                                                                                                     
    </app_version>                                                                                                                                                                 
    <app_version>                                                                                                                                                                   
        <app_name>setiathome_enhanced</app_name>                                                                                                                                   
        <version_num>608</version_num>                                                                                                                                             
        <plan_class>cuda</plan_class>                                                                                                                                               
        <flops>244630444899.2773714</flops>                                                                                                                                         
        <avg_ncpus>0.1400</avg_ncpus>
        <max_ncpus>0.1400</max_ncpus>
    <coproc>
        <type>CUDA</type>
        <count>1</count>
    </coproc>
    <file_ref>
        <file_name>setiathome-6.08.CUDA_2.2_x86_64-pc-linux-gnu</file_name>
        <main_program/>
    </file_ref>
    <file_ref>
        <file_name>libcudart.so.2</file_name>
    </file_ref>
    <file_ref>
        <file_name>libcufft.so.2</file_name>
    </file_ref>
    </app_version>
</app_info>


-Change the yellow filenames with the ones you've got.
- Change the red number with something much lower, I don't know what a proper number for your card will be, maybe half of it.

- You have to make the cuda app run at 0 priority. I use the following script that runs in infinite loop:

#!/bin/sh
while true
do
for arg in `ps -Lo lwp --no-headers -p $(pgrep setiathome-6)`
do
renice 0 -p ${arg}
done
sleep 5
done

for that script to work you'll need the procps and bsdutils packages installed.

- Fire up boinc and see what we've done  :D
« Last Edit: 23 Aug 2010, 12:50:22 pm by sunu »

Offline Metod, S56RKO

  • Alpha Tester
  • Knight o' The Realm
  • ***
  • Posts: 51
Re: SETI MB CUDA for Linux
« Reply #713 on: 25 Aug 2010, 10:20:08 am »
- Change the red number with something much lower, I don't know what a proper number for your card will be, maybe half of it.

I assume that this flops stuff is more or less aesthetics.

Other than few differences in appinfo.xml (missing flops stanza, different values for avg_ncpus and max_ncpus - was 0.04) I've managed quite a different setup. I've left toolkit 3.1 while installing 2.0 libraries systemwide as well. ldd shows that correct (2.0) libraries are then loaded by seti/cuda app.

I've updated toolkit to 2.3 now and Crunch3r's app to the one you proposed. I'll check and see how it works out.

- You have to make the cuda app run at 0 priority. I use the following script that runs in infinite loop:

Could this be reason for quite long run times on my card? Meanwhile two results got verified and peers were running stock GPU app for Windows. One peer had GTX295, the other GTX285, both had from 5 to 8 times shorter run time. Or is such a speed difference expected between more modern and almost ancient GPUs?

Offline sunu

  • Alpha Tester
  • Knight who says 'Ni!'
  • ***
  • Posts: 771
Re: SETI MB CUDA for Linux
« Reply #714 on: 25 Aug 2010, 08:34:29 pm »
I assume that this flops stuff is more or less aesthetics.
Absolutely NOT! Check http://lunatics.kwsn.net/3-linux/seti-mb-cuda-for-linux.msg19829.html#msg19829 and http://lunatics.kwsn.net/5-windows/appinfo-flops-question.0.html


I've left toolkit 3.1 while installing 2.0 libraries systemwide as well. ldd shows that correct (2.0) libraries are then loaded by seti/cuda app.

I've updated toolkit to 2.3 now and Crunch3r's app to the one you proposed. I'll check and see how it works out.
If you don't need them for other things DELETE the 3.1 and 2.0 and KEEP the 2.3.

- You have to make the cuda app run at 0 priority. I use the following script that runs in infinite loop:

Could this be reason for quite long run times on my card?

Yes, you ABSOLUTELY have to run the cuda app at priority 0.


Meanwhile two results got verified and peers were running stock GPU app for Windows. One peer had GTX295, the other GTX285, both had from 5 to 8 times shorter run time. Or is such a speed difference expected between more modern and almost ancient GPUs?
Of course a gtx295 or gtx 285 will be multiple times faster than a 8600gt. If they weren't, why people then would fork out multiple hundred euros/dollars to buy them?
Also note that the linux cuda app is slower in general than the windows one.

Offline Metod, S56RKO

  • Alpha Tester
  • Knight o' The Realm
  • ***
  • Posts: 51
Re: SETI MB CUDA for Linux
« Reply #715 on: 26 Aug 2010, 02:54:18 am »
I assume that this flops stuff is more or less aesthetics.
Absolutely NOT! Check http://lunatics.kwsn.net/3-linux/seti-mb-cuda-for-linux.msg19829.html#msg19829 and http://lunatics.kwsn.net/5-windows/appinfo-flops-question.0.html

Well, for me that's more aesthetic than functionality. I'm running SAH along other projects with not-so-high resource share, hence effect of wrong DCF is not so crucial.

- You have to make the cuda app run at 0 priority. I use the following script that runs in infinite loop:

Could this be reason for quite long run times on my card?

Yes, you ABSOLUTELY have to run the cuda app at priority 0.

Given that CUDA/Win app is generally faster than CUDA/Linux and that peers' GPUs are faster HW-wise than mine, then 5-8 times longer runtimes on my ancient GPU seems reasonable. Even without running CUDA app at priority 0.

Or is it that normal (not niced that is) priority for CUDA app is necessary for flawless operation?

[edit]
Ah, I've read a couple more articles about it and now it became more clear that this helps a lot. At the same time I'll try to lower avg_ncpus and max_ncpus to 0.00. My reasoning: if the task runs with mich higher priority than the rest of CPU-hungry apps, then it'll just steal CPU time from them. Hence no need for BOINC to allocate CPU cycles for it - decent OS tends to do it much better and with much finer granularity.

This might cure my concerns below.

I'll report back with results.
[/edit]

Back to my concerns: what should I put in app_info.xml to make BOINC run 4 CPU tasks and 1 GPU task in parallel? I've got a quad CPU and single GPU so this system should be able to run 5 tasks in parallel. Currently it suspends one CPU task when running the GPU task.
« Last Edit: 26 Aug 2010, 03:47:16 am by Metod, S56RKO »

Offline sunu

  • Alpha Tester
  • Knight who says 'Ni!'
  • ***
  • Posts: 771

Offline riofl

  • Knight o' The Round Table
  • ***
  • Posts: 240
Re: SETI MB CUDA for Linux
« Reply #717 on: 28 Aug 2010, 08:03:32 pm »
sheesh. ever since they started this 3days off thing my pending has skyrocketed. is that true for everyone? for the past few weeks i have been hovering between 310,000 and 370,000 !

Offline sunu

  • Alpha Tester
  • Knight who says 'Ni!'
  • ***
  • Posts: 771
Re: SETI MB CUDA for Linux
« Reply #718 on: 29 Aug 2010, 01:01:49 am »
Yes, I'm seeing this also. Before that, I had about 200,000-250,000 pending, now it's about 600,000 pending.

Offline riofl

  • Knight o' The Round Table
  • ***
  • Posts: 240
Re: SETI MB CUDA for Linux
« Reply #719 on: 31 Aug 2010, 05:09:44 am »
i'm currently using nvidia-drivers-195.36.31.  noticed an upgrade available to nvidia-drivers-256.52.
i'm always a bit suspicious of large jumps in upgrade versions. worth it? avoid it?

 

Welcome, Guest.
Please login or register.
 
 
 
Forgot your password?
Members
Total Members: 97
Latest: ToeBee
New This Month: 0
New This Week: 0
New Today: 0
Stats
Total Posts: 59559
Total Topics: 1672
Most Online Today: 18
Most Online Ever: 983
(20 Jan 2020, 03:17:55 pm)
Users Online
Members: 0
Guests: 12
Total: 12
Powered by EzPortal