SETI MB CUDA for Linux

[riofl]

i think i will keep my 2.6.29 kernel on my desktop then... unfortunately i have to update several of my servers to 2.6.31 due to application requirements, so those machines, unfortunately, will probably suffer. i have the script on them for cpu setting them to 10 to see how things go and how server performance is affected if at all. the rest of the servers i will probably leave with the 2.6.26 kernels they now have (if it ain't broke dont fix it) .

[riofl]

over the short term, i am seeing an average of 2 to 3 minutes lower processing times per workunit on the gpus at nice 0. nice boost.

[pp]

can you run gpus at higher levels sucn as -1 or so or is it the nature of the gpu system to not go below 0? just wondering if there is only marginal benefit at running them at -1 or maybe even -5.

You can run them at -5 if you like but I didn't notice any difference in speed running them at -5 compared to 0.

[sunu]

can you run gpus at higher levels sucn as -1 or so or is it the nature of the gpu system to not go below 0? just wondering if there is only marginal benefit at running them at -1 or maybe even -5.

Some time ago, I've tried with negative priorities in my 2.6.27-14 ubuntu kernel and they had... negative effect, the tasks ran slower, but I don't remember if the slowdown was for the GPU tasks or the CPU ones or for both.

I'm trying right now the same in sidux with 2.6.31-5.slh.1-sidux-amd64 (-5 nice for cuda tasks, 19 for cpu) and it doesn't seem to make any difference over 0 for GPU and 19 for CPU ( http://setiathome.berkeley.edu/results.php?hostid=3690316 ). All results dated 2 Nov are with -5 for GPU and 19 for CPU, the rest are with 0 for GPU and 19 for CPU. They seem about the same to me. In this machine using the "default" (10 for GPU and 19 for CPU) results in doubling up computation times for CUDA tasks.

[riofl]

i dont get that radical a change in processing times.. seems i never do compared to you. what mobo are you using? i believe you said you had a q6600 processor which is the same one i have. i did notice an apparent slow down using -5 so i went back to 0 but again that was short term. for some reason i don't get nearly the response you do so i am beginning to suspect it may be my gigabyte ga-p35-ds4 mobo. i may try upgrading it to an asus rampage formula or something similar. this machine does not need more than 2 pci-e slots although i suspect if there were more than 2 i could spread the 2 gpu devices out farther away from each other to help minimize heat buildup. then i might not need that little 1 in fan moving air between them.

[sunu]

Yes I still have a Q6600, it's my other "big" machine http://setiathome.berkeley.edu/show_host_detail.php?hostid=3281360 . This has the old ubuntu kernel that saw the slowdown with negative nice levels. This has an ASUS P5K vanilla motherboard. With this machine I didn't see much difference going from 10-19 GPU-CPU to 0-19 GPU-CPU.

i did notice an apparent slow down using -5 so i went back to 0 but again that was short term.

So we see the same result, at least with my ubuntu machine using the old, 2.6.27-14 kernel.

The 3690316 is a new lightweight experimental/testing build I've made to use it for... well experimental/testing things. Right now it runs sidux and boinc with all cuda and stuff over livecd.

[riofl]

ahh ok so the difference was not that great with the 'big' machine compared to the others... now i dont feel so bad

my workstation is using 2.6.29 kernel and i doubt i will update it knowing what i do now.. i have the script running to set cpu nice to 10 to see if it helps on those servers that must run 2.6.31. i did notice rac dropped on those machines so i hope this helps.  the rest are 2.6.26 and will be left alone unless my test on one of those proves nice 10 is considerably faster with little bother to the server.

[lordvader]

Well I had the units run overnight with the renice script running in the background, and it really made no difference.

CUDA workunits under the 2.6.31 kernel take between 2-4 times longer than they should. I may add, that while these units are running, compositing effects (such as wobbly windows and other eye candy) are extremely stuttery. This wasn't the case with the 2.6.27 kernel.

I'm running a phenom 955 CPU, GTX 275 GPU.

[sunu]

Now your machine says 2.6.31-14-generic kernel. Couple of days ago I think I saw it showing 2.6.31-14-custom or something like that.

Looking at your tasks I see most of them with good times and some others with abnormal times all seemingly mixed up.

We are now nearing midnight UTC. Please stick with one kernel, run the script and let it crunch so all workunits reported 3 Nov are with the new 2.6.31 kernel with script running so we can see what's going on.

[riofl]

after running gpu at 0 all day with the 2.6.29 kernel it appears my average is down by a few hundred points, so i am cancelling the script for the next 24 hrs to see if it goes back up

[sunu]

Riofl, check your pending credits. Mine have gone up almost 8000 the last day or so and my RAC has taken a dive.

[riofl]

ahh forgot about them. ok im writing current down and will put the script back into operation till sometime tomorrow

[pp]

Well I had the units run overnight with the renice script running in the background, and it really made no difference.

CUDA workunits under the 2.6.31 kernel take between 2-4 times longer than they should. I may add, that while these units are running, compositing effects (such as wobbly windows and other eye candy) are extremely stuttery. This wasn't the case with the 2.6.27 kernel.

I'm running a phenom 955 CPU, GTX 275 GPU.

I'm beginning to suspect that someone left some debug options enabled in your kernel. What's the output of :

Code: [Select]

zgrep DEBUG /proc/config.gz

[lordvader]

Interesting ...
Here's the output.

Code: [Select]

CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y                                       
CONFIG_SLUB_DEBUG=y                                                         
CONFIG_HAVE_DMA_API_DEBUG=y                                                 
CONFIG_X86_DEBUGCTLMSR=y                                                     
CONFIG_X86_CPU_DEBUG=m                                                       
CONFIG_PM_DEBUG=y
CONFIG_IRDA_DEBUG=y
CONFIG_CFG80211_REG_DEBUG=y
CONFIG_CFG80211_DEBUGFS=y
CONFIG_MAC80211_DEBUGFS=y
CONFIG_WIMAX_DEBUG_LEVEL=8
CONFIG_PNP_DEBUG_MESSAGES=y
CONFIG_CB710_DEBUG_ASSUMPTIONS=y
CONFIG_AIC7XXX_DEBUG_ENABLE=y
CONFIG_AIC7XXX_DEBUG_MASK=0
CONFIG_AIC79XX_DEBUG_ENABLE=y
CONFIG_AIC79XX_DEBUG_MASK=0
CONFIG_SCSI_MVSAS_DEBUG=y
CONFIG_SCSI_LPFC_DEBUG_FS=y
CONFIG_SCSI_DEBUG=m
CONFIG_FIREWIRE_OHCI_DEBUG=y
CONFIG_MLX4_DEBUG=y
CONFIG_ATH9K_DEBUG=y
CONFIG_LIBIPW_DEBUG=y
CONFIG_B43LEGACY_DEBUG=y
CONFIG_WIMAX_I2400M_DEBUG_LEVEL=8
CONFIG_ATM_FORE200E_DEBUG=0
CONFIG_USB_SERIAL_DEBUG=m
CONFIG_INFINIBAND_MTHCA_DEBUG=y
CONFIG_INFINIBAND_AMSO1100_DEBUG=y
CONFIG_INFINIBAND_IPOIB_DEBUG=y
CONFIG_THINKPAD_ACPI_DEBUGFACILITIES=y
CONFIG_OCFS2_DEBUG_MASKLOG=y
CONFIG_JFFS2_FS_DEBUG=0
CONFIG_DEBUG_FS=y
CONFIG_DEBUG_KERNEL=y
CONFIG_SCHED_DEBUG=y
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_DEBUG_RODATA=y


[riofl]

interesting... doesnt look like i will have the large increase you did but my rac went down by about 100 points but my pending went up by almost 300 points.. wonder why pending increases when running more aggressively?