[Split] PowerSpectrum Unit Test

[Miep]

Thanks for the tolerances.  Being largely memory bound, the FLops tolerances are more than enough, and indicate +/- 10% variation of worst case on that.  I presume that's driving a display, so that's reasonable.

You're welcome - now what exactly makes you think the mobile GPU of a laptop might be driving a display?
No bluescreens with the lastest driver yet - touch wood...

I'll do statistics on all the numbers next time round then.

[PatrickV2]

OK, I ran version 6 of the tool on my system (Q6600/8GB/8800GTX) under both WinXP32 as well as Win7-64. If you want me to (re-)run other versions of the tool, let me know.

Both loggings below each-other, first the oldest, WinXP32:

Device: GeForce 8800 GTX, 1350 MHz clock, 768 MB memory.
Compute capability 1.0
Compiled with CUDA 3020.
      PowerSpectrum+summax Unit test #6 (pinned mem)
Stock:
 PwrSpec<    64>   18.3 GFlops   73.1 GB/s 1183.3ulps

 SumMax (    64)    1.3 GFlops    5.5 GB/s
Every ifft average & peak OK

 PS+SuMx(    64)    4.3 GFlops   17.6 GB/s


GetPowerSpectrum() choice for Opt1: 64 thrds/block
     64 threads:       18.3 GFlops   73.1 GB/s 121.7ulps


Opt1 (PSmod3+SM): 64 thrds/block
   64 threads, fftlen 64: (worst case: full summax copy)
         6.4 GFlops   26.1 GB/s 121.7ulps
Every ifft average & peak OK
   64 threads, fftlen 64: (best case, nothing to update)
         8.1 GFlops   32.7 GB/s 121.7ulps



Then Win7-64:

Device: GeForce 8800 GTX, 1350 MHz clock, 731 MB memory.
Compute capability 1.0
Compiled with CUDA 3020.
      PowerSpectrum+summax Unit test #6 (pinned mem)
Stock:
 PwrSpec<    64>   18.1 GFlops   72.5 GB/s 1183.3ulps

 SumMax (    64)    1.1 GFlops    4.8 GB/s
Every ifft average & peak OK

 PS+SuMx(    64)    3.8 GFlops   15.4 GB/s


GetPowerSpectrum() choice for Opt1: 64 thrds/block
     64 threads:       18.1 GFlops   72.6 GB/s 121.7ulps


Opt1 (PSmod3+SM): 64 thrds/block
   64 threads, fftlen 64: (worst case: full summax copy)
         5.4 GFlops   21.9 GB/s 121.7ulps
Every ifft average & peak OK
   64 threads, fftlen 64: (best case, nothing to update)
         6.6 GFlops   26.8 GB/s 121.7ulps


Regards, Patrick.

[Jason G]

Ahhh, hi Patrick.  Looks like your card  should still be able to use pinned host memory, but isn't  .  It indeed doesn't support mapped memory (a different kind), but didn't engage the pinned memory improvement because I need to change how I detect that feature.  I'm checking the wrong feature flags it looks like.... ooops 

Will make a #7 end of week, and pay special attention to making sure that engages properly on compute capability 1.0 cards (that don't support mapped memory).

Cheers for finding the problem 

[PatrickV2]

Ahhh, hi Patrick.  Looks like your card  should still be able to use pinned host memory, but isn't  .  It indeed doesn't support mapped memory (a different kind), but didn't engage the pinned memory improvement because I need to change how I detect that feature.  I'm checking the wrong feature flags it looks like.... ooops 

Will make a #7 end of week, and pay special attention to making sure that engages properly on compute capability 1.0 cards (that don't support mapped memory).

Cheers for finding the problem 

I have no idea what I did, but you're quite welcome.

Regards, Patrick.

[Jason G]

Thanks,

    It's what you (the test #6 anyway) didn't do 

This line's missing:

Opt1 (PSmod3+SM): 64 thrds/block
PowerSpectrumSumMax array pinned in host memory.
   64 threads, fftlen 64: (worst case: full summax copy)
         1.5 GFlops    5.9 GB/s 121.7ulps
Every ifft average & peak OK
   64 threads, fftlen 64: (best case, nothing to update)
         1.6 GFlops    6.7 GB/s 121.7ulps

When operational, that feature seems to add a touch of throughput to both XP & Vista/Win7, and seems to close the performance difference. (we've been so worried about).  You should get a boost when I fix that.

Jason

[PatrickV2]

Thanks,

    It's what you (the test #6 anyway) didn't do 

This line's missing:

Opt1 (PSmod3+SM): 64 thrds/block
PowerSpectrumSumMax array pinned in host memory.
   64 threads, fftlen 64: (worst case: full summax copy)
         1.5 GFlops    5.9 GB/s 121.7ulps
Every ifft average & peak OK
   64 threads, fftlen 64: (best case, nothing to update)
         1.6 GFlops    6.7 GB/s 121.7ulps

When operational, that feature seems to add a touch of throughput to both XP & Vista/Win7, and seems to close the performance difference. (we've been so worried about).  You should get a boost when I fix that.

Jason

Ah, ok, thanks for the elaboration. Looking forward to test #7 then!

Regards, Patrick.

[Frizz]

Windows XP32. GTX 570. Nvidia Driver 263.09.


Device: GeForce GTX 570, 1464 MHz clock, 1280 MB memory.
Compute capability 2.0
Compiled with CUDA 3020.
      PowerSpectrum+summax Unit test #6 (pinned mem)
Stock:
 PwrSpec<    64>   25.6 GFlops  102.5 GB/s   0.0ulps

 SumMax (    64)    1.9 GFlops    7.9 GB/s
Every ifft average & peak OK

 PS+SuMx(    64)    6.2 GFlops   25.1 GB/s


GetPowerSpectrum() choice for Opt1: 256 thrds/block
    256 threads:       33.3 GFlops  133.3 GB/s 121.7ulps


Opt1 (PSmod3+SM): 256 thrds/block
PowerSpectrumSumMax array pinned in host memory.
  256 threads, fftlen 64: (worst case: full summax copy)
        10.9 GFlops   44.0 GB/s 121.7ulps
Every ifft average & peak OK
  256 threads, fftlen 64: (best case, nothing to update)
        13.5 GFlops   54.7 GB/s 121.7ulps

[Jason G]

570 wooot! 

[Frizz]

570 wooot! 

Borrowed it from a friend. It's hot, almost non-overclockable, and slightly slower than 480.

I am really looking forward to AMD HD6950/6970 !

[EDIT] Seems I got a bad sample. I've seen reports where the 570 has been overclocked to 840@4250 (stock: 732@3800) with air cooling.

[Jason G]

It's hot, almost non-overclockable

Why bother ?  harvesting faulty parts you think ?

[Edit:] that worst case is slightly better than my 480 worst case, but the best cases are inferioir.   From the powerspectrum I see the constraint is memory ( again  ) ... So indeed these may not be be a good choice for seti in the short term ... probably do Batman really well though 

[Frizz]

probably do Batman really well though 

LOL

Yeah ... memory. They chopped the memory interface. I guess they did this so it's not getting to close to the 580 - and not to far ahead of the 480.

GTX570: 320bit
GTX480 & GTX580: 384bit

[Jason G]

@All:  In the meantime, having identified the major issues at pllay with these code areas, along with appropriate techniques to use,  I have come up with some ideas for a major redesign of the FFT->Powerspectrum->Summax(reduction)->FindSpikes pipeline, which currently accounts for around ~40%-60% of processing. 

I'll change the format of the next test quite a bit, and spend time tomorrow to get things underway toward #7.

Jason

[Raistmer]

I would suggest to  test these samples of code at different GPU freq to mem freq ratios.
SubSpace's experiment with beta OpenCL apps showed that it's very informative approach.
(He established that HD5 wins over usual OpenCL MB if GPU engine is relative fast and memory relative slow, while if GPU clocks lowed usual app wins).
I think it's quite explains why other testers see bigger execution time on VLAR for HD5 than for usual app - their GPUs not so fast relative their memory.
Memory influence can be quite highlighted this way.

[Jason G]

Yes, in fact that's exactly what happened to confirm memory bound nature of what's going on, ( from yet another angle ).

Steve's 480 core is clocked considerably higher than mine, yet he was initially achieving lower throughput than my card.  He tweaked his memory throughput for some improvement. 

After that, a discrepancy between throughput on XP Vs Win7 was then noted, somewhere around the familiar 10% difference.  I added use of pinned memory for the transfers, to try hide them.  With Ghost's help, In the heavy transfer case ( worst case full summax array copy, as with stock code) the XP-Win7 difference was narrowed to ~4% or less, while the WDDM performance proved more efficient with the raw processing in best case (No transfers needed)

Now Steve's 480 achieves some 27% more throughput, in the worst case,  than mine does.  I take this as an indication that the transfer hiding is shifting the bottleneck around as intended, and that it's time to move on to more sophisticated code portions with the acquired tools & techniques.

Still learning stuff every day with these things.

Jason

[SciManStev]

I have kicked my 480 memory speed up to 1975 MHz, with plenty of room to go. The 480 cores are clocked at 860 MHz. I tried to increase my CPU memory, but 1774 MHz is as fast as I can get it. I was able to increase my CPU speed to 4.26 GHz with hypethreading enabled, while maintaining about 57°C to 60°C core temps.

Steve