Large number of Errors when processing.

[_Geordie_]

I recently updated my drivers and noticed that I'm getting a large number of Cuda errors (Compute error) with the latest V12 cuda app (driver stopped).

I've taken the driver back to 190.38 but am still getting a large number of compute errors and using up the daily quota rapidly.

I'm not sure whether its hardware or software - I think my hardware is good - no overheating etc etc.

Is there any way I can diagnose?

[Raistmer]

first of all you could post link to host under question.

[_Geordie_]

This is the host:

http://setiathome.berkeley.edu/results.php?hostid=4672231 - I've just changed this machine back to the SETI cuda client to see if that makes a difference - I'll know in the morning. (10:30pm here)

I also have another couple of machines that I don't have immediate terminal access to that are also posting errors according to SETI.

http://setiathome.berkeley.edu/results.php?hostid=4612287

http://setiathome.berkeley.edu/results.php?hostid=4093238

Edit: Just checked and the second 2 hosts are all VLAR kills as far as I can tell - (I took a sample of around 10 units from each host from the entire list of WU's errored for each host)

[Pepi]

You are using VLAR kill app    VLAR WU (AR: 0.059975 )detected... autokill initialized, so there is no error It suppose to work in that way

[Raistmer]

no, his first host has true errors:

Work Unit Info:
...............
WU true angle range is :  2.722896
Optimal function choices:
-----------------------------------------------------
name               
-----------------------------------------------------
              v_BaseLineSmooth (no other)
            v_GetPowerSpectrum 0.00023 0.00000
                   v_ChirpData 0.01420 0.00000
                  v_Transpose4 0.00362 0.00000
               FPU opt folding 0.00234 0.00000
CUFFT error in file 'c:/sw/gpgpu/seti/seti_boinc/client/cuda/cudaAcc_fft.cu' in line 62.

</stderr_txt>

But it's stock app, not opt one.
[
most likely some problem with CUFFT libraries. Maybe incompatible driver version...
]

[_Geordie_]

Pepi - read above please.....

Raistmer - Yep that's because I've reverted back to the stock app in the past day or so to see if my hardware is a problem - I'm getting some app errors but no more than I've seen on other machines - its not screaming through the entire days quota in a couple of minutes now and I've not had one driver crash.

http://setiathome.berkeley.edu/result.php?resultid=1540425786

http://setiathome.berkeley.edu/result.php?resultid=1540423298

http://setiathome.berkeley.edu/result.php?resultid=1540423294

http://setiathome.berkeley.edu/result.php?resultid=1540423290

http://setiathome.berkeley.edu/result.php?resultid=1540423281

That's just a few of the many.

[Raistmer]

Reverting to stock not helped much it seems.
Opt app has better means for issues debugging, not only bettter speed.
For example:
After app init: total GPU memory 536870912    free GPU memory 128245760
That is, your GPU has ~512 onboard memory, but only ~128MB is available at moment app started to allocate memory.
This amount on border line, sometimes it's not enough, that's why that result (firts of listed by you) errored out with memory allocation error.
Maybe errors in stock app induced by same shortage of GPU memory, it's the primary cause you need to fight with.
Do you running some graphic-intencive app (like 3D ganes) while crunching with CUDA MB? Do you use some 3D screen-savers? Any another GPU memory consumers maybe?
Try to free more GPU memory, 512MB is more than enough for CUDA MB, but 128MB sometimes not enough...

[_Geordie_]

mmmm not sure what could be causing that - definately no game playing.  The CUDA app only runs when the machine is idle also - there's nothing running on the box other than 8 optimized clients when the CUDA app starts AFAIK.

Just looked even further back and its the same problem with every one of the sample (10 or so) I looked at.....and yes it looks as if the error with the stock app even though less frequent and not crashing the driver is a consistent error.

Looking at GPU-Z each GPU is apparently only using 21Mb of memory right now. (providing it interogates the GPU correctly?)

As for driver version I'm running 190.38 at the moment - going to try 191.07.

Are there any quirks with the 9800GX2 that could be causing this?

[Raistmer]

Are there any quirks with the 9800GX2 that could be causing this?

Sorry, don't know. Try to ask on SETI main forums. There are some owners of 9800 cards, maybe they could give some insights...

[_Geordie_]

Will do thanks.

Fairly new to CUDA/GPU crunching so learned a few things here anyhow! Appart from obviously not playing games when the app starts is there any way to check and ensure that nothing is running on the GPU?

Just running a memtest on each GPU RAM. Seems ok.

I've changed to 191.07 and reverted back to the optimised app to see if that helps.

I'll see if I can find any 9800GX2 card specific issues.

If I resolve it I'll post back here.

[Twidget]

Receiving large number of "errors while computing" using optimized apps. Installed Lunatics_Win64v0.2 on March 8th, that is when they started. Vast majority have same information in "stderr out" field:

<core_client_version>6.10.18</core_client_version>
<![CDATA[
<message>
 - exit code -6 (0xfffffffa)
</message>
<stderr_txt>

VLAR WU (AR: 0.008783 )detected... autokill initialised
SETI@home error -6 Bad workunit header

File: ..\worker.cpp
Line: 144


</stderr_txt>
]]>

http://setiathome.berkeley.edu/results.php?hostid=5210381

Any help will be greatly appreciated. Thanks

[Ghost0210]

Hi Twidget,
I take it from your post your running the vLar kill version of the opt app which will stop processing any vLar wus on your GPU with this error and send them back to seti to be sent out to another client.
There is another version of the opt app that you can download that doesn't kill these workunits, but vLars are extremely slow on GPUs.
If you want to download the Reschedule tool from here this will move the vLars from your GPU and onto your CPU. Saving you from having all these tasks erroring out on you

[Twidget]

Hi Ghost0210,
Thanks for the quick response. I'm a newbie to this site. Poked around a bit and can't find the Reschedule app. Would you please point me in the right direction.

Thanks

[Raistmer]

http://lunatics.kwsn.net/12-gpu-crunching/cpu-gpu-rebranding-perl-script.0.html

[Twidget]

Thanks Raistmer! Will give it a try. Be back in touch if I have questions.