When corrupted results get validated...

[sunu]

... and valid results get thrown out of the window...

See this workunit http://setiathome.berkeley.edu/workunit.php?wuid=609263674

My result was the only valid result from all that garbage. And was marked as invalid 

[Raistmer]

Hm.... looks like our hope that incorrect overflow gives quite random pulses not fulfilled
So such disrupted GPU state even more dangerous for project than was thought before!

[perryjay]

Wish I hadn't seen that. I've got a couple of pendings waiting for a match that look a lot like that.

[Claggy]

Here's one on my host: workunit.php?wuid=618018953 

Claggy

[perryjay]

Just found this one today  http://setiathome.berkeley.edu/workunit.php?wuid=618796380

[sunu]

Unfortunately they are many. Yesterday I had another one but today it must have been deleted from the database and I can't post the link.

[perryjay]

I think they are trying to keep it a secret!! Looks like as soon as we post a link to one of them they erase it from the database. Must be a conspiracy.

[Richard Haselgrove]

Could people spotting / reporting this problem please check and report the hardware involved?

I have a horrible feeling that people who just throw a Fermi card into a host and attach, are being issued with the stock Cuda23 application and immediately start trashing WUs.

But we need robust reports from reliable witnesses....

[Jason G]

...I have a horrible feeling that people who just throw a Fermi card into a host and attach, are being issued with the stock Cuda23 application and immediately start trashing WUs....

  From the few I saw, that was the case (470's & 480'wingmen trashing & validating against one another).  I do suspect that was the 'errors with 2.3' situation that Eric alluded to a while back (around fermi release IIRC), which might suggest some sortof flag raised somewhere that let him know, like the noisy wus figure that used to show somewhere (?) .  Further conjecturing (& hoping), some double -9 intercept may be in place, explaining the rapid result removal sooner than the normal 24 hour assimilation/deletion period.  If that's the case, I hope they put those through again for reprocessing.

[Richard Haselgrove]

Eric's comment (he actually said cuda24, which we're taking to be a typing error) was made on 20 May - well, actually late afternoon 19 May in his time zone - in the course of a conversation with David and me about Fermi issues at Beta. It came just after the corrected Fermi 6.10 application version was loaded for stock download at Beta.

The 'noisy WUs' figure is still showing on the Science status page. Since it's "science", I assume it's driven off the validated results transferred from the BOINC to the science database. Historically, it's been 'about 5%'. Last time I looked, it was down to 1.2%, which I took as a compliment to the Radar removal team. Now, it's showing as 4.8%, which probably reflects the scale of the "pseudo -9" problem.

I'm coming to the conclusion that nobody saw this one coming. I certainly hadn't thought about it until this afternoon, and yet I've been working closely with David / Eric / Jason on BOINC+Fermi issues. Even when I told David (much to his surprise) that the Fermi card wouldn't run the cuda23 app at Beta (during the quota overflow discussion), the penny didn't drop that the situation was already building up at Main.

I have now suggested - on boinc_dev, which is the wrong mailing list, but the only one we've got in the absence of an official seti_technical channel - that 6.10_fermi should be installed as a stock application at Main. I think that's the only sensible way to rescue the situation.

Let's hope that no eager young project puppy runs into the lab this afternoon and loads a pristine box of tapes.....

[sunu]

Could people spotting / reporting this problem please check and report the hardware involved?

I have a horrible feeling that people who just throw a Fermi card into a host and attach, are being issued with the stock Cuda23 application and immediately start trashing WUs.

But we need robust reports from reliable witnesses....

Well, some of the reported workunits have a fermi card involved while others don't. The workunit from the first post here had 3-4 corrupted results, only one, if I remember correctly, was from a fermi with a cuda23 app. The other workunit I mentioned above had two 2xx cards involved with massive amounts of corrupted -9 results.

[Josef W. Segur]

Suggestion: make copies of the WU and Task detail pages before BOINC purges them. Even better would be to find examples at SETI Beta where purging is disabled, and probably file deletion too. I started looking there, but no luck in finding any. Besides I got distracted by some of the nonsensical credit granting, one of Tetsuji's hosts recently did a set of reissued 0.448 tasks on 6.09 CUDA 23 with claims of 94.12 and grants ranging from 5.87 to 52.01 
                                                                               Joe

[perryjay]

Sorry, I didn't notice what my wingmen were running but I'm pretty sure the WUs were showing as 6.09s. I'm running a GT9500  on my vista X86 machine.


edit: Forgot to add I'm running the renamed cudart32_30_14 and cufft32_30_14 DLLs with the 197.45 driver.

[Richard Haselgrove]

Suggestion: make copies of the WU and Task detail pages before BOINC purges them. Even better would be to find examples at SETI Beta where purging is disabled, and probably file deletion too. I started looking there, but no luck in finding any. Besides I got distracted by some of the nonsensical credit granting, one of Tetsuji's hosts recently did a set of reissued 0.448 tasks on 6.09 CUDA 23 with claims of 94.12 and grants ranging from 5.87 to 52.01 
                                                                               Joe

Unfortunately, searching at Beta probably won't turn up many errors, because I've been leaning on David to get them fixed, and this particular problem (issuing work associated with a non-Fermi app, to a Fermi-equipped host) should no longer happen at Beta. There are just a few still visible on 12316.

All that's left is - why am I still trapped by yesterday's quota?

Code: [Select]

Max tasks per day 153
Number of tasks today 273
- and why is credit so erratic?

[Raistmer]

Suggestion: make copies of the WU and Task detail pages before BOINC purges them. Even better would be to find examples at SETI Beta where purging is disabled, and probably file deletion too. I started looking there, but no luck in finding any. Besides I got distracted by some of the nonsensical credit granting, one of Tetsuji's hosts recently did a set of reissued 0.448 tasks on 6.09 CUDA 23 with claims of 94.12 and grants ranging from 5.87 to 52.01 
                                                                               Joe

[offtopic]
Credit granting on beta absolutely screwed. If you look on granting for AP tasks it will be even more obviously
[/offtopic]

And ontopic: cause first listed WU in this thread had no 2 Fermi GPUs, this problem not only Fermi-related (unfortunately). Looks like _any_ invalid overflow has some probability to be validated