Seti@Home optimized science apps and information

Optimized Seti@Home apps => Discussion Forum => Topic started by: Leaps-from-Shadows on 21 Nov 2008, 04:39:30 pm

Title: Report on new optimized Astropulse apps for Windows
Post by: Leaps-from-Shadows on 21 Nov 2008, 04:39:30 pm
On Cruiser-

v4.37 SSE3 EPF:  Slower than v4.35 SSE3 app, with a projected completion time of 31 hours or so (21.1% complete).  Slowest time for v4.35 SSE3 app was around 29 hours.

v5.00r69 SSE3:  Faster than all previous apps, with a projected completion time of 26 hours or so (63.2% complete).

I'll post more updates once the units complete.  The unit crunching with v4.37 is the last one for that app - all others have been branded for the v5.00 app.  I've got 10 of those including the one crunching (six for Main, four for Beta).  I'm currently using the v5.00r69 SSE3 app for both Main and Beta.
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Richard Haselgrove on 21 Nov 2008, 05:42:32 pm
My testing note on that build, running on a Q6600, reads "Did shave an hour+ off the runtime, though - 18:32:47 down to 17:16:42.": which is why I chose it for the release (final build in the ongoing quest for v4 optimisation, before programming effort switched to v5.00 compatibility).

Since v4 is now effectively dead, I think the only sensible thing to do is to soldier on until the last one is crunched, accepting the 2 hr penalty with as much grace as you can muster (through gritted ;D, no doubt).

But it's still a useful reminder of the unpredicable variability of the optimisations on AMD versus Intel, and and indication of the sort of tests we need to run (project timing permitting) in the run up to future releases.
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Raistmer on 21 Nov 2008, 05:46:12 pm
@Leaps-from-Shadows 
could you post link on your host ?
I need timings from stderr to see what part of program become slower...
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Richard Haselgrove on 21 Nov 2008, 06:04:18 pm
I think it must be RID 1065144181 (http://setiathome.berkeley.edu/result.php?resultid=1065144181) on host 4521634 (http://setiathome.berkeley.edu/show_host_detail.php?hostid=4521634) (Phenom) - as yet unfinished.

Cosmic_Archer has a similar report for RID 1060056077 (http://setiathome.berkeley.edu/result.php?resultid=1060056077) on host 2889590 (http://setiathome.berkeley.edu/show_host_detail.php?hostid=2889590) (Opteron) - also as yet unfinished.
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Raistmer on 21 Nov 2008, 06:05:57 pm
Thanks Richard.
Unfinished tasks have no timings, so will await when they become finished :)
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Richard Haselgrove on 21 Nov 2008, 06:08:39 pm
Thanks Richard.
Unfinished tasks have no timings, so will await when they become finished :)


Yeah, just did clickable links so you can check often - these tasks are on Main, so subject to 24-hour purge (assuming validation ;D).
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Leaps-from-Shadows on 21 Nov 2008, 06:30:48 pm
Note:  I do have one work unit that started crunching with the v4.35 optimized app before I switched to the v4.37 optimized app.

It had approximately two hours on v4.35 app and approximately eight hours on v4.37 app (it was about 33% complete) before it ended with Found 30 single pulses and 30 repeating pulses, exiting message.  I don't know how useful it will be, but here's a link to it anyway:  Task details (http://setiathome.berkeley.edu/result.php?resultid=1064406375).

The work unit done with only the v4.37 optimized app will complete approximately 23 hours from this post (27.1% complete).
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Jason G on 21 Nov 2008, 10:32:19 pm
Version 4.37 contains some radar blanking related overhead.  You'll notice also in the stderr.txt that SPLIT_COMPLEX is enabled in 4.37, but not in newer builds.  V5 showed it to be faster without that, and also has an improved filter that lab tests showed little/no improvement, but yielded a nice speedup over previous builds.

Given the lifespan of 4.37 is going to be a total of a couple of days, I wouldn't worry about that edition too much, especially considering it wasn't meant for release, only to pad the transition of code-base, which happened to be useful (it seems by accident) for hoisting checkpoints up a notch [Well less that, than squeezing out a few obscure bugs].

Jason

[Later: ooops :o]
E8400         cpusecs   credits   cr/cpusec
MB akv8SSE41   2260   44.1   0.019513274
ap 4.37   42941.67   760.36   0.01770681
ap 5.00   36760.94   759.74   0.020667045
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Raistmer on 22 Nov 2008, 04:07:50 am


[Later: ooops :o]
E8400         cpusecs   credits   cr/cpusec
MB akv8SSE41   2260   44.1   0.019513274
ap 4.37   42941.67   760.36   0.01770681
ap 5.00   36760.94   759.74   0.020667045


LoL :) Now AstroPulse will attract credit likers until the point of boykott for too big credits for hour :)
It's clear that state of optimization in MB build higher, but credit is lower.... Good reason to blame that credit system again :)
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Leaps-from-Shadows on 22 Nov 2008, 04:09:07 am
Woohoo!  Cruiser has finished its first v5.00 work unit at Beta!

Task details (http://setiweb.ssl.berkeley.edu/beta/result.php?resultid=4972322).

Credits (868.56) divided by CPU seconds (94123.61) = 0.009227865 credits per CPU second.

Other work units on Cruiser:
Shorty Multibeam - AK_v8_win_SSE3 app - Credits (16.84) divided by CPU seconds (2264.34) = 0.007037046 credits per CPU second.
Average Multibeam - AK_v8_win_SSE3 app - Credits (44.12) divided by CPU seconds (5268.02) = 0.008375063 credits per CPU second.

Once again confirming that AMD's Phenom efficiency is very low in comparison to Intel's Core 2 efficiency for SETI@home work units.
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Jason G on 22 Nov 2008, 04:18:19 am
Nice, That's the first Live WU I've seen that is granted & against stock wingman, because all my v5's so far have gone to pending.

@Raistmer: Sure for 32 bit users,  I'm pretty sure 64 bit multibeam is still a faster option, but it is in the right direction, and some machines 32 bit AP might indeed show equal or slightly better.  Some time will tell.
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Richard Haselgrove on 22 Nov 2008, 04:50:58 am
My v5s have been validating just fine at Beta, and here's a nice one at Main: 367818871 (http://setiathome.berkeley.edu/workunit.php?wuid=367818871). Be quick - turns into a pumpkin at midnight UTC!

Leaps - I don't want to burst your bubble, but Beta is currently overpaying credit - the tracking average was skewed by a disasterously slow v4.37 build. Main is paying as near as dammit 760 cobbles, making Cruiser's AP rate ~0.00807454. Still respectable, but not such an outright winner.

I won't be commenting on this at Main: the difference isn't all that significant, and it's nice to see someone saying something nice about AP credit for a change.
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Raistmer on 22 Nov 2008, 05:00:18 am
Hm...second result I see today and it contains 30 repetitive pulses... (Joe's addition of pulse counters is very handy :) )
It seems overflow is pretty common for AP. Some troubles with threshoulds or with blanking?
@ Richard - is it possible to say had this result non-zero indices.txt or not ?


[probably some "blanking counter" should be added to stderr too]

@All testers
Please keep eye on indices.txt in working directory (if it become non-zero length or not).
And if you keep result file, please, keep indices.txt too along with result and WU files.
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Jason G on 22 Nov 2008, 05:24:06 am
The high incidence of overflows seen might possibly be related to what I mentioned when I realised they were using psuedo-random number generators to generate the noise data.  Unfortunately most psuedo random number generators are based on shift register designs that exhibit an approximation to a maximum length sequence, a special kind of signal used to excite an impulse response from active systems due to the fact that the impulse response is a broadband pulse. .  We have to follow the project's lead in this matter, though I would have suggested using a fixed signal deformation function with known characteristics that could be subtracted as known artefacts from the results, much like the FFT splitter artefact in MB.  But I haven;t been practising my bone pointing enough, and Australia is just too far away to reach over and knock some heads together.
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Richard Haselgrove on 22 Nov 2008, 05:31:52 am

@ Richard - is it possible to say had this result non-zero indices.txt or not ?


Sorry, can't say - I didn't look. But I'll start spot-checking now. Also, I haven't been preserving data/result files since we went live, but I see that box has had two short ones already: I'll start preserving them again, in view of this discussion.

Jason, could you teach me to bone-point? I must be nearer.... Or better yet, can you teach Josef?
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Jason G on 22 Nov 2008, 05:36:16 am
I think Joe needs the most bone pointing training,   Skilled as he is in the ways that matter, placing curses is certainly not his forte.  Yourself and Joe should refer to the 'HowstuffWorks' site , article on 'Pointing the Bone' (http://reference.howstuffworks.com/pointing-the-bone-encyclopedia.htm)
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Richard Haselgrove on 22 Nov 2008, 05:39:39 am
There's an even fuller description at www.everything2.org (http://www.everything2.org/e2node/Bone%2520pointing)

Maybe Joe sees himself more as the Nangarri?
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Jason G on 22 Nov 2008, 05:42:18 am
Ha!, now that's a good one!.  Just rememeber this stuff's the real deal and not for kids to play with  ;D
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Richard Haselgrove on 22 Nov 2008, 05:58:09 am
The high incidence of overflows ...
Getting back (almost) on topic ...

It's interesting that there's such a high incidence of overflows at Main: whereas at Beta, I've only had that one short result, against 50 (so far) full-length ones at Beta. It's another weakness of the Beta testing paradigm that they tend to load a single 'tape' and let it split to the end, rather than taking shorter segments from a variety of data sources. Murphy's Law dictates that the sample tape chosen often turns out not to be a very interesting one.
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Jason G on 22 Nov 2008, 06:06:37 am
mmmm.  I'd have to put that into the 'not our problem' category... On the wacko extreme interpretation of things you see it is entirely possible the solar system is surrounded by rapidly rotating pulsars or otherwise an approaching invasion fleet uses spread spectrum technology for its communications... Of course the algorithm could just be wrong, but I'm no phd, so I'll leave that to the experts.
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Richard Haselgrove on 22 Nov 2008, 08:52:17 am

@All testers
Please keep eye on indices.txt in working directory (if it become non-zero length or not).
And if you keep result file, please, keep indices.txt too along with result and WU files.


ap_03no08af_B2_P1_00143_20081121_30048 (1065721003 (http://setiathome.berkeley.edu/result.php?resultid=1065721003))
ap_03no08ag_B2_P1_00100_20081121_20392 (1066329522 (http://setiathome.berkeley.edu/result.php?resultid=1066329522))

- both just started - have chunky indices.txt files (attached). If I'm reading the guesstimates right, these both have a high chance of pulsing out before their allotted span. Have suspended networking for a result capture.

[attachment deleted by admin]
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Raistmer on 22 Nov 2008, 10:11:06 am
Fine, thanks!
It would be usefult to get these WU + results + final indices.txt in their final state. To have some reference point.

"At sample 1024, data_chunk_now 256, strength 13.369141 > 12.100000"
So, we can see what limit established for pulses. If pulse is bigger it should be quenched ...
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Jason G on 22 Nov 2008, 10:23:29 am
NoNo, not Pulse I think... sample... Big difference... Correct me if I am wrong please.
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Leaps-from-Shadows on 22 Nov 2008, 10:44:09 am
Leaps - I don't want to burst your bubble, but Beta is currently overpaying credit - the tracking average was skewed by a disasterously slow v4.37 build. Main is paying as near as dammit 760 cobbles, making Cruiser's AP rate ~0.00807454. Still respectable, but not such an outright winner.

I won't be commenting on this at Main: the difference isn't all that significant, and it's nice to see someone saying something nice about AP credit for a change.
Don't worry ... I figured that Beta would have a different pay scale, which is why I pointed out that it was a work unit from Beta.  I'll be adding new posts to all the threads with updated results for all the v5 work units completed for Main and Beta, so people can see the differences.  I've got 11 more in my queues (six for Main, five for Beta), plus the last v4.36-branded work unit being crunched with the v4.37 SSE3 EPF app.

Some questions from a new tester...

I have to work in a couple of hours, and the v4.36-branded work unit will complete while I'm gone.  If I have it set for no network activity, will the results files be preserved?  If they are preserved ... then where?  The slots directories?

Looking in the slots directories, I can already see a couple of non-zero indeces.txt files.  Will these be preserved as well, or overwritten by new ones when new work units start?
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Jason G on 22 Nov 2008, 10:50:39 am
I have to work in a couple of hours, and the v4.36-branded work unit will complete while I'm gone.  If I have it set for no network activity, will the results files be preserved?  If they are preserved ... then where?  The slots directories?

Looking in the slots directories, I can already see a couple of non-zero indeces.txt files.  Will these be preserved as well, or overwritten by new ones when new work units start?

- Yes, they'll stay there with network activity disabled.
- Yes the files will be preserved
- WU file will be in the project folder
- corresponding results and indices.txt will be in the slot (I think  ;D)
- What we are mostly after is those that early exit.  If you can keep/copy the WU file, result and indices, then report the file, the result page online will have useful info too, I find making a text file with the result web address at the top, followed by entire task page copy/pasted, is very useful as it has stderr contents.

[Note: If you have a 'nice' early exit, say <50% progress, then I'll point you to a FTP place to deposit the files]

[EDIT: example stderr & details text file attached to clarify  ;D]
jason


[attachment deleted by admin]
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Raistmer on 22 Nov 2008, 11:41:40 am
NoNo, not Pulse I think... sample... Big difference... Correct me if I am wrong please.
Don't know what "sample" is then.
Sample can be (0,0), (0,1), (1,0), (1,1). How it can be more than 12 ?
Probably it's power[] sample, so - pulse.
But sure need to check with actual code...
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Jason G on 22 Nov 2008, 12:09:12 pm
Right, polarised signal already decoded,  complex signal  partially blanked.

Comparison:
- A single sample amplitude variation = AM radio (benefit = simple, Disadvantage = short range, noisy )
- peak moving in frequency per time interval = FM radio (benefit = longer range, disadvantage = complex decoding)
- broadband pulses = primitive marconi/morse code telegraph style( very high efficiency/ low power because part of signal can be enough to sufficiently detect, because other interferece will only mask a small part.  Long Range. Highly illegal now, because they saturate all neighbouring frequencies. (this would mess up things like telescopes and police radio  ;D) [Square wave shape at source]

So broadband pulse is not one high sample:
   - Fundamental frequency + harmonics of the fundamental in constructive interference produce the pulse. [ multiple sine waves]
   - shape is determined by bandwidth + power [ impure square wave]

Now shift the signal source out 100 light years [ but stationary] :
   - different frequency components are 'dispersed' at different rates, so phase alignment is 'stuffed'
   - square wave scattered now looks like noise
   - dedispersion reconstitutes square wave through constructive interference generating highest peak.

Now put signal on orbiting planet:
   - Doppler effect causes chirp and its reverse as planet approaches and accelerates away.
   - could be coming toward us too.
   - Now dispersed signal is also chirped, so we need dedisperson and dechirp....  :o... big cpu cycles

A pulse is not a AM signal peak, but a combination of sine wave frequencies added together.  What we detect will only ever be approximations, however, we can do this approximation on many more places/times in the sky than anybody before in history.  The worst radar removal could do is remove the 'guts' of a pulse, which we would probably detect anyway.  My 'feeling' is the abundance of overflows we get for now are either radar blanking artefacts, or genuine interstellar sources.

Jason

   
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Raistmer on 22 Nov 2008, 12:19:51 pm
My 'feeling' is the abundance of overflows we get for now are either radar blanking artefacts, or genuine interstellar sources.

Jason
 

Or wrong threshould value ;)
sngle pulse finding algorthm compares power with threshould.
"
if (power[m] > state.thresh[l]) {
"
So, at least this simplest form of pulse is power[] elment. (next round - coadded power[] elements, next - coadded and summarized through different data chunks in FFA).
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Jason G on 22 Nov 2008, 12:23:46 pm
Don't be too quick, Better too low a threshold than too high  ;).  I reckon some of these may be early human transmissions bouncing around out there too,  Ever heard of the Queen Mary phenomenon ?
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Raistmer on 22 Nov 2008, 12:36:42 pm
Don't be too quick, Better too low a threshold than too high  ;).  I reckon some of these may be early human transmissions bouncing around out there too,  Ever heard of the Queen Mary phenomenon ?

What is queen Mary?
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Jason G on 22 Nov 2008, 12:43:44 pm
A ship.  It supposedly was supposed to get a wartime transmission/order that was received by it's successor, Queen Mary 2 Elizabeth II, (~35 years? later )... basically the point is that radio waves do weird stuff, and that messing with a single sample isn't likely to upset the kind of transmission we're looking for, especially when you're talking broadband pulses that saturate surrounding frequencies... (Will check name of etc.... Though that sounds about right  ;) )

[Edit:  intended recipient was the 'Queen Mary' during world war II, recipient was the  QE2 , "Queen Elizabeth the 2nd" in 1978... No subscription by myself about paranormal factors or ' Blackwell Bracewell probe' theories either, but can attest that radio waves do weird stuff  anyway :P]
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Josef W. Segur on 22 Nov 2008, 07:16:07 pm
@All testers
Please keep eye on indices.txt in working directory (if it become non-zero length or not).
And if you keep result file, please, keep indices.txt too along with result and WU files.

ap_03no08af_B2_P1_00143_20081121_30048 (1065721003 (http://setiathome.berkeley.edu/result.php?resultid=1065721003))
ap_03no08ag_B2_P1_00100_20081121_20392 (1066329522 (http://setiathome.berkeley.edu/result.php?resultid=1066329522))

- both just started - have chunky indices.txt files (attached). If I'm reading the guesstimates right, these both have a high chance of pulsing out before their allotted span. Have suspended networking for a result capture.

That second one is a jewel. 2141 blanking indices, leaves about 6422528 samples to process so roughly 81% is blanked. A waste of CPU, but intellectually interesting. If it exits early I suspect the percent blanking reported in the result file won't agree with my calculations, it's done incrementally. In theory it should not be likely to exit early, in practice I'm not even willing to guess.

The indices.txt file isn't used for anything, just written during the initialization step which generates the internal array. It's basically the same information which was put into stderr.txt by the Linux 4.37 stock build, but BOINC will only report the last 64K of stderr. I'm glad they preserved it in this form even if it is ridiculously verbose.
                                                                       Joe
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Raistmer on 22 Nov 2008, 07:19:07 pm
We can use that file for determining blanked areas and to see if these blanked areas gave some signals or not....
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Jason G on 22 Nov 2008, 07:33:12 pm
We can use that file for determining blanked areas and to see if these blanked areas gave some signals or not....

  I should hope so, if there's no signal there then there was no point blanking it right?
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Raistmer on 22 Nov 2008, 07:50:31 pm
After blanking there should be NO signal. OR this blanking is very bad idea...

I mean we can correlate blanking index and signals in result file. They should not point on the same area in data...
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Jason G on 22 Nov 2008, 08:01:01 pm
Yes, I'm *attempting* Joe's 'full blank' now.  Don't know if I put it in the right place, We'll see.  I presume it should erase the 30 signals in JasonShort.dat without too much hassle. [If it appears to work I'll post SSE & SSE3 binary for playing with.]

[Later: Looks like 'Scorched earth' here, apart from best pulses all gone.]

Code: [Select]
Starting parse: .\testDatas\ref-astropulse_5.00_windows_intelx86.exe-JasonShort.
dat.res
Successfully parsed: .\testDatas\ref-astropulse_5.00_windows_intelx86.exe-JasonS
hort.dat.res
--- Found <ap_signal> tags: 40
Starting parse: .\testDatas\result-ap_5.00r69_SSE3_blankOverride.exe-JasonShort.
dat.res
Successfully parsed: .\testDatas\result-ap_5.00r69_SSE3_blankOverride.exe-JasonS
hort.dat.res
--- Found <ap_signal> tags: 10
Result      : Weakly similar or Different.

Unfortunately my hack doesn;t write indices.txt, but I do print a line to stderr whenever the randomize_indices() if block executes (stderr attached)

[Pending result from a full WU blank,I'll stop whining about the possibility of artefacts... and start whining about the blanking width instead  :D]


[attachment deleted by admin]
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Leaps-from-Shadows on 23 Nov 2008, 02:31:21 am
Update from Cruiser:

First (and last) Astropulse work unit crunched with v4.37 optimized app completed on Main.  Task details (http://setiathome.berkeley.edu/result.php?resultid=1065144181).
Credits (759.31) divided by CPU seconds (112876.70) = 0.006726898 credits per CPU second.

Second Astropulse work unit crunched with v5.00 optimized app completed on Beta.  Task details (http://setiweb.ssl.berkeley.edu/beta/result.php?resultid=4972254).
Credits (868.56) divided by CPU seconds (89492.05) = 0.009705443 credits per CPU second.

I should have my first v5.00 unit crunched on Main in six hours, plus three more around this time tomorrow (two on Main, one on Beta).
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Richard Haselgrove on 23 Nov 2008, 06:49:25 am
@All testers
Please keep eye on indices.txt in working directory (if it become non-zero length or not).
And if you keep result file, please, keep indices.txt too along with result and WU files.

ap_03no08af_B2_P1_00143_20081121_30048 (1065721003 (http://setiathome.berkeley.edu/result.php?resultid=1065721003))
ap_03no08ag_B2_P1_00100_20081121_20392 (1066329522 (http://setiathome.berkeley.edu/result.php?resultid=1066329522))

- both just started - have chunky indices.txt files (attached). If I'm reading the guesstimates right, these both have a high chance of pulsing out before their allotted span. Have suspended networking for a result capture.

That second one is a jewel. 2141 blanking indices, leaves about 6422528 samples to process so roughly 81% is blanked. A waste of CPU, but intellectually interesting. If it exits early I suspect the percent blanking reported in the result file won't agree with my calculations, it's done incrementally. In theory it should not be likely to exit early, in practice I'm not even willing to guess.

The indices.txt file isn't used for anything, just written during the initialization step which generates the internal array. It's basically the same information which was put into stderr.txt by the Linux 4.37 stock build, but BOINC will only report the last 64K of stderr. I'm glad they preserved it in this form even if it is ridiculously verbose.
                                                                       Joe

Both tasks ran full-length and have now been allowed to report. They both found a limited number of single pulses (4/7 respectively) plus 30 repetitive pulses. I have captured the output files and will upload the 'jewel' (only) to FTP. The other one can stay in cold storage until Jason's mum has got her bandwidth cap removed....

Note for anyone else assisting in this research: the indices.txt file is deleted from the slot directory when the WU finishes processing, whether networking is disabled or not - and of course, a WU can finish early at any moment. Since it's created at the beginning for the run and not changed thereafter, it would be best to copy it to a safe place as soon as the WU starts.
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Jason G on 23 Nov 2008, 07:05:03 am
 :'( 30  repetitive pulses in the ~81% blanked task? duh,  I'm just going to take a look and see what I can see... If full-blanked shows a similar pattern we (or the project, rather) has a problem IMO.
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Jason G on 23 Nov 2008, 07:10:57 am
Hmmm, arrchive doesn't seem to open.  Any problem during upload (besides speed ?)  LoL ... too quick on the trigger  ;D  I'll wait 'til the upload finishes eh  ;)
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Richard Haselgrove on 23 Nov 2008, 07:35:25 am
Upload complete - you can pull that trigger now.
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Jason G on 23 Nov 2008, 07:50:11 am
Got it, and looking at the result file.  All signals listed are same <time>, which doesn't make sense to me uury, so I'm looking for some periodic relationships between one repeating pulse and the next...
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Richard Haselgrove on 23 Nov 2008, 08:18:17 am
Not quite all. The 'jewel' has seven pulses with different times, and the other one (not yet uploaded) has four - so I guess that the single pulses are accurately timed, and the repetitive pulses just have a dummy value inserted - which happens to match the <start> time in <data_desc>.

I'm slightly surprised by the blanking report at the start of the result file analysis: <percent_blanked> -18.750000 for the 'jewel', and <percent_blanked> -21.785156 for the other.
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Raistmer on 23 Nov 2008, 08:37:51 am

I'm slightly surprised by the blanking report at the start of the result file analysis: <percent_blanked> -18.750000 for the 'jewel', and <percent_blanked> -21.785156 for the other.
So, Joe's % estimate and Josh's differ... Interesting, how much % will be in our current totally-blanked run....
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Jason G on 23 Nov 2008, 08:40:29 am
Picked one of the non-repetitive signals at random to look at: [playing with numbers still...]

<ap_signal>
  <fft_num>30015488</fft_num>
  <peak_bin>30024704</peak_bin>

Which I (with less than novice skill) make to be be data_chunk_now: 7503872 + 9216 samples (A bit less than a third into the data chunk?) at sample 30024704

That would place that somewhere: in the vicinity of the blanking index:

"At sample 30035968, data_chunk_now 7508992, strength 12.390625 > 12.100000"

So the peak is 11264 samples before the index... On the edge of the blank perhaps shifted by dedispersion?  I'm not sure, but I think this shouldn't be there  :-\  [Now where did I see that with before?  20480...]

Oh $@!$#... here (http://lunatics.kwsn.net/5-windows/astropulse-signal-record-formate.msg10925.html#msg10925)  I'll wait for Joe's expert analysis, but I reckon that one's an artefact [of some sort]
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Richard Haselgrove on 23 Nov 2008, 08:57:02 am
I'm slightly surprised by the blanking report at the start of the result file analysis: <percent_blanked> -18.750000 for the 'jewel', and <percent_blanked> -21.785156 for the other.

So, Joe's % estimate and Josh's differ... Interesting, how much % will be in our current totally-blanked run....

Probably not - since a negative percentage doesn't make much sense, I'm reading that as (100-18.750000) or 81.25%, which is perfectly in line with Joe's "roughly 81%".

But the equivalent calculation for the other WU suggests <percent_blanked> 78.214844 - which is odd, since there are far fewer blanking indices (787) in that file.
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Jason G on 23 Nov 2008, 09:21:55 am
probably a misplaced division of integer components with the typecasts in the wrong place.  That should definitely be in the range 0.0-->1.0 looking at the code (so proportion rather than percentage).  The value is likely a fluke that it bears any apparent relationship IMO.
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Leaps-from-Shadows on 23 Nov 2008, 09:50:39 am
First Astropulse work unit crunched with v5.00 optimized app completed on Main.  Task details (http://setiathome.berkeley.edu/result.php?resultid=1065309955).
Credits (759.31) divided by CPU seconds (92202.00) = 0.008235288 credits per CPU second.

Compare to other work units:
Shorty Multibeam - AK_v8_win_SSE3 app - Credits (16.84) divided by CPU seconds (2264.34) = 0.007037046 credits per CPU second.
Average Multibeam - AK_v8_win_SSE3 app - Credits (44.12) divided by CPU seconds (5268.02) = 0.008375063 credits per CPU second.

This version finally seems to bring Astropulse and Multibeam units into equality as far as credits per CPU second - at least on AMD processors.  Woohoo!
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Jason G on 23 Nov 2008, 10:08:53 am
Good for the AMD chips then.  Keep an eye on the wingman for that task though,  looks like a machine that might be having some issues. If not fixed, best hope might be that they error out quickly for reissue to a new wingman, otherwise could be waiting awhile for validation.
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Richard Haselgrove on 23 Nov 2008, 10:13:47 am
Attaching result files for my two blanking cases, as a courtesy to Joe's dialup.

Edit - it would be even more of a courtesy if I attached the zipped files!

[attachment deleted by admin]
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Jason G on 23 Nov 2008, 11:50:07 am
Chose to examine another single pulse, and this one is not so near blanked sections it seems

<ap_signal>
  <fft_num>10190848</fft_num>
  <peak_bin>10199552</peak_bin>

closest index is at 11808768

immediately following one similar story, <fft_num>9093120</fft_num>
  <peak_bin>9105920</peak_bin>

 but notice that both these seem to lie between blanked regions around the middle, so wonder how much a part interactions with the higher dm figure on these around the middle of processing might play, and so how far to factor in adjacent data / proximity due to dedispersion.
 
[perhaps later on, a comparison run of this WU on a pre-blanking build might confirm whether these are legitimate pulses or not (assuming can get it to run w/o really early exit) ]

Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Raistmer on 23 Nov 2008, 12:11:43 pm

[perhaps later on, a comparison run of this WU on a pre-blanking build might confirm whether these are legitimate pulses or not (assuming can get it to run w/o really early exit) ]


Good idea!
To look if all new pulses could be found inside old app.
Think we should do this investigation.
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Jason G on 23 Nov 2008, 12:59:27 pm
Yeah, still a while to go on my full-blank run... will see after that tomorrow. [We seem to have gravitated development discussion unintentionally out to discussion forum (probably my fault).  I suggest we migrate back to avoid confusing users]
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Raistmer on 23 Nov 2008, 01:54:41 pm
Yeah, still a while to go on my full-blank run... will see after that tomorrow. [We seem to have gravitated development discussion unintentionally out to discussion forum (probably my fault).  I suggest we migrate back to avoid confusing users]
Sure :) You can move needed part as moderator if you think it's worth efforts...
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Jason G on 23 Nov 2008, 01:58:41 pm
No it isn't worth mucking about, just reminding myself before I any more out of hand and start swearing in the wrong place  ;D
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Richard Haselgrove on 23 Nov 2008, 02:21:04 pm
No it isn't worth mucking about, just reminding myself before I any more out of hand and start swearing in the wrong place  ;D

No, please don't move anything - I posted (http://setiweb.ssl.berkeley.edu/beta/forum_thread.php?id=1418&nowrap=true#35452) at SETI Beta several hours ago, letting Josh know that much of this discussion would be visible to him even before he registered on this site (though of course he'll have access to much more once we have an account ID to upgrade).
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Jason G on 23 Nov 2008, 02:32:35 pm
LoL, OK fair enough.  Next theories though lead to movie night at arecibo involving lots of microwaved popcorn.
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Richard Haselgrove on 23 Nov 2008, 02:51:23 pm
Get the popcorn ready - got another chunky indices.txt file to watch through - 1344 sample points..

ap_04no08ac_B4_P1_00145_20081123_17576 (1067934018 (http://setiathome.berkeley.edu/result.php?resultid=1067934018))

[attachment deleted by admin]
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Jason G on 23 Nov 2008, 03:09:53 pm
If I didn't know any better (which I don't) I'd swear that pickup has a loose ground wire.
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Josef W. Segur on 24 Nov 2008, 01:02:04 am
Get the popcorn ready - got another chunky indices.txt file to watch through - 1344 sample points..

ap_04no08ac_B4_P1_00145_20081123_17576 (1067934018 (http://setiathome.berkeley.edu/result.php?resultid=1067934018))

You might find it interesting to watch the changes in percent_blanked in the result file (it is rewritten with each checkpoint). The count of blanked samples should be limited to one DM, but it keeps on adding AFAICT. Because the count is kept in a signed integer, it can overflow into negative territory. The limits may be +/- 6400%, but if so I'm surprised we haven't seen larger values already. My current offline test on a slow system progressed from -38.121094 to -31.998047 in the last 40 minutes or so.

Edit: Aha! It doesn't really convert to percent, it simply divides the blanked sample count by the total sample count, so if they were equal the field would show 1.00000 rather than 100.0000 With that, the max range becomes +/-64. The name is wrong as well as the method.
                                                                         Joe
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Jason G on 24 Nov 2008, 01:29:03 am
Yep, noticed the nice integer division followed by typecast to double.  Always did find 32 bit long & normal signed ints funny (or the contexts I've seen them used in anyway), changing the values to unsigned would probably be enough to fix it.  (but obviously still wouldn't be percent)

Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Richard Haselgrove on 24 Nov 2008, 09:13:40 am
Get the popcorn ready - got another chunky indices.txt file to watch through - 1344 sample points..

ap_04no08ac_B4_P1_00145_20081123_17576 (1067934018 (http://setiathome.berkeley.edu/result.php?resultid=1067934018))

Finished at full runtime - 7 single pulses and 30 repetitive pulses. Result file attached - full WU datapak available if you want it uploading.

[attachment deleted by admin]
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Jason G on 24 Nov 2008, 09:46:23 am
Might as well wack it up,  Speed's back up now  ;D
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Richard Haselgrove on 24 Nov 2008, 09:54:08 am
What's the next description beyond 'jewel'?

 ap_04no08ae_B5_P0_00273_20081124_23735 (1068934402 (http://setiathome.berkeley.edu/result.php?resultid=1068934402))

has 3548 blanking indices, 448 of them with signal strength over 100. Highest signal strength is 213.925781

Would anyone like to estimate the likely scientific value of the next 15 hours work?

[attachment deleted by admin]
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Richard Haselgrove on 24 Nov 2008, 09:54:56 am
Might as well wack it up,  Speed's back up now  ;D

Will do, give us a mo.

Wow, that's better. >283Kbps sustained. Zip contains v5 result, indices, std_err.

What's the best way to get a v4 result offline to compare?
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Raistmer on 24 Nov 2008, 09:55:01 am
Get the popcorn ready - got another chunky indices.txt file to watch through - 1344 sample points..

ap_04no08ac_B4_P1_00145_20081123_17576 (1067934018 (http://setiathome.berkeley.edu/result.php?resultid=1067934018))

Finished at full runtime - 7 single pulses and 30 repetitive pulses. Result file attached - full WU datapak available if you want it uploading.
Try to do the same WU with 4.x version, then send both result files along with indices.txt, stderr.txt and WU itself to FTP, please.
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Raistmer on 24 Nov 2008, 10:24:33 am

What's the best way to get a v4 result offline to compare?
Think opt app will be fine. No need to use 4.x stock one.
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Leaps-from-Shadows on 24 Nov 2008, 11:21:15 am
Got some more completed work units on Cruiser:

Main:368768645 (http://setiathome.berkeley.edu/workunit.php?wuid=368768645) (Pending), 368768653 (http://setiathome.berkeley.edu/workunit.php?wuid=368768653) (Validated)
Beta:  1563494 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1563494) (Validated)

Cruiser's compiled results so far:

Main:
368768645 (http://setiathome.berkeley.edu/workunit.php?wuid=368768645) (Pending) - 0.008021076 credits per CPU second
368768653 (http://setiathome.berkeley.edu/workunit.php?wuid=368768653) (Validated) - 0.008183099 credits per CPU second
368496971 (http://setiathome.berkeley.edu/workunit.php?wuid=368496971) (Pending) - 0.008235288 credits per CPU second

368418713 (http://setiathome.berkeley.edu/workunit.php?wuid=368418713) (v4.37, Pending) - 0.006726898 credits per CPU second

Beta:
1563494 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1563494) (Validated) - 0.009484371 credits per CPU second
1565756 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1565756) (Validated) - 0.009705443 credits per CPU second
1564992 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1564992) (Validated) - 0.009227865 credits per CPU second

Fairly consistent on both Main and Beta...

For comparison:
Shorty Multibeam (16.84 credits):  0.007037046 credits per CPU second
Average Multibeam (44.12 credits):  0.008375063 credits per CPU second
Long Multibeam (63.86 credits):  0.010619245 credits per CPU second
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Raistmer on 24 Nov 2008, 01:58:13 pm
Again, rep pulses overflow....
single pulses: 8
repetitive pulses: 30

single pulses: 12
repetitive pulses: 30

    single pulses: 7
repetitive pulses: 30

All main ended with rep overflow but no such effect on beta. It seems tape w/o radar used on beta .
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Josef W. Segur on 24 Nov 2008, 05:50:30 pm
...main ended with rep overflow but no such effect on beta. It seems tape w/o radar used on beta .

About a week ago I went through Richard's Beta reports from opt 5.00. There were 26 of them, 1 was a 30+30 overflow, 3 more had 30 repetitive pulses, 4 had no reportable pulses of either type. In total, there were 108 single pulses reported and 179 repetitive pulses. That's a very small sample, but at least indicates the "tape" covers a wide range of conditions.

The "radar removal" is looking for a DC level from the length 1024 FFTs it uses, which I think is an effect of getting a strong enough input to saturate the receiver channel, or at least cause compression. Assuming those incidents are caused by radar signals bouncing back from an airplane, the strongest ones are going to be when the bounce arrives on one of the side-lobe peaks and/or when the aircraft is oriented to produce the strongest reflection. It's a hugely complicated set of interactions.
                                                                      Joe
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Leaps-from-Shadows on 24 Nov 2008, 11:20:58 pm
Quote
Got some more completed work units on Cruiser:

Main:  368768645 (http://setiathome.berkeley.edu/workunit.php?wuid=368768645) (Pending), 368768653 (http://setiathome.berkeley.edu/workunit.php?wuid=368768653) (Validated)
Beta:  1563494 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1563494) (Validated)

Cruiser's compiled results so far:

Main:
368768645 (http://setiathome.berkeley.edu/workunit.php?wuid=368768645) (Pending) - 0.008021076 credits per CPU second
368768653 (http://setiathome.berkeley.edu/workunit.php?wuid=368768653) (Validated) - 0.008183099 credits per CPU second
368496971 (http://setiathome.berkeley.edu/workunit.php?wuid=368496971) (Pending) - 0.008235288 credits per CPU second

368418713 (http://setiathome.berkeley.edu/workunit.php?wuid=368418713) (v4.37, Pending) - 0.006726898 credits per CPU second

Beta:
1563494 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1563494) (Validated) - 0.009484371 credits per CPU second
1565756 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1565756) (Validated) - 0.009705443 credits per CPU second
1564992 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1564992) (Validated) - 0.009227865 credits per CPU second

Fairly consistent on both Main and Beta...

For comparison:
Shorty Multibeam (16.84 credits):  0.007037046 credits per CPU second
Average Multibeam (44.12 credits):  0.008375063 credits per CPU second
Long Multibeam (63.86 credits):  0.010619245 credits per CPU second
Add another one to the Main list:
368768654 (http://setiathome.berkeley.edu/workunit.php?wuid=368768654) (Validated, canonical result) - 0.008136883 credits per CPU second

Still, it has this:
    single pulses: 4
repetitive pulses: 30
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Josef W. Segur on 25 Nov 2008, 12:08:34 am
What's the next description beyond 'jewel'?

 ap_04no08ae_B5_P0_00273_20081124_23735 (1068934402 (http://setiathome.berkeley.edu/result.php?resultid=1068934402))

has 3548 blanking indices, 448 of them with signal strength over 100. Highest signal strength is 213.925781

Would anyone like to estimate the likely scientific value of the next 15 hours work?

That's a 90% blanker. There's one place with slightly over 2.7 M unblanked samples, another with over 400 K, and two more very short ones.

With considerable refinement, the app could just process the unblanked portions using only the appropriate tests. The short FFA would be appropriate if shifted to work within that 2.7 M section, the long FFA makes no sense, single pulse searches work on only 32 K samples and could be tried on all the unblanked data. Processing time that way should be less than 10% of the full run, and scientific value would be maximized. That assumes it's possible to have actual clean data in short sections surrounded by noise, we're not on a path to discover that yet.

As it stands, scientific value is going to lie in the realm of illustrating flaws in the implementation; sometimes that's the most important part of the scientific method.
                                                                          Joe
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Richard Haselgrove on 25 Nov 2008, 07:48:55 am

That's a 90% blanker. There's one place with slightly over 2.7 M unblanked samples, another with over 400 K, and two more very short ones.

With considerable refinement, the app could just process the unblanked portions using only the appropriate tests. The short FFA would be appropriate if shifted to work within that 2.7 M section, the long FFA makes no sense, single pulse searches work on only 32 K samples and could be tried on all the unblanked data. Processing time that way should be less than 10% of the full run, and scientific value would be maximized. That assumes it's possible to have actual clean data in short sections surrounded by noise, we're not on a path to discover that yet.

As it stands, scientific value is going to lie in the realm of illustrating flaws in the implementation; sometimes that's the most important part of the scientific method.
                                                                          Joe

Turned out to be over 16 hours - slightly slower than normal for r69 on this box. Don't know if that's significant.

Found 6 single pulses and 27 repetitive pulses - I suppose there are fewer edges to have edge artefacts on, when the whole thing is blanked ;). Result attached.

Oh, and please contribute to to my 'dilemma (http://setiathome.berkeley.edu/forum_thread.php?id=50396)' thread at SETI :-\

[ooops, wrong file]

[attachment deleted by admin]
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Jason G on 25 Nov 2008, 07:53:40 am
..
Oh, and please contribute to to my 'dilemma (http://setiathome.berkeley.edu/forum_thread.php?id=50396)' thread at SETI :-\
..

Done.  Feel free to disagree.
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Richard Haselgrove on 25 Nov 2008, 07:57:53 am

Done.  Feel free to disagree.


Yours is probably the nearest thing to a 'right' answer, but I just wanted to get the debate out into the open.....

I'll probably end up doing it with v5, just so we get another indices.txt file to play with.
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Jason G on 25 Nov 2008, 08:09:35 am
I'm seeing a similar thing in your result file posted before , to my 'full blanked' result... A bunch of repeating pulses collected at the end of the WU.

My 3 results... (attached) 4.35, 5r69 and 5r69FullBlank for a WU that had lots of signals, but didn't overflow with either 4.35 or v5,

Note full blank repeating pulses x 30 all around <peak_bin> 33538050 give or take.  They also seem to be all around <scale> 4, so we probably can work out the average power there.  I think a step from the randomised to off-the edge or similar phenomemon might be occurring.  Exactly where does last dm get its overlap data from during dechirp? and is full blank randomising that too? or have we a nice step to raw data, zero or some averaged value?

Jason

[Note on FFT/High pass filter thing: A quick look it does *appear to me* to be the full complex fft result we're processing.  [But note that I always hated digging through buried functionality just to find what should be part of the function name, or a direct parameter] . If so that would make the frequency range -pi to +pi periodic,  low frequencies in the middle, a highpass look like an inverted 'U', what dechirp does with this I don't know yet, but later in coadds after inverse fft redundant half of *something* appears to be 'chucked out' though I'm dubious on the in-code comments as to the reasoning]

Whoops [ forgot attachment .. attached... ]

[attachment deleted by admin]
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Raistmer on 25 Nov 2008, 09:59:07 am
@Richard
Done too. Maybe it's pretty radical proposal, but there is plenty of SETI search for MB tasks still to probably waste resourses on app under big suspiction for now....

@Jason
LoL, comments... I liked that comments on first reading :))))) But it's still unclear, did he check or not...
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Leaps-from-Shadows on 25 Nov 2008, 11:22:34 am
I have suspended the seven queued work units that Cruiser has for Main.  There are two in progress that I'm going to let finish, then it's up to the coding experts to decide whether the suspended units get aborted or crunched.

I'll be awaiting your determination.
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Jason G on 25 Nov 2008, 11:45:28 am
Personally I would continue crunching in case Berkeley need data to analyse the performance characteristics themselves.  Whether the results are fundamentally correct or not, in scientific terms they still have great value for verification, and further refinement if needed.  I don't think stopping crunching them based on the anomalies we've seen so far is necessarily the ideal situation, as they could contain important pointers toward further development, or even indicate some real, little understood phenomenon lurking in some WUs.

Just my 2 cents.

Jason
 
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Raistmer on 25 Nov 2008, 12:20:53 pm
Sure, maybe....  or maybe not.
As usual there is no announcement what is the aim of current run and, recall situation with beta testing, I personally has no trust at all to ability of AP responsible person to correctly judge what processing power is needed for current aim. It can be just debug run, installed on few thousands of hosts, with easy. In such situation most of results will go into trash because just unneeded.
I will continue running AP offline to clear situation, but no "production" run. Wanna understand what is going on and why.
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Josef W. Segur on 25 Nov 2008, 04:12:45 pm
I'm seeing a similar thing in your result file posted before , to my 'full blanked' result... A bunch of repeating pulses collected at the end of the WU.

My 3 results... (attached) 4.35, 5r69 and 5r69FullBlank for a WU that had lots of signals, but didn't overflow with either 4.35 or v5,

Note full blank repeating pulses x 30 all around <peak_bin> 33538050 give or take.  They also seem to be all around <scale> 4, so we probably can work out the average power there.
 

Repetitive pulses with <scale> 4 are from the the short FFA, and although <peak_bin> shows a high number they are actually working on data from the first 1M samples. Subtract 33538048 from the <peak_bin> value to correct it. Even then, it looks like the remaining value needs to be scaled up by 2^(scale + ffa_scale).

<scale> 4 says the power was x16 when copied to the FFA buffer, if there's any non-zero <ffa_scale> value those are also doublings. But the actual folding also increases power by a factor between 137 and 274, it should be possible to figure that out from the <period> but I haven't looked at that closely enough to know what units it's using.

Quote
I think a step from the randomised to off-the edge or similar phenomemon might be occurring.  Exactly where does last dm get its overlap data from during dechirp? and is full blank randomising that too? or have we a nice step to raw data, zero or some averaged value?

Jason
...

For "blanking", a 32K pseudo-random data sample chunk replaces 32K real samples before the forward FFT, there's no chance that any deeper parts of the single pulse processing will see a mix.

There IS a flaw in copying data to the FFA buffers, though. Because the data chunks are overlapped by half, only the first half of the coadded chunk is copied, then when the last chunk to be copied for a dm is reached there's half a chunk worth which needs special handling. Rather than putting the last half of the chunk there, the code puts a second copy of the first half. That's wrong and should be corrected, but I doubt it is contributing much to the weird results.
                                                                    Joe
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Leaps-from-Shadows on 26 Nov 2008, 12:51:34 am
Cruiser's compiled results so far:

Main:
368768645 (http://setiathome.berkeley.edu/workunit.php?wuid=368768645) (Validated) - 0.008021076 credits per CPU second
368768653 (http://setiathome.berkeley.edu/workunit.php?wuid=368768653) (Validated) - 0.008183099 credits per CPU second
368496971 (http://setiathome.berkeley.edu/workunit.php?wuid=368496971) (Pending) - 0.008235288 credits per CPU second
368768654 (http://setiathome.berkeley.edu/workunit.php?wuid=368768654) (Validated, canonical result) - 0.008136883 credits per CPU second
368768675 (http://setiathome.berkeley.edu/workunit.php?wuid=368768675) (Pending) - 0.008161555 credits per CPU second

368418713 (http://setiathome.berkeley.edu/workunit.php?wuid=368418713) (v4.37, Pending) - 0.006726898 credits per CPU second

Beta:
1563494 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1563494) (Validated) - 0.009484371 credits per CPU second
1565756 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1565756) (Validated) - 0.009705443 credits per CPU second
1564992 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1564992) (Validated) - 0.009227865 credits per CPU second
1565745 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1565745) (Pending) (Task details (http://setiweb.ssl.berkeley.edu/beta/result.php?resultid=4972425)) - 0.009846161 credits per CPU second

For comparison:
Shorty Multibeam (16.84 credits):  0.007037046 credits per CPU second
Average Multibeam (44.12 credits):  0.008375063 credits per CPU second
Long Multibeam (63.86 credits):  0.010619245 credits per CPU second

I have decided to keep crunching Astropulse units on Main, but only one at a time.
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Leaps-from-Shadows on 26 Nov 2008, 09:39:29 pm
Cruiser's compiled results so far:

Main:
368768645 (http://setiathome.berkeley.edu/workunit.php?wuid=368768645) (Validated) - 0.008021076 credits per CPU second
368768653 (http://setiathome.berkeley.edu/workunit.php?wuid=368768653) (Validated) - 0.008183099 credits per CPU second
368496971 (http://setiathome.berkeley.edu/workunit.php?wuid=368496971) (Pending) - 0.008235288 credits per CPU second
368768654 (http://setiathome.berkeley.edu/workunit.php?wuid=368768654) (Validated, canonical result) - 0.008136883 credits per CPU second
368768675 (http://setiathome.berkeley.edu/workunit.php?wuid=368768675) (Pending) - 0.008161555 credits per CPU second
368768681 (http://setiathome.berkeley.edu/workunit.php?wuid=368768681) (Validated) - 0.008024405 credits per CPU second

368418713 (http://setiathome.berkeley.edu/workunit.php?wuid=368418713) (v4.37, Pending) - 0.006726898 credits per CPU second

Beta:
1563494 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1563494) (Validated) - 0.009484371 credits per CPU second
1565756 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1565756) (Validated) - 0.009705443 credits per CPU second
1564992 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1564992) (Validated) - 0.009227865 credits per CPU second
1565745 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1565745) (Pending) (Task details (http://setiweb.ssl.berkeley.edu/beta/result.php?resultid=4972425)) - 0.009846161 credits per CPU second
1565021 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1565021) (Pending) (Task details (http://setiweb.ssl.berkeley.edu/beta/result.php?resultid=4972419)) - 0.009777995 credits per CPU second

For comparison:
Shorty Multibeam (16.84 credits):  0.007037046 credits per CPU second
Average Multibeam (44.12 credits):  0.008375063 credits per CPU second
Long Multibeam (63.86 credits):  0.010619245 credits per CPU second
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Leaps-from-Shadows on 27 Nov 2008, 03:32:54 pm
Cruiser's compiled results so far:

Main:
368768645 (http://setiathome.berkeley.edu/workunit.php?wuid=368768645) (Validated) - 0.008021076 credits per CPU second
368768653 (http://setiathome.berkeley.edu/workunit.php?wuid=368768653) (Validated) - 0.008183099 credits per CPU second
368496971 (http://setiathome.berkeley.edu/workunit.php?wuid=368496971) (Pending) - 0.008235288 credits per CPU second
368768654 (http://setiathome.berkeley.edu/workunit.php?wuid=368768654) (Validated, canonical result) - 0.008136883 credits per CPU second
368768675 (http://setiathome.berkeley.edu/workunit.php?wuid=368768675) (Pending) - 0.008161555 credits per CPU second
368768681 (http://setiathome.berkeley.edu/workunit.php?wuid=368768681) (Validated) - 0.008024405 credits per CPU second
369656439 (http://setiathome.berkeley.edu/workunit.php?wuid=369656439) (Pending) - 0.007830532 credits per CPU second
349739781 (http://setiathome.berkeley.edu/workunit.php?wuid=349739781) (Validated) - 0.008531656 credits per CPU second

368418713 (http://setiathome.berkeley.edu/workunit.php?wuid=368418713) (v4.37, Pending) - 0.006726898 credits per CPU second

Beta:
1563494 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1563494) (Validated) - 0.009484371 credits per CPU second
1565756 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1565756) (Validated) - 0.009705443 credits per CPU second
1564992 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1564992) (Validated) - 0.009227865 credits per CPU second
1565745 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1565745) (Pending) (Task details (http://setiweb.ssl.berkeley.edu/beta/result.php?resultid=4972425)) - 0.009846161 credits per CPU second
1565021 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1565021) (Pending) (Task details (http://setiweb.ssl.berkeley.edu/beta/result.php?resultid=4972419)) - 0.009777995 credits per CPU second
1565898 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1565898) (Pending) (Task details (http://setiweb.ssl.berkeley.edu/beta/result.php?resultid=4972760)) - 0.009551190 credits per CPU second

For comparison:
Shorty Multibeam (16.84 credits):  0.007037046 credits per CPU second
Average Multibeam (44.12 credits):  0.008375063 credits per CPU second
Long Multibeam (63.86 credits):  0.010619245 credits per CPU second

For the first time, I had two v5.00 work units on Main (369656439 (http://setiathome.berkeley.edu/workunit.php?wuid=369656439) and 349739781 (http://setiathome.berkeley.edu/workunit.php?wuid=349739781)) that did not end with pulse overflows, and one on Beta (Task details (http://setiweb.ssl.berkeley.edu/beta/result.php?resultid=4972760)) that did end with a pulse overflow.
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Raistmer on 27 Nov 2008, 04:08:24 pm
this one near overflow too
single pulses: 8
repetitive pulses: 27

Pity that we haven's % of blanking in stderr too....
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Richard Haselgrove on 27 Nov 2008, 04:45:58 pm
..
Oh, and please contribute to to my 'dilemma (http://setiathome.berkeley.edu/forum_thread.php?id=50396)' thread at SETI :-\
..

Done.  Feel free to disagree.


And now the WU is done, too. I decided to let it run with opti v5.00: it validated the v5.00 stock, and the v4.36 stock got zilch. Not even weakly similar. Similar situation for the first of Geek@Play's dilemma cases, which has now reported. I can't say I'm surprised, but I am disappointed. I very much doubt Eric is likely to waste his holiday weekend writing a manual credit script - and the results will be purged by then, anyway.
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Raistmer on 27 Nov 2008, 05:34:42 pm
Well, I'm more concerned by the meaning (or meaningless) of current AP work than credits actually...
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Leaps-from-Shadows on 28 Nov 2008, 04:33:29 pm
Cruiser's compiled results so far:

Main:
368768645 (http://setiathome.berkeley.edu/workunit.php?wuid=368768645) (Validated) - 0.008021076 credits per CPU second
368768653 (http://setiathome.berkeley.edu/workunit.php?wuid=368768653) (Validated) - 0.008183099 credits per CPU second
368496971 (http://setiathome.berkeley.edu/workunit.php?wuid=368496971) (Pending) - 0.008235288 credits per CPU second
368768654 (http://setiathome.berkeley.edu/workunit.php?wuid=368768654) (Validated, canonical result) - 0.008136883 credits per CPU second
368768675 (http://setiathome.berkeley.edu/workunit.php?wuid=368768675) (Pending) - 0.008161555 credits per CPU second
368768681 (http://setiathome.berkeley.edu/workunit.php?wuid=368768681) (Validated) - 0.008024405 credits per CPU second
369656439 (http://setiathome.berkeley.edu/workunit.php?wuid=369656439) (Pending) - 0.007830532 credits per CPU second
349739781 (http://setiathome.berkeley.edu/workunit.php?wuid=349739781) (Validated) - 0.008531656 credits per CPU second
369656217 (http://setiathome.berkeley.edu/workunit.php?wuid=369656217) (Pending) - 0.008278782 credits per CPU second

368418713 (http://setiathome.berkeley.edu/workunit.php?wuid=368418713) (v4.37, Pending) - 0.006726898 credits per CPU second

Beta:
1563494 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1563494) (Validated) - 0.009484371 credits per CPU second
1565756 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1565756) (Validated) - 0.009705443 credits per CPU second
1564992 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1564992) (Validated) - 0.009227865 credits per CPU second
1565745 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1565745) (Pending) (Task details (http://setiweb.ssl.berkeley.edu/beta/result.php?resultid=4972425)) - 0.009846161 credits per CPU second
1565021 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1565021) (Pending) (Task details (http://setiweb.ssl.berkeley.edu/beta/result.php?resultid=4972419)) - 0.009777995 credits per CPU second
1565898 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1565898) (Pending) (Task details (http://setiweb.ssl.berkeley.edu/beta/result.php?resultid=4972760)) - 0.009551190 credits per CPU second
1628888 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1628888) (Pending) (Task details (http://setiweb.ssl.berkeley.edu/beta/result.php?resultid=5037376)) - 0.009486180 credits per CPU second

For comparison:
Shorty Multibeam (16.84 credits):  0.007037046 credits per CPU second
Average Multibeam (44.12 credits):  0.008375063 credits per CPU second
Long Multibeam (63.86 credits):  0.010619245 credits per CPU second
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Leaps-from-Shadows on 30 Nov 2008, 01:47:09 am
Cruiser's compiled results so far:

Main:
368768645 (Validated) - 0.008021076 credits per CPU second
368768653 (Validated) - 0.008183099 credits per CPU second
368496971 (http://setiathome.berkeley.edu/workunit.php?wuid=368496971) (Pending) - 0.008235288 credits per CPU second
368768654 (Validated) - 0.008136883 credits per CPU second
368768675 (http://setiathome.berkeley.edu/workunit.php?wuid=368768675) (Validated) - 0.008161555 credits per CPU second
368768681 (Validated) - 0.008024405 credits per CPU second
369656439 (http://setiathome.berkeley.edu/workunit.php?wuid=369656439) (Validated) - 0.007830532 credits per CPU second
349739781 (Validated) - 0.008531656 credits per CPU second
369656217 (http://setiathome.berkeley.edu/workunit.php?wuid=369656217) (Pending) - 0.008278782 credits per CPU second
370050505 (http://setiathome.berkeley.edu/workunit.php?wuid=370050505) (Validated) - 0.008334832 credits per CPU second
370050504 (http://setiathome.berkeley.edu/workunit.php?wuid=370050504) (Validated) - 0.008610554 credits per CPU second

368418713 (http://setiathome.berkeley.edu/workunit.php?wuid=368418713) (v4.37, Pending) - 0.006726898 credits per CPU second

Beta:
1563494 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1563494) (Validated) - 0.009484371 credits per CPU second
1565756 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1565756) (Validated) - 0.009705443 credits per CPU second
1564992 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1564992) (Validated) - 0.009227865 credits per CPU second
1565745 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1565745) (Validated) - 0.009846161 credits per CPU second
1565021 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1565021) (Pending) (Task details (http://setiweb.ssl.berkeley.edu/beta/result.php?resultid=4972419)) - 0.009777995 credits per CPU second
1565898 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1565898) (Pending) (Task details (http://setiweb.ssl.berkeley.edu/beta/result.php?resultid=4972760)) - 0.009551190 credits per CPU second
1628888 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1628888) (Validated) - 0.009486180 credits per CPU second
1565146 (http://setiweb.ssl.berkeley.edu/beta/workunit.php?wuid=1565146) (Validated) - 0.009490379 credits per CPU second

For comparison:
Shorty Multibeam (16.84 credits):  0.007037046 credits per CPU second
Average Multibeam (44.12 credits):  0.008375063 credits per CPU second
Long Multibeam (63.86 credits):  0.010619245 credits per CPU second
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: [B^S] zioriga on 02 Dec 2008, 03:25:20 pm
Leaps-from-Shadows excuse me, but if you use Vista x64 why don't you use BOINC 6.2.19 64b client ??
Title: Re: Report on new optimized Astropulse apps for Windows
Post by: Leaps-from-Shadows on 02 Dec 2008, 05:44:09 pm
I tend to use 32-bit apps for everything.  I figure that the 32-bit apps have had more time to mature and have had more thorough testing to eliminate problems and bugs.  Eventually, I'll switch to 64-bit apps ... but for now I just have 64-bit Vista to get full use out of my 4GB of RAM.