Seti@Home optimized science apps and information
Optimized Seti@Home apps => Windows => Topic started by: Raistmer on 10 Jul 2009, 01:54:28 pm
-
Typical situation:
BOINC runs 4 SETI tasks w/o any MW tasks.
ATI GPU stays idle.
Another situation:
BOINC runs more than 10 MW tasks (from them only 3 actually do progress thanks to well-designed opt app by Gipsel, so it's not so bad), but only 3 SETI tasks. That is, one core stays idle (MW tasks consume almost no CPU, less tank CUDA MB).
Please, help to set project shares and other options to balance load.
Ideally I wanna see 4 SETI tasks + 3 or more (actuall no matter how more) MW tasks running simultaneously.
Maybe avg and max CPU in app_info should be changed too?
ADDON: additional info:
current project shares (only 2 projects active):
SETI - 10000
MW - 15000
IMHO bigger MW share should stimulate BOINC to run MW tasks.
But what I observe:
2 MW tasks started then were put in waiting for run state.
Short debt for MW (observing it via BOINCView) ~5000 s now and continues to increase.
Debt for SETI ~ -5000 s correspondingly....
Why BOINC doesn't allow MW tasks to run in this conditions?
-
Even more info:
I suspended SETI for 1-2 seconds then resumed it.
BOINC started lot of MW tasks >10 then, after SETI resuming it starts 3 SETI tasks.
After few MW tasts completed it started 4th SETI task.
Short debts were resetted to zero. Now MW short debt increasing again and number of running MW tasks decreasing (that is BOINC refuses to start additional MW tasks when some of them completed).
What is wrong???
ADDON:
current app_info for MW:
<flops>1.0e11</flops>
<avg_ncpus>0.05</avg_ncpus>
<max_ncpus>0.2</max_ncpus>
<cmdline></cmdline>
-
How are these MW tasks supposed to work?
I know they do the actual work on a GPU that BOINC knows nothing about. So they must be scheduled as CPU tasks, right? How do you get seven tasks, all believed by BOINC to be CPU tasks, to run at the same time? Non-computationally-intensive? 175% CPU utilisation?
What (and this is a serious question) does <cpu_sched_debug> put out?
-
yes, MW use avg_cpu setting in app_info to claim it almost no use of CPU. That allows BOINC to run multiple instances of MW + some more CPU intensive tasks like SETI.
MW taks are processed completely on GPU indeed.
Will enable option you mentioned.
-
Right, Ascribe a plan class with avg & max cpus of ~0.95 to SaH CPU tasks (AKv8 I presume). .... Initially should get 5 then drop back :o
-
10/07/2009 22:33:13 [cpu_sched_debug] Request enforce CPU schedule: schedule_cpus
10/07/2009 22:33:13 [cpu_sched_debug] enforce_schedule(): start
10/07/2009 22:33:13 [cpu_sched_debug] preliminary job list:
10/07/2009 22:33:13 Milkyway@home [cpu_sched_debug] 0: ps_sgr_210F5_3s_hiw_15791565_1247247343_0
10/07/2009 22:33:13 SETI@home [cpu_sched_debug] 1: 13se08ab.29372.13569.8.8.136_1
10/07/2009 22:33:13 SETI@home [cpu_sched_debug] 2: 22dc08ac.28597.14904.7.8.164_1
10/07/2009 22:33:13 Milkyway@home [cpu_sched_debug] 3: ps_sgr_222F5_3s_hiw_15793323_1247247510_0
10/07/2009 22:33:13 SETI@home [cpu_sched_debug] 4: 29au08aa.21644.18477.7.8.4_1
10/07/2009 22:33:13 Milkyway@home [cpu_sched_debug] 5: ps_sgr_214F5_3s_hiw_15793750_1247247570_0
10/07/2009 22:33:13 SETI@home [cpu_sched_debug] 6: 13se08ab.29372.17250.8.8.237_1
10/07/2009 22:33:13 [cpu_sched_debug] final job list:
10/07/2009 22:33:13 Milkyway@home [cpu_sched_debug] 0: ps_sgr_210F5_3s_hiw_15791565_1247247343_0
10/07/2009 22:33:13 Milkyway@home [cpu_sched_debug] 1: ps_sgr_230F5_3s_hiw_15810913_1247249472_0
10/07/2009 22:33:13 SETI@home [cpu_sched_debug] 2: 13se08ab.29372.13569.8.8.136_1
10/07/2009 22:33:13 SETI@home [cpu_sched_debug] 3: 22dc08ac.28597.14904.7.8.164_1
10/07/2009 22:33:13 Milkyway@home [cpu_sched_debug] 4: ps_sgr_222F5_3s_hiw_15793323_1247247510_0
10/07/2009 22:33:13 SETI@home [cpu_sched_debug] 5: 29au08aa.21644.18477.7.8.4_1
10/07/2009 22:33:13 Milkyway@home [cpu_sched_debug] 6: ps_sgr_214F5_3s_hiw_15793750_1247247570_0
10/07/2009 22:33:13 SETI@home [cpu_sched_debug] 7: 13se08ab.29372.17250.8.8.237_1
10/07/2009 22:33:13 Milkyway@home [cpu_sched_debug] scheduling ps_sgr_210F5_3s_hiw_15791565_1247247343_0
10/07/2009 22:33:13 Milkyway@home [cpu_sched_debug] scheduling ps_sgr_230F5_3s_hiw_15810913_1247249472_0
10/07/2009 22:33:13 SETI@home [cpu_sched_debug] scheduling 13se08ab.29372.13569.8.8.136_1
10/07/2009 22:33:13 SETI@home [cpu_sched_debug] scheduling 22dc08ac.28597.14904.7.8.164_1
10/07/2009 22:33:13 Milkyway@home [cpu_sched_debug] scheduling ps_sgr_222F5_3s_hiw_15793323_1247247510_0
10/07/2009 22:33:13 SETI@home [cpu_sched_debug] scheduling 29au08aa.21644.18477.7.8.4_1
10/07/2009 22:33:13 Milkyway@home [cpu_sched_debug] scheduling ps_sgr_214F5_3s_hiw_15793750_1247247570_0
10/07/2009 22:33:13 SETI@home [cpu_sched_debug] scheduling 13se08ab.29372.17250.8.8.237_1
10/07/2009 22:33:13 SETI@home [cpu_sched_debug] 13se08ab.29372.13569.8.8.136_1 sched state 2 next 2 task state 1
10/07/2009 22:33:13 SETI@home [cpu_sched_debug] 22dc08ac.28597.14904.7.8.164_1 sched state 2 next 2 task state 1
10/07/2009 22:33:13 SETI@home [cpu_sched_debug] 29au08aa.21644.18477.7.8.4_1 sched state 2 next 2 task state 1
10/07/2009 22:33:13 Milkyway@home [cpu_sched_debug] ps_sgr_230F5_3s_hiw_15787913_1247246910_0 sched state 1 next 1 task state 9
10/07/2009 22:33:13 Milkyway@home [cpu_sched_debug] ps_sgr_210F5_2s_hiw_15788517_1247247017_0 sched state 1 next 1 task state 9
10/07/2009 22:33:13 Milkyway@home [cpu_sched_debug] ps_sgr_210F5_2s_hiw_15788518_1247247017_0 sched state 1 next 1 task state 9
10/07/2009 22:33:13 Milkyway@home [cpu_sched_debug] ps_sgr_210F5_3s_hiw_15791565_1247247343_0 sched state 2 next 2 task state 1
10/07/2009 22:33:13 Milkyway@home [cpu_sched_debug] ps_sgr_222F5_3s_hiw_15793323_1247247510_0 sched state 2 next 2 task state 1
10/07/2009 22:33:13 Milkyway@home [cpu_sched_debug] ps_sgr_214F5_3s_hiw_15793750_1247247570_0 sched state 2 next 2 task state 1
10/07/2009 22:33:13 Milkyway@home [cpu_sched_debug] ps_sgr_230F5_3s_hiw_15810913_1247249472_0 sched state 2 next 2 task state 1
10/07/2009 22:33:13 SETI@home [cpu_sched_debug] 13se08ab.29372.17250.8.8.237_1 sched state 2 next 2 task state 1
10/07/2009 22:33:13 [cpu_sched_debug] enforce_schedule: end
-
Right, Ascribe a plan class with avg & max cpus of ~0.95 to SaH CPU tasks (AKv8 I presume). .... Initially should get 5 then drop back :o
Ok, will add.
EDIT:
After restart BOINC started 4 SETI + 4 MW tasks. It just OK. Will see what will be next...
ADDON: current MW short debt approaching to 3000s....
-
EDIT:
After restart BOINC started 4 SETI + 4 MW tasks. It just OK. Will see what will be next...
It's a packing algorithm instead of graph colouring (otherwise known as Richard's Back of the envelope calculation). Debt will accumulate according to the amount specified in Max, when task isn't running, even if it is running, but scheduler thinsk it has no room :o
-
Now, after few MW tasks completed still 4+4. Will see what will be after one of SETI tasks completed....
Debt is ~3500.
And what max value is ?
-
32 bit unsigned int ?
-
:) hope less :)
Oops, "communicating with BOINC"... and 99% of idle CPU. boinc.exe crashed it seems...
I use patched version now .
-
:) hope less :)
Oops, "communicating with BOINC"... and 99% of idle CPU. boinc.exe crashed it seems...
I use patched version now (with project- wide network activity suspend).
LoL. It doesn't like being watched.
-
It seems so :)
Well, no crash dump. It seems it was not a boinc.exe crash, just all apps exited with "no heartbeat" state.
-
...just all apps exited with "no heartbeat" state.
Because you deliberately terminated 8 apps simultaneously?... no... didn't really think so.
-
No, I didn't terminated them.
There was delay in manager communication with daemon. It seems daemon was hung for more than 30 seconds - all apps exited with no heartbeat message. I see that message in SETI's stderr (MW tasks already gone of course).
Maybe just because BOINC's stderr with debug enabled grows too big (more 2MB now)... don't know. Will see if it will repeat again.
Will leave it for awhile and check later. Now still 4+4 that can be considered as perfect state :)
-
No, I didn't terminated them.
There was delay in manager communication with daemon. It seems daemon was hung for more than 30 seconds - all apps exited with no heartbeat message. I see that message in SETI's stderr (MW tasks already gone of course).
I know I was being facetious (Another term for 'Australian').
-
Ah, I see :)
-
Now 5 MW + 4 SETI - that's OK too.
STD ~13500s now and still increasing.
-
...
STD ~13500s now and still increasing.
...
OK, now make max & avg equal, then increase them gradually untill debt accumualtion stops ;)
-
OK. Been out for a meal - and to learn about funeral arrangements for next week. Not nice.
But I see you've made progress while I've been away. Can we agree that v.36 is not broken like an old banger, but a Formula 1 racer that needs a skilled driver to exploit - otherwise it stalls on the starting line?
If so - and this is the difficult bit - how do we get it to manage all that tricksy CPU / GPU / GPU pretending to be a part of a CPU / multithreaded CPU (active at AQUA) stuff, add GSTD resource-share scheduling, and still make it driveable by both a teenager and their granny?
-
....
If so - and this is the difficult bit - how do we get it to manage all that tricksy CPU / GPU / GPU pretending to be a part of a CPU / multithreaded CPU (active at AQUA) stuff, add GSTD resource-share scheduling, and still make it driveable by both a teenager and their granny?
I'm working on the premise that we can fudge the input figures, such that it stabilises and does what it's told. It seems to be somewhat similar in intelligence to my goldfish that only ever floated upside down, and a little below my dog 'Bozley', which is relatively intelligent compared to the goldfish (He responds to the word 'Biscuits' favourably, anyway.).
-
....
If so - and this is the difficult bit - how do we get it to manage all that tricksy CPU / GPU / GPU pretending to be a part of a CPU / multithreaded CPU (active at AQUA) stuff, add GSTD resource-share scheduling, and still make it driveable by both a teenager and their granny?
I'm working on the premise that we can fudge the input figures, such that it stabilises and does what it's told. It seems to be somewhat similar in intelligence to my goldfish that only ever floated upside down, and a little below my dog 'Bozley', which is relatively intelligent compared to the goldfish (He responds to the word 'Biscuits' favourably, anyway.).
But does he respond to the word 'biscuits' in all 182,898 regional accents and dialects spoken by active SETI users - including those from cultures where the concept of 'biscuit' is entirely missing? ???
-
But does he respond to the word 'biscuits' in all 182,898 regional accents and dialects spoken by active SETI users - including those from cultures where the concept of 'biscuit' is entirely missing? ???
No, but he ignores the word 'out', yet dutifully obeys 'oot', despite being born in the middle of Western Australia, and AFAIK never has met a scotchman, so maybe he's more of a linguist than I thoiught.
-
:)
Actually I hope that STD accumulation will be solved by running more copies of MW .Time to time BOINC starts 5 MW tasks instead of 4.
As long as it doesn't decrease number of SETI tasks it's OK.
Will leave it over night and see at morning what happened.
My MW RAC curve declined from almost vertical line, but current RAC is only about 1/2 of theoretical limit.
I will post screenshot later - all CPU + 9600GSO achievements are almost on bottom line of graph %) Current RAC ~ 44000 :o :o :o
-
Hm... 5 SETI + 0 MW... it's not good....
MW has bigger project share so why BOINC does this...?
How STD calculated? Basing on elapsed time x number of running apps or how?
60% of MW share and only 40% os SETI but BOINC runs 5 SETI tasks and no MW ones...
-
My STD is 6.26 with my 850-SETI/150-MW settings.
-
My STD is 6.26 with my 850-SETI/150-MW settings.
That is, you did just reverse - decreased MW share? Interesting. Do you use ATI GPU app too (and only GPU app for MW) ?
-
I have a 4830 in my Quad, it averages about 33000 RAC on the GPU only. My other 2 computers do not have compatible GPU's.
I just checked my STD and it is up to 64.93 currently.
I also use a 0.1 and 3 for network settings.
-
Well, my current config is Q9450 + HD4870. ATI GPU RAC >50K for now and still increasing.
avg_cpu=max_cpu for both projects and:
MW avg_cpu==0.01, SETI avg_cpu==0.99 (as suggested on MW forums).
Project shares are 10k for SETI and 12k for MW.
Currently BOINC runs 4 SETI tasks (it's completely OK) and 22 MW tasks (it's acceptable, but can be improved IMHO).
From these 22 tasks only 3 actually running (by opt app design). So, BOINC's time to complete estimations are completely screwed.
And what is absolutely not OK : 17 of 22 MW tasks are running in high priority mode. This will prevent additional work fetch from MW AFAIK (will it affect on work fetch for SETI ? ).
Another relevant fact about MW project - it doesn't allow to have more than 6*number of CPUs (24 in my case) tasks in cache.
Ideal case I want to reach somehow:
~24 MW tasks in cache always + few days of work for SETI.
4 SETI tasks running always + 2 to ~4-5 MW tasks (running always). Too many waiting MW tasks still increase system overhead IMHO.
And no high priority mode at all of course, for both projects.
I think current BOINC's estimations for all parameters are pretty distorted by my attempts to manually balance workload.
What should I do from current state of things?
ADDON: typical deadline for MW tasks ~4 days.
My current network settings:
0+6, work fetch enabled only for MW, SETI setted to "no new tasks" few days already.
-
I would start playing with cc_config and ncpus.
4+2
ncpus -> 6
seti avg-max-cpus -> 1.51
MW avg-max-cpus -> 0.5 (?), 1(?)
4+4
ncpus ->8
seti avg-max-cpus -> 2.01
MW avg-max-cpus ->0.5 (?), 1 (?)
-
Actually I don't want to touch ncpus.
What I need is to run both projects simultaneously.
changing ncpus value will allow to run more than 4 tasks at once, yes, but it could be 8 SETI only tasks or complete bunch of 24 MW tasks... ncpus option can't govern what project will use all these virtual CPUs (this was main problem of running my first "teamed" apps w/o cuda plan...).
Currently, I reached stable execution for both projects, but only because BOINC can't start more than 24 MW tasks at once (no more tasks in queue).
Surely it's suboptimal state.
-
I have a 4830 in my Quad, it averages about 33000 RAC on the GPU only. My other 2 computers do not have compatible GPU's.
I just checked my STD and it is up to 64.93 currently.
I also use a 0.1 and 3 for network settings.
But do you have sometimes situation when no MW tasks active (running) at all ?
With MW share lowe than SETI one i'm afraid to get GPU idle. DO you have idle GPU time to time with your config?
-
changing ncpus value will allow to run more than 4 tasks at once, yes, but it could be 8 SETI only tasks or complete bunch of 24 MW tasks... ncpus option can't govern what project will use all these virtual CPUs (this was main problem of running my first "teamed" apps w/o cuda plan...).
Using for example ncpus=6 with avg-max-cpus=1.50 or 1.51 for seti you force boinc to run only 4 seti tasks (4*1.50=6, 4*1.51=6.04) while really using 4 cores of its supposedly 6 available. Experimenting with MW avg-max-cpus I think you can accomplish the desired effect.
-
Ok, will try , looks reasonable... Thanks!
-
changing ncpus value will allow to run more than 4 tasks at once, yes, but it could be 8 SETI only tasks or complete bunch of 24 MW tasks... ncpus option can't govern what project will use all these virtual CPUs (this was main problem of running my first "teamed" apps w/o cuda plan...).
Using for example ncpus=6 with avg-max-cpus=1.50 or 1.51 for seti you force boinc to run only 4 seti tasks (4*1.50=6, 4*1.51=6.04) while really using 4 cores of its supposedly 6 available. Experimenting with MW avg-max-cpus I think you can accomplish the desired effect.
Hm... after more thinking:
What will prevent MW from taking whole CPUs (beside that limit of 24 tasks available)?
-
Hm... after more thinking:
What will prevent MW from taking whole CPUs (beside that limit of 24 tasks available)?
What do you mean?
-
Hm... after more thinking:
What will prevent MW from taking whole CPUs (beside that limit of 24 tasks available)?
What do you mean?
I mean if MW task takes less than 1 CPU what will prevent BOINC not to run SETI tasks at all (or run less than 4 of them) and fill all remaining CPU capacity with MW tasks?
For now it wants run almost as many MW tasks it could find zero to 2 MW tasks are "ready to run", all others are running, most part in "high priority state"
What I currently need is to decrease number of MW taks running not changing number of SETI tasks running at the same time.
So, if more CPUs will be available (from BOINC point of view) it will try to run even more MW taks.
If I increas avg_cpu for MW task BOINC could decide to run less SETI tasks but still allow as many MW tasks as fin in increased ncpus to run.
I.e., I don't see how increased ncpus number can help in reducing number of running MW tasks (not sacrificing SETI)....
-
What I currently need is to decrease number of MW taks running not changing number of SETI tasks running at the same time.
So, if more CPUs will be available (from BOINC point of view) it will try to run even more MW taks.
If I increase avg_cpu for MW task BOINC could decide to run less SETI tasks but still allow as many MW tasks as fin in increased ncpus to run.
Well, resource share should also play a role here.
I.e., I don't see how increased ncpus number can help in reducing number of running MW tasks (not sacrificing SETI)....
I'm not saying that this will solve your problems, just experiment with it and see if it does. My own experience with boinc tells me that boinc likes values of avg-max-cpus >=1.0000 more than <1.0000.
-
I see,will try then...
-
I have a 4830 in my Quad, it averages about 33000 RAC on the GPU only. My other 2 computers do not have compatible GPU's.
I just checked my STD and it is up to 64.93 currently.
I also use a 0.1 and 3 for network settings.
But do you have sometimes situation when no MW tasks active (running) at all ?
With MW share lowe than SETI one i'm afraid to get GPU idle. DO you have idle GPU time to time with your config?
The only time I have an idle GPU is when I am uploading work and getting new.
I am going to experiment with 700/300
-
The only time I have an idle GPU is when I am uploading work and getting new.
I am going to experiment with 700/300
Well, sometimes my config starts to work in that mode too.
I.e. reciving work from server only after all MW work is uploaded and reporting. It surely decrease effectiveness.
Yesterday, after extended brook AP benchmarking NW server started refuse fetch requests with messages like "BOINC running 90% of time, computationa enabled 100% of that, not completed in time." How it could not be completed in time if MW has bigger share, SETI has no debt and each MW task took (even from BOINC's own point of view) less than 1,5h - it's beyond my understanding :-\
I switched to 6 ncpus and 1,51+0,01 avg_cpus - MW tasks were not started at all. Then I switched to 1,5+0.01 - situation resemble that was before: 4 SETI tasks running + ~20 MW tasks, some of them in high priority mode.
At evening I changed project sheres to 10000(SETI)+9500(MW) in hope that will reduce MW debt and number of running MW tasks.
STD of MW now (after whole night crunching) in ~86k sec, almost all available MW tasks in running state.
Apparently I should decrease MW share even more, but I don't quite understand why debt still increases.
How STD is calculated? I have project with lower share, it runs bigger number of tasks (i.e. consumes more elapsed time) than project with higher share, but still BOINC things that lower share project needs to work more. How is this possible? Or SDT calculations based not on elapsed time bu CPU time instead (in this case avg_cpu == 0.01 setting will tell BOINC that almost no work done by MW, correct? )
-
I switched to 6 ncpus and 1,51+0,01 avg_cpus - MW tasks were not started at all. Then I switched to 1,5+0.01 - situation resemble that was before: 4 SETI tasks running + ~20 MW tasks, some of them in high priority mode.
Well if it starts 20 MW tasks with 0.01 avg_cpus then increase avg_cpus.
You might get a better feeling of all the dynamics involved by leveling things of. Zero your debt and put ncpus=8 and avg-max-cpus=1.0000 to both seti and MW. Playing with resource share you should be able to get the desired 4+4 effect. Then try to replicate that with ncpus=4.