Simplified stats gathering

[Simon]

Hiya,

I'm currently working on making a simple web entry form where any registered user can easily input their test results. That should make stats generation way easier, as I don't have to do it manually anymore.

This web form will first store submissions in a temporary place, and when I have verified they're legit (i.e. no intentional fiddling with times), they will get added to the results collection.

I can draw automated graphs from that info, too.

This will make a few things necessary - a test script that produces a fixed format output file, for one. Kna has sent me a much improved benchmark script which has given me a few ideas how to include a simple anti-tamper mechanism as well as simplified the calculation of time taken (it now does that for you, thanks Kna!).

I will be posting a new test package containing his excellent script with a few modifications (filenames, mmx removed).

The web form/automated stats processing might take a bit to get to work reliably, but should be worth it in the long run.

Regards,
Simon.

[pepperammi]

Maybe you could make a script/programme that took the numbers straight out of the testData files? Not sure how difficult that would be but that way would be more difficult to tamper and you'd have the evidence in the same files? Could make your life a lot easyer if its easily possible because you wouldn't have to go through manually verifying peoples numbers. You could just run that programme and its done in half a sec.

When I think about it it starts to sound more and more complicated. Maybe not a good idea. No idea how to write a programme to do the math on time values.

[EDIT]
Also if was setup to work on the web data input form your talking about then people could upload a copy of the testData folder and it does it all automatic?

[Kna]

Maybe you could make a script/programme that took the numbers straight out of the testData files?
...

Hi Pepperammi

I get in touch with Simon to produce better scripts, based on his first ones.
I can't promise quick and fantastic results, I'm not a cmd-guru nor a real programmer with plenty of spare time, but i'll try my best 

Scripts can have multiple purposes, benching newly compiled clients by the test team, and benching several computers to determine the best perf's client for each one.
Also, automated and simplified results reporting for personnal use or for building global stats is to be considered.

All this have an impact in future scripts, and I admit  that exceeds already my starting idea, simply to make better scripts than the original one, provided with the Simon's sources package 

Regards, Kna

[Raistmer]

Hello
Is there any table with results aquired with benchmarks (KWSN Knabench 1.43 Benchmark for example) on different platforms to compare ?

[Gecko_R7]

Hello
Is there any table with results aquired with benchmarks (KWSN Knabench 1.43 Benchmark for example) on different platforms to compare ?

There really isn't anything like this consolidated w/ the recent aps.
Early-on, a few charts were created w/ some of the first opt aps on different platforms for S-Enhanced @ 1 year ago.
Since then, speed of later aps has improved dramatically and the hardware mix has advanced w/ more C2D and xeon rigs.  So....the earlier charts are rather worthless to your question.

However, there have been a couple of recent discussions re: how to measure FPU and INT performance accross platforms for comparison.  As Crunch3r points-out, as long as bitness (x32 or x64) is consistant between aps and hardware, then S@H and ABC could provide a decent set of benches using a uniform WU for each project that different platforms could be measured and logged.
This would not only give a better relative comparison of CPUs for S@H, but would also give better insight into relative processor performance for those that run projects that have both FPU and INT processing bias.

In some cases, many older p4 and AMD processors that are rather melba-toast today on S@H (compared to C2D), may perform comparatively much better in 64-bit mode w/ projects like ABC that focus on INT processing.  For those that run multiple projects, this could be helpful info.

Could be a fun side-project and not too hard other than the maintance and updating of the central database.

[Raistmer]

So such database still doesnt' exist, right?
Thanks for answer.
BTW, both benchmarks I found on this site measure elapsed system time, not CPU time. These can be pretty different in multitask OS (with big network activity on PC being tested or if another background processes that consume CPU are running).

[Simon]

Nope, not for public consumption yet, anyway.

I've been working (when I find time) on a multi-platform benchmark suite (should work on Windows and Linux, and probably other platforms as well). The Windows version will even have a GUI, and it'll be able to transparently grab and upload results for comparison. Only catch: for "trusted" results, you'll have to be a registered user here - everyone else will be able to use it, too, but their results will be in the "unverified" department (but still viewable).

Every registered user will find a new link on the left when it's ready/released. This link will lead to a page that issues each user their own unique ID, so results can be associated with people who submitted them.

Since benchmarks are only useful for comparison if run as intended (with the WUs we put in there), this should make it easier to single out bogus/unwanted results, if any.

Also, you'll be able to graph your results vs. others according to some parameters - CPU/MHz/OS/...

Obviously, this will include runs of the - current at the time - stock executable from Berkeley, which may even help the project itself gather some performance data they did not have until now.

Regards,
Simon.

[Vyper]

Great Simon..

That is really cool indeed.. If used together with various AR Wu's as before we will soon be able to see where the strong and weak part of every different compiled app lies..

Very nice and keep up the good work.. This will be usefull in the future truly..

Kind Regards Vyper