+- +-
Say hello if visiting :) by Gecko
11 Jan 2023, 07:43:05 pm

Seti is down again by Mike
09 Aug 2017, 10:02:44 am

Some considerations regarding OpenCL MultiBeam app tuning from algorithm view by Raistmer
11 Dec 2016, 06:30:56 am

Loading APU to the limit: performance considerations by Mike
05 Nov 2016, 06:49:26 am

Better sleep on Windows - new round by Raistmer
26 Aug 2016, 02:02:31 pm

Author Topic: optimized sources  (Read 615548 times)

Offline _heinz

  • Volunteer Developer
  • Knight who says 'Ni!'
  • *****
  • Posts: 2117
Re: optimized sources
« Reply #675 on: 05 Jan 2011, 06:13:35 pm »
Today 5th of January "Intel C++ Studio XE/SSR" Premier support expires,
renewal for one year costs 467,10 EUR zzgl. MwSt.
my other two "Intel Parallel Studio XE WIN/SSR/ESD/SU" Premier support expired 31.12.2010
renewal  each--> 584,10 EUR zzgl. MwSt.

summary really a lot of money, price of a new product itself....phhh... :'(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
New product:
Intel Parallel Studio XE WIN/COM/ESD/SU
1.439,10 EUR

must think about what todo
heinz

Offline Jason G

  • Construction Fraggle
  • Knight who says 'Ni!'
  • *****
  • Posts: 8980
Re: optimized sources
« Reply #676 on: 05 Jan 2011, 06:16:28 pm »
must think about what todo
heinz

Your choice Heinz, but I stopped renewing my premier support quite a while ago, since the prices went up and support received with non-functioning advanced features in earlier versions was not satisfactory.

Don't worry too much, as careful hand coding of AVX would end up better anyway  ;)

Offline _heinz

  • Volunteer Developer
  • Knight who says 'Ni!'
  • *****
  • Posts: 2117
Re: optimized sources
« Reply #677 on: 05 Jan 2011, 06:39:26 pm »
At the moment I'm activating my old P4 2.6GHz as a developer machine, install all necessary software
VS2008Prof
ICC / Parallel Studio XE
...
fighting with not readable DVD again on this machine....

must connect my extern DVD writer from LG, it does it.
After installing VS2008 Service Pack 1 a lot of update followed....
« Last Edit: 05 Jan 2011, 09:34:51 pm by _heinz »

Offline _heinz

  • Volunteer Developer
  • Knight who says 'Ni!'
  • *****
  • Posts: 2117
Re: optimized sources
« Reply #678 on: 06 Jan 2011, 03:52:59 pm »
Today the machine is ready to use.
Some first tests show it works. MSC and ICC compiles.
To proof full functionality some more complex project must be compiled.

heinz


Offline _heinz

  • Volunteer Developer
  • Knight who says 'Ni!'
  • *****
  • Posts: 2117
Re: optimized sources
« Reply #679 on: 06 Jan 2011, 08:28:48 pm »
To proof full functionality some more complex project must be compiled.
Used Release_vc90.sln from CUDA3.2 SDK (it contains 93 Projects)

Using MSC  Compiler
========== Alles neu erstellen: 92 erfolgreich, Fehler bei 0, 1 übersprungen ==========

hint: 1 übersprungen -->cudaEncode was not marked in Konfigurations-Manager

we compile now cudaEncode
~~~~~~~~~~~~~~~~~~
cudaEncode - 0 Fehler, 2 Warnung(en)
========== Alles neu erstellen: 1 erfolgreich, Fehler bei 0, 0 übersprungen ==========

summary we compiled 93 cuda32-projects now. That's correct.

heinz
modify: recompiled CUDA32
========== Alles neu erstellen: 93 erfolgreich, Fehler bei 0, 0 übersprungen ==========
« Last Edit: 07 Jan 2011, 06:10:02 pm by _heinz »

Offline _heinz

  • Volunteer Developer
  • Knight who says 'Ni!'
  • *****
  • Posts: 2117
Re: optimized sources
« Reply #680 on: 07 Jan 2011, 08:20:41 pm »
Intel Compiler Xe2011

step1 converting project solution
Rebuild these 93 projects to ensure all intermediate files are rebuilt with the new compiler.

Detailed update log was saved at "file://C:\CUDA32SDK\C\src\IcUpdateLog.htm"

step2 compiling with Intel compiler
...
...
Compiling with CUDA Build Rule... (Microsoft VC++ Environment)
"C:\CUDA32\v3.2\\bin\nvcc.exe"  -G0  -gencode=arch=compute_10,code=\"sm_10,compute_10\" -gencode=arch=compute_20,code=\"sm_20,compute_20\"  --machine 32 -ccbin "C:\Programme\Microsoft Visual Studio 9.0\VC\bin" -D_NEXUS_DEBUG -g   -Xcompiler "/EHsc /W3 /nologo /O2 /Zi  /MT  " -I"C:\CUDA32\v3.2\/include" -I"./" -I"../../common/inc" -I"../../../shared/inc" -I"../../shared/inc/" -I"C:\CUDA32\v3.2\\include" -maxrregcount=[Value]  --compile -o "Release/bandwidthTest.cu.obj" bandwidthTest.cu
nvcc fatal   : '[Value]': expected a number
Project bandwidthTest : error: A tool returned an error code from "Compiling with CUDA Build Rule..."
Build log was saved at "file://C:\CUDA32SDK\C\src\bandwidthTest\Release\BuildLog.htm"
bandwidthTest - 1 error(s), 0 warning(s), 0 remark(s)
------ Neues Erstellen gestartet: Projekt: asyncAPI, Konfiguration: Release Win32 ------
Deleting intermediate files and output files for project 'asyncAPI', configuration 'Release|Win32'.
Compiling with CUDA Build Rule... (Microsoft VC++ Environment)
"C:\CUDA32\v3.2\\bin\nvcc.exe"  -G0  -gencode=arch=compute_10,code=\"sm_10,compute_10\" -gencode=arch=compute_20,code=\"sm_20,compute_20\"  --machine 32 -ccbin "C:\Programme\Microsoft Visual Studio 9.0\VC\bin" -D_NEXUS_DEBUG -g   -Xcompiler "/EHsc /W3 /nologo /O2 /Zi  /MT  " -I"C:\CUDA32\v3.2\/include" -I"./" -I"../../common/inc" -I"../../../shared/inc" -I"C:\CUDA32\v3.2\\include" -maxrregcount=[Value]  --compile -o "Release/asyncAPI.cu.obj" asyncAPI.cu
nvcc fatal   : '[Value]': expected a number
Project asyncAPI : error: A tool returned an error code from "Compiling with CUDA Build Rule..."
Build log was saved at "file://C:\CUDA32SDK\C\src\asyncAPI\Release\BuildLog.htm"
asyncAPI - 1 error(s), 0 warning(s), 0 remark(s)
------ Neues Erstellen gestartet: Projekt: alignedTypes, Konfiguration: Release Win32 ------
Deleting intermediate files and output files for project 'alignedTypes', configuration 'Release|Win32'.
Compiling with CUDA Build Rule... (Microsoft VC++ Environment)
"C:\CUDA32\v3.2\\bin\nvcc.exe"  -G0  -gencode=arch=compute_10,code=\"sm_10,compute_10\" -gencode=arch=compute_20,code=\"sm_20,compute_20\"  --machine 32 -ccbin "C:\Programme\Microsoft Visual Studio 9.0\VC\bin" -D_NEXUS_DEBUG -g   -Xcompiler "/EHsc /W3 /nologo /O2 /Zi  /MT  " -I"./" -I"../../common/inc" -I"../../../shared/inc" -I"C:\CUDA32\v3.2\\include" -maxrregcount=[Value]  --compile -o "Release/alignedTypes.cu.obj" alignedTypes.cu
nvcc fatal   : '[Value]': expected a number
Project alignedTypes : error: A tool returned an error code from "Compiling with CUDA Build Rule..."
Build log was saved at "file://C:\CUDA32SDK\C\src\alignedTypes\Release\BuildLog.htm"
alignedTypes - 1 error(s), 0 warning(s), 0 remark(s)
========== Alles neu erstellen: 10 erfolgreich, Fehler bei 83, 0 übersprungen ==========

hey, thats the rules file error...maxrregcount get no value...
phh, I installed latest compiler
Compiling with Intel(R) C++ Compiler XE 12.0.1.127 [IA-32]... (Intel C++ Environment)

looking up:
Intel(R) C++ Compiler XE on IA-32, version 12.0.1 Package ID: w_ccompxe_2011.1.127
got it with latest download from 04.01.2011
It's a little bit frustrating... that this known error is  in package w_ccompxe_2011.1.127 till today.  :'(

heinz

Offline _heinz

  • Volunteer Developer
  • Knight who says 'Ni!'
  • *****
  • Posts: 2117
Re: optimized sources
« Reply #681 on: 08 Jan 2011, 02:06:11 pm »
Hi Jason,

CUDA3.2 & INTEL's Compiler
Package ID: w_ccompxe_2011.1.127
I used the patch CompilerIDEPluginUpdate.zip to avoid the rules-file error sucessful.
The rules-file error is gone.  ;)

Projekt: MonteCarloMultiGPU and Projekt: MonteCarlo does not sucessful compile with Intel Compiler
ptxas hung up  :'(
========== Alles neu erstellen: 91 erfolgreich, Fehler bei 2, 0 übersprungen ==========

So far, we must wait for the next update.

heinz

Offline _heinz

  • Volunteer Developer
  • Knight who says 'Ni!'
  • *****
  • Posts: 2117
Re: optimized sources
« Reply #682 on: 11 Jan 2011, 07:33:58 am »
Sandy Bridge is available in Germany

Intel® Core™ i5-2500K (Boxed, FC-LGA4, "Sandy Bridge") € 219,90*
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Der Intel® Core™ i5-2500K Prozessor basiert auf der neuen 32nm-Sandy-Bridge-Architektur und ist eine native Quad-Core-CPU mit integriertem Grafikkern (Intel® HD 3000). Die Weiterentwicklung der bisherigen Intel® Core™ Prozessoren bringt als wichtigste Neuerung mit AVX eine überarbeitete und auf 256-Bit erweiterte Version der SSE-Befehle. Der integrierte Speichercontroller untertstützt jetzt offiziell DDR3-Speicher bis 1600 MHz und der verbesserte "Dynamic Turbomode" kann den Takt bei niedrigen Systemtemperaturen (wie nach dem Rechnerstart) kurzzeitig um bis zu 30% über den Maximalwert anheben.
Die integrierte GPU läuft mit 850 Mhz (max. 1150 mit Turbo) und teilt sich den gemeinsamen, 6 MB großen L3-Cache mit der  CPU.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Bezeichnung: Intel® Core™ i5-2500K
Anzahl Prozessorkerne: 4
Taktfrequenz: 3300 MHz
Cache Level 1: 4x 64 kB
Level 2: 4x 256 kB
Level 3: 6144 kB
Befehlssätze: SSE 4.x, AVX, EIST, Intel 64, XD bit, Intel VT-x, Smart Cache, Clear Video, Turbo Boost, AES-NI
Speicher-Controller Speicherstandards: DDR3-1066, DDR3-1333, DDR3-1600
Speicherkanäle: 2
Prozessorkern: Sandy Bridge, 32 nm
Bemerkung Turbo-Modus: CPU bis max. 3,7 GHz; GPU-Takt: 850 / 1.100 MHz (Standard / Turbo)
max. Leistungsaufnahme: 95 Watt
Bauform: FC-LGA4
Sockel: 1155
Lüfter: Befestigung für ATX-Mainboards
Anschlüsse PWM-Lüfteranschluss
Besonderheiten: mit aktivem CPU-Kühler

Weitere Infos: Die K-Modelle der Intel® Core™ Prozessoren sind für Übertakter und Technik-Enthusiasten gedacht und bieten einen frei wählbaren Multiplikator
~~~~~~~

Intel® Core™ i7-2600K (Boxed, FC-LGA4, "Sandy Bridge") € 339,-*
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Der Intel® Core™ i7-2600K Prozessor basiert auf der neuen 32nm-Sandy-Bridge-Architektur und ist eine native Quad-Core-CPU mit integriertem Grafikkern (Intel® HD 3000). Die Weiterentwicklung der bisherigen Intel® Core™ Prozessoren bringt als wichtigste Neuerung AVX, eine überarbeitete und auf 256-Bit erweiterte Version der SSE-Befehle. Der integrierte Speichercontroller untertstützt jetzt offiziell DDR3-Speicher bis 1600 MHz und der verbesserte "Dynamic Turbomode" kann den Takt bei niedrigen Systemtemperaturen (wie nach dem Rechnerstart) kurzzeitig um bis zu 30% über den Maximalwert anheben.
Die integrierte GPU läuft mit 850 Mhz (max. 1350 mit Turbo) und teilt sich den gemeinsamen, 8 MB großen L3-Cache mit der CPU.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Bezeichnung: Intel® Core™ i7-2600K
Anzahl Prozessorkerne: 4
Taktfrequenz: 3400 MHz
Cache Level 1: 4x 64 kB
Level 2: 4x 256 kB
Level 3: 8192 kB
Befehlssätze: SSE 4.x, AVX, EIST, Intel 64, XD bit, Intel VT-x, Smart Cache, Clear Video, Turbo Boost, HyperThreading, AES-NI
Speicher-Controller:
Speicherstandards: DDR3-1066, DDR3-1333, DDR3-1600
Speicherkanäle: 2
Prozessorkern: Sandy Bridge, 32 nm
Bemerkung Turbo-Modus: CPU bis max. 3,8 GHz; GPU-Takt: 850 / 1.350 MHz (Standard / Turbo)
max. Leistungsaufnahme: 95 Watt
Bauform: FC-LGA4
Sockel: 1155
Lüfter: Befestigung für ATX-Mainboards
Anschlüsse: PWM-Lüfteranschluss
Besonderheiten mit aktivem CPU-Kühler

Weitere Infos: Die K-Modelle der Intel® Core™ Prozessoren sind für Übertakter und Technik-Enthusiasten gedacht und bieten einen frei wählbaren Multiplikator
 
~~~~~~~~~~~~~~~~~~~~~~
astropulse AVX app is in preparation.

heinz

Offline _heinz

  • Volunteer Developer
  • Knight who says 'Ni!'
  • *****
  • Posts: 2117
Re: optimized sources
« Reply #683 on: 20 Jan 2011, 11:03:25 am »
For all of you who are interested to know some more about IPP/AVX


/* Win32:
px - C-optimized for all IA-32 processors
a6 - Optimized for Pentium III processors remark:thru 5.3 only
w7 - Optimized for Pentium 4 processors (SSE + SSE2)
t7 - Optimized for Pentium 4 processors with Streaming SIMD Extensions 3 (SSE3)
v8 - New Optimizations for 32-bit applications on Intel® Core™ 2 and Intel® Xeon® 5100 Processors
p8 - New Optimizations for 32-bit applications on 45nm Intel® Core™2 Duo (Penryn,Nehalem,Westmere) family processors
s8 - New Optimizations for 32-bit applications on Intel® Atom™ family processors
e9 - Not in Win32
*/
/* Win64:
mx - C-optimized for all Intel 64-based Platforms
m7 - Optimized for Intel 64-Based Platforms for Pentium 4 processors with Streaming SIMD Extensions 3 (SSE3)
u8 - New Optimizations for 64-bit applications on Intel® CoreTM 2 and Intel® Xeon® 5100 Processors
y8 - New Optimizations for 64-bit applications on 45nm Intel® CoreTM2 Duo (Penryn,Nehalem,Westmere) family processors and

Intel® CoreTM i7 processors (Nehalem and Westmer)
n8 - New Optimizations for 64-bit applications on Intel® AtomTM family processors
e9 - AVX required,New Optimizations for 64-bit applications on Sandy Bridge µarchitecture
*/
// Description:
// introduced USE_xxx for better prefix-handling
// USE_AVX is still for 64Bit available.
// USE_AVX exclude all other
// USE_AVX and USE_ATOM exclude each other.
// USE_ATOM exclude all other
// USE_SSE42 and USE_SSE41 use the same prefixes(y8, p8)
/*
USE_IPP         Prefix
---------------------------------      
                   X64   X32
USE_AVX      e9   -(p8)
USE_ATOM   n8   s8
USE_SSE42   y8   p8
USE_SSE41   y8   p8
USE_SSSE3   u8   v8
USE_SSE3    m7   t7
USE_SSE2    m7  w7
USE_SSE      m7  w7
no order      mx   px
*/

We use macros to handle all that IPP stuff in the right way.
The PREFIX will be inserted into the different macros to generate the IPP statements.
sample: #define ippsFFTFree_C_32fc PREFIX(ippsFFTFree_C_32fc)
Further it is necessary that generated objectfile must be static. Linked as static.
64 Bit libs are necessary to link.



------ Neues Erstellen gestartet: Projekt: ap_client, Konfiguration: AP_QaxAVX_CSP x64 ------
Deleting intermediate files and output files for project 'ap_client', configuration 'AP_QaxAVX_CSP|x64'.
Compiling with Intel(R) C++ Compiler XE 12.0.1.127 [Intel(R) 64]... (Intel C++ Environment)
ap_schema.cpp
Compiling with Intel(R) C++ Compiler XE 12.0.1.127 [Intel(R) 64]... (Intel C++ Environment)
ap_fold.cpp
-----USE_AVX activ-----
-----AVX x64 e9 activ-----
Compiling with Intel(R) C++ Compiler XE 12.0.1.127 [Intel(R) 64]... (Intel C++ Environment)
dm_chunk_parallel.cpp
-----USE_AVX activ-----
-----AVX x64 e9 activ-----
mtrand.cpp
Compiling with Intel(R) C++ Compiler XE 12.0.1.127 [Intel(R) 64]... (Intel C++ Environment)
ap_timer.cpp
-----USE_AVX activ-----
-----AVX x64 e9 activ-----
Compiling with Intel(R) C++ Compiler XE 12.0.1.127 [Intel(R) 64]... (Intel C++ Environment)
ap_fileio.cpp
-----USE_AVX activ-----
-----AVX x64 e9 activ-----
Compiling with Intel(R) C++ Compiler XE 12.0.1.127 [Intel(R) 64]... (Intel C++ Environment)
ap_remove_radar.cpp
-----USE_AVX activ-----
-----AVX x64 e9 activ-----
Compiling with Intel(R) C++ Compiler XE 12.0.1.127 [Intel(R) 64]... (Intel C++ Environment)
ap_client_main.cpp
-----USE_AVX activ-----
-----AVX x64 e9 activ-----
Compiling with Intel(R) C++ Compiler XE 12.0.1.127 [Intel(R) 64]... (Intel C++ Environment)
ap_debug.cpp
Compiling with Intel(R) C++ Compiler XE 12.0.1.127 [Intel(R) 64]... (Intel C++ Environment)
version.cpp
Compiling with Intel(R) C++ Compiler XE 12.0.1.127 [Intel(R) 64]... (Intel C++ Environment)
sbtf.cpp
Compiling with Intel(R) C++ Compiler XE 12.0.1.127 [Intel(R) 64]... (Intel C++ Environment)
ap_gfx_main.cpp
-----USE_AVX activ-----
-----AVX x64 e9 activ-----
Compiling with Intel(R) C++ Compiler XE 12.0.1.127 [Intel(R) 64]... (Intel C++ Environment)
intrinsics.cpp
-----USE_AVX activ-----
-----AVX x64 e9 activ-----
Compiling with Intel(R) C++ Compiler XE 12.0.1.127 [Intel(R) 64]... (Intel C++ Environment)
ap_science.cpp
-----USE_AVX activ-----
-----AVX x64 e9 activ-----
-----USE_AVX activ-----
-----AVX x64 e9 activ-----
Linking... (Intel C++ Environment)
xilink: executing 'link'
So far we can compile, for linking we need the 64Bit libs of libboinc, libboincapi, setiboincdb

To create the 64Bit libs(libboinc, libboincapi, setiboincdb) of the project will take some time.


heinz

Offline _heinz

  • Volunteer Developer
  • Knight who says 'Ni!'
  • *****
  • Posts: 2117
Re: optimized sources
« Reply #684 on: 20 Jan 2011, 05:09:22 pm »
V8-Xeon
btw crunching:
precalculation says: will reach 100 Mio in 30 days..... on 20th february.
Since 20th december V8-Xeon produced constant 1Mio per day with two well clocked GTX470.
RAC is now: 1,030,867
So we hope the machine run stable next month too.
happy crunching  ;)

heinz




Offline _heinz

  • Volunteer Developer
  • Knight who says 'Ni!'
  • *****
  • Posts: 2117
Re: optimized sources
« Reply #685 on: 02 Feb 2011, 06:27:06 am »
V8-Xeon, 2xGTX470
cr/per day       week         month         average
1,101,713   7,014,636   30,799,731   1,014,102 

RAC dropped a bit by several driver restarts and twice I found the machine down in the morning,
although I could hold the average over 1 Mio.

heinz

technical news: a new dual bord for XEON is available
EVGA Classified SR-2 (Super Record 2)PART NUMBER: 270-WS-W555-A2
assembled machines you can find: there
I like this SUB-ZERO PC and the CUBE
have fun to explore it.

Offline _heinz

  • Volunteer Developer
  • Knight who says 'Ni!'
  • *****
  • Posts: 2117
Re: optimized sources
« Reply #686 on: 10 Feb 2011, 04:30:56 pm »
V8-Xeon
10th february 90,095,533.50 total
average credit: 1,091,774
we have still 10 days, we are on the line to catch 100 Mio total on 20th february.

heinz
 

Offline _heinz

  • Volunteer Developer
  • Knight who says 'Ni!'
  • *****
  • Posts: 2117
Re: optimized sources
« Reply #687 on: 16 Feb 2011, 01:35:51 pm »
16th february: now in the first 500
Pos   Name  Summury       cr/day          cr/week     cr/month       average
496 _heinz  96,392,212   1,316,379   9,851,843   31,132,429   1,090,954 

although I found the machine yesterday down again in the morning I was able to hold the average today.

heinz


Offline _heinz

  • Volunteer Developer
  • Knight who says 'Ni!'
  • *****
  • Posts: 2117
Re: optimized sources
« Reply #688 on: 18 Feb 2011, 07:09:28 pm »
samedi 19th february
I got the 100 Mio total today.
100,114,951.92

number 33 in France
Pos  Name    Summary        cr/day           cr/week     cr/month      average
33   _heinz  100,114,952   2,116,920   8,843,266   32,506,661   1,191,680

have a look at the full statistic

It shows it is possible to produce constant ~1 Mio credit/day with two  well clocked GTX470, started on 10th of december.
Yesterday I added a GTX570 and V8-Xeon runs now with 3 cards.
v8-xeon_2-GTX470
v8-xeon_3_cards
The card in the middle is the GTX570 and a bit longer than the GTX470.
1.6 Mio/day are precalculated. We will see if it will become true.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The upper card runs 10 grd celsius hotter than the other two, althought the case is well aired
slot0 GTX470  ~87 grd Celsius
slot1 GTX570  ~75 grd
slot3 GTX470  ~77  grd
roomtemp=23,5 grd Celsius
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
happy crunching
heinz
 

« Last Edit: 20 Feb 2011, 01:57:31 pm by _heinz »

Offline _heinz

  • Volunteer Developer
  • Knight who says 'Ni!'
  • *****
  • Posts: 2117
Re: optimized sources
« Reply #689 on: 22 Feb 2011, 12:23:31 pm »
1.6 Mio/day are precalculated. We will see if it will become true.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ 1,669,906.80 seither, today I got it.
It's not easy to hold the machine stable, GPU 0 slot 0 overheated, 86 grd celsius. It seems the card got hot air from the main memory, its cooler blows the air onto the memory slots and hot air comes out left and right. Right under it is GPU0 and sucks this hot air.
Temperatur-difference between GPU0 and GPU1 are ~10 grd celsius. Must do something.

 

Welcome, Guest.
Please login or register.
 
 
 
Forgot your password?
Members
Total Members: 97
Latest: ToeBee
New This Month: 0
New This Week: 0
New Today: 0
Stats
Total Posts: 59559
Total Topics: 1672
Most Online Today: 355
Most Online Ever: 983
(20 Jan 2020, 03:17:55 pm)
Users Online
Members: 0
Guests: 39
Total: 39
Powered by EzPortal