Business Winstone 2002 Benchmark Results with the 100th edition PC Pro 100 Essential Utilities.

The Experiment

We tested Windows 98 (ME to come) 2000 and XP to destruction by loading them up with as many as possible of PC Pro Magazines Feb 2003  Essential Utilities. When installing these applications we always chose to launch the application at start up and to put icons on the desktop where given the option. We then tested the loaded up systems with the Business Winstone 2002 Benchmark as follows.

Business Winstone is a thorough real word Benchmark written by Ziff Davis. It gives your PC a good workout. It is designed to emulate  a 'day in the office'. The Benchmark opens several typical office and internet applications at once.

The PC used was an Athlon XP 1800+ Machine, with 512MB of DDRAM, running on a 133Mhz FSB. The Motherboard was an ABIT KR7A - RAID, an IBM 120GXP 60GB Hard disk driven from the motherboard ATA100 motherboard controller and a bog standard Geforce 2MX graphics card. Windows 98, ME,  had the AMD miniport driver installed that came with the Motherboard, and Hyperion 4 in 1 VIA Southbridge drivers and the Sparkle Geforce2MX drivers that were provided with the Graphics card. Windows 2000 had the Hyperion 4 in 1 Via Southbridge drivers and the Sparkle Geforce2MX drivers only. Windows XP was left with its own integral drivers. 

The Methodology

The methodology was not to defragment between tests, and to reboot every other test, and not to terminate applications running in the background, and not to allow XP time to finish its housekeeping if possible (as the Business Winstone 2002 Benchmark provides the option to do this with XP) so as to as closely as possible mimic a day in the office. Business Winstone 2002 was uninstalled and then re-installed before every series of 3/4 tests.

The Controversy

We hope to have presented these present results in such a way that they can easily be duplicated. The results that we have published in the past have caused a great debate in the Industry. Some editors accept that Windows has max loading problems and were not surprised by our previous results. Others frankly did not believe them or accept our conclusions. PC Pro editorial in particular accepted that our results and conclusions may hold true for 98 Me, but not for 2K or XP. Although they suspected that questionably coded software apps on their competitors cover disks played a part in the results.

So we have some good old controversy here. Hence we decided to present these new results as below so that any reader can see precisely what we have done and replicate the tests himself if he desires (or has the time available).

We tested the first 57 installable applications on the PC Pro 100 Essential Utilities Cover Disk that came free with the 100th edition of the UKs pre-eminent Computer Magazine PC Pro, February 2003. We found that 8 of these apps were 'incompatible' with BW2002 for various reasons. 49 of them worked fine with BW2002.

Here is a list of the 49 apps we tried that ran well with BW2002, in the order they appear (subject to choice of classes of software) on the PC Pro CD. We installed them in the order below.

Apps 1- 20

EzDesk 1.8
InfoTray 1.10
KeyText 2000 v2.22
Recent Documents 2002 v2
TClockEx 1.4.2
ToggleMouse v4.5.9
TrayDay v6.5
TreeSize Professional v2.4.3
HyperSnap DX 4.54.02
Irfan View 3.75
Microangelo 5.56
PicaView 2
Real-Draw Pro 2.45
Dun Manager 3
GetRight 4.5e
Hyper Terminal Private Edition 6.3
Microsoft IE6 SP1
LinkStash 1.1.0.14
mIRC 6.03
Mozilla 1.1

Apps 21 - 30

NeoTrace Express 3.25
Opera 6.05 with Java and SSL patch
Pegasus Mail 4.02
Sam Spade for Windows 1.14
Smart FTP 1.0.970
WAR FTP Daemon 1.65
Clone CD v4.0.1.10
Exp Print v1.6.1.0
IsoBuster 1.1
NetSwitcher 3.2.5

Apps 31 - 49

PowerToys for Windows 95 98 ME
SiSoft Sandra 2002 6.8.97
VoptXP 7.13
Accent Composer 1.09c
Adobe Acrobat 5.05
ClipCache Plus 2.9
Jot+ Notes v2.5.5
OutBack Plus v3.1
PGP Freeware Desktop Security for Windows 32 Bit 7.03
Secure-IT v1.44
TextPad v4.5
To-do List 2.0.7
Virtual Network Computing VNC
Winrar 3.0
BrainStorm 3 Build 17
Macro Express 3.0f
Agnitum Outpost Firewall Free 1.0.1817.1645
Trillian
WinKey 2.8

8 Apps which caused BW2002 to fail

Netscape 7                           Could not be used because Winstone uses this apps itself in its testing routines
Winzip 8.1                            Could not be used because Winstone uses this apps itself in its testing routines
The GIMP for Windows       We couldn't install it
FilterGate 5.05                      Caused Winstone to Freeze.
Zone Alarm                           Caused Winstone to freeze and irrelevant popups to occur  - the PC did not even have a modem!
FinePrint 2000                      Caused Winstone to stop with a printing error.
pdfFactory v1.51                  Caused Winstone to crash
SuperGee 12 Ghosts 6.02    Caused Winstone to crash (way too spooky)

The last 6 of these are incompatible with either Windows, BW2002 itself of most likely one or other of the BW2002 components which are Microsoft Office XP (pro), Lotus Notes, Winzip, Norton AntiVirus and Netscape 6.02.

Windows 98 could run BW2002 with any of the above 49 apps installed in groups of 10 or 20 or 30, but it could not run BW2002 with all 49 installed. Many of the apps automatically start on PC start up (if the respective application setup programs enable this). So Windows 98 was carrying out around 20 processes for the apps in addition to the 10 or so BW2002 processes at once. It is the number of simultaneous processes that is causing the problem here. If you install all 49 apps, and disable each of the from starting at startup (which is pointless in the case of firewalls etc.) then you could probably run them individually, one at a time, without any difficulty, but this is unrealistic. It is better to split them up between 2 Windows systems.

For XP we introduced a further dozen titles from the PC Pro disk, and then 15 more larger apps from 3 other cover disks

Apps 50 - 61

Ad-Aware 5.83
RefUpdate 2.1 for Ad-Aware
Agent 1.92
Ameol 2.53
Audacity 1
Eraser 5.5.2
HammerSnipe PowerTool
LauncherASM
Research-Desk 3.1.889 Pro
The Bat! 1.61
WebCompiler 2000 1.67
Winamp 3

SmartboardXP 3.12 was incompatible with BW2002, because it is effectively an always on top Window.

Larger Apps 62 - 76

PC Pro CD June 2003
4D 6.7.4
AMAPI 3D 5.15
SkyMap Pro 9
The Newwws
Creative Thinker
Kaspersky Anti Virus Personal 4.0

Computer Shopper CD June 2003
Mediator 5
WinRoute Lite 4.2
Movie Collector 3.1
VCDEasy v.1.1.4

Computer Buyer CD January 2003
MultiMedia Builder 3.0
Adobe Photoshop 7.0.1
PhotoImpact 8
Xara 3D v5
Windates 4.25

Adding 2 mins reduces the score by around 20% (since the test takes around 8 mins). Adding 4 mins reduces the score by 33%

XP seems to take around 2 to 5 minutes to do some system work when it starts up and periodically thereafter. Business Winstone gives the user the chance to account for this by waiting until it detects that this work has ended and then starting the test. But with more than 30 apps installed XP refused to start running the test until it had finished its system business. Ignoring this wait gives a non real world figure. In the real world one does not wait by one's PC until XP has finished doing what it wants to do!! So XP is actually slower the 98 due to this system activity. It has a faster engine than 98 (27 compared with 24) but it has a chauffeur who has his own agenda. He will not listen to you if he is busy. The effect of this is that the test which typically took between 7 and 8 minutes can take an extra 5 minutes, which takes it up to to 13 minutes. This would translate to a score of around 14!! Which is truly diabolical! This is why XP 'feels' so slow. The order of installation seems to be relevant as well, since the larger apps on their own slowed down XP more than they did when loaded on top of 61 smaller apps!

XP became unuseably slow with 76 apps installed. 

Windows 2000 is considerably faster than XP when carrying a reasonable application load. It is slightly less stable than XP at high application load (above 50 apps), but the increase in stability of XP is at the expense of some kind of system housekeeper which slows XP down to a crawl rather than letting it crash.

The Windows ME results will follow shortly, but our previous tests have shown us that ME is similar to 98. It is slightly slower and slightly less stable in our experience.

Conclusions

More research needs to be done, but clearly Windows 2000 is the all round best operating system. 2K is 20% faster than 98, and 98 is 15% faster than XP with BW2002. XP home, is not a performance OS at all, however it does make admirable, if rather time consuming, attempts not to crash. 4 Copies of 98 and HyperOs with 60 apps split amongst them, gives you a faster and more stable system than one copy of XP with 60 applications in it, which many of our customers already realise. But the King of Windows Operating Systems is Windows 2000 pro.

Based on the above we would recommend everyone to have at least one copy of Windows 2000 as well as their favourite 98 or XP system. For us the perfect PC would have at least 2 x 2K 2 x 98 and 2 x XP. You can purchase such PCs from Holly Computers, visit www.hollycomputers.co.uk

HyperOs Systems and PC Pro Editorial Debate

Let's now turn to the debate between HyperOs and PC Pro editorial. Well, HyperOs asserted that if you load all 57 Essential Utilities (and specials) into XP then Business Winstone will not run, so your Office PC would effectively be scuppered. The results show that this is certainly true. However HyperOs had asserted that the failure was due to the max load of XP being breached. They were wrong! PC Pro asserted that it was due to Questionable software - They were right! The following 5 apps are questionable for an Office System:

PC Pro editorial further asserted that the max load might well be 30 odd apps with 9x but not with 2K or XP systems. They were right here again. The bottom line is that XP can run every decent app on the PC Pro February 2003 cover CD (61 of them!!). Microsoft have done very well here. Although XP becomes very slow with just 15 larger applications. Perhaps with XP it is the total size of all apps that causes the trouble? 2K can carry all 49 PC Pro Essential Applications and run them and BW2002 at blistering speeds faster than a Bare Windows 98 or a Bare Windows XP system can run the Benchmark. But it got into trouble with 61 of them, yet it could run these 61 together with 15 larger apps, so possibly installation order is a feature here.

But in the real world, assuming that the PC Pro cover disk is a typical representation of what a user would install (it ought to be better than typical), around 10% of these carefully chosen apps conflict with Major Office apps (if you run a lot of them all together as BW2002 does). So the poor user, is going to get in trouble with one in every 10 apps he installs. Sadly he may not realise which one it is immediately, since the conflict may not become apparent until he tries the Office app or combination of Office apps that don't like the utility.

The best way to avoid these sorts of problems is plainly to separate out your mission critical apps and your 'fun' apps into 2 different Windows systems. In this way there is less chance of an application conflict. The more you separate the less the chance of a conflict.

Check back for our Windows ME results soon!