|Anandtech's 'No More Mysteries' Server Benchmarks
In June of 2005, Johan De Gelas of the Anandtech.com site published a comparison of Mac OS X Server running on a PowerMac G5 with three PC servers running SuSE Linux. The report concluded, in part, that:
Mac OS X is incredibly slow, between 2 and 5(!) times slower, in creating new threads, as it doesn't use kernel threads, and has to go through extra layers (wrappers). No need to continue our search: the G5 might not be the fastest integer CPU on earth - its database performance is completely crippled by an asthmatic operating system that needs up to 5 times more time to handle and create threads.
From that report, it sounds like their testing carefully isolated a comparison between the kernel performance of Mac OS X and Linux. Does this mean that Mac OS X would benefit from using the Linux kernel? Certainly, if Linux is 5 times faster in its core functions, it would warrant significant efforts by Apple to get that technology in Mac OS X.
As it turned out, Anandtech's benchmarking efforts were so poorly executed that the numbers they generated were worthless. On top of that, the conclusions they drew were based in conjecture that demonstrated ignorance of the software architectures they were attempting to profile. What did they get wrong?
For starters, the there was a poor control of variables; what was it they were testing? Was it:
- the G5 vs. x86 hardware?
- Mac OS X vs. Linux operating systems?
- the degree of optimizations available for a software package?
- their own understanding of application optimization across platforms?
Anandtech didn't seem to know what they were trying to test, because they didn't make any controlled comparisons. Further, when they did make efforts to control variables, they did so unfairly.
They reported that, in order to be consistent, they used the GCC 3.x compiler across both platforms. This isn't a controlled variable; rather, they were just using a compiler with more optimization on x86 and in Linux against one that is outdated for use on either the G5 or the version of Mac OS X they were using. The GCC 4 compiler that Mac OS X needed wasn't available in the Linux distro they used, so they used the old version on both. Why?
||Similarly, they chose to compile the MySQL database software manually on both platforms to supposedly keep the tests "consistent," but then they chose not to include the Mac OS X specific patches for MySQL that would alter its threading mechanism.
This caused a significant performance hit against Mac OS X, because MySQL was written for the Linux threading model, not Mach or more generic model. The correct patches would have put MySQL on more equal ground, but they chose not to install them. Why?
Manually building MySQL from source was unnecessary anyway, because binary builds with Mac OS X specific patches were available for download via links from the official software project site.
It's difficult to fathom why they would go out of their way to do all this unless they intentionally wanted to compile a crippled version for Mac OS X in order to generate more sensationalistic performance discrepancies.
More interesting than the numbers Anandtech published were the numbers they didn't publish.