On Benchmarketing

This blog contains only two entries until today but both have a very interesting view to different parts of the x86-world: Scientias Blog.

Interesting article about new TPC benchmark suites. It´s good to see, that the media support Sun opinion, that TPC-C is absolutly useless this days. It should be subsituted by something more realistic sooner than later:

One problem with TPC-C was the ease with which certain servers could generate unrealistically high scores using hardware and software configurations that are highly improbable in the real world. For example, high TPC-C scores come from servers with colossal numbers of hard drives--6,548 in the case of IBM's top score.

and

Another problem hinged on the fact that TPC-C's test was easily distributed among relatively independent servers linked in a cluster. That gave the impression that a number of inexpensive machines were as good as a single multiprocessor behemoth, leading the consortium to list results separately for clustered and non-clustered systems.

I looked for a project into the SPEC website today and saw that the T2000 still leads the SPECweb2005 benchmark by roughly 4000 points. And this benchmark was made on the T2000 in November last year. Even the new Woodcrests stopped short of 10000 points and that were already fully blown 4 core systems (maximum for Woodcrest right now). Now keep in consideration that Niagara II already runs in our labs. Interesting times . Well ... Rolf, we should do the power meter gag again next cebit

Okay, very impressive number: 1.000.000 operation per second. But look at the bill of material: 24 Filerheads, not the small one, the NetApp 6040 is one of the bigbadass machines, 768 Gigabytes of Cache, 48 GB NVRAM, 1152 Harddisks. Does anybody uses really a environment like this? It´s seems to me like using an SF25K for fileserving.
This is the nature of benchmarketing. It´s not a matter of being realistic of realistic workloads, it´s a expression of the fine art of understanding and outsmart a benchmark to make a marketing statement.