Whetstone Benchmark on PowerPC 440 CPU

Performance Assessment using the Whetstone v1.2 Benchmark on the Xilinx ML507 FPGA Platform
 

Table of Contents [Toc]

Overview
Test Setup
Hardware Architecture
   Clocking Infrastructure
   Glossary
The Whetstone Benchmark
IBM PowerPC Performance Libraries
Test Results
   Measurement Results w/ FPU
   Measurement Results w/o FPU
   Inter-/Extrapolated Results
Summary & Conclusions
References
 

Overview   [Toc] [Top]

• Assessment of the Whetstone performance of the embedded PowerPC 440 CPU with present and absent double-precision floating-point unit (FPU) implemented as hardware soft-IP on the CPU's fabric coprocessor bus (FCB) and serving as auxiliary processing unit (APU) [1, 2].
• Impact analysis of the compiler's different code optimization techniques on the specifics of the Whetstone benchmark.
• It is noted that the code size is intended to be kept within 32 kB in order to run entirely from cache, hence preventing any penalties due to cache miss and detrimental aspects of the hierarchical memory subsystem such as interconnect latencies.

Well, I'm aware of the paraphrase 'fake, lie, benchmark', therefore I seek to follow best practices by clearly specifying the overall setup employed including hardware system, software and compiler versions used, and employed compilation flags. Moreover, I do not intend to tweak any system specifics in order to reach highest scores, but rely on ordinary compiler flags such as '-O2' and '-O3'. The corresponding results are listed below. Any interpretation is left to the prospective reader.

Note that in general, benchmark numbers are meaningless without proper specification of compiler settings and benchmarking conditions.

Last but not least, keep in mind Dilbert:

Test Setup   [Toc] [Top]

Hardware Architecture   [Toc] [Top]

The hardware architecture is depicted below. Only essential peripheral blocks have been attached to the CPU.

Hardware architecture for the performance assessment of the embedded IBM PowerPC 440 CPU using the Whetstone benchmark. A double-precision floating point unit (DP-FPU) as soft IP block clocked at 200 MHz has been connected through the fabric coprocessor bus (FCB) to the CPU core running at 400 MHz. As peripheral blocks, 32 kB on-board memory, timer, interrupt controller and RS232 UART for serial communication are connected.

Clocking Infrastructure   [Toc] [Top]

The clocking scheme of the system architecture needs to adhere to a bunch of different rules imposed by various interconnect and device specifics. The applied clocking scheme and the corresponding important clock ratios are listed below.

Glossary   [Toc] [Top]

The Whetstone Benchmark   [Toc] [Top]

The Whetstone benchmark is a synthetic benchmark initially composed in 1972 using the Algol 60 programming language [3]. Later on, the source code of the benchmark was ported to FORTRAN and C [4, 5]. The Whetstone benchmark intends to qualify the CPU performance by employing various CPU-centric operations within approx. 150 code statements. The benchmarks itself performs a number of iterations on a set of modules. Each module performs a loop on certain instructions such as procedure calls, branching statements, fixed- and floating-point arithmetics, and trigonometric operations - with a predefined weighted amount of executions for each loop derived from the initially specified number of iterations. The dominant code section usually accounts for 30%-50% of execution time and performs floating-point operations [4]. As stated in the code, a loop number of 10 corresponds to one million Whetstone instructions. By specifying the loop number and by measuring the overall execution time of the benchmark on a given CPU, the individual CPU performance can then be reflected in a benchmark-specific metric called Million Whetstone Instructions per Second (MWIPS).

In general, a synthetic benchmark like Whetstone has to satisfy contradicting design requirements [3]: On one hand, the benchmark must be simple enough to be easily portable from one machine to another and to be translated to different programming languages. On the other hand, the benchmark has to exhibit sufficient complexity as to represent a sound measure for CPU performance without inferring language-specific peculiarities, while at the same time preventing excessive compiler optimizations. According to [3], at design time of the Whetstone benchmark, code movement out of loops and similar extensive optimization techniques were rarely performed by ALGOL 60 compilers, hence procedure calls and loops were viable techniques when confined to the ALGOL 60 programming language.

Citing Roy Longbottom's [4] interesting statement about compiler optimizations attributed to the Whetstone benchmark:

"The code was designed to be non-optimisable and optimising compilers did not have a significant impact until the introduction of in-lining of subroutine instructions. Although this produces code outside the definition of Whetstone instructions, which include a specific proportion of procedure calls, it is a valid technique to obtain the best performance out of modern systems and may well be the compiler default optimisation level. As reflected in the PC results, a good compiler can halve the execution time by in-lining, careful choice of instructions and sequence, and omission of intermediate stores/loads.

With in-lining and global optimisation, a small number of compilers identified that the dominant loop did not have to be executed and immediately lead to an apparent more than doubling of MWIPS speeds. This was identified by the 1980 enhancements and fixed in 1987, essentially by changing the name of one variable. Unlike some other standard benchmarks, Whetstone results were generally verified as part of the Central Computer and Telecommunications Agency (CCTA) system appraisal, in project related benchmarking sessions or during acceptance trials. It was also standard practice to run the tests with different levels of optimisation and obvious over - optimised results were not published."

IBM PowerPC Performance Libraries   [Toc] [Top]

Selected tests with absent floating-point unit (FPU) employ the compilation flag '-mppcperflib' which infers the IBM PowerPC Performance Libraries for optimized low-level integer and floating-point emulation, and optimized string handling routines [6]. According to Xilinx, the IBM PowerPC Performance Libraries may show an average of three times increase in speed on applications that heavily use these routines.

Caution: The IBM PowerPC Performance Libraries are only intended for improving the execution of emulated floating-point arithmetics and hence cannot be used in conjunction with floating-point hardware, i.e., with active '-mfpu' switch.

Test Results   [Toc] [Top]

The source code has been compiled using GCC wit compiler flags indicated below.
Any interpretation is left to the prospective reader.

Measurement Results w/ FPU   [Toc] [Top]

IBM PowerPC 440 @ 400 MHz with dedicated double-precision FPU @ 200 MHz

The gray shaded table entries represent compiler settings violating the intended compilation rules of the synthetic benchmark.

Measurement Results w/o FPU   [Toc] [Top]

IBM PowerPC 440 @ 400 MHz without dedicated FPU, using software-emulated floating-point

The gray shaded table entries represent compiler settings violating the intended compilation rules of the synthetic benchmark.

Inter-/Extrapolated Results   [Toc] [Top]

PPC w/ FPU, using 134.7 MWIPS @ 400 MHz as reference data:

PPC w/o FPU, using 35.3 MWIPS @ 400 MHz as reference data:

Summary & Conclusions   [Toc] [Top]

By using an embedded PowerPC 440 CPU running at 400 MHz in conjunction with a double-precision floating-point unit running at 200 MHz, a Whetstone performance of 135 MWIPS was measured with cache-adjusted code size and compiler settings not violating the rules of the benchmark. Extrapolated to the fastest available Virtex-5 FPGA speed grade (i.e., a CPU clock frequency of 550 MHz), the PowerPC 440 would achieve 185 MWIPS for the version 1.2 benchmark.

When using a PowerPC 440 CPU running at 400 MHz without dedicated FPU coprocessor, all floating-point arithmetics have to be performed by software. In this context, a Whetstone performance of 35 MWIPS was measured by inferring the IBM PowerPC Performance Libraries during compilation. Without inferring the PowerPC Performance Libraries, the Whetstone performance of the employed system decreases to merely 15 MWIPS.

However, the relevance of these performance numbers are in general questionable: The code must be running entirely from cache without any I/O transfers to show best results. As soon as larger code size, costly I/O transfers, and different compiler options are involved, these numbers are merely theoretical.

Last but not least, it is very impressive to see how different code optimization techniques of the compiler significantly influence the execution time of the identical piece of code. Here at last it becomes obvious, that a CPU performance measuring tool like a benchmark needs to be designed by keeping clearly in mind hardware, software and compiler architectures and capabilities.
 

References   [Toc] [Top]

[1] Xilinx Inc., Virtex-5 Family Brochure, Dec 2008

[2] Xilinc Inc., Virtex-5 Website

[3] H.J. Curnow, B.A. Wichmann, "A synthetic benchmark" (Whetstone), Computer Journal, Vol 19, No 1, pp 43-49, 1976

[4] Roy Longbottom, Whetstone Benchmark History and Results

[5] netlib.org, Benchmark Programs and Reports

[6] sourceforge.net, IBM PowerPC Performance Libraries

Last updated: 2012/12/30