Version 4.1.0 -- 1997.08.26
================================================================
To use Unixbench:
make
Run
================================================================
You may run individual benchmark programs and groups of programs.
There are also a few options.
Run [<bench> ...] [<group> ...] [-qvd]
-q quiet no output to standard out
-v verbose extra output to standard out
-d debug Run is displayed as it is run
-<number> Use <number> as maximum number of iterations
(e.g. Run -1 index)
GROUP BENCHMARK EXPLANATION SOURCE
-------- ----------- -------------------- -------------
index only those need for
generating the BYTE
index
dhry2reg dhrystone 2 with regs src/dhry_1.c
src/dhry_2.c
src/dhry.h
whetstone-double double-prec. whetstone src/whets.c
execl execl call src/execl.c
src/big.c
fstime filesystem throughput src/fstime.c
fsbuffer filesystem throughput src/fstime.c
fsdisk filesystem throughput src/fstime.c
pipe pipe throughput src/pipe.c
context1 pipe context switch src/context1.c
spawn process creation src/spawn.c
shell load system with pgms/multi.sh
concurrent shell pgms/tst.sh
scripts src/looper.c
syscall system call overhead src/syscall.c
arithmetic
whetstone-double double-prec. whetstone src/whets.c
arithoh arithmetic overhead src/arith.c
register register arithmetic src/arith.c
short short arithmetic src/arith.c
int int arithmetic src/arith.c
long long arithmetic src/arith.c
float float arithmetic src/arith.c
double double arithmetic src/arith.c
system
syscall system call overhead src/syscall.c
pipe pipe throughput src/pipe.c
context1 pipe context switch src/context1.c
spawn process creation src/spawn.c
execl execl call src/execl.c
fstime filesystem throughput src/fstime.c
fsbuffer filesystem throughput src/fstime.c
fsdisk filesystem throughput src/fstime.c
shell load system with pgms/multi.sh
concurrent shell pgms/tst.sh
scripts src/looper.c
misc
C compile and link /bin/cc
src/looper.c
dc calculations with dc testdir/dc.dat
src/looper.c
hanoi recursion src/hanoi.c
src/looper.c
dhry
dhry2 dhrystone 2 w/o regs src/dhry_1.c
src/dhry_2.c
src/dhry.h
dhry2reg dhrystone 2 with regs src/dhry_1.c
src/dhry_2.c
src/dhry.h
shell load system with pgms/multi.sh
concurrent shell pgms/tst.sh
scripts
** Most C programs also include src/timeit.c
===================== RELEASE NOTES =====================================
v4.1.0
Double precision Whetstone put in place instead of the old "double" benchmark.
Removal of some obsolete files.
"system" suite adds shell8.
perlbench and poll added as "exhibition" (non-index) benchmarks.
Incorporates several suggestions by Andre Derrick Balsa <andrewbalsa@usa.net>
Code cleanups to reduce compiler warnings by David C Niemi <niemi@tux.org>
and Andy Kahn <kahn@zk3.dec.com>; Digital Unix options by Andy Kahn.
======================== Jun 97 ==========================
v4.0.1
Minor change to fstime.c to fix overflow problems on fast machines. Counting
is now done in units of 256 (smallest BUFSIZE) and unsigned longs are used,
giving another 23 dB or so of headroom ;^) Results should be virtually
identical aside from very small rounding errors.
======================== Dec 95 ==========================
v4.0
Byte no longer seems to have anything to do with this benchmark, and I was
unable to reach any of the original authors; so I have taken it upon myself
to clean it up.
This is version 4. Major assumptions made in these benchmarks have changed
since they were written, but they are nonetheless popular (particularly for
measuring hardware for Linux). Some changes made:
- The biggest change is to put a lot more operating system-oriented
tests into the index. I experimented for a while with a decibel-like
logarithmic scale, but finally settled on using a geometric mean for
the final index (the individual scores are a normalized, and their
logs are averaged; the resulting value is exponentiated).
"George", certain SPARCstation 20-61 with 128 MB RAM, a SPARC Storage
Array, and Solaris 2.3 is my new baseline; it is rated at 10.0 in each
of the index scores for a final score of 10.0.
Overall I find the geometric averaging is a big improvement for
avoiding the skew that was once possible (e.g. a Pentium-75 which got
40 on the buggy version of fstime, such that fstime accounted for over
half of its total score and hence wildly skewed its average).
I also expect that the new numbers look different enough from the old
ones that no one is too likely to casually mistake them for each other.
I am finding new SPARCs running Solaris 2.4 getting about 15-20, and
my 486 DX2-66 Compaq running Linux 1.3.45 got a 9.1. It got
understandably poor scores on CPU and FPU benchmarks (a horrible
1.8 on "double" and 1.3 on "fsdisk"); but made up for it by averaging
over 20 on the OS-oriented benchmarks. The Pentium-75 running
Linux gets about 20 (and it *still* runs Windows 3.1 slowly. Oh well).
- It is difficult to get a modern compiler to even consider making
dhry2 without registers, short of turning off *all* optimizations.
This is also not a terribly meaningful test, even if it were possible,
as noone compiles without registers nowadays. Replaced this benchmark
with dhry2reg in the index, and dropped it out of usage in general as
it is so hard to make a legitimate one.
- fstime: this had some bugs when compiled on modern systems which return
the number of bytes read/written for read(2)/write(2) calls. The code
assumed that a negative return code was given for EOF, but most modern
systems return 0 (certainly on SunOS 4, Solaris2, and Linux, which is
what counts for me). The old code yielded wildly inflated read scores,
would eat up tens of MB of disk space on fast systems, and yielded
roughly 50% lower than normal copy scores than it should have.
Also, it counted partial blocks *fully*; made it count the proportional
part of the block which was actually finished.
Made bigger and smaller variants of fstime which are designed to beat
up the disk I/O and the buffer cache, respectively. Adjusted the
sleeps so that they are short for short benchmarks.
- Instead of 1,2,4, and 8-shell benchmarks, went to 1, 8, and 16 to
give a broader range of information (and to run 1 fewer test).
The only real problem with this is that not many iterations get
done with 16 at a time on slow systems, so there are some significant
rounding errors; 8 therefore still used for the benchmark. There is
also the problem that the last (uncompleted) loop is counted as a full
loop, so it is impossible to score below 1.0 lpm (which gave my laptop
a break). Probably redesigning Shell to do each loop a bit more
quickly (but with less intensity) would be a good idea.
This benchmark appears to be very heavily influenced by the speed
of the loader, by which shell is being used as /bin/sh, and by how
well-compiled some of the common shell utilities like grep, sed, and
sort are. With a consistent tool set it is also a good indicator of
the bandwidth between main memory and the CPU (e.g. Pentia score about
twice as high as 486es due to their 64-bit bus). Small, sometimes
broken shells like "ash-linux" do particularly well here, while big,
robust shells like bash do not.
- "dc" is a somewhat iffy benchmark, because there are two versions of
it floating around, one being small, very fast, and buggy, and one
being more correct but slow. It was never in the index anyway.
- Execl is a somewhat troubling benchmark in that it yields much higher
scores if compiled statically. I frown on this practice because it
distorts the scores away from reflecting how programs are really used
(i.e. dynamically linked).
- Arithoh is really more an indicator of the compiler quality than of
the computer itself. For example, GCC 2.7.x with -O2 and a few extra
options optimizes much of it away, resulting in about a 1200% boost
to the score. Clearly not a good one for the index.
I am still a bit unhappy with the variance in some of the benchmarks, most
notably the fstime suite; and with how long it takes to run. But I think
it gets significantly more reliable results than the older version in less
time.
If anyone has ideas on how to make these benchmarks faster, lower-variance,
or more meaningful; or has nice, new, portable benchmarks to add, don't
hesitate to e-mail me.
David C Niemi <niemi@tux.org> 7 Dec 1995
======================== May 91 ==========================
This is version 3. This set of programs should be able to determine if
your system is BSD or SysV. (It uses the output format of time (1)
to see. If you have any problems, contact me (by email,
preferably): ben@bytepb.byte.com
---
The document doc/bench.doc describes the basic flow of the
benchmark system. The document doc/bench3.doc describes the major
changes in design of this version. As a user of the benchmarks,
you should understand some of the methods that have been
implemented to generate loop counts:
Tests that are compiled C code:
The function wake_me(second, func) is included (from the file
timeit.c). This function uses signal and alarm to set a countdown
for the time request by the benchmark administration script
(Run). As soon as the clock is started, the test is run with a
counter keeping track of the number of loops that the test makes.
When alarm sends its signal, the loop counter value is sent to stderr
and the program terminates. Since the time resolution, signal
trapping and other factors don't insure that the test is for the
precise time that was requested, the test program is also run
from the time (1) command. The real time value returned from time
(1) is what is used in calculating the number of loops per second
(or minute, depending on the test). As is obvious, there is some
overhead time that is not taken into account, therefore the
number of loops per second is not absolute. The overhead of the
test starting and stopping and the signal and alarm calls is
common to the overhead of real applications. If a program loads
quickly, the number of loops per second increases; a phenomenon
that favors systems that can load programs quickly. (Setting the
sticky bit of the test programs is not considered fair play.)
Test that use existing UNIX programs or shell scripts:
The concept is the same as that of compiled tests, except the
alarm and signal are contained in separate compiled program,
looper (source is looper.c). Looper uses an execvp to invoke the
test with its arguments. Here, the overhead includes the
invocation and execution of looper.
--
The index numbers are generated from a baseline file that is in
pgms/index.base. You can put tests that you wish in this file.
All you need to do is take the results/log file from your
baseline machine, edit out the comment and blank lines, and sort
the result (vi/ex command: 1,$!sort). The sort in necessary
because the process of generating the index report uses join (1).
You can regenerate the reports by running "make report."
--
========================= Jan 90 =============================
Tom Yager has joined the effort here at BYTE; he is responsible
for many refinements in the UNIX benchmarks.
The memory access tests have been deleted from the benchmarks.
The file access tests have been reversed so that the test is run
for a fixed time. The amount of data transfered (written, read,
and copied) is the variable. !WARNING! This test can eat up a
large hunk of disk space.
The initial line of all shell scripts has been changed from the
SCO and XENIX form (:) to the more standard form "#! /bin/sh".
But different systems handle shell switching differently. Check
the documentation on your system and find out how you are
supposed to do it. Or, simpler yet, just run the benchmarks from
the Bourne shell. (You may need to set SHELL=/bin/sh as well.)
The options to Run have not been checked in a while. They may no
longer function. Next time, I'll get back on them. There needs to
be another option added (next time) that halts testing between
each test. !WARNING! Some systems have caches that are not getting flushed
before the next test or iteration is run. This can cause
erroneous values.
========================= Sept 89 =============================
The database (db) programs now have a tuneable message queue space.
queue space. The default set in the Run script is 1024 bytes.
Other major changes are in the format of the times. We now show
Arithmetic and Geometric mean and standard deviation for User
Time, System Time, and Real Time. Generally, in reporting, we
plan on using the Real Time values with the benchs run with one
active user (the bench user). Comments and arguments are requested.
contact: BIX bensmith or rick_g