Comparison-based File Server Verification
Yuen-Lin Tan, Terrence Wong, John D. Strunk, Gregory R. Ganger
Carnegie Mellon University
Comparison-based server verification involves testing a
server by comparing its responses to those of a reference
server. An intermediary, called a “server Tee,” interposes
between clients and the reference server, synchronizes
the system-under-test (SUT) to match the reference
server’s state, duplicates each request for the SUT,
and compares each pair of responses to identify any discrepancies.
The result is a detailed view into any differences
in how the SUT satisfies the client-server protocol
specification, which can be invaluable in debugging
servers, achieving bug compatibility, and isolating performance
differences. This paper introduces, develops,
and illustrates the use of comparison-based server verification.
As a concrete example, it describes a NFSv3
Tee and reports on its use in identifying interesting differences
in several production NFS servers and in debugging
a prototype NFS server. These experiences confirm
that comparison-based server verification can be a useful
tool for server implementors.
Debugging servers is tough. Although the client-server
interface is usually documented in a specification, there
are often vague or unspecified aspects. Isolating specification
interpretation flaws in request processing and in
responses can be a painful activity. Worse, a server that
works with one type of client may not work with another,
and testing with all possible clients is not easy.
The most common testing practices are RPC-level test
suites and benchmarking with one or more clients. With
enough effort, one can construct a suite of tests that exercises
each RPC in a variety of cases and verifies that each
response conforms to what the specification dictates.
This is a very useful approach, though time-consuming
to develop and difficult to perfect in the face of specification
vagueness. Popular benchmark programs, such as
SPEC SFS  for NFS servers, are often used to stresstest
servers and verify that they work for the clients used
in the benchmark runs.
This paper proposes an additional tool for server testing:
comparison-based server verification. The idea is sim-
Currently works for VMware.
Figure 1: Using a server Tee for comparison-based verification.
The server Tee is interposed between unmodified clients and the unmodified
reference server, relaying requests and responses between
them. The Tee also sends the same requests to the system-under-test
and compares the responses to those from the reference server. With
the exception of performance interference, this latter activity should be
invisible to the clients.
ple: each request is sent to both the system-under-test
(SUT) and a reference server, and the two responses are
compared. This can even be done in a live environment
with real clients to produce scenarios that artificial test
suites may miss. The reference server is chosen based on
the belief that it is a valid implementation of the relevant
interface specification. For example, it might be a server
that has been used for some time by many user communities.
The reference server thus becomes a “gold standard”
against which the SUT’s conformity can be evaluated.
Given a good reference server, comparison-based
server verification can assist with debugging infrequent
problems, achieving “bug compatibility,” and isolating
This paper specifically develops the concept of
comparison-based verification of file servers via use of
a file server Tee (See Figure 1).1 A file server Tee interposes
on communication between clients and the reference
server. The Tee automatically sets and maintains
SUT state (i.e., directories, files, etc.) to match the reference
server’s state, forwards client requests to the reference
server, duplicates client requests for the SUT, and
compares the two responses for each request. Only the
reference server’s responses are sent to clients, which
1The name, “server Tee,” was inspired by the UNIX tee command,
which reads data from standard input and writes it to both standard
output and one or more output files.