Comparison-based File Server Verification Yuen-Lin Tan, Terrence Wong, John D. Strunk, Gregory R. Ganger Carnegie Mellon University Abstract Comparison-based server verification involves testing a server by comparing its responses to those of a reference server. An intermediary, called a “server Tee,” interposes between clients and the reference server, synchronizes the system-under-test (SUT) to match the reference server’s state, duplicates each request for the SUT, and compares each pair of responses to identify any discrepancies. The result is a detailed view into any differences in how the SUT satisfies the client-server protocol specification, which can be invaluable in debugging servers, achieving bug compatibility, and isolating performance differences. This paper introduces, develops, and illustrates the use of comparison-based server verification. As a concrete example, it describes a NFSv3 Tee and reports on its use in identifying interesting differences in several production NFS servers and in debugging a prototype NFS server. These experiences confirm that comparison-based server verification can be a useful tool for server implementors. 1 Introduction Debugging servers is tough. Although the client-server interface is usually documented in a specification, there are often vague or unspecified aspects. Isolating specification interpretation flaws in request processing and in responses can be a painful activity. Worse, a server that works with one type of client may not work with another, and testing with all possible clients is not easy. The most common testing practices are RPC-level test suites and benchmarking with one or more clients. With enough effort, one can construct a suite of tests that exercises each RPC in a variety of cases and verifies that each response conforms to what the specification dictates. This is a very useful approach, though time-consuming to develop and difficult to perfect in the face of specification vagueness. Popular benchmark programs, such as SPEC SFS [15] for NFS servers, are often used to stresstest servers and verify that they work for the clients used in the benchmark runs. This paper proposes an additional tool for server testing: comparison-based server verification. The idea is sim- Currently works for VMware. Unmodified Clients SUT Reference Server Server Tee RPC Requests Responses Figure 1: Using a server Tee for comparison-based verification. The server Tee is interposed between unmodified clients and the unmodified reference server, relaying requests and responses between them. The Tee also sends the same requests to the system-under-test and compares the responses to those from the reference server. With the exception of performance interference, this latter activity should be invisible to the clients. ple: each request is sent to both the system-under-test (SUT) and a reference server, and the two responses are compared. This can even be done in a live environment with real clients to produce scenarios that artificial test suites may miss. The reference server is chosen based on the belief that it is a valid implementation of the relevant interface specification. For example, it might be a server that has been used for some time by many user communities. The reference server thus becomes a “gold standard” against which the SUT’s conformity can be evaluated. Given a good reference server, comparison-based server verification can assist with debugging infrequent problems, achieving “bug compatibility,” and isolating performance differences. This paper specifically develops the concept of comparison-based verification of file servers via use of a file server Tee (See Figure 1).1 A file server Tee interposes on communication between clients and the reference server. The Tee automatically sets and maintains SUT state (i.e., directories, files, etc.) to match the reference server’s state, forwards client requests to the reference server, duplicates client requests for the SUT, and compares the two responses for each request. Only the reference server’s responses are sent to clients, which 1The name, “server Tee,” was inspired by the UNIX tee command, which reads data from standard input and writes it to both standard output and one or more output files. |