next up previous
Next: Still challenging to test Up: Automatic Regression testing of Previous: eXtreme Programming

How to test with physical hardware

Figure 1: Basic Physical Network configuration
\includegraphics[height=4in,width=5in]{testsetup1.eps}

The basic network is shown in Figure 1. The taxonomy for our test setup is that the Sun rises in the east and the sun sets in the west. Thus one can easily remember where each host is.

EAST and WEST, shown with firewall icons, are FreeS/WAN IPsec gateway boxes.

SUNRISE and SUNSET are just ordinary hosts whose traffic will be protected by their respective gateways.

The machine SKY is used to do network analysis (``sniff''). There are frequently problems that occur when trying examine the traffic produced by a machine itself, so a seperate machine to make unbiased observations if necessary.2

The two gateway boxes are not directly attached, but rather are connected via a router. There are two reasons for this:

This setup is very representative of the typically deployed scenario for FreeS/WAN systems in a VPN. It does not cover every single situation - most of the most difficult to reproduce bugs have occured in other setups. More machines are needed to create such setups.

Aside from the space and cost involved in providing each developer with six machines (it is often the case that sky is the developers desktop), there are a number of other factors that make this difficult.

The major problem is maintaining this setup. There are many machines with many files that must be maintained. The systems must be kept up-to-date so that the latest kernels can be tested, yet at the same time, testing against older kernels is necessary. Different distributions need to be tested. The combinatorics are quite high.

The other major problem is work environment. Sitting in a room with six computers is a lot of noise. Getting access to each systems' console is difficult (one can not rely upon network logins!). If a monitor is attached to each system (vs a monitor switch), then the developer probably gets too much exersize.

One answer to this is serial consoles. See figure 2. Terminals attached to serial ports was the primary way that people used Unix until the advent of the X-terminal, and Linux continues this grand tradition.

Figure 2: Basic Network with console access
\includegraphics[height=4in,width=5in]{testsetup2.eps}

One simply puts the following in /etc/lilo.conf:

serial=0,38400n

...
image=/boot/vmlinuz-2.4.18-6mdk
        label=linux2418
        root=/dev/hda1
        append=" devfs=mount console=ttyS0,38400 console=tty0"
        read-only

The console then appears on both ``COM1'' and on the VGA screen. In this situation, the machines may be located in another room, connected to a console server. One logins from one's (quiet) desktop to the console server, accessing each machine via a serial port. Serial interfacesd are readily available with either PCI or USB interfaces. This makes building a 6 port console server rather easy.

The developer now has ready access to each machine, can reboot each machine, select different kernels and can configure it without even having networking on. In addition, kernel panics (``kernel oops'') or other strange output on the console can be cut and pasted into emails, etc..



Subsections
next up previous
Next: Still challenging to test Up: Automatic Regression testing of Previous: eXtreme Programming
Michael Richardson
2002-06-26