Performance Testing – Tools of the Trade

In the previous article we covered a brief load test scenario. It involved running load against a webserver and driving the server to failure to determine its capacity. What I did not discuss were the specific tools that could have been used during the testing period.

Client Side:

Wget – A non-interactive command line tool to fetch https, http, and ftp items. It can easily be run from inside a script in a tight loop or used in recursive/mirror mode for crawling. It can be found at http://www.gnu.org/software/wget/.
cURL- Another non-interactive command line tool. Much more extensive in its protocol support. Available from http://curl.haxx.se/. This tool was designed to fetch single URLs. It requires integration into a script to mirror/copy websites if such a behavior is desired.

The server side has several options depending on the OS being run. There are a few commands that are present on all OSes by default, and then there are OS specific commands as well.

Server Side:

Iostat provides thorough IO statstics for the machine. The command is native on Solaris as well as linux with the installation of the sysstat package. After watching the output run over time you can tell if the disk is lightly used, overloaded, or somewhere in-between. You can also tell if the drives are overwhelmed with many small transactions or content with large streaming writes. This is a good all round tool to get familiar with.

Next in line in the *stat family is vmstat. While iostat gives us a view into the disk subsystem, vmstat provides similar insight into the memory subsystem on the host. The command is natively present on Solaris and Linux. Vmstat is capable of showing you statistics on swap, memory paging characteristics, memory fault characteristics, and some additional information about run queues on the machine and CPU utilization. This is another good general tool to be familiar with.

Top is the next tool I used, and is probably the single most popular tool for getting an at-a-glance view to what the system is doing. It is a curses based application that periodically updates itself and constantly shows the top 20ish running items sorted by CPU utilization. While present in Linux, the application is not in the default Solaris installation. www.sunfreeware.com offers the package for download. Top typically requires minimal skill to interpret its output which makes it a good first line tool to see why a machine is behaving oddly.

Solaris offers two additional tools for observing system behavior: prstat and mpstat [edit: Seems Linux offers mpstat as well. It contains similar information to the Solaris version].

Prstat is very similar to top and shows the top 20 or so processes by CPU utilization but through various command line options, can provide insight into threads and offer microstate accounting information as well.

Mpstat is more in line with the vm/iostat line of commands and shows per cpu, or processor set, statistics such as context switches, systems calls, etc. The command can be used to determine if an application is thrashing the cpus in the machine or generally what the machine’s CPUs are doing.

Netstat is a utility which allows the user to see what the network stack on a machine is doing. It is useful for looking at the number of open sessions as well as the number of sessions in TIME_WAIT.

Finally, the sar command is also available to see what a given host is doing. The command is natively present on both linux and solaris. I personally tend not to use that as I can generally determine what I need from watching the *stat commands .

There is one additional tool which I use on Solaris and OSX, and that is dtrace. Dtrace is capable of instrumenting virtually anything on the fly and can provide amazing insight into what an application, or even the host, is doing at that moment without resulting to the instrumentation of binaries or adding typical splatterprint debug code to the binaries.

Essentially, I use the commands in approximately this order: top/prstat first to get a general overview of what the machine is doing. For more insight I open additional sessions on the machine and run iostat, vmstat, netstat, and possibly mpstat if available. Finally if I’m still unable to determine what the host is doing, and the server is running Solaris or OSX, I’ll use dtrace to look into the kernel or at the application for insight into whatever problem I’m chasing.

Whichever commands you decide to use, and whatever order you run them in, keep in mind one important point. Observing an experiment can affect the output of said experiment. This means that running all of those tools on a very burdened server may be enough to push the machine over the edge of unresponsiveness.

This concludes our brief introduction to the tools of the trade. If there is a demand and time permits I intend to write articles on the best practices for each of the commands.

If you like what you’ve read, please share the blog with others. If you have any questions or comments, feel free to send me email at kmajer at karlmajer dot com.