OSX 10.6.2 + SCR3110 CAC reader + new GX4 CAC card == No love on OSX. Keychain sees the card as empty.

However, 10.6.2 + VMWare + win7-64 + reader + CAC card + IE works just fine without any of the add-on software used on XP. You’ll need to pass the reader through to the VM by clicking on the little USB icons in the bottom right.

On another note, a Verisign EAC Certificate loaded in your keychain will cause codesign to hang for 8-10 minutes while it asks oscpd to validate the cert. This also happens when you use Keychain Access to go try and figure out why its taking so long to sign things. Work around it by either dropping your network when you need to sign things, or more permanently, drop your network and then use Keychain Access to remove the cert altogether. Save yourself the pain and load the EAC cert directly into firefox and use that browser to access the EAC enabled sites.

And finally, if you have the reader plugged in with a card in it and try to sign an iPhone application you will probably get the error: CSSMERR_DL_MISSING_VALUE. Keychain Access on 10.6.2 recognizes the reader and if the card is plugged in, Keychain Access seems to want to try and use it for signing. Take the card out of the reader and try again.

>>> Karl

Client Side:

Wget – A non-interactive command line tool to fetch https, http, and ftp items. It can easily be run from inside a script in a tight loop or used in recursive/mirror mode for crawling. It can be found at http://www.gnu.org/software/wget/.
cURL- Another non-interactive command line tool. Much more extensive in its protocol support. Available from http://curl.haxx.se/. This tool was designed to fetch single URLs. It requires integration into a script to mirror/copy websites if such a behavior is desired.

The server side has several options depending on the OS being run. There are a few commands that are present on all OSes by default, and then there are OS specific commands as well.

Server Side:

Iostat provides thorough IO statstics for the machine. The command is native on Solaris as well as linux with the installation of the sysstat package. After watching the output run over time you can tell if the disk is lightly used, overloaded, or somewhere in-between. You can also tell if the drives are overwhelmed with many small transactions or content with large streaming writes. This is a good all round tool to get familiar with.

Next in line in the *stat family is vmstat. While iostat gives us a view into the disk subsystem, vmstat provides similar insight into the memory subsystem on the host. The command is natively present on Solaris and Linux. Vmstat is capable of showing you statistics on swap, memory paging characteristics, memory fault characteristics, and some additional information about run queues on the machine and CPU utilization. This is another good general tool to be familiar with.

Top is the next tool I used, and is probably the single most popular tool for getting an at-a-glance view to what the system is doing. It is a curses based application that periodically updates itself and constantly shows the top 20ish running items sorted by CPU utilization. While present in Linux, the application is not in the default Solaris installation. www.sunfreeware.com offers the package for download. Top typically requires minimal skill to interpret its output which makes it a good first line tool to see why a machine is behaving oddly.

Solaris offers two additional tools for observing system behavior: prstat and mpstat [edit: Seems Linux offers mpstat as well. It contains similar information to the Solaris version].

Prstat is very similar to top and shows the top 20 or so processes by CPU utilization but through various command line options, can provide insight into threads and offer microstate accounting information as well.

Mpstat is more in line with the vm/iostat line of commands and shows per cpu, or processor set, statistics such as context switches, systems calls, etc. The command can be used to determine if an application is thrashing the cpus in the machine or generally what the machine’s CPUs are doing.

Netstat is a utility which allows the user to see what the network stack on a machine is doing. It is useful for looking at the number of open sessions as well as the number of sessions in TIME_WAIT.

Finally, the sar command is also available to see what a given host is doing. The command is natively present on both linux and solaris. I personally tend not to use that as I can generally determine what I need from watching the *stat commands .

There is one additional tool which I use on Solaris and OSX, and that is dtrace. Dtrace is capable of instrumenting virtually anything on the fly and can provide amazing insight into what an application, or even the host, is doing at that moment without resulting to the instrumentation of binaries or adding typical splatterprint debug code to the binaries.

Essentially, I use the commands in approximately this order: top/prstat first to get a general overview of what the machine is doing. For more insight I open additional sessions on the machine and run iostat, vmstat, netstat, and possibly mpstat if available. Finally if I’m still unable to determine what the host is doing, and the server is running Solaris or OSX, I’ll use dtrace to look into the kernel or at the application for insight into whatever problem I’m chasing.

Whichever commands you decide to use, and whatever order you run them in, keep in mind one important point. Observing an experiment can affect the output of said experiment. This means that running all of those tools on a very burdened server may be enough to push the machine over the edge of unresponsiveness.

This concludes our brief introduction to the tools of the trade. If there is a demand and time permits I intend to write articles on the best practices for each of the commands.

If you like what you’ve read, please share the blog with others. If you have any questions or comments, feel free to send me email at kmajer at karlmajer dot com.

There is nothing stopping a tester from performing their load test using a browser or other GUI based load generation tool. Providing the tool has some manner of minimal controllability and repeatability, feel free to use any tool you so desire. Should you choose to use a browser, such as Firefox, be sure to recognize that the plugins are capable of altering the behavior and characteristics of the browser considerably and all future testing would need to be done using the same exact browser, machine, and plugin combination for the numbers to be comparable.

Essentially, I choose to use a CLI tool, such as wget, because it behaves the same every time and I can wrap the application in a shell script to guarantee both the same behavior, and instrument things further.

Second, the first cut performance numbers. When trying to rapidly determine a speed of light, its more important to get to the right magnitude first and then refine than it is to try and get the exact answer on the first pass.

For example, we know that we cannot move more packets across the network than we can fit through a single network interface. Therefore, if the size of the interface is N bytes per second, regardless of what we do, we will never be able to push more than N+1 total bytes through the interface. Similarly, if we know that we have M sized pages, then we will never be able to push N/M pages through the interface at a single point in time.

What this does for us is provide an upper bound for our testing. If our results indicated that we were able to do N/M + 5302 transactions per second, we know know that something was wrong with our calculation as those last 5302 operations/second would simply not fit through the pipe. However, if our results indicated 5302, and N/M is 8192, then we know that our result is a reasonable number.

It is important to obtain the speed of light at the start and then refine using the bounding boxes we know to be true. This ideal holds whether testing network, disk I/O, cpu, etc. If we know what the fastest possible speed of a given component is, then we know that if our results report numbers faster than what we know to be possible, then we must question the results and find them to be true because of another reason, or discard them and start over.

And a few thoughts just to provide a bit of context as to when speed of light may be false. When testing disk I/O, if the file size is sufficiently small, the file will be loaded directly into the buffer cache. This is a section of machine memory dedicated to buffering all disk I/O transactions and provides access times and speeds comparable to system memory and not the disk subsystem. Ie, the speed of light for the buffer cache is based off of a 50ns access time and not an 80ms access time.

I’d like to thank the reader that brought the questions to me and invite others to comment or email me with any questions they may have. I can be reached at kmajer at karlmajer dot com.

>>> Karl

The developers you support as a systems administrator are considering moving the static content for the company website www.somefamoussite.com to its own webserver to free up resources for the dynamic page generation and generally speed things up. They are curious how fast an apache serving only static content will be able to serve requests.

In order to accomplish this task, we must examine the steps of the scientific method and see how each plays its part in providing us a sound and thorough roadmap to provide the developers with their answer.

1. First, define the question: “How fast can an apache webserver go serving static content.”
2. Next gather information and resources:

It would benefit us to know the number of static items and the size of them to determine how best to answer the question, so we ask development that question and are handed a tarball containing 256 images with an average size of 6k each.

Since we setup the hardware that is being used, we know that the servers have gigabit ethernet cards in them. We also expand the tarball into the tree on our webserver and use a find to create a logfile full of relative links to use to fetch the static content.

find . -type f | awk ‘{print “/path/to/images/$1”}’ > logfile.out
cat logfile.out logfile.out logfile.out logfile.out > logfile.1k
cat logfile.1k logfile.1k logfile.1k logfile.1k > logfile.4k
cat logfile.4k logfile.4k logfile.4k logfile.4k > logfile.16k

3. Form the hypothesis:

Based on an average size of 6KBytes, and knowing that the hardware has gigabit ethernet, we can compute that in lab conditions with perfect network, the machines can do no more than approximately 21,845 requests/second.

( (1Gbit/sec == 128 MBytes/sec) / 6KBytes avg size == 21,845.3 objects/sec)

Our hypothesis, therefore, based solely on the network capacity of the hardware, is “A server can do no more than 21,845 operations/second.”

4. Perform the experiment(s) and collect data:

You’ll want to run top on the webserver to get a rough idea of how much free cpu there is.

Copy the logfile.16k to each of the load generators. In this example there will be 4 load generators.

Use wget on one of the load generators to mark the logfile with something we can search for later.

wget http://www.myserver.com/images/TESTSTARTEDHERE

Use wget on each load generator to fetch the images

wget -i logfile.16k -o wget.out

Fire off all four wget’s at the same time and let it run.

Watch top running on the webserver and keep rough track of our idle cpu. Time passes and the load generators will eventually run out of logs, probably within a few minutes.

Use wget to mark the logfile again.

wget http://www.myserver.com/images/TETENDEDHERE

5. Analyze the data:

With each load generator having a 16k logfile, we had the potential load capability of 64k instantaneous requests. This is unlikely, however, as there is a certain amount of overhead between requests that must be accounted for. A reasonable assumption would be that each generator could generate close to 8k instantaneous requests, the four of which still totaling over the ~22k maximum of the network.

Using the logfile from the apache we can determine how much traffic we received.

First use sed or perl or your language of choice and extract the logfile.

sed -n /TESTSTARTEDHERE/,/TESTENDEDHERE/p access.log > test.log

Next determine the starting time by looking at the next line after STARTED line in the test.log and looking at the timestamp on the line

head -2 test.log

Do the same for the ending time by looking at the second to last line of test.log.

tail -2 test.log

Determine the total test time by subtracting the two times from each other.

Determine the total number of lines in the logfile and removing 2 for the header and footers.

wc -l test.log

This will likely be 64k unless you interrupted the test prematurely.

Now extract the successful image retrievals using grep.

grep “HTTP/…. 200” test.log > 200.log

Count the number of successful requests

wc -l 200.log

Now compute the average request speed

Good Requests / Total Time in minutes == Average Good Requests/Minute

Next, determine the number of requests per unit. Typically a 5 minute unit works best but for simplicity we will use a 1 minute unit.
Extract column 4 (the timestamp in a CLF apache log) from the 200.log file

awk ‘{print $4}’ 200.log > timestamp.out

Truncate the seconds from the logfile using either cut or awk

cut -d: -f 1,2,3 timestamp.out > trunctime.out

Uniq and count the truncated timestampts to get the number occuring during each minute:

uniq -c trunctime.out > counttime.out

Now reverse the two columns to make the graphic easier and add a closing bracket.

awk ‘{print $2”] ”$1}’ counttime.out > graphdata.out

It is now possible to look through the logfile at this time and see a rough estimate of how the webserver did, however it is more valuable if we can graph this data and examine it visually.

Import the data into excel, pages, or use gnuplot on the command line and plot the graph using a line graph.



The graph above was manufactured to illustrate the desired point. Note that the middle of the graph plateaus around 8500 requests/second. The flatness of the graph suggests that we’ve hit a bottleneck of some point. Since we know that the network is capable of nearly 22k request/second, and the network on each load client is presumably similar, we know that we’re either hitting the limitations of the disk subsystem or that we’ve pushed the webserver out of CPU.

If, during the test, you saw the idle CPU approach single digits, then we have reasonable confidence that we pushed the machine to its limit of CPU otherwise we may be pushing the limits of I/O.

6. Interpret the data and draw conclusions:

Now, by using the graph, we can decide on a limit for the webserver. The flat line starts approximately around 8500 req/sec. A reasonable buffer is 10% of that number, and so we would say that the max capacity of the webserver is 7650 req/sec. If you wish to be more conservative, and you should, you could leave yourself 25% capacity and call it 6325.

As a general rule you want to leave sufficient capacity on the machines to handle any excess load from failed hosts. If you have 2 machines, each machine should be able to handle all of the load. If you have 3 machines, then each should be able to handle 66% of the total load, and so forth.

7. Publish Findings:

With this simplistic testing done, you could approach development and tell them that you have some confidence that based on the preliminary testing you’ve done, the webserver can do 6325 ops/sec. Additionally, you should then provide the developers with your step by step guide as to how to get their own numbers to both allow them to validate your work, and to enable them to do this level of testing on their own in the future.

This concludes our first example. There are several more to come.

If you like what you’ve read, please share the blog with others. If you have any questions or comments, feel free to send me email at kmajer at karlmajer dot com.

>>> Karl

As a working systems administrators, or programmer, there will be times in which you are called upon to determine the performance of a system, an application, or even an algorithm. While performance testing is frequently time consuming, finding the answer to the question “how fast is it” is neither arcane nor rocket science. All that is required is patience, sound and repeatable methodology, and use of the principles of the scientific method.

Information detailing the scientific method is available on multiple websites. For simplicity Wikipedia’s entries will be used. Wikipedia tells us that the Classical Model of the scientific method is:

• Characterization – Observations, definitions, measurements,
• Hypothesis – explanations of characterization, often theoretical/hypothetical
• Prediction – reasoning from the hypothesis
• Experiment – tests all of the above

The Hypothetico-Deductive model, perhaps more in line with the working professional, is:

• Use your experience.
• Conjecture an explanation.
• Deduce a prediction from that explanation.
• Test

And an even more modern interpretation of the classical model is:

• Define the question
• Gather information and resources
• Form hypothesis
• Perform experiment and collect data
• Analyze data
• Interpret data and draw conclusions that form starting point for next hypothesis
• Publish results
• Retest, often done by third parties

The modern definition would extend the classical model with post experiment work to analyzie, publish, and retest the work done.

Using this information as background, I will be writing a series of articles on understanding performance. These articles will include a few in depth examples to help performance testers understand not only the methods and techniques used, but also the depth and thoroughness required. In the examples, both the hypothetico-deductive model and the modern interpretation of the scientific method will be used.

If you have any examples or questions you’d like covered, feel free to drop me email at kmajer at thisdomainname.com.
>>> Karl

Configuring Intel Solaris is similar to Linux. The low level startup boot process is the same as Windows and Solaris, mainly the use of a PXE loader to start the process, however in order to keep the top level menuing consistent for the client, and to work around a dhcpd configuration issue, it was required to chain together Sun’s version of PXEgrub with PXELinux.

In both the Intel and Sparc cases, a request to boot is dropped on the wire, the dhcpd server accepts the request and issues a reply packet containing the IP address for the PXE client, the IP address of the next server to speak to, and the filename that the PXE client needs to request, still PXELinux for the first stage. For Sparc Solaris, the filename that the bootloader needs to request is not a PXE loader, but instead is the actual Solaris supplied boot file.

The operating system makes use of the PXE Loader configuration. Please go read that post if you have yet to do so as general configuration details are described in that article.

Please take note that a conscious design decision was made at this point to chain the bootloaders to present a mostly unified menu system to the end user. It is possible to create an Intel Solaris stanza in the dhcpd.conf and add the mac addresses of each host that would need to boot off of pxegrub instead of pxelinux. However, this would require the modification of the configuration and restarting of the dhcpd service for each new intel host added.

Since, in the environment, there were an exceptionally small number of Sparc hosts to be installed, and new Sparcs were added intermittently at best, it was considered more desirable to touch the dhcpd files only for the Sparcs rather than the often multiple installs/day required of the Intel machines. This would then allow the scheduling of dhcpd restarts during a maintenance window.

There was one modification made to the dhcpd.conf file which needs to be made regardless of the architecture. The following option statements need to be added to the file somewhere near the PXElinux lines that were added to support PXE booting.

option SUNW.root-mount-options code 1 = text;
option SUNW.root-server-ip-address code 2 = ip-address;
option SUNW.root-server-hostname code 3 = text;
option SUNW.root-path-name code 4 = text;
option SUNW.swap-server-ip-address code 5 = ip-address;
option SUNW.swap-file-path code 6 = text;
option SUNW.boot-file-path code 7 = text;
option SUNW.posix-timezone-string code 8 = text;
option SUNW.boot-read-size code 9 = unsigned integer 16;
option SUNW.install-server-ip-address code 10 = ip-address;
option SUNW.install-server-hostname code 11 = text;
option SUNW.install-path code 12 = text;
option SUNW.sysid-config-file-server code 13 = text;
option SUNW.JumpStart-server code 14 = text;
option SUNW.terminal-name code 15 = text;

These vendor specific lines assign human readable names to the various SUNW codes that are used to set various parameters such as the root path, the swap file, and the like.

The pxelinux menu entry needs to be changed to point to the location of the Sun PXEgrub file placed in the boot tree, and a menu.lst file will need to be created to support that bootloader. The single Solaris entry in my environment looks like this:

label solaris
kernel /boot/grub/pxegrub.0

In my boot directory you can find a grub directory containing the files pxegrub.0 and menu.lst. The menu.lst file is grub’s version of the default entry. It must always be named menu.lst either via the actual filename or through a tftp filename remap. After reading through the Solaris pxegrub source it seems highly likely that this filename cannot be changed via a vendor ID code in the dhcpd.conf file despite what the manpages say.

The three line Solaris 10 entry in my sample menu.lst looks like:

title Solaris 10 Interactive
kernel /images/SOL_10_U3_x86_64/boot/multiboot kernel/unix – install nowin dhcp -B install_media=172.16.32.16:/export/install/images/SOL_10_U3_x86_64
module /images/SOL_10_U3_x86_64/boot/x86.miniroot

The first line is the title as it will appear in the grub menu. The second describes the kernel file to load including all switches passed to the kernel. Given the strange wrapping, please take note that the line ends with the install_media stanza found in the SOL_10_U3_x86_64 directory. Finally, the final line specifies the module the kernel will need to load in order to run properly.

For an Intel installation, the prep work is completed and there remains only the image to prepare. For the sparc image, the dhcpd.conf file will need to be edited each time a new sparc host is added and the following stanza will need to be added and/or updated:

group {
vendor-option-space SUNW;
option SUNW.JumpStart-server “172.16.32.16:/export/JS/sol8/configs”;
option SUNW.install-server-hostname “gorgon”;
option SUNW.install-server-ip-address 172.16.32.16;
option SUNW.install-path “/export/install/images/SOL_9_U8_x86″;
option SUNW.root-server-hostname “gorgon”;
option SUNW.root-server-ip-address 172.16.32.16;
option SUNW.root-path-name “/export/install/images/SOL_9_U8_x86/Solaris_9/Tools/Boot”;
next-server 172.16.32.16;

host solclient01 {
hardware ethernet 00:0c:29:e3:e9:94;
fixed-address 172.16.33.69;
option host-name “solclient01″;
option SUNW.sysid-config-file-server=”172.16.32.16:/export/install/jumpstart”;
}
}

Each new sparc addition will require the addition of a new host substanza in this group. You’ll need to gather the mac address of the new host as well as assign it a static IP address. Once the configuration file has been updated, simply restart dhcpd and the configuration changes are complete. As a side note, installations were done using a small preallocated pool of statics outside the dhcpd range and then the host was reconfigured once the installation was complete.

Now that dhcpd has been setup, we can setup the images. Substitute x86_64 with SPARC as appropriate in the examples. Note that the solaris media comes on DVDs and so there is only a single image to copy.

mkdir /export/install/images/SOL_10_U3_x86_64 # or the path for the real vers.
mkdir -p /mnt/{1,2,3,4…N} # one per ISO
mount -o loop /path/to/iso/SOL_10_U3_x86.iso /mnt/1
…
mount -o loop /path/to/iso/SOL_10_U3_x86.iso /mnt/N
.
tar -C /mnt/1 cpvf – | tar -C /export/install/images/SOL_10_U3_x86_64/ -xpf -

When finished, in /export/install/images/SOL_10_U3_x86/ directory you should have the following directories: boot, Solaris_10, and a Copyright and installer file. Note that there is only a single ISO released from Sun. If the host is 64 bit capable the Solaris OS will boot in 64 bit mode. If the host is only 32 bit capable the OS will boot on its 32 bit kernel. The directory naming of _64 was chosen to remain consistent with the other directory names.

Next confirm that the boot/ directory contains the files you supplied paths to in the grub menu.lst, mainly multiboot and x86.miniroot:

ls -al /export/install/images/SOL_10_U3_x86_64/boot/
grep boot /export/install/boot/menu.lst

And that concludes all of the image prep necessary. The next item to configure is the apache server to ensure it can serve the http content from the /export/install mountpoint. To do so we need to edit the configuration of apache and add a section allowing the webserver to find our content at /images/. Edit the httpd.conf and add the following in an appropriate location, or use this snippet to create a new file uniquely named along the lines of install.conf in the httpd/conf.d directory, if present.

Alias /images/ /export/install/images/
< directory >
Options Indexes FollowSymLinks
AllowOverride Limit
Order allow,deny
Allow from all
< /directory >

Apache needs to then be restarted or HUP’ed for the configuration changes to take effect.

Unless the entire installation is being done over HTTP, Solaris also introduces the need for an NFS server for the image shares. How this is done will be dependent on your operating system, but assuming the nfs software is installed on the machine, in order to nfs export /export/install one needs to try one of the following:

Linux:
edit /etc/exports and add
/export/install *(ro)
service nfsd restart

Solaris:
edit /etc/dfs/dfstab and add
share -F nfs -o ro -d “Image Mounts” /export/install

All that is left now is to test the individual components:

First we need to make sure dhcpd is running:

ps -ef | grep dhcpd

Next we ensure apache is running:

ps -ef | grep httpd

Next ensure that nfsd is running:

ps -ef | grep nfs

Now we make sure that the repository is in proper order:

ls -al /export/install/boot/pxelinux # check for pxelinux
ls -al /export/install/boot/grub/pxegrub # check for pxegrub
ls -al /export/install/boot/grub/menu.lst # check for pxegrub’s menu file
ls -al /export/install/boot/default # check for the default menu file for your entry
ls -ald /export/install/images/SOL_10_U3_x86_64 # ensure the image is there
ls -ald /export/install/images/SOL_10_U3_x86_64/boot # should return 3 files: grub, multiboot, x86.miniroot

Now we make sure that tftp is working as expected (remember that tftp was chrooted to /export/install so everything will be relative to that path):

tftp localhost
cd images/SOL_10_U3_x86_64/boot/
get x86.miniroot

Next we ensure the apache is serving content properly.

wget http://localhost/images/SOL_10_U3_x86_64/boot/x86.miniroot

if, for some reason, you dont have wget on your server you can do it manually using telnet:

telnet localhost 80
HEAD /images/SOL_10_U3_x86_64/boot/x86.miniroot HTTP/1.0

Finally we test the NFS shares. Mount the filesystem on another machine if possible:

mount -o ro yourserver:/export/install /mnt
cd /mnt/images/SOL_10_U3_x86_64/boot
ls -al x86.miniroot

This essentially concludes the work required to net install a Solaris installation.

After finishing the configuration, all that is left is to reboot the machine and netboot using the F12 key, and then select the newly added menu entry from the pxelinux menu when prompted. Grub will then start and present its menu to you. From there select the entry you desire and then just wait until you reach the Solaris blue and green configuration screens.

Again, If you have any problems or questions, drop me a line and I’ll see what I can do to help. If there is interest in the topic, moving from an interactive installation to the configuring of jumpstart may be discussed in a later article should there be interest in the topic. If this article sounds familiar it is because I followed the guidelines I established in the Linux article. ;)

>>> Karl

Configuring Linux is nowhere near as complex as the Windows environment. The low level Linux boot process is the same as Windows and Solaris, mainly a PXE request is dropped onto the wire. The dhcpd server accepts the request and issues a reply packet containing the IP address for the PXE client, the IP address of the next server to speak to, and the filename that the PXE client needs to request. The filename passed to the client used is the pxelinux bootloader which has been described in detail elsewhere.

The operating system makes use of the PXE Loader configuration. Please go read that post if you have yet to do so as general configuration details are described in that article.

The pxelinux menu entry needs to be updated to point to the linux kernel and an appropriate initrd stanza needs to be added to the menu entry as well. Substitute your own directory name for all instances of RHEL_4_U5_x86_32. That said, a sample entry in my environment looks like this:

label rhel4_32
kernel /images/RHEL_4_U5_x86_32/install/images/pxeboot/vmlinuz
append initrd=/images/RHEL_4_U5_x86_32/install/images/pxeboot/initrd.img nofb text noipv6 method=http://install/images/RHEL_4_U5_x86_32/install ksdevice=eth0

Remember that the tftpd server is chrooted to /export/install, so on the filesystem, the kernel resides in the base directory /export/install/images/RHEL_4_U5_x86_32/. The nomenclature used at the customer site allows multiple variants of RHEL 4 32 and 64 bit to live in the tree concurrently. Menu entries need only be added for each desired variant to install other versions.

Once the PXE menu entries have been added, the image can be prepared. This is nowhere near as complex as windows and requires only a few steps:

mkdir /export/install/images/RHEL_4_U5_x86_32 # or the path for the real vers.
mkdir -p /mnt/{1,2,3,4…N} # one per ISO
mount -o loop /path/to/iso/RHEL4-U5-i386-AS-disc1.iso /mnt/1
…
mount -o loop /path/to/iso/RHEL4-U5-i386-AS-discN.iso /mnt/N
# note, if you run out of loopback devices, add “options loop max_loop=256”
# to /etc/modprobe.conf and reboot.
tar -C /mnt/1 cpvf – | tar -C /export/install/images/RHEL_4_U5_x86_32/ -xpf -
…
tar -C /mnt/N cpvf – | tar -C /export/install/images/RHEL_4_U5_x86_32/ -xpf -

When finished, in /export/install/images/RHEL_4_U5_x86_32/ directory you should have the following directories: RedHat, SRPMS, install, images, isolinux, and a large number of README html files.

Confirm that the install/images/pxeboot directory contains a file called vmlinuz and the path matches what was indicated in the pxelinux menu entry and that the initrd.img is present in the same directory and that the paths to the images match:

ls -al /export/install/images/RHEL_4_U5_x86_32/install/images/pxeboot
grep RHEL_4_U5_86_32 /export/install/default

And that concludes all of the image prep necessary. The next item to configure is the apache server to ensure it can serve the http content as requested by the method= line of the menu entry. To do so we need to edit the configuration of apache and add a section allowing the webserver to find our content at /images/. Edit the httpd.conf and add the following in an appropriate location, or use this snippet to create a new file uniquely named along the lines of install.conf in the httpd/conf.d directory, if present.

Alias /images/ /export/install/images/
< directory >
Options Indexes FollowSymLinks
AllowOverride Limit
Order allow,deny
Allow from all
< /directory >

Apache needs to then be restarted or HUP’ed for the configuration changes to take effect.

All that is left now is to test the individual components:

First we need to make sure dhcpd is running:

ps -ef | grep dhcpd

Next we ensure apache is running;

ps -ef | grep httpd

Now we make sure that the repository is in proper order:

ls -al /export/install/boot/pxelinux # check for pxelinux
ls -al /export/install/boot/default # check for the default menu file for your entry
ls -ald /export/install/images/RHEL_4_U5_x86_32/ # make sure the RedHat dir is there
ls -ald /export/install/images/RHEL_4_U5_x86_32/install/images/pxeboot # should return 4 files: initrd.img, README, TRANS.TBL, vmlinuz

Now we make sure that tftp is working as expected (remember that tftp was chrooted to /export/install so everything will be relative to that path):

tftp localhost
cd images/RHEL_4_U5_x86_32/install/images/pxeboot
get vmlinuz

Next we ensure the apache is serving content properly.

wget http://localhost/images/RHEL_4_U5_x86_32/install/images/pxeboot/vmlinuz

if for some reason you dont have wget on your server you can do it manually using telnet:

telnet localhost 80
HEAD /images/RHEL_4_U5_x86_32/install/images/pxeboot/vmlinuz HTTP/1.0

This essentially concludes the work required to netboot a RHEL installer to the point where the user can make their desired selections to complete the installation.

After finishing the configuration, all that is left is to reboot the machine and netboot using the F12 key, and then select the newly added menu entry from the pxelinux menu when prompted. When the installer loads and begins the interactive installation you will need to walk through the installer options and answer any question.

If you have any problems or questions, drop me a line and I’ll see what I can do to help. If there is interest in the topic, moving from an interactive installation to the configuring of kickstart may be discussed in a later article should there be interest in the topic.

>>> Karl

So, there are a few things left to do before we can attempt a network boot. Primarily we need to add/create the winnt.sif configuration file, update the binl daemon files and restart the daemon, and then we need to test the components to make sure that everything is in order.

An absolute minimal winnt.sif created by the addnewos.sh script looks like the following:

[data]
floppyless = "1"
msdosinitiated = "1"
; Needed for second stage
Orisrc="\\\\172.16.32.16\REMINST\\W50AA\i386"
OriTyp = "4"
LocalSourceOnCD = 1
;DisableAdminAccountOnDomainJoin = 1
[SetupData]
OsLoadOptions = "/fastdetect"
; Needed for first stage
SetupSourceDevice = "\Device\LanmanRedirector\172.16.32.16\REMINST\\W50AA"
[UserData]
ComputerName = *
FullName = *
OrgName = *
ProductID = *

Briefly, the [data] section specifies where the media will be coming from and what type it is. The [SetupData] stanza contains any arguments to be passed to the kernel, as well as establishing the source device from which setup is to pull the additional datafiles. The listed path corresponds to the samba share that was setup earlier.

Finally, the [UserData] section allows for the customization of the computer name, the user name, and the company name, as well as provide a slot to prefill the appropriate license key obtained from Microsoft.

This sif file will allow setup to execute in a fully interrogative mode, asking the user performing the install the same series of questions as if they’d used the CD media for the installation. There are quite a few options available for the sif file allowing for the full unattended installation of windows. If there is an interest I can expand on it in a future piece.

Once the w50aa.sif file is in place, we can go refresh the binlsrv.py database. Assuming the copies of binlsrv.py, infparser.py, and windirs.conf.sample were obtained from this site, this is accomplished by doing the following:

• edit windirs.conf and add the new directory name, i.e. W50AA
• run infparser.py
• run binlsrv.py

With the binlsrv.py script running, we can begin to do the testing on the system. Of course you could at this point simply reboot the box you wish to install, cross your fingers, and hit F12 to netboot while hoping for the best. Those of you willing to use some methodology in their testing, please read on. Please note: I’m assuming the path to your install tree is /export/install/ and you have the windows image in images/W50AA. You’ll need to adjust it for your environment otherwise.

First we need to make sure dhcpd is running:

ps -ef | grep dhcpd

Next we ensure binlsrv.py is running;

ps -ef | grep binl

Now we make sure that the repository is in proper order:

ls -al /export/install/pxelinux	   # check for pxelinux
ls -al /export/install/default     # check for the default menu file for pxelinux
ls -ald /export/install/images/W50AA/i386 # make sure the i386 dir is there
ls -ald /export/install/images/W50AA/w50aa* # we should get 3 files back, w50aa, w50aa.0, w50aa.sif.

Now we make sure that tftp is working as expected (remember that tftp was chrooted to /export/install so everything will be relative to that path):

tftp localhost
cd images/W50AA/boot
get w50aa.sif

Now we ensure that the rewriting rules for the tfpd map files worked properly

grep W50AA /var/log/messages  # on linux
grep W50AA /var/adm/messages  # on solaris

You should have seen several log lines spit out as a match to the grep statement. If not, there is the possibility that the mapping isn’t working. Try a tail -f on the logfile and run a few tftp fetches by hand looking for the matches.

At this point the only thing left to do is the actual boot of the computer. So, reboot the test machine, netboot it, and if all goes well you will be greeted by the pxelinux menu. From there pick the entry you created for the windows installation and you will soon be greeted by the windows setup program.

Hopefully there is enough in here to allow you to troubleshoot any issues which you may run into. If this is not the case, please post in the comments and I’ll do what I can to assist you.

>>> Karl

The configuration of the PXE environment requires several components:

• dhcpd server
• tftpd server
• PXE bootloader

Each of these items has customized configuration files which will be described below.

At the lowest level there is a dhcpd server listening for initial boot requests. ISC’s dhcpd server, specifically version 3.0.6 which was the current version at the time. On the production RHEL machine, version 3.0.5 from the RPM was used.

The full intricacies of configuring the dhcpd file can be found in the manpage, however at a minimum one needs to specify the following pxelinux entries and create a single subnet stanza for basic functionality.

# pxelinux settings
option space pxelinux;
option pxelinux.magic code 208 = string;
option pxelinux.configfile code 209 = text;
option pxelinux.pathprefix code 210 = text;
option pxelinux.reboottime code 211 = unsigned integer 32;

#and

# Servers subnet
subnet 172.16.32.0 netmask 255.255.254.0 {
authoritative;
option routers 172.16.32.1;
option subnet-mask 255.255.254.0;
option broadcast-address 172.16.33.255;
option domain-name “dhcp.karlmajer.com karlmajer.com”;
range dynamic-bootp 172.16.33.201 172.16.33.251;

ddns-domainname “dhcp.karlmajer.com”;
ddns-rev-domainname “dhcp.33.30.172.in-addr.arpa.”;

next-server 172.16.32.16;

filename “boot/pxelinux.0″;
site-option-space “pxelinux”;
option pxelinux.magic f1:00:74:7e;
option pxelinux.configfile “pxelinux.cfg”;
option pxelinux.pathprefix “/tftpboot/pxelinux/files/”;
option pxelinux.reboottime 30;
}

Several assumptions are made by that configuration snippet that will need updating:

1. *There is no other dhcpd server on the network*
2. The tftpd server is at 172.16.32.16
3. The default router is at 172.16.32.1
4. You’ve placed your pxelinux and configuration files in $TFTPROOT/boot directory
5. Your network spans 2 class Cs, i.e. the 255.255.254 (/23) netmask and 33.255 broadcast.

After making these changes in the dhcpd.conf file, start up the daemon using the method of your choice, i.e., service dhcpd start, svcadm enable dhcpd, or even simply ‘dhcpd’ from the command line.

At this point there should be a functioning dhcpd server on the network. The next thing to configure is the tftpd server. Tftp-hpa version 0.42 was chosen as the tftp server of choice as it supported the remapping of file paths in requests, and offered tcp wrappers as an added bonus. Edit the tftpd.map file, often sought by tftpd at /etc/tftpd.map, and add the following lines to it:

rgi ^pxelinux.0 /boot/pxelinux.0
rgi ^pxelinux.cfg/ /boot/
rgi ^boot/pxelinux.cfg/ /boot/
rgi ^boot// /

This sets up the location of the boot files relative to the tftpd root which is specified on startup. In the environment that was built, all provisioning related items were placed in /export/install and the tftpd daemon was chrooted to this path. Therefore the pxelinux related files were in /export/install/boot, the menu files were in /export/install/hostmenu.

Now the tftpd server can be run from either inetd or as a standalone. Given the low usage of the install system, it was decided that tftpd would run from xinetd on RHEL ignoring the startup/shutdown costs of the daemon which may be noticeable were the install frequency several installs per hour rather than week.

Configuration of the inetd process will differ depending on the flavor of unix employed. On RHEL one need simply add a service definition to the xinetd.d directory and the xinetd daemon will pick it up on the next invocation. A sample RHEL stanza looks as follows:

# default: off
# description: The tftp server serves files using the trivial file transfer \
# protocol. The tftp protocol is often used to boot diskless \
# workstations, download configuration files to network-aware printers, \
# and to start the installation process for some operating systems. service tftp
{
socket_type = dgram
protocol = udp
wait = yes
user = root
server = /usr/sbin/in.tftpd
server_args = -m /etc/tftpd.map -v -v -v -s /export/install -t 10
disable = no
per_source = 11
cps = 100 2
flags = IPv4
}

Note the explicit paths to the tftpd.map as well as the chroot to /export/install.

At this point we have a daemon to respond to our PXE requests, and we have a daemon to serve the content using the tftp protocol for the PXE requests. Now we need to supply the file to be served.

PXELinux from http://syslinux.zytor.com/pxe.php was used as the bootloader. The files were unbundled and placed into /export/install/boot. A menu file was created for pxelinux, the contents of which look like:

#Note, this menu starts the local disk, a memtest utility, and a Windows 2000 RIS #Installation.
default local
prompt 1
timeout 0
display /boot/install.msg
F1 /boot/install.msg

label local
localboot 0

label w50aa
kernel /images/W50AA/boot/w50aa.0

label memtest
kernel /boot/memtest
append -

For safety the default configuration chosen is to always favor the internal drives and only attempt to boot from the w50aa stanza on request. Now copy the menu containing the described data into the /export/install/boot/ (adjusted appropriately for your filesystem layout) and either name the file default, or name the file something else and symlink it to default.

At this point you should have the basic workings of a PXE loader environment. To test it first make sure that dhcpd is running:

ps -ef | grep dhcpd

Next use tftp to connect to localhost and request the file pxelinux:

tftp localhost
get pxelinux

Finally, reboot a host with a PXEboot capable network adapter and hit F12 to netboot.

Please let me know if you have any questions or comments.

>>> Karl