The following hack will start Chrome up in Incognito mode by default. If you want a “normal” browser, just open a new browser window. Note, this is a ‘I’m not afraid of the shell prompt, I like voiding warranties, and if my computer explodes I’m fine with it’ sort of modification.

In windows land you can do this by right clicking on the desktop icon and editing the properties to add a ‘-incognito’ at the end of the line. Things don’t quite work the same way in OSX.

Without further delay:

bash-3.2$ cd /Applications/Google\ Chrome.app/Contents/MacOS/
bash-3.2$ mv Google\ Chrome Google\ Chrome.real
bash-3.2$ echo '#!/bin/sh' > Google\ Chrome
bash-3.2$ echo 'exec "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome.real" -incognito' >> Google\ Chrome
bash-3.2$ chmod 775 Google\ Chrome


Click on the normal icon, if the app starts up it should be in Incognito mode. If it doesn’t start, fire up the console app and look at the last couple of entries, you may have a typo. If all else fails you can fall back with this:


bash-3.2$ mv Google\ Chrome.real Google\ Chrome


So what does this do? OSX .app bundles contain all the resources for the application. The binary is usually hidden in the MacOS directory. We’ve just created a shell wrapper for that which calls the real binary and passes it the -incognito switch, just like they do on Windows.

Like I said earlier, you have been warned. I’ve been using this for a while now without any problems but my setup is likely different than yours, and this definitely would not fall under “normal” usage for Chrome. Also, note that you will need to do this anytime you update your Chrome installation.

Good luck,

>>> Karl

Context is everything with this device. If you were hoping to replace a laptop or skip out on getting that netbook by buying an iPad, I’m afraid you’re in for a rude awakening. I have gotten in the habit of reading technical manuals on my laptop prior to going to bed at night. I wanted to simplify things and purchase something smaller and lighter to let me continue my light reading in the evenings. I contemplated buying a Kindle at first, but decided to hold off after Apple first made mention of their device.

With that point of view in mind, the device has been quite successful in meeting, if not exceeding, my expectations. I have iBooks, Kindle for iPad, and GoodReader for iPad installed on the iPad and between the three applications, there is little I cannot read. I do wish that the iTunes Bookstore carried technical books, but I can always pull down the kindle versions of those, or buy pdf versions and view them in GoodReader.

Just a quick kudos to the GoodReader guys. The app offers almost a dozen different ways to get data into it from your iPad. When I first tried it, I was having issues with the iTunes integrated sync, but then realized that the app has an integrated webserver that can be connected to as a network folder (I’m on a mac) and then it was literally just drag and drop. I’ve also have yet to find a pdf that it can’t open and read. Well done guys.

So, using the iPad as an ebook works splendidly. The screen doesn’t require ambient light like the kindle does, and the controls on the various readers let you adjust the brightness/contrast as needed. I’ve already chewed through a few books in my nightly habit of reading for at least an hour.

As to the bonuses, the iPad is just the right size to comfortably lay in bed and watch content from iTunes U. Some of the Stanford lectures are outstanding and are free as an added bonus. I have also found its just the right size to keep up with my RSS feeds that I tend to consume as well. And when I feel like relaxing, the free Netflix client (account required) streams a decent selection of movies right to me.

There are a few downsides. The iWorks suite that can be purchased is tolerable at best on the virtual keyboard. I’m used to Excel and so caught a double whammy with the mind shift to how Numbers does things, and not having a full keyboard for data entry. Keynote was actually the best of the three, I was able to throw together a quick presentation without too much effort, and Pages works fine for a quick note, but I wouldn’t want to try and take notes in a class or a meeting on the device. I can almost touch type on the virtual keyboard until I hit a need for a special character. Then you’re flipping to another keyboard screen to get access to those. They should really move the apostrophe to the first keyboard.

So as I said, based on what I intended to use the device for, the iPad exceeds my expectations. I know there are other people whining about not being able to code on it, and/or it still being limited because its running iPhoneOS instead of MacOSX (which would have absolutely rocked), but frankly, at the end of the day I’d rather read a book than stare at more code.

>>> Karl

OSX 10.6.2 + SCR3110 CAC reader + new GX4 CAC card == No love on OSX. Keychain sees the card as empty.

However, 10.6.2 + VMWare + win7-64 + reader + CAC card + IE works just fine without any of the add-on software used on XP. You’ll need to pass the reader through to the VM by clicking on the little USB icons in the bottom right.

On another note, a Verisign EAC Certificate loaded in your keychain will cause codesign to hang for 8-10 minutes while it asks oscpd to validate the cert. This also happens when you use Keychain Access to go try and figure out why its taking so long to sign things. Work around it by either dropping your network when you need to sign things, or more permanently, drop your network and then use Keychain Access to remove the cert altogether. Save yourself the pain and load the EAC cert directly into firefox and use that browser to access the EAC enabled sites.

And finally, if you have the reader plugged in with a card in it and try to sign an iPhone application you will probably get the error: CSSMERR_DL_MISSING_VALUE. Keychain Access on 10.6.2 recognizes the reader and if the card is plugged in, Keychain Access seems to want to try and use it for signing. Take the card out of the reader and try again.

>>> Karl

• Always read the entire recipe first
• Always read the entire recipe first
• Dropping eggs on carpets tends to be messy
• Go fetch everything the recipe calls for before starting
• Timing, timing, timing …

I altered the recipe a bit based on what I had available, here is what I used:

Ingredients
• 1/2 of elbow macaroni
• 3 tbs butter + 3 tbs butter
• 3 tbs flour
• 1 tbs powdered mustard
• 3 cups 2% milk
• 1/2 small yellow onion, finely diced
• 1/4 teaspoon paprika
• 1 medium egg
• 5 oz mild yellow cheddar
• 5 oz colby
• 5 oz monterey jack
• salt/pepper to taste
• 1 cup panko breadcrumbs

Prep

Preheat oven to 350 degrees F.

To save you a little pain, do all this first…
Shred the cheeses if they aren’t already and mix together. Dice the onion and set aside. Mix the flour and mustard together and set aside. Lightly whip egg in small bowl and set aside. Premeasure the 3 cups of milk and set aside. Measure out the butter into 2 containers and let come to room temperature.

Directions

First, realize the following things which affect timing:

• Al dente elbows takes roughly 6-7 minutes
• Starting the sauce to simmer below will be roughly 15 minutes before mixing in the cheese
• melting in 10oz of the cheese while stirring and keeping everything smooth took about 3-4 minutes
• Prepping the panko took 2 minutes as my butter was straight from the fridge

So, In a large pot of boiling, salted water cook the pasta to al dente, pull from stove and drain completely when the sauce is at the simmering stage.

While the pasta is cooking, in a separate pot, melt the butter. Whisk in the flour and mustard and keep it moving for about five minutes. Make sure it’s free of lumps. Stir in the milk, onion, and paprika. Simmer for ten minutes and temper in the egg. Stir in 3/4 of the cheese. Season with salt and pepper. Fold the macaroni into the mix and pour into a 2-quart casserole dish. Top with remaining cheese.

Melt the butter in a saute pan and toss the bread crumbs to coat. Top the macaroni with the bread crumbs. Bake for 30 minutes. Remove from oven and rest for five minutes before serving.

The end result was happily eaten by my picky 3 year old who was happy to be served Mac n Cheese for dinner.

Original Recipe here: link

Karl’s Chicken Soup – Makes roughly 4 bowls.

Ingredients

• Half a chicken. Follow the 55 cloves and a chicken recipe from Jan 2nd.
• One large white onion
• 2 large carrots
• 1 large stalk of celery
• 16oz of white wine
• 16oz of water
• 16oz of chicken stock
• salt/pepper to taste
• 1/2lb of pasta (Farfalle) if desired
• cheesecloth

Directions

A traditional Mirepoix has a 2:1:1 ratio of onions, carrots, and celery. While I think they carrots and celery contribute to the taste of the final product, I’m not a big fan of those vegetables cooked so I tend to skimp on them. I dice the onion but the carrots and celery I purposely rough chop so its easy to fish back out of my bowl when I want to eat around them.

Remove all of the meat from the chicken bones. Put the meat back in the fridge and wrap the bones tightly in cheesecloth. I find this keeps the bones in one place and makes them easier to remove as a whole later. Fishing chicken bones out of individual portions of soup is not fun.

Take all the gelled fats/greases from the bottom of the chicken bowl and toss it in a large stockpot. Turn on high heat until it has all degelled and then add the onions and reduce the heat. If you tossed the fat/gell add a few swirls of olive oil to the pot and cook the onions in that.

Once the onions are a nice translucent color, add the carrots and celery. Cook for a few more minutes and then add the liquids to the pot. As to the wine, I use whatever white is currently open. I am always sure to use something that I would actually drink and not a cooking wine or anything ultra cheap. Add the chicken stock and water as well. Let this cook for a few minutes and then drop in the cheesecloth full of bones.

Bring the entire pot to a boil and then reduce to a simmer. Let it cook for as long as you’d like but pay attention to the amount of fluid left in the pot. If it gets too low, add some water. I tend to let it simmer for at least an hour, sometimes longer.

About 20 minutes before I plan on pulling it from the heat I add all of the chicken meat back in and let it warm up. If you planned on adding pasta, now is the time to start cooking that in a separate pot, otherwise just turn off the burner and cover the pot when the meat is warm.

Cook the pasta until its al dente. Al dente will be a moving target depending on the type and size of the pasta you chose, but for the Barilla brand Farfalle pasta I used most recently, it was about 7 minutes. Drain the pasta and pour directly into the soup pot. Let the soup simmer for another 10-15 minutes then just turn off the heat and cover.

Serve with fresh bread and top with Parmesan cheese if desired.

]]> http://www.karlmajer.com/2010/01/07/010410-karls-chicken-soup/feed/ 0 http://www.karlmajer.com/2010/01/06/new-for-2010-010210-40-cloves-and-a-chicken/ http://www.karlmajer.com/2010/01/06/new-for-2010-010210-40-cloves-and-a-chicken/#comments Wed, 06 Jan 2010 20:38:42 +0000 Karl Majer http://www.karlmajer.com/?p=98 So we’ve split up chores a bit more at home. I’m doing the cooking now on all odd days. I’ll be posting recipes and reviews of foods as I cook them.

Jan 2 2010

I love Good Eats on Food TV. Its one of the few shows that I watch with any regularity. The first recipe for the year is slightly modified from his episode guide.

40 Cloves and a Chicken.
Ingredients

• 1 whole chicken (broiler/fryer) cut into 8 pieces - I used an already quartered chicken from Harris Teeter.
• 1/2 cup plus 2 tablespoons olive oil
• 10 sprigs fresh thyme – I used 3, my wife isn’t a big fan
• 40 peeled cloves garlic – I used 55 or so, I am a big fan. I took me 20 minutes to peel it all…
• Salt / Pepper

Directions

Preheat oven to 350 degrees F. My Jenn-Airre oven seems to take about 25 minutes to preheat. It was the first time I’d ever baked anything in the oven that I was cooking (in 5 years…) and waiting for it to preheat threw off my meal time. Be warned or know your kitchen appliances

Season chicken with salt and pepper. Toss with a 2 tablespoons olive oil and brown on both sides in a wide fry pan or skillet over high heat. Remove from heat, add oil, thyme, and garlic cloves. Cover and bake for 1 1/2 hours.

Remove chicken from the oven, let rest for 5 to 10 minutes, carve, and serve. I let it rest for closer to 20 while I prepped dinner for my 3 year old. I also had some fresh bread from Harris Teeters to go with it.


“You’re cooking more often.”

Total prep time was close to 30 minutes, oven time was 90 minutes and then another 30 to rest. The first dinner of the year was a success. Half the chicken was consumed and I put everything else including all of the delicious oil/chicken drippings at the bottom of the pan into a tupperware container and saved it for the next meal.

Original Recipe Here: link

]]> http://www.karlmajer.com/2010/01/06/new-for-2010-010210-40-cloves-and-a-chicken/feed/ 0 http://www.karlmajer.com/2008/05/19/experiences-with-cfengine/ http://www.karlmajer.com/2008/05/19/experiences-with-cfengine/#comments Mon, 19 May 2008 12:45:19 +0000 Karl Majer http://www.karlmajer.com/?p=27 At a customer site I’ve had both the pleasure and pain of working with cfengine as a means to control the environment from a single touchpoint. The client decided to save the precious little time they had and to use readily available packages for the software instead of building the software from scratch. This resulted in mixed versions between the RedHat and Solaris machines with RedHat running the 2.2.x series code and Solaris on 2.1.x. What followed over the months were several exercises in frustration.
Cfengine was picked after an evaluation showed that it was capable of updating system files across machines and pushing files from the repository to the individual hosts. With the proof of concept working and with high hopes from the functionality referred to in the documentation, cfengine was rolled onto the network.
The first task was relatively straightforward; insure that the staff member’s logins were always present in the sudoers file. The site was using several group aliases to define various sudoers functions, which leaded to order dependencies in the file. The solution found was to delete all of the aliases and groups definitions and re-add them each pass. This was found to work until it was realized that this would happen from cron every 15 minutes. Not quite what was hoped for. The configuration was ultimately altered to explicitly look for a lack of team members before beginning any of the updates.
The next task was to distribute a dozen or so files from the central repository to all 250 or so machines. This worked almost perfectly and required only a bit of tuning on the MaxConnections, MaxCfengins, and Splay time directives. The current challenge with this task now is to ensure that all of the installed machines do the pull on an update as it seems that occasionally some small percentage of them, typically Solaris, refuse to pull the files without some manual cfrun intervention.
An Editfiles directive is provided to allow the editing of arbitrary files. Using this for the next task, editing the crontab on linux was straightforward and linux cron noticed the change immediately. On solaris this was not the case. Editing the files was the easy part. After editing the file the cron daemon needs to be made aware of the change, and so either the process needs to be restarted or the crontab command needs to be used to reread the file. Unfortunately, the Process directive proved to be little use on the older cfengine and the Shell directive was used to write custom inline shell scripts to handle the stopping and starting of cron.
The most recent change to the network involved manipulating the fstab on both OSes, removing an NFS mount, and then changing the former mount point from a directory to a symlink. There is an Unmount directive that would have worked beautifully had it worked on both OSes. It worked fine with Linux, unmounting the FS and deleting the entry and directory for us. Sadly, on Solaris it did nothing. To promote overall consistency it was opted to do manipulations on both operating systems in the same manner. An Editfiles stanza was used to manipulate the (v)fstabs, while a long chain of shell commands to ensure that the filesystem was unmounted, the fstabs were edited, and the symlink was created. Each component set a success or failure flag to enable the script to proceed to the next step or fail and alert as the case may be. This is probably the most complex script used at the customer site.
I’m unable to say for certain how much of the pain with the software is caused by the mixed versions being used on the network. It could be that if there was time to build and upgrade the software across the machines to the same version that the problems would magically disappear. It could also be that while powerful and seemingly capable, cfengine does things in a slightly different manner than a sysadmin, making the would-be cfengine administrator configuring it have to double check everything they write. (An example of this is the symlink directive which uses target->source whereas the ln command uses source->target.) Documentation and error reporting for the product was also a challenge, especially with the mixed version. The version specific source was often referred to understand what an error means or to determine why a specific section of code was being ignored. Cfengine will remain in place at the customer site for the foreseeable future, as despite its issues it is still better than touching each machine individually. Only time will tell if a complete upgrade to a consistent version will remove some of the pain or if this is simply the way of cfengine.
]]> http://www.karlmajer.com/2008/05/19/experiences-with-cfengine/feed/ 1 http://www.karlmajer.com/2008/02/18/performance-testing-tools-of-the-trade/ http://www.karlmajer.com/2008/02/18/performance-testing-tools-of-the-trade/#comments Tue, 19 Feb 2008 04:15:20 +0000 Karl Majer http://www.karlmajer.com/2008/02/18/performance-testing-tools-of-the-trade/ In the previous article we covered a brief load test scenario. It involved running load against a webserver and driving the server to failure to determine its capacity. What I did not discuss were the specific tools that could have been used during the testing period.

Client Side:

Wget – A non-interactive command line tool to fetch https, http, and ftp items. It can easily be run from inside a script in a tight loop or used in recursive/mirror mode for crawling. It can be found at http://www.gnu.org/software/wget/.
cURL- Another non-interactive command line tool. Much more extensive in its protocol support. Available from http://curl.haxx.se/. This tool was designed to fetch single URLs. It requires integration into a script to mirror/copy websites if such a behavior is desired.

The server side has several options depending on the OS being run. There are a few commands that are present on all OSes by default, and then there are OS specific commands as well.

Server Side:

Iostat provides thorough IO statstics for the machine. The command is native on Solaris as well as linux with the installation of the sysstat package. After watching the output run over time you can tell if the disk is lightly used, overloaded, or somewhere in-between. You can also tell if the drives are overwhelmed with many small transactions or content with large streaming writes. This is a good all round tool to get familiar with.

Next in line in the *stat family is vmstat. While iostat gives us a view into the disk subsystem, vmstat provides similar insight into the memory subsystem on the host. The command is natively present on Solaris and Linux. Vmstat is capable of showing you statistics on swap, memory paging characteristics, memory fault characteristics, and some additional information about run queues on the machine and CPU utilization. This is another good general tool to be familiar with.

Top is the next tool I used, and is probably the single most popular tool for getting an at-a-glance view to what the system is doing. It is a curses based application that periodically updates itself and constantly shows the top 20ish running items sorted by CPU utilization. While present in Linux, the application is not in the default Solaris installation. www.sunfreeware.com offers the package for download. Top typically requires minimal skill to interpret its output which makes it a good first line tool to see why a machine is behaving oddly.

Solaris offers two additional tools for observing system behavior: prstat and mpstat [edit: Seems Linux offers mpstat as well. It contains similar information to the Solaris version].

Prstat is very similar to top and shows the top 20 or so processes by CPU utilization but through various command line options, can provide insight into threads and offer microstate accounting information as well.

Mpstat is more in line with the vm/iostat line of commands and shows per cpu, or processor set, statistics such as context switches, systems calls, etc. The command can be used to determine if an application is thrashing the cpus in the machine or generally what the machine’s CPUs are doing.

Netstat is a utility which allows the user to see what the network stack on a machine is doing. It is useful for looking at the number of open sessions as well as the number of sessions in TIME_WAIT.

Finally, the sar command is also available to see what a given host is doing. The command is natively present on both linux and solaris. I personally tend not to use that as I can generally determine what I need from watching the *stat commands .

There is one additional tool which I use on Solaris and OSX, and that is dtrace. Dtrace is capable of instrumenting virtually anything on the fly and can provide amazing insight into what an application, or even the host, is doing at that moment without resulting to the instrumentation of binaries or adding typical splatterprint debug code to the binaries.

Essentially, I use the commands in approximately this order: top/prstat first to get a general overview of what the machine is doing. For more insight I open additional sessions on the machine and run iostat, vmstat, netstat, and possibly mpstat if available. Finally if I’m still unable to determine what the host is doing, and the server is running Solaris or OSX, I’ll use dtrace to look into the kernel or at the application for insight into whatever problem I’m chasing.

Whichever commands you decide to use, and whatever order you run them in, keep in mind one important point. Observing an experiment can affect the output of said experiment. This means that running all of those tools on a very burdened server may be enough to push the machine over the edge of unresponsiveness.

This concludes our brief introduction to the tools of the trade. If there is a demand and time permits I intend to write articles on the best practices for each of the commands.

If you like what you’ve read, please share the blog with others. If you have any questions or comments, feel free to send me email at kmajer at karlmajer dot com.

]]> http://www.karlmajer.com/2008/02/18/performance-testing-tools-of-the-trade/feed/ 1 http://www.karlmajer.com/2008/02/04/performance-testing-first-example-notes/ http://www.karlmajer.com/2008/02/04/performance-testing-first-example-notes/#comments Mon, 04 Feb 2008 12:05:41 +0000 Karl Majer http://www.karlmajer.com/2008/02/04/performance-testing-first-example-notes/ In our first example we did several things that I perhaps took for granted. A reader asked why I chose to use CLI load generation tools instead of surfing with a browser, and if the initial numbers generated are even reasonable. Allow me to address both points.

There is nothing stopping a tester from performing their load test using a browser or other GUI based load generation tool. Providing the tool has some manner of minimal controllability and repeatability, feel free to use any tool you so desire. Should you choose to use a browser, such as Firefox, be sure to recognize that the plugins are capable of altering the behavior and characteristics of the browser considerably and all future testing would need to be done using the same exact browser, machine, and plugin combination for the numbers to be comparable.

Essentially, I choose to use a CLI tool, such as wget, because it behaves the same every time and I can wrap the application in a shell script to guarantee both the same behavior, and instrument things further.

Second, the first cut performance numbers. When trying to rapidly determine a speed of light, its more important to get to the right magnitude first and then refine than it is to try and get the exact answer on the first pass.

For example, we know that we cannot move more packets across the network than we can fit through a single network interface. Therefore, if the size of the interface is N bytes per second, regardless of what we do, we will never be able to push more than N+1 total bytes through the interface. Similarly, if we know that we have M sized pages, then we will never be able to push N/M pages through the interface at a single point in time.

What this does for us is provide an upper bound for our testing. If our results indicated that we were able to do N/M + 5302 transactions per second, we know know that something was wrong with our calculation as those last 5302 operations/second would simply not fit through the pipe. However, if our results indicated 5302, and N/M is 8192, then we know that our result is a reasonable number.

It is important to obtain the speed of light at the start and then refine using the bounding boxes we know to be true. This ideal holds whether testing network, disk I/O, cpu, etc. If we know what the fastest possible speed of a given component is, then we know that if our results report numbers faster than what we know to be possible, then we must question the results and find them to be true because of another reason, or discard them and start over.

And a few thoughts just to provide a bit of context as to when speed of light may be false. When testing disk I/O, if the file size is sufficiently small, the file will be loaded directly into the buffer cache. This is a section of machine memory dedicated to buffering all disk I/O transactions and provides access times and speeds comparable to system memory and not the disk subsystem. Ie, the speed of light for the buffer cache is based off of a 50ns access time and not an 80ms access time.

I’d like to thank the reader that brought the questions to me and invite others to comment or email me with any questions they may have. I can be reached at kmajer at karlmajer dot com.

>>> Karl

]]> http://www.karlmajer.com/2008/02/04/performance-testing-first-example-notes/feed/ 0 http://www.karlmajer.com/2008/02/01/performance-testing-first-example/ http://www.karlmajer.com/2008/02/01/performance-testing-first-example/#comments Fri, 01 Feb 2008 23:30:25 +0000 Karl Majer http://www.karlmajer.com/2008/02/01/performance-testing-first-example/ This article is second in a series on performance and performance testing. The principles of the scientific method were discussed in the first article and now this article will detail a basic and slightly simplistic example of a performance task.

The developers you support as a systems administrator are considering moving the static content for the company website www.somefamoussite.com to its own webserver to free up resources for the dynamic page generation and generally speed things up. They are curious how fast an apache serving only static content will be able to serve requests.

In order to accomplish this task, we must examine the steps of the scientific method and see how each plays its part in providing us a sound and thorough roadmap to provide the developers with their answer.

1. First, define the question: “How fast can an apache webserver go serving static content.”
2. Next gather information and resources:

It would benefit us to know the number of static items and the size of them to determine how best to answer the question, so we ask development that question and are handed a tarball containing 256 images with an average size of 6k each.

Since we setup the hardware that is being used, we know that the servers have gigabit ethernet cards in them. We also expand the tarball into the tree on our webserver and use a find to create a logfile full of relative links to use to fetch the static content.

find . -type f | awk ‘{print “/path/to/images/$1”}’ > logfile.out
cat logfile.out logfile.out logfile.out logfile.out > logfile.1k
cat logfile.1k logfile.1k logfile.1k logfile.1k > logfile.4k
cat logfile.4k logfile.4k logfile.4k logfile.4k > logfile.16k

3. Form the hypothesis:

Based on an average size of 6KBytes, and knowing that the hardware has gigabit ethernet, we can compute that in lab conditions with perfect network, the machines can do no more than approximately 21,845 requests/second.

( (1Gbit/sec == 128 MBytes/sec) / 6KBytes avg size == 21,845.3 objects/sec)

Our hypothesis, therefore, based solely on the network capacity of the hardware, is “A server can do no more than 21,845 operations/second.”

4. Perform the experiment(s) and collect data:

You’ll want to run top on the webserver to get a rough idea of how much free cpu there is.

Copy the logfile.16k to each of the load generators. In this example there will be 4 load generators.

Use wget on one of the load generators to mark the logfile with something we can search for later.

wget http://www.myserver.com/images/TESTSTARTEDHERE

Use wget on each load generator to fetch the images

wget -i logfile.16k -o wget.out

Fire off all four wget’s at the same time and let it run.

Watch top running on the webserver and keep rough track of our idle cpu. Time passes and the load generators will eventually run out of logs, probably within a few minutes.

Use wget to mark the logfile again.

wget http://www.myserver.com/images/TETENDEDHERE

5. Analyze the data:

With each load generator having a 16k logfile, we had the potential load capability of 64k instantaneous requests. This is unlikely, however, as there is a certain amount of overhead between requests that must be accounted for. A reasonable assumption would be that each generator could generate close to 8k instantaneous requests, the four of which still totaling over the ~22k maximum of the network.

Using the logfile from the apache we can determine how much traffic we received.

First use sed or perl or your language of choice and extract the logfile.

sed -n /TESTSTARTEDHERE/,/TESTENDEDHERE/p access.log > test.log

Next determine the starting time by looking at the next line after STARTED line in the test.log and looking at the timestamp on the line

head -2 test.log

Do the same for the ending time by looking at the second to last line of test.log.

tail -2 test.log

Determine the total test time by subtracting the two times from each other.

Determine the total number of lines in the logfile and removing 2 for the header and footers.

wc -l test.log

This will likely be 64k unless you interrupted the test prematurely.

Now extract the successful image retrievals using grep.

grep “HTTP/…. 200” test.log > 200.log

Count the number of successful requests

wc -l 200.log

Now compute the average request speed

Good Requests / Total Time in minutes == Average Good Requests/Minute

Next, determine the number of requests per unit. Typically a 5 minute unit works best but for simplicity we will use a 1 minute unit.
Extract column 4 (the timestamp in a CLF apache log) from the 200.log file

awk ‘{print $4}’ 200.log > timestamp.out

Truncate the seconds from the logfile using either cut or awk

cut -d: -f 1,2,3 timestamp.out > trunctime.out

Uniq and count the truncated timestampts to get the number occuring during each minute:

uniq -c trunctime.out > counttime.out

Now reverse the two columns to make the graphic easier and add a closing bracket.

awk ‘{print $2”] ”$1}’ counttime.out > graphdata.out

It is now possible to look through the logfile at this time and see a rough estimate of how the webserver did, however it is more valuable if we can graph this data and examine it visually.

Import the data into excel, pages, or use gnuplot on the command line and plot the graph using a line graph.



The graph above was manufactured to illustrate the desired point. Note that the middle of the graph plateaus around 8500 requests/second. The flatness of the graph suggests that we’ve hit a bottleneck of some point. Since we know that the network is capable of nearly 22k request/second, and the network on each load client is presumably similar, we know that we’re either hitting the limitations of the disk subsystem or that we’ve pushed the webserver out of CPU.

If, during the test, you saw the idle CPU approach single digits, then we have reasonable confidence that we pushed the machine to its limit of CPU otherwise we may be pushing the limits of I/O.

6. Interpret the data and draw conclusions:

Now, by using the graph, we can decide on a limit for the webserver. The flat line starts approximately around 8500 req/sec. A reasonable buffer is 10% of that number, and so we would say that the max capacity of the webserver is 7650 req/sec. If you wish to be more conservative, and you should, you could leave yourself 25% capacity and call it 6325.

As a general rule you want to leave sufficient capacity on the machines to handle any excess load from failed hosts. If you have 2 machines, each machine should be able to handle all of the load. If you have 3 machines, then each should be able to handle 66% of the total load, and so forth.

7. Publish Findings:

With this simplistic testing done, you could approach development and tell them that you have some confidence that based on the preliminary testing you’ve done, the webserver can do 6325 ops/sec. Additionally, you should then provide the developers with your step by step guide as to how to get their own numbers to both allow them to validate your work, and to enable them to do this level of testing on their own in the future.

This concludes our first example. There are several more to come.

If you like what you’ve read, please share the blog with others. If you have any questions or comments, feel free to send me email at kmajer at karlmajer dot com.

>>> Karl

]]> http://www.karlmajer.com/2008/02/01/performance-testing-first-example/feed/ 0