May 19 2008
At a customer site Iâ€™ve had both the pleasure and pain of working with cfengine as a means to control the environment from a single touchpoint. The client decided to save the precious little time they had and to use readily available packages for the software instead of building the software from scratch. This resulted in mixed versions between the RedHat and Solaris machines with RedHat running the 2.2.x series code and Solaris on 2.1.x. What followed over the months were several exercises in frustration.
Cfengine was picked after an evaluation showed that it was capable of updating system files across machines and pushing files from the repository to the individual hosts. With the proof of concept working and with high hopes from the functionality referred to in the documentation, cfengine was rolled onto the network.
The first task was relatively straightforward; insure that the staff memberâ€™s logins were always present in the sudoers file. The site was using several group aliases to define various sudoers functions, which leaded to order dependencies in the file. The solution found was to delete all of the aliases and groups definitions and re-add them each pass. This was found to work until it was realized that this would happen from cron every 15 minutes. Not quite what was hoped for. The configuration was ultimately altered to explicitly look for a lack of team members before beginning any of the updates.
The next task was to distribute a dozen or so files from the central repository to all 250 or so machines. This worked almost perfectly and required only a bit of tuning on the MaxConnections, MaxCfengins, and Splay time directives. The current challenge with this task now is to ensure that all of the installed machines do the pull on an update as it seems that occasionally some small percentage of them, typically Solaris, refuse to pull the files without some manual cfrun intervention.
An Editfiles directive is provided to allow the editing of arbitrary files. Using this for the next task, editing the crontab on linux was straightforward and linux cron noticed the change immediately. On solaris this was not the case. Editing the files was the easy part. After editing the file the cron daemon needs to be made aware of the change, and so either the process needs to be restarted or the crontab command needs to be used to reread the file. Unfortunately, the Process directive proved to be little use on the older cfengine and the Shell directive was used to write custom inline shell scripts to handle the stopping and starting of cron.
The most recent change to the network involved manipulating the fstab on both OSes, removing an NFS mount, and then changing the former mount point from a directory to a symlink. There is an Unmount directive that would have worked beautifully had it worked on both OSes. It worked fine with Linux, unmounting the FS and deleting the entry and directory for us. Sadly, on Solaris it did nothing. To promote overall consistency it was opted to do manipulations on both operating systems in the same manner. An Editfiles stanza was used to manipulate the (v)fstabs, while a long chain of shell commands to ensure that the filesystem was unmounted, the fstabs were edited, and the symlink was created. Each component set a success or failure flag to enable the script to proceed to the next step or fail and alert as the case may be. This is probably the most complex script used at the customer site.
Iâ€™m unable to say for certain how much of the pain with the software is caused by the mixed versions being used on the network. It could be that if there was time to build and upgrade the software across the machines to the same version that the problems would magically disappear. It could also be that while powerful and seemingly capable, cfengine does things in a slightly different manner than a sysadmin, making the would-be cfengine administrator configuring it have to double check everything they write. (An example of this is the symlink directive which uses target->source whereas the ln command uses source->target.) Documentation and error reporting for the product was also a challenge, especially with the mixed version. The version specific source was often referred to understand what an error means or to determine why a specific section of code was being ignored. Cfengine will remain in place at the customer site for the foreseeable future, as despite its issues it is still better than touching each machine individually. Only time will tell if a complete upgrade to a consistent version will remove some of the pain or if this is simply the way of cfengine.