Pro Puppet (23 page)

The system administrator has merged the changes using the
--no-ff
option in order to create a merge commit for each of the two topic branches. In the future, this merge commit will allow the team to refer back to the change list as a whole rather than having to tease apart which commit is associated with which topic. We’re able to verify that both the changes from the operator and the developer are now in the master branch of the system administrator’s repository, by using the
git log
command:

Notice that this time, the system administrator has used the
git log
command to display abbreviated log messages from the current head of the
origin/master
branch to the current head of the local checked out branch. He chose
origin/master
because he has not pushed the newly merged changes to the central repository and this command therefore shows a list of changes that will be pushed if he decides to do so.

Everything looks good, as he expected. He also doesn’t see the commit the developer made on his own topic branch to add the missing curly brace, because the developer chose to rebase his topic branch against the master branch before publishing his change list.

The team members decide to make the newly merged master branch the first testing branch, and they decide to continue developing on the master branch over the next couple of weeks. In a few days or weeks, the team will come together and decide on which of the change lists that each member has contributed are ready for merging into the testing branch. The system administrator starts this process by merging the changes he just made to the master branch into the testing branch, then pushing all of these changes to the central repository:

Notice that the system administrator executes two different push commands: one plain
git push origin
, and one
git push origin testing:testing
. This is because
git push,
by default, will only push the changes made to local branches into a remote repository if there is a branch with the same name in both locations.

Previously, the operator and developer logged into the puppet master and activated their changes by checking out their code in
/etc/puppet/environments/development/modules
. Similarly, the system administrator needs to fetch and checkout the new testing branch into the
/etc/puppet/environments/testing/modules
repository to activate the new configuration in the testing environment. Before doing so, he verifies that the remote named “origin” is configured to connect to the central repository at
[email protected]:git/modules.git:


Now that the testing environment repository has an up-to-date list of the branches, including the new testing branch, the system administrator performs a
git checkout
to activate the new changes on the system:

The system administrator is finally able to test a Puppet agent against the new testing environment, which now contains both changes: the SSH contribution from the operator, and the Postfix contribution from the developer. The testing environment is the only place where both changes are currently active in the configuration management system.

Our team of Puppet contributors at Example.com has been effectively making changes to the configuration management system. Using Puppet Environments and a version control system, they’re able to work efficiently and independently of one another without creating conflicts or obstructing another person’s work. We’ve seen how the operator and the developer were able to make two changes in parallel, publishing those changes in a branch in the central version control repository for the system administrator to merge into a testing branch.

The team has also tested a number of machines using the Puppet agent in the testing environment, and is now ready to release the configuration to the production systems. This section covers how the team creates their first release, and provides a process to follow for subsequent releases.

You’ll also see how a Git feature called “tagging” is useful to provide a method of referring to a specific point in time when the production configuration was active. You’ll see how tags provide the ability to quickly roll back changes that might not be desirable in the production environment.

First, the team decides to release the current testing branch into production. Before doing so, the system administrator creates a tag so this release can be easily referred back to in the future. The system administrator does this in his own personal repository in his home directory:

The process of creating a tag is often called “cutting a release.” The system administrator has done just this, tagged the current testing branch as a release to production, and then published the new tagged release into the central repository.

New branches, such as the testing or topic branches, were activated in the development and testing environments in the previous section. The process of activating a new production release is very similar, except instead of checking out a branch, which may change over time, a specific tag is checked out, which is static and refers to a very specific point in the history of configuration changes.

To activate the new production release, the system administrator logs into the Puppet master system as the user
puppet
, fetches the new tag from the central repository, and then checks out the tagged production release. Unlike the development and testing environments, Example.com has chosen to configure the production environment to use the working copy at
/etc/puppet/modules
rather than as a sub directory of
/etc/puppet/environments
where the development and testing active working copies reside.

Remember that the
git fetch
command does not affect the currently checked out configuration; it only updates the internal git index of data. The system administrator then checks out the newly-released production environment using the same familiar syntax we’ve seen so far:

The note about moving to a non-local branch may be safely ignored. A tag is static reference, and the team should not have any need to directly modify the files in
/etc/puppet/modules
or make commits from the active production environment repository.

After executing the
git checkout
command to place the 1.0.0 release of the configuration into the production environment, everything is now active for the puppet agents. The system administrator verifies this by executing
puppet agent
in the default environment:

You will also remember that the default environment is the production environment, and as such, the system administrator did not need to set the
--environment
command line option. If something were to have gone wrong in the production environment, a previous tag may be activated quickly, rolling back the changes introduced by the release of a new production configuration. One of the team members simply needs to execute
git checkout tags/x.y.z
to roll back the configuration.

The changes and workflow we’ve seen the operator, developer, and system administrator undertake in this chapter may now be repeated in a cycle. This development, testing, and release cycle provides an effective method to make changes to the configuration management system in a safe and predictable manner. Changes to the production system can be made with confidence: They’ve been vetted through the development and testing phases of the release process, they’ve been explicitly tagged in a release, and they can be quickly and easily backed out if things go awry.

You’ve seen how Puppet environments enable a team of contributors to work effectively and efficiently. Puppet environments, combined with a modern version control system, enable three people to make changes simultaneously and in parallel without obstructing each other’s work. Furthermore, the tagging and branching features of modern version control systems provide an effective release management strategy. The process a single team member may follow in order to make changes is summarized as:

• Develop changes in a local topic branch
  
• Rebase against the master branch to remove any unnecessary commits
  
• Publish the topic branch to the central repository
  
• Activate and try the changes in the development puppet environment
  
• Periodically merge and activate change lists from multiple people into a testing branch
  
• Periodically cut a release of the testing branch using version control tags.
  

• Debian stable, testing, unstable releases and distributions -
  http://www.debian.org/doc/FAQ/ch-ftparchives.en.html

• Puppet Labs Environments Curated Documentation -
  http://docs.puppetlabs.com/guides/environment.html

• Puppet Labs Environments Wiki Article -
  http://projects.puppetlabs.com/projects/1/wiki/Using_Multiple_Environments


We’ve seen that the Puppet agent and master require very little work to get up and running on a handful of nodes using the default configuration. It is, however, a significantly more involved undertaking to scale Puppet to handle hundreds of nodes. Yet many installations are successfully using Puppet to manage hundreds, thousands and tens of thousands of nodes. In this chapter, we cover a number of proven strategies that are employed to scale Puppet.

In this chapter you’ll see how to enable a single Puppet master system to handle hundreds of nodes using the Apache web server. We also demonstrate how to configure more than one Puppet master system to handle thousands of nodes using a load balancer. Throughout, we make a number of recommendations to help you avoid the common pitfalls related to performance and scalability.

Finally, you’ll learn how to measure the performance of the Puppet master infrastructure in order to determine when it’s time to add more capacity. We also provide two small scripts to avoid the “thundering herd effect” and to measure catalog compilation time.

First, though, we need review some of the challenges you’ll be facing along the way.

Earlier in the book, you learned a bit about Puppet’s client-server configuration and the use of SSL to secure connections between the agent and the master. Puppet uses SSL, specifically the HTTPS protocol, to communicate. As a result, when we’re scaling Puppet we are in fact scaling a web service, and many of the problems (and the solutions) overlap with traditional web scaling. Consequently, the two challenges we’re going to need to address when scaling Puppet are:

• Scaling the transport
  
• Scaling SSL
  

The first challenge requires that we increase the performance and potential number of possible master and agent connections. The second challenge requires that we implement good management of the SSL certificates that secure the connection between the master and the agent. Both challenges require changes to Puppet’s “out-of-the-box” configuration.

In
Chapter 1
we started the Puppet Master using the
puppet master
command. The default
puppet master
configuration makes use of the WEBRick Ruby-based HTTP server. Puppet ships WEBRick to eliminate the need to set up a web server like Apache to handle the HTTPS requests out of the box. While the WEBrick server provides quick and easy testing, it does not provide a scalable solution and should not be used except to evaluate, test and develop Puppet. In production situations, a more robust web server such as Apache or Nginx is necessary to handle the number of client requests.

Therefore, the first order of business when scaling Puppet is to replace the default WEBRick HTTP server. In the following section, we first replace WEBrick with the Apache web server on a single Puppet master system and then show how this strategy can be extended to multiple Puppet master systems working behind a load balancer.

The second change to Puppet’s out-of-the-box configuration is the management of the SSL certificates that Puppet uses to secure the connection between agent and master. The Puppet master stores a copy of every certificate issued, along with a revocation list. This information needs to be kept in sync across the Puppet worker nodes. So, together with the transport mechanism between the agent and master, we’ll explore the two main options of handling SSL certificates in a scalable Puppet deployment:

• Using a single Certificate Authority Puppet master
  
• Distributing the same Certificate Authority across multiple Puppet masters
  

The first scaling example we’re going to demonstrate is the combination of the Apache web server with a module called Phusion Passenger, which is also known as mod_rails, mod_passenger, or just Passenger. Passenger is an Apache module that allows the embedding of Ruby applications, much like mod_php or mod_perl allow the embedding of PHP and Perl applications. The Passenger module is not a standard module that ships with Apache web server, and as a result, must be installed separately. Passenger is available as a Ruby gem package, or may be downloaded and installed from
http://www.modrails.com/
.

For networks of one to two thousand Puppet managed nodes, a single Puppet master system running inside of Apache with Passenger is often sufficient. Later in this chapter, we examine how to run multiple Puppet master systems if you want a highly available system or support for an even larger number of Puppet-managed nodes. These more complex configurations all build on the basic Apache and Passenger configuration we introduce to you. We also build upon the Puppet master configuration we created in
Chapter 2
and the environment structure we introduced in
Chapter 3
.

First, you need to install Apache and Passenger, then configure Apache to handle the SSL authentication and verification of the Puppet agent, and finally connect Apache to the Puppet master and ensure everything is working as expected.

As we scale Puppet up, it is important to draw the distinction between the idea of a front-end HTTP request handler and a back-end Puppet master worker process. The front-end request handler is responsible for accepting the TCP connection from the Puppet agent, selecting an appropriate back-end worker, routing the request to the worker, accepting the response and finally serving it back to the Puppet agent. This distinction between a front-end request handler and a back-end worker process is a common concept when scaling web services.

To get started, you need to install Apache and Passenger. Apache and Passenger are a relatively simple and easy to set up. Pre-compiled Passenger packages may not be available for your platform, however, making configuration a little more complex. This section covers the installation of Apache and Passenger on the Enterprise Linux family of systems such as CentOS, RedHat Enterprise Linux, and Oracle Enterprise Linux.

In
Listing 4-1
, we’ve used the
puppet resource
command to ensure that Apache and the Apache SSL libraries are installed. We’ve also ensured that the Apache service is not currently running. The next step is to obtain Passenger, which is implemented as an Apache loadable module, similar to mod_ssl or mod_perl.

In order to install Passenger on our Enterprise Linux system, configure
yum
to access a local yum repository with packages for Puppet and
rubygem_passenger
. An example of the yum repository configuration for the x86_64 architecture is:

We’ve verified that the
rubygem-passenger
package is now available on this system, so we’re able to install the package using
puppet resource
, as shown in
Listing 4-2
.

At the time of writing, Passenger packages are available in Debian 5, “Lenny.” The packages available in the stable repository have known issues, however, and we recommend installing version 2.2.11 of Passenger from the backports package repository.

Installing Apache on Debian is very straightforward (see
Listing 4-3
). The packages available in the stable release of the Debian operating system work extremely well with Puppet. Please ensure you’ve
enabled Debian backports as per the instructions at http://backports.debian.org/ before attempting to install the passenger package.

As an alternative to the Puppet resource commands shown in
Listing 4-3
, Passenger may be installed from Debian backports using the command
aptitude -t lenny-backports install libapache2-mod-passener
.

Compiled binary packages of Passenger 2.2.11 are available for some platforms, but not all. Ruby Gems provide an alternative way to install the Passenger module. The passenger gem behaves slightly differently from most binary packages; the source code for Passenger is installed using the Gem format complete with a shell script to assist in the compilation of the Apache module.

For this installation method to succeed, the Apache development packages for your platform will need to be installed and present. The Passenger build scripts will link the library using the available version of Apache development libraries (
Listing 4-4
).

The output of the passenger-install-apache2-module script is quite long and has been truncated. For additional information and troubleshooting tips related to installing Passenger using Ruby Gems please see:
http://www.modrails.com/install.html

If you haven’t already done so, make sure you’ve started the Puppet master at least once to create the SSL certificates you’re going to configure Apache to use. Apache will then verify that the Puppet agent certificate is signed with the generated Puppet CA, and present a certificate that the Puppet agent uses to verify the authenticity of the server. Once you have you SSL certificates in place, configure Apache by enabling the Passenger module and creating an Apache virtual host for the Puppet master service.

First, enable mod_passenger with the following configuration provided in
Listing 4-5
.