Pro Puppet (36 page)

Once the configuration file has been created, a single line must be added to the activemq.xml configuration file. As you can see in
Listing 6-28
, a stomp URI line is inserted in the
transportConnectors
section.

Once the XML configuration file has been updated to work with stomp, the operator is ready to start the service (
Listing 6-29
).

The
sudo
command in
Listing 6-29
ensures the ActiveMQ service is running under the unprivileged activemq account, with the home directory environment variable reset to /var/lib/activemq using the
-H
flag.

With ActiveMQ up and running, we now need to connect the Puppet queue daemon process to the message bus to handle queued messages and write them to the database. The first step in this process is to install the stomp gem. This Ruby library provides a stomp protocol interface for Ruby applications (
Listing 6-30
).

After the installation of the stomp gem, a slight change to
/etc/puppet/puppet.conf
is needed to configure the Puppet master to hand off configuration information to ActiveMQ rather than performing the database writes itself. In the
[main]
section of the
puppet.conf
file, the
puppet queue
application will be configured to read from ActiveMQ and write to the database, offloading the work from the master (shown in
Listing 6-31
).

Note that the operator has changed the configuration slightly from
Listing 6-7
and
6-23
, where he had initially configured stored configurations. When queue support is enabled, the
puppet queue
daemon will pick up the settings in the
[main]
section to read data from the ActiveMQ stomp interface and write the information to the SQL database. In addition, the Puppet master application will use the settings in
[main]
and the settings in
[master]
to read from the SQL database and write to the ActiveMQ queue.

With these settings in place, let's see how the operator tests out the new queue system. First, he starts the Puppet master as he normally would:

Then, he does a normal Puppet agent run. He hasn't yet started the Puppet queue application, so we expect any configuration updates to be queued in the ActiveMQ system until the Puppet queue application is running and drains the queue.

Everything looks good so far. The Puppet Agent is able to communicate with the master and obtain a configuration catalog, but exported resources won't be written to the SQL database until the Puppet queue application is started in
Listing 6-32
.

Once the operator starts the Puppet queue daemon, it immediately connects to the ActiveMQ stomp port specified in the
puppet.conf
[main]
section and drains the queued configuration update. As additional Puppet agents retrieve catalogs, the Puppet master will place configuration updates in the queue which the Puppet queue daemon will drain and write to the SQL database. It is important to remember the Puppet master still reads directly from the database, so the Puppet queue daemon isn't required to be running for catalogs to be compiled.

In this section, we've seen how the Example.com operator configures the ActiveMQ messaging service to queue up expensive database updates. The Puppet queue daemon is then responsible for asynchronously handling each of these configuration updates and writing to the SQL database. This combination greatly reduces the performance impact of many Puppet agent systems checking in with the Puppet master. In the next section, we'll see how the operator periodically prunes the SQL database when a node with exported resources and stored configurations is retired.

A potential pitfall of using stored configurations is the situation where nodes retired from service still have configuration resources stored in the configuration database. Without periodically pruning the configuration database, these stale resources will linger indefinitely, tainting the configurations of remaining nodes. The Example.com operator incorporates a small utility to remove nodes from the configuration database when they take them offline.

Pruning a single node is quite straightforward using the
puppetstoredconfigclean.rb
script in the ext directory of the Puppet source code. If this script is not installed on your system, it may be downloaded from
https://github.com/puppetlabs/puppet/tree/2.6.4/ext
.

To clean out a node from the configuration database, simply give its short hostname as an argument to the script. In
Listing 6-33
, the operator removes the stored configurations for the nodes
mail01dev,
mail02dev, and
mail03dev
.

After running the stored configuration cleaning script, any resources exported by these nodes will no longer be collected in any Puppet manifest.

Exported resources and stored configuration are two very powerful features in Puppet. Using the central stored configuration database, each Puppet master system is capable of exporting and collecting resources as new hosts are brought online.

In this chapter, you've learned about the basics of virtual resources and how to export resources from one catalog and collect them in another. You saw three examples of how we might use exported resources:

• SSH public host keys that are easily stored centrally and distributed
  
• Adding load balancer members to an Apache configuration
  
• Exported resources to allow Nagios to automatically add new systems
  

You also saw how to scale stored configuration using ActiveMQ message queues, and the Puppet Queuing daemon. And finally, you learned how to prune expired hosts and resources from your stored configuration database.

• Virtual Resources
  http://docs.puppetlabs.com/guides/virtual_resources.html

• Exported Resources
  http://docs.puppetlabs.com/guides/exported_resources.html

• Using Stored Configuration
  http://projects.puppetlabs.com/projects/1/wiki/Using_Stored_Configuration


Until recently, you needed to manage Puppet via its manifest files and from the command line. As Puppet has matured, a small ecosystem of tools has emerged, including two console products: Puppet Dashboard and The Foreman.

Both console products are relatively new. The company that supports Puppet development, Puppet Labs, created Puppet Dashboard. Israeli developer Ohad Levy in turn wrote The Foreman. Both are Ruby on Rails applications and both are undergoing regular development.

Each tool suits a slightly different sort of environment. Puppet Dashboard can be used as an External Node Classifier (ENC) as well as a reporting tool, and is moving towards being an integration interface for a variety of new Puppet functions including audit and inventory capabilities. The Foreman has a stronger focus on provisioning and data center management and already includes some inventory capabilities.

In this chapter, we show you how to install and configure both consoles and demonstrate some of their features and capabilities. We show you how to use both consoles:

• As ENCs (we learned about ENCs in
  Chapter 5
  )
  
• To display data about the status and state of your hosts
  
• To display and analyze Puppet reports
  
• To make use of additional capabilities to provision and manage Puppet and your hosts
  

Later, in
Chapter 8
, you'll learn more about Puppet's integration with other tools.

Puppet Dashboard is a Ruby on Rails application designed to display information about your Puppet masters and agents. It allows you to view graphs and reporting data aggregated from one or more Puppet masters. It also makes inventory data (your host's Facts and other information) from your Puppet agents
available on one or more Puppet masters. Lastly, it can be used as an ENC to configure your Puppet nodes and specify the classes and parameters available on those nodes.

We're going to take you through installing the Dashboard, making use of its features, integrating with Puppet masters, and maintaining the Dashboard, including backups and management of data.

Let's start by installing the Dashboard.

Installing the Puppet Dashboard requires some basic prerequisites, typical of a Ruby on Rails application. These include Ruby 1.8.x (Dashboard doesn't yet work with 1.9.x) and a MySQL database to store data in. Currently Dashboard only supports MySQL as a database back end. There are plans to support additional back ends in later releases.

We're going to take you through installing Version 1.0.4 of Dashboard on both Red Hat and Ubuntu, including how to install the required prerequisites.

First, we need to add the Extra Packages for Enterprise Linux (EPEL) package repository (which we first saw in
Chapter 1
).

1. Add the
   epel-release
RPM Package Manager (as of this writing, this is the current RPM – you should go to the EPEL website to find the latest version):
   
   $ sudo rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/
epel-release-5.4.noarch.rpm

2. Install any of the required packages you haven't already added:
   
   $ sudo yum install -y mysql mysql-devel mysql-server ruby ruby-devel ruby-irb ruby-mysql
 ruby-rdoc ruby-ri

3. Start MySQL and configure it to start at boot. Use the
   service
command to start the service:
   
   $ sudo service mysqld start

4. Then use the
   chkconfig
command to configure MySQL to start when the host boots:
   
   $ sudo chkconfig mysqld on

5. We also need to install the RubyGems package manager. Unfortunately, the RubyGems package manager provided with Red Hat (and CentOS) 5.x releases isn't suitable. We need to manually install an appropriate version. To do this, we download the RubyGems source and install it. This will download, unpack and install the
   gem
command and required libraries.
   
   $ cd /tmp
$ wget http://production.cf.rubygems.org/rubygems/rubygems-1.3.5.tgz
$ tar -xzf rubygems-1.3.5.tgz
$ cd rubygems-1.3.5
$ sudo ruby setup.rb

6. Once you've installed RubyGems, you need to install the
   rake
gem.
   

That's it, you're done installing the Red Hat prerequisites.

On Ubuntu 10.04 and later, you need to install several packages.

1. Start with the following:
   
   $ sudo apt-get install -y build-essential irb libmysql-ruby libmysqlclient-dev
 libopenssl-ruby libreadline-ruby mysql-server rake rdoc ri ruby ruby-dev

2. Install the RubyGems package manager. Unfortunately, the RubyGems package manager provided with Ubuntu 10.04 and earlier is not a recent enough version to support the required RubyGems. We need to manually install an appropriate version. To do this we download the RubyGems source and install it. This will download, unpack and install the
   gem
command and required libraries:
   
   $ cd /tmp
$ wget http://production.cf.rubygems.org/rubygems/rubygems-1.3.7.tgz
$ tar –xzf rubygems-1.3.7.tgz
$ cd rubygems-1.3.7
$ sudo ruby setup.rb

3. Use the
   update-alternatives
command to add the newly installed RubyGems version as an alternative command.
   

Now that we have all the prerequisites, we can install the Dashboard itself.

The Dashboard is available in package form as RPMs (Red Hat, et al) and DEBs (Debian and Ubuntu, et al) from the Puppet Labs package repositories. For RPMs this is
http://yum.puppetlabs.com
, and for DEBs this is
http://apt.puppetlabs.com
. Puppet Dashboard can also be installed from source, via tarball available from the Puppet Labs download site (
http://www.puppetlabs.com/downloads/
) or by cloning the GitHub repository at
https://github.com/puppetlabs/puppet-dashboard
.

To install the Dashboard from an RPM, you need to add the Puppet Labs Yum repository to your Dashboard host.

1. Create an entry:
   
   $ sudo vi /etc/yum.repos.d/puppetlabs.repo


   The entry should be:
   

    [puppetlabs]
name=Puppet Labs Packages
baseurl=http://yum.puppetlabs.com/base/
enabled=1
gpgcheck=1
gpgkey=http://yum.puppetlabs.com/RPM-GPG-KEY-puppetlabs

2. Run the Yum package manager:
   
   $ sudo yum update


   This will update the Yum package repository data.
   

3. Run the Yum package manager again to install the Dashboard itself:
   

The Dashboard package will be installed and the Dashboard site itself will be installed into the
/usr/share/puppet-dashboard
directory.

To install the Debian or Ubuntu DEB packages, you need to add details of the Puppet Labs APT repository to your Dashboard host.

1. Edit the /etc/apt/sources.list file by adding the following lines:
   
   deb http://apt.puppetlabs.com/ubuntu lucid main
deb-src http://apt.puppetlabs.com/ubuntu lucid main

2. Add the Puppet Labs GPG key to validate the downloaded packages, like so:
   
   $ sudo gpg --recv-key 4BD6EC30
$ sudo gpg -a --export 4BD6EC30 > /tmp/key
$ sudo apt-key add /tmp/key

3. Then, run an update to refresh APT:
   
   $ sudo apt-get update

4. Finally, install the Puppet Dashboard package:
   

The Dashboard package will be installed and the Dashboard site will be installed in the
/usr/share/puppet-dashboard
directory.