Pro Puppet (60 page)

The output shown in
Listing 11-27
comes from information stored in the DDL file accompanying each MCollective agent. When installing agent plugins, the DDL file should b installed on the system where the
mc-rpc
command is invoked to provide documentation on the command line.

So far we've seen how MCollective is useful for starting Puppet agent runs on demand on all hosts in the collective. What if the operator wants to perform actions on only a subset of the collection? MCollective allows systems to be addressed by any value returned from Facter. Let's see how the operator configures MCollective to work with Facter to obtain this information.

MCollective allows systems to be addressed by metadata about the each system in addition to the system host name. This provides much more flexibility because any relevant information about each node can be used to group systems into collectives. MCollective integrates with the Facter library to collect this metadata on each server and on demand. By default, the metadata MCollective uses is statically defined in the file
/etc/mcollective/facts.yaml
. In most situations, a library like Facter should be used to dynamically generate metadata for each system.

Let's see how the Example.com operator reconfigures MCollective in
Listing 11-28
to obtain metadata about each system from Facter.

On Enterprise Linux-based systems, the MCollective plugin directory is located in
/usr/libexec/mcollective
rather than
/usr/share/mcollective/plugins
on Debian. The operator installs the Facter plugin on Enterprise Linux using the command shown in
Listing 11-29
.

Once the Facter plugin is installed, configuration is simply a matter of adding a few lines to the
server.cfg
file and restarting the MCollective servers on all of the nodes. The output shown in
Listing 11-30
are the lines the operator uses to configure MCollective for use with Facter in
/etc/mcollective/server.cfg
.

We can see the operator has changed the default configuration of
factsource = yaml
to use Facter instead. Finally, the operator restarts the MCollective server daemon in
Listing 11-31
to activate the change.

The operator is ready to test if MCollective is properly obtaining information about each system from Facter. This is easily accomplished with the
mc-facts
command. This command accepts a Facter variable and returns a count of the number of systems with each value set. Let's see what this looks like in
Listing 11-32
.

We can see from the output of the
mc-facts
command that two systems are in the collection, one of them a CentOS system and one of them a Debian-based system. In the next section we'll show how to make more advanced use of the rich metadata Facter provides. Specifically, this information about each node may be used to divide the nodes into collections and only execute commands on systems matching specific criteria.

The mcollective-plugins project mentioned in this chapter contains a number of useful agent plugins for MCollective. However, you may find the need to write your own agents and actions to carry out deployment or administrative tasks on your systems. Please visit the latest MCollective documentation at
http://docs.puppetlabs.com/
to learn more about writing agents for MCollective.

We also recommend you fork the mcollective-plugins project on GitHub and use some of the small agent plugins as a reference to writing your own. The
filemgr.rb
plugin is a great starting point to get started with MCollective. If you do write a new agent, please don't hesitate to submit a pull request to share your work with the rest of the MCollective community.

In the previous section, you learned how the Example.com operator uses MCollective with Facter to obtain metadata about each system. This dynamic information provides a unique way to execute commands on a large number of systems. Specific systems matching exact criteria may also be selected to execute MCollective commands on. The operator no longer needs to maintain cumbersome spreadsheets with all of the hostnames for his systems. If a command needs to be executed, MCollective provides a simple and straightforward way to do so rather than connecting to each machine in succession over SSH.

When working with Puppet and MCollective, hosts may be addressed by any Facter value or any class the host has been assigned from Puppet. These classes are read from
/var/lib/puppet/state/classes.txt
. This file may be in a different location on your system and can be found using the command
puppet agent --configprint classfile
.

To address all systems that are Debian or CentOS, the operator uses the
--with-fact
option of MCollective client commands. Let's see how the operator finds the amount of free memory without knowing the hostname of the systems (
Listing 11-33
).

The operator uses a regular expression to execute the facts agent on systems where Facter reports the
operatingsystem
to be CentOS or Debian. By using this regular expression, the operator is easily able to exclude systems and obtain information only from the systems he's interested in.

Host filters work nearly everywhere in MCollective. The pervasiveness of filters is a key differentiator between MCollective and other command and control tools. Notice in
Listing 11-34
how the
mc-rpc
command is able to execute agent plugins using host filtering.

The operator uses the
--with-fact
option to send the status action to the
puppetd
agent to the collection of Debian systems. In addition to specifying one filter, the operator is able to narrow down the selection of nodes for the collective using multiple filters. Let's see how this works in
Listing 11-35
.

The Example.com operator uses the short version of the
--with-fact
option to filter against both the
operatingsystem
and
fqdn
facts. With this command, any CentOS system with the word “mail” in the fully qualified hostname will match the filter. This regular expression matching, in real-time, allows the operator to write scripts that will take into account additional systems. Perhaps mail01 and mail02 will come online in the future in addition to the single mail system. With the ability to filter on any Facter value and combine multiple filters, actions may be carried out that take into account the number of systems automatically. Scripts no longer need to be updated as hosts are added to the network.

In this chapter, you learned how MCollective provides real-time, metadata-driven command and control of Puppet-managed systems. MCollective takes an innovative and unique approach to the problem of orchestrating a large number of systems. Instead of using hostnames to uniquely identify and access systems, MCollective integrates with Facter, allowing the operator to filter out machines he does not want to carry out actions on.

In addition to the unique approach of addressing machines through metadata, MCollective uses the STOMP messaging protocol to communicate. The MCollective client (most commonly accessed through
the
mc-rpc
command) and the MCollective server take advantage of the proven scalability and performance of asynchronous messaging services.

You also learned how to configure the RabbitMQ messaging service for use with MCollective as well. While RabbitMQ is relatively easy to configure and get running, there may be performance and scalability considerations that make ActiveMQ a better choice for your deployment. In addition, MCollective is most heavily developed and tested with ActiveMQ. RabbitMQ support was recently added and the STOMP connector for RabbitMQ works well with MCollective as of version 2.3.

MCollective gives you the ability to obtain information from your systems in real-time, without the tedium of scripting SSH connections to each and every hostname on the network. Systems may be added and removed from the network quickly without the need to update scripts or other programs communicating with these systems. In addition, MCollective works extremely well with Facter and Puppet, enabling control of the Puppet agent and filtering of hosts through Facter with ease.

• http://devco.net/


  The blog of R.I. Pienaar, the author of MCollective.
  

• http://docs.puppetlabs.com/


  MCollective Documentation is located on the Puppet Labs curated documentation site.
  

• http://docs.puppetlabs.com/mcollective/reference/basic/messageflow.html


  An architectural overview of how messages travel from client to server processes in MCollective.
  

• http://docs.puppetlabs.com/mcollective/reference/integration/activemq_clusters.html


  Information about setting up multiple ActiveMQ middleware services for use with MCollective. This information may be useful for deployments among multiple data centers or geographic locations.
  

• http://en.wikipedia.org/wiki/Publish/subscribe


  Overview of the publish and subscribe methodology used by MCollective.
  

• http://forge.puppetlabs.com/


  The RabbitMQ Puppet module is available for installation using the puppet-module tool from the Puppet Forge.
  

• http://github.com/puppetlabs/puppetlabs-rabbitmq


  The source code for the RabbitMQ Puppet module is published on GitHub.
  

• http://www.rabbitmq.com/plugins.html


  RabbitMQ Stomp protocol plugins are available as a separate download from the main RabbitMQ website. Please download the version of the AMQP and Stomp plugins from this location.
  

• http://www.rabbitmq.com


  The main website for the RabbitMQ messaging middleware service.
  

• http://www.rabbitmq.com/admin-guide.html


  Additional information about user accounts and access control in RabbitMQ is located in the administrative guide.
  

• http://puppetlabs.com/downloads/


  MCollective packages and source may be downloaded from the Puppet Labs website.
  

• https://github.com/puppetlabs/mcollective-plugins


  Many agent plugins for MCollective are located in the mcollective-plugins Git repository on GitHub.
  

It is very important to remember that Puppet is an ever-developing tool with an ever-widening community. Not only is the Puppet community growing quickly but many new ideas, developments, patches, and recipes appear every day. This is important for two major reasons:

• More often than not, someone has already solved the issue, problem or challenge you are trying to address
  
• New features, functions, and fixes are available in every release
  

It is a good idea to check out the various resources we talk about in the Resources section below, such as the mailing list archives and the Wiki, when you have an issue. These forums are also where announcements are made about new releases of Puppet and related tools.

Puppet is an open-source tool and there are a lot of sources of information and support available for it (including this book!). In addition, Puppet's parent company, Puppet Labs, offers the Puppet Enterprise product (the pre-packaged commercial edition of Puppet), support contracts, and custom development, consulting and training programs worldwide. You can find details of these offerings at
http://www.puppetlabs.com/
.

There are also a number of members of the Puppet community who offer services, implementation support and consulting services. Many local systems implementers and consultants also have Puppet, skills should you require assistance. Posting a message on the Puppet mailing list, or your Linux User Group or Open Source Business Association forums seeking help will usually result in offers of assistance.

There are a number of useful resources available to get you started with Puppet. We'll refer to these and other references throughout this book. We also strongly recommend subscribing to the Puppet mailing lists (see below) as a lot of useful information, tips and tricks, and trouble-shooting assistance is presented there. Currently the mailing list has over 3000 subscribers and is an active and helpful community.

The Puppet IRC channel, #puppet on the Freenode network, is also a useful place to visit and ask for help. There are 500 people regularly on the channel and while they are all generally busy system administrators, they can usually spare some time to help people new to Puppet.

Like all requests for help, when asking on email or IRC, you should try to ask a good question. Include your Puppet version, your platform and the exact error you are receiving. The more information you provide, the easier it is for people to help you.

Another good resource for information on asking good questions on the Internet is
http://catb.org/esr/faqs/smart-questions.html
.

You can also find a searchable log of IRC conversations available at
http://pelin.lovedthanlost.net/puppet/
.

• Puppet Bug Tracker:
  
• http://projects.puppetlabs.com

• Puppet Source Code:
  
• https://github.com/puppetlabs/puppet

• Facter Source Code:
  
• https://github.com/puppetlabs/facter

• MCollective Source Code:
  
• https://github.com/puppetlabs/marionette-collective

• Puppet Documentation:
  
• http://docs.puppetlabs.com/

• http://docs.puppetlabs.com/learning/

• http://docs.puppetlabs.com/references/

• Puppet Documentation in PDF:
  
• http://www.puppetlabs.com/resources/downloads/

• Puppet Wiki:
  
• http://projects.puppetlabs.com/projects/puppet/wiki

• Puppet FAQ:
  
• http://docs.puppetlabs.com/guides/faq.html

• Puppet Style Guide:
  
• http://docs.puppetlabs.com/guides/style_guide.html

• Puppet Labs:
  
• http://www.puppetlabs.com


• Puppet User Group:
  
• http://groups.google.com/group/puppet-users
/
  
• Puppet Developer Group:
  
• http://groups.google.com/group/puppet-dev/


• http://forge.puppetlabs.com


• http://www.puppetlabs.com/puppet/puppet-enterprise/


• http://puppetlabs.com/services/


• http://puppetlabs.com/training/


• Puppet IRC Channel:
  
• irc://irc.freenode.net/puppet

• Puppet Developers IRC Channel:
  
• irc://irc.freenode.net/puppet-dev