Pro Puppet (56 page)

We mentioned earlier that functions run on the Puppet master. This means we only have access to the resources and data available on the master, but this does include some quite useful information, most importantly fact data. You can look up and use the value of facts in your functions using the
lookupvar
function, like so:

Replace
fqdn
with the name of the fact whose value you wish to look up.

You can see how easy it is to create some very powerful functions in only a few lines of code. We recommend having a look at the existing functions (most of which are very succinct) as a way to get started on your first functions. Some of the common functions include tools to manipulate paths, regular expressions and substitutions, and functions to retrieve data from external sources. There are numerous examples (many on Github or searchable via Google) of functions that you can copy or adapt for your environment.

After you've created your function you should test that it works correctly. There are a couple of ways you can do this. Some basic testing of the function can be performed by executing the function file with Ruby, like so:

This loads the Puppet library (Puppet must be installed on the host) and then runs the file containing the function we created in
Listing 10-12
. This will allow us to determine whether the file parses without error. It does not tell us if the function performed correctly.

We can also use the Ruby IRB (Interactive Ruby Shell) to confirm our function is properly defined, like so:

Here we've launched
irb
and then required Puppet and our new function. We then confirm that Puppet can see the new function and that it parses as a correct function.

The best way to test a function is to use it in a manifest, and the easiest way to do that is to add your functions to Puppet's
libdir
and run a stand-alone manifest. Assuming Puppet is installed, first find your
libdir
:

Then create a directory to hold our functions:

Then copy in our function:

Then a manifest to execute our new function:

And finally run the function:

We can see that our
notify
resource returned a 512-bit hash generated by our
sha512
function.

In this chapter, you learned how to extend Puppet and Facter with your own custom types, providers, functions and facts. We demonstrated how to:

• Configure Puppet to distribute your custom facts in your modules
  
• Write your own custom facts
  
• Test your new custom facts
  
• Utilize two ensure-style types and providers
  
• Use a ParsedFile type and provider to edit simple configuration files
  
• Write Puppet functions
  
• Test Puppet functions
  

There are also a lot of examples of extensions and additions to Puppet that are available for you to add to your Puppet installation, or which can serve as examples of how to develop particular extensions. A good place to start looking for these is on GitHub (
http://www.github.com
).

• Adding custom facts
  http://puppetlabs.com/trac/puppet/wiki/AddingFacts

• Try Ruby (
  http://tryruby.org/
  ) online tutorial
  
• Learn to Program tutorial (
  http://pine.fm/LearnToProgram/
  )
  
• Programming Ruby (
  http://ruby-doc.org/docs/ProgrammingRuby/
  )
  
• Beginning Ruby (
  http://beginningruby.org/
  ).
  
• Documentation on how to create custom types:
  http://docs.puppetlabs.com/guides/custom_types.html
  
• A complete example of resource type creation:
  http://projects.puppetlabs.com/projects/puppet/wiki/Development_Complete_Resource_Example
  
• Documentation on detailed provider development:
  http://projects.puppetlabs.com/projects/puppet/wiki/Development_Provider_Development
  
• Practical set of documentation covering type development:
  http://projects.puppetlabs.com/projects/puppet/wiki/Development_Practical_Types
  
• Writing your own functions:
  http://projects.puppetlabs.com/projects/1/wiki/Writing_Your_Own_Functions
  
• Writing tests for Puppet:
  http://projects.puppetlabs.com/projects/puppet/wiki/Development_Writing_Tests
  
• Try Ruby (
  http://tryruby.org/
  ) online tutorial
  
• Learn to Program tutorial(
  http://pine.fm/LearnToProgram/
  )
  
• Programming Ruby (
  http://ruby-doc.org/docs/ProgrammingRuby/
  )
  
• Beginning Ruby (
  http://beginningruby.org/
  ).
  

In
Chapter 10
, you learned about the puppet-module and cucumber-puppet tools. Both of these tools help automate the process of developing and testing Puppet modules. Similarly, Marionette Collective (MCollective) is an orchestration framework closely related to Puppet. Puppet excels at managing the state of your systems; however, the default 30-minute run interval of the Puppet agent makes it unsuitable for real-time command and control. MCollective addresses the need to execute commands in real-time on a large number of systems in a novel and unique manner. With MCollective, nodes are easily divided into collections based on information about the node itself rather than hostnames. The use of metadata means you don't need to maintain long lists of hostnames or IP addresses. All systems in the collection can report information about themselves in real-time on demand. Armed with this information, the overall population of machines can be divided into collectives. Procedures are carried out remotely against a collective rather than against a single machine.

MCollective was created to provide an API for the orchestration tasks that systems engineers and developers frequently need to perform. Command and control tools are numerous and effectively provide the same functionality of the Unix shell. Though powerful, the shell interface is not an ideal application-programming interface. In addition, commands dispatched to systems in this manner are difficult to manage using the same tools and processes that you manage code with. With a robust API, orchestration actions may be implemented as small agent plugins and treated like other pieces of code in a software development lifecycle. MCollective agents are testable, version-controlled, and consistently repeatable.

There are a number of problems and use cases that MCollective is particularly well-suited to address. Through the use of real-time messaging and metadata addressing, a number of tasks previously carried out with SSH or other deployment tools are more efficiently solved with MCollective. In particular, the following actions and questions are addressed extremely well with MCollective:

• How many systems have 32 GB of memory?
  
• What systems are online
  right now
  ?
  
• Deploy version 1.2.3 of my application to all systems.
  
• Deploy version 1.2.4 of my application to the quality assurance systems.
  
• Deploy version 1.2.5rc2 of my application to the development systems.
  
• Run Puppet on all systems, ensuring that at most 10 runs are happening at once.
  
• Restart the Apache service on all systems in North America.
  

In addition to the actions MCollective already handles, it is quite straightforward to write custom agents in Ruby to carry out your own actions on all of your systems. The MCollective RPC framework alleviates much of the effort you would otherwise have to spend writing code to connect to your machines, issue commands to them, and handle logging and exceptions. If you need to take action on all
of your systems, MCollective agents distributed through Puppet are an excellent way to quickly tackle the problem.

MCollective takes advantage of modern technologies to handle communication between the nodes in a collective. In this chapter, you'll learn how to install the RabbitMQ message bus and connect MCollective servers to a message queue. Once that's installed, you'll also learn how integrate MCollective with Facter to provide a large amount of metadata that's useful to divide the population into collectives and then command them. In addition, you'll learn how Puppet works well with MCollective to orchestrate and reconfigure your nodes on demand and in a controlled manner. Plugins for MCollective provide these integrations with Puppet, specifically the Puppet Agent plugin and Puppet Commander plugin. Let's get started with the installation of the RabbitMQ messaging bus.

MCollective makes use of publish and subscription messaging techniques. These publications and subscriptions are often implemented using asynchronous messaging software such as ActiveMQ and RabbitMQ. The broad category of messaging software is often referred to as messaging middleware. MCollective is developed and tested with the Apache ActiveMQ middleware, however the requirement of Java and XML configuration files have driven increased attention and interest in the RabbitMQ middleware service.

MCollective sends and receives messages using the Stomp protocol. Any messaging middleware implementing a robust Stomp listener should work with MCollective. However, ActiveMQ and RabbitMQ are the two most widely deployed and tested middleware services used with MCollective. It is important to keep in mind that only one messaging service on one system is required to get started with MCollective. A single RabbitMQ server will easily support hundreds of connected MCollective server processes. Advanced configurations with multiple security zones and tens of thousands of nodes may consider deploying multiple, federated messaging services to scale with demand. In a multi-datacenter configuration, ActiveMQ is an excellent choice. ActiveMQ and MCollective have been deployed together across multiple data centers and geographic continents in redundant and reliable configurations.

It is worth noting that MCollective and RabbitMQ are not bundled with the distribution of Enterprise Linux or Debian and Ubuntu systems as of the writing of this book. However, both RabbitMQ and MCollective will be included in the Natty Narwhal release of Ubuntu, scheduled to be version 11.04 of the operating system. If you are running a Natty or a more recent version of Ubuntu, the easiest installation route will be to simply install the packages from the distribution repositories using aptitude.

MCOLLECTIVE MESSAGING ARCHITECTURE

RabbitMQ is a message queue service implementing the Advanced Message Queuing Protocol, or AMQP. RabbitMQ is built using the OTP (Open Telecom Platform) and implemented in the Erlang language and runtime environment. To get started with RabbitMQ, first see how the Example.com operator installs and configures the Erlang runtime on each platform, then install and configure RabbitMQ and plugins required for MCollective. The information in this chapter is specific to RabbitMQ for ease of configuration and installation on both Debian and Enterprise Linux systems. While ActiveMQ is equally suitable to the task, many people find the XML configuration file more complex than the direct command line interface of RabbitMQ.

Debian and Ubuntu systems provide the Erlang runtime as precompiled binary packages. The operator installs Erlang before RabbitMQ using the
aptitude install
command as shown in
Listing 11-1
.

On Debian-based systems, installation of the Erlang runtime is straightforward. The operator simply uses aptitude to install the
erlang
package. Next, in
Listing 11-2
, the operator installs the RabbitMQ server package. This package is not available in the main Debian repositories, so the best way to install RabbitMQ is by adding the RabbitMQ repositories to the apt packaging system.

In order to enable the RabbitMQ repository, the operator uses the
puppet resource
command to set the contents of the rabbitmq apt source file. If the version of Debian you are running does not have an
/etc/apt/sources.list.d
directory, you'll need to append the line listed in the content attribute to the
/etc/apt.sources.list
file instead of creating a new file.

Next, the operator downloads the RabbitMQ public key to verify the package signatures and adds this key to the apt package management system. Finally, the apt-get update command should not return any errors about verifying the authenticity of the package repository. If you receive any such errors, please make sure the RabbitMQ public key has been added successfully.

With the apt repository configured and updated, the RabbitMQ server software may be installed, as shown in
Listing 11-3
using a straightforward
aptitude install
command.