Pro Puppet (28 page)

Note that the primary Puppet CA worker is using the default CA directory. If the active worker falls offline, requests will be redirected to the Puppet CA worker on 18143 and therefore the
/var/lib/puppet/ssl/ca.standby/
directory.

Once the certificate has been signed, you can run the Puppet agent on the new node again to make sure the agent is able to download its signed certificate from the master.

You can also check the logs again to make sure HTTP Status 200 is present, now that the Puppet agent has the signed certificate:

You can see two log entries, matching up with the Puppet agent downloading its signed certificate and the certificate revocation list maintained by the Puppet CA worker. Both entries contain HTTP Status 200 “OK” codes, indicating that the Puppet agent successfully transferred the certificate and revocation list from the Puppet CA.

You can also see that the access logs for the active Puppet CA worker does not contain any catalog requests. Check the access logs of the two load balanced Puppet master workers running on ports 18140 and 18141 in order to make sure catalog requests are being routed correctly to only those two systems:

Note that the catalog requests are still being directed by the front-end load balancer to the worker running on Port 18140, while certificate requests are being directed to the active Puppet CA on port 18142.

With this, you’ve configured the front-end HTTP load balancer to direct all certificate related requests to a single Puppet CA worker. This redirection ensures that the certificate revocation list and serial.txt files are maintained properly.

Now that the certificate requests are being handled properly, the next step is to configure the hot standby Puppet CA worker. If the primary Puppet CA worker fails, another worker should quickly take over responsibility for responding to certificate requests. We will take advantage of the load balancer’s ability to redirect requests in order to quickly fail over to the backup Puppet CA worker.

You first need to configure a periodic task to automatically synchronize the CA directory across the primary and secondary workers. Next, you’ll configure the load balancer to use the secondary worker as a hot standby, automatically activated in the event the primary worker goes offline. Finally, you will test the new configuration and work through the exercise of testing the fail-over and fail-back to the primary Puppet CA worker.

In the following examples, we have configured both Puppet CA workers on the same system using two distinct certificate authority directories. Copy the existing certificate authority directory to the hot
standby using
rsync
. This command could also be configured as a
cron
task, to keep the hot standby directory contents up to date:

This command sets an environment variable named
$cadir
set to the default value of the
cadir
configuration setting used by the primary CA worker.

This rsync command synchronizes the primary CA directory into the standby CA directory, deleting any files existing in the destination and not in the source.

Once the certificate data has been synchronized, you can test failover between the two virtual hosts on ports 18142 and 18143. In the previous section, we configured the virtual host running on port 18143 as a hot standby. We also configured
/etc/puppet/rack/puppetmaster_18143/config.ru
to use a unique CA directory named
ca.standby
.

We’re going to use the iptables firewall to block traffic to the primary Puppet CA worker, effectively simulating a failure of the service. We expect the load balancer configuration to automatically redirect certificate requests to the hot standby.

Once the primary Puppet CA worker is inaccessible, you can test that certificate requests are automatically redirected to the secondary worker listening on port 18143, using the
curl
command

You can see the results of this curl in the logs:

And:

The first command we’ve executed is a standard curl HTTP request. Rather than display the contents of the request body, we display the HTTP Response headers on standard output. The HTTP header output provides an indication of the status of the response, with anything other than status 200 indicating an error.

After requesting the Puppet CA certificate, you can look at the error log file of the front-end load balancer to see how the request as handled. As expected, the
iptables
firewall rule prevented the load balancer from forwarding the request to the primary Puppet CA worker listening on port 18142. The load balancer properly failed over to the hot standby Puppet CA worker listening on port 18143 and forwarded the request to the virtual host.

Looking at the access logs in
puppetmaster_worker_access_18143.log
of the Puppet CA worker listening on port 18143, we can see the incoming request and resulting HTTP 200 OK status code for the response.

Now we want to make sure we can still provision new Puppet managed nodes while the hot standby certificate authority is currently active. With a different CA directory configured on the hot standby, the process of listing and signing certificates is slightly different, requiring the use of the
puppet cert --cadir
configuration option:

The first time a Puppet agent is run, a new certificate request is generated and submitted to the Puppet master. Since the primary Puppet CA worker is offline, we expect to see the pending certificate request in the standby directory:

In order to see the pending certificate-signing request in the standby directory, you need to provide the
--cadir
option, as we’ve done using the
puppet cert
command shown in
Listing 4-25
.

Similar to listing the pending requests, you’re able to sign a specific request waiting in the standby directory using
puppet cert
with the
--cadir
option.

Finally, you can reconnect the Puppet agent and obtain the signed certificate using the same command we started with. The peer certificate warning in this output may be safely ignored.

The failover to the secondary CA is now working properly, and new certificates can be signed. Let’s test the process of re-activating the primary Puppet CA Worker. The load balancer will automatically start using the primary worker when it comes online again, so the process becomes a matter of synchronizing the secondary certificate authority back to the primary CA directory. You need to synchronize changes before re-activating the firewall rule to allow traffic back to the primary certificate authority.

Similar to the rsync command synchronizing the primary CA directory into the standby location, the rsync command shown in
Listing 4-26
reverses the direction and synchronizes the standby CA directory into the primary location before re-enabling the primary CA using the host firewall.

The two
iptables
commands first list the rules associated with incoming traffic. We see that rule #1 is rejecting traffic destined for the port used by the primary CA worker. The second
iptables
command deletes rule #1 to allow traffic once again.

You performed three simple tasks to re-activate the primary Puppet CA Worker. First, you synchronized the CA directory from the standby in
Listing 4-23
. Notice that three files have changed since the hot standby worker has become active. These three files changed when you signed the certificate request for
test.example.lan
in
Listing 4-22
. Immediately after synchronizing the CA directory, you removed the iptables rule blocking access to the primary Puppet CA worker. The first iptables command lists rules in the INPUT chain by number and the second iptables command removes rule number one from the firewall policy.