Puppet: Mastering Infrastructure Automation电子书

Puppet: Mastering Infrastructure Automation

Table of Contents

Puppet: Mastering Infrastructure Automation

Puppet: Mastering Infrastructure Automation

Credits

Preface

What this learning path covers

What you need for this learning path

Who this learning path is for

Reader feedback

Customer support

Downloading the example code

Errata

Piracy

Questions

1. Module 1

1. Writing Your First Manifests

Getting started

Introducing resources and properties

Interpreting the output of the puppet apply command

Dry-testing your manifest

Adding control structures in manifests

Using variables

Variable types

Controlling the order of evaluation

Declaring dependencies

Error propagation

Avoiding circular dependencies

Implementing resource interaction

Examining the most notable resource types

The user and group types

The exec resource type

The cron resource type

The mount resource type

Summary

2. The Master and Its Agents

The Puppet Master

Puppet Master and Puppet Server

Setting up the master machine

Creating the master manifest

Inspecting the configuration settings

Setting up the Puppet Agent

The agent's life cycle

Renewing an agent's certificate

Running the agent from cron

Performance considerations

Tuning puppetserver

Using Phusion Passenger with Nginx

Comparing Passenger with puppetserver

Completing the stack with PuppetDB

Troubleshooting SSL issues

Summary

3. A Peek under the Hood – Facts, Types, and Providers

Summarizing systems with Facter

Accessing and using fact values

Extending Facter with custom facts

Simplifying things using external facts

Goals of Facter

Understanding the type system

The resource type's life cycle on the agent side

Substantiating the model with providers

Resource types with generic providers

Summarizing types and providers

Putting it all together

Summary

4. Modularizing Manifests with Classes and Defined Types

Introducing classes and defined types

Defining and declaring classes

Creating and using defined types

Understanding and leveraging the differences

Structured design patterns

Writing comprehensive classes

Writing component classes

Using defined types as resource wrappers

Using defined types as resource multiplexers

Using defined types as macros

Exploiting array values using defined types

Using iterator functions

Including classes from defined types

Establishing relationships among containers

Passing events between classes and defined types

Ordering containers

Limitations

The performance implications of container relationships

Mitigating the limitations

The anchor pattern

The contain function

Making classes more flexible through parameters

The caveats of parameterized classes

Preferring the include keyword

Summary

5. Extending Your Puppet Infrastructure with Modules

An overview of Puppet's modules

Parts of a module

Module structure

Documentation in modules

Maintaining environments

Configuring environment locations

Obtaining and installing modules

Modules' best practices

Putting everything in modules

Avoiding generalization

Testing your modules

Safe testing with environments

Building a specific module

Naming your module

Making your module available to Puppet

Implementing basic module functionality

Creating utilities for derived manifests

Adding configuration items

Allowing customization

Removing unwanted configuration items

Dealing with complexity

Enhancing the agent through plugins

Replacing a defined type with a native type

Naming your type

Creating the resource type interface

Designing sensible parameter hooks

Using resource names

Adding a provider

Declaring management commands

Implementing the basic functionality

Allowing the provider to prefetch existing resources

Making the type robust during provisioning

Enhancing Puppet's system knowledge through facts

Refining the interface of your module through custom functions

Making your module portable across platforms

Finding helpful Forge modules

Identifying modules' characteristics

Summary

6. Leveraging the Full Toolset of the Language

Templating dynamic configuration files

Learning the template syntax

Using templates in practice

Avoiding performance bottlenecks from templates

Creating virtual resources

Realizing resources more flexibly using collectors

Exporting resources to other agents

Exporting and importing resources

Configuring the master to store exported resources

Exporting SSH host keys

Managing hosts files locally

Automating custom configuration items

Simplifying the configuration of Nagios

Maintaining your central firewall

Removing obsolete exports

Overriding resource parameters

Saving redundancy using resource defaults

Avoiding antipatterns

Summary

7. New Features from Puppet 4

Upgrading to Puppet 4

Using Puppet 3.8 and environment directories

The new Puppet 4 Master

Updating the Puppet agent

Testing Puppet DSL code

Using the type system

Learning lambdas and functions

Creating Puppet 4 functions

Leveraging the new template engine

Handling multiline with HEREDOC

Breaking old practices

Converting node inheritance

Dealing with bool algebra on Strings

Using strict variable naming

Learning the new reference syntax

Cleaning hyphens in names

No Ruby DSL anymore

Relative class name resolution

Dealing with different data types

Summary

8. Separating Data from Code Using Hiera

Understanding the need for separate data storage

Consequences of defining data in the manifest

Structuring configuration data in a hierarchy

Configuring Hiera

Storing Hiera data

Choosing your backends

Retrieving and using Hiera values in manifests

Working with simple values

Binding class parameter values automatically

Handling hashes and arrays

Converting resources to data

Choosing between manifest and Hiera designs

A practical example

Debugging Hiera lookups

Implementing the Roles and Profiles pattern

Summary

2. Module 2

1. Puppet Essentials

The Puppet ecosystem

Why configuration management matters

Puppet components

Installing and configuring Puppet

Puppet in action

Resources

The resource abstraction layer

Nodes

Classes and defines

Class inheritance

Resource defaults

Resource references

Variables, facts, and scopes

System's facts

External facts

User variables in Puppet DSL

Type-checking

User variables in an ENC

User variables in Hiera

Puppet built-in variables

Variables scope

Meta parameters

Managing order and dependencies

Run stages

Reserved names and allowed characters

Conditionals

Comparison operators

Iteration and lambdas

The in operator

Expressions combinations

Exported resources

Virtual resources

Modules

Paths of a module and auto loading

Templates

Restoring files from a filebucket

Summary

2. Managing Puppet Data with Hiera

Installing and configuring Hiera

The hiera.yaml configuration file

Global settings

Backend specific settings

Examples

Working with the command line on a YAML backend

Using Hiera in Puppet

Dealing with hashes in Puppet code

Puppet 3 automatic parameter lookup

Evolving usage patterns for class parameters

Additional Hiera backends

Hiera-file

Encryption plugins

Hiera-gpg

Hiera-eyaml

Hiera-http, hiera-mysql

Using Hiera as an ENC

Summary

3. Introducing PuppetDB

Installation and configuration

Installing PuppetDB

PuppetDB configurations

Init script configuration

Configuration settings

Logging configuration

Configurations on the Puppet Master

Dashboards

PuppetDB performance dashboard

Puppetboard—Querying PuppetDB from the Web

PuppetDB API

Querying PuppetDB (read)

PuppetDB commands (write)

Querying PuppetDB for fun and profit

/facts endpoint

/resources endpoint

/nodes endpoint

/catalogs endpoint

/fact-names endpoint

/metrics endpoint

/reports endpoint

/events endpoint

/event-counts endpoint

/aggregated-event-counts endpoint

/server-time endpoint

/version endpoint

The puppetdbquery module

Query format

Querying from the command line

Querying from Puppet manifests

The PuppetDB Hiera backend

How Puppet code may change in the future

Summary

4. Designing Puppet Architectures

Components of a Puppet architecture

Definition of the classes to include in each node

Definition of the parameters to use for each node

Definition of the configuration files provided to the nodes

Definition of custom resources and classes

The Foreman

Roles and profiles

The data and the code

Sample architectures

The default approach

Basic ENC, logic on site module, Hiera backend

The Foreman and Hiera

Hiera-based setup

Foreman smart variables

Facts-driven truths

Nodeless site.pp

Summary

5. Using and Writing Reusable Modules

Modules layout evolution

Class parameters—from zero to data bindings

Params pattern

Data in modules

Files and class names

The anchor pattern

The parameters dilemma

Naming standards

Reusability patterns

Managing files

Managing configuration hash patterns

Managing multiple configuration files

Managing users and dependencies

Managing installation options

Managing extra resources

Summary

6. Higher Abstraction Modules

The OpenStack example

Component (application) modules

Raising abstraction – the official OpenStack module

Raising abstraction – Puppet Labs OpenStack module

Taking an alternate approach

An approach to reusable stack modules

Tiny Puppet

Summary

7. Puppet Migration Patterns

Examining potential scenarios and approaches

New infrastructures

Existing manually managed infrastructures

Node migration

Node update

Existing automated infrastructures

Patterns for extending Puppet coverage

Raising the bar, step by step

Knowing the territory

Defining priorities

Evaluating solutions

Coding

Applying changes

Things change

Puppet friendly infrastructure

Puppet-friendly software

Summary

8. Code Workflow Management

Write Puppet code

Geppetto

Vim

Git workflows

Code management using Git

Git hooks

Environments and branches

Branch based automatic environments

Simplified developer workdir environments

Code review

Gerrit

Online resources for peer review

Testing Puppet code

Using rspec-puppet

Vagrant

Beaker and beaker-rspec

Deploying Puppet code

Using librarian-puppet for deployments

Deploying code with r10k

Propagating Puppet changes

Puppet continuous integration

Travis

Jenkins

Summary

9. Scaling Puppet Infrastructures

Scaling Puppet

As simple as puppet apply

Default Puppet Master

Puppet Master with Passenger

Installing and configuring Passenger

Puppet Master based on Trapperkeeper

Multi-Master scaling

Managing certificates

Managing SSL termination

Managing code and data

Load balancing alternatives

Masterless Puppet

Configurations and infrastructure optimizations

Traffic compression

Caching

Distributing Puppet run times

File system changes check

Scaling stored configs

Measuring performance

Puppet Metrics

Optimizing code

Testing different Puppet versions

Summary

10. Extending Puppet

Anatomy of a Puppet run, under the hood

Puppet extension alternatives

ENC and Hiera Extendibility

Modules pluginsync

Puppet indirector and its termini

Custom functions

Custom facts

Ruby facts distributed via pluginsync

External facts in facts.d directory

Custom types and providers

Custom report handlers

Custom faces

Summary

11. Beyond the System

Puppet on network equipment

Proxy mode with Puppet device application

Native Puppet on network equipment

Cisco onePK

Juniper and the netdev_stdlib

Puppet for cloud and virtualization

VMware

VM provisioning on vCenter and vSphere

vCenter configuration

vSphere virtual machine management with resource types

Amazon Web Services

Cloud provisioning on AWS

AWS provisioning and configuration with resource types

Managing CloudFormation

Cloud provisioning on Google Compute Engine

Puppet on storage devices

Puppet and Docker

Summary

12. Future Puppet

Changing the serialization format

Direct Puppet

File sync

Other changes

Certificate authority in Clojure

Package management

Changes in faces

Beyond Puppet 4.x

Summary

3. Module 3

1. Dealing with Load/Scale

Divide and conquer

Certificate signing

Reporting

Storeconfigs

Catalog compilation

puppetserver

Building a Puppet master

Certificates

systemd

Creating a load balancer

Apache proxy

TLS headers

SELinux

Configuring masters

Keeping the code consistent

rsync

NFS

Clustered filesystem

Git

One more split

One last split or maybe a few more

Conquer by dividing

Creating an rpm

Using Puppet resource to configure cron

Creating the yum repository

Summary

2. Organizing Your Nodes and Data

Getting started

Organizing the nodes with an ENC

A simple example

Hostname strategy

Modified ENC using hostname strategy

LDAP backend

OpenLDAP configuration

Hiera

Configuring Hiera

Using hiera_include

Summary

3. Git and Environments

Environments

Environments and Hiera

Multiple hierarchies

Single hierarchy for all environments

Directory environments

Git

Why Git?

A simple Git workflow

Git hooks

Using post-receive to set up environments

Puppet-sync

Using Git hooks to play nice with other developers

Not playing nice with others via Git hooks

Git for everyone

Summary

4. Public Modules

Getting modules

Using GitHub for public modules

Updating the local repository

Modules from the Forge

Using Librarian

Using r10k

Using Puppet-supported modules

concat

inifile

firewall

Logical volume manager

Standard library

Summary

5. Custom Facts and Modules

Module manifest files

Module files and templates

Naming a module

Creating modules with a Puppet module

Comments in modules

Multiple definitions

Custom facts

Creating custom facts

Creating a custom fact for use in Hiera

CFacter

Summary

6. Custom Types

Parameterized classes

Data types

Defined types

Types and providers

Creating a new type

Summary

7. Reporting and Orchestration

Turning on reporting

Store

Logback

Internet relay chat

Foreman

Installing Foreman

Attaching Foreman to Puppet

Using Foreman

Puppet GUIs

mcollective

Installing ActiveMQ

Configuring nodes to use ActiveMQ

Connecting a client to ActiveMQ

Using mcollective

Ansible

Summary

8. Exported Resources

Configuring PuppetDB – using the Forge module

Manually installing PuppetDB

Installing Puppet and PuppetDB

Installing and configuring PostgreSQL

Configuring puppetdb to use PostgreSQL

Configuring Puppet to use PuppetDB

Exported resource concepts

Declaring exported resources

Collecting exported resources

Simple example – a host entry

Resource tags

Exported SSH keys

sshkey collection for laptops

Putting it all together

Summary

9. Roles and Profiles

Design pattern

Creating an example CDN role

Creating a sub-CDN role

Dealing with exceptions

Summary

10. Troubleshooting

Connectivity issues

Catalog failures

Full trace on a catalog compilation

The classes.txt file

Debugging

Personal and bugfix branches

Echo statements

Scope

Profiling and summarizing

Summary

A. Bibliography

Index