Hands-On Infrastructure Monitoring with Prometheus电子书

About Packt

Why subscribe?

Packt.com

Contributors

About the authors

About the reviewer

Packt is searching for authors like you

Preface

Introduction to the book and the technology

Who this book is for

What this book covers

To get the most out of this book

Download the example code files

Download the color images

Conventions used

Get in touch

Reviews

Section 1: Introduction

Monitoring Fundamentals

Defining of monitoring

The value of monitoring

Organizational contexts

Monitoring components

Whitebox versus blackbox monitoring

Understanding metrics collection

An overview of the two collection approaches

Push versus pull

What to measure

Google's four golden signals

Brendan Gregg's USE method

Tom Wilkie's RED method

Summary

Questions

Further reading

An Overview of the Prometheus Ecosystem

Metrics collection with Prometheus

High-level overview of the Prometheus architecture

Exposing internal state with exporters

Exporter fundamentals

Alert routing and management with Alertmanager

Alerting routes

Visualizing your data

Summary

Questions

Further reading

Setting Up a Test Environment

Code organization

Machine requirements

Hardware requirements

Recommended software

VirtualBox

Vagrant

Minikube

kubectl

Spinning up a new environment

Automated deployment walkthrough

Prometheus

Grafana

Alertmanager

Cleanup

Advanced deployment walkthrough

Prometheus

Grafana

Alertmanager

Node Exporter

Validating your test environment

Summary

Questions

Further reading

Section 2: Getting Started with Prometheus

Prometheus Metrics Fundamentals

Understanding the Prometheus data model

Time series data

Time series databases

Prometheus local storage

Data flow

Memory

Write ahead log

Disk

Layout

Prometheus data model

Notation

Metric names

Metric labels

Samples

Cardinality

A tour of the four core metric types

Counter

Gauge

Histogram

Summaries

Longitudinal and cross-sectional aggregations

Cross-sectional aggregation

Longitudinal aggregation

Summary

Questions

Further reading

Running a Prometheus Server

Deep dive into the Prometheus configuration

Prometheus startup configuration

The config section

The storage section

The web section

The query section

Prometheus configuration file walkthrough

Global configuration

Scrape configuration

Managing Prometheus in a standalone server

Server deploy

Configuration inspection

Cleanup

Managing Prometheus in Kubernetes

Static configuration

Kubernetes environment

Prometheus server deployment

Adding targets to Prometheus

Dynamic configuration – the Prometheus Operator

Kubernetes environment

Prometheus Operator deployment

Prometheus server deployment

Adding targets to Prometheus

Summary

Questions

Further reading

Exporters and Integrations

Test environments for this chapter

Static infrastructure test environment

Kubernetes test environment

Operating system exporter

The Node Exporter

Configuration

Deployment

Container exporter

cAdvisor

Configuration

Deployment

kube-state-metrics

Configuration

Deployment

From logs to metrics

mtail

Configuration

Deployment

Grok exporter

Configuration

Deployment

Blackbox monitoring

Blackbox exporter

Configuration

Deployment

Pushing metrics

Pushgateway

Configuration

Deployment

More exporters

JMX exporter

HAProxy exporter

Summary

Questions

Further reading

Prometheus Query Language - PromQL

The test environment for this chapter

Getting to know the basics of PromQL

Selectors

Label matchers

Instant vectors

Range vectors

The offset modifier

Subqueries

Operators

Binary operators

Arithmetic

Comparison

Vector matching

One-to-one

Many-to-one and one-to-many

Logical operators

Aggregation operators

Binary operator precedence

Functions

absent()

label_join() and label_replace()

predict_linear()

rate() and irate()

histogram_quantile()

sort() and sort_desc()

vector()

Aggregation operations over time

Time functions

Info and enum

Common patterns and pitfalls

Patterns

Service-level indicators

Percentiles

The health of scrape jobs

Pitfalls

Choosing the right functions for the data type

Sum-of-rates versus rate-of-sums

Having enough data to work with

Unexpected results when using increase

Not using enough matchers to select a time series

Losing statistical significance

Knowing what to expect when constructing complex queries

The query of death

Moving on to more complex queries

In which node is Node Exporter running?

Scenario rationale

PromQL approach

Comparing CPU usage across different versions

Scenario rationale

PromQL approach

Summary

Questions

Further reading

Troubleshooting and Validation

The test environment for this chapter

Deployment

Cleanup

Exploring promtool

Checks

check config

check rules

check metrics

Queries

query instant

query range

query series

query labels

Debug

debug pprof

debug metrics

debug all

Tests

Logs and endpoint validation

Endpoints

Logs

Analyzing the time series database

Using the tsdb tool

Summary

Questions

Further reading

Section 3: Dashboards and Alerts

Defining Alerting and Recording Rules

Creating the test environment

Deployment

Cleanup

Understanding how rule evaluation works

Using recording rules

Naming convention for recording rules

Setting up alerting in Prometheus

What is an alerting rule?

Configuring alerting rules

Prometheus server configuration file

Rule file configuration

Labels and annotations

Delays on alerting

Testing your rules

Recording rules tests

Alerting rules tests

Summary

Questions

Further reading

Discovering and Creating Grafana Dashboards

Test environment for this chapter

Deployment

Cleanup

How to use Grafana with Prometheus

Login screen

Data source

Explore

Dashboards

Grafana running on Kubernetes

Building your own dashboards

Dashboard fundamentals

Panels

Variables

Time picker

Creating a basic dashboard

Exporting dashboards

Discovering ready-made dashboards

Grafana dashboards gallery

Publishing your dashboards

Default Prometheus visualizations

Out-of-the-box console templates

Console template basics

Summary

Questions

Further reading

Understanding and Extending Alertmanager

Setting up the test environment

Deployment

Cleanup

Alertmanager fundamentals

The notification pipeline

Dispatching alert groups to the notification pipeline

Inhibition

Silencing

Routing

Alertmanager clustering

Alertmanager configuration

Prometheus configuration

Configuration file overview

global

route

inhibit_rules

receiver

templates

The amtool command-line tool

alert

silence

check-config

config

Kubernetes Prometheus Operator and Alertmanager

Common Alertmanager notification integrations

Email

Chat

Pager

Webhook

null

Customizing your alert notifications

Default message format

Creating a new template

Who watches the Watchmen?

Meta-monitoring and cross-monitoring

Dead man's switch alerts

Summary

Questions

Further reading

Section 4: Scalability, Resilience, and Maintainability

Choosing the Right Service Discovery

Test environment for this chapter

Deployment

Cleanup

Running through the service discovery options

Cloud providers

Container orchestrators

Service discovery systems

DNS-based service discovery

File-based service discovery

Using a built-in service discovery

Using Consul service discovery

Using Kubernetes service discovery

Building a custom service discovery

Custom service discovery fundamentals

Recommended approach

The service discovery adapter

Custom service discovery example

Using the custom service discovery

Summary

Questions

Further reading

Scaling and Federating Prometheus

Test environment for this chapter

Deployment

Cleanup

Scaling with the help of sharding

Logical grouping of jobs

The single job scale problem

What to consider when sharding

Alternatives to sharding

Having a global view using federation

Federation configuration

Federation patterns

Hierarchical

Cross-service

Using Thanos to mitigate Prometheus shortcomings at scale

Thanos' global view components

Sidecar

Query

Summary

Questions

Further reading

Integrating Long-Term Storage with Prometheus

Test environment for this chapter

Deployment

Cleanup

Remote write and remote read

Remote write

Remote read

Options for metrics storage

Local storage

Remote storage integrations

Thanos remote storage and ecosystem

Thanos ecosystem

Thanos components

Test environment specifics

Thanos query

Thanos sidecar

Thanos store gateway

Thanos compact

Thanos bucket

Thanos receive

Thanos rule

Summary

Questions

Further reading

Assessments

Chapter 1, Monitoring Fundamentals

Chapter 2, An Overview of the Prometheus Ecosystem

Chapter 3, Setting Up a Test Environment

Chapter 4, Prometheus Metrics Fundamentals

Chapter 5, Running a Prometheus Server

Chapter 6, Exporters and Integrations

Chapter 7, Prometheus Query Language - PromQL

Chapter 8, Troubleshooting and Validation

Chapter 9, Defining Alerting and Recording Rules

Chapter 10, Discovering and Creating Grafana Dashboards

Chapter 11, Understanding and Extending Alertmanager

Chapter 12, Choosing the Right Service Discovery

Chapter 13, Scaling and Federating Prometheus

Chapter 14, Integrating Long-Term Storage with Prometheus

Other Books You May Enjoy

Leave a review - let other readers know what you think