Mastering Distributed Tracing电子书

Mastering Distributed Tracing

Mastering Distributed Tracing

Why subscribe?

Packt.com

Contributors

About the author

About the reviewer

About the illustrator

Packt is searching for authors like you

Preface

Who this book is for

What this book covers

To get the most out of this book

Download the example code files

Download the color images

Conventions used

Get in touch

Reviews

I. Introduction

1. Why Distributed Tracing?

Microservices and cloud-native applications

What is observability?

The observability challenge of microservices

Traditional monitoring tools

Metrics

Logs

Distributed tracing

My experience with tracing

Why this book?

Summary

References

2. Take Tracing for a HotROD Ride

Prerequisites

Running from prepackaged binaries

Running from Docker images

Running from the source code

Go language development environment

Jaeger source code

Start Jaeger

Meet the HotROD

The architecture

The data flow

Contextualized logs

Span tags versus logs

Identifying sources of latency

Resource usage attribution

Summary

References

3. Distributed Tracing Fundamentals

The idea

Request correlation

Black-box inference

Schema-based

Metadata propagation

Anatomy of distributed tracing

Sampling

Preserving causality

Inter-request causality

Trace models

Event model

Span model

Clock skew adjustment

Trace analysis

Summary

References

II. Data Gathering Problem

4. Instrumentation Basics with OpenTracing

Prerequisites

Project source code

Go development environment

Java development environment

Python development environment

MySQL database

Query tools (curl or wget)

Tracing backend (Jaeger)

OpenTracing

Exercise 1 – the Hello application

Hello application in Go

Hello application in Java

Hello application in Python

Exercise summary

Exercise 2 – the first trace

Step 1 – create a tracer instance

Create a tracer in Go

Create a tracer in Java

Create a tracer in Python

Step 2 – start a span

Start a span in Go

Start a span in Java

Start a span in Python

Step 3 – annotate the span

Annotate the span in Go

Annotate the span in Java

Annotate the span in Python

Exercise summary

Exercise 3 – tracing functions and passing context

Step 1 – trace individual functions

Trace individual functions in Go

Trace individual functions in Java

Trace individual functions in Python

Step 2 – combine multiple spans into a single trace

Combine multiple spans into a single trace in Go

Combine multiple spans into a single trace in Java

Combine multiple spans into a single trace in Python

Step 3 – propagate the in-process context

In-process context propagation in Python

In-process context propagation in Java

In-process context propagation in Go

Exercise summary

Exercise 4 – tracing RPC requests

Step 1 – break up the monolith

Microservices in Go

Microservices in Java

Microservices in Python

Step 2 – pass the context between processes

Passing context between processes in Go

Passing context between processes in Java

Passing context between processes in Python

Step 3 – apply OpenTracing-recommended tags

Standard tags in Go

Standard tags in Java

Standard tags in Python

Exercise summary

Exercise 5 – using baggage

Using baggage in Go

Using baggage in Java

Using baggage in Python

Exercise summary

Exercise 6 – auto-instrumentation

Open source instrumentation in Go

Auto-instrumentation in Java

Auto-instrumentation in Python

Exercise 7 – extra credit

Summary

References

5. Instrumentation of Asynchronous Applications

Prerequisites

Project source code

Java development environment

Kafka, Zookeeper, Redis, and Jaeger

The Tracing Talk chat application

Implementation

The lib module

AppId

Message

KafkaConfig and KafkaService

RedisConfig and RedisService

GiphyService

The chat-api service

The storage-service microservice

The giphy-service microservice

Running the application

Observing traces

Instrumenting with OpenTracing

Spring instrumentation

Tracer resolver

Redis instrumentation

Kafka instrumentation

Producing messages

Consuming messages

Instrumenting asynchronous code

Summary

References

6. Tracing Standards and Ecosystem

Styles of instrumentation

Anatomy of tracing deployment and interoperability

Five shades of tracing

Know your audience

The ecosystem

Tracing systems

Zipkin and OpenZipkin

Jaeger

SkyWalking

X-Ray, Stackdriver, and more

Standards projects

W3C Trace Context

W3C "Data Interchange Format"

OpenCensus

OpenTracing

Summary

References

7. Tracing with Service Meshes

Service meshes

Observability via a service mesh

Prerequisites

Project source code

Java development environment

Kubernetes

Istio

The Hello application

Distributed tracing with Istio

Using Istio to generate a service graph

Distributed context and routing

Summary

References

8. All About Sampling

Head-based consistent sampling

Probabilistic sampling

Rate limiting sampling

Guaranteed-throughput probabilistic sampling

Adaptive sampling

Local adaptive sampling

Global adaptive sampling

Goals

Theory

Architecture

Calculating sampling probability

Implications of adaptive sampling

Extensions

Context-sensitive sampling

Ad-hoc or debug sampling

How to deal with oversampling

Post-collection down-sampling

Throttling

Tail-based consistent sampling

Partial sampling

Summary

References

III. Getting Value from Tracing

9. Turning the Lights On

Tracing as a knowledge base

Service graphs

Deep, path-aware service graphs

Detecting architectural problems

Performance analysis

Critical path analysis

Recognizing trace patterns

Look for error markers

Look for the longest span on the critical path

Look out for missing details

Avoid sequential execution or "staircase"

Be wary when things finish at exactly the same time

Exemplars

Latency histograms

Long-term profiling

Summary

References

10. Distributed Context Propagation

Brown Tracing Plane

Pivot tracing

Chaos engineering

Traffic labeling

Testing in production

Debugging in production

Developing in production

Summary

References

11. Integration with Metrics and Logs

Three pillars of observability

Prerequisites

Project source code

Java development environment

Running the servers in Docker

Declaring index pattern in Kibana

Running the clients

The Hello application

Integration with metrics

Standard metrics via tracing instrumentation

Adding context to metrics

Context-aware metrics APIs

Integration with logs

Structured logging

Correlating logs with trace context

Context-aware logging APIs

Capturing logs in the tracing system

Do we need separate logging and tracing backends?

Summary

References

12. Gathering Insights with Data Mining

Feature extraction

Components of a data mining pipeline

Tracing backend

Trace completion trigger

Feature extractor

Aggregator

Feature extraction exercise

Prerequisites

Project source code

Running the servers in Docker

Defining index mapping in Elasticsearch

Java development environment

Microservices simulator

Running as a Docker image

Running from source

Verify

Define an index pattern in Kibana

The Span Count job

Trace completion trigger

Feature extractor

Observing trends

Beware of extrapolations

Historical analysis

Ad hoc analysis

Summary

References

IV. Deploying and Operating Tracing Infrastructure

13. Implementing Tracing in Large Organizations

Why is it hard to deploy tracing instrumentation?

Reduce the barrier to adoption

Standard frameworks

In-house adapter libraries

Tracing enabled by default

Monorepos

Integration with existing infrastructure

Where to start

Building the culture

Explaining the value

Integrating with developer workflows

Tracing Quality Metrics

Troubleshooting guide

Don't be on the critical path

Summary

References

14. Under the Hood of a Distributed Tracing System

Why host your own?

Customizations and integrations

Bandwidth cost

Own the data

Bet on emerging standards

Architecture and deployment modes

Basic architecture: agent + collector + query service

Client

Agent

Collector

Query service and UI

Data mining jobs

Streaming architecture

Multi-tenancy

Cost accounting

Complete isolation

Granular access controls

Security

Running in multiple DCs

Capturing origin zone

Cross-zone federation

Monitoring and troubleshooting

Resiliency

Over-sampling

Debug traces

Traffic spikes due to DC failover

Perpetual traces

Very long traces

Summary

References

15. Afterword

References

Other Books You May Enjoy

Leave a review - let other readers know what you think

Index