Learning ELK Stack电子书

Learning ELK Stack

Table of Contents

Learning ELK Stack

Credits

About the Author

About the Reviewers

www.PacktPub.com

Support files, eBooks, discount offers, and more

Why subscribe?

Free access for Packt account holders

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Downloading the color images of this book

Errata

Piracy

Questions

1. Introduction to ELK Stack

The need for log analysis

Issue debugging

Performance analysis

Security analysis

Predictive analysis

Internet of things and logging

Challenges in log analysis

Non-consistent log format

Tomcat logs

Apache access logs – combined log format

IIS logs

Variety of time formats

Decentralized logs

Expert knowledge requirement

The ELK Stack

Elasticsearch

Logstash

Kibana

ELK data pipeline

ELK Stack installation

Installing Elasticsearch

Running Elasticsearch

Elasticsearch configuration

Network Address

Paths

The cluster name

The node name

Elasticsearch plugins

Installing Logstash

Running Logstash

Logstash with file input

Logstash with Elasticsearch output

Configuring Logstash

Installing Logstash forwarder

Logstash plugins

Input plugin

Filters plugin

Output plugin

Installing Kibana

Configuring Kibana

Running Kibana

Kibana interface

Discover

Visualize

Dashboard

Settings

Summary

2. Building Your First Data Pipeline with ELK

Input dataset

Data format for input dataset

Configuring Logstash input

Filtering and processing input

Putting data to Elasticsearch

Visualizing with Kibana

Running Kibana

Kibana visualizations

Building a line chart

Building a bar chart

Building a Metric

Building a data table

Summary

3. Collect, Parse and Transform Data with Logstash

Configuring Logstash

Logstash plugins

Listing all plugins in Logstash

Data types for plugin properties

Array

Boolean

Codec

Hash

String

Comments

Field references

Logstash conditionals

Types of Logstash plugins

Input plugins

file

Configuration options

add_field

codec

delimiter

exclude

path

sincedb_path

sincedb_write_interval

start_position

tags

type

stdin

Configuration options

add_field

codec

tags

type

twitter

Configuration options

add_field

codec

consumer_key

consumer_secret

full_tweet

keywords

oauth_token

oauth_token_secret

tags

type

lumberjack

Configuration options

add_field

codec

host

port

ssl_certificate

ssl_key

ssl_key_passphrase

tags

type

redis

Configuration options

add_field

codec

data_type

host

key

password

port

Output plugins

csv

Configuration options

codec

csv_options

fields

gzip

path

file

Configuration options

email

Configuration options

attachments

body

cc

from

to

htmlbody

replyto

subject

elasticsearch

Configuration options

ganglia

Configuration options

metric

unit

value

jira

Configuration options

kafka

Configuration options

topic_id

lumberjack

Configuration options

hosts

port

ssl_certificate

redis

Configuration options

rabbitmq

stdout

mongodb

Configuration options

collection

database

uri

Filter plugins

csv

Configuration options

date

Configuration options

drop

Configuration options

geoip

Configuration options

source

grok

Custom grok patterns

mutate

Configuration options

sleep

Codec plugins

json

line

multiline

plain

rubydebug

Summary

4. Creating Custom Logstash Plugins

Logstash plugin management

Plugin lifecycle management

Installing a plugin

Updating a plugin

Uninstalling a plugin

Structure of a Logstash plugin

Required dependencies

Class declaration

Configuration name

Configuration options setting

Plugin methods

Input plugin

Filter plugin

Output plugin

Codec plugin

Writing a Logstash filter plugin

Building the plugin

Summary

5. Why Do We Need Elasticsearch in ELK?

Why Elasticsearch?

Elasticsearch basic concepts

Index

Document

Field

Type

Mapping

Shard

Primary shard and replica shard

Cluster

Node

Exploring the Elasticsearch API

Listing all available indices

Listing all nodes in a cluster

Checking the health of the cluster

Health status of the cluster

Creating an index

Retrieving the document

Deleting documents

Deleting an index

Elasticsearch Query DSL

Elasticsearch plugins

Bigdesk plugin

Elastic-Hammer plugin

Head plugin

Summary

6. Finding Insights with Kibana

Kibana 4 features

Search highlights

Elasticsearch aggregations

Scripted fields

Dynamic dashboards

Kibana interface

Discover page

Time filter

Quick time filter

Relative time filter

Absolute time filter

Kibana Auto-refresh setting

Querying and searching data

Freetext search

AND

OR

NOT

Groupings

Wildcard searches

Field searches

Range searches

Special characters escaping

New search

Saving the search

Loading a search

Field searches using field list

Summary

7. Kibana – Visualization and Dashboard

Visualize page

Creating a visualization

Visualization types

Metrics and buckets aggregations

Buckets

Date Histogram

Histogram

Range

Date Range

Terms

Metrics

Count

Average, Sum, Min, and Max

Unique Count

Advanced options

Visualizations

Area chart

Data table

Line chart

Markdown widget

Metric

Pie chart

Tile map

Vertical bar chart

Dashboard page

Building a new dashboard

Saving and loading a dashboard

Sharing a dashboard

Summary

8. Putting It All Together

Input dataset

Configuring Logstash input

Grok pattern for access logs

Visualizing with Kibana

Running Kibana

Searching on the Discover page

Visualizations – charts

Building a Line chart

Building an Area chart

Building a Bar chart

Building a Markdown

Dashboard page

Summary

9. ELK Stack in Production

Prevention of data loss

Data protection

System scalability

Data retention

ELK Stack implementations

ELK Stack at LinkedIn

Problem statement

Criteria for solution

Solution

Kafka at LinkedIn

Operational challenges

Logging using Kafka at LinkedIn

ELK at SCA

How is ELK used in SCA?

How is it helping in analytics?

ELK for monitoring at SCA

ELK at Cliffhanger Solutions

Kibana demo – Packetbeat dashboard

Summary

10. Expanding Horizons with ELK

Elasticsearch plugins and utilities

Curator for index management

Curator commands

Curator installation

Shield for security

Shield installation

Adding users and roles

Using Kibana4 on shield protected Elasticsearch

Marvel to monitor

Marvel installation

Marvel dashboards

ELK roadmap

Elasticsearch roadmap

Logstash roadmap

Event persistence capability

End-to-end message acknowledgement

Logstash monitoring and management API

Kibana roadmap

Summary

Index