Learning Elastic Stack 6.0电子书

Title Page

Copyright

Learning Elastic Stack 6.0

Credits

Disclaimer

About the Authors

About the Reviewer

www.PacktPub.com

Why subscribe?

Customer Feedback

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Downloading the color images of this book

Errata

Piracy

Questions

Introducing Elastic Stack

What is Elasticsearch, and why use it?

Schemaless and document-oriented

Searching

Analytics

Rich client library support and the REST API

Easy to operate and easy to scale

Near real time

Lightning fast

Fault tolerant

Exploring the components of Elastic Stack

Elasticsearch

Logstash

Beats

Kibana

X-Pack

Security

Monitoring

Reporting

Alerting

Graph

Elastic Cloud

Use cases of Elastic Stack

Log and security analytics

Product search

Metrics analytics

Web search and website search

Downloading and installing

Installing Elasticsearch

Installing Kibana

Summary

Getting Started with Elasticsearch

Using the Kibana Console UI

Core concepts

Index

Type

Document

Node

Cluster

Shards and replicas

Mappings and data types

Data types

Core datatypes

Complex datatypes

Other datatypes

Mappings

Creating an index with the name catalog

Defining the mappings for the type of product

Inverted index

CRUD operations

Index API

Indexing a document by providing an ID

Indexing a document without providing an ID

Get API

Update API

Delete API

Creating indexes and taking control of mapping

Creating an index

Creating type mapping in an existing index

Updating a mapping

REST API overview

Common API conventions

Formatting the JSON response

Dealing with multiple indices

Searching all documents in one index

Searching all documents in multiple indexes

Searching all documents of a particular type in all indices

Summary

Searching-What is Relevant

Basics of text analysis

Understanding Elasticsearch analyzers

Character filters

Tokenizer

Standard Tokenizer

Token filters

Using built-in analyzers

Standard Analyzer

Implementing autocomplete with a custom analyzer

Searching from structured data

Range query

Range query on numeric types

Range query with score boosting

Range query on dates

Exists query

Term query

Searching from full text

Match query

Operator

minimum_should_match

Fuzziness

Match phrase query

Multi match query

Querying multiple fields with defaults

Boosting one or more fields

With types of multi match queries

Writing compound queries

Constant score query

Bool query

Combining OR conditions

Combining conditions AND and OR conditions

Adding NOT conditions

Summary

Analytics with Elasticsearch

The basics of aggregations

Bucket aggregations

Metric aggregations

Matrix aggregations

Pipeline aggregations

Preparing data for analysis

Understanding the structure of data

Loading the data using Logstash

Metric aggregations

Sum, average, min, and max aggregations

Sum aggregation

Average aggregation

Min aggregation

Max aggregation

Stats and extended stats aggregations

Stats aggregation

Extended stats Aggregation

Cardinality aggregation

Bucket aggregations

Bucketing on string data

Terms aggregation

Bucketing on numeric data

Histogram aggregation

Range aggregation

Aggregations on filtered data

Nesting aggregations

Bucketing on custom conditions

Filter aggregation

Filters aggregation

Bucketing on date/time data

Date Histogram aggregation

Creating buckets across time

Using a different time zone

Computing other metrics within sliced time intervals

Focusing on a specific day and changing intervals

Bucketing on geo-spatial data

Geo distance aggregation

GeoHash grid aggregation

Pipeline aggregations

Calculating the cumulative sum of usage over time

Summary

Analyzing Log Data

Log analysis challenges

Logstash

Installation and configuration

Prerequisites

Downloading and installing Logstash

Installing on Windows

Installing on Linux

Running Logstash

Logstash architecture

Overview of Logstash plugins

Installing or updating plugins

Input plugins

Output plugins

Filter plugins

Codec plugins

Exploring plugins

Exploring Input plugins

File

Beats

JDBC

IMAP

Output plugins

Elasticsearch

CSV

Kafka

PagerDuty

Codec plugins

JSON

Rubydebug

Multiline

Filter plugins

Ingest node

Defining a pipeline

Ingest APIs

Put pipeline API

Get Pipeline API

Delete pipeline API

Simulate pipeline API

Summary

Building Data Pipelines with Logstash

Parsing and enriching logs using Logstash

Filter plugins

CSV filter

Mutate filter

Grok filter

Date filter

Geoip filter

Useragent filter

Introducing Beats

Beats by Elastic.co

Filebeat

Metricbeat

Packetbeat

Heartbeat

Winlogbeat

Auditbeat

Community Beats

Logstash versus Beats

Filebeat

Downloading and installing Filebeat

Installing on Windows

Installing on Linux

Architecture

Configuring Filebeat

Filebeat prospectors

Filebeat global options

Filebeat general options

Output configuration

Filebeat modules

Summary

Visualizing data with Kibana

Downloading and installing Kibana

Installing on Windows

Installing on Linux

Configuring Kibana

Data preparation

Kibana UI

User interaction

Configuring the index pattern

Discover

Elasticsearch query string

Elasticsearch DSL query

Visualize

Kibana aggregations

Bucket aggregations

Metric

Creating a visualization

Visualization types

Line, area, and bar charts

Data table

MarkDown widget

Metric

Goal

Gauge

Pie charts

Co-ordinate maps

Region maps

Tag cloud

Visualizations in action

Response codes over time

Top 10 URLs requested

Bandwidth usage of top five countries over time

Web traffic originating from different countries

Most used user agent

Dashboards

Creating a dashboard

Saving the dashboard

Cloning the dashboard

Sharing the dashboard

Timelion

Timelion UI

Timelion expressions

Using plugins

Installing plugins

Removing plugins

Summary

Elastic X-Pack

Installing X-Pack

Installing X-Pack on Elasticsearch

Installing X-Pack on Kibana

Uninstalling X-Pack

Configuring X-Pack

Security

User authentication

User authorization

Security in action

New user creation

Deleting a user

Changing the password

New role creation

How to Delete/Edit a role

Document-level security or field-level security

X-Pack security APIs

User management APIs

Role management APIs

Monitoring Elasticsearch

Monitoring UI

Elasticsearch metrics

Overview tab

Nodes tab

The Indices tab

Alerting

Anatomy of a watch

Alerting in action

Create a new alert

Threshold Alert

Advanced Watch

How to Delete/Deactivate/Edit a Watch

Summary

Running Elastic Stack in Production

Hosting Elastic Stack on a managed cloud

Getting up and running on Elastic Cloud

Using Kibana

Overriding configuration

Recovering from a snapshot

Hosting Elastic Stack on your own

Selecting hardware

Selecting an operating system

Configuring Elasticsearch nodes

JVM heap size

Disable swapping

File descriptors

Thread pools and garbage collector

Managing and monitoring Elasticsearch

Running in Docker containers

Special considerations while deploying to a cloud

Choosing instance type

Changing default ports; do not expose ports!

Proxy requests

Binding HTTP to local addresses

Installing EC2 discovery plugin

Installing S3 repository plugin

Setting up periodic snapshots

Backing up and restoring

Setting up a repository for snapshots

Shared filesystem

Cloud or distributed filesystems

Taking snapshots

Restoring a specific snapshot

Setting up index aliases

Understanding index aliases

How index aliases can help

Setting up index templates

Defining an index template

Creating indexes on the fly

Modeling time series data

Scaling the index with unpredictable volume over time

Unit of parallelism in Elasticsearch

The effect of the number of shards on the relevance score

The effect of the number of shards on the accuracy of aggregations

Changing the mapping over time

New fields get added

Existing fields get removed

Automatically deleting older documents

How index-per-timeframe solves these issues

Scaling with index-per-timeframe

Changing the mapping over time

Automatically deleting older documents

Summary

Building a Sensor Data Analytics Application

Introduction to the application

Understanding the sensor-generated data

Understanding the sensor metadata

Understanding the final stored data

Modeling data in Elasticsearch

Defining an index template

Understanding the mapping

Setting up the metadata database

Building the Logstash data pipeline

Accept JSON requests over the web

Enrich the JSON with the metadata we have in the MySQL database

The jdbc_streaming plugin

The mutate plugin

Move the looked-up fields that are under lookupResult directly in JSON

Combine the latitude and longitude fields under lookupResult as a location field

Remove the unnecessary fields

Store the resulting documents in Elasticsearch

Sending data to Logstash over HTTP

Visualizing the data in Kibana

Set up an index pattern in Kibana

Build visualizations

How does the average temperature change over time?

How does the average humidity change over time?

How do temperature and humidity change at each location over time?

Can I visualize temperature and humidity over a map?

How are the sensors distributed across departments?

Create a dashboard

Summary

Monitoring Server Infrastructure

Metricbeat

Downloading and installing Metricbeat

Installing on Windows

Installing on Linux

Architecture

Event structure

Configuring Metricbeat

Module configuration

Enabling module configs in the modules.d directory

Enabling module config in the metricbeat.yml file

General settings

Output configuration

Logging

Capturing system metrics

Running Metricbeat with the system module

Specifying aliases

Visualizing system metrics using Kibana

Deployment architecture

Summary