Learning Elastic Stack 7.0电子书

About Packt

Why subscribe?

Packt.com

Contributors

About the authors

About the reviewer

Packt is searching for authors like you

Preface

Who this book is for

What this book covers

To get the most out of this book

Download the example code files

Download the color images

Conventions used

Get in touch

Reviews

Section 1: Introduction to Elastic Stack and Elasticsearch

Introducing Elastic Stack

What is Elasticsearch, and why use it?

Schemaless and document-oriented

Searching capability

Analytics

Rich client library support and the REST API

Easy to operate and easy to scale

Near real-time capable

Lightning–fast

Fault-tolerant

Exploring the components of the Elastic Stack

Elasticsearch

Logstash

Beats

Kibana

X-Pack

Security

Monitoring

Reporting

Alerting

Graph

Machine learning

Elastic Cloud

Use cases of Elastic Stack

Log and security analytics

Product search

Metrics analytics

Web search and website search

Downloading and installing

Installing Elasticsearch

Installing Kibana

Summary

Getting Started with Elasticsearch

Using the Kibana Console UI

Core concepts of Elasticsearch

Indexes

Types

Documents

Nodes

Clusters

Shards and replicas

Mappings and datatypes

Datatypes

Core datatypes

Complex datatypes

Other datatypes

Mappings

Creating an index with the name catalog

Defining the mappings for the type of product

Inverted indexes

CRUD operations

Index API

Indexing a document by providing an ID

Indexing a document without providing an ID

Get API

Update API

Delete API

Creating indexes and taking control of mapping

Creating an index

Creating type mapping in an existing index

Updating a mapping

REST API overview

Common API conventions

Formatting the JSON response

Dealing with multiple indexes

Searching all documents in one index

Searching all documents in multiple indexes

Searching all the documents of a particular type in all indexes

Summary

Section 2: Analytics and Visualizing Data

Searching - What is Relevant

The basics of text analysis

Understanding Elasticsearch analyzers

Character filters

Tokenizer

Standard tokenizer

Token filters

Using built-in analyzers

Standard analyzer

Implementing autocomplete with a custom analyzer

Searching from structured data

Range query

Range query on numeric types

Range query with score boosting

Range query on dates

Exists query

Term query

Searching from the full text

Match query

Operator

Minimum should match

Fuzziness

Match phrase query

Multi match query

Querying multiple fields with defaults

Boosting one or more fields

With types of multi match queries

Writing compound queries

Constant score query

Bool query

Combining OR conditions

Combining AND and OR conditions

Adding NOT conditions

Modeling relationships

has_child query

has_parent query

parent_id query

Summary

Analytics with Elasticsearch

The basics of aggregations

Bucket aggregations

Metric aggregations

Matrix aggregations

Pipeline aggregations

Preparing data for analysis

Understanding the structure of the data

Loading the data using Logstash

Metric aggregations

Sum, average, min, and max aggregations

Sum aggregation

Average aggregation

Min aggregation

Max aggregation

Stats and extended stats aggregations

Stats aggregation

Extended stats aggregation

Cardinality aggregation

Bucket aggregations

Bucketing on string data

Terms aggregation

Bucketing on numerical data

Histogram aggregation

Range aggregation

Aggregations on filtered data

Nesting aggregations

Bucketing on custom conditions

Filter aggregation

Filters aggregation

Bucketing on date/time data

Date Histogram aggregation

Creating buckets across time periods

Using a different time zone

Computing other metrics within sliced time intervals

Focusing on a specific day and changing intervals

Bucketing on geospatial data

Geodistance aggregation

GeoHash grid aggregation

Pipeline aggregations

Calculating the cumulative sum of usage over time

Summary

Analyzing Log Data

Log analysis challenges

Using Logstash

Installation and configuration

Prerequisites

Downloading and installing Logstash

Installing on Windows

Installing on Linux

Running Logstash

The Logstash architecture

Overview of Logstash plugins

Installing or updating plugins

Input plugins

Output plugins

Filter plugins

Codec plugins

Exploring plugins

Exploring input plugins

File

Beats

JDBC

IMAP

Output plugins

Elasticsearch

CSV

Kafka

PagerDuty

Codec plugins

JSON

Rubydebug

Multiline

Filter plugins

Ingest node

Defining a pipeline

Ingest APIs

Put pipeline API

Get pipeline API

Delete pipeline API

Simulate pipeline API

Summary

Building Data Pipelines with Logstash

Parsing and enriching logs using Logstash

Filter plugins

CSV filter

Mutate filter

Grok filter

Date filter

Geoip filter

Useragent filter

Introducing Beats

Beats by Elastic.co

Filebeat

Metricbeat

Packetbeat

Heartbeat

Winlogbeat

Auditbeat

Journalbeat

Functionbeat

Community Beats

Logstash versus Beats

Filebeat

Downloading and installing Filebeat

Installing on Windows

Installing on Linux

Architecture

Configuring Filebeat

Filebeat inputs

Filebeat general/global options

Output configuration

Logging

Filebeat modules

Summary

Visualizing Data with Kibana

Downloading and installing Kibana

Installing on Windows

Installing on Linux

Configuring Kibana

Preparing data

Kibana UI

User interaction

Configuring the index pattern

Discover

Elasticsearch query string/Lucene query

Elasticsearch DSL query

KQL

Visualize

Kibana aggregations

Bucket aggregations

Metric

Creating a visualization

Visualization types

Line, area, and bar charts

Data tables

Markdown widgets

Metrics

Goals

Gauges

Pie charts

Co-ordinate maps

Region maps

Tag clouds

Visualizations in action

Response codes over time

Top 10 requested URLs

Bandwidth usage of the top five countries over time

Web traffic originating from different countries

Most used user agent

Dashboards

Creating a dashboard

Saving the dashboard

Cloning the dashboard

Sharing the dashboard

Timelion

Timelion

Timelion expressions

Using plugins

Installing plugins

Removing plugins

Summary

Section 3: Elastic Stack Extensions

Elastic X-Pack

Installing Elasticsearch and Kibana with X-Pack

Installation

Activating X-Pack trial account

Generating passwords for default users

Configuring X-Pack

Securing Elasticsearch and Kibana

User authentication

User authorization

Security in action

Creating a new user

Deleting a user

Changing the password

Creating a new role

Deleting or editing a role

Document-level security or field-level security

X-Pack security APIs

User Management APIs

Role Management APIs

Monitoring Elasticsearch

Monitoring UI

Elasticsearch metrics

Overview tab

Nodes tab

The Indices tab

Alerting

Anatomy of a watch

Alerting in action

Creating a new alert

Threshold Alert

Advanced Watch

Deleting/deactivating/editing a watch

Summary

Section 4: Production and Server Infrastructure

Running Elastic Stack in Production

Hosting Elastic Stack on a managed cloud

Getting up and running on Elastic Cloud

Using Kibana

Overriding configuration

Recovering from a snapshot

Hosting Elastic Stack on your own

Selecting hardware

Selecting an operating system

Configuring Elasticsearch nodes

JVM heap size

Disable swapping

File descriptors

Thread pools and garbage collector

Managing and monitoring Elasticsearch

Running in Docker containers

Special considerations while deploying to a cloud

Choosing instance type

Changing default ports; do not expose ports!

Proxy requests

Binding HTTP to local addresses

Installing EC2 discovery plugin

Installing the S3 repository plugin

Setting up periodic snapshots

Backing up and restoring

Setting up a repository for snapshots

Shared filesystem

Cloud or distributed filesystems

Taking snapshots

Restoring a specific snapshot

Setting up index aliases

Understanding index aliases

How index aliases can help

Setting up index templates

Defining an index template

Creating indexes on the fly

Modeling time series data

Scaling the index with unpredictable volume over time

Unit of parallelism in Elasticsearch

The effect of the number of shards on the relevance score

The effect of the number of shards on the accuracy of aggregations

Changing the mapping over time

New fields get added

Existing fields get removed

Automatically deleting older documents

How index-per-timeframe solves these issues

Scaling with index-per-timeframe

Changing the mapping over time

Automatically deleting older documents

Summary

Building a Sensor Data Analytics Application

Introduction to the application

Understanding the sensor-generated data

Understanding the sensor metadata

Understanding the final stored data

Modeling data in Elasticsearch

Defining an index template

Understanding the mapping

Setting up the metadata database

Building the Logstash data pipeline

Accepting JSON requests over the web

Enriching the JSON with the metadata we have in the MySQL database

The jdbc_streaming plugin

The mutate plugin

Moving the looked-up fields that are under lookupResult directly in JSON

Combining the latitude and longitude fields under lookupResult as a location field

Removing the unnecessary fields

Store the resulting documents in Elasticsearch

Sending data to Logstash over HTTP

Visualizing the data in Kibana

Setting up an index pattern in Kibana

Building visualizations

How does the average temperature change over time?

How does the average humidity change over time?

How do temperature and humidity change at each location over time?

Can I visualize temperature and humidity over a map?

How are the sensors distributed across departments?

Creating a dashboard

Summary

Monitoring Server Infrastructure

Metricbeat

Downloading and installing Metricbeat

Installing on Windows

Installing on Linux

Architecture

Event structure

Configuring Metricbeat

Module configuration

Enabling module configs in the modules.d directory

Enabling module configs in the metricbeat.yml file

General settings

Output configuration

Logging

Capturing system metrics

Running Metricbeat with the system module

Specifying aliases

Visualizing system metrics using Kibana

Deployment architecture

Summary

Other Books You May Enjoy

Leave a review - let other readers know what you think