Hadoop 2.x Administration Cookbook电子书

Hadoop 2.x Administration Cookbook

Table of Contents

Hadoop 2.x Administration Cookbook

Credits

About the Author

About the Reviewers

www.PacktPub.com

eBooks, discount offers, and more

Why subscribe?

Customer Feedback

Preface

What this book covers

What you need for this book

Who this book is for

Sections

Getting ready

How to do it…

How it works…

There's more…

See also

Conventions

Reader feedback

Customer support

Downloading the example code

Downloading the color images of this book

Errata

Piracy

Questions

1. Hadoop Architecture and Deployment

Introduction

Overview of Hadoop Architecture

Building and compiling Hadoop

Getting ready

How to do it...

How it works...

Installation methods

Getting ready

How to do it...

How it works...

Setting up host resolution

Getting ready

How to do it...

How it works...

Installing a single-node cluster - HDFS components

Getting ready

How to do it...

How it works...

There's more...

Setting up ResourceManager and NodeManager

Installing a single-node cluster - YARN components

Getting ready

How to do it...

How it works...

There's more...

See also

Installing a multi-node cluster

Getting ready

How to do it...

How it works...

Configuring the Hadoop Gateway node

Getting ready

How to do it...

How it works...

See also

Decommissioning nodes

Getting ready

How to do it...

How it works...

See also

Adding nodes to the cluster

Getting ready

How to do it...

How it works...

There's more...

2. Maintaining Hadoop Cluster HDFS

Introduction

Overview of HDFS

Configuring HDFS block size

Getting ready

How to do it...

How it works...

Setting up Namenode metadata location

Getting ready

How to do it...

How it works...

Loading data in HDFS

Getting ready

How to do it...

How it works...

Configuring HDFS replication

Getting ready

How to do it...

How it works...

See also

HDFS balancer

Getting ready

How to do it...

How it works...

Quota configuration

Getting ready

How to do it...

How it works...

HDFS health and FSCK

Getting ready

How to do it...

How it works...

See also

Configuring rack awareness

Getting ready

How to do it...

How it works...

See also

Recycle or trash bin configuration

Getting ready

How to do it...

How it works...

There's more...

Distcp usage

Getting ready

How to do it...

How it works...

Control block report storm

Getting ready

How to do it...

How it works...

Configuring Datanode heartbeat

Getting ready

How to do it...

How it works...

3. Maintaining Hadoop Cluster – YARN and MapReduce

Introduction

Running a simple MapReduce program

Getting ready

How to do it...

Hadoop streaming

Getting ready

How to do it...

How it works...

Configuring YARN history server

Getting ready

How to do it...

How it works...

There's more...

Job history web interface and metrics

Getting ready

How to do it...

How it works...

Configuring ResourceManager components

Getting ready

How to do it...

How it works...

There's more...

See also

YARN containers and resource allocations

Getting ready

How to do it...

How it works...

There's more...

See also

ResourceManager Web UI and JMX metrics

Getting ready

How to do it...

How it works...

Preserving ResourceManager states

Getting ready

How to do it...

How it works...

There's more...

4. High Availability

Introduction

Namenode HA using shared storage

Getting ready

How to do it...

How it works...

See also

ZooKeeper configuration

Getting ready

How to do it...

How it works...

Namenode HA using Journal node

Getting ready

How to do it...

How it works...

Resourcemanager HA using ZooKeeper

Getting ready

How to do it...

How it works…

Rolling upgrade with HA

Getting ready

How to do it...

How it works...

Configure shared cache manager

Getting ready

How to do it...

There's more...

See also

Configure HDFS cache

Getting ready

How to do it...

How it works...

See also

HDFS snapshots

Getting ready

How to do it...

How it works...

Configuring storage based policies

Getting ready

How to do it...

How it works...

Configuring HA for Edge nodes

Getting ready

How to do it...

How it works...

5. Schedulers

Introduction

Configuring users and groups

Getting ready

How to do it...

How it works...

See also

Fair Scheduler configuration

Getting ready

How to do it...

How it works...

Fair Scheduler pools

Getting ready

How to do it...

How it works...

Configuring job queues

Getting ready

How to do it...

How it works...

See also

Job queue ACLs

Getting ready

How to do it...

How it works...

See also

Configuring Capacity Scheduler

Getting ready

How to do it...

How it works...

See also

Queuing mappings in Capacity Scheduler

Getting ready

How to do it...

How it works...

YARN and Mapred commands

Getting ready

How to do it...

How it works...

YARN label-based scheduling

Getting ready

How to do it...

How it works...

YARN SLS

Getting ready

How to do it...

How it works...

6. Backup and Recovery

Introduction

Initiating Namenode saveNamespace

Getting ready

How to do it...

How it works...

Using HDFS Image Viewer

Getting ready

How to do it...

How it works...

Fetching parameters which are in-effect

Getting ready

How to do it...

How it works...

Configuring HDFS and YARN logs

Getting ready

How to do it...

How it works...

See also

Backing up and recovering Namenode

Getting ready

How to do it...

How it works...

See also

Configuring Secondary Namenode

Getting ready

How to do it...

How it works…

Promoting Secondary Namenode to Primary

Getting ready

How to do it...

How it works...

See also

Namenode recovery

Getting ready

How to do it...

How it works...

Namenode roll edits – online mode

Getting ready

How to do it...

How it works...

Namenode roll edits – offline mode

Getting ready

How to do it...

How it works...

Datanode recovery – disk full

Getting ready

How to do it...

How it works...

Configuring NFS gateway to serve HDFS

Getting ready

How to do it...

How it works...

Recovering deleted files

Getting ready

How to do it...

How it works...

7. Data Ingestion and Workflow

Introduction

Hive server modes and setup

Getting ready

How to do it...

How it works...

Using MySQL for Hive metastore

How to do it…

How it works...

Operating Hive with ZooKeeper

Getting ready

How to do it...

How it works...

Loading data into Hive

Getting ready

How to do it...

How it works...

See also

Partitioning and Bucketing in Hive

Getting ready

How to do it...

How it works...

See also

Hive metastore database

Getting ready

How to do it...

How it works...

See also

Designing Hive with credential store

Getting ready

How to do it...

How it works...

Configuring Flume

Getting ready

How to do it...

How it works...

Configure Oozie and workflows

Getting ready

How to do it...

How it works...

8. Performance Tuning

Tuning the operating system

Getting ready

How to do it...

How it works...

See also

Tuning the disk

Getting ready

How to do it...

How it works...

Tuning the network

Getting ready

How to do it...

How it works...

Tuning HDFS

Getting ready

How to do it...

How it works...

Tuning Namenode

Getting ready

How to do it...

There's more...

See also

Tuning Datanode

Getting ready

How to do it...

How it works...

See also

Configuring YARN for performance

Getting ready

How to do it...

How it works...

Configuring MapReduce for performance

Getting ready

How to do it...

How it works...

Hive performance tuning

Getting ready

How to do it...

There's more...

How it works...

Benchmarking Hadoop cluster

Getting ready

How to do it...

Benchmark 1--Testing HDFS with TestDFSIO

Benchmark 2--Stress testing Namenode

Benchmark 3--MapReduce testing by generating small files

Benchmark 4--TeraGen, TeraSort, and TeraValidate benchmarks

There's more...

How it works...

9. HBase Administration

Introduction

Setting up single node HBase cluster

Getting ready

How to do it...

How it works...

Setting up multi-node HBase cluster

Getting ready

How to do it...

How it works...

Inserting data into HBase

Getting ready

How to do it...

How it works...

Integration with Hive

Getting ready

How to do it...

How it works...

See also

HBase administration commands

Getting ready

How to do it...

How it works...

See also

HBase backup and restore

Getting ready

How to do it...

How it works...

Tuning HBase

Getting ready

How to do it...

How it works...

HBase upgrade

Getting ready

How to do it...

How it works...

Migrating data from MySQL to HBase using Sqoop

Getting ready

How to do it...

10. Cluster Planning

Introduction

Disk space calculations

Getting ready

How to do it...

How it works...

Nodes needed in the cluster

Getting ready

How to do it...

How it works...

See also

Memory requirements

Getting ready

How to do it...

How it works...

See also

Sizing the cluster as per SLA

Getting ready

How to do it...

How it works...

See also

Network design

Getting ready

How to do it...

How it works...

Estimating the cost of the Hadoop cluster

How to do it...

How it works...

Hardware and software options

How it works...

11. Troubleshooting, Diagnostics, and Best Practices

Introduction

Namenode troubleshooting

Getting ready

How to do it...

How it works...

See also

Datanode troubleshooting

Getting ready

How to do it...

How it works...

See also

Resourcemanager troubleshooting

Getting ready

How to do it…

How it works...

See also

Diagnose communication issues

Getting ready

How to do it...

How it works...

Parse logs for errors

Getting ready

How to do it...

How it works...

Hive troubleshooting

Getting ready

How to do it...

How it works...

See also

HBase troubleshooting

Getting ready

How to do it...

How it works...

Hadoop best practices

How it works...

12. Security

Introduction

Encrypting disk using LUKS

Getting ready

How to do it...

How it works...

See also

Configuring Hadoop users

Getting ready

How to do it...

How it works...

HDFS encryption at Rest

Getting ready

How to do it...

How it works...

Configuring SSL in Hadoop

Getting ready

How to do it...

How it works...

See also

In-transit encryption

Getting ready

How to do it...

There's more...

See also

Enabling service level authorization

Getting ready

How to do it...

How it works...

See also

Securing ZooKeeper

Getting ready

How to do it...

How it works...

Configuring auditing

Getting ready

How to do it...

How it works...

Configuring Kerberos server

Getting ready

How to do it...

How it works...

Configuring and enabling Kerberos for Hadoop

Getting ready

How to do it...

How it works...

Index