YARN Essentials电子书

YARN Essentials

Table of Contents

YARN Essentials

Credits

About the Authors

About the Reviewers

www.PacktPub.com

Support files, eBooks, discount offers, and more

Why subscribe?

Free access for Packt account holders

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Errata

Piracy

Questions

1. Need for YARN

The redesign idea

Limitations of the classical MapReduce or Hadoop 1.x

YARN as the modern operating system of Hadoop

What are the design goals for YARN

Summary

2. YARN Architecture

Core components of YARN architecture

ResourceManager

ApplicationMaster (AM)

NodeManager (NM)

YARN scheduler policies

The FIFO (First In First Out) scheduler

The fair scheduler

The capacity scheduler

Recent developments in YARN architecture

Summary

3. YARN Installation

Single-node installation

Prerequisites

Platform

Software

Starting with the installation

The standalone mode (local mode)

The pseudo-distributed mode

The fully-distributed mode

HistoryServer

Slave files

Operating Hadoop and YARN clusters

Starting Hadoop and YARN clusters

Stopping Hadoop and YARN clusters

Web interfaces of the Ecosystem

Summary

4. YARN and Hadoop Ecosystems

The Hadoop 2 release

A short introduction to Hadoop 1.x and MRv1

MRv1 versus MRv2

Understanding where YARN fits into Hadoop

Old and new MapReduce APIs

Backward compatibility of MRv2 APIs

Binary compatibility of org.apache.hadoop.mapred APIs

Source compatibility of org.apache.hadoop.mapred APIs

Practical examples of MRv1 and MRv2

Preparing the input file(s)

Running the job

Result

Summary

5. YARN Administration

Container allocation

Container allocation to the application

Container configurations

YARN scheduling policies

The FIFO (First In First Out) scheduler

The FIFO (First In First Out) scheduler

The capacity scheduler

Capacity scheduler configurations

The fair scheduler

Fair scheduler configurations

YARN multitenancy application support

Administration of YARN

Administrative tools

Adding and removing nodes from a YARN cluster

Administrating YARN jobs

MapReduce job configurations

YARN log management

YARN web user interface

Summary

6. Developing and Running a Simple YARN Application

Running sample examples on YARN

Running a sample Pi example

Monitoring YARN applications with web GUI

YARN's MapReduce support

The MapReduce ApplicationMaster

Example YARN MapReduce settings

YARN's compatibility with MapReduce applications

Developing YARN applications

The YARN application workflow

Writing the YARN client

Writing the YARN ApplicationMaster

Responsibilities of the ApplicationMaster

Summary

7. YARN Frameworks

Apache Samza

Writing a Kafka producer

Writing the hello-samza project

Starting a grid

Storm-YARN

Prerequisites

Hadoop YARN should be installed

Apache ZooKeeper should be installed

Setting up Storm-YARN

Getting the storm.yaml configuration of the launched Storm cluster

Building and running Storm-Starter examples

Apache Spark

Why run on YARN?

Apache Tez

Apache Giraph

HOYA (HBase on YARN)

KOYA (Kafka on YARN)

Summary

8. Failures in YARN

ResourceManager failures

ApplicationMaster failures

NodeManager failures

Container failures

Hardware Failures

Summary

9. YARN – Alternative Solutions

Mesos

Omega

Corona

Summary

10. YARN – Future and Support

What YARN means to the big data industry

Journey – present and future

Present on-going features

Future features

YARN-supported frameworks

Summary

Index