Scaling Big Data with Hadoop and Solr - Second Edition电子书

Scaling Big Data with Hadoop and Solr Second Edition

Table of Contents

Scaling Big Data with Hadoop and Solr Second Edition

Credits

About the Author

About the Reviewers

www.PacktPub.com

Support files, eBooks, discount offers, and more

Why subscribe?

Free access for Packt account holders

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Errata

Piracy

Questions

1. Processing Big Data Using Hadoop and MapReduce

Apache Hadoop's ecosystem

Core components

Understanding Hadoop's ecosystem

Configuring Apache Hadoop

Prerequisites

Setting up ssh without passphrase

Configuring Hadoop

Running Hadoop

Setting up a Hadoop cluster

Common problems and their solutions

Summary

2. Understanding Apache Solr

Setting up Apache Solr

Prerequisites for setting up Apache Solr

Running Apache Solr on jetty

Running Solr on other J2EE containers

Hello World with Apache Solr!

Understanding Solr administration

Solr navigation

Common problems and solutions

The Apache Solr architecture

Configuring Solr

Understanding the Solr structure

Defining the Solr schema

Solr fields

Dynamic fields in Solr

Copying the fields

Dealing with field types

Additional metadata configuration

Other important elements of the Solr schema

Configuration files of Apache Solr

Working with solr.xml and Solr core

Instance configuration with solrconfig.xml

Understanding the Solr plugin

Other configuration

Loading data in Apache Solr

Extracting request handler – Solr Cell

Understanding data import handlers

Interacting with Solr through SolrJ

Working with rich documents (Apache Tika)

Querying for information in Solr

Summary

3. Enabling Distributed Search using Apache Solr

Understanding a distributed search

Distributed search patterns

Apache Solr and distributed search

Working with SolrCloud

Why ZooKeeper?

The SolrCloud architecture

Building an enterprise distributed search using SolrCloud

Setting up SolrCloud for development

Setting up SolrCloud for production

Adding a document to SolrCloud

Creating shards, collections, and replicas in SolrCloud

Common problems and resolutions

Sharding algorithm and fault tolerance

Document Routing and Sharding

Shard splitting

Load balancing and fault tolerance in SolrCloud

Apache Solr and Big Data – integration with MongoDB

What is NoSQL and how is it related to Big Data?

MongoDB at glance

Installing MongoDB

Creating Solr indexes from MongoDB

Summary

4. Big Data Search Using Hadoop and Its Ecosystem

Understanding NoSQL

Working with the Solr HDFS connector

Big data search using Katta

How Katta works?

Setting up the Katta cluster

Creating Katta indexes

Using Solr 1045 Patch – map-side indexing

Using Solr 1301 Patch – reduce-side indexing

Distributed search using Apache Blur

Setting up Apache Blur with Hadoop

Apache Solr and Cassandra

Working with Cassandra and Solr

Single node configuration

Integrating with multinode Cassandra

Scaling Solr through Storm

Getting along with Apache Storm

Advanced analytics with Solr

Integrating Solr and R

Summary

5. Scaling Search Performance

Understanding the limits

Optimizing search schema

Specifying default search field

Configuring search schema fields

Stop words

Stemming

Index optimization

Limiting indexing buffer size

When to commit changes?

Optimizing index merge

Optimize option for index merging

Optimizing the container

Optimizing concurrent clients

Optimizing Java virtual memory

Optimizing search runtime

Optimizing through search query

Filter queries

Optimizing the Solr cache

The filter cache

The query result cache

The document cache

The field value cache

The lazy field loading

Optimizing Hadoop

Monitoring Solr instance

Using SolrMeter

Summary

A. Use Cases for Big Data Search

E-Commerce websites

Log management for banking

The problem

How can it be tackled?

High-level design

Index