Building Python Real-Time Applications with Storm电子书

Building Python Real-Time Applications with Storm

Table of Contents

Building Python Real-Time Applications with Storm

Credits

About the Authors

About the Reviewers

www.PacktPub.com

Support files, eBooks, discount offers, and more

Why subscribe?

Free access for Packt account holders

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Errata

Piracy

Questions

1. Getting Acquainted with Storm

Overview of Storm

Before the Storm era

Key features of Storm

Storm cluster modes

Developer mode

Single-machine Storm cluster

Multimachine Storm cluster

The Storm client

Prerequisites for a Storm installation

Zookeeper installation

Storm installation

Enabling native (Netty only) dependency

Netty configuration

Starting daemons

Playing with optional configurations

Summary

2. The Storm Anatomy

Storm processes

Supervisor

Zookeeper

The Storm UI

Storm-topology-specific terminologies

The worker process, executor, and task

Worker processes

Executors

Tasks

Interprocess communication

A physical view of a Storm cluster

Stream grouping

Fault tolerance in Storm

Guaranteed tuple processing in Storm

XOR magic in acking

Tuning parallelism in Storm – scaling a distributed computation

Summary

3. Introducing Petrel

What is Petrel?

Building a topology

Packaging a topology

Logging events and errors

Managing third-party dependencies

Installing Petrel

Creating your first topology

Sentence spout

Splitter bolt

Word Counting Bolt

Defining a topology

Running the topology

Troubleshooting

Productivity tips with Petrel

Improving startup performance

Enabling and using logging

Automatic logging of fatal errors

Summary

4. Example Topology – Twitter

Twitter analysis

Twitter's Streaming API

Creating a Twitter app to use the Streaming API

The topology configuration file

The Twitter stream spout

Splitter bolt

Rolling word count bolt

The intermediate rankings bolt

The total rankings bolt

Defining the topology

Running the topology

Summary

5. Persistence Using Redis and MongoDB

Finding the top n ranked topics using Redis

The topology configuration file – the Redis case

Rolling word count bolt – the Redis case

Total rankings bolt – the Redis case

Defining the topology – the Redis case

Running the topology – the Redis case

Finding the hourly count of tweets by city name using MongoDB

Defining the topology – the MongoDB case

Running the topology – the MongoDB case

Summary

6. Petrel in Practice

Testing a bolt

Example – testing SplitSentenceBolt

Example – testing SplitSentenceBolt with WordCountBolt

Debugging

Installing Winpdb

Add Winpdb breakpoint

Launching and attaching the debugger

Profiling your topology's performance

Split sentence bolt log

Word count bolt log

Summary

A. Managing Storm Using Supervisord

Storm administration over a cluster

Introducing supervisord

Supervisord components

Supervisord installation

Configuration of supervisord.conf

Configuration of supervisord.conf on 172-31-19-62

Summary

Index