Learning Concurrency in Python电子书

Title Page

Copyright

Learning Concurrency in Python

Credits

About the Author

About the Reviewer

www.PacktPub.com

Why subscribe?

Customer Feedback

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Downloading the example code

Errata

Piracy

Questions

Speed It Up!

History of concurrency

Threads and multithreading

What is a thread?

Types of threads

What is multithreading?

Processes

Properties of processes

Multiprocessing

Event-driven programming

Turtle

Breaking it down

Reactive programming

ReactiveX - RxPy

Breaking it down

GPU programming

PyCUDA

OpenCL

Theano

The limitations of Python

Jython

IronPython

Why should we use Python?

Concurrent image download

Sequential download

Breaking it down

Concurrent download

Breaking it down

Improving number crunching with multiprocessing

Sequential prime factorization

Breaking it down

Concurrent prime factorization

Breaking it down

Summary

Parallelize It

Understanding concurrency

Properties of concurrent systems

I/O bottlenecks

Understanding parallelism

CPU-bound bottlenecks

How do they work on a CPU?

Single-core CPUs

Clock rate

Martelli model of scalability

Time-sharing - the task scheduler

Multi-core processors

System architecture styles

SISD

SIMD

MISD

MIMD

Computer memory architecture styles

UMA

NUMA

Summary

Life of a Thread

Threads in Python

Thread state

State flow chart

Python example of thread state

Breaking it down

Different types of threads

POSIX threads

Windows threads

The ways to start a thread

Starting a thread

Inheriting from the thread class

Breaking it down

Forking

Example

Breaking it down

Daemonizing a thread

Example

Breaking it down

Handling threads in Python

Starting loads of threads

Example

Breaking it down

Slowing down programs using threads

Example

Breaking it down

Getting the total number of active threads

Example

Breaking it down

Getting the current thread

Example

Breaking it down

Main thread

Example

Breaking it down

Enumerating all threads

Example

Breaking it down

Identifying threads

Example

Breakdown

Ending a thread

Best practice in stopping threads

Example

Output

Orphan processes

How does the operating system handle threads

Creating processes versus threads

Example

Breaking it down

Multithreading models

One-to-one thread mapping

Many-to-one

Many-to-many

Summary

Synchronization between Threads

Synchronization between threads

The Dining Philosophers

Example

Output

Race conditions

Process execution sequence

The solution

Critical sections

Filesystem

Life-critical systems

Shared resources and data races

The join method

Breaking it down

Putting it together

Locks

Example

Breaking it down

RLocks

Example

Breaking it down

Output

RLocks versus regular locks

Condition

Definition

Example

Our publisher

Our subscriber

Kicking it off

The results

Semaphores

Class definition

Example

The TicketSeller class

Output

Thread race

Bounded semaphores

Events

Example

Breaking it down

Barriers

Example

Breaking it down

Output

Summary

Communication between Threads

Standard data structures

Sets

Extending the class

Exercise - extending other primitives

Decorator

Class decorator

Lists

Queues

FIFO queues

Example

Breaking it down

Output

LIFO queues

Example

Breaking it down

Output

PriorityQueue

Example

Breakdown

Output

Queue objects

Full/empty queues

Example

Output

The join() function

Example

Breakdown

Output

Deque objects

Example

Breakdown

Output

Appending elements

Example

Breaking it down

Output

Popping elements

Example

Breaking it down

Output

Inserting elements

Example

Breaking it down

Output

Rotation

Example

Breaking it down

Output

Defining your own thread-safe communication structures

A web Crawler example

Requirements

Design

Our Crawler class

Our starting point

Extending the queue object

Breaking it down

Output

Future enhancements

Conclusion

Exercise - testing your skills

Summary

Debug and Benchmark

Testing strategies

Why do we test?

Testing concurrent software systems

What should we test?

Unit tests

PyUnit

Example

Output

Expanding our test suite

Unit testing concurrent code

Integration tests

Debugging

Make it work as a single thread

Pdb

An interactive example

Catching exceptions in child threads

Benchmarking

The timeit module

Timeit versus time

Command-line example

Importing timeit into your code

Utilizing decorators

Timing context manager

Output

Profiling

cProfile

Simple profile example

The line_profiler tool

Kernprof

Memory profiling

Memory profile graphs

Summary

Executors and Pools

Concurrent futures

Executor objects

Creating a ThreadPoolExecutor

Example

Output

Context manager

Example

Output

Maps

Example

Output

Shutdown of executor objects

Example

Output

Future objects

Methods in future objects

The result() method

The add_done_callback() method

The .running() method

The cancel() method

The .exception() method

The .done() method

Unit testing future objects

The set_running_or_notify_cancel() method

The set_result() method

The set_exception() method

Cancelling callable

Example

Output

Getting the result

Example

Output

Using as_completed

Example

Output

Setting callbacks

Example

Output

Chaining callbacks

Exception classes

Example

Output

ProcessPoolExecutor

Creating a ProcessPoolExecutor

Example

Output

Context Manager

Example

Output

Exercise

Getting started

Improving the speed of computationally bound problems

Full code sample

Output

Improving our crawler

The plan

New improvements

Refactoring our code

Storing the results in a CSV file

Exercise - capture more info from each page crawl

concurrent.futures in Python 2.7

Summary

Multiprocessing

Working around the GIL

Utilizing sub-processes

Example

Output

The life of a process

Starting a process using fork

Spawning a process

Forkserver

Daemon processes

Example

Breaking it down

Output

Identifying processes using PIDs

Example

Output

Terminating a process

Example

Getting the current process

Subclassing processes

Example

Output

Multiprocessing pools

The difference between concurrent.futures.ProcessPoolExecutor and Pool

Context manager

Example

Output

Submitting tasks to a process pool

Apply

Apply_async

Map

Map_async

Imap

Imap_unordered

Starmap

Starmap_async

Maxtasksperchild

Communication between processes

Pipes

Anonymous pipes

Named pipes

Working with pipes

Example

Handling Exceptions

Using pipes

Multiprocessing managers

Namespaces

Example

Queues

Example

Output

Listeners and clients

Example

The Listener class

The Client class

Output

Logging

Example

Communicating sequential processes

PyCSP

Processes in PyCSP

Output

Summary

Event-Driven Programming

Event-driven programming

The event loop

Asyncio

Getting started

Event loops

The run_forever() method

The run_until_complete() method

The stop() method

The is_closed() method

The close() function

Tasks

Example

The all_tasks(loop=None) method

The current_tasks() function

The cancel() function

Task functions

The as_completed(fs, *, loop=None, timeout=None) function

The ensure_future(coro_or_future, *, loop=None) function

The wrap_future(future, *, loop=None) function

The gather(*coroes_or_futures, loop=None, return_exceptions=False) function

The wait() function

Futures

Example

Output

Coroutines

Chaining coroutines

Output

Transports

Protocols

Synchronization between coroutines

Locks

Queues

Events and conditions

Semaphores and BoundedSemaphores

Sub-processes

Debugging asyncio programs

Debug mode

Twisted

A simple web server example

Gevent

Event loops

Greenlets

Simple example-hostnames

Output

Monkey patching

Summary

Reactive Programming

Basic reactive programming

Maintaining purity

ReactiveX, or RX

Installing RxPY

Observables

Creating observers

Example

Example 2

Breaking it down

Output

Lambda functions

Example

Breaking it down

On_next, on_completed, and on_error in lambda form

Output

Operators and chaining

Filter example

Breaking it down

Chained operators

The different operators

Creating observables

Transforming observables

Filtering observables

Error-handling observables

Hot and cold observables

Emitting events

Example

Breaking it down

Output

Multicasting

Example

Output

Combining observables

Zip() example

Output

The merge_all() operator

Output

Concurrency

Example

Output

PyFunctional

Installation and official docs

Simple example

Output

Streams, transformations, and actions

Filtering lists

Output

Reading/writing SQLite3

Compressed files

Parallel execution

Summary

Using the GPU

Basic reactive programming

Maintaining purity

ReactiveX, or RX

Installing RxPY

Observables

Creating observers

Example

Example 2

Breaking it down

Output

Lambda functions

Example

Breaking it down

On_next, on_completed, and on_error in lambda form

Output

Operators and chaining

Filter example

Breaking it down

Chained operators

The different operators

Creating observables

Transforming observables

Filtering observables

Error-handling observables

Hot and cold observables

Emitting events

Example

Breaking it down

Output

Multicasting

Example

Output

Combining observables

Zip() example

Output

The merge_all() operator

Output

Concurrency

Example

Output

PyFunctional

Installation and official docs

Simple example

Output

Streams, transformations, and actions

Filtering lists

Output

Reading/writing SQLite3

Compressed files

Parallel execution

Summary

Choosing a Solution

Libraries not covered in this book

GPU

PyGPU

Event-driven and reactive libraries

Tornado

Flask

Celery

Data science

Pandas

Matplotlib

TensorFlow

Designing your systems

Requirements

Functional requirements

Non-functional requirements

Design

Computationally expensive

Event-heavy applications

I/O-heavy applications

Recommended design books

Software Architecture with Python

Python: Master the Art of Design Patterns

Research

Summary