Advanced Python Programming电子书

Title Page

Copyright

Advanced Python Programming

About Packt

Why Subscribe?

Packt.com

Contributors

About the Authors

Packt Is Searching for Authors Like You

Preface

Who This Book Is For

What This Book Covers

To Get the Most out of This Book

Download the Example Code Files

Conventions Used

Get in Touch

Reviews

Benchmarking and Profiling

Designing your application

Writing tests and benchmarks

Timing your benchmark

Better tests and benchmarks with pytest-benchmark

Finding bottlenecks with cProfile

Profile line by line with line_profiler

Optimizing our code

The dis module

Profiling memory usage with memory_profiler

Summary

Pure Python Optimizations

Useful algorithms and data structures

Lists and deques

Dictionaries

Building an in-memory search index using a hash map

Sets

Heaps

Tries

Caching and memoization

Joblib

Comprehensions and generators

Summary

Fast Array Operations with NumPy and Pandas

Getting started with NumPy

Creating arrays

Accessing arrays

Broadcasting

Mathematical operations

Calculating the norm

Rewriting the particle simulator in NumPy

Reaching optimal performance with numexpr

Pandas

Pandas fundamentals

Indexing Series and DataFrame objects

Database-style operations with Pandas

Mapping

Grouping, aggregations, and transforms

Joining

Summary

C Performance with Cython

Compiling Cython extensions

Adding static types

Variables

Functions

Classes

Sharing declarations

Working with arrays

C arrays and pointers

NumPy arrays

Typed memoryviews

Particle simulator in Cython

Profiling Cython

Using Cython with Jupyter

Summary

Exploring Compilers

Numba

First steps with Numba

Type specializations

Object mode versus native mode

Numba and NumPy

Universal functions with Numba

Generalized universal functions

JIT classes

Limitations in Numba

The PyPy project

Setting up PyPy

Running a particle simulator in PyPy

Other interesting projects

Summary

Implementing Concurrency

Asynchronous programming

Waiting for I/O

Concurrency

Callbacks

Futures

Event loops

The asyncio framework

Coroutines

Converting blocking code into non-blocking code

Reactive programming

Observables

Useful operators

Hot and cold observables

Building a CPU monitor

Summary

Parallel Processing

Introduction to parallel programming

Graphic processing units

Using multiple processes

The Process and Pool classes

The Executor interface

Monte Carlo approximation of pi

Synchronization and locks

Parallel Cython with OpenMP

Automatic parallelism

Getting started with Theano

Profiling Theano

Tensorflow

Running code on a GPU

Summary

Advanced Introduction to Concurrent and Parallel Programming

Technical requirements

What is concurrency?

Concurrent versus sequential

Example 1 – checking whether a non-negative number is prime

Concurrent versus parallel

A quick metaphor

Not everything should be made concurrent

Embarrassingly parallel

Inherently sequential

Example 2 – inherently sequential tasks

I/O bound

The history, present, and future of concurrency

The history of concurrency

The present

The future

A brief overview of mastering concurrency in Python

Why Python?

Setting up your Python environment

General setup

Summary

Questions

Further reading

Amdahl's Law

Technical requirements

Amdahl's Law

Terminology

Formula and interpretation

The formula for Amdahl's Law

A quick example

Implications

Amdahl's Law's relationship to the law of diminishing returns

How to simulate in Python

Practical applications of Amdahl's Law

Summary

Questions

Further reading

Working with Threads in Python

Technical requirements

The concept of a thread

Threads versus processes

Multithreading

An example in Python

An overview of the threading module

The thread module in Python 2

The threading module in Python 3

Creating a new thread in Python

Starting a thread with the thread module

Starting a thread with the threading module

Synchronizing threads

The concept of thread synchronization

The threading.Lock class

An example in Python

Multithreaded priority queue

A connection between real-life and programmatic queues

The queue module

Queuing in concurrent programming

Multithreaded priority queue

Summary

Questions

Further reading

Using the with Statement in Threads

Technical requirements

Context management

Starting from managing files

The with statement as a context manager

The syntax of the with statement

The with statement in concurrent programming

Example of deadlock handling

Summary

Questions

Further reading

Concurrent Web Requests

Technical requirements

The basics of web requests

HTML

HTTP requests

HTTP status code

The requests module

Making a request in Python

Running a ping test

Concurrent web requests

Spawning multiple threads

Refactoring request logic

The problem of timeout

Support from httpstat.us and simulation in Python

Timeout specifications

Good practices in making web requests

Consider the terms of service and data-collecting policies

Error handling

Update your program regularly

Avoid making a large number of requests

Summary

Questions

Further reading

Working with Processes in Python

Technical requirements

The concept of a process

Processes versus threads

Multiprocessing

Introductory example in Python

An overview of the multiprocessing module

The process class

The Pool class

Determining the current process, waiting, and terminating processes

Determining the current process

Waiting for processes

Terminating processes

Interprocess communication

Message passing for a single worker

Message passing between several workers

Summary

Questions

Further reading

Reduction Operators in Processes

Technical requirements

The concept of reduction operators

Properties of a reduction operator

Examples and non-examples

Example implementation in Python

Real-life applications of concurrent reduction operators

Summary

Questions

Further reading

Concurrent Image Processing

Technical requirements

Image processing fundamentals

Python as an image processing tool

Installing OpenCV and NumPy

Computer image basics

RGB values

Pixels and image files

Coordinates inside an image

OpenCV API

Image processing techniques

Grayscaling

Thresholding

Applying concurrency to image processing

Good concurrent image processing practices

Choosing the correct way (out of many)

Spawning an appropriate number of processes

Processing input/output concurrently

Summary

Questions

Further reading

Introduction to Asynchronous Programming

Technical requirements

A quick analogy

Asynchronous versus other programming models

Asynchronous versus synchronous programming

Asynchronous versus threading and multiprocessing

An example in Python

Summary

Questions

Further reading

Implementing Asynchronous Programming in Python

Technical requirements

The asyncio module

Coroutines, event loops, and futures

Asyncio API

The asyncio framework in action

Asynchronously counting down

A note about blocking functions

Asynchronous prime-checking

Improvements from Python 3.7

Inherently blocking tasks

concurrent.futures as a solution for blocking tasks

Changes in the framework

Examples in Python

Summary

Questions

Further reading

Building Communication Channels with asyncio

Technical requirements

The ecosystem of communication channels

Communication protocol layers

Asynchronous programming for communication channels

Transports and protocols in asyncio

The big picture of asyncio's server client

Python example

Starting a server

Installing Telnet

Simulating a connection channel

Sending messages back to clients

Closing the transports

Client-side communication with aiohttp

Installing aiohttp and aiofiles

Fetching a website's HTML code

Writing files asynchronously

Summary

Questions

Further reading

Deadlocks

Technical requirements

The concept of deadlock

The Dining Philosophers problem

Deadlock in a concurrent system

Python simulation

Approaches to deadlock situations

Implementing ranking among resources

Ignoring locks and sharing resources

An additional note about locks

Concluding note on deadlock solutions

The concept of livelock

Summary

Questions

Further reading

Starvation

Technical requirements

The concept of starvation

What is starvation?

Scheduling

Causes of starvation

Starvation's relationship to deadlock

The readers-writers problem

Problem statement

The first readers-writers problem

The second readers-writers problem

The third readers-writers problem

Solutions to starvation

Summary

Questions

Further reading

Race Conditions

Technical requirements

The concept of race conditions

Critical sections

How race conditions occur

Simulating race conditions in Python

Locks as a solution to race conditions

The effectiveness of locks

Implementation in Python

The downside of locks

Turning a concurrent program sequential

Locks do not lock anything

Race conditions in real life

Security

Operating systems

Networking

Summary

Questions

Further reading

The Global Interpreter Lock

Technical requirements

An introduction to the Global Interpreter Lock

An analysis of memory management in Python

The problem that the GIL addresses

Problems raised by the GIL

The potential removal of the GIL from Python

How to work with the GIL

Implementing multiprocessing, rather than multithreading

Getting around the GIL with native extensions

Utilizing a different Python interpreter

Summary

Questions

Further reading

The Factory Pattern

The factory method

Real-world examples

Use cases

Implementing the factory method

The abstract factory

Real-world examples

Use cases

Implementing the abstract factory pattern

Summary

The Builder Pattern

Real-world examples

Use cases

Implementation

Summary

Other Creational Patterns

The prototype pattern

Real-world examples

Use cases

Implementation

Singleton

Real-world examples

Use cases

Implementation

Summary

The Adapter Pattern

Real-world examples

Use cases

Implementation

Summary

The Decorator Pattern

Real-world examples

Use cases

Implementation

Summary

The Bridge Pattern

Real-world examples

Use cases

Implementation

Summary

The Facade Pattern

Real-world examples

Use cases

Implementation

Summary

Other Structural Patterns

The flyweight pattern

Real-world examples

Use cases

Implementation

The model-view-controller pattern

Real-world examples

Use cases

Implementation

The proxy pattern

Real-world examples

Use cases

Implementation

Summary

The Chain of Responsibility Pattern

Real-world examples

Use cases

Implementation

Summary

The Command Pattern

Real-world examples

Use cases

Implementation

Summary

The Observer Pattern

Real-world examples

Use cases

Implementation

Summary

Appendix

Chapter 8

Chapter 9

Chapter 10

Chapter 11

Chapter 12

Chapter 13

Chapter 14

Chapter 15

Chapter 16

Chapter 17

Chapter 18

Chapter 19

Chapter 20

Chapter 21

Chapter 22

Other Books You May Enjoy

Leave a Review - Let Other Readers Know What You Think