Python: Journey from Novice to Expert电子书

Python: Journey from Novice to Expert

Table of Contents

Python: Journey from Novice to Expert

Python: Journey from Novice to Expert

Credits

Preface

What this learning path covers

What you need for this learning path

Who this learning path is for

Reader feedback

Customer support

Downloading the example code

Errata

Piracy

Questions

1. Module 1

1. Introduction and First Steps – Take a Deep Breath

A proper introduction

Enter the Python

About Python

Portability

Coherence

Developer productivity

An extensive library

Software quality

Software integration

Satisfaction and enjoyment

What are the drawbacks?

Who is using Python today?

Setting up the environment

Python 2 versus Python 3 – the great debate

Installing Python

Setting up the Python interpreter

About virtualenv

Your first virtual environment

Your friend, the console

How you can run a Python program

Running Python scripts

Running the Python interactive shell

Running Python as a service

Running Python as a GUI application

How is Python code organized

How do we use modules and packages

Python's execution model

Names and namespaces

Scopes

Object and classes

Guidelines on how to write good code

The Python culture

A note on the IDEs

Summary

2. Built-in Data Types

Everything is an object

Mutable or immutable? That is the question

Numbers

Integers

Booleans

Reals

Complex numbers

Fractions and decimals

Immutable sequences

Strings and bytes

Encoding and decoding strings

Indexing and slicing strings

Tuples

Mutable sequences

Lists

Byte arrays

Set types

Mapping types – dictionaries

The collections module

Named tuples

Defaultdict

ChainMap

Final considerations

Small values caching

How to choose data structures

About indexing and slicing

About the names

Summary

3. Iterating and Making Decisions

Conditional programming

A specialized else: elif

The ternary operator

Looping

The for loop

Iterating over a range

Iterating over a sequence

Iterators and iterables

Iterating over multiple sequences

The while loop

The break and continue statements

A special else clause

Putting this all together

Example 1 – a prime generator

Example 2 – applying discounts

A quick peek at the itertools module

Infinite iterators

Iterators terminating on the shortest input sequence

Combinatoric generators

Summary

4. Functions, the Building Blocks of Code

Why use functions?

Reduce code duplication

Splitting a complex task

Hide implementation details

Improve readability

Improve traceability

Scopes and name resolution

The global and nonlocal statements

Input parameters

Argument passing

Assignment to argument names don't affect the caller

Changing a mutable affects the caller

How to specify input parameters

Positional arguments

Keyword arguments and default values

Variable positional arguments

Variable keyword arguments

Keyword-only arguments

Combining input parameters

Avoid the trap! Mutable defaults

Return values

Returning multiple values

A few useful tips

Recursive functions

Anonymous functions

Function attributes

Built-in functions

One final example

Documenting your code

Importing objects

Relative imports

Summary

5. Saving Time and Memory

map, zip, and filter

map

zip

filter

Comprehensions

Nested comprehensions

Filtering a comprehension

dict comprehensions

set comprehensions

Generators

Generator functions

Going beyond next

The yield from expression

Generator expressions

Some performance considerations

Don't overdo comprehensions and generators

Name localization

Generation behavior in built-ins

One last example

Summary

6. Advanced Concepts – OOP, Decorators, and Iterators

Decorators

A decorator factory

Object-oriented programming

The simplest Python class

Class and object namespaces

Attribute shadowing

I, me, and myself – using the self variable

Initializing an instance

OOP is about code reuse

Inheritance and composition

Accessing a base class

Multiple inheritance

Method resolution order

Static and class methods

Static methods

Class methods

Private methods and name mangling

The property decorator

Operator overloading

Polymorphism – a brief overview

Writing a custom iterator

Summary

7. Testing, Profiling, and Dealing with Exceptions

Testing your application

The anatomy of a test

Testing guidelines

Unit testing

Writing a unit test

Mock objects and patching

Assertions

A classic unit test example

Making a test fail

Interface testing

Comparing tests with and without mocks

Boundaries and granularity

A more interesting example

Test-driven development

Exceptions

Profiling Python

When to profile?

Summary

8. The Edges – GUIs and Scripts

First approach – scripting

The imports

Parsing arguments

The business logic

Second approach – a GUI application

The imports

The layout logic

The business logic

Fetching the web page

Saving the images

Alerting the user

How to improve the application?

Where do we go from here?

The tkinter.tix module

The turtle module

wxPython, PyQt, and PyGTK

The principle of least astonishment

Threading considerations

Summary

9. Data Science

IPython and Jupyter notebook

Dealing with data

Setting up the notebook

Preparing the data

Cleaning the data

Creating the DataFrame

Unpacking the campaign name

Unpacking the user data

Cleaning everything up

Saving the DataFrame to a file

Visualizing the results

Where do we go from here?

Summary

10. Web Development Done Right

What is the Web?

How does the Web work?

The Django web framework

Django design philosophy

The model layer

The view layer

The template layer

The Django URL dispatcher

Regular expressions

A regex website

Setting up Django

Starting the project

Creating users

Adding the Entry model

Customizing the admin panel

Creating the form

Writing the views

The home view

The entry list view

The form view

Tying up URLs and views

Writing the templates

The future of web development

Writing a Flask view

Building a JSON quote server in Falcon

Summary

11. Debugging and Troubleshooting

Debugging techniques

Debugging with print

Debugging with a custom function

Inspecting the traceback

Using the Python debugger

Inspecting log files

Other techniques

Profiling

Assertions

Where to find information

Troubleshooting guidelines

Using console editors

Where to inspect

Using tests to debug

Monitoring

Summary

12. Summing Up – A Complete Example

The challenge

Our implementation

Implementing the Django interface

The setup

The model layer

A simple form

The view layer

Imports and home view

Listing all records

Creating records

Updating records

Deleting records

Setting up the URLs

The template layer

Home and footer templates

Listing all records

Creating and editing records

Talking to the API

Deleting records

Implementing the Falcon API

The main application

Writing the helpers

Coding the password validator

Coding the password generator

Writing the handlers

Coding the password validator handler

Coding the password generator handler

Running the API

Testing the API

Testing the helpers

Testing the handlers

Where do you go from here?

Summary

2. Module 2

1. Object-oriented Design

Introducing object-oriented

Objects and classes

Specifying attributes and behaviors

Data describes objects

Behaviors are actions

Hiding details and creating the public interface

Composition

Inheritance

Inheritance provides abstraction

Multiple inheritance

Case study

Exercises

Summary

2. Objects in Python

Creating Python classes

Adding attributes

Making it do something

Talking to yourself

More arguments

Initializing the object

Explaining yourself

Modules and packages

Organizing the modules

Absolute imports

Relative imports

Organizing module contents

Who can access my data?

Third-party libraries

Case study

Exercises

Summary

3. When Objects Are Alike

Basic inheritance

Extending built-ins

Overriding and super

Multiple inheritance

The diamond problem

Different sets of arguments

Polymorphism

Abstract base classes

Using an abstract base class

Creating an abstract base class

Demystifying the magic

Case study

Exercises

Summary

4. Expecting the Unexpected

Raising exceptions

Raising an exception

The effects of an exception

Handling exceptions

The exception hierarchy

Defining our own exceptions

Case study

Exercises

Summary

5. When to Use Object-oriented Programming

Treat objects as objects

Adding behavior to class data with properties

Properties in detail

Decorators – another way to create properties

Deciding when to use properties

Manager objects

Removing duplicate code

In practice

Case study

Exercises

Summary

6. Python Data Structures

Empty objects

Tuples and named tuples

Named tuples

Dictionaries

Dictionary use cases

Using defaultdict

Counter

Lists

Sorting lists

Sets

Extending built-ins

Queues

FIFO queues

LIFO queues

Priority queues

Case study

Exercises

Summary

7. Python Object-oriented Shortcuts

Python built-in functions

The len() function

Reversed

Enumerate

File I/O

Placing it in context

An alternative to method overloading

Default arguments

Variable argument lists

Unpacking arguments

Functions are objects too

Using functions as attributes

Callable objects

Case study

Exercises

Summary

8. Strings and Serialization

Strings

String manipulation

String formatting

Escaping braces

Keyword arguments

Container lookups

Object lookups

Making it look right

Strings are Unicode

Converting bytes to text

Converting text to bytes

Mutable byte strings

Regular expressions

Matching patterns

Matching a selection of characters

Escaping characters

Matching multiple characters

Grouping patterns together

Getting information from regular expressions

Making repeated regular expressions efficient

Serializing objects

Customizing pickles

Serializing web objects

Case study

Exercises

Summary

9. The Iterator Pattern

Design patterns in brief

Iterators

The iterator protocol

Comprehensions

List comprehensions

Set and dictionary comprehensions

Generator expressions

Generators

Yield items from another iterable

Coroutines

Back to log parsing

Closing coroutines and throwing exceptions

The relationship between coroutines, generators, and functions

Case study

Exercises

Summary

10. Python Design Patterns I

The decorator pattern

A decorator example

Decorators in Python

The observer pattern

An observer example

The strategy pattern

A strategy example

Strategy in Python

The state pattern

A state example

State versus strategy

State transition as coroutines

The singleton pattern

Singleton implementation

The template pattern

A template example

Exercises

Summary

11. Python Design Patterns II

The adapter pattern

The facade pattern

The flyweight pattern

The command pattern

The abstract factory pattern

The composite pattern

Exercises

Summary

12. Testing Object-oriented Programs

Why test?

Test-driven development

Unit testing

Assertion methods

Reducing boilerplate and cleaning up

Organizing and running tests

Ignoring broken tests

Testing with py.test

One way to do setup and cleanup

A completely different way to set up variables

Skipping tests with py.test

Imitating expensive objects

How much testing is enough?

Case study

Implementing it

Exercises

Summary

13. Concurrency

Threads

The many problems with threads

Shared memory

The global interpreter lock

Thread overhead

Multiprocessing

Multiprocessing pools

Queues

The problems with multiprocessing

Futures

AsyncIO

AsyncIO in action

Reading an AsyncIO future

AsyncIO for networking

Using executors to wrap blocking code

Streams

Executors

Case study

Exercises

Summary

3. Module 3

1. Getting Started – One Environment per Project

Creating a virtual Python environment using venv

Creating your first venv

venv arguments

Differences between virtualenv and venv

Bootstrapping pip using ensurepip

ensurepip usage

Manual pip install

Installing C/C++ packages

Debian and Ubuntu

Red Hat, CentOS, and Fedora

OS X

Windows

Summary

2. Pythonic Syntax, Common Pitfalls, and Style Guide

Code style – or what is Pythonic code?

Formatting strings – printf-style or str.format?

PEP20, the Zen of Python

Beautiful is better than ugly

Explicit is better than implicit

Simple is better than complex

Flat is better than nested

Sparse is better than dense

Readability counts

Practicality beats purity

Errors should never pass silently

In the face of ambiguity, refuse the temptation to guess

One obvious way to do it

Now is better than never

Hard to explain, easy to explain

Namespaces are one honking great idea

Conclusion

Explaining PEP8

Duck typing

Differences between value and identity comparisons

Loops

Maximum line length

Verifying code quality, pep8, pyflakes, and more

flake8

Pep8

pyflakes

McCabe

flake8

Pylint

Common pitfalls

Scope matters!

Function arguments

Class properties

Modifying variables in the global scope

Overwriting and/or creating extra built-ins

Modifying while iterating

Catching exceptions – differences between Python 2 and 3

Late binding – be careful with closures

Circular imports

Import collisions

Summary

3. Containers and Collections – Storing Data the Right Way

Time complexity – the big O notation

Core collections

list – a mutable list of items

dict – unsorted but a fast map of items

set – like a dict without values

tuple – the immutable list

Advanced collections

ChainMap – the list of dictionaries

counter – keeping track of the most occurring elements

deque – the double ended queue

defaultdict – dictionary with a default value

namedtuple – tuples with field names

enum – a group of constants

OrderedDict – a dictionary where the insertion order matters

heapq – the ordered list

bisect – the sorted list

Summary

4. Functional Programming – Readability Versus Brevity

Functional programming

list comprehensions

dict comprehensions

set comprehensions

lambda functions

The Y combinator

functools

partial – no need to repeat all arguments every time

reduce – combining pairs into a single result

Implementing a factorial function

Processing trees

itertools

accumulate – reduce with intermediate results

chain – combining multiple results

combinations – combinatorics in Python

permutations – combinations where the order matters

compress – selecting items using a list of Booleans

dropwhile/takewhile – selecting items using a function

count – infinite range with decimal steps

groupby – grouping your sorted iterable

islice – slicing any iterable

Summary

5. Decorators – Enabling Code Reuse by Decorating

Decorating functions

Why functools.wraps is important

How are decorators useful?

Memoization using decorators

Decorators with (optional) arguments

Creating decorators using classes

Decorating class functions

Skipping the instance – classmethod and staticmethod

Properties – smart descriptor usage

Decorating classes

Singletons – classes with a single instance

Total ordering – sortable classes the easy way

Useful decorators

Single dispatch – polymorphism in Python

Contextmanager, with statements made easy

Validation, type checks, and conversions

Useless warnings – how to ignore them

Summary

6. Generators and Coroutines – Infinity, One Step at a Time

What are generators?

Advantages and disadvantages of generators

Pipelines – an effective use of generators

tee – using an output multiple times

Generating from generators

Context managers

Coroutines

A basic example

Priming

Closing and throwing exceptions

Bidirectional pipelines

Using the state

Summary

7. Async IO – Multithreading without Threads

Introducing the asyncio library

The async and await statements

Python 3.4

Python 3.5

Choosing between the 3.4 and 3.5 syntax

A simple example of single-threaded parallel processing

Concepts of asyncio

Futures and tasks

Event loops

Event loop implementations

Event loop policies

Event loop usage

Processes

Asynchronous servers and clients

Basic echo server

Summary

8. Metaclasses – Making Classes (Not Instances) Smarter

Dynamically creating classes

A basic metaclass

Arguments to metaclasses

Accessing metaclass attributes through classes

Abstract classes using collections.abc

Internal workings of the abstract classes

Custom type checks

Using abc.ABC before Python 3.4

Automatically registering a plugin system

Importing plugins on-demand

Importing plugins through configuration

Importing plugins through the file system

Order of operations when instantiating classes

Finding the metaclass

Preparing the namespace

Executing the class body

Creating the class object (not instance)

Executing the class decorators

Creating the class instance

Example

Storing class attributes in definition order

The classic solution without metaclasses

Using metaclasses to get a sorted namespace

Summary

9. Documentation – How to Use Sphinx and reStructuredText

The reStructuredText syntax

Getting started with reStructuredText

Inline markup

Headers

Lists

Enumerated list

Bulleted list

Option list

Definition list

Nested lists

Links, references, and labels

Images

Substitutions

Blocks, code, math, comments, and quotes

Conclusion

The Sphinx documentation generator

Getting started with Sphinx

Using sphinx-quickstart

Using sphinx-apidoc

Sphinx directives

The table of contents tree directive (toctree)

Autodoc, documenting Python modules, classes, and functions

Sphinx roles

Documenting code

Documenting a class with the Sphinx style

Documenting a class with the Google style

Documenting a class with the NumPy style

Which style to choose

Summary

10. Testing and Logging – Preparing for Bugs

Using examples as tests with doctest

A simple doctest example

Writing doctests

Testing with pure documentation

The doctest flags

True and False versus 1 and 0

Normalizing whitespace

Ellipsis

Doctest quirks

Testing dictionaries

Testing floating-point numbers

Times and durations

Testing with py.test

The difference between the unittest and py.test output

The difference between unittest and py.test tests

Simplifying assertions

Parameterizing tests

Automatic arguments using fixtures

Cache

Custom fixtures

Print statements and logging

Plugins

pytest-cov

pytest-pep8 and pytest-flakes

Configuring plugins

Mock objects

Using unittest.mock

Using py.test monkeypatch

Logging

Configuration

Basic logging configuration

Dictionary configuration

JSON configuration

Ini file configuration

The network configuration

Logger

Usage

Summary

11. Debugging – Solving the Bugs

Non-interactive debugging

Inspecting your script using trace

Debugging using logging

Showing call stack without exceptions

Debugging asyncio

Handling crashes using faulthandler

Interactive debugging

Console on demand

Debugging using pdb

Breakpoints

Catching exceptions

Commands

Debugging using ipdb

Other debuggers

Debugging services

Summary

12. Performance – Tracking and Reducing Your Memory and CPU Usage

What is performance?

Timeit – comparing code snippet performance

cProfile – finding the slowest components

First profiling run

Calibrating your profiler

Selective profiling using decorators

Using profile statistics

Line profiler

Improving performance

Using the right algorithm

Global interpreter lock

Try versus if

Lists versus generators

String concatenation

Addition versus generators

Map versus generators and list comprehensions

Caching

Lazy imports

Using optimized libraries

Just-in-time compiling

Converting parts of your code to C

Memory usage

Tracemalloc

Memory profiler

Memory leaks

Reducing memory usage

Generators versus lists

Recreating collections versus removing items

Using slots

Performance monitoring

Summary

13. Multiprocessing – When a Single CPU Core Is Not Enough

Multithreading versus multiprocessing

Hyper-threading versus physical CPU cores

Creating a pool of workers

Sharing data between processes

Remote processes

Distributed processing using multiprocessing

Distributed processing using IPyparallel

ipython_config.py

ipython_kernel_config.py

ipcontroller_config.py

ipengine_config.py

ipcluster_config.py

Summary

14. Extensions in C/C++, System Calls, and C/C++ Libraries

Introduction

Do you need C/C++ modules?

Windows

OS X

Linux/Unix

Calling C/C++ with ctypes

Platform-specific libraries

Windows

Linux/Unix

OS X

Making it easy

Calling functions and native types

Complex data structures

Arrays

Gotchas with memory management

CFFI

Complex data structures

Arrays

ABI or API?

CFFI or ctypes?

Native C/C++ extensions

A basic example

C is not Python – size matters

The example explained

static

PyObject*

Parsing arguments

C is not Python – errors are silent or lethal

Calling Python from C – handling complex types

Summary

15. Packaging – Creating Your Own Libraries or Applications

Installing packages

Setup parameters

Packages

Entry points

Creating global commands

Custom setup.py commands

Package data

Testing packages

Unittest

py.test

Nosetests

C/C++ extensions

Regular extensions

Cython extensions

Wheels – the new eggs

Distributing to the Python Package Index

Summary

A. Bibliography

Index