Python Reinforcement Learning电子书

About Packt

Why subscribe?

Packt.com

Contributors

About the authors

Packt is searching for authors like you

Preface

Who this book is for

What this book covers

To get the most out of this book

Download the example code files

Conventions used

Get in touch

Reviews

Introduction to Reinforcement Learning

What is RL?

RL algorithm

How RL differs from other ML paradigms

Elements of RL

Agent

Policy function

Value function

Model

Agent environment interface

Types of RL environment

Deterministic environment

Stochastic environment

Fully observable environment

Partially observable environment

Discrete environment

Continuous environment

Episodic and non-episodic environment

Single and multi-agent environment

RL platforms

OpenAI Gym and Universe

DeepMind Lab

RL-Glue

Project Malmo

ViZDoom

Applications of RL

Education

Medicine and healthcare

Manufacturing

Inventory management

Finance

Natural Language Processing and Computer Vision

Summary

Questions

Further reading

Getting Started with OpenAI and TensorFlow

Setting up your machine

Installing Anaconda

Installing Docker

Installing OpenAI Gym and Universe

Common error fixes

OpenAI Gym

Basic simulations

Training a robot to walk

OpenAI Universe

Building a video game bot

TensorFlow

Variables, constants, and placeholders

Variables

Constants

Placeholders

Computation graph

Sessions

TensorBoard

Adding scope

Summary

Questions

Further reading

The Markov Decision Process and Dynamic Programming

The Markov chain and Markov process

Markov Decision Process

Rewards and returns

Episodic and continuous tasks

Discount factor

The policy function

State value function

State-action value function (Q function)

The Bellman equation and optimality

Deriving the Bellman equation for value and Q functions

Solving the Bellman equation

Dynamic programming

Value iteration

Policy iteration

Solving the frozen lake problem

Value iteration

Policy iteration

Summary

Questions

Further reading

Gaming with Monte Carlo Methods

Monte Carlo methods

Estimating the value of pi using Monte Carlo

Monte Carlo prediction

First visit Monte Carlo

Every visit Monte Carlo

Let's play Blackjack with Monte Carlo

Monte Carlo control

Monte Carlo exploration starts

On-policy Monte Carlo control

Off-policy Monte Carlo control

Summary

Questions

Further reading

Temporal Difference Learning

TD learning

TD prediction

TD control

Q learning

Solving the taxi problem using Q learning

SARSA

Solving the taxi problem using SARSA

The difference between Q learning and SARSA

Summary

Questions

Further reading

Multi-Armed Bandit Problem

The MAB problem

The epsilon-greedy policy

The softmax exploration algorithm

The upper confidence bound algorithm

The Thompson sampling algorithm

Applications of MAB

Identifying the right advertisement banner using MAB

Contextual bandits

Summary

Questions

Further reading

Playing Atari Games

Introduction to Atari games

Building an Atari emulator

Getting started

Implementation of the Atari emulator

Atari simulator using gym

Data preparation

Deep Q-learning

Basic elements of reinforcement learning

Demonstrating basic Q-learning algorithm

Implementation of DQN

Experiments

Summary

Atari Games with Deep Q Network

What is a Deep Q Network?

Architecture of DQN

Convolutional network

Experience replay

Target network

Clipping rewards

Understanding the algorithm

Building an agent to play Atari games

Double DQN

Prioritized experience replay

Dueling network architecture

Summary

Questions

Further reading

Playing Doom with a Deep Recurrent Q Network

DRQN

Architecture of DRQN

Training an agent to play Doom

Basic Doom game

Doom with DRQN

DARQN

Architecture of DARQN

Summary

Questions

Further reading

The Asynchronous Advantage Actor Critic Network

The Asynchronous Advantage Actor Critic

The three As

The architecture of A3C

How A3C works

Driving up a mountain with A3C

Visualization in TensorBoard

Summary

Questions

Further reading

Policy Gradients and Optimization

Policy gradient

Lunar Lander using policy gradients

Deep deterministic policy gradient

Swinging a pendulum

Trust Region Policy Optimization

Proximal Policy Optimization

Summary

Questions

Further reading

Balancing CartPole

OpenAI Gym

Gym

Installation

Running an environment

Atari

Algorithmic tasks

MuJoCo

Robotics

Markov models

CartPole

Summary

Simulating Control Tasks

Introduction to control tasks

Getting started

The classic control tasks

Deterministic policy gradient

The theory behind policy gradient

DPG algorithm

Implementation of DDPG

Experiments

Trust region policy optimization

Theory behind TRPO

TRPO algorithm

Experiments on MuJoCo tasks

Summary

Building Virtual Worlds in Minecraft

Introduction to the Minecraft environment

Data preparation

Asynchronous advantage actor-critic algorithm

Implementation of A3C

Experiments

Summary

Learning to Play Go

A brief introduction to Go

Go and other board games

Go and AI research

Monte Carlo tree search

Selection

Expansion

Simulation

Update

AlphaGo

Supervised learning policy networks

Reinforcement learning policy networks

Value network

Combining neural networks and MCTS

AlphaGo Zero

Training AlphaGo Zero

Comparison with AlphaGo

Implementing AlphaGo Zero

Policy and value networks

preprocessing.py

features.py

network.py

Monte Carlo tree search

mcts.py

Combining PolicyValueNetwork and MCTS

alphagozero_agent.py

Putting everything together

controller.py

train.py

Summary

References

Creating a Chatbot

The background problem

Dataset

Step-by-step guide

Data parser

Data reader

Helper methods

Chatbot model

Training the data

Testing and results

Summary

Generating a Deep Learning Image Classifier

Neural Architecture Search

Generating and training child networks

Training the Controller

Training algorithm

Implementing NAS

child_network.py

cifar10_processor.py

controller.py

Method for generating the Controller

Generating a child network using the Controller

train_controller method

Testing ChildCNN

config.py

train.py

Additional exercises

Advantages of NAS

Summary

Predicting Future Stock Prices

Background problem

Data used

Step-by-step guide

Actor script

Critic script

Agent script

Helper script

Training the data

Final result

Summary

Capstone Project - Car Racing Using DQN

Environment wrapper functions

Dueling network

Replay memory

Training the network

Car racing

Summary

Questions

Further reading

Looking Ahead

The shortcomings of reinforcement learning

Resource efficiency

Reproducibility

Explainability/accountability

Susceptibility to attacks

Upcoming developments in reinforcement learning

Addressing the limitations

Transfer learning

Multi-agent reinforcement learning

Summary

References

Assessments

Chapter 1: Introduction to Reinforcement Learning

Chapter 2: Getting Started with OpenAI and TensorFlow

Chapter 3: The Markov Decision Process and Dynamic Programming

Chapter 4: Gaming with Monte Carlo Methods

Chapter 5: Temporal Difference Learning

Chapter 6: Multi-Armed Bandit Problem

Chapter 8: Atari Games with Deep Q Network

Chapter 9: Playing Doom with a Deep Recurrent Q Network

Chapter 10: The Asynchronous Advantage Actor Critic Network

Chapter 11: Policy Gradients and Optimization

Chapter 19: Capstone Project – Car Racing Using DQN

Other Books You May Enjoy

Leave a review - let other readers know what you think