Python for Secret Agents - Volume II电子书

Python for Secret Agents Volume II

Table of Contents

Python for Secret Agents Volume II

Credits

About the Author

About the Reviewer

www.PacktPub.com

Support files, eBooks, discount offers, and more

Why subscribe?

Free access for Packt account holders

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Errata

Piracy

Questions

1. New Missions – New Tools

Background briefing on tools

Doing a Python upgrade

Preliminary mission to upgrade pip

Background briefing: review of the Python language

Using variables to save results

Using the sequence collections: strings

Using other common sequences: tuples and lists

Using the dictionary mapping

Comparing data and using the logic operators

Using some simple statements

Using compound statements for conditions: if

Using compound statements for repetition: for and while

Defining functions

Creating script files

Mission One – upgrade Beautiful Soup

Getting an HTML page

Navigating the HTML structure

Doing other upgrades

Mission to expand our toolkit

Scraping data from PDF files

Sidebar on the ply package

Building our own gadgets

Getting the Arduino IDE

Getting a Python serial interface

Summary

2. Tracks, Trails, and Logs

Background briefing – web servers and logs

Understanding the variety of formats

Getting a web server log

Writing a regular expression for parsing

Introducing some regular expression rules and patterns

Finding a pattern in a file

Using regular expression suffix operators

Capturing characters by name

Looking at the CLF

Reading and understanding the raw data

Reading a gzip compressed file

Reading remote files

Studying a log in more detail

What are they downloading?

Trails of activity

Who is this person?

Using Python to run other programs

Processing whois queries

Breaking a request into stanzas and lines

Alternate stanza-finding algorithm

Making bulk requests

Getting logs from a server with ftplib

Building a more complete solution

Summary

3. Following the Social Network

Background briefing – images and social media

Accessing web services with urllib or http.client

Who's doing the talking?

Starting with someone we know

Finding our followers

What do they seem to be talking about?

What are they posting?

Deep Under Cover – NLTK and language analysis

Summary

4. Dredging up History

Background briefing–Portable Document Format

Extracting PDF content

Using generator expressions

Writing generator functions

Filtering bad data

Writing a context manager

Writing a PDF parser resource manager

Extending the resource manager

Getting text data from a document

Displaying blocks of text

Understanding tables and complex layouts

Writing a content filter

Filtering the page iterator

Exposing the grid

Making some text block recognition tweaks

Emitting CSV output

Summary

5. Data Collection Gadgets

Background briefing: Arduino basics

Organizing a shopping list

Getting it right the first time

Starting with the digital output pins

Designing an external LED

Assembling a working prototype

Mastering the Arduino programming language

Using the arithmetic and comparison operators

Using common processing statements

Hacking and the edit, download, test and break cycle

Seeing a better blinking light

Simple Arduino sensor data feed

Collecting analog data

Collecting bulk data with the Arduino

Controlling data collection

Data modeling and analysis with Python

Collecting data from the serial port

Formatting the collected data

Crunching the numbers

Creating a linear model

Reducing noise with a simple filter

Solving problems adding an audible alarm

Summary

Index