Linux Shell Scripting Cookbook - Third Edition电子书

Title Page

Copyright

Credits

About the Authors

About the Reviewer

www.PacktPub.com

Customer Feedback

Preface

What this book covers

What you need for this book

Who this book is for

Sections

Getting ready

How to do it…

How it works…

There's more…

See also

Conventions

Reader feedback

Customer support

Downloading the example code

Downloading the color images of this book

Errata

Piracy

Questions

Shell Something Out

Introduction

Displaying output in a terminal

Getting ready

How to do it...

How it works...

There's more...

Escaping newline in echo

Printing a colored output

Using variables and environment variables

Getting ready

How to do it...

There's more...

Finding the length of a string

Identifying the current shell

Checking for super user

Modifying the Bash prompt string (username@hostname:~$)

Function to prepend to environment variables

How to do it...

How it works...

Math with the shell

How to do it...

Playing with file descriptors and redirection

Getting ready

How to do it...

How it works...

There's more...

Redirection from a file to a command

Redirecting from a text block enclosed within a script

Custom file descriptors

Arrays and associative arrays

Getting ready

How to do it...

There's more...

Defining associative arrays

Listing of array indexes

Visiting aliases

How to do it...

There's more...

Escaping aliases

Listing aliases

Grabbing information about the terminal

Getting ready

How to do it...

Getting and setting dates and delays

Getting ready

How to do it...

How it works...

There's more...

Producing delays in a script

Debugging the script

How to do it...

How it works...

There's more...

Shebang hack

Functions and arguments

How to do it...

There's more...

The recursive function

Reading the return value (status) of a command

Passing arguments to commands

Sending output from one command to another

Getting ready

How to do it...

There's more...

Spawning a separate process with subshell

Subshell quoting to preserve spacing and the newline character

Reading n characters without pressing the return key

How to do it...

Running a command until it succeeds

How to do it...

How it works...

There's more...

A faster approach

Adding a delay

Field separators and iterators

Getting ready

How to do it...

Comparisons and tests

How to do it...

Customizing bash with configuration files

How to do it...

Have a Good Command

Introduction

Concatenating with cat

How to do it...

There's more...

Getting rid of extra blank lines

Displaying tabs as ^I

Line numbers

Recording and playing back terminal sessions

Getting ready

How to do it...

How it works...

Finding files and file listing

Getting ready

How to do it...

There's more...

Search based on name or regular expression match

Negating arguments

Searching based on the directory depth

Searching based on file type

Searching by file timestamp

Searching based on file size

Matching based on file permissions and ownership

Performing actions on files with find

Deleting based on file matches

Executing a command

Skipping specified directories when using the find command

Playing with xargs

Getting ready

How to do it...

How it works...

There's more...

Passing formatted arguments to a command by reading stdin

Using xargs with find

Counting the number of lines of C code in a source code directory

While and subshell trick with stdin

Translating with tr

Getting ready

How to do it...

How it works...

There's more...

Deleting characters using tr

Complementing character sets

Squeezing characters with tr

Character classes

Checksum and verification

Getting ready

How to do it...

How it works...

There's more...

Checksum for directories

Cryptographic tools and hashes

How to do it...

Sorting unique and duplicate lines

Getting ready

How to do it...

How it works...

There's more...

Sorting according to keys or columns

uniq

Temporary file naming and random numbers

How to do it...

How it works...

Splitting files and data

How to do it...

There's more...

Specifying a filename prefix for the split files

Slicing filenames based on extensions

How to do it...

How it works...

Renaming and moving files in bulk

Getting ready

How to do it...

How it works...

Spell–checking and dictionary manipulation

How to do it...

How it works...

Automating interactive input

Getting ready

How to do it...

How it works...

There's more...

Automating with expect

Making commands quicker by running parallel processes

How to do it...

How it works...

There's more...

Examining a directory, files and subdirectories in it

Getting ready

How to do it...

Generating a tree view of a directory.

Generating a summary of files and sub-directories

File In, File Out

Introduction

Generating files of any size

How to do it...

The intersection and set difference (A-B) on text files

Getting ready

How to do it...

How it works...

Finding and deleting duplicate files

Getting ready

How to do it...

How it works...

Working with file permissions, ownership, and the sticky bit

How to do it...

There's more...

Changing ownership

Setting the sticky bit

Applying permissions recursively to files

Applying ownership recursively

Running an executable as a different user (setuid)

Making files immutable

Getting ready

How to do it...

Generating blank files in bulk

Getting ready

How to do it...

Finding symbolic links and their targets

How to do it...

How it works...

Enumerating file type statistics

Getting ready

How to do it...

How it works...

Using loopback files

How to do it...

How it works...

There's more...

Creating partitions inside loopback images

Mounting loopback disk images with partitions more quickly

Mounting ISO files as loopback

Flush changing immediately with sync

Creating ISO files and hybrid ISO

Getting ready

How to do it...

There's more...

Hybrid ISO that boots off a flash drive or hard disk

Burning an ISO from the command line

Playing with the CD-ROM tray

Finding the difference between files, and patching

How to do it...

There's more...

Generating difference against directories

Using head and tail for printing the last or first 10 lines

How to do it...

Listing only directories - alternative methods

Getting ready

How to do it...

How it works...

Fast command-line navigation using pushd and popd

Getting ready

How to do it...

There's more...

pushd and popd are useful when there are more than three directory paths used. However, when you use only two locations, there is an alternative and easier way, that is, cd -.

Counting the number of lines, words, and characters in a file

How to do it...

Printing the directory tree

Getting ready

How to do it...

There's more...

HTML output for tree

Manipulating video and image files

Getting ready

Extracting Audio from a movie file (mp4)

How to do it...

Making a video from a set of still images

How to do it...

How it works...

Creating a panned video from a still camera shot

How to do it...

How it works...

Texting and Driving

Introduction

Using regular expressions

How to do it...

Position markers

Identifiers

Count modifiers

Other

There's more...

How it works...

There's more...

Treatment of special characters

Visualizing regular expressions

Searching and mining text inside a file with grep

How to do it...

There's more...

Recursively searching many files

Ignoring case in patterns

grep by matching multiple patterns

Including and excluding files in a grep search

Using grep with xargs with the zero-byte suffix

Silent output for grep

Printing lines before and after text matches

Cutting a file column-wise with cut

How to do it...

There's more

Specifying the range of characters or bytes as fields

Using sed to perform text replacement

How to do it...

There's more...

Removing blank lines

Performing replacement directly in the file

Matched string notation ()

Substring match notation (\1)

Combining multiple expressions

Quoting

Using awk for advanced text processing

Getting ready

How to do it...

How it works...

There's more...

Special variables

Passing an external variable to awk

Reading a line explicitly using getline

Filtering lines processed by awk with filter patterns

Setting delimiters for fields

Reading the command output from awk

Associative arrays in Awk

Using loop inside awk

String manipulation functions in awk

Finding the frequency of words used in a given file

Getting ready

How to do it...

How it works...

See also

Compressing or decompressing JavaScript

Getting ready

How to do it...

How it works...

See also

Merging multiple files as columns

How to do it...

See also

Printing the nth word or column in a file or line

How to do it...

See also

Printing text between line numbers or patterns

Getting ready

How to do it...

See also

Printing lines in the reverse order

Getting ready

How to do it...

How it works...

Parsing e-mail address and URLs from text

How to do it...

How it works...

See also

Removing a sentence in a file containing a word

Getting ready

How to do it...

How it works...

See also

Replacing a pattern with text in all the files in a directory

How to do it...

How it works...

There's more...

Text slicing and parameter operations

How to do it...

See also

Tangled Web? Not At All!

Introduction

Downloading from a web page

Getting ready

How to do it...

How it works...

There's more...

Restricting the download speed

Resume downloading and continue

Copying a complete website (mirroring)

Accessing pages with HTTP or FTP authentication

Downloading a web page as plain text

Getting ready

How to do it...

A primer on cURL

Getting ready

How to do it...

How it works...

There's more...

Continuing and resuming downloads

Setting the referer string with cURL

Cookies with cURL

Setting a user agent string with cURL

Specifying a bandwidth limit on cURL

Specifying the maximum download size

Authenticating with cURL

Printing response headers excluding data

See also

Accessing unread Gmail e-mails from the command line

How to do it...

How it works...

See also

Parsing data from a website

How to do it...

How it works...

See also

Image crawler and downloader

How to do it...

How it works...

See also

Web photo album generator

Getting ready

How to do it...

How it works...

See also

Twitter command-line client

Getting ready

How to do it...

How it works...

See also

Accessing word definitions via a web server

Getting ready

How to do it...

How it works...

See also

Finding broken links in a website

Getting ready

How to do it...

How it works...

See also

Tracking changes to a website

Getting ready

How to do it...

How it works...

See also

Posting to a web page and reading the response

Getting ready

How to do it...

See also

Downloading a video from the Internet

Getting ready

How to do it...

How it works...

Summarizing text with OTS

Getting ready

How to do it...

How it works...

Translating text from the command line

Getting ready

How to do it...

How it works...

Repository Management

Introduction

Creating a new git repository

Getting ready

How to do it...

How it works...

Cloning a remote git repository

How to do it...

Adding and committing changes with git

How to do it...

Creating and merging branches with git

Getting ready...

How to do it...

How it works...

There's more...

Merging branches

How to do it...

How it works...

There's more...

Sharing your work

How to do it...

Pushing a branch to a server

How to do it...

Checking the status of a git repository

How to do it...

How it works...

Viewing git history

How to do it...

Finding bugs

How to do it...

There's more...

How to do it...

How it works...

Tagging snapshots

How to do it...

Committing message ethics

How to do it...

Using fossil

Getting ready

How to do it...

Creating a new fossil repository

How to do it...

How it works...

There's more...

Web interface to fossil

How to do it...

Making a repository available to remote users

Cloning a remote fossil repository

How to do it...

How it works...

Opening a fossil project

How to do it...

How it works...

There's more...

Adding and committing changes with fossil

How to do it...

There's more...

Using branches and forks with fossil

How to do it

How it works...

There's more...

Merging forks and branches

How to do it...

Sharing your work with fossil

How to do it...

How it works...

Updating your local fossil repository

How to do it...

Checking the status of a fossil repository

How to do it...

Viewing fossil history

How to do it...

Finding bugs

How to do it...

There's more...

Tagging snapshots

How to do it...

There's more...

The Backup Plan

Introduction

Archiving with tar

Getting ready

How to do it...

How it works...

There's more...

Appending files to an archive

Extracting files and folders from an archive

stdin and stdout with tar

Concatenating two archives

Updating files in an archive with a timestamp check

Comparing files in the archive and filesystem

Deleting files from the archive

Compression with the tar archive

Excluding a set of files from archiving

Excluding version control directories

Printing the total bytes

See also

Archiving with cpio

How to do it...

How it works...

Compressing data with gzip

How to do it...

There's more...

Gzip with tarball

zcat - reading gzipped files without extracting

Compression ratio

Using bzip2

Using lzma

See also

Archiving and compressing with zip

How to do it...

How it works...

Faster archiving with pbzip2

Getting ready

How to do it...

How it works...

There's more...

Manually specifying the number of CPUs

Specifying the compression ratio

Creating filesystems with compression

Getting ready

How to do it...

There's more...

Excluding files while creating a squashfs file

Backing up snapshots with rsync

How to do it...

How it works...

There's more...

Excluding files while archiving with rsync

Deleting non-existent files while updating rsync backup

Scheduling backups at intervals

Differential archives

How to do it...

How it works...

Creating entire disk images using fsarchiver

Getting ready

How to do it...

How it works...

The Old-Boy Network

Introduction

Setting up the network

Getting ready

How to do it...

There's more...

Printing the list of network interfaces

Displaying IP addresses

Spoofing the hardware address (MAC address)

Name server and DNS (Domain Name Service)

DNS lookup

Showing routing table information

See also

Let us ping!

How to do it...

There's more...

Round Trip Time

Sequence number

Time to live

Limiting the number of packets to be sent

Return status of the ping command

Tracing IP routes

How to do it...

Listing all available machines on a network

Getting ready

How to do it...

How it works...

There's more...

Parallel pings

Using fping

See also

Running commands on a remote host with SSH

Getting ready

How to do it...

There's more...

SSH with compression

Redirecting data into stdin of remote host shell commands

Running graphical commands on a remote machine

How to do it...

See also

Transferring files through the network

Getting ready

How to do it...

There's more...

Automated FTP transfer

SFTP (Secure FTP)

The rsync command

SCP (secure copy program)

Recursive copying with scp

See also

Connecting to a wireless network

Getting ready

How to do it...

How it works...

See also

Password-less auto-login with SSH

Getting ready

How to do it...

Port forwarding using SSH

How to do it...

There's more...

Non-interactive port forward

Reverse port forwarding

Mounting a remote drive at a local mount point

Getting ready

How to do it...

See also

Network traffic and port analysis

Getting ready

How to do it...

How it works...

There's more...

Opened port and services using netstat

Measuring network bandwidth

How to do it...

Creating arbitrary sockets

Getting ready

How to do it...

There's more...

Quickly copying files over the network

Creating a broadcasting server

How it works...

Building a bridge

Getting ready

How to do it...

Sharing an Internet connection

Getting ready

How to do it...

How it works

Basic firewall using iptables

How to do it...

How it works...

There's more...

Creating a Virtual Private Network

Getting ready

How to do it...

Creating certificates

Configuring OpenVPN on the server

Configuring OpenVPN on the client

Starting the server

Starting and testing a client

Put On the Monitors Cap

Introduction

Monitoring disk usage

Getting ready

How to do it...

There's more...

Displaying disk usage in KB, MB, or blocks

Displaying the grand total sum of disk usage

Printing sizes in specified units

Excluding files from the disk usage calculation

Finding the ten largest size files from a given directory

Disk free information

Calculating the execution time for a command

How to do it...

How it works...

Collecting information about logged in users, boot logs, and boot failures

Getting ready

How to do it...

Listing the top ten CPU– consuming processes in an hour

Getting ready

How to do it...

How it works...

See also

Monitoring command outputs with watch

How to do it...

There's more

Highlighting the differences in the watch output

Logging access to files and directories

Getting ready

How to do it...

How it works...

Logging with syslog

Getting ready

How to do it...

See also

Managing log files with logrotate

Getting ready

How to do it...

How it works...

Monitoring user logins to find intruders

Getting ready

How to do it...

How it works...

Monitoring remote disk usage health

Getting ready

How to do it...

How it works...

See also

Determining active user hours on a system

Getting ready

How to do it...

How it works...

Measuring and optimizing power usage

Getting ready

How to do it...

Monitoring disk activity

Getting ready

How to do it...

Checking disks and filesystems for errors

Getting ready

How to do it...

How it works...

Examining disk health

Getting ready

How to do it...

How it works

Getting disk statistics

Getting ready

How to do it...

How it works

There's more

Administration Calls

Introduction

Gathering information about processes

Getting ready

How to do it...

How it works...

There's more...

Showing environment variables for a process

Creating a tree view of processes

Sorting ps output

Filters with ps for real user or ID, effective user or ID

TTY filter for ps

Information about process threads

Specifying the output width and columns to be displayed

What's what – which, whereis, whatis, and file

How to do it...

Finding the process ID from the given command names

Determining how busy a system is

The top command

See also...

Killing processes, and sending and responding to signals

Getting ready

How to do it...

There's more...

The kill family of commands

Capturing and responding to signals

Sending messages to user terminals

Getting ready

How to do it...

Sending one message to one user

Holding a conversation with another user

Sending a message to all users

The /proc filesystem

How to do it...

Gathering system information

How to do it...

Scheduling with a cron

Getting ready

How to do it...

How it works...

There's more...

Specifying environment variables

Running commands at system start-up/boot

Viewing the cron table

Removing the cron table

Database styles and uses

Getting ready

How to do it...

There's more...

Creating a table

Inserting a row into an SQL database

Selecting rows from a SQL database

Writing and reading SQLite databases

Getting ready

How to do it...

How it works...

There's more...

Writing and reading a MySQL database from Bash

Getting ready

How to do it...

How it works...

User administration scripts

How to do it...

How it works...

Bulk image resizing and format conversion

Getting ready

How to do it...

How it works...

See also

Taking screenshots from the terminal

Getting ready

How to do it...

Managing multiple terminals from one

Getting ready

How to do it...

Tracing the Clues

Introduction

Tracing packets with tcpdump

Getting ready

How to do it...

Displaying only HTTP packets

Displaying only HTTP packets generated by this host

Viewing the packet payload as well as headers

How it works...

Finding packets with ngrep

Getting ready

How to do it...

How it works...

There's more...

Tracing network routes with ip

Getting ready

How to do it...

Reporting routes with ip route

Tracing recent IP connections and the ARP table

Tracing a route

How it works...

Tracing system calls with strace

Getting ready

How to do it...

How it works...

Tracing dynamic library functions with ltrace

Getting ready

How to do it...

How it works...

There's more...

Tuning a Linux System

Introduction

Identifying services

Getting ready

How to do it...

systemd-based computers

RedHat-based computers

Debian-based computers

There's more

Gathering socket data with ss

Getting ready

How to do it...

Displaying the status of tcp sockets

Tracing applications listening on ports

How it works

Gathering system I/O usage with dstat

Getting ready

How to do it...

Viewing system activity

How it works

There's more...

Identifying a resource hog with pidstat

Getting ready

How to do it...

How it works

Tuning the Linux kernel with sysctl

Getting started

How to do it...

Tuning the task scheduler

Tuning a network

How it works

There's more...

Tuning a Linux system with config files

Getting ready

How to do it...

How it works

Changing scheduler priority using the nice command

How to do it...

How it works

There's more

Containers, Virtual Machines, and the Cloud

Introduction

Using Linux containers

Getting ready

How to do it...

Creating a privileged container

Starting a container

Stopping a container

Listing known containers

Displaying container information

Creating an unprivileged container

Creating an Ethernet bridge

How it works...

Using Docker

Getting ready

How to do it...

Finding a container

Downloading a container

Starting a Docker container

Listing the Docker sessions

Attaching your display to a running Docker container

Stopping a Docker session

Removing a Docker instance

How it works

Using Virtual Machines in Linux

Getting ready

How to do it...

Linux in the cloud

Getting ready

Ubuntu 16.10

OpenSuSE Tumbleweed

How to do it...

Configuring OwnCloud

There's more...