Squid Proxy Server 3.1: Beginner's Guide电子书

Squid Proxy Server 3.1 Beginner's Guide

Table of Contents

Squid Proxy Server 3.1 Beginner's Guide

Credits

About the Author

About the Reviewers

www.PacktPub.com

Support files, eBooks, discount offers, and more

Why Subscribe?

Free Access for Packt account holders

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Time for action - heading

What just happened?

Pop quiz

Have a go hero - heading

Reader feedback

Customer support

Errata

Piracy

Questions

1. Getting Started with Squid

Proxy server

Reverse proxy

Getting Squid

Time for action – identifying the right version

What just happened?

Methods of obtaining Squid

Using source archives

Time for action – downloading Squid

What just happened?

Obtaining the latest source code from Bazaar VCS

Time for action – using Bazaar to obtain source code

What just happened?

Have a go hero – fetching the source code

Using binary packages

Installing Squid

Installing Squid from source code

Compiling Squid

Why compile?

Uncompressing the source archive

Configure or system check

--prefix

--enable-gnuregex

--disable-inline

--disable-optimizations

--enable-storeio

--enable-removal-policies

--enable-icmp

--enable-delay-pools

--enable-esi

--enable-useragent-log

--enable-referer-log

--disable-wccp

--disable-wccpv2

--disable-snmp

--enable-cachemgr-hostname

--enable-arp-acl

--disable-htcp

--enable-ssl

--enable-cache-digests

--enable-default-err-language

--enable-err-languages

--disable-http-violations

--enable-ipfw-transparent

--enable-ipf-transparent

--enable-pf-transparent

--enable-linux-netfliter

--enable-follow-x-forwarded-for

--disable-ident-lookups

--disable-internal-dns

--enable-default-hostsfile

--enable-auth

Old Syntax

New Syntax

--enable-auth-basic

--enable-auth-ntlm

--enable-auth-negotiate

--enable-auth-digest

--enable-ntlm-fail-open

--enable-external-acl-helpers

--disable-translation

--disable-auto-locale

--disable-unlinkd

--with-default-user

--with-logdir

--with-pidfile

--with-aufs-threads

--without-pthreads

--with-openssl

--with-large-files

--with-filedescriptors

Have a go hero – file descriptors

Time for action – running the configure command

What just happened?

Have a go hero – debugging configure errors

Time for action – compiling the source

What just happened?

Time for action – installing Squid

What just happened?

Time for action – exploring Squid files

bin

bin/squidclient

etc

etc/squid.conf

etc/squid.conf.default

etc/squid.conf.documented

libexec

libexec/cachemgr.cgi

sbin

sbin/squid

share

share/errors

share/icons

share/man

var

var/cache

var/logs

What just happened?

Installing Squid from binary packages

Fedora, CentOS or Red Hat

Debian or Ubuntu

FreeBSD

OpenBSD or NetBSD

Dragonfly BSD

Gentoo

Arch Linux

Pop quiz

Summary

2. Configuring Squid

Quick start

Syntax of the configuration file

Types of directives

Single valued directives

Boolean-valued or toggle directives

Multi-valued directives

Directives with time as a value

Directives with file or memory size as values

Have a go hero – categorize the directives

HTTP port

Time for action – setting the HTTP port

What just happened?

Access control lists

Time for action – constructing simple ACLs

What just happened?

Have a go hero – understanding the pre-defined ACLs

Controlling access to the proxy server

HTTP access control

Time for action – combining ACLs and HTTP access

What just happened?

HTTP reply access

ICP access

HTCP access

HTCP CLR access

Miss access

Ident lookup access

Cache peers or neighbors

Declaring cache peers

Time for action – adding a cache peer

What just happened?

Quickly restricting access to domains using peers

Advanced control on access using peers

Caching web documents

Using main memory (RAM) for caching

In-transit objects or current requests

Hot or popular objects

Negatively cached objects

Specifying cache space in RAM

Time for action – specifying space for memory caching

What just happened?

Have a go hero – calculating cache_mem for your machine

Maximum object size in memory

Memory cache mode

Using hard disks for caching

Specifying the storage space

Storage types

Choosing a directory name or location

Time for action – creating a cache directory

What just happened?

Declaring the size of the cache

Configuring the number of sub directories

Read-only cache

Time for action – adding a cache directory

What just happened?

Cache directory selection

Cache object size limits

Setting limits on object replacement

Cache replacement policies

Least recently used (LRU)

Greedy dual size frequency (GDSF)

Least frequently used with dynamic aging (LFUDA)

Tuning Squid for enhanced caching

Selective caching

Time for action – preventing the caching of local content

What just happened?

Refresh patterns for cached objects

Time for action – calculating the freshness of cached objects

What just happened?

Options for refresh pattern

override-expire

override-lastmod

reload-into-ims

ignore-reload

ignore-no-cache

ignore-no-store

ignore-must-revalidate

ignore-private

ignore-auth

refresh-ims

Have a go hero – forcing the Google homepage to be cached for longer

Aborting the partial retrievals

Caching the failed requests

Playing around with HTTP headers

Controlling HTTP headers in requests

Controlling HTTP headers in responses

Replacing the contents of HTTP headers

DNS server configuration

Specifying the DNS program path

Controlling the number of DNS client processes

Setting the DNS name servers

Time for action – adding DNS name servers

What just happened?

Setting the hosts file

Default domain name for requests

Timeout for DNS queries

Caching the DNS responses

Setting the size of the DNS cache

Logging

Log formats

Log file rotation or log file backups

Log access

Buffered logs

Strip query terms

URL rewriters and redirectors

Other configuration directives

Setting the effective user for running Squid

Configuring hostnames for the proxy server

Hostname visible to everyone

Unique hostname for the server

Controlling the request forwarding

Always direct

Never direct

Hierarchy stoplist

Broken posts

TCP outgoing address

PID filename

Client netmask

Pop quiz

Summary

3. Running Squid

Command line options

Getting a list of available options

Time for action – listing the options

What just happened?

Getting information about our Squid installation

Time for action – finding out the Squid version

What just happened?

Creating cache or swap directories

Time for action – creating cache directories

What just happened?

Have a go hero – adding cache directories

Using a different configuration file

Getting verbose output

Time for action – debugging output in the console

What just happened?

Full debugging output on the terminal

Running as a normal process

Parsing the Squid configuration file for errors or warnings

Time for action – testing our configuration file

What just happened?

Sending various signals to a running Squid process

Reloading a new configuration file in a running process

Shutting down the Squid process

Interrupting or killing a running Squid process

Checking the status of a running Squid process

Have a go hero – check the return value

Sending a running process in to debug mode

Rotating the log files

Forcing the storage metadata to rebuild

Double checking swap during rebuild

Automatically starting Squid at system startup

Adding Squid command to /etc/rc.local file

Adding init script

Time for action – adding the init script

What just happened?

Pop quiz

Summary

4. Getting Started with Squid's Powerful ACLs and Access Rules

Access control lists

Fast and slow ACL types

Source and destination IP address

Time for action – constructing ACL lists using IP addresses

What just happened?

Time for action – using a range of IP addresses to build ACL lists

What just happened?

Have a go hero – make a list of the client IP addresses in your network

Identifying local IP addresses

Client MAC addresses

Source and destination domain names

Time for action – constructing ACL lists using domain names

What just happened?

Have a go hero – make a list of domains hosted in your local network

Regular expressions for domain names

Destination port

Time for action – building ACL lists using destination ports

What just happened?

Local port name

HTTP methods

Identifying requests using the request protocol

Time for action – using a request protocol to construct access rules

What just happened?

Time-based ACLs

URL and URL path-based identification

Have a go hero – ACL list for audio content

Matching client usernames

Regular expressions for client usernames

Proxy authentication

Time for action – enforcing proxy authentication

Regular expressions for usernames

What just happened?

User limits

Maximum number of connections per client

Maximum logins per user

Identification based on various HTTP headers

User-agent or browser

Referer identification

Content type-based identification

Other HTTP headers

HTTP reply status

Identifying random requests

Access list rules

Access to HTTP protocol

Adapted HTTP access

HTTP access for replies

Access to other ports

ICP port

HTCP port

Purge access via HTCP

SNMP port

Enforcing limited access to neighbors

Time for action – denying miss_access to neighbors

What just happened?

Requesting neighbor proxy servers

Have a go hero – make a list of proxy servers in your network

Forwarding requests to remote servers

Ident lookup access

Controlled caching of web documents

URL rewrite access

HTTP header access

Custom error pages

Have a go hero – custom access denied page

Maximum size of the reply body

Logging requests selectively

Mixing ACL lists and rules – example scenarios

Handling caching of local content

Time for action – avoiding caching of local content

What just happened?

Denying access from external networks

Denying access to selective clients

Blocking the download of video content

Time for action – blocking video content

What just happened?

Special access for certain clients

Time for action – writing rules for special access

What just happened?

Limited access during working hours

Allowing some clients to connect to special ports

Testing access control with squidclient

Options for squidclient

Using the squidclient

Time for action – testing our access control example with squidclient

What just happened?

Time for action – testing a complex access control

What just happened?

Pop quiz

Summary

5. Understanding Log Files and Log Formats

Log messages

Cache log or debug log

Time for action – understanding the cache log

What just happened?

Have a go hero – exploring the cache log

Access log

Understanding the access log

Time for action – understanding the access log messages

What just happened?

Access log syntax

Time for action – analyzing a syntax to specify access log

What just happened?

Have a go hero – logging messages to the syslog module

Log format

Time for action – learning log format and format codes

What just happened?

Log formats provided by Squid

Time for action – customizing the access log with a new log format

What just happened?

Selective logging of requests

Time for action – using access_log to control logging of requests

What just happened?

Referer log

Time for action – enabling the referer log

What just happened?

Time for action – translating the referer logs to a human-readable format

What just happened?

Have a go hero – referer log

User agent log

Time for action – enabling user agent logging

What just happened?

Emulating HTTP server-like logs

Time for action – enabling HTTP server log emulation

What just happened?

Log file rotation

Have a go hero – rotate log files

Other log related features

Cache store log

Pop quiz

Summary

6. Managing Squid and Monitoring Traffic

Cache manager

Installing the Apache Web server

Time for action – installing Apache Web server

What just happened?

Configuring Apache for providing the cache manager web interface

Time for action – configuring Apache to use cachemgr.cgi

What just happened?

Accessing the cache manager web interface

Configuring Squid

Log in to cache manger

General Runtime Information

IP Cache Stats and Contents

FQDN Cache Statistics

HTTP Header Statistics

Traffic and Resource Counters

Request Forwarding Statistics

Cache Client List

Memory Utilization

Internal DNS Statistics

Have a go hero – exploring cache manager

Log file analyzers

Calamaris

Installing Calamaris

Time for action – installing Calamaris

What just happened?

Using Calamaris to generate statistics

Time for action – generating stats in plain text format

What just happened?

Have a go hero – exploring the reports

Time for action – generating graphical reports with Calamaris

What just happened?

Have a go hero – exploring Calamaris

Pop quiz

Summary

7. Protecting your Squid Proxy Server with Authentication

HTTP authentication

Basic authentication

Time for action – exploring Basic authentication

What just happened?

Database authentication

Configuring database authentication

NCSA authentication

Time for action – configuring NCSA authentication

What just happened?

NIS authentication

LDAP authentication

SMB authentication

PAM authentication

Time for action – configuring PAM service

What just happened?

MSNT authentication

Time for action – configuring MSNT authentication

What just happened?

MSNT multi domain authentication

SASL authentication

Time for action – configuring Squid to use SASL authentication

What just happened?

getpwnam authentication

POP3 authentication

RADIUS authentication

Time for action – configuring RADIUS authentication

What just happened?

Fake Basic authentication

Digest authentication

Time for action – configuring Digest authentication

What just happened?

File authentication

LDAP authentication

eDirectory authentication

Microsoft NTLM authentication

Samba's NTLM authentication

Fake NTLM authentication

Negotiate authentication

Time for action – configuring Negotiate authentication

What just happened?

Using multiple authentication schemes

Writing a custom authentication helper

Time for action – writing a helper program

What just happened?

Have a go hero – implementing the validation function

Making non-concurrent helpers concurrent

Common issues with authentication

Whitelisting selected websites

Challenge loops

Authentication in the intercept or transparent mode

Pop quiz

Summary

8. Building a Hierarchy of Squid Caches

Cache hierarchies

Reasons to use hierarchical caching

Problems with hierarchical caching

Avoiding a forwarding loop

Joining a cache hierarchy

Time for action – joining a cache hierarchy

What just happened?

ICP options

no-query

multicast-responder

closest-only

background-ping

HTCP options

htcp

htcp=oldsquid

htcp=no-clr

htcp=only-clr

htcp=no-purge-clr

htcp=forward-clr

Peer or neighbor selection

default

round-robin

weighted-round-robin

userhash

sourcehash

carp

multicast-siblings

Options for peer selection methods

weight

basetime

ttl

no-delay

digest-URL

no-digest

ssl

sslcert

sslkey

sslversion

sslcipher

ssloptions

sslcafile

sslcapath

sslcrlfile

sslflags

ssldomain

front-end-https

Other cache peer options

login=username:password

login=PASS

login=PASSTHRU

login=NEGOTIATE

connect-timeout

connect-fail-limit

max-conn

name

proxy-only

allow-miss

Controlling communication with peers

Domain-based forwarding

Time for action – configuring Squid for domain-based forwarding

What just happened?

Cache peer access

Time for action – forwarding requests to cache peers using ACLs

What just happened?

Have a go hero – join a cache hierarchy

Switching peer relationship

Time for action – configuring Squid to switch peer relationship

What just happened?

Controlling request redirects

hierarchy_stoplist

always_direct

never_direct

prefer_direct

nonhierarchical_direct

Have a go hero – proxy server behind a firewall

Peer communication protocols

Internet Cache Protocol

Cache digests

Squid and cache digest configuration

Digest generation

Digest bits per entry

Digest rebuild period

Digest rebuild chunk percentage

Digest swapout chunk

Digest rewrite period

Hypertext Caching Protocol

Pop quiz

Summary

9. Squid in Reverse Proxy Mode

What is reverse proxy mode?

Exploring reverse proxy mode

Configuring Squid as a server surrogate

HTTP port

HTTP options in reverse proxy mode

defaultsite

vhost

vport

allow-direct

protocol

ignore-cc

HTTPS port

HTTPS options in reverse proxy mode

defaultsite

vhost

version

cipher

options

clientca

cafile

capath

crlfile

dhparams

sslflags

NO_DEFAULT_CA

NO_SESSION_REUSE

VERIFY_CRL

VERIFY_CRL_ALL

sslcontext

vport

Have a go hero – exploring OpenSSL

Adding backend web servers

Cache peer options for reverse proxy mode

originserver

forcedomain

Time for action – adding backend web servers

What just happened?

Support for surrogate protocol

Understanding the surrogate protocol

Configuration options for surrogate support

httpd_accel_surrogate_id

httpd_accel_surrogate_remote

Support for ESI protocol

Configuring Squid for ESI support

Logging messages in web server log format

Ignoring the browser reloads

Time for action – configuring Squid to ignore the browser reloads

Using ignore-cc

Using ignore-reload

Using reload-into-ims

What just happened?

Access controls in reverse proxy mode

Squid in only reverse proxy mode

Squid in reverse proxy and forward proxy mode

Example configurations

Web server and Squid server on the same machine

Accelerating multiple backend web servers hosting one website

Accelerating multiple web servers hosting multiple websites

Have a go hero – set up a Squid proxy server in reverse proxy mode

Pop quiz

Summary

10. Squid in Intercept Mode

Interception caching

Time for action – understanding interception caching

What just happened?

Advantages of interception caching

Zero client configuration

Better control

Increased reliability

Problems with interception caching

Violates TCP/IP standards

Susceptible to routing problems

No authentication

Supports only HTTP interception

Client exposure

IP filtering

Protocol support

Security vulnerabilities

Have a go hero – interception caching for your network

Diverting HTTP traffic to Squid

Using a router's policy routing to divert requests

Using rule-based switching to divert requests

Using Squid server as a bridge

Using WCCP tunnel

Implementing interception caching

Configuring the network devices

Configuring the operating system

Enabling IP forwarding

Time for action – enabling IP forwarding

What just happened?

Redirecting packets to Squid

Time for action – redirecting HTTP traffic to Squid

What just happened?

Have a go hero – testing the traffic diversion

Configuring Squid

Configuring HTTP port

Pop quiz

Summary

11. Writing URL Redirectors and Rewriters

URL redirectors and rewriters

Understanding URL redirectors

HTTP status codes for redirection

Understanding URL rewriters

Issues with URL rewriters

Squid, URL redirectors, and rewriters

Communication interface

Time for action – exploring the message flow between Squid and redirectors

What just happened?

Time for action – writing a simple URL redirector program

What just happened?

Have a go hero – modify the redirector program

Concurrency

Handling whitespace in URLs

Using the uri_whitespace directive

Strip whitespaces

Deny URLs with whitespaces

Encode whitespaces in URLs

Chop URLs

Allow URLs with whitespaces

Making redirector programs intelligent

Writing our own URL redirector program

Time for action – writing our own template for a URL redirector

What just happened?

Have a go hero – extend the redirector program

Configuring Squid

Specifying the URL redirector program

Controlling redirector children

Controlling requests passed to the redirector program

Bypassing URL redirector programs when under heavy load

Rewriting the Host HTTP header

A special URL redirector – deny_info

Popular URL redirectors

SquidGuard

Squirm

Ad Zapper

Pop quiz

Summary

12. Troubleshooting Squid

Some common issues

Cannot write to log files

Time for action – changing the ownership of log files

What just happened?

Could not determine hostname

Cannot create swap directories

Time for action – fixing cache directory permissions

What just happened?

Failed verification of swap directories

Time for action – creating swap directories

What just happened?

Address already in use

Time for action – finding the program listening on a specific port

For Linux-based operating systems

For OpenBSD and NetBSD

For FreeBSD and DragonFlyBSD

What just happened?

URLs with underscore results in an invalid URL

Enforce hostname checks

Allow underscore

Squid becomes slow over time

The request or reply is too large

Access denied on the proxy server

Connection refused when reaching a sibling proxy server

Debugging problems

Time for action – debugging HTTP requests

What just happened?

Time for action – debugging access control

What just happened?

Have a go Hero – debugging HTTP responses

Getting help online and reporting bugs

Pop quiz

Summary

A. Pop Quiz Answers

Chapter 1, Getting Started with Squid

Chapter 2, Configuring Squid

Chapter 3, Running Squid

Chapter 4, Getting Started with Squid’s Powerful ACLs and Access Rules

Chapter 5, Understanding Log Files and Log Formats

Chapter 6, Managing Squid and Monitoring Traffic

Chapter 7, Protecting your Squid with Authentication

Chapter 8, Building a Hierarchy of Squid Caches

Chapter 9, Squid in Reverse Proxy Mode

Chapter 10, Squid in Intercept Mode

Chapter 11: Writing URL Redirectors and Rewriters

Chapter 12: Troubleshooting Squid

Index