Scrapy and Tor

Hard to categorize, so it ended up under "Programming Concepts" — you know how it is.

Integrating Scrapy with Tor is yet another rabbit hole.

Installation

Tor (The Onion Router) is free software for enabling anonymous communication. For a more accessible introduction, see Common Questions About TOR — if the link doesn't open, you're not quite ready to step through that door yet.

Installing Tor is straightforward:

$ brew install tor
# $ pip install nyx

Configuration

Currently, Tor cannot be used directly from mainland China — you need to configure a "front proxy" for it. This guide uses shadowsocks as the front proxy.

Create a configuration file named torrc under /usr/local/etc/tor. You can refer to the torrc.sample file there to see all available options and descriptions.

Key parameters:

Parameter	Description
ControlPort	Port for control programs; required if you use tools like nyx
Socks5Proxy	Upstream SOCKS proxy port
HTTPProxy	Upstream HTTP proxy port
HTTPSProxy	Upstream HTTPS proxy port
SocksPort	Port through which external programs access Tor
MaxCircuitDirtiness	Interval for automatic IP rotation

Final configuration:

ControlPort 9051
CookieAuthentication 1

# My local shadowsocks client provides an HTTP proxy at 127.0.0.1:1087
HTTPProxy 127.0.0.1:1087
HTTPSProxy 127.0.0.1:1087

# Port through which external programs (e.g., a browser) access Tor
SocksPort 9000
MaxCircuitDirtiness 1 minutes

Now in Chrome, use SwitchySharp to create a new profile named Tor with manual proxy settings: SOCKS proxy at 127.0.0.1, port 9000 (matching the configured SocksPort).

Select the newly created profile in SwitchySharp and test browsing — it works.

I actually spent an entire afternoon on this step. I finally got it right only after carefully reading the documentation to find the correct HTTPProxy configuration. (ಥ _ ಥ)...

At this point, Scrapy still cannot use Tor directly. There needs to be an intermediate layer that converts the SOCKS proxy provided by Tor into an HTTP proxy.

Privoxy

Privoxy is an HTTP filtering proxy that is commonly used alongside Tor.

$ brew install privoxy

Find the config file under /usr/local/etc/privoxy (create it if it doesn't exist), and add the following:

listen-address 0.0.0.0:8118
forward-socks5 / 127.0.0.1:9000 .

The first line sets Privoxy to listen on port 8118 for any IP address. The second line sets the local SOCKS5 proxy client port — don't forget the trailing space and period at the end.

Then run:

privoxy

This starts Privoxy using the /usr/local/etc/privoxy/config configuration file.

Run the following command to check the running status:

netstat -na | grep 8118

If you see output like this, it's working:

......
tcp4       0      0  127.0.0.1.8118         *.*                    LISTEN

You can also use SwitchySharp to create a proxy profile and verify it works. For more details, see How to Use Privoxy.

Scrapy Integration

class ProxyMiddleware(object):
    # Single proxy

    def __init__(self, settings):
        self.proxy = settings.get('PROXY')

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings)

    def process_request(self, request, spider):
        request.meta['proxy'] = self.proxy

In settings.py:

PROXY = 'http://127.0.0.1:8118'
DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.retry.RetryMiddleware': 90,
    'scrapydemo.middlewares.ProxyMiddleware': 100,
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
}