Datacenter proxies with Scrapy

Scrapy sends a request through a proxy when you set meta["proxy"]. This page uses the datacenter tier: a US datacenter IP that changes for every new connection.

What you need

• Your proxy API key, from the Proxies page in the portal.
• Scrapy 2.13 or later (pip install scrapy). The examples use the current async def start() entry point.

export PB_PROXY_KEY="YOUR_KEY"

A spider that uses the proxy

import os
import scrapy

PROXY = f"http://{os.environ['PB_PROXY_KEY']}:[email protected]:3128"


class IpSpider(scrapy.Spider):
    name = "ip"

    async def start(self):
        for i in range(4):
            yield scrapy.Request(
                f"https://api.ipify.org/?n={i}",
                meta={"proxy": PROXY},
                dont_filter=True,
            )

    def parse(self, response):
        self.logger.info("exit address: %s", response.text)

Run it with scrapy runspider ip_spider.py. Scrapy's proxy middleware reads the credentials from the URL, so you do not set a header yourself.

On older Scrapy, define start_requests() instead of start(). The request itself is the same.

One IP, or a new one each time

By default Scrapy reuses connections. All four requests above leave from the same address, because the proxy changes the exit when a connection opens, not on every request.

To get a new IP for every request, tell Scrapy to close each connection:

class IpSpider(scrapy.Spider):
    name = "ip"
    custom_settings = {"DEFAULT_REQUEST_HEADERS": {"Connection": "close"}}

Reuse is cheaper and gives you a steady address for a few requests in a row. Closing is what you want for wide crawls where each request should look like a different visitor. See how usage is counted for the cost difference.

Be polite and be fast

Set DOWNLOAD_DELAY, AUTOTHROTTLE_ENABLED and ROBOTSTXT_OBEY for the sites you crawl. A proxy hides your address, not your request rate.

Errors

What you see	What it means
407 on the CONNECT request	The API key is wrong or missing.
400 Invalid proxy password settings	The password asked for something this tier does not do: a country other than US, a session or a TTL.
403 from the target	The site is refusing datacenter addresses. Use a residential tier such as standard.
A 400 that names an unknown tier	The tier word is a close misspelling of a real tier, such as datacentre. Fix the spelling. A password that is not close to any tier, such as x, uses your account's default tier.

Related

• Proxy connection format and tiers
• Datacenter proxies with Python requests
• Datacenter proxies