Zyte Scrapy Cloud

The Manager for Your Robot Workers. Automated web scraping, proxies, and cloud spiders.

What is Zyte?

Think of Zyte Scrapy Cloud as a manager for your robot workers.

When you write a web crawler (a "spider"), it's like building a robot that goes out to the internet to collect information for you. But running this robot on your own computer can be annoying: you have to keep your computer on, internet might drop, or your IP address might get blocked.

Zyte is a cloud platform where you can send your robots to live and work 24/7.

You upload your spider code to Zyte.
Zyte runs your spiders on their servers 24/7.
Schedule them to run automatically (e.g. "every morning at 6 AM").
Inspect all collected datasets in a clean dashboard.

Developer Tier & Features

Zyte offers a generous free tier for developers:

1 Free Forever Scrapy Cloud Unit: Run one spider continuously.
Unlimited Team Members: Invite your entire dev team.
Unlimited Projects & Requests: No arbitrary cap on requests.
120-Day Data Retention: Scraped data is preserved for 4 months.

Technical Examples for Developers

Here is how to deploy a standard Scrapy project to Zyte Scrapy Cloud.

1. Prerequisites

Install the shub command-line deployment tool:

pip install shub

2. Configuration (`scrapinghub.yml`)

Create a scrapinghub.yml file in your project root:

projects:
  default: 12345

stacks:
  default: scrapy:2.11

requirements:
  file: requirements.txt

3. Deploying to Cloud

Deploy your code with a single command:

shub deploy

4. Scheduling a Spider

shub schedule quotes

5. Example Scrapy Spider (`quotes.py`)

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = ['http://quotes.toscrape.com/page/1/']

    def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').get(),
                'author': quote.css('small.author::text').get(),
            }

        next_page = response.css('li.next a::attr(href)').get()
        if next_page is not None:
            yield response.follow(next_page, callback=self.parse)

Navigation Menu