Python SDK - Scrapely

The official Python client for Scrapely.io, a powerful API for web scraping and CAPTCHA solving. Our SDK provides a seamless developer experience with fully typed responses, auto-polling for task completion, and both synchronous and asynchronous support for high-concurrency applications.

Installation

You can install the SDK via pip:

pip install scrapely-python-client

Usage Examples

The SDK provides two clients: Scrapely for standard synchronous scripts, and AsyncScrapely for high-performance, concurrent applications using asyncio.

Synchronous (Sync)
Asynchronous (Async)

from scrapely import Scrapely

# Initialize the client
client = Scrapely(api_key="YOUR_API_KEY")

# 1. Crawl a website
response = client.crawler.crawl(
    website_url="https://example.com",
    return_page_text=True
)
print(response.result.text)

# 2. Solve reCAPTCHA V3
captcha = client.google.RecaptchaV3(
    website_url="https://example.com",
    website_key="6LdKlZEpAAAAAAOQjzC2v_d36tWxCl6dWsozdSy9"
)
print(captcha.result.solution)

import asyncio
from scrapely import AsyncScrapely

async def main():
    # Initialize the async client
    client = AsyncScrapely(api_key="YOUR_API_KEY")

    # 1. Crawl a website asynchronously
    response = await client.crawler.crawl(
        website_url="https://example.com",
        screenshot=True
    )
    print(f"Screenshot URL: {response.result.screenshot}")

    # 2. Solve Cloudflare Turnstile asynchronously
    captcha = await client.cloudflare.Turnstile(
        website_url="https://example.com",
        website_key="0x4AAAAAA..."
    )
    print(f"Token: {captcha.result.solution}")

if __name__ == "__main__":
    asyncio.run(main())

Advanced Usage

Automation Instructions

You can pass a list of instructions to interact with the page before scraping. The SDK provides helper classes (SendKeys, Click, Wait, etc.) for type safety and auto-completion.

from scrapely import Scrapely
from scrapely.models.types import SendKeys, Click, Wait

client = Scrapely(api_key="YOUR_API_KEY")

instructions = [
    SendKeys(selector="#search", text="scraping"),
    Click(selector="#submit-button"),
    Wait(timeout=2) # Wait 2 seconds
]

response = client.crawler.crawl(
    website_url="https://example.com",
    instructions=instructions,
    return_page_text=True
)

print(response.result.text)

Typed Response Objects

All API responses are returned as strictly typed Python dataclasses for better IDE support.

CrawlerResponse: Returned by crawler tasks. Contains a result (CrawlerResult) object with fields like html, text, cookies, screenshot, and user_agent.
CaptchaResponse: Returned by CAPTCHA tasks. Contains a result (CaptchaResult) object with the solved solution string.