Smart Scraping

Fresh Source: docs.airtop.ai

The scrapeContent method extracts page content as clean markdown. It handles complex pages including Office365 documents, Google Docs, dynamically rendered content, and more.

Basic Usage

TypeScript

const result = await client.windows.scrapeContent(sessionId, windowId);
console.log(result.data.content); // Clean markdown string

Python

result = client.windows.scrape_content(session_id, window_id)
print(result.data.content)  # Clean markdown string

With Output Schema

Extract structured data using a JSON schema.

TypeScript

const result = await client.windows.scrapeContent(sessionId, windowId, {
  config: {
    outputSchema: {
      type: 'object',
      properties: {
        title: { type: 'string' },
        author: { type: 'string' },
        publishDate: { type: 'string' },
        content: { type: 'string' },
        tags: { type: 'array', items: { type: 'string' } },
      },
    },
  },
});
const article = JSON.parse(result.data.content);

Python

result = client.windows.scrape_content(session_id, window_id, config={
    "outputSchema": {
        "type": "object",
        "properties": {
            "title": {"type": "string"},
            "author": {"type": "string"},
            "publishDate": {"type": "string"},
            "content": {"type": "string"},
            "tags": {"type": "array", "items": {"type": "string"}}
        }
    }
})
import json
article = json.loads(result.data.content)

Supported Content Types

Content Type	Notes
Standard HTML pages	Clean markdown extraction
JavaScript-rendered pages	Waits for dynamic content
Office365 documents	Word, Excel, PowerPoint online
Google Docs/Sheets/Slides	Full content extraction
PDF viewers	Extracts text from embedded PDFs
SPAs (React, Vue, Angular)	Renders and extracts

scrapeContent vs pageQuery

Feature	scrapeContent	pageQuery
Purpose	Full page extraction	Specific questions
Output	Markdown (or JSON with schema)	LLM-generated answer
Best for	Archiving, indexing, full content	Targeted data points
Cost	Lower (single extraction)	Higher (LLM inference)

Best Practices

1. Use for full content — when you need everything on the page
2. Add output schema for structured extraction instead of raw markdown
3. Wait for page load — ensure content is rendered before scraping
4. Combine with pageQuery — scrape first for overview, query for specifics
5. Handle large pages — very long pages may be truncated; use paginatedExtraction for multi-page content