Scraping Pipeline

Fresh Source: docs.airtop.ai

A complete workflow for intelligent web scraping using Airtop's AI APIs: scrapeContent for raw extraction, pageQuery for targeted questions, and paginatedExtraction for multi-page data.

Architecture

Step 1: Create Session and Window

TypeScript

import Airtop from '@airtop/sdk';

const client = new Airtop({ apiKey: process.env.AIRTOP_API_KEY });

const session = await client.sessions.create({
  configuration: { timeoutMinutes: 15 },
});
const sessionId = session.data.id;

const window = await client.windows.create(sessionId, {
  url: 'https://example.com/products',
});
const windowId = window.data.windowId;

Python

from airtop import Airtop

client = Airtop(api_key="your-api-key")

session = client.sessions.create(
    configuration={"timeoutMinutes": 15}
)
session_id = session.data.id

window = client.windows.create(
    session_id, url="https://example.com/products"
)
window_id = window.data.window_id

Step 2a: scrapeContent — Full Page Extraction

Extracts the entire page as clean markdown. Works with Office365 documents, Google Docs, and complex pages.

TypeScript

const scraped = await client.windows.scrapeContent(sessionId, windowId);
console.log(scraped.data.content); // Clean markdown

Python

scraped = client.windows.scrape_content(session_id, window_id)
print(scraped.data.content)  # Clean markdown

Step 2b: pageQuery — Targeted Extraction

Ask specific questions about the page content. Supports JSON schema for structured output.

TypeScript

// Simple query
const answer = await client.windows.pageQuery(sessionId, windowId, {
  prompt: 'What are all the product names and prices on this page?',
});
console.log(answer.data.modelResponse);

// Structured output with JSON schema
const structured = await client.windows.pageQuery(sessionId, windowId, {
  prompt: 'Extract all products with name, price, and rating',
  configuration: {
    outputSchema: {
      type: 'object',
      properties: {
        products: {
          type: 'array',
          items: {
            type: 'object',
            properties: {
              name: { type: 'string' },
              price: { type: 'number' },
              rating: { type: 'number' },
            },
          },
        },
      },
    },
  },
});
const products = JSON.parse(structured.data.modelResponse);

Python

# Simple query
answer = client.windows.page_query(
    session_id, window_id,
    prompt="What are all the product names and prices on this page?"
)
print(answer.data.model_response)

# Structured output with JSON schema
structured = client.windows.page_query(
    session_id, window_id,
    prompt="Extract all products with name, price, and rating",
    configuration={
        "outputSchema": {
            "type": "object",
            "properties": {
                "products": {
                    "type": "array",
                    "items": {
                        "type": "object",
                        "properties": {
                            "name": {"type": "string"},
                            "price": {"type": "number"},
                            "rating": {"type": "number"}
                        }
                    }
                }
            }
        }
    }
)
import json
products = json.loads(structured.data.model_response)

Step 2c: paginatedExtraction — Multi-Page Scraping

Automatically navigates through paginated content, collecting data across multiple pages.

TypeScript

const paginated = await client.windows.paginatedExtraction(
  sessionId,
  windowId,
  {
    config: {
      outputSchema: {
        type: 'object',
        properties: {
          items: {
            type: 'array',
            items: {
              type: 'object',
              properties: {
                title: { type: 'string' },
                url: { type: 'string' },
              },
            },
          },
        },
      },
    },
  }
);
console.log(paginated.data);

Python

paginated = client.windows.paginated_extraction(
    session_id, window_id,
    config={
        "outputSchema": {
            "type": "object",
            "properties": {
                "items": {
                    "type": "array",
                    "items": {
                        "type": "object",
                        "properties": {
                            "title": {"type": "string"},
                            "url": {"type": "string"}
                        }
                    }
                }
            }
        }
    }
)
print(paginated.data)

Step 3: Terminate

TypeScript

await client.sessions.terminate(sessionId);

Python

client.sessions.terminate(session_id)

When to Use Which

Method	Use Case	Output
scrapeContent	Full page dump, document extraction	Markdown text
pageQuery	Specific questions, structured extraction	LLM response or JSON
paginatedExtraction	Multi-page listings, search results	Aggregated structured data