Batch Processing

Fresh Source: docs.airtop.ai

Process multiple URLs in parallel using Airtop's batch operations. Control concurrency with maxConcurrentSessions and maxWindowsPerSession.

Architecture

Basic Batch Operation

TypeScript

import Airtop from '@airtop/sdk';

const client = new Airtop({ apiKey: process.env.AIRTOP_API_KEY });

const urls = [
  'https://example.com/page1',
  'https://example.com/page2',
  'https://example.com/page3',
  'https://example.com/page4',
  'https://example.com/page5',
];

const results = await client.windows.batchOperate({
  urls,
  maxConcurrentSessions: 3,
  maxWindowsPerSession: 2,
  operation: async (windowContext) => {
    const { sessionId, windowId } = windowContext;
    const scraped = await client.windows.scrapeContent(sessionId, windowId);
    return scraped.data.content;
  },
});

for (const result of results) {
  console.log(`URL: ${result.url}`);
  console.log(`Status: ${result.status}`);
  console.log(`Content length: ${result.data?.length || 0}`);
}

Python

from airtop import Airtop

client = Airtop(api_key="your-api-key")

urls = [
    "https://example.com/page1",
    "https://example.com/page2",
    "https://example.com/page3",
    "https://example.com/page4",
    "https://example.com/page5",
]

results = client.windows.batch_operate(
    urls=urls,
    max_concurrent_sessions=3,
    max_windows_per_session=2,
    operation=lambda ctx: client.windows.scrape_content(
        ctx.session_id, ctx.window_id
    ).data.content
)

for result in results:
    print(f"URL: {result.url}")
    print(f"Status: {result.status}")

Configuration Options

Option	Type	Description
urls	string[]	List of URLs to process
maxConcurrentSessions	number	Max parallel sessions (controls resource usage)
maxWindowsPerSession	number	Max tabs per session (reduces session overhead)
operation	function	Async function to run on each window

Dynamic URL Discovery

Add new URLs during batch processing as you discover them.

TypeScript

const results = await client.windows.batchOperate({
  urls: ['https://example.com/sitemap.xml'],
  maxConcurrentSessions: 5,
  maxWindowsPerSession: 3,
  operation: async (windowContext, addUrls) => {
    const { sessionId, windowId, url } = windowContext;

    if (url.includes('sitemap')) {
      // Extract URLs from sitemap and add them to the queue
      const links = await client.windows.pageQuery(sessionId, windowId, {
        prompt: 'List all URLs found on this page',
      });
      const newUrls = links.data.modelResponse.split('\n').filter(Boolean);
      addUrls(newUrls);
      return { type: 'sitemap', count: newUrls.length };
    }

    // Process regular pages
    const content = await client.windows.scrapeContent(sessionId, windowId);
    return content.data.content;
  },
});

Python

def process(ctx, add_urls):
    if "sitemap" in ctx.url:
        links = client.windows.page_query(
            ctx.session_id, ctx.window_id,
            prompt="List all URLs found on this page"
        )
        new_urls = links.data.model_response.strip().split("\n")
        add_urls(new_urls)
        return {"type": "sitemap", "count": len(new_urls)}

    content = client.windows.scrape_content(ctx.session_id, ctx.window_id)
    return content.data.content

results = client.windows.batch_operate(
    urls=["https://example.com/sitemap.xml"],
    max_concurrent_sessions=5,
    max_windows_per_session=3,
    operation=process
)

Batch with Structured Extraction

TypeScript

const results = await client.windows.batchOperate({
  urls: productUrls,
  maxConcurrentSessions: 5,
  maxWindowsPerSession: 2,
  operation: async (ctx) => {
    const data = await client.windows.pageQuery(ctx.sessionId, ctx.windowId, {
      prompt: 'Extract product name, price, availability, and rating',
      configuration: {
        outputSchema: {
          type: 'object',
          properties: {
            name: { type: 'string' },
            price: { type: 'number' },
            available: { type: 'boolean' },
            rating: { type: 'number' },
          },
        },
      },
    });
    return JSON.parse(data.data.modelResponse);
  },
});

Python

results = client.windows.batch_operate(
    urls=product_urls,
    max_concurrent_sessions=5,
    max_windows_per_session=2,
    operation=lambda ctx: json.loads(
        client.windows.page_query(
            ctx.session_id, ctx.window_id,
            prompt="Extract product name, price, availability, and rating",
            configuration={"outputSchema": {
                "type": "object",
                "properties": {
                    "name": {"type": "string"},
                    "price": {"type": "number"},
                    "available": {"type": "boolean"},
                    "rating": {"type": "number"}
                }
            }}
        ).data.model_response
    )
)

Cost Considerations

• Each session incurs per-30-second billing
• More concurrent sessions = faster but more expensive
• More windows per session = efficient (fewer sessions needed)
• Balance speed vs cost with maxConcurrentSessions

Best Practices

1. Start small — test with 2-3 URLs before scaling up
2. Balance concurrency — 3-5 concurrent sessions is usually optimal
3. Use multiple windows per session — reduces total session cost
4. Handle errors — individual URL failures shouldn't crash the batch
5. Monitor credits — batch operations can consume credits quickly