Training Hub › Data Types & Ingestion

Scraping & Remote Content

Data Engineering & Processing beginner Building with AI 12 min read

Overview

AI systems are only as useful as the data they can access. In most real-world applications, the knowledge an LLM needs doesn’t live in a single database — it’s scattered across websites, APIs, internal wikis, cloud storage, CMS platforms, and third-party services. Data ingestion is the Extract phase of ETL (Extract, Transform, Load), and it’s where your AI pipeline begins.

Scraping and remote content ingestion encompass the techniques and tools for programmatically collecting raw data from external and internal sources. This includes HTML scraping from websites, consuming REST and GraphQL APIs, pulling documents from cloud storage (S3, GCS, SharePoint), fetching content from CMS platforms, and crawling internal knowledge bases. The goal is to bring all relevant information into your pipeline where it can be parsed, chunked, embedded, and made searchable.

For AI engineers, mastering ingestion is critical because the downstream quality of your RAG system, knowledge base, or fine-tuning dataset is entirely dependent on the completeness and cleanliness of what you extract. A sophisticated retrieval pipeline built on incomplete or poorly extracted data will always underperform.

How It Works

The ETL Pipeline for AI

The data ingestion process for AI systems follows the classic ETL pattern but with AI-specific considerations at each stage:

ETL for AI

Extract: Fetch raw content from sources (web pages, APIs, file storage). Transform: Parse, clean, normalize, chunk, and embed the content. Load: Store processed content in vector databases, document stores, or search indices for retrieval.

┌─────────────────────────────────────────────────────┐
│                   DATA SOURCES                       │
├──────────┬──────────┬───────────┬───────────────────┤
│ Websites │   APIs   │  Cloud    │ Internal Systems  │
│ (HTML)   │(REST/GQL)│(S3/GCS)  │ (Wiki/CMS/Email)  │
└────┬─────┴────┬─────┴─────┬─────┴─────────┬────────┘
     │          │           │               │
     ▼          ▼           ▼               ▼
┌─────────────────────────────────────────────────────┐
│              EXTRACT (Ingestion Layer)                │
│  Scrapers, API clients, SDK calls, file downloaders  │
└──────────────────────┬──────────────────────────────┘
                       ▼
┌─────────────────────────────────────────────────────┐
│              TRANSFORM (Processing Layer)            │
│  Parse HTML/PDF → Clean → Normalize → Chunk → Embed │
└──────────────────────┬──────────────────────────────┘
                       ▼
┌─────────────────────────────────────────────────────┐
│              LOAD (Storage Layer)                     │
│  Vector DB, Document Store, Search Index             │
└─────────────────────────────────────────────────────┘

Web Scraping Techniques

Web scraping extracts content from HTML pages. The approach depends on how the target site renders its content:

Static HTML scraping — For server-rendered pages where content is in the initial HTML response. Use HTTP clients to fetch the page and HTML parsers to extract content.

import requests
from bs4 import BeautifulSoup

def scrape_property_page(url):
    """Extract property details from a static HTML page."""
    response = requests.get(url, headers={
        "User-Agent": "PropertyBot/1.0 (data-team@company.com)"
    })
    response.raise_for_status()

    soup = BeautifulSoup(response.text, "html.parser")

    # Extract structured elements
    title = soup.find("h1", class_="property-name").get_text(strip=True)
    description = soup.find("div", class_="description").get_text(strip=True)
    amenities = [
        li.get_text(strip=True)
        for li in soup.select("ul.amenities li")
    ]

    return {
        "title": title,
        "description": description,
        "amenities": amenities,
        "source_url": url,
        "scraped_at": datetime.utcnow().isoformat()
    }

Dynamic content scraping — For JavaScript-rendered pages (React, Vue, Angular SPAs) where content loads after the initial HTML. Requires a headless browser to execute JavaScript before extracting content.

from playwright.sync_api import sync_playwright

def scrape_dynamic_page(url):
    """Scrape a JavaScript-rendered page using Playwright."""
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()
        page.goto(url, wait_until="networkidle")

        # Wait for specific content to render
        page.wait_for_selector(".property-details", timeout=10000)

        # Extract rendered content
        content = page.query_selector(".property-details").inner_text()
        images = [
            img.get_attribute("src")
            for img in page.query_selector_all(".gallery img")
        ]

        browser.close()
        return {"content": content, "images": images}

API Ingestion

APIs are the preferred data source when available — they provide structured, versioned, and rate-limited access to data. Common patterns include:

import httpx
import asyncio

async def ingest_from_api(base_url, api_key, endpoint="/properties"):
    """Paginated API ingestion with rate limiting."""
    headers = {"Authorization": f"Bearer {api_key}"}
    all_records = []
    page = 1

    async with httpx.AsyncClient() as client:
        while True:
            response = await client.get(
                f"{base_url}{endpoint}",
                headers=headers,
                params={"page": page, "per_page": 100}
            )
            response.raise_for_status()
            data = response.json()

            records = data.get("results", [])
            if not records:
                break

            all_records.extend(records)
            page += 1

            # Respect rate limits
            await asyncio.sleep(0.5)

    return all_records

Cloud and Internal Sources

Enterprise AI systems frequently need to ingest from internal infrastructure:

Scraping Tool Selection Guide

Choose the right tool based on your source characteristics:

Practical Applications

Building a Knowledge Base Pipeline

A typical AI knowledge base ingestion pipeline handles multiple source types simultaneously. Consider a hotel intelligence system that needs to aggregate:

  1. Property websites: Scrape descriptions, amenity lists, and image galleries from hotel websites.
  2. Review platforms: Ingest guest reviews via API (TripAdvisor, Google Places) for sentiment and topic extraction.
  3. Internal documents: Pull operational manuals, SOPs, and training materials from SharePoint or Google Drive.
  4. CRM data: Extract structured property records (room counts, pricing, contacts) from the CRM API.

Incremental vs Full Ingestion

Production systems rarely re-ingest everything from scratch. Two patterns dominate:

# Incremental ingestion with change detection
def ingest_incremental(source, last_run_timestamp):
    """Only fetch documents modified since last ingestion run."""
    new_docs = source.list_modified_since(last_run_timestamp)

    for doc in new_docs:
        content = source.fetch(doc.id)
        parsed = parse_document(content)
        chunks = chunk_text(parsed)
        embeddings = embed_chunks(chunks)

        # Upsert: update if exists, insert if new
        vector_db.upsert(
            ids=[f"{doc.id}_chunk_{i}" for i in range(len(chunks))],
            embeddings=embeddings,
            metadatas=[{"source_id": doc.id, "updated_at": doc.modified_at}
                       for _ in chunks]
        )

    return {"processed": len(new_docs), "timestamp": datetime.utcnow()}

Responsible Scraping Practices

Web scraping carries legal and ethical obligations that AI engineers must respect:

Orchestrating Multi-Source Ingestion

Production ingestion pipelines use workflow orchestrators to manage scheduling, retries, and dependencies:

Key Takeaways

Further Reading