Every AI system is only as good as the data it can access. Before you can build embeddings, run retrieval pipelines, or fine-tune models, you need to understand the fundamental nature of your data sources. The distinction between structured and unstructured data is the first architectural decision in any AI data pipeline — it determines your storage layer, your retrieval strategy, and ultimately the quality of your model’s outputs.
Approximately 80% of enterprise data is unstructured — emails, PDFs, Slack messages, wikis, support tickets, images, and video. The remaining 20% lives in databases, spreadsheets, and APIs with well-defined schemas. AI engineering requires building pipelines that can ingest, process, and reason over both types, often simultaneously.
For AI engineers at Visiting Media, this distinction is especially relevant: property data may live in structured CRM databases (room counts, pricing, amenities as typed fields), while the rich descriptive content — property descriptions, guest reviews, marketing copy, tour scripts — is deeply unstructured. Building effective AI features means bridging both worlds.
Structured data conforms to a predefined schema with explicit types, constraints, and relationships. Each data point has a known shape before it arrives.
Structured data is information organized into a fixed format — rows and columns in relational databases, key-value pairs in JSON with known schemas, or typed fields in APIs. It is queryable via SQL or structured query languages and can be validated against a schema at write time.
Common forms of structured data in AI systems:
-- Structured data: precise SQL query returns exact results
SELECT property_name, room_count, avg_nightly_rate
FROM properties
WHERE city = 'New York'
AND star_rating >= 4
ORDER BY avg_nightly_rate DESC
LIMIT 10;
-- Result: deterministic, typed, complete
-- | property_name | room_count | avg_nightly_rate |
-- | The Plaza | 282 | 895.00 |
-- | Four Seasons | 368 | 825.00 |Unstructured data has no predefined schema. Its meaning is encoded in natural language, pixel patterns, audio waveforms, or other formats that require interpretation rather than simple lookup.
Unstructured data is information that does not conform to a tabular or typed schema — free-form text, images, audio, video, PDFs, and HTML. Extracting meaning requires parsing, embedding, or machine learning rather than SQL queries.
Common forms of unstructured data in AI systems:
Semi-structured data has some organizational markers but doesn’t fit neatly into tables. Examples include:
The retrieval and reasoning strategies differ fundamentally based on data type:
┌─────────────────────────────────────────────────────────┐
│ AI Data Pipeline Architecture │
├──────────────────────┬──────────────────────────────────┤
│ STRUCTURED PATH │ UNSTRUCTURED PATH │
│ │ │
│ SQL Database │ Raw Documents (PDF, HTML, etc.) │
│ ↓ │ ↓ │
│ Text-to-SQL │ Document Parsing │
│ (NL → SQL query) │ (Extract text, preserve layout) │
│ ↓ │ ↓ │
│ Query Execution │ Chunking │
│ (exact results) │ (Split into passages) │
│ ↓ │ ↓ │
│ Result Formatting │ Embedding │
│ (table → text) │ (Text → vector) │
│ ↓ │ ↓ │
│ ┌───────────────────┤ Vector Search │
│ │ LLM Context │ (Similarity retrieval) │
│ │ Window │ ↓ │
│ │ ├──────→│ │
│ └───────────────────┴───────┘ │
└─────────────────────────────────────────────────────────┘Structured data path: A user asks “What are the top 5 most expensive properties in New York?” The system translates this to SQL, executes the query, and gets exact results. No embedding or vector search needed. The challenge is accurate text-to-SQL translation, especially with complex schemas.
Unstructured data path: A user asks “Which properties have the best rooftop dining experience?” There’s no database column for “rooftop dining experience.” The system must search embedded property descriptions, reviews, and marketing content using semantic similarity. The challenge is retrieval quality and preventing hallucination.
The type of data you’re working with determines your entire technology stack:
Unstructured data projects typically require 2-3x more development time than structured data projects. The additional complexity comes from:
The most effective production AI systems combine both data types. For example, a property recommendation system might:
# Hybrid retrieval example
structured_results = db.query("""
SELECT name, city, star_rating, price
FROM properties WHERE city = %s AND price <= %s
""", [city, max_price])
# Embed the user's qualitative query
query_embedding = embed("rooftop dining with city views")
# Search only among the filtered property IDs
unstructured_context = vector_db.search(
embedding=query_embedding,
filter={"property_id": {"$in": [r.id for r in structured_results]}},
top_k=5
)
# Combine both in the LLM prompt
prompt = f"""Based on these properties and their descriptions,
recommend the best option for rooftop dining:
Property Data: {format_table(structured_results)}
Relevant Descriptions: {format_chunks(unstructured_context)}
"""