Projects

Guided tour

Marketing Warehouse. Ten sources in one place a model can read.

A BigQuery warehouse that pulls ten marketing sources into one place every morning and joins ad spend through to booked patients, with a catalog an LLM reads and a weekly graded eval.

Data platform · LLM-ready analytics 6 steps · synthetic data

Walk through it, six steps

Step 1 of 6

Ten sources, one morning sync

Cloud Scheduler fires the connector jobs at 6 AM: ads, calls, CRM, web, search and social, with one platform arriving via BigQuery Data Transfer. All connectors share retry, load, and secret code, packaged as a single container image run as serverless jobs.

Nightly runs (excerpt)Synthetic data
SourceRowsStatusFinished
Search ads (Platform A)1,204ok06:04
Social ads (Platform B)912ok06:06
Call tracking388ok06:07
CRM boards2,730ok06:11
Search ads (transfer)·managed transfer06:15
Step 1 of 6

Every name, transcript and document in this tour is synthetic, and so are the numbers on the sample screens. The numbers under The problem and Where it landed are real. The production system and its data stay private.

The problem.

Marketing questions like which channel actually produces patients required manual exports from each platform and stitched spreadsheets. And once an LLM was allowed to answer questions over the data, a harder problem appeared: how do you know its answers stay right as the schema evolves?

  • Sources unified 10 ads, calls, CRM, web, search and social
  • Funnel linkage 98.5% patients joined to leads; 99.3% agreement with the weekly exports
  • LLM evals 22 golden Qs weekly graded run, fails under 80%
  • Fresh by 6:45 AM freshness and drift check every morning

How it fits together.

4 layers, left to right
  1. Ingest

    • 10 sync jobs, one Docker image, Cloud Run jobs
    • Shared retry, load and secret modules
    • 6 AM schedule + managed transfer
  2. Warehouse

    • BigQuery: raw → staging → marts (+ evals dataset)
    • Inline descriptions on every view and column
    • data_guide catalog written for LLM readers
  3. Consume

    • LLM access via MCP Toolbox
    • Looker Studio dashboards
    • Plain-English answers with mart citations
  4. Trust

    • 6:45 freshness check + z-score drift sentinel
    • 22 golden questions graded weekly, 80% gate
    • Glass monitoring app with per-node health

Where it landed.

Marketing spend, calls, leads and booked patients live in one queryable place: 98.5% of patients link to a lead, with 99.3% agreement against the weekly exports. The part worth stealing: treating the LLM's access as a product surface with its own weekly, graded, gated eval suite instead of assuming yesterday's good answers persist. All dollar figures shown are synthetic.

Concepts

Dimensional modelingLLM-over-SQLGolden-question evalsDrift detectionSelf-documenting schemasFreshness monitoring

Stack

PythonBigQueryCloud Run JobsCloud SchedulerMCP ToolboxLooker StudioDocker