Data Sources and Gathering
Knowing where numbers come from (and where they break)
Analytical data comes from transactional systems, product events, SaaS exports, files and third parties — each with its own capture mechanics, latency and quality profile.
What you will learn
- Describe the main analytical data source categories
- Explain event vs entity vs aggregate data
- Ask the right questions about coverage, latency and reliability of a source
What is this?
Analysts rely on several families of sources: transactional databases (the system of record), product event streams (what users did), SaaS tool exports (billing, ads, CRM), flat files (finance, log dumps) and third-party data. Each source has a capture mechanism, a latency profile and failure modes that define what questions it can honestly answer.
Why do you need it?
- Choosing the wrong source produces confident-but-wrong metrics (e.g., billing vs event totals).
- Knowing capture mechanics tells you what a metric actually counts.
- Latency and backfill behavior determine which dashboards can cite which numbers.
- Documenting source semantics prevents daily definition arguments.
Mental model
Reading a map made by a different surveyor
Gathering data from a new source
Every surveyor measures differently — one uses peaks, another meant river banks. Before trusting the map you must learn the surveyor's conventions.
How it works
- Transactional data is written by the app's database on every action — authoritative but operational, not event-shaped.
- Event data is emitted by SDKs on the client/server as users act — rich and granular but prone to ad-blockers, offline loss and schema drift.
- SaaS tools expose data via API or automated exports (CSV/JSON drops into your warehouse).
- Files and spreadsheets arrive manually or via FTP and need validation that they arrived whole.
- A data contract (schema, semantics, SLA) is agreed between source and consumer to keep quality honest.
Common analytical sources and their nature.
In production
- Revenue is measured from the billing system of record, not from purchase events, because events can drop.
- Product activation uses event data because 'completed onboarding' only exists as events.
- A marketing team reconciles ad-platform spends against internal paid channels to spot tracking drift.
Best practices
- Document each source's grain, definitions, latency and known quirks in one place.
- Reconcile new sources against an existing one (counts, totals) before trusting them.
- Design event tracking with a contract: schema version, identity, timestamps with timezone.
- Keep raw source data immutable forever; never transform it in place.
Common mistakes
- Mixing event-based and entity-based counts without reconciling (double counting).
- Trusting an automated export without validating row counts and freshness.
- Ignoring timezone misalignment between sources (UTC vs local).
- Replacing authority with convenience: using SaaS dashboards instead of raw data.
Debugging tips
- When two sources disagree, reconcile at the grain: same filters, same window, same day boundary.
- Check a source's last modified time before debugging joins — the source may be stale.
- Spot test IDs: pull one userId through both systems and compare its story.
- Log source version in pipelines so you can attribute metric jumps to a schema change.
Interview checkpoint
Quick quiz
Which source should be authoritative for company revenue?
Event data is lossless because every user action is automatically captured.