DATA FOUNDATIONS
From source to confident use.
A business-first guide to the decisions behind enterprise data. Start with the outcome, then choose how information should move, be governed and be used.
01SourcesSystems, events, documents
02IntegrationBatch, replication, streams
03Trusted productsQuality, context, ownership
04UseAnalytics, operations, AI
Conceptual information flow. Governance, security and monitoring apply throughout; this is not a deployed architecture.
01 / CONNECT
Choose the movement your use case needs.
A daily finance report and a live operational alert do not require the same pipeline. Batch moves grouped records at intervals. Change data capture replicates changes from a source system. Streaming processes an ongoing event flow.
Before choosing a pattern, document freshness expectations, source-system impact, failure handling, recovery and the cost of moving data. Hybrid execution may keep processing close to sensitive or difficult-to-move sources.
Questions for your brief
- Which source is authoritative for each important field?
- How fresh must the result be, and why?
- What happens when a source or connection fails?
02 / TRUST
Make responsibility explicit.
A dataset needs an accountable owner, a clear definition and quality rules tied to its intended use. Completeness, validity, uniqueness and timeliness are different checks; one aggregate number cannot prove fitness for every purpose.
Lineage records how information is produced and changed. Access rules describe who can use it. Retention and deletion expectations should be documented with the relevant business owners.
Questions for your brief
- Who approves changes to this data product?
- Which quality failures block its use?
- Can a consumer trace a field back to its source?
03 / ACTIVATE
Publish context alongside the data.
A useful data product includes purpose, owner, definitions, access conditions, freshness expectations and known limitations. Analytics teams need consistent measures; operational teams need reliability; AI applications also need permission-aware retrieval, provenance and evaluation suited to the task.
Open table formats, catalog vocabularies and lineage standards solve different interoperability problems. Apache Iceberg addresses analytic tables; W3C DCAT describes catalog resources; OpenLineage captures metadata about datasets, jobs and runs. None substitutes for the others.
Read the primary sources →Start with one important data product. Define its owner, consumers, quality rules and recovery needs before extending the pattern.
Create a planning brief →