Data Architecture

Implementing Data Mesh at Scale: Architecture and Governance

28 April 202515 min readBy Bayseian Engineering

Practical guide to transitioning from monolithic data warehouses to a decentralized data mesh architecture with domain-driven ownership.

Introduction: From Data Lake to Data Mesh

Traditional centralized data platforms (data lakes, data warehouses) create bottlenecks as organizations scale. A single data team becomes responsible for ingesting, transforming, and serving data for the entire company, which leads to slow delivery, poor data quality, and frustrated business teams.

Data Mesh is a paradigm shift that treats data as a product, owned by domain teams who understand it best. Instead of centralizing data in one platform, Data Mesh distributes ownership while providing shared infrastructure and governance.

Core Principles:

1. Domain Ownership: Business domains own their data as products
2. Data as a Product: Treat data with product thinking (quality, discoverability, SLAs)
3. Self-Serve Infrastructure: Platform team provides tools, domain teams build products
4. Federated Governance: Automated policies, not manual reviews

•Scalability: Distribute ownership across teams so growth in data volume or domains doesn't funnel through one central bottleneck.
•Agility: Domains move independently. A domain team can ship a new data product without waiting on a shared backlog.
•Quality: Owners closest to the data ensure accuracy. The team that generates the data understands its quirks better than a central team ever will.
•Innovation: Self-service reduces dependencies, so domain teams can experiment without filing a ticket and waiting weeks.

•100+ data engineers: roughly the headcount at which a centralized data team becomes the bottleneck everyone is waiting on.
•50+ data sources: past this point, a single team can't maintain context on every source well enough to serve every consumer.
•Multiple business domains with unique needs: when domains have genuinely different schemas, SLAs, and quality bars, forcing them into one platform creates friction for everyone.
•Central data team is a bottleneck: if requests queue for weeks regardless of headcount, the problem is architectural, not staffing.

•<20 engineers: the governance and platform overhead of a mesh costs more than the coordination problem it solves at this scale.
•Single domain business: there's no real domain boundary to distribute ownership across, so a mesh just adds process without a payoff.
•Simple reporting needs: a data warehouse and a BI tool solve this faster and cheaper than standing up domain-owned data products.
•No organizational buy-in: data mesh requires domain teams to accept new ownership responsibilities. Without executive sponsorship, that ownership never actually transfers.

Architecture Overview

Each domain owns ingestion, quality, catalog publishing, and SLAs; the platform team owns only the shared rails.

Implementation: Building a Data Product

Step-by-Step: Creating a Data Product

•Example: "Customer 360", a unified view of customer data
•Consumers: Marketing, Sales, Support teams
•SLA: Daily refresh by 8am
•Quality: 99% completeness, <1% duplicates

•SQL table in data warehouse: the default for analysts and BI tools, and the lowest friction for the most common consumer.
•REST API for real-time lookups: for applications that need a single record on demand, not a bulk query.
•Event stream for downstream processing: for consumers who need to react to changes as they happen, not poll for them.
•Pre-computed metrics/aggregations: expensive computations done once by the producer instead of recomputed by every consumer.

•Extract from operational databases (CDC): change data capture keeps the data product fresh without hammering production databases with batch queries.
•Transform with dbt (tested, documented): transformations live as version-controlled, testable code rather than opaque stored procedures.
•Publish to Snowflake/BigQuery: land the result in the same warehouse most consumers already query, so there's no new tool to adopt.
•Register in data catalog: an undiscoverable data product might as well not exist. The catalog entry is what lets other domains find and trust it.

•Schema validation (column types, nullability): catches upstream schema drift before it silently breaks every downstream consumer.
•Data quality rules (freshness, completeness, accuracy): encodes the SLA as an automated test instead of a promise nobody checks.
•Anomaly detection (unexpected distributions): catches the failures schema validation can't, like a metric suddenly dropping to zero.
•Lineage tracking (upstream dependencies): when a source table changes, lineage tells you exactly which data products and consumers are affected.

•SLA monitoring (freshness, availability): proves the SLA is being met instead of assuming it, and pages someone before a consumer notices stale data.
•Usage analytics (who's using it, how often): tells the owning team whether the product is actually valuable or safe to deprecate.
•Alerting on quality failures: quality checks are only useful if a failure reaches a human before it reaches a dashboard.
•Support channel for consumers: every data product needs an obvious place for consumers to ask questions or report problems.

•Clear ownership (team + point of contact). Every product needs a name attached, not "the data team."
•Versioned schema, so consumers can upgrade deliberately instead of breaking on a silent change.
•Quality SLAs defined and monitored. A target with no monitoring isn't an SLA, it's a hope.
•Documentation (README, examples): the difference between a discoverable product and one nobody dares touch.
•Discoverable in catalog. If it's not in the catalog, no one outside the owning team will find it.
•Access controls (who can read/write), especially critical once the data product includes any PII.
•Lineage tracked, required to answer "what breaks if I change this upstream table."
•Usage monitored, the basis for deprecation decisions and for proving the product's value.
•Support process defined, so consumers know who to page when the data looks wrong.

SQL

# Data Product Example: Customer 360
# Implemented using dbt (data build tool)

# models/customer_360/schema.yml
version: 2

models:
  - name: customer_360
    description: >
      Unified customer view combining profile, activity, and support data.
      Owner: customer-analytics@company.com
      SLA: Daily refresh by 8am UTC
      Quality: 99% completeness, <1% duplicates
    
    meta:
      owner: customer-analytics@company.com
      domain: customer
      sla_freshness_hours: 24
      quality_score_target: 0.99
      
    columns:
      - name: customer_id
        description: Unique customer identifier
        tests:
          - unique
          - not_null
      
      - name: email
        description: Customer email address
        tests:
          - unique
          - not_null
      
      - name: first_order_date
        description: Date of customer's first order
        tests:
          - not_null
          - dbt_expectations.expect_column_values_to_be_of_type:
              column_type: date
      
      - name: total_revenue
        description: Lifetime customer revenue
        tests:
          - not_null
          - dbt_expectations.expect_column_values_to_be_between:
              min_value: 0
              max_value: 1000000
      
      - name: last_activity_date
        description: Most recent customer activity
        tests:
          - not_null
          - dbt_expectations.expect_column_values_to_be_between:
              min_value: "2020-01-01"
              max_value: "{{ var('current_date') }}"

# models/customer_360/customer_360.sql
{{
  config(
    materialized='incremental',
    unique_key='customer_id',
    on_schema_change='fail',
    tags=['customer', 'core', 'pii'],
    meta={
      'owner': 'customer-analytics',
      'domain': 'customer'
    }
  )
}}

WITH customer_profile AS (
  SELECT
    customer_id,
    email,
    first_name,
    last_name,
    signup_date,
    country,
    segment
  FROM {{ ref('stg_customers') }}
  {% if is_incremental() %}
  WHERE updated_at >= (SELECT MAX(updated_at) FROM {{ this }})
  {% endif %}
),

order_metrics AS (
  SELECT
    customer_id,
    MIN(order_date) as first_order_date,
    MAX(order_date) as last_order_date,
    COUNT(*) as total_orders,
    SUM(order_amount) as total_revenue,
    AVG(order_amount) as avg_order_value
  FROM {{ ref('stg_orders') }}
  GROUP BY customer_id
),

activity_metrics AS (
  SELECT
    customer_id,
    MAX(event_timestamp) as last_activity_date,
    COUNT(*) as total_events,
    COUNT(DISTINCT DATE(event_timestamp)) as active_days
  FROM {{ ref('stg_events') }}
  WHERE event_timestamp >= DATEADD('day', -90, CURRENT_DATE())
  GROUP BY customer_id
),

support_metrics AS (
  SELECT
    customer_id,
    COUNT(*) as total_tickets,
    AVG(resolution_time_hours) as avg_resolution_time,
    SUM(CASE WHEN status = 'closed' THEN 1 ELSE 0 END) as resolved_tickets
  FROM {{ ref('stg_support_tickets') }}
  GROUP BY customer_id
),

final AS (
  SELECT
    p.customer_id,
    p.email,
    p.first_name,
    p.last_name,
    p.signup_date,
    p.country,
    p.segment,
    
    -- Order metrics
    COALESCE(o.first_order_date, NULL) as first_order_date,
    COALESCE(o.last_order_date, NULL) as last_order_date,
    COALESCE(o.total_orders, 0) as total_orders,
    COALESCE(o.total_revenue, 0) as total_revenue,
    COALESCE(o.avg_order_value, 0) as avg_order_value,
    
    -- Activity metrics
    COALESCE(a.last_activity_date, NULL) as last_activity_date,
    COALESCE(a.total_events, 0) as total_events_90d,
    COALESCE(a.active_days, 0) as active_days_90d,
    
    -- Support metrics
    COALESCE(s.total_tickets, 0) as total_support_tickets,
    COALESCE(s.avg_resolution_time, 0) as avg_ticket_resolution_hours,
    
    -- Computed fields
    CASE 
      WHEN o.total_orders >= 10 THEN 'champion'
      WHEN o.total_orders >= 5 THEN 'loyal'
      WHEN o.total_orders >= 2 THEN 'returning'
      WHEN o.total_orders = 1 THEN 'new'
      ELSE 'prospect'
    END as customer_lifecycle_stage,
    
    DATEDIFF('day', COALESCE(a.last_activity_date, p.signup_date), CURRENT_DATE()) as days_since_last_activity,
    
    -- Metadata
    CURRENT_TIMESTAMP() as updated_at,
    '{{ run_started_at }}' as pipeline_run_timestamp
  
  FROM customer_profile p
  LEFT JOIN order_metrics o ON p.customer_id = o.customer_id
  LEFT JOIN activity_metrics a ON p.customer_id = a.customer_id
  LEFT JOIN support_metrics s ON p.customer_id = s.customer_id
)

SELECT * FROM final

# Quality check SQL (runs after build)
# tests/customer_360/test_completeness.sql
SELECT
  COUNT(*) as total_customers,
  COUNT(CASE WHEN email IS NULL THEN 1 END) as missing_email,
  COUNT(CASE WHEN first_name IS NULL THEN 1 END) as missing_name,
  ROUND(100.0 * COUNT(CASE WHEN email IS NOT NULL THEN 1 END) / COUNT(*), 2) as completeness_pct
FROM {{ ref('customer_360') }}
HAVING completeness_pct < 99.0  -- Fail if <99% complete

# Freshness check (SLA monitoring)
# macros/check_sla.sql
{% macro check_data_freshness(model_name, max_hours) %}
  SELECT
    '{{ model_name }}' as data_product,
    MAX(updated_at) as last_update,
    DATEDIFF('hour', MAX(updated_at), CURRENT_TIMESTAMP()) as hours_since_update,
    {{ max_hours }} as sla_hours
  FROM {{ ref(model_name) }}
  HAVING hours_since_update > {{ max_hours }}
{% endmacro %}

# Run this query to check SLAs
SELECT * FROM {{ check_data_freshness('customer_360', 24) }}

Governance and Standards

Federated Governance: Automate policies, don't centralize decisions

1.Schema Evolution: Backwards compatibility required
2.Data Quality: Automated tests on every build
3.Security: Column-level access controls
4.Privacy: PII auto-classification and masking
5.Lineage: Automatic dependency tracking

Global Standards (Platform enforced):

•Tables: domain_entity_grain (e.g., sales_orders_daily)
•Columns: snake_case, no abbreviations
•Metrics: metric_name_period (e.g., revenue_monthly)

•Bronze: Raw, as-is from source (no guarantees)
•Silver: Cleaned, validated, deduplicated
•Gold: Business logic applied, aggregated, production-ready

•Tier 1: <1 hour freshness, 99.9% availability (critical dashboards)
•Tier 2: <6 hour freshness, 99% availability (reporting)
•Tier 3: Daily, best-effort (exploratory analysis)

•Clear description (what, why, how)
•Owner contact information
•Sample queries / usage examples
•Schema with column descriptions
•SLA and quality metrics
•Lineage (upstream dependencies)
•Access request process

•Provide self-service infrastructure: the tooling (Airflow, dbt, catalog) that lets domain teams ship without filing a platform ticket.
•Enforce global policies (automated): schema, security, and naming standards get checked in CI, not in a manual review meeting.
•Build shared components (auth, monitoring, catalog): the pieces that would otherwise be reimplemented, inconsistently, by every domain team.
•Training and enablement: domain teams can't own data products well if they were never taught the platform's conventions.
•NOT responsible for domain-specific data products: this is the boundary that makes the mesh work. If the platform team starts owning domain pipelines, you're back to a centralized bottleneck.

Migration Strategy

Migrating from Centralized to Data Mesh:

•Set up self-serve infrastructure (Airflow, dbt, observability)
•Implement data catalog (Datahub, Atlan)
•Define governance standards
•Train first domain team

•Select high-value, well-defined domain
•Migrate 2-3 data products
•Establish patterns and best practices
•Document learnings

•Onboard 1-2 domains per quarter
•Refine platform based on feedback
•Build community of practice
•Migrate legacy pipelines gradually

•All domains managing own data products
•Central team focuses on platform
•Continuous improvement
•Federated governance operational

•Time to create new data product (target: <2 weeks)
•Data product quality score (target: >95%)
•Consumer satisfaction (target: NPS >50)
•Platform uptime (target: 99.9%)
•Domain team autonomy (% of work self-served)

Common Challenges:

•Solution: Start with volunteer domains, demonstrate wins
•Show improved velocity and quality
•Executive sponsorship critical

•Solution: Training programs, pair programming
•Hire embedded data engineers per domain
•Provide templates and examples

•Solution: Start simple, add features incrementally
•Prioritize self-service over features
•Comprehensive documentation

•Solution: Automate policies, don't rely on manual reviews
•Shift left: Catch issues in CI/CD
•Clear escalation paths

## Conclusion Data Mesh moves from centralized data platforms to distributed ownership, treating data as a product owned by domain teams. It lets organizations scale data capabilities without creating bottlenecks, while maintaining quality and governance through automated policies. The core principles driving Data Mesh success: - Domain ownership empowers teams closest to the data to ensure accuracy and relevance - Product thinking applied to data creates discoverable, high-quality, well-documented data assets - Self-serve infrastructure enables domain teams to move independently without dependencies on central teams - Federated governance automates policy enforcement rather than relying on manual reviews The transition from centralized to decentralized data architecture is not trivial. It requires organizational change, platform investment, and new skills. For organizations with 100+ data engineers and multiple business domains, though, Data Mesh can improve time-to-insight, data quality, and team velocity. At Bayseian, we've helped enterprises implement Data Mesh, migrating from monolithic data lakes to distributed data products while maintaining governance and quality. Our approach emphasizes incremental adoption, starting with pilot domains and scaling based on proven patterns. The key is balancing decentralization (domain ownership) with standardization (platform and governance). Domain teams gain autonomy while the platform team provides guardrails through automated policy enforcement, shared infrastructure, and best practice templates. Ready to explore Data Mesh for your organization? Contact us at contact@bayseian.com to discuss whether Data Mesh is right for you and how to implement it successfully.

Data MeshArchitectureData EngineeringGovernance

Working on something like this?

No pitch, just a practical conversation with the team that builds and operates these systems in production.

Start a conversation