Data Mesh Governance / Policies / Isolation / Project Structure

One project per data product

Category: Interoperability
Platform: BigQuery

Context

For consistency, we want a uniform structure and naming of our BigQuery projects.

The structure must fit to BigQuery’s strict 3-level-hierarchy:

BigQuery has some naming restrictions: Project IDs must be 6-30 characters, contain letters, numbers, and hyphens and are globally unique, cannot be in use or have previously been used. Datasets and table names can contain up to 1024 characters, numbers and underscores.

Decision

We agree on a set of conventions for our BigQuery projects, datasets, and tables:

Project ID

Format:

<orgname>[-<env>]-dp-<domain>-<dataproduct>

Elements:

Examples:

Datasets

If applicable, more datasets can be defined by adding a suffix, separated by an underscore, e.g. source_googleanalytics, source_salesforce, source_kafka.

Tables and Views

Example

Consequences

Automation

The BigQuery project structure can be set up through a self-service web-app, when a new data product is created.

A dbt hook can be implemented that makes sure that all models use the defined prefixes.