Data Mesh Governance / Policies / Isolation / Project Structure
For consistency, we want a uniform structure and naming of our BigQuery projects.
The structure must fit to BigQuery’s strict 3-level-hierarchy:
BigQuery has some naming restrictions: Project IDs must be 6-30 characters, contain letters, numbers, and hyphens and are globally unique, cannot be in use or have previously been used. Datasets and table names can contain up to 1024 characters, numbers and underscores.
We agree on a set of conventions for our BigQuery projects, datasets, and tables:
A BigQuery dataset equals one data product.
We use prefixes to structure the data models within a data product:
Further naming conventions:
The BigQuery project structure can be set up through a self-service web-app, when a new data product is created.
A dbt hook can be implemented that makes sure that all models use the defined prefixes.