Governed Data Lake | The Advanced Guide 2021

Governed Data Lake | The Advanced Guide 2021

Subscription

Table of content

Introduction to Data Lake

Data Lake is a secured centralized repository that stores data in its original form, ready for analysis. It uses the flat architecture to store data. It helps to break down data silos and analyze data to gain insights.

What is a Governed Data Lake?

A Governed data lake is a reliable and secured platform that contains clean data from unstructured and structured sources, easily accessible and protected.

Why Data lakes should be Governed?

It is recommended to use a Governed Data Lake rather than a normal data lake. A governed data lake enables data consumers to make data-driven decisions to business-ready data. As one’s data grows, it can be scaled and ingested in the data lake, irrespective of its type and structure. It governs the data present to be in a better position to meet increasingly strict regulations.


Click to explore about Data Catalog with Data Discovery


Governed Data Lake’s Building Blocks

Data exchange, Governance, Catalog, and Self-service access are the four building blocks of a Governed Data Lake.

  1. Data Exchange – This process works on extracting, analyzing, refining, transforming, and exchanging data between data lakes and IT systems. In doing so, it transports the data from data puddles to lakes.
  2. Governance – This is a governing process whose main aim is to provide security, privacy, and quality control of the data.
  3. Catalog – This process describes data present in the Data Lake. It shows the meaning of the data, how it’s classified, and the required governance.
  4. Self-service Access – This process provides access to the data lake on-demand. Analytics users can access raw data with the help of this process.

Architecture of a Governed Data Lake

The essential tiers in Data Lake Architecture are as follows:

  1. Ingestion Tier: This tier depicts the data sources. Here the data can be loaded into the data lake in batches or in real-time.
  2. Insights Tier: This represents the research side where insights from the system are used.
  3. HDFS: This tier is a landing zone for all resting data in the system.
  4. Distillation Tier: This tier converts data taken from the storage tire to structured data for more straightforward and better analysis.
  5. Processing Tier: This tier runs analytical algorithms, and user queries in real-time to generate structured data for analysis.
  6. Unified Operations Tier: This tier monitors system management and auditing of data.

Infrastructure of a Governed Data Lake

A Governed Data lake requires a robust data integration process to store data with meaningful metadata, containing a proper data lineage to retrieve data. If these attributes are lacking, then that Data Lake may turn into a Data Swamp.


Read more about Graph Database: Working | Advantages | Use-Cases


Building a Governed Data Lake

There are two options to build Governed data lake:

On-Premises

  1. Involves RDBMS and/or Big Data infrastructures
  2. Self-Managed with controlled/secure access
  3. Represents the SOURCE data

This option depicts Talend being installed and running locally in a data center while Snowflake runs on a hosted AWS platform. Execution servers run Talend jobs that connect to Snowflake and process data as needed.

Cloud

  1. Involves SaaS applications
  2. Hosted with user roles/permissions for access
  3. Cloud-2-Cloud, Cloud-2-Ground, or Ground-2-Cloud procedures are available
  4. Global usability guaranteed

Execution Servers run jobs in the cloud. These jobs can connect to any other data available in the Cloud ecosystem. This can present the best option when data directly ingest into Data Lake from files stored in the cloud and where users who want access to Talend are dispersed globally.

Advantages and Disadvantages of Building a Governed Data Lake

Advantages Disadvantages
Enables all data consumers in an organization to make smart, data-driven decisions. Data lakes are at risk of losing relevance and becoming data swamps if not properly governed.
As data grows, one can scale and ingest it in Data Lake regardless of its type and structure. Difficult to ensure data security as some data is dumped in the lake without proper oversight.
Saves time and resources on data preparation and data transformation. Storage and processing costs may increase as more data is added into the lake.
 Applies governance to the data in Data Lake. High cost, less space, increasing hardware setup demand.

Conclusion

Data that comes into any data lake must be appropriately cleaned, classified, and protected in controlled data feeds. It helps us populate and document the data with reliable information assets and metadata. If we do not govern how data is managed within our Data Lake, it can get polluted easily. This makes our Data Lake unusable and turns it into a Data Swamp.

It is recommended to use a modern cloud-based Data Warehouse as a service(DWaaS), which helps us address the Data Management challenges and scale our data easily, and Data Integration tools to build a Governed Data Lake. It is also recommended to use the Data Vault model, which helps to provide long-term historical storage of data from multiple sources. This helps us deal with issues such as auditing, tracing of data, loading speed. It helps us to trace where all the data in the database came from.

  1. Discover more about What is a Data Pipeline? Benefits and its Importance
  2. Read more about Composable Data Processing with a Case study

Fresh news directly to your mailbox