I am currently working on the CRM Project. As part of the project, we are taking data from on-premises MySQL DB and some CSV files from the FTP site, and loading data into the staging area of Azure Data Lake Gen2 by using the Azure Data Factory ETL Tool in the Incremental Loading mechanism.
Once new and latest modified records are loaded into the staging area of Data-Lake gen2 by using the control table,
We are creating some external tables using PolyBase in Synapse Data Warehousing (DW). By using these external tables, we are implementing scd-1( slowly changing dimension method-1) to load data into dimensions and fact tables available in the Synapse data warehouse.
Once data is available in the Synapse Data Warehouse, we are connected from Power BI and consume data like dashboards and reports.
Subjects
-
SQL Intermediate
-
Databricks PySpark Intermediate
-
Azure Data Engineer - Data Factory Expert
-
Microsoft Fabric Intermediate
Experience
-
Sr software Engineer (Dec, 2022
–Present) at Wipro
Fee details
₹500–1,000/hour
(US$5.26–10.53/hour)