Built a scalable data pipeline for traffic data analytics
Ingested data using Azure Data Factory into Azure Data Lake Storage (ADLS)
Processed and transformed data in Azure Databricks
Implemented Unity Catalog for centralized governance and access control
Integrated CI/CD pipeline for automated deployment of Databricks notebooks
Azure Databricks Details
This project focuses on creating a scalable and secure data pipeline to ingest, store, process, and analyze traffic data using Azure Data Lake Storage (ADLS), Azure Databricks, and Unity Catalog. Data is ingested through Azure Data Factory, stored in ADLS, and processed in Databricks to transform raw traffic data into valuable insights. Unity Catalog ensures centralized governance and access control, while a CI/CD pipeline facilitates continuous integration and deployment of Databricks notebooks, enabling efficient management and monitoring of traffic data.
