Blogs

End-to-End Data Pipeline on GCP with Airflow: A Social Media Case Study

In Part 1, we covered how to set up the GCP environment, create datasets, and prepare the schema for our social media project. Now in Part 2, we’ll focus on building an Apache Airflow DAG that automatically reads SQL files from Cloud Storage and executes them in BigQuery. This approach simplifies transformation logic and brings automation into the data pipeline.

The Future of Multi-Agent Collaboration in Generative AI Systems

In the evolving landscape of artificial intelligence, one concept that has gained significant momentum is agentic AI—AI systems capable of reasoning, planning, and acting autonomously.

Part 2: End-to-End Lakeflow Job Pipeline in Databricks — From Ingestion to DAG Scheduling

In this part, we dive into the full ETL pipeline using Databricks Lakeflow Jobs — starting from ingesting raw voter data, transforming it through silver and gold layers, and automating every task using DAG-based orchestration, scheduling, and alerts. This hands-on walkthrough shows how to build production-ready data workflows inside the Databricks Lakehouse Platform.

Part 1: Introduction to Lakeflow Jobs and ETL Workflow in Databricks.

Databricks has rebranded Jobs as Lakeflow Jobs to support modern, end-to-end workflow orchestration with DAG-based task flows, scheduling, retries, and monitoring — all in a unified interface.

Integrating DBT Cloud Semantic Layer with Power BI

The integration of dbt Cloud’s Semantic Layer with Power BI creates a unified environment were trusted business metrics flow directly into interactive dashboards. Instead of redefining logic in every report, dbt centralizes models, sources, and metrics, then exposes them through a connector that Power BI can consume in real time.