talk-data.com talk-data.com

N

Speaker

Nayan Sharma

1

talks

Lead Data Engineer CKDelta

Nayan Sharma brings over 14 years of experience in Big Data and ETL pipeline development, with expertise in gathering business requirements, designing workflows in Agile environments, and building efficient ETL processes. Proficient in Databricks, AWS (EMR, EC2, S3), Python, Talend, Informatica, SQL, Spark, and R, he has delivered scalable solutions across industries such as healthcare, finance, consulting, and R&D. Holding a Master’s in Data Analytics, Nayan has played a key role in optimizing cloud-based data workflows and enabling actionable insights. His background includes working with global organizations, aligning technical solutions with complex business needs.

Bio from: Data + AI Summit 2025

Filter by Event / Source

Talks & appearances

1 activities · Newest first

Search activities →
Real-Time Analytics Pipeline for IoT Device Monitoring and Reporting

This session will show how we implemented a solution to support high-frequency data ingestion from smart meters. We implemented a robust API endpoint that interfaces directly with IoT devices. This API processes messages in real time from millions of distributed IoT devices and meters across the network. The architecture leverages cloud storage as a landing zone for the raw data, followed by a streaming pipeline built on Lakeflow Declarative Pipelines. This pipeline implements a multi-layer medallion architecture to progressively clean, transform and enrich the data. The pipeline operates continuously to maintain near real-time data freshness in our gold layer tables. These datasets connect directly to Databricks Dashboards, providing stakeholders with immediate insights into their operational metrics. This solution demonstrates how modern data architecture can handle high-volume IoT data streams while maintaining data quality and providing accessible real-time analytics for business users.