talk-data.com talk-data.com

P

Speaker

Padraic Kirrane

1

talks

Data Scientist CK Delta

Padraic Kirrane, Data Scientist at CK Delta, brings 12 years of expertise in data analytics across many sectors such as energy, logistics, utilities and finance. Known for his accessible approach to complex technical concepts, his sessions combine cutting-edge strategies with immediately applicable frameworks that bridge the gap between data science and business value. Padraic will showcase how his team built a cloud-based solution for high-frequency IoT data ingestion from millions of smart meters, implementing a multi-layer medallion architecture with real-time processing capabilities.

Bio from: Data + AI Summit 2025

Filter by Event / Source

Talks & appearances

1 activities · Newest first

Search activities →
Real-Time Analytics Pipeline for IoT Device Monitoring and Reporting

This session will show how we implemented a solution to support high-frequency data ingestion from smart meters. We implemented a robust API endpoint that interfaces directly with IoT devices. This API processes messages in real time from millions of distributed IoT devices and meters across the network. The architecture leverages cloud storage as a landing zone for the raw data, followed by a streaming pipeline built on Lakeflow Declarative Pipelines. This pipeline implements a multi-layer medallion architecture to progressively clean, transform and enrich the data. The pipeline operates continuously to maintain near real-time data freshness in our gold layer tables. These datasets connect directly to Databricks Dashboards, providing stakeholders with immediate insights into their operational metrics. This solution demonstrates how modern data architecture can handle high-volume IoT data streams while maintaining data quality and providing accessible real-time analytics for business users.