talk-data.com talk-data.com

A

Speaker

Ashish Saraswat

1

talks

Resident Solutions Architect Databricks

Ashish Saraswat is a Resident Solutions Architect at Databricks with over 11 years of experience in Software and Data Engineering. He is passionate about building scalable, reliable, and secure data architecture & platforms that drive business value and innovation. In his role within Databricks' Professional Services team, he guides strategic customer engagements, helping clients architect, design, and implement robust Data Intelligence Platforms using Databricks.

Bio from: Data + AI Summit 2025

Filter by Event / Source

Talks & appearances

1 activities · Newest first

Search activities →
Breaking Barriers: Building Custom Spark 4.0 Data Connectors with Python

Building a custom Spark data source connector once required Java or Scala expertise, making it complex and limiting. This left many proprietary data sources without public SDKs disconnected from Spark. Additionally, data sources with Python SDKs couldn't harness Spark’s distributed power. Spark 4.0 changes this with a new Python API for data source connectors, allowing developers to build fully functional connectors without Java or Scala. This unlocks new possibilities, from integrating proprietary systems to leveraging untapped data sources. Supporting both batch and streaming, this API makes data ingestion more flexible than ever. In this talk, we’ll demonstrate how to build a Spark connector for Excel using Python, showcasing schema inference, data reads/writes and streaming support. Whether you're a data engineer or Spark enthusiast, you’ll gain the knowledge to integrate Spark with any data source — entirely in Python.